A call for identifying your custom changes

Reviewing code changes is difficult at best, but when reviewing a "basic_change" in Magik, it would be very helpful if the changes were very clearly bracketed with private comments.

For example, you might have an #EMQ like this...

_method some_class.some_method()
## some_method() : some_return_value
##
## does something crazy

_global a

(a.b[c], a.b[d]) << (a.b[d], a.b[c])
_endmethod
$

... which you might want to refactor to be more robust. It would be really, really helpful to future code reviewers if you very explicitly indicated which lines of code you changed. Like this...

_method some_class.some_method()
## some_method() : some_return_value
##
## does something crazy

_global a

# BEGIN CHANGES
#(a.b[c], a.b[d]) << (a.b[d], a.b[c])
_try _with cond
(a.b[c], a.b[d]) << (a.b[d], a.b[c])
_when error
write("!!! EMQ !!!",cond.report_string)
_endtry
# END CHANGES
_endmethod
$

You can see that I actually commented out the original line and then put a copy of the line back in the _try block. This makes it very easy for a reviewer (and text diff tools) to see what has changed. I would not recommend using this approach for your own product code. Source Code control systems can identify these changes for you. What I am more concerned about is cases where you have redefined a core Smallworld method to address some business requirement. Once compiled into the image, it is very difficult to find the original version of the method, so the more that you can document as a customizer, the better it will be for future code readers.

Igor over at MagikEmacs pointed out to me that Alt-c in his extended version of Emacs does exactly this "changed code block" highlighting for you. You can see an example in his Screen-cast #3 at 3:50 where he uses exactly this technique.

Another Mapping Blog

I am expanding my mapping experiences beyond Smallworld. Particularly, I am getting more involved with OpenStreetMap. In order to spare those of you interested mainly in Smallworld the details of non-Smallworld blog posts, I have created a new blog at boulderalfmaps.blogspot.com

SEPM Launches a YouTube Channel

Robert at SEPM took my suggesting in a recent post seriously enough to create a SEPM YouTube Channel. The first video is up and shows some very cool stuff that he has been working on.

If you are considering creating your own channel, please consider tagging the videos with "swgis". On Twitter you can use the #swgis hashtag.

Smallworld on the Social Web

I have become very interested in the use of social media to provide a place to share ideas about Smallworld. Obviously the social web is a place where companies and individuals can market their products and services. But I have begun to see the social web as an extension of the informal technical meetings I have had with colleagues over the years where new ideas have been generated and old ideas refined.

I would recommend that you watch the TED Talk by Steven Johnson: Where good ideas come from. He gives some great examples of how networks of people get together in chaotic situations to share ideas and out of those situations arise great new ideas. One of the very interesting examples he gives at the end is how some researchers trying to hear the telemetry signals from Sputnik triggered a series of events that today allows us all to use our mobile phones as navigation devices.

In that same stream of thought, I am beginning to see the Social Web as a place that the Smallworld community can use to foster new ideas and innovation.

Some in the Smallworld community have moved onto Twitter. That is a great tool for updating others on the status of ideas and projects (remember to tag your tweets with #swgis).

Others are carrying on Smallworld-related conversations in Linkedin "GE Smallworld" group.

There is the well-used Yahoo group sw-gis that many of us use. It is a great forum for answering questions to problems.

But what if we don't know what the problem definition is? I have recently begun to see the utility of YouTube in fostering this idea sharing. Another TED Talk by Chris Anderson: How YouTube is driving innovation got me to start thinking about how I could use video to share ideas and add them to the chaotic mix that stimulates innovation. I have started a channel at http://www.youtube.com/user/boulderalf that shares my Smallworld-related ideas and experiences.

I'm curious to know if there are other Smallworld-related YouTube videos/channels out there. Please let me know in a comment on this post. I know of only one other one: http://www.youtube.com/user/cliffhangersolutions

When you post to YouTube, remember to tag it with "swgis". It turns out that the "Smallworld" tag is more likely to take you to Disney-related content that it is to a GIS :(

Magikfun

I wanted to announce a new Smallworld-related blog at http://magikfun.blogspot.com. Welcome, Bhimesh, to the relatively new space of blogging about Magik. One of his first posts is about creating a 3D pie graph in Excel using Magik (http://magikfun.blogspot.com/2009/11/3d-pie-diagram-in-excel-from-magik.html). Looks really cool and the code is not that complex.

Another Magik Blog

Pedro Miranda has started a new blog...

dedicated to Smallworld magik, it will be mainly a place to post assorted bits of useful code, targeting developers....and information regarding SW.

You can find it at http://magikbites.blogspot.com/

Announcing "Programar en Magik de SmallWorld" blog

For Magik practitioners wishing to read about Smallworld Magik in Spanish, Joan in Barcelona has created Programar en Magik de SmallWorld.

Congratulations Joan!

What is RSS?

I noticed today that about 45% of subscribers to this blog use the e-mail subscription option instead of RSS. While e-mail is definitely a good way to be notified of updates to the blog I would suggest that you learn about RSS and how it might lighten your inbox load.

The best explanation I have found so far is a 3 minute video called RSS in Plain English (the website also has links to information in other languages).

One advantage of RSS over e-mail notification is that your inbox is not filled with update notifications at inconvenient times. Using RSS you go to your RSS page (or reader application) when you want to.

SECO Energy gives customers access to outage data

There is a good article in the September issue of Transmission & Distribution World titled Customers Access Online Outage Data. It was written by Barry Owens of SECO Energy. In case you missed it, Barry also gave the featured Customer Presentation at the recent Smallworld 2007 Americas Users Conference (presentations are here).

This is a project that I had heard about before and one of the things I had wondered was: "in a power outage situation, how will affected customers have the power to run their computers to check their outage status online?". Barry answers that question (and many others) in a well written presentation of their project.

Congratulations also to Peter Manskopf at GE for leading this successful project.

Where did Geography go?

One thing I regret about my undergraduate studies is that I took no geography courses. I only discovered GIS after I graduated. I enjoy the technical nature of GIS very much but have always been intrigued by the geographic questions that the GIS technologies strive to address.

So you might ask: "if you are intrigued by Geography and GIS why are you working with Smallworld?". Good question... GE seems to have made a decision not to market the Smallworld Core Technology as a GIS but rather to present it as a network management suite of tools. I know that Smallworld can be used for GIS analysis but the product development has not emphasized that at all. (I would be interested to read how you are using the Smallworld tools to understand non-utility geographic issues.)

So because I am still interested in GIS and Geography, Jerome E. Dobson's article "Bring Back Geography!" resonated with me. If you are at all interested in Geography (or wonder what it is) this article is definitely worth reading. (Yes, it is an ESRI site. But if you are interested in Geography and GIS in general, you go to where the information is)

Dobson writes of the role of Geography largely in the United States of America. He writes about how during the 1st and 2nd World Wars, geographers were an important part of preparing for peace after the war. And then he points out that Geography was purged from the major USA Universities after World War II. His call is for geographers to reassert themselves into the public consciousness so that they can make important contributions for the common good.

Because of my daily work with GIS technologies, I always fancied myself a bit of a geographer. Dobson writing about GIS says...

In contrast, advances in GIS alone are likely to cast us as clerks handling data for the ecologists, political scientists, economists, and other current leaders in these topics

I do actually seem myself as a bit of a clerk (or consultant to clerks) that manages data for others. Not that being a clerk is a bad thing... it puts food on my family's table and is something that is also technically challenging and rewarding. But this article has got me thinking about how I can take the next step beyond technology and start learning more about Geography to understand my world better and then also contribute to its common good.

progress counters and dialogs

I recently had a colleague ask me how to pop up a progress bar dialog to indicate to a user the percentage complete of a process. I did not know how to answer that at first... Smallworld has historically not used progress bar dialogs as much as some users would expect. So I will present two options for showing progress status.

Text-based process

If you have some kind of text based process use progress_counter...

_block
 _local some_collection << rope.new(20)

 # create a progress_counter class that reports every 5
 # increments.  Because we know some_collection.size we can also
 # estimate completion time
 _local pc << progress_counter.new_on_count(5,some_collection.size)
 
 _for idx,an_el _over some_collection.fast_keys_and_elements()
 _loop
  # the write() and sleep() statements are here to make a simple
  # example...
  write(an_el)
  _thisthread.sleep(1000)

  # ... but the :next() method is important to let the
  # progress_counter (aka PC) know that you have completed one
  # iteration of the loop.
  pc.next()
 _endloop
_endblock
$
unset
unset
unset
unset
unset
Done 5 (15 left) time taken  5 seconds 297 milliseconds  (ETC. 15 seconds 891 milliseconds ) average rate 0.9439305267
unset
unset
unset
unset
unset
Done 10 (10 left) time taken 10 seconds 625 milliseconds  (ETC. 10 seconds 625 milliseconds ) average rate 0.9411844826
unset
unset
unset
unset
unset
Done 15 (5 left) time taken 15 seconds 812 milliseconds  (ETC.  5 seconds 270 milliseconds ) average rate 0.9487724325
unset
unset
unset
unset
unset
Done 20 (0 left) time taken 21 seconds 15 milliseconds  (ETC.  0 milliseconds ) average rate 0.9562876356

... you can also ask the progress_counter to report at a given time interval...

_block
  _local some_collection << rope.new(20)

  # create a progress_counter class that reports every 2
  # seconds. Because we know some_collection.size we can also
  # estimate completion time
  _local pc << progress_counter.new_on_time(3,some_collection.size)
 
  _for idx,an_el _over some_collection.fast_keys_and_elements()
  _loop
   # the write() and sleep() statements are here to make a simple
   # example...
   write(an_el)
   _thisthread.sleep(1000)
  
   # ... but the :next() method is important to let the
   # progress_counter (aka PC) know that you have completed one
   # iteration of the loop.
   pc.next()
  _endloop
_endblock
$
unset
unset
unset
Done 3 (17 left) time taken  3 seconds 125 milliseconds  (ETC. 17 seconds 708 milliseconds ) average rate 0.9600000000
unset
unset
unset
Done 6 (14 left) time taken  6 seconds 250 milliseconds  (ETC. 14 seconds 583 milliseconds ) average rate 0.9600000000
unset
unset
unset
Done 9 (11 left) time taken  9 seconds 359 milliseconds  (ETC. 11 seconds 439 milliseconds ) average rate 0.9616468318
unset
unset
unset
Done 12 (8 left) time taken 12 seconds 453 milliseconds  (ETC.  8 seconds 291 milliseconds ) average rate 0.9648530373
unset
unset
unset
Done 15 (5 left) time taken 15 seconds 578 milliseconds  (ETC.  5 seconds 199 milliseconds ) average rate 0.9616468318
unset
unset
unset
Done 18 (2 left) time taken 18 seconds 703 milliseconds  (ETC.  2 seconds 81 milliseconds ) average rate 0.9612351238
unset
unset
Done 20 (0 left) time taken 20 seconds 765 milliseconds  (ETC.  0 milliseconds ) average rate 0.9612351238

If you want to use a more user-friendly indicator, use the progress_indicator_dialog. It is loaded into the core 4.1 image and you can find details about it in the header of product/modules/sw_core/progress_manager/source/progress_indicator_dialog.magik

_block
 _local q << progress_indicator_dialog.new ( "Generating" )
 q.interrupt_handler << handler.new ( _unset, _unset )
 q.info_string << "The current series will contain approx 10 frames."
 q.bar_label << "Generating frames..."
 q.image << { :help, _unset }
 q.activate ()

 q.max_count << 10

 _for n _over range(1,10,1)
 _loop
  _thisthread.sleep(1000)

  q.progress_changed(n)
 _endloop
_endblock
$

And there you have it... easy ways to let your users know the progress of a process.

Smallworld 2007 Americas Users Conference dates

The Smallworld 2007 Americas Users Conference dates have been set for September 9-12. You can read the announcement here.

getting a list of all users currently connected to Smallworld

I learned a new aspect of Windows command-line scripting the other day that I thought might be worth sharing here.

The requirement was to get a list of all users connected to a particular swmfs server and notify them that the server was about to go down.

I discovered that the FOR command is just what the doctor ordered for this task. The basic form is as follows...

FOR /F "skip=5 delims=@" %%i IN ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds') DO net send %%i Smallworld server going down in 2 minutes.

Where...

• FOR /F ... IN ...
• Command key words that tell the system that you want to iteratively process the results of some file or command
• ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds')
• the command that will have each line processed individually for some information. This particular command provides a list of all the user@computername that are connected to message.ds in the specified folder on %COMPUTERNAME%
• %%i
• the internal variable representing each line of the file/command
• "skip=5 delims=@"
• instructions to the command to skip the first 5 lines of swmfs_list and then also split the processed lines by the @ character.
• DO
• a command key word that indicates that what follows next is the command to be processed on each line of iterator
• net send %%i Smallworld server going down in 2 minutes.
• the NET SEND command is used to send the message "Smallworld server going down in 2 minutes" to the user specified in variable %%i. Remember that the variable %%i represents the first element of each line after it has been split by the @ character.
  

Special note about the variables. If you are running this script from within a BAT file, you need to reference the variable with two percent signs: eg., %%i. If you want to test this functionality at the command line, you need to reference the variable with only one percent sign: eg., %i.

I hope you find this useful.

getting a list of all users currently connected to Smallworld

I learned a new aspect of Windows command-line scripting the other day that I thought might be worth sharing here.

The requirement was to get a list of all users connected to a particular swmfs server and notify them that the server was about to go down.

I discovered that the FOR command is just what the doctor ordered for this task. The basic form is as follows...

FOR /F "skip=5 delims=@" %%i IN ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds') DO net send %%i Smallworld server going down in 2 minutes.

Where...

• FOR /F ... IN ...
• Command key words that tell the system that you want to iteratively process the results of some file or command
  
• ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds')
• the command that will have each line processed individually for some information. This particular command provides a list of all the user@computername that are connected to message.ds in the specified folder on %COMPUTERNAME%
  
• %%i
• the internal variable representing each line of the file/command
  
• "skip=5 delims=@"
• instructions to the command to skip the first 5 lines of swmfs_list and then also split the processed lines by the @ character.
  
• DO
• a command key word that indicates that what follows next is the command to be processed on each line of iterator
  
• net send %%i Smallworld server going down in 2 minutes.
• the NET SEND command is used to send the message "Smallworld server going down in 2 minutes" to the user specified in variable %%i. Remember that the variable %%i represents the first element of each line after it has been split by the @ character.
  

I hope you find this useful.

talk to me GE

If you were able to attend GITA 30, you would have likely heard the announcement "GE Energy to Introduce New Oracle-Based Product Portfolio for Electric Disturbition Utilities" (be sure to read the comments for this article). It is also likely that if you had any questions and concerns at that meeting that there would have been plenty of GE representatives there to answer your questions and assuage your concerns. It sounds to me that GE had a consistent message that they were able to get out to the attendees.

If you were not able to attend GITA 30 it is likely that you now have more questions than answers about this strategic announcement by GE. I was unable to attend GITA so I count myself in that group of under informed Smallworld users/partners. If this year's attendance was any indication, then I imagine that many others in the Smallworld community were also not able to make use of the direct interaction with GE representatives to have their questions/concerns heard. And GITA attendance is largely from the Western Hemisphere. European, Asian and African Smallworld community members would have had even less chance to get their questions answered.

I suspect that this year's Smallworld Users Conferences will be dominated by presentations on this new architecture (and hopefully demonstrations), but it seems like a long time to wait until September/October to have our questions answered.

Maybe GE would consider some of the following communication options:

• press release
• been there done that
• send a more detailed note out to all Smallworld users/partners
• Smallworld Helpdesk has a fairly efficient mechanism in the SupportCentral site to send out blanket e-mails to all users. Use this feature to give a more-detailed-than-the-press-release announcement to the Smallworld community
• Online forum
• I understand that there might be concerns about making information available on an uncontrolled list, so extend the Global Customer Forums (see On-line Smallworld resources (closed and open)) to include a forum where customers/partners are invited to post any questions they have about the new product.
• Commit to having GE management/technical staff answer each post in a timely manner.
  
• FAQ
• As the questions start coming in to GE as phone calls and e-mails, compile a list of Frequently Asked Questions and post them in the forum.
• online demonstrations
• provide WebEx demonstrations of a more technical nature where we can see how the new tools are supposed to work.

Most importantly, please talk to us sooner than later. It would be helpful even to send out a note (another press release, maybe) that indicates dates and types of information will be made available. For example "GE will have a new customer forum opened by April 1", etc...

As always, I welcome readers' comments. In particular, I am interested in what kind of communication you would find helpful to address your questions or concerns about this new "non-GIS" spatial product announcement by GE.

Recent Smallworld Highlights

I found some interesting links on the GE Energy Geospatial Asset Management page.

• An official view from GE on the outlook of the GIS industry in 2007.
• Cambridge office holding an Open Day. I am curious about what an "Open Day" is. The flyer does not give too many details. Anyone in the UK or Ireland attending and willing to report on it afterwards?
• I'm always interested in hearing about large companies that see Smallworld as the best solution for them. Find out more by reading a GE Energy executive interview with Dr.-Ing. Andreas Bordonetti, Head of Power Distribution, Azienda Energetica S.p.A.

Cutting out the GIS Middleman

As the days roll on from the original GE announcement to move to a 100% Oracle/Java solution, more details are slowly trickling in.

Directions Magazine has an article (Cutting Out the GIS Middleman: GE Energy Cements Relationship with Oracle to Build Enterprise Applications) that sheds some more light on GE's direction.

It seems to me that GE is trying to leap-frog over some of its GIS competitors who have used Oracle in the past but used it in a proprietary fashion.

[Updated: 11:46 am 15March07... I did not know that Oracle ships with landbase data. Have a look at http://www.oracle.com/technology/products/spatial/pdf/navteq_faq.pdf]

North American Daylight Saving Time

(Thank you to everyone that posted comments in response to the announcement that GE will be introducing a new version of Smallworld on Oracle/Java.)

I'm not sure about Mexico, but Canada and USA switched to Daylight Saving Time on March 11. If your Smallworld system times are off by an hour, please read "USA/Canada Daylight Saving Time is 4 weeks longer this year. Are you ready?"

You can tell if you have the appropriate Smallworld patch loaded by executing the following command at the Magik prompt.

MagikSF> date_time_now()
$

If the date and time are different than you expect, it is likely that your patch has not been loaded.

An e-mail from the Smallworld helpdesk also mentioned that the most recent time zone patches include adjustments for Western Australia.

goodbye VMDS/Magik, hello Oracle/Java?

Well, not quite.

Earlier this week, GE issued a press release "GE Energy to Introduce New Oracle-Based Product Portfolio for Electric Disturbition Utilities". I had heard rumblings of such an announcement in the days leading up to the release and was very anxious to understand what GE Energy would be announcing.

At first glance, this announcement seems to suggest another significant paradigm shift in the Smallworld application. The most recent significant paradigm shift was the move to the Smallworld Application Framework (SWAF). Some customers are still preparing for the SWAF upgrade. To be faced with an announcement like this (Smallworld on Oracle/Java) might come as a bit of a shock. But if you read the press release carefully, you will see that this announcement is geared to small- and mid-sized electric utility companies.

If you are a larger utility company that has invested considerably in VMDS and Magik technology, Bob Gilligan states that this new product line will complement the current Smallworld offerings. I find it significant that Mr. Gilligan used "complement" and not "replace".

Clearly GE Energy needs to grow more license sales and it seems to me that offering Smallworld in an Oracle/Java environment will make the sale more appealing to smaller users who do not have the resources to commit to learning another database and programming language. Having said that, I suspect that as some of the mid-sized utilities start asking the Oracle/Java application to do greater and greater things (especially around long transaction management and customising the data model), it might be a more natural step to direct them to VMDS/Magik. I think VMDS/Magik will go away at GE Energy's peril. I am of the opinion that the current VMDS/Magik package provides better solutions for some of the Smallworld user's business requirements than Oracle/Java is capable of.

GE Energy may spend more energy developing the Oracle/Java side of things because that is where the revenue growth may be, but I would strongly urge them to continue to support VMDS/Magik either directly or via some Open Source framework so that the Smallworld user community can service it.

I have included some highlights (with my comments) of the press release. I have formatted direct quotes from the press release in Georgia font. All other comments are mine.

The highlights of the announcement include:

• GE Energy is collaborating with Oracle to expand GE’s geospatial product portfolio by developing a network design and maintenance application based on Oracle® Database 10g, Oracle Spatial 10g and Oracle Fusion Middleware.
• This solution is designed specifically to meet the needs of the small- to mid-size North American electric distribution market.
• One can take a half-full or half-empty view of this press release. The statement above makes me lean towards the half-full view. It seems that GE Energy is trying to increase its market share and trying to offer something that the smaller utilities are asking for. My half-full view continues when I realize that there are opportunities in this approach for current Smallworld customers. Let's say your large utility acquires a small- to mid-size utility. Would you rather integrate your GIS system with another Smallworld platform or with an entirely different GIS vendor? I think that any move that keeps Smallworld viable in the market place and does not take away the product that already works for current customers can only be good for the entire Smallworld user community.
  
• GE Energy has been working with selected customers in the development of the application and expects beta testing to commence later this year.
• I'm looking forward to seeing how beta testing goes.
  
• As a result, companies increasingly seek products to meet their needs with minimum customization, ease of install/upgrade and low maintenance.
• said Bob Gilligan, general manager of GE Energy’s transmission and distribution business. “Our new approach will complement our current offerings and result in products that leverage Oracle’s spatial software."
• GE Energy’s Smallworld Core Spatial Technology* remains a key technology for large utilities and telecommunications customers with complex solution needs. No existing products will be retired as a result of this new offering. The addition of applications based on Oracle Spatial will provide more options for companies with specific IT requirements.
• This is an important statement that no existing products will be retired. I think it is very important that GE Energy repeat that statement until it becomes very clear to all of its current customers. I have frozen this statement in time as a PDF file. In my view this is a very import statement and I will do what I can to hold GE Energy to it.
  

What is not included in the announcement:

• VMDS/Magik will be retired.
• Actually, the press release states the current CST will continue to exist for customers with complex networks.
• GE Energy’s Smallworld Core Spatial Technology* remains a key technology for large utilities and telecommunications customers with complex solution needs. No existing products will be retired as a result of this new offering.

I would be interested in your comments.

Cheers,

Alfred

writing Smallworld symbols/styles to JPG images

Recently a question appeared on sw-gis asking how to output your Smallworld styles to some external format.

If you download the SourceForge MagikComponents (http://sourceforge.net/projects/magikcmpnts) [MagikComponents has been moved to https://www.assembla.com/code/magik-components/subversion/nodes/mclib/trunk ] you should see a number of modules that provide results similar to what you are asking or provide examples of how you might render styles to JPG files.

• modules\mclib_style_images provides code to create jpg files from the various style records.
• modules\mclib_case_html_reporter provides code that creates an HTML output for your CASE tool. Part of this code (see case_html_report_plugin.output_style_info_for() ) creates jpeg files of the various geometry field styles.