Smallworld Newsletter

A newsletter arrived in my inbox today that I had forgotten that I had subscribed to. It is called "Smallworld Applications: Technically Speaking". According to the newsletter footer

You are receiving this electronic newsletter as a customer of the General Electric Company.

So this post might not be news to many of you, but it may be of interest to people not on the regular Smallworld mailing list.

The newsletter has one page and then links to various GE Smallworld marketing material. The first page of the newsletter typically has a detailed customer story that is worth reading. All issues are kept in an archive.

Where is that error coming from? (an update)

Roger Neal posted a helpful comment to my original post about how to force a traceback on User Errors for debugging purposes.

If you have CST v4.1 you can use the Development Tools application option to "Show User Tracebacks". If find this Development Tools toolkit very helpful. (For day-to-day code profiling, I use the xprf_dialog class alot. This is probably another useful topic to post about later.)

If you are already at v4.0, you could upgrade to 4.1 to get these development tools or you might try loading the product\modules\sw_dev_tools product from the 4.1 CD into your 4.0 image.

If you are at version 4.0, you can do this from Magik...

MagikSF> model.enable_user_error_tracebacks? << _true
$

The method :enable_user_error_tracebacks? was only introduced at version 4.0. If you have an earlier version you will still need to do something like the hack I described in my original post.

Copying Data into Smallworld Using SOMs

I recently received a question from a colleague concerning a Data Migration Module. I suspect that other users might have the same question so I am posting the question and answer here.

Question:

I'm curious, have you ever seen or heard of Data Migration Module from GE? I know Smallworld Spatial Object Managers provide the capability to 'plug-in' data readers (SOMs) for a variety of external data formats. These plug-ins enable data from the external sources to be treated as real world objects in the same way as data held internally to the Smallworld Core virtual database but beyond that...

Through SOMs, using the external data as source and with certain mapping, migrate external data objects as PNI objects in SW. I'm certain these are customisable but is there any ready made product out there?

Answer:

I wonder if you are referring to the sas_data_copier module that ships as an example with the SOM product. In SOMs 4.0 it can be found at ...\SOMs\modules\sas_core\magik_gui\source\examples\sas_data_copier

It is not a supported product but it provides an example of how you can copy data using SOMs. The module also has some simple help documentation.

I think that the Smallworld team in Denver (USA) had done some work to make the sas_data_copier more user friendly. You may consider contacting your Smallworld account representative to see if that is something that they can make available to you.

Do you happen to have FME (see a related post here) at your site? If you do, you might consider using that as a migration tool.

FME as a Spatial Intelligence tool for Smallworld

Safe Software's FME suite of tools are usually thought of in terms of translating data in or out of a GIS database or for viewing foreign data using some kind of vendor-specific FME plugin. ESRI has one, Smallworld has one, and I imagine many other vendors do to.

But have you ever considered using FME as a Business Geographics or Spatial Intelligence tool?

FME has a very rich suite of geospatial query and manipulation tools. Its FME Workbench graphical user interface is very easy to use and extremely well documented.

I recently worked on a task where a customer wanted to know how many valves they had in each grid of a matrix covering their service area. The idea was that field staff supervisors wanted to know the number of valves per area so they could know how to assign their staff to the various areas and also to check up on the quality of the work once the valve inspections were complete. The supervisors wanted this information in a spreadsheet.

So I went ahead and wrote some Magik code to create a CSV file based on various spatial predicates.

After I had delivered the code to the customer I started thinking about whether this could have been achieved another way. Specifically, I wondered if I could use the FME Workbench tool to create the same results.

Well, the answer is YES! It took me about an hour to create the Workbench file but it finally created the same CSV output report as the Magik approach did.

I think this opens up many possibilities for an organization's Business Analysts that want to use spatial information. Granted, they still need to know something about spatial analysis, but...

... think about the analysts in your organization...

To query some geographic data from your database, would they prefer to learn and write code like this ...




... or would they prefer to learn a Graphical User Interface that cleanly documents their query like this...



And if you are using FME anyways, then you are only minutes away from creating reports like...


(the height of each grid indicates the relative number of valves inside that grid)

I discovered that the extract code in Magik created the report faster than using FME, but that is mainly due to how the Smallworld-FME interface is architected. I imagine Smallworld could improve the interface to speed things up. Spatial Business Systems have also created their own Smallworld-FME interface. I have not tested it and would be interested to hear from anyone who has compared the two interfaces.

In summary, I think that the FME suite is a very useful tool that an organization can pass along to business analysts to help them query data from Smallworld. And because they will be using FME, they can build queries that pull facility data from Smallworld, real estate data from Shape, customer data from Oracle and create a nice looking map for the managers using Google KML.

FME is typically thought of for data translation, but I suggest that it also opens up your data for easier access for query, analysis and reporting.

Smallworld version 4.1 Feature Review

I recently had the opportunity to install and review the new Smallworld Core Spatial Technology version 4.1 release. The documentation refers to many improvements and I wanted to highlight some of them here.

Incidentally, I created an internal briefing document for my coworkers that "summarizes" the changes in 4.1. I say "summary" because it is about 17 pages long but is itself only a snapshot of the very descriptive documentation that accompanies the 4.1 release. If you would like a copy of this summary (it is not a polished document, but contains good details about new features) please feel free to contact me at one of the following e-mail addresses and request the "Smallworld CST 41 Upgrade Features Review" PDF file.


The following is a list of feature enhancements in version 4.1 for Users, Administrators and Application Developers. This list is not exhaustive, but it does include those features that I consider especially noteworthy.

Improvements that Users will see:

• Enhancements to layouts and layout facilities including:
• Customized legends
• better control over the exact position and size of layout elements on the page.
• New text formatting properties
• Linked view ports
  
• Performance improvements to the Smallworld Datastore.
• Improvements to Short Transaction Manager performance.
• Smallworld Explorer performance improved with large data volumes.
• Plotter driver enhancements
  
• New user interface facilities for an interactive application.
• A new TINs editor and Constructions and Reference Origin plugins are provided.
  
• Field editors for raster geometry are updated to be consistent with other geometry field editors.
• Keyboard interaction with object editors is improved
• System bookmarks are introduced, available to all users of an application. In addition, bookmarks are assigned to bookmark groups which can be shared by different applications.

Improvements that Administrators will see:

• New plotting system configuration tools
• The Set Up Smallworld Printer Wizard provides an easy-to-use GUI for configuring new plot filters.
• The Smallworld Printer Properties dialog box can be used to modify plot filters.
• The Print Test Plots dialog box can be used to test newly-configured plot filters.
• Oracle
• Extended and updated Smallworld on Oracle® facilities for use with Oracle 10g data types and Oracle Real Application Clusters.
• Extension of the Smallworld Authorisation system to use LDAP user authentication.
• Changes to the Smallworld Datastore Server.
• These improve the consistency of facilities on UNIX® and Windows® servers and allow increased tolerance to network failure.
  
• New tools for administrators and configurers
  
• The Administration application is extended to give access to additional facilities.
• A remote_cli exemplar is added, to allow one Smallworld session to run a command line interface on a second session. This is intended for development and debugging only.
• The new utilities swmfs_monitor and swmfs_lock_monitor are added to allow improved monitoring by a Smallworld Datastore Server (swmfs).
• new ds_transfer module version and Datastore Transfer dialog
• The key features introduced by ds_transfer version 3 are:
• Graphical user interface for starting and monitoring an operation.
• Online compression. This enables incremental transfers to be made after an initial one, and as these simply transfer changes they should be much faster, minimising the window when the dataset is unavailable for use.
• Progress reporting, and more controllable logging information.
• Tests show that ds_transfer version 3 is generally slightly faster than version 2, but not significantly so. However, online compression should dramatically reduce the downtime required for a transfer.

Improvements that Application Developers will see:

• simple areas can now contain holes. This means that simple areas are now more compatible with other area geometry.
• 64-bit UVA values
• New tools and enhanced facilities for application developers who need direct access to the Magik prompt of a Smallworld session.
  
• A remote_cli exemplar is added, to allow one Smallworld session to run a command line interface on a second session.
  
• Development Tools including:
• object inspector
• profiler
• system monitor
• and many more useful tools.
• Keyboard shortcuts for map actions can now be configured. [my note: this should make it easier to revert zoom/pan actions back to what users were comfortable with in earlier versions]
  
• progress counter enhancements.
• The Rate and Estimated Time to Completion (ETC) are now calculated over a number of recent samples and exclude outliers. This should give a much more accurate ETC.
  
• Incrementing the counter manually, instead of doing next(), is now treated correctly.
  
• When processing records in batches using get_upto_n(), report output is at more consistent intervals.
• There is now a reset() interface, so a counter can be set back to 0 after a potential wait while other things were done while it was created.
• total slot is now writable, so a progress counter can be supplied to a piece of code which can populate the total itself.
• workspace layouts
• This functionality has been used by Physical Network Inventory (PNI) for some time. Now it can be used in other applications as well.
• A remote version of the sw_databus class called sw_remote_databus is implemented. This class allows multiple computers to talk to each other using protocols based on the databus communication channels. with the limitation that only strings can be passed.
• [my note: this remote databus behaviour could spawn a whole new set of distributed GIS applications in Smallworld. It reminds me of the GeoRSS protocol]
  

As mentioned before, this list is not exhaustive but it does give you a sense of the enhancements that caught my attention. If you want a more detailed summary, please send me a note and I will forward it to you. And if you want a really detailed treatment of the enhancements, you can look at the Documentation that accompanies the SW 4.1 CD. The documentation is well laid out and very informative.

Pictometry on Smallworld

A press release today by Pictometry announced a collaborative effort with AML Solutions Group that enables customers who use Smallworld 4 Product Suites from GE Energy to use Pictometry software directly from Smallworld 4 Products.

What a great idea! I checked the AML web site and there was no mention yet of this collaborative effort. I am looking forward to seeing a demonstration (or at least some screen snapshots) in the near future. I will check with AML to see what they can "show and tell".

On-line Smallworld resources (closed and open)

Have you ever wondered where you could find help on-line for your Smallworld questions? Actual Smallworld documents are scarce on the web, but there are some other online resources you can use to get your questions answered.

GE Smallworld
If you are a paying customer you may have access to on-line help desk support. You will need to contact your local Smallworld support organization to get registered for this. You can use the help desk to report bugs and request enhancements but also to request Smallworld related answers for questions that you cannot find anywhere else.

Recently, Smallworld announced two new Global Customer Forums (you will also need to be registered with GE ):

• The Geospatial Forum consists of topics related to:
• Gas
• Water
• Electric
• Communications
  
• Developers
• The PowerOn Forum intends for PowerOn customers to discuss topics related to that product.

[I had wanted to post the URLs to these Forums, but I just read the Terms of Use for these sites (that I agreed to by accessing the Forums) and discovered that I cannot, without GE approval, give you links any deeper than http://www.GEPower.com.]

I wonder if these new "paying customers only" forums by GE will spell the demise of such well-used forums as the Yahoo Groups' sw-gis? I doubt that they will for a number of reasons:

• inertia. Developers are used to the sw-gis community.
  
• legacy. The sw-gis group has a significant archive of technical questions/answers that have not been ported to the GE forums.
• ease of use. I find it easier to use Yahoo's groups interface than the SupportCentral interface used by the GE forums.
• fresh air. I suspect that many people prefer the "fresh air" of open discussion. I think this can be accomplished more comfortable in an open community of ideas rather than a closed environment that may be perceived as too closely tied to the product vendor.
  

My personal preference is to see the open groups continue and thrive. They introduce new users or potential users to the community (how will they find out about Smallworld if they need to be a paying customer first?). GE created the secure forums in response to customer requests for such a venue, but I think I need some enlightenment as to what purpose that serves that cannot be accomplished with open groups. Your comments would be greatly appreciated.

Enough of my editorializing. Now on to the open Smallworld communities.

Yahoo Groups
There have been a number of active Smallworld groups on Yahoo Groups and many not so active. Here is a list (sorted by registered members) of the ones that I could find. In my opinion, the first few are helpful and the rest are either very language/region-specific or have too little activity. You can find these groups by browsing to http://groups.yahoo.com/group/.

• sw-gis
• The sw-gis list provides a forum for developers and system administrators working with Smallworld GIS. This list addresses technical issues concerning Smallworld GIS, such as Magik programming, CASE tool, interfacing to other systems, but also system and database administration.
• 2356 members
• sw-gis-announce
• This list is for general announcements related to Smallworld GIS or other Smallworld related products or applications. Acceptable announcements include: New Smallworld-related products or new releases of existing products, Smallworld related job openings, New Smallworld partner companies or new offices of existing Smallworld partner companies
• 624 users
• sw-gis-users
• This list provides a forum for users of Smallworld GIS and other Smallworld related applications to share information and knowledge about the use and configuration of those products and applications.
• 312 users
• swgis-indonesia
• GIS / SIG Indonesia:FORUM Diskusi pengembang aplikasi Geographic Information System (GIS) di Indonesia. Untuk pengguna software GIS :GE SMALLWORLD, ESRI ARCGIS, ARCVIEW, ARCSDE, , MAPINFO, GEOMEDIA, AUTOCAD , SICAD, ORACLE SPATIAL, CADCORP, Bentley Microstation atau yang lainnya. Software Webgis :SIAS (Smallworld Internet Application Server), ARCIMS, MAPGUIDE, MAPSERVER dan yang lainnya Lowongan Kerja GIS
• 191 members
• mal-sw-gis
• For intelligence users of Smallworld Systems
• 98 members
• SmallWorld-GIS
• El pequeño mundo de los sistemas de información geográfica, tanto bajo SMALLWORLD como otros sistemas. Además tambien se pueden tratar temas de programación más genéricos.
• 84 members
• smallworld_tw
• Taiwan - SMALLWORLD Developers Group. The Technical Transfer Discussion eGroup of the SMALLWORLD GIS Technology for all the Cradle internal and partner-ship developers in Taiwan and Worldwide!!
• 53 members
• SWConnector · Smallworld (.NET) Connector
• This Smallworld (.NET) Connector group is used for the purpose of informing those interested in the advances of developing .NET applications over a Smallworld GIS database using SWConnector.
• 6 members
• smallworldgroup
• Kindly help me with Process Methodology for Merge/Append Process in Smallworld ( Merge/Append of 2 different data sets with one case tool ), Workflow in this regard would also help. If we need to do merge/append in live database what additional precautions needs to be taken. Thanking You in anticipation.
• 1 member
• smallworld_GIS
• This group is specifically for the users of GEsmallworld. You can post your query and problems. An online resource for all your query and doubts solved. This group provides a podium for GESmallworld Application, development and Programming.It emcompasses technical issues concerning GESmallworld, CASE Tool, Database Creation and Administration, Magik programming, interfacing with/to external systems. A Guide to usage and development.
• 1 member
• p_khoodaverdi
• For GE Smallworld users in Middle East
• 2 members
• sw-gis-group
• The GIS Professional discussion group for GE Smallworld, ArcGIS, AutoCAD and ESRI products.
• 1 member
  

Google Groups
Google Groups also has some Smallworld resources, but they seem to have very low membership and usually require registering with the group owner before you can access the archives. Another interesting thing I noticed was that many of the groups had names similar to the groups listed in Yahoo Groups. I'm not sure the purpose of that, but I note that there do not seem to be cross links between Google and Yahoo with these groups. You can find these groups by browsing to http://groups.google.com/group/.

• swgis-indonesia
• Description: GIS / SIG Indonesia: FORUM Diskusi pengembang aplikasi Geographic Information System (GIS) di Indonesia. Untuk pengguna software GE SMALLWORLD, ESRI, MAPINFO, GEOMEDIA, AUTOCAD , SICAD, ORACLE SPATIAL, CADCORP, Bentley Microstation atau yang lain
• 7 members
• smallworld-gis
• Description: Smallworld GIS Forum. neXgen Technologies International Ltd.
• 4 members
• Smallworld-Core
• Description: Smallworld kullanıcıları ve uzmanları
• 2 members
• sw-gis
• Group description: Smallworld GIS Technical Group
• 2 members
• sw-gis-users
• Description: GIS users forum for discussion on Smallworld, ArcSDE, Mapinfo etc...
• 2 members
• sw-gis-announce
• Smallworld Announce
• 1 member

SourceForge
This is not a regular "groups" community, but it is a place where people with common interests and ideas congregate to discuss and then make tangible contributions. Have a look at http://sourceforge.net/projects/magikcmpnts. It is an open contribution of Magik Components that others have found useful and want to share with the Smallworld community.

Blogs
I have done a search for blogs related to general Smallworld discussion and have found no active ones. That is why I started the SWORLDWATCH blog. I hope you find it helpful.

USA/Canada Daylight Saving Time is 4 weeks longer this year. Are you ready?

You may have heard that the Daylight Saving Time (DST) start and end dates are changing in the USA and Canada in 2007. They are a result of the USA Energy Policy Act of 2005.

Have you considered the changes you will need to make to your Smallworld installation to account for these changes?

Thanks to Bill's post at sw-gis, we have learned that you should contact your local authorized Smallworld Support team for the new timezone fixes. The switch to DST for this year occurs on March 11 so it is a good idea to have the fixes in place before then.

[Updated 05Jan07: I contacted GE Smallworld help and they confirmed that TSB111 addresses this Daylight Saving Time change.
Note: when updating to TSB111, please also read the document title "Issue with TSB 111 - Update". It provides important information that supersedes the original TSB111 document.]

National Resource Council Canada has an article that describes how the USA Daylight Saving Time has now become a "North American" standard to include Canada as well. (it is not clear to me whether Mexico has implemented these changes)

Items to consider:

• contact your Smallworld Support team for the new timezone fixes before March 11, 2007
  
• If you are a Canadian customer, confirm that Smallworld has changed the timezone information for Canadian zones as well. When the USA Energy Policy Act of 2005 was first introduced, it apparently caught the Canadian government by surprise and there was some confusion about whether Canada would follow suit.
• The Energy Policy Act of 2005 has a stipulation in it that Congress should at the end of 2007 review the energy savings realized by the DST shift. The implication is that the DST changes could be rolled back to what they used to be. You might consider keeping a copy of your old product\data\zoneinfo data handy in case this grand experiment proves ineffective.
  

Related to all this, David posts some helpful links describing how Windows, Outlook and Java will address these DST changes.

be very careful when overwriting core image files

WARNING: TECHNICAL DISCUSSION AHEAD!!!

(If you have made changes to your module.def files and the changes do not seem to "take hold" AND your core images may have been overwritten since you installed them from the CD, OR;
you have installed Telco TSB 47 and now notice unexpected module/product behaviour, read on...)


It is usually not good practice to overwrite any core images that are shipped by Smallworld or a 3rd party vendor. A typical build script will use a core image as its starting point but then save a new custom image with a different name or path.

But there are some rare instances where a customer needs to overwrite a core image. One such case is required by Telco TSB 47. That TSB includes a document "Upgrading PNI 4/LNI 4 to Core Schematics TSB 1". It instructs the customer to open pni40_closed.msf, update the schematics UDS dataset definition and then resave the pni40_closed.msf image with the same name. This is an efficient way of ensuring that all PNI/LNI applications will see the updated schematics UDS dataset definitions.

BUT, the instructions do not account for something subtle yet very important: the SW_PRODUCTS_PATH environment variable. If a customer has correctly used the SW_PRODUCTS_PATH environment variable then it will point to one or more product.def files which also include image/application modules unique to that customer. Typically an image or application module will "require" other modules that provide the desired functionality for the customer. A customer can add new functionality to their application by "requiring" other modules (created internally or supplied by 3rd party vendors). If someone overwrites a core image file without accounting for the SW_PRODUCTS_PATH environment variable, they run the risk of rendering future module definition changes as unreadable.

Why? Consider the following. When an image is started, its smallworld_product class has a slot that contains references to all known product.def files (and by inference, all module.def files). At image startup, the smallworld_product class augments its list of known product.def files by also looking at the %SW_PRODUCTS_PATH%. A shipped core image does not "know" about any of your custom products or modules. But if you start up your environment and then overwrite the core image, the act of opening the core image has "introduced" that core image to your custom products and modules and overwriting the core image will now save the knowledge of your custom products and modules in the core image itself.

Why is this a problem? Storing a module definition in an image makes the image a bit arrogant. That is to say, the image now thinks it knows everything there is to know about the saved module definitions and will ignore any changes you make to the actual module.def file from which it derived its information.

This is especially significant when it comes to changing a module.def in the future. Typically the change is to the "requires" stanza. Because the image already knows about a "requires" stanza for the module, it will see no need to reinitialize its definition based on the changes you entered into the module.def file. You can force a product or module to reinitialize itself, but I think the better solution is to not store custom product/module definitions in core images in the first place.

How do we fix this problem? We go back to the SW_PRODUCTS_PATH environment variable. If you will be overwriting a core image, first ensure that the SW_PRODUCTS_PATH environment variable has no value. You may need to temporarily modify your environment BAT file (or whatever BAT file you use) to accomplish this. Once you have overwritten your image, you can revert SW_PRODUCTS_PATH back to what it needs to be for your custom environment.