Wednesday, December 1, 2010

Smallworld Emacs tutorials

There are three more video tutorials for using Igor's Emacs extensions with Magik. There are so many features available in Emacs and you are bound to find out some useful tip that will make your coding tasks more efficient.

Visit to find out more.

My favorite features...

F12-m - creates a temporary Magik file where you can hack without affecting the original source code files.

Alt-c - useful for tagging a block of code with START CHANGE/END CHANGE when making your own patches.

F2-t - creates debug write() statement around a variable in the code.

F4-s - creates #DEBUG show() statements around a variable (use F3-d to toggle #DEBUG mode)

F3-t - creates a debug write() statement at the top of the method.

Alt-x all - this is one of my all-time favourite tools within Emacs. Great for finding all the occurrences of a string (using regexp) in a file. See 3:20 of MagikEmacs Screen-cast 4 for a demo.

F9 - selects the entire method.

Ctrl-Alt-\ - indents the Magik code in current selection

Alt-space - drops breadcrumbs (ie bookmarks in Emacs buffers)

F5-b - loops through the current breadcrumbs

Alt-F5 - manage your breadcrumbs

Thursday, November 18, 2010

SEPM Launches a YouTube Channel

Robert at SEPM took my suggesting in a recent post seriously enough to create a SEPM YouTube Channel. The first video is up and shows some very cool stuff that he has been working on.

If you are considering creating your own channel, please consider tagging the videos with "swgis". On Twitter you can use the #swgis hashtag.

Wednesday, November 17, 2010

Visualization in Smallworld (Part 2)

Back in January, I posted an article about Visualization in Smallworld. The article was long on words and short on visuals, so now I have created a video that shows more clearly what the post about.

Wednesday, November 10, 2010

Another use of OpenStreetMap within Smallworld

Up until today, I was aware of only two Smallworld-based products that incorporate OpenStreetMap (aka OSM. see data into a Smallworld application:
  1. iFactor Consulting's (my employer) Web Maps Connector; and
  2. another company working on OSM to be released soon.

So I was pleasantly surprised to see an announcement from SEPM today that one of the recent products they have released is SEPM X-Database "Open Street Map". You can read more about the product by downloading SEPM_2010_02_New_Features.pdf and browsing to page 20.

At the Smallworld Americas Users Conference this year I presented on OpenStreetMap integration with Smallworld (download presentation here). I presented heavily on using tile based OSM rendering in Smallworld together with downloading vector data on demand. I also mentioned other approaches such as setting up your own server or downloading OSM data into a VMDS partition. Now it seems that SEPM has taken that latter approach and is presenting OSM in VMDS. Clearly there are advantages/disadvantages to the different approaches (e.g., fetching tiles via a web service vs. having all the OSM data in VMDS). But the beautiful thing of having different options is that you can pick the one that fits yours needs best. And the OSM price is not one to be argued with :)

Now if we can just get SEPM to put a video up on YouTube demoing this product...

Friday, November 5, 2010

Designing Simplicity into Smallworld

At the Smallworld America's Users Conference this September in Baltimore, I heard a comment from an attendee that has stuck with me... "Smallworld needs to innovate simplicity into its product". Clearly there are many ways to accomplish this, but I thought I would share one area where I have been able to add some simplicity. I would very much like to hear from others about how they have designed simplicity into Smallworld applications or where they would like to see a simpler design.

Tuesday, November 2, 2010

Smallworld on the Social Web

I have become very interested in the use of social media to provide a place to share ideas about Smallworld. Obviously the social web is a place where companies and individuals can market their products and services. But I have begun to see the social web as an extension of the informal technical meetings I have had with colleagues over the years where new ideas have been generated and old ideas refined.

I would recommend that you watch the TED Talk by Steven Johnson: Where good ideas come from. He gives some great examples of how networks of people get together in chaotic situations to share ideas and out of those situations arise great new ideas. One of the very interesting examples he gives at the end is how some researchers trying to hear the telemetry signals from Sputnik triggered a series of events that today allows us all to use our mobile phones as navigation devices.

In that same stream of thought, I am beginning to see the Social Web as a place that the Smallworld community can use to foster new ideas and innovation.

Some in the Smallworld community have moved onto Twitter. That is a great tool for updating others on the status of ideas and projects (remember to tag your tweets with #swgis).

Others are carrying on Smallworld-related conversations in Linkedin "GE Smallworld" group.

There is the well-used Yahoo group sw-gis that many of us use. It is a great forum for answering questions to problems.

But what if we don't know what the problem definition is? I have recently begun to see the utility of YouTube in fostering this idea sharing. Another TED Talk by Chris Anderson: How YouTube is driving innovation got me to start thinking about how I could use video to share ideas and add them to the chaotic mix that stimulates innovation. I have started a channel at that shares my Smallworld-related ideas and experiences.

I'm curious to know if there are other Smallworld-related YouTube videos/channels out there. Please let me know in a comment on this post. I know of only one other one:

When you post to YouTube, remember to tag it with "swgis". It turns out that the "Smallworld" tag is more likely to take you to Disney-related content that it is to a GIS :(

Not happy with Magik in Emacs? See the MDT Webinar.

I've written a few posts related to Astec's MDT Eclipse plugin over the years. I have gotten used to using Emacs but I realize that is not for everyone (especially people coming to Magik from other development environments). So I was happy to read that Astec would be hosting a MDT webinar at the beginning of December. The timing of it is aimed at participants in the western hemisphere.

(Maybe Astec can set up a YouTube channel with additional demo videos for those who cannot attend the webinar).

Here's the announcement...

We would like to invite all Magik developers to an online live demo presentation of MDT (Magik Development Tools): the new and modern IDE for Magik.

Session is planned to be held on December, the 2nd, 1 p.m. Washington D.C. time.

MDT live demo presents main MDT features from simple basics to most advanced stuff. Topics included in this demo are as follows:
- creating project, runtime, session;
- importing and exporting (preferences, sessions, products etc.);
- working with code;
- browsing through code;
- managing project dependencies;
- search;
- version control systems;
- debugging

duration: approx. 60 min.

Due to limited number of seats please confirm your interest on If needed, we will hold another session 90 minutes after the first one. Joining details and link will be posted one week prior to the presentation.

MDT Team

Thursday, October 28, 2010

Creating a 3D PDF from Smallworld

Lassi Tani (@Lassitani) has posted a very cool example of creating a 3D PDF visualization of Smallworld data using FME at

Thursday, October 21, 2010

Some cool geospatial events in Denver in the next year

There are some really exciting Geospatial events happening in and around Denver in the next year. While they are not strictly related to Smallworld, I think these would be great venues for spreading the word about Smallworld technology to a wide geospatial audience.

At the Smallworld Americas User Conference in Baltimore this year, a GE-hosted forum discussed how Smallworld could gain exposure in the big world. I would like to suggest that one way to do this is to provide sponsorship for these events and get staff/partners to provide presentations for these events. At the very least, send observers.

PS. If you are travelling from out of state/country, you can attend the two September 2011 events in Denver, then spend a week travelling around the Rocky Mountain west and then travel over to Chicago for the Americas User Conference (September 25-29, 2011). What a great month that is going to be!

Friday, October 15, 2010

Decimal Degrees vs DMS

I had someone ask me today how to configure Smallworld to show Decimal Degrees or Degrees/Minutes/Seconds.

I found two ways. The one way is to use the Administration application's Unit Display Groups tool to change the "Sub Units?" value

The other option is to create some Magik code in your image build process that changes the sub units setting.

unit_manager.unit(:degree).default_format.use_sub_units? << _false # (or _true)

Smallworld GeoSpatial Analysis examples

I recently had the good fortune of being able to teach a course on Smallworld GeoSpatial Analysis (GSA). If you haven't seen the product, you really should try to get a demo from your GE Smallworld representative. As the GE documentation states, "Smallworld GeoSpatial Analysis is a new, exciting, and easy to use business intelligence product from GE Energy Services."

During the training, some of my students asked me to put together some videos of use case scenarios that were more relevant to their situation. What follows here are 4 videos (each less than 5 minutes) that I made for them but now make available for everyone to get a quick introduction to some of the possibilities of GSA. In my mind, GSA has many great spatial intelligence features, but these videos touch on only a very few of them.

Thursday, September 23, 2010

Using #FME to mash open data into #OSM addresses

While this video is not strictly related to Smallworld, it does describe mashing open data from various sources into something usable in a destination database. The "Big World" forum at the #swuc recently covered the area of open data and I thought this might be of interest to some of you out there.

In this case, the demo takes open parcel and address information from the Boulder City and Boulder County (Colorado, USA), processes it with FME and writes data into a format that can be uploaded into OpenStreetMap for the world to see.

Wednesday, September 22, 2010

Smallworld Latin America Users Conference #swgis #swuc

I learned today that Smallworld will be hosting a Latin America Users Conference later this year. What a great idea! I likely won't be attending but would love to hear feedback from attendees. I'd be interested to see if Latin American attendees are more avid Tweeters. Please tweet on under #swuc. I guess now is a good time to get familiar with :)

Here is the announcement I received from Smallworld...

Please mark your calendars.

GE Energy will host a Smallworld Latin America Users Conference November 30-December 1 in São Paulo, Brazil.

The venue will be CENESP (Centro Empresarial de São Paulo) on Avenida Maria Coelho Aguiar 215 - Bloco G, 2nd Floor

v Join fellow customers, channels, third party companies related to our industry, and GE staff.

v Listen to industry presentations, case studies, and product announcements.

v Spend time at the Exhibition to learn about the latest software developments and innovations.

v Network and socialize with colleagues from across the Latin America.

v Simultaneous translations to be provided for Spanish and Portuguese.

Please RSVP to Denise Buciotti Coelho by phone +55 11 3067-8223 or email

Tuesday, September 21, 2010

New Smallworld Twitter Hashtag #swgis

Mark Field at Smallworld GIS Wiki suggests that the Twitter hashtag we use for all things Smallworld be #swgis instead of #sworld. The #swgis tag is shorter and not prone to use in other domains. I heartily agree with his suggestion and will use #swgis going forward.

Wednesday, September 15, 2010

A low-tech mobile field solution #sworld #osm

With all the cool hi-tech mobile field solutions available out there, I thought I would mention a very low-tech one. provides a way of printing out your maps onto paper. These paper maps have georeferencing marks on them. You take them out into the field, make your redlining changes and then when you come back into the office, you scan them and then upload the scan. The scanned maps are georeferenced onto the background map and made available for a back-office technician to digitize the data and red-lined attributes into the database.

Seems very simple yet effective. I'm not suggesting that Smallworld customers adopt this approach, but when thinking about mobile solutions I think it is good to understand the broad range of options and learn from them.

Smallworld Twitter after the Conference #swuc #sworld

This year was the first year that the Smallworld Users conference was covered by Twitter. The turnout was fairly decent with the likes of @ifactor, @pmbatty, @mweisman, @JEJusticeJR, @mbfbsae90, @SafeSoftware, @cliffhangerSolu, @Fasterre, @aardvark179 and @boulderalf all posting from the conference.

I met an attendee at the conference who was very new to Magik/Smallworld. He mentioned that he had been watching the #swuc tweets to see what to expect leading up to the conference. Because this seems to be fairly new to the #sworld community, he couldn't get a good read on it. I am hopeful that this new communication medium will find its way into the Smallworld community in the next year.

By the way, I have started tweeting about all things Smallworld under the #sworld hashtag. It turns out that #smallworld has alot of other connotations on Twitter but #sworld has not yet been used.

Saturday, September 11, 2010

What are advantages of Smallworld #swuc

At the "Smallworld Visibility Forum" today there was a question about "what does Smallworld do well and what doesn't it do well and has that been documented?" I thought that was an worthwhile question and was wondering if you have had exposure to other GIS systems to compare advantages/disadvantages of the systems?

For example...

1. I think wrapping the database records as objects in Magik provides for realistic real-world
2. Long transaction support is still an industry leader.
3. External access to VMDS has a performance bottle neck in Magik.

I would love to read your comments pro/con in the comments of this blog. Or you can send me an e-mail directly. If there is a lot of response, I can set up a wiki page at that lists the advantages/disadvantages.

Friday, September 10, 2010

Using SQL from Magik against VMDS #swuc

At the Smallworld Users Conference yesterday, I heard someone ask when GE would provide an interface to write SQL from Magik against VMDS tables. It turns out that the functionality already exists. Here are some examples.

1. You can get a SQL prompt directly at the Magik prompt...

MagikSF> sql(gis_program_manager.cached_dataset(:gis))
SQL> SELECT name, address1 FROM hotel ORDER BY address1
Name Address 1

Acorn Guest House 154 Chesterton Road
Hamilton Lodge 156 Chesterton Road
De Freville House 166 Chesterton Road
Rosswill 17 Chesterton Road
Cam Guest House 17 Elizabeth Way
Kirkwood House 172 Chesterton Road
All Seasons 219 Chesterton Road
Lyngamore 35 Chesterton Road
Suffolk House 69 Milton Road
Southampton House 7 Elizabeth Way
Ashley Hotel 74 Chesterton Road
Arundel House Chesterton Road
Gonville Hotel Gonville Place
Garden House Granta Place
University Arms Regent Street

15 record(s) selected
SQL> SQL> select count(*) from min_road


1 record(s) selected

NOTE: the sql() procedure is defined in module :sql2. You can load that module if your image doesn't have the sql() procedure loaded.

2. You can also use the sql2_parser object directly in your Magik code.
You can find out more by searching the Smallworld documentation for "SQL Support". To be clear, this is not the same as Smallworld's SQL Server image code. This is strictly related to writing SQL in Magik to query Smallworld tables.

I have never used this in code, but I suppose it could come in handy if you were more comfortable writing a SQL statement compared to writing a selection predicate. I'm curious to hear if anyone else uses this SQL functionality within Smallworld.

Friday, September 3, 2010

Smallworld and OpenStreetMap at #swuc

My presentation at the Smallworld Users Conference is titled "Let Other People Maintain Your Landbase (Without Having to Pay Them)". It could just as easily have been titled "Integrating OpenStreetMap with Smallworld".

I will be covering some options for how the utility can move out of the business of maintaining landbase. Most of the presentation will be an introduction to OpenStreetMap and how you might make use of that in your Smallworld applications. If you cannot make it to the presentation, let me know and I will send you the slides.

The presentation is on Friday, September 10 in Key Ballroom 3 from 3:45-4:25pm.

On Saturday, September 11, GE will be hosting a forum titled "Smallworld and the Big World". One of the agenda items is related to "open data". I will be attending that forum and hopefully actively participating in the "open data" bit. This forum will be in Key Ballroom 4 from 2:45-4:30pm.

See you in Baltimore! I am looking forward to meeting lots of people. Please stop by the iFactor booth and say hi.

Should you attend FME Users Exchange #swuc

If you are involved with moving data in or out of Smallworld I maintain that the most powerful way to do that is using Safe Software's FME product. If you are already using FME or if you are just wondering how FME can help you integrate Smallworld data with other data formats, I recommend that you attend the FME Users Exchange on Thursday, September 9 from 1pm-4pm in Baltimore at the site of the Smallworld Americas 2010 Users Conference.

There will be great presentations by customers, GE and Safe representatives. I will be presenting on how to enhance your GE FME-Smallworld Magik interface to let you do things like updating and deleting records and also creating Smallworld joins via FME.

As a bit of a trailer, here is a 5 minute video that highlights some of the items I will be presenting in more detail (video:

You can see my prioritized presentation points here:

Wednesday, September 1, 2010

Sign up for the FME Users Exchange at #swuc

The Smallworld-FME Users Exchange is happening at the Smallworld Americas Users Conference in Baltimore, MD this next Thursday, September 9 from 1:00-4:00pm. It will be in the Key Ballroom 4, Hilton Ballroom. If you are a Smallworld/FME user, I would highly recommend that you attend this event. There will be a customer case study, some best practices and demos and also a product roadmap presentation from GE.

You can find out more information about it here:

Wednesday, August 25, 2010

Where was that method or exemplar defined?

Do you ever want to know the source file for a particular method? You can use the class browser to jump to the code or you can do this...

MagikSF> circle.method(:new|()|).source_file

Most of the time, jumping to a method is faster in the Class Browser than using this code snippet.

But what if you want to jump to the file where the actual class exemplar (not the method) is defined? Then you can do this...

MagikSF> circle.method(:exemplar).source_file

I know of no way to do this latter action from the Class Browser.

Tuesday, August 24, 2010

#FME Users Exchange at #swuc topics

The FME Users Exchange at the 2010 Smallworld Users Conference will take place in Baltimore, Maryland on Thursday, September 9 from 1pm to 4pm (EDT). You can read more details of the agenda at

I will be presenting a section on "Best Practices: Tweaking FME-Magik code to improve your FME experience in Smallworld". It turns out that I have more material to cover than the approximately 30 minutes allotted to me. So in the spirit of conference collaboration, I thought I would put all my proposed topics up for a vote. The topics with the most votes get moved to the top of my presentation list. The ones with the least votes get used as filler at the end :)

Please vote on the topics here ( you are also welcome to add your own topics).

See you in Baltimore!

Tuesday, August 17, 2010

How much does an alternative contribute to file size?

If you want determine how much a particular alternative contributes to the overall size of a .ds file, use these debug methods...

_local v << gis_program_manager.cached_dataset(:gis)


See the public comments for ds_version_view.file_usage() for a description of the data that is returned. Based on that, you can figure out what percentage of total blocks used is attributable to a particular alternative.

This can give you guidance for which alternatives to merge/post/delete, etc.

Wednesday, August 4, 2010

Promote your #SWUC 2010 items here

When I attend Smallworld Users Conferences, I typically don't get around to all the booths because I get busy with other things. So I usually feel like I have missed some good demonstrations and presentations by the time the conference is done. So I thought that it would be helpful for the various Smallworld vendors/exhibitors/presenters to promote what they were planning to share at the conference.

So here's my idea.
1. Get your Twitter account going (if you don't already have one)
2. start posting short tidbits about what you are planning for the conference. Be sure that you use the hashtag #swuc.

From now until the conference, I will keep a dynamic Twitter feed on that hashtag running at the top of this blog. This will be a way for us to get Smallworld-related commercial content onto this blog without anyone having to receive them in their RSS readers or inboxes. Come back to this blog often to see the latest updates. Or you can browse to #swuc to see the results on the Twitter search page. You could embed the Twitter search results in your own blog or company web page as well to spread the love.

I would love to see posts about stuff like:
  • will you be attending?
  • what demos you will give at your booth
  • if you don't have a booth, say that you will be representing your company anyways and would love to meet with customers (tell them which bar to meet you at)
  • presentations you will be doing or are interested in
  • thoughts on trends/topics at the conference
I'm thinking that a lively contribution stream will make the conference that much more interesting. And if for no other reason, please post so that it doesn't become just an iFactor Consulting promotion stream :)

Monday, August 2, 2010

Smallworld and the Big World #swuc

Recently the organizers for the Smallworld Americas Users Conference sent out an e-mail related to booking your exhibit booth. If you saw the subject line and ignored the e-mail (maybe because you are not a vendor), you would have missed a significant announcement. I have copied the announcement here in case you missed it...

====begin announcement====

This year we are introducing a new session on the agenda that may interest you…it is open to all attendees including customers, partners, and GE staff:

Smallworld and the Big World – Saturday September 11th 2:45-4:30pm, with a 15 minute break at 3:30

After a week of concentrating on products and applications within the Smallworld community, we’d like to shift the focus outside: how we relate to the bigger world.

This session is an open forum to discuss things like:

· visibility (making Smallworld and MapFrame more visible in the geospatial arena)

· standards (OGC and others)

· policy (stimulus, other government actions)

· technology trends (smart phones, neo-geo, social networking)

· organizations (sources of information and influence).

====end announcement====

I particularly am interested in the "technology trends" item. I have started doing work with integrating OpenStreetMap (neo-geo) with Smallworld (paleo-geo?) and now seeing that GE is going to have a forum about that is very encouraging. My presentation on OSM and Smallworld integration is currently scheduled for the same time as the open forum, but I am trying to get the time changed so I can be part of the forum as well.

I also see that they want to discuss social networking. In a past blog I have suggested that conference attendees try tweeting about all things conference-related under the hashtag #swuc. I think the amount of traffic on that hashtag could be gauge of how much the Smallworld community is already into this technology trend.

Could GE encourage their attending employees to start tweeting/blogging about conference proceedings? That would be very encouraging.

Exciting things happening in the Smallworld realm!

Saturday, July 31, 2010

Smallworld and OpenStreetMap #swuc

Lately I have become very interested in the possibilities of integrating data from (aka OSM) with Smallworld. You can find out more about the mission of OSM at, but in summary…

“OpenStreetMap creates and provides free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways.” (

Think of OSM as Wikipedia but on a world-wide geographic database. I am presenting at this year’s Smallworld Americas Users Conference in September a presentation entitled “Let other people maintain your landbase (without having to pay them)”. It will be a discussion of some options available to Smallworld users to get out of the landbase maintenance business. Most of it will be centered around OpenStreetMap and how you can integrate that freely available data as a Smallworld data layer (both raster and vector data). The session will be in the last time slot of the conference on Saturday, so if you are interested in the topic but cannot attend the presentation, please let me know and I will send you the slides.

As an example of OSM data, if you are a utility that has Bakersfield, California, USA in your service area, have a look at this link This is the OSM data that is available for this area. If you read the attribution on the features in this view, you can see that someone got the data from the City of Bakersfield GIS download site. You don’t get that kind of building detail (even individual trees!) from Bing or Google. Although if you do integrate aerial imagery into Smallworld you could just see the building profiles. But if you are testing the viability of integrating external datasets with your Smallworld environment, you might want to consider OSM. The price is right and the data quality/quantity keeps increasing. The beauty of OSM data is that if you know that some data is missing or incorrect in the map, you can make edits to the database yourself and the changes will be available on the map within an hour or two. That kind of turn-around is not available from most other data vendors. Clearly there are trade-offs between using OSM vs. Bing/Navteq/TeleAtlas but that is what I hope to cover in my presentation.

If anyone has questions about how OSM could be integrated into their Smallworld database, please let me know.

Thursday, July 22, 2010

Smallworld Americas Users Conference #swuc

The 2010 Smallworld Americas Users Conference is coming up soon (September 8-11, 2010). You can read more about it here. An important thing to note is that the order as well as the start day are different compared to previous North American Smallworld Users Conferences:
  • Training sessions occur on Wednesday and Thursday (September 8 and 9)
  • The opening Cocktail Reception and first Technology Showcase Exhibition is on Thursday night.
  • Opening session and workshops start on Friday morning and finish on Saturday afternoon.

So this year there is 1 less day of workshops than in other recent North American Smallworld Users Conference. So now there is no excuse for anyone to leave early :)

Twitter alert!!! If you will be sending tweets about/from the conference, I suggest that you use the hash tag #swuc

Here is why I will be using Twitter at the conference:
  • coordinating information technical meetups. (Kind of like an informal unconference)
  • provide commentary about workshops. Hopefully if enough other people send tweets, I can find out what is happening in workshops that I am unable to attend
  • reach out to new attendees and international guests. The Smallworld Users Conference is a great way to network and get caught up with friends and colleagues from years gone by. Unfortunately, if you are new to the Smallworld community in North America you might find it more difficult to access this great network. Using a well-known channel such as Twitter will allow anyone to post questions/answers related to the conference during the conference.
I'm curious to see how this will take. I have had great experiences with this kind of approach at other conferences. I imagine there could be benefits to the Smallworld Conference attendees as well.

See you in September!

Friday, July 9, 2010

FME Import Zoom to Smallworld Map

In the SWAF version of the Smallworld-FME plugin, it would be nice to have the current Smallworld map "goto" the bounds after the FME import has completed. This comes in handy when you want to easily find your just-imported data for subsequent review.

Apparently this functionality used to exist in the pre-SWAF version of the FME plugin.

I have uploaded a sample of some tweaks you can make that will enable this functionality at Smallworld version 4.x (SWAF).

The basic steps are:
  1. in fme_tics_client.private_run(), determine if there are any smallworld_product.applications that might be interested in knowing where the FME-imported geometry bounding box is
  2. if there are any such applications, then create a dynamic variable geometry_set that!received_feature() can add newly-created geometries to
  3. back in fme_tics_client.private_run(), send a :goto_request databus message to all the relevant applications with the geometry_set populated in!received_feature()

Have a look at this file (fme_goto_import_geometry_set.magik) to see the details of how it is done.

Thursday, July 8, 2010

WMC Webinar Next Week

iFactor Consulting (my employer) is putting on a webinar next week featuring the Web Maps Connector. There's some pretty cool stuff in there in terms of integrating Smallworld with outside data sources (landbase, weather, satellite imagery, street routing, geocoding, reverse geocoding). If you are interested, please click here for more information.

Thursday, July 1, 2010

FME SmartCleaner makes things easy

I recently had a customer requirement to:
  1. read Lot lines and lot labels as two separate layers from a single DWG file
  2. where possible, stitch the lot lines together to make a polygon
  3. where possible, figure out which lot label is inside each of the newly generated polygons and aggregate those two geometries into one RWO in Smallworld

It turns out that step#3 can easily be done with PointOnAreaOverlayer and Aggregator.

But step#2 didn't seem that easy. I had tried Intersector, TopologyBuilder and Snapper but I never seemed to be able to get the over/under shoots to snap. Then I remembered to check and discovered that there is a custom transformer that does exactly what I needed. It is called the SmartCleaner and is described here.

You can choose to either link or embed this Custom Transformer. I chose to embed it into my existing workspace. This made it transportable across user machines without having to ensure that each user explicitly installed the custom transformer.

When I looked at the internal workings of the SmartCleaner, I realized why my efforts to use Intersector and Snapper were inadequate. The following screen snapshot shows the complexity of the SmartCleaner.

All I can say is a hearty "thank you" to Dave Campanas at Safe Software for creating this Custom Transformer and making this over/under shoot snapping so easy to implement.

Thursday, June 24, 2010

Very Cool Charts in Smallworld

Ever tried showing pie charts in Magik? Or bar charts or scatter plots? Then tried to label the data points? And create a matching chart legend? Technically, it is all possible in Magik, but the code to manage all that could be quite complex.

Check out this post ( about how Graham at iFactor Consulting has written a Magik interface to the Google Charts API. The possibilities are endless for various kinds of dashboard applications directly in Magik. And I think the charts are also clickable (ie you can drill down on data points). But you might want to ask Graham for more details.

Tuesday, June 8, 2010

Configuring FME to Write Multiple Datasets More Efficiently

This post by the FME Evangelist describes how to configure your Workspace so that a minimum number of features are cached in memory. The less amount of caching, the more efficient your workspace. This applies to any workspace that has more than one writer (aka destination) dataset in it. Always put the dataset that has the potential to contain the most features to the top of the writer list.

The post gives a more detailed description.

Weather Radar Layer in Smallworld

iFactor Consulting now has a service that allows you to see the current NOAA Weather Radar layer as an overlay on your Smallworld map (see the post here). The service is integrated with Smallworld using the Web Maps Connector.

Clearly NOAA data applies only to data in the USA, but it seems like it wouldn't take much more effort to integrate weather layers from other regions.

(Disclosure: I am happily employed by iFactor Consulting so this isn't a completely unbiased post)

PS. If you are a Web Maps Connector user you can also get updates on Twitter at

Thursday, May 27, 2010

MDT on Smallworld 4.2

In recent MDT news from ASTEC, I noticed an article that stated that "MDT will be included in new Smallworld CST 4.2". Basically, during the Smallworld installation process you will be able to choose whether you want to use Emacs or MDT. As far as I can tell, you will initially be able to evaluate MDT for 30 days and then have to purchase a licensed version to continue using it.

For those of you who don't know, MDT (Magik Development Tools) is a visual Integrated Development Environment for Smallworld developers that is based on the Eclipse framework. You can read more about the details here.

I think that it promises faster familiarization with Magik particularly for those developers just getting started. I have to admit that it took me quite a while to get comfortable with Emacs.

I haven't used the tool much myself, but would love to hear from users that have made the switch from Emacs to MDT to learn about their experiences.

Thursday, May 13, 2010

FME and Smallworld Text Join fields

Smallworld ignores data from FME that is destined for text_join fields.

Smallworld has a data type called "text_join" which allows you to store a string of indefinite length on a Smallworld record. Under the hood, this data type is actually modeled as a join relationship to an internal table that stores chunks of finite-length strings that can be added to each other to emulate an indefinite-length string. So a text_join is actually considered a join field and not a physical field.

This becomes a problem when you want to copy data into one of these text_join fields. This is not an unusual requirement. Many Smallworld objects have at least one text_join field (e.g., Electric Office objects have a "Remarks" field; other datamodels might have a "Comments" field). Not being able to import attribute values via FME into those Smallworld objects is not an insignificant limitation.

I raised a case with GE about this recently and was told that this was a duplicate of a same case that I had raised a few years ago. I knew that there had been something familiar with this issue, but I couldn't remember that I had already encountered it a few years ago. Because there is no patch in sight any time soon for this issue, I present to you the fix I made. Like most of the FME/Magik tweaks I have had to make, the implementation is fairly easy.

(click on image to enlarge)

Debugging Smallworld-FME Error Messages

Sometimes, when I am importing data into Smallworld using FME, I get a Smallworld error message in the FME log. Unfortunately I get only the condition string and not the accompanying traceback so it is usually very difficult to debug. The traceback is also not written out to the Magik standard output.

This simple hack will cause any error conditions raised during an FME import to have a traceback dumped to the Magik standard output.

(click on image below to enlarge)

Monday, May 3, 2010

Update Triggers Fired Unnecessarily

Recently, I was working on some performance profiling on some QAQC scripts in Electric Office. It turns out that the big bottleneck was in the trigger and business rule code that was being called during the QAQC routine run. In my review of the business rules, I saw that a lot of the business rules were being called unnecessarily. That is, the update triggers were being called on objects where no material change had occurred to their attributes. If you call record_transaction.new_update(proposed_values).run() and the proposed_values are all the same as the existing values for those attributes, the triggers and business rules will still be fired. Because some of the business rules called code that was expensive to run, this added unnecessary time to the whole process.

Please be aware of this whenever you write code with record_transaction.new_update() and make sure that you only update attributes where the value has actually changed. The record_transaction API will not optimize your proposed_values list for you.

If you compile the following code into your image and then run any “slow” process, you will be notified of all the times a record gets updated with nothing new, causing the business rules to be fired unnecessarily…

_method record_transaction.new_update( record, values, _optional description )
## Creates an instance which specifies the update of a record.
## RECORD - the record to be updated
## VALUES - a property_list keyed on field name.
## The elements of this property_list provide proposed values
## for physical, geometry and simple valued join fields, and
## information for updating multi valued join fields.
## DESCRIPTION - String description on transactions to be used
## for undo messages.

_local anything_changed? << _false
_for k,v _over values.fast_keys_and_elements()
_if record.perform(k) <> v
anything_changed? << _true

_if _not anything_changed?
write("Nothing changed for: ",record)
# put a traceback in here to figure out which code is calling
# unnecessary new_update()

>> _clone.init( :update, _unset, record, values, description )


Tuesday, April 27, 2010

Debugging batch files

Creating Smallworld images often requires one or more batch files with myriads of environment variables. This link gives a good summary of various batch file debugging techniques.

[NOTE: a reader notified me that the referenced link is broken. You can find a cached version of it at Google here]

Friday, April 16, 2010

PDF as Export Format (not just for plotting anymore)

You are all aware that Smallworld has the ability to print maps to PDF.

Now, with FME, you can export Smallworld data to PDF. In this example I:
  • Imported some Cambridge demo db data (substations, cables, supply points)
  • Imported a Google Static Map based on the bounds of the Smallworld data
  • Exported to Geospatial PDF format which is readable with Adobe Reader
I apologize for not providing the actual files, but it seems that my online file repository is no longer available. I will include screen shots instead. You can click on the image to get the full size.

FME workspace...

Adobe Reader. Note the ability to show attributes for selected objects. You can also do "goto selected object".

This was just a very simple example of mashing together data from multiple sources (including Smallworld) and exporting to a format that even "Ken's grandpa can read*".

Adobe Reader is free and widely available on most computers. It seems to me that this would be one format that would lend itself well to an inexpensive mobile field solution.

* inside joke from the FMEOdyssey 2010 meeting in Denver in March.

Tuesday, April 13, 2010

SIAS client using Silverlight

As you are no doubt aware, most SIAS client applications use Java to build their front end. I saw something really cool today done by long-time Smallworld developer Poorya Khodaverdi. He has created a Silverlight-based web application that uses SIAS services. You can see his 5-minute demo here.

As he pointed out to me, this was just a rapid application development implementation. It should be easy enough to modify the client to whatever specs a user would want. While we may all have our differing opinions of Microsoft and their Silverlight framework, I do think that any technology that makes the user interface more pleasant to work with is worth having a look at.

You can find out more about this Silverlight/SIAS integration by contacting Poorya directly at

Saturday, April 3, 2010

Converting Coordinate System From Smallworld to FME

When setting up an FME Workspace to interact with Smallworld, you always want to be sure that your SWORLDSWAF reader/writer coordinate system is the same as your Smallworld session's Application Coordinate System (see discussion here). The post "Smallworld/FME Configuration Tips and Tricks" has a link to a slideshow that describes how to make sure that your Smallworld Application Coordinate System and FME SWORLDSWAF reader/writer coordinate systems match . The relevant information starts on slide 31.

I recently discovered another way of figuring out the FME COORDINATE_SYSTEM_DEF that corresponds to a Smallworld coordinate_system.

The FME Workbench's CoordinateSystemDescriptioConvertor Transformer (see example here) can take a Proj.4 coordinate system definition string and convert it into a suitable FME definition.

And how do you get the Proj.4 definition string? Easy... just use the following method.

_method coordinate_system_mixin.proj4_string
## proj4_string : string
## returns a string that represents self in the Proj.4 format.
## This can be used by coordinate system converters to convert
## self into other representations.

# according to the comments in
# coordinate_system_mixin.new_proj(): "PROJ_TYPE is a symbol
# specifying the projection type. The naming convention is
# similar, but not identical, to that used in the USGS Proj4
# library. See below for details."
# because the Magik names are not always identical to the Proj4
# names, we provide a lookup table here.
_local projection_name_lookup << hash_table.new_with(:accurate_tmerc,:tmerc,

_local (proj,params) << _self.proj_type_and_params

_local info <<

_for k,v _over params.fast_keys_and_elements()

_local int_str <<


_return int_str.string


The next step will be to write a Magik method that actually converts the Smallworld coordinate_system definition directly into the FME COORDINATE_SYSTEM_DEF string all within Magik. That should allow for some greater flexibility in automating various Smallworld/FME processes.


Thursday, April 1, 2010

How to tell your emacs apart

If you, like me, tend to have more than one emacs session running, your OS toolbar will show a bunch of emacs applications running all titled "emacs@". It gets a bit tiring to have to click on the multiple emacs applications to see which one I need to jump to. I discovered that you can easily rename the title for each emacs session and that title change will also appear in your OS toolbar.

The command is...

M-x set-frame-name

... and needs to be run for each emacs session.

I imagine that someone could easily modify the .emacs file to set the frame name based on some environment variable. If anyone does that, please post a comment with the code.

Wednesday, March 31, 2010

Getting a resource file from a module

I am looking at some product upgrade code that is trying to find a resource file relative to a module definition and I thought this would be a good example of how not to do it.

The existing code is...

_for a_file _over sw_module_manager.module(_self.module_name).resource_files(:data, _true).fast_elements()
_if a_file.index_of_seq(xml_file_name) _isnt _unset
fn << a_file

... and can be replaced with ...

fn << smallworld_product.get_data_file(xml_file_name,office_upgrade_runner.module_name)

Not only is the latter easier to read than the former, the former does not actually return the expected files.[Note: I spoke too soon. The former code does work but in my opinion is not as clean.] Additionally, the former bit of code calls :resource_files() which is a restricted method. The latter code calls :get_data_file() which is classified as advanced and may legitimately be called from other methods.

FME Odyssey Slides Now Available

Safe Software staff recently did their '2010: An FME Odyssey' tour of select North American cities. I was fortunate to attend the one in Denver. This was my first time at a FME User Group meeting and found it very helpful and informative. The interaction with the Safe staff was great. I also found very helpful the presentations given by real FME users explaining how they were using FME to solve a particular problem. You can find all the presentation slides here.

While this blog is focussed on Smallworld, I would still like to suggest that even the presentations that do not cover Smallworld would provide benefit to Smallworld users because they describe how problems in various industry segments were solved using FME. The Safe folks liked to say "data format is irrelevant" when it comes to FME. I agree heartily and look forward to discovering ways that FME can provide business benefit to customers that store their data in Smallworld.

Tuesday, March 23, 2010

Solving the Travelling Salesman problem with Smallworld and Cloud Resources

Here's a tool that can solve the Travelling Salesman problem in Smallworld. It makes use of Smallworld features/trail to create a list of points to visit and then uses a combination of cloud-based services as well as other Magik classes to suggest a more efficient ordering of the points.

(Disclosure: I work for iFactor Consulting and have been involved in the development of the Web Maps Connector that the above-mentioned link references)

Friday, March 5, 2010

Emacs 23

My friend Igor Lapitski recently announced that he was releasing his customizations for Emacs 23. I started using it this week and the most noticeable feature is the slick auto-completion functionality. As you start to type variable names or method names in a .magik file, the system tries to do auto completion based on both previous examples of that string in the current file and matching method names based on the method finder for that image. It works surprisingly fast and gives good results.

There are a lots of other cool features in this release. You can download it from here.

As I find other ones that stick out, I will post about them, too.

Wednesday, March 3, 2010

Smallworld EMEAI 2010

I arrived at the conference venue today. Nice weather in Noordwijk (on the beach south east of Amsterdam).
My main purpose here this week is to demonstrate the Web Maps Connector (the iFactor Consulting product that lets you view Bing Maps as a dataset in your Smallworld applications). I am fortunate to be hosted by the team from Realworld. You can see my demos at the Realworld booth starting at the reception this evening.

I also have interests in Smallworld/FME integration, visualization techniques, and any cool Magik code my colleague Graham has put together. (If you are a Magik GUI developer and you haven't tried his Dialog Designer you owe it to yourself to see a demo). Please stop by the booth and introduce yourself. I am very excited about being at this conference and look forward to meeting many of you.

Friday, February 12, 2010

When a foot is not a foot

When integrating FME input/output with Smallworld there is one unit of measure that you need to be particularly wary about... the "foot". Depending on whom you are talking to a foot can be different things.

Consider this...

When people talk about FME calls it Smallworld calls it
International Foot IFOOT :feet
US Survey Foot FOOT :usfeet

This is important in two places when setting up your FME mapping files:
  1. Finding the correct FME coordinate system for the non-Smallworld end of the mapping file
    • If a customer tells you that they are giving you data to import into Smallworld and its units is "feet" make sure that you clarify whether it is in International Feet or US Survey Feet.
    • Looking at the FME coordinate system browser, I notice that many of the State Plane Coordinate Systems use the US Survey Foot.
    • You will need to clarify the same question when a customer asks you to export data from Smallworld. Typically, though, they will ask you for the data in a particular coordinate system and FME will know what units comprise that coordinate system.
  2. Finding the correct FME coordinate system to match your current Smallworld application coordinate system
    • Probably the biggest confusion of feet/usfeet/internationalFeet comes when you are trying to configure an FME coordinate system to match your Smallworld application coordinate system. You need to be absolutely sure that you are using the same "feet" unit in the FME version of the Smallworld application coordinate system.
    • Most application coordinate systems use "cm" or "mm" for units so you won't run into this confusion. Even in US installations where units are measured in feet and inches, the application coordinate system and database coordinate systems are often set to metric units. (Remember that what a user sees on the GIS can be set to show Imperial units even if the underlying coordinate systems use metric units).
A classic sign that you are dealing with the Foot/USFoot issue is if you try loading in data from an external source and it looks like it "almost but not quite" lines up with your existing data. In that case you should carefully review all the coordinate system projections in your data flow process to ensure that the correct "feet" units are used throughout.

Wednesday, February 10, 2010

'2010: An FME Odyssey' coming to a North American city soon

Safe Software is promoting the recent release of FME 2010 with a series of one-day complimentary events in cities throughout Canada and the USA. Check out the full-day program overview here. I plan to attend the session in Denver on March 24. I hope to see some of you there!

Thursday, February 4, 2010

Smallworld Product Release History

I recently received a note from a reader of this blog asking if I knew the history of the Smallworld Product release versions and dates. I have never kept close tabs on that so I thought it would be interesting to research it a bit. Thanks to the folks at GE for providing this information!

Source: GE's Americas Customer Support Organization


Release Date

Core Spatial Technology (GIS)

Smallworld 2.0(0)

July 1993

Smallworld 2.0(1)

August 1993

Smallworld 2.0(2)

December 1993

Smallworld 2.0(3)

January 1994

Smallworld 2.0(4)

March 1994

Smallworld 2.1(0)

November 1994

Smallworld 2.1(1)

March 1995

Smallworld 2.1(2)

January 1995

Smallworld 2.1(3)

March 1996

Smallworld 2.1(3)NT

May 1996

Smallworld 2.2(0)

December 1996

Smallworld 2.1(3) G

April 1997

Smallworld 2.2(1)

September 1997

Smallworld 3.0(0)

September 1998

Smallworld 3.1(0)

November 1999

Smallworld 3.1(0)SP1

March 2000

Smallworld 3.1(0)SP2

August 2000

Core Spatial Technology 3.2

June 2001

Core Spatial Technology 3.2.1

November 2001

Smallworld Core Spatial Technology 3.3

June 2002

Smallworld Core Spatial Technology 4

December 2003

Smallworld Core Spatial Technology 4.1

December 2006

Smallworld Core Spatial Technology 4.1.1

February 2008

In addition to this, a conversation I had with Peter Batty recently brought up the fact that he had worked on a version 1.9.6 back in 1992.

A lot has changed in Smallworld in the 20+ years of its existence. In my opinion, the product only gets better with each release. I'm looking forward to seeing more of version 4.2 at the EMEAI Conference.

Wednesday, February 3, 2010

Going to 2010 Smallworld EMEAI Users Conference

I am very excited to announce that I will be attending the 2010 Smallworld EMEAI Users Conference in Noordwijk, The Netherlands this March. I have been to quite a few Smallworld Users Conferences in the Americas but never in EMEAI. For those of you not familiar with the acronym, EMEAI stands for Europe, Middle East, Africa, India.

The kind folks at Realworld Systems have offered me a spot in their booth to demo the iFactor Web Maps Connector product. If you are interested in other iFactor product offerings, I will also be able to demonstrate or discuss them.

Aside from those product-demo goals, if you would like to talk about Smallworld/FME, SmallworldCamp, Visualization, or just introduce yourself, please come find me at the Realworld booth.

And if I have time, I think it would be fun to do some informal code demoing and sharing. Kind of like a mini BarCamp at Smallworld. Please send me a note if you are interested in this kind of informal sharing and we can coordinate a meeting time.

Over the years I have made many contacts in the Smallworld realm in the EMEAI region but have never met them in person. I'm hoping that on this trip I will be able to meet many of you and also make new acquaintances.

See you in March!

Monday, January 18, 2010

Visualization in Smallworld

(the cool stuff/demo is at the end of this longer-than-normal post)

Lately I have taken an interest in Visualization techniques within the Smallworld context. I am using the term "Visualization" in the sense of using visual representations to simplify complex data relationships. The term has various other related meanings as well.

In a sense, we work with Visualization techniques in Smallworld every day... we deal with a visualization of geographic relationships between objects that are of interest to us. When you create a report using the Schematics Generator you are using visualization techniques to help you comprehend complex relationships by keeping only the salient information. Any Thematic Mapping tool that you might use also lets you visualize complex data relationships easier.

I have been intrigued by the use of heat maps to provide a way of showing intensity of some attribute spatially. I still haven't got my mind around all the applications that might be used in the utility space for heat maps. Some that come to mind are: heat map of all the transformers that are about to blow up; heat map of all the outages in a storm. While the pictures look cool, I'm trying to better understand what value a heatmap image would bring to GIS users that cannot be already provided by other kinds of thematic mapping applications.

I have a feeling that the full potential for visualizations such as heat maps and density maps has not been realized within the Smallworld realm but I sure could use some ideas from readers about what possible applications might be.

My colleague Graham Garlick has done alot of work with generating heat maps, density maps and TIN density maps within Smallworld and I want to put a plug in for him in this post. If you are interested in knowing more about how to generate these kinds of maps in Smallworld, please contact iFactor Consulting and ask for Graham's heat map demo.

So now to the final point of this post. The visualizations that I have already referred to are typically georeferenced rasters that overlay on a map and give you a sense of the spatial relationships of your data. I have been exploring another area that involves network visualizations.

My first foray into this realm was by making use of a software package called Graphviz to visually show the results of network follower trace results in PNI. This was really helpful in debugging network follower results. Typically, a Magik developer debugs network follower trace results by querying the nf_links of a network follower and trying to understand how they are linked. Using a visualization tool like Graphviz allowed me to easily create a .jpg file that I could view in any image viewer. By looking at the image output, I could easily spot places where the trace stopped unexpectedly or went down an unexpected branch. Sometimes a picture really is worth a thousand words (or a thousand minutes of Magik debugging!!!)

My second attempt at network visualization came from a slightly different tack. If visualization techniques are meant to represent complex interdependencies in an easy-to-understand manner, what is a really complex difficult-to-understand scenario in Smallworld that I could try to visualize? The answer to that question is: XML configuration file dependencies. Anyone that has ever tried to modify config.xml, gui.xml or a host of other XML configuration files in Smallworld will tell you that it can be somewhat mystifying as to which version of the file to change. Any given XML file can inherit from another XML file and its elements can supersede the same elements of a "superclass" XML file. In addition to that, XML files are considered module resource files. That means that they can be placed in customisation products which have a different pathname than the same module found in the core product. If this all sounds confusing, rest assured it is. I have seen many instances (and caused some myself) where I have changed the wrong XML file or changed too many "inheriting" XML files when it would have been more efficient to change a "superclass" XML file. But it was always difficult to get my mind wrapped around the big picture of XML file dependencies. Until now...

I have take a bit of time to write a Magik wrapper around the Graphviz engine. That in itself is not that remarkable and anyone with interest in that could do it. What is of more interest to me is the result that I was able to create with this Graphviz tool showing the interdependencies of XML files. As an example, click here to see the dependencies of the Electric Office config.xml files for applications.

  • If you look at the image, you will see information about which XML files inherit from others.
  • You will see the full pathname of each file. This will help you know whether to look in the actual module resource file or in a corresponding customization product file
  • You will see some cells that are green. These are the cells that represent XML file elements that have a material change relative to a "superclass" file.
  • You will see some cells that are red. These are cells that represent XML elements that exist in a configuration file but whose contents are no different than the same element in the inherited XML file. Many times it can be confusing to have the exact same XML element contents in two files, so looking at this visualization gives you an easy-to-understand representation of which duplicate XML elements you can remove.
  • You will see some cells that are yellow. These represent XML elements that require further investigation. They indicate that the current XML element is an exact duplicate of another XML element somewhere in the graph. If you notice yellow cells in two XML files that share the same parent, you might want to consider moving those XML elements up the inheritance ladder.
  • And finally, just seeing the overall flow of XML inheritance can give you a quick indication of whether you have too many dependencies or maybe where it would make the most sense to insert a suitable XML file "superclass" that would serve your own customizations most efficiently.
This was just a quick demonstration of how a network visualization tool can be used to simplify complex interdependencies in the Smallworld environment. I would love to hear from readers about other complex scenarios that might lend themselves to this sort of network visualization.