Exporting internal world geometries from Smallworld using FME

I recently had a reader ask me...

I exported from Smallworld into ESRI shape format using FME. However, I noticed that the only features/collections that have a shape file output are the objects in geographic world. I checked the logs of objects in internal world but the log shows that the features were skipped. I would like to ask if you know the approach in these types of objects.

My response...

Have a look at method fme_tics_client.int!create_feature()

You will see a line that looks like...

geoms << rec.geometry_set_for_world(world)

... that is the one that is causing only geometries in the main GIS world to be considered.

You could try replacing that line with something like this...

geoms << geometry_set.new_from_iter(rec,:all_geometry|()|,_false)

... I haven't tested this, but I expect it might work. Use at your own risk :)

Exporting Node Coordinates via FME

I received a query today from a reader that wants to export the node coordinates for each electric conductor sent to FME. The sw_gis!node table is not a user table in Smallworld and is therefore not typically exposed to the FME plugin. But if you use the GE FME plugin, you can make use of the pseudo_fields functionality to gather up the relevant node information and send it to FME as a conductor attribute.

The basic steps are:

1. define pseudo_fields on the conductor class (or whatever class you are interested in)
2. write Magik support code that populates the pseudo_fields with appropriate node-based coordinates.
3. create a FME Workspace that reads the new pseudo_fields and processes the appropriately.

This video describes these steps in more detail.

Creating a 3D PDF from Smallworld

Lassi Tani (@Lassitani) has posted a very cool example of creating a 3D PDF visualization of Smallworld data using FME at http://geolassi.wordpress.com/2010/10/28/from-smallworld-to-3d-pdf/

Using #FME to mash open data into #OSM addresses

While this video is not strictly related to Smallworld, it does describe mashing open data from various sources into something usable in a destination database. The "Big World" forum at the #swuc recently covered the area of open data and I thought this might be of interest to some of you out there.

In this case, the demo takes open parcel and address information from the Boulder City and Boulder County (Colorado, USA), processes it with FME and writes data into a format that can be uploaded into OpenStreetMap for the world to see.

Should you attend FME Users Exchange #swuc

If you are involved with moving data in or out of Smallworld I maintain that the most powerful way to do that is using Safe Software's FME product. If you are already using FME or if you are just wondering how FME can help you integrate Smallworld data with other data formats, I recommend that you attend the FME Users Exchange on Thursday, September 9 from 1pm-4pm in Baltimore at the site of the Smallworld Americas 2010 Users Conference.

There will be great presentations by customers, GE and Safe representatives. I will be presenting on how to enhance your GE FME-Smallworld Magik interface to let you do things like updating and deleting records and also creating Smallworld joins via FME.

As a bit of a trailer, here is a 5 minute video that highlights some of the items I will be presenting in more detail (video: http://bit.ly/ainbzt)

You can see my prioritized presentation points here: http://bit.ly/9nFvkd

Sign up for the FME Users Exchange at #swuc

The Smallworld-FME Users Exchange is happening at the Smallworld Americas Users Conference in Baltimore, MD this next Thursday, September 9 from 1:00-4:00pm. It will be in the Key Ballroom 4, Hilton Ballroom. If you are a Smallworld/FME user, I would highly recommend that you attend this event. There will be a customer case study, some best practices and demos and also a product roadmap presentation from GE.

You can find out more information about it here: http://bit.ly/cvok0h

#FME Users Exchange at #swuc topics

The FME Users Exchange at the 2010 Smallworld Users Conference will take place in Baltimore, Maryland on Thursday, September 9 from 1pm to 4pm (EDT). You can read more details of the agenda at http://bit.ly/cvok0h.

I will be presenting a section on "Best Practices: Tweaking FME-Magik code to improve your FME experience in Smallworld". It turns out that I have more material to cover than the approximately 30 minutes allotted to me. So in the spirit of conference collaboration, I thought I would put all my proposed topics up for a vote. The topics with the most votes get moved to the top of my presentation list. The ones with the least votes get used as filler at the end :)

Please vote on the topics here http://bit.ly/9nFvkd ( you are also welcome to add your own topics).

See you in Baltimore!

FME Import Zoom to Smallworld Map

In the SWAF version of the Smallworld-FME plugin, it would be nice to have the current Smallworld map "goto" the bounds after the FME import has completed. This comes in handy when you want to easily find your just-imported data for subsequent review.

Apparently this functionality used to exist in the pre-SWAF version of the FME plugin.

I have uploaded a sample of some tweaks you can make that will enable this functionality at Smallworld version 4.x (SWAF).

The basic steps are:

1. in fme_tics_client.private_run(), determine if there are any smallworld_product.applications that might be interested in knowing where the FME-imported geometry bounding box is
2. if there are any such applications, then create a dynamic variable geometry_set that fme_tics_client.int!received_feature() can add newly-created geometries to
3. back in fme_tics_client.private_run(), send a :goto_request databus message to all the relevant applications with the geometry_set populated in fme_tics_client.int!received_feature()

Have a look at this file (fme_goto_import_geometry_set.magik) to see the details of how it is done.

FME SmartCleaner makes things easy

I recently had a customer requirement to:

1. read Lot lines and lot labels as two separate layers from a single DWG file
2. where possible, stitch the lot lines together to make a polygon
3. where possible, figure out which lot label is inside each of the newly generated polygons and aggregate those two geometries into one RWO in Smallworld

It turns out that step#3 can easily be done with PointOnAreaOverlayer and Aggregator.

But step#2 didn't seem that easy. I had tried Intersector, TopologyBuilder and Snapper but I never seemed to be able to get the over/under shoots to snap. Then I remembered to check http://fmepedia.com and discovered that there is a custom transformer that does exactly what I needed. It is called the SmartCleaner and is described here.

You can choose to either link or embed this Custom Transformer. I chose to embed it into my existing workspace. This made it transportable across user machines without having to ensure that each user explicitly installed the custom transformer.

When I looked at the internal workings of the SmartCleaner, I realized why my efforts to use Intersector and Snapper were inadequate. The following screen snapshot shows the complexity of the SmartCleaner.

All I can say is a hearty "thank you" to Dave Campanas at Safe Software for creating this Custom Transformer and making this over/under shoot snapping so easy to implement.

Configuring FME to Write Multiple Datasets More Efficiently

This post by the FME Evangelist describes how to configure your Workspace so that a minimum number of features are cached in memory. The less amount of caching, the more efficient your workspace. This applies to any workspace that has more than one writer (aka destination) dataset in it. Always put the dataset that has the potential to contain the most features to the top of the writer list.

The post gives a more detailed description.

FME and Smallworld Text Join fields

Smallworld ignores data from FME that is destined for text_join fields.

Smallworld has a data type called "text_join" which allows you to store a string of indefinite length on a Smallworld record. Under the hood, this data type is actually modeled as a join relationship to an internal table that stores chunks of finite-length strings that can be added to each other to emulate an indefinite-length string. So a text_join is actually considered a join field and not a physical field.

This becomes a problem when you want to copy data into one of these text_join fields. This is not an unusual requirement. Many Smallworld objects have at least one text_join field (e.g., Electric Office objects have a "Remarks" field; other datamodels might have a "Comments" field). Not being able to import attribute values via FME into those Smallworld objects is not an insignificant limitation.

I raised a case with GE about this recently and was told that this was a duplicate of a same case that I had raised a few years ago. I knew that there had been something familiar with this issue, but I couldn't remember that I had already encountered it a few years ago. Because there is no patch in sight any time soon for this issue, I present to you the fix I made. Like most of the FME/Magik tweaks I have had to make, the implementation is fairly easy.

(click on image to enlarge)

Debugging Smallworld-FME Error Messages

Sometimes, when I am importing data into Smallworld using FME, I get a Smallworld error message in the FME log. Unfortunately I get only the condition string and not the accompanying traceback so it is usually very difficult to debug. The traceback is also not written out to the Magik standard output.

This simple hack will cause any error conditions raised during an FME import to have a traceback dumped to the Magik standard output.

(click on image below to enlarge)

PDF as Export Format (not just for plotting anymore)

You are all aware that Smallworld has the ability to print maps to PDF.

Now, with FME, you can export Smallworld data to PDF. In this example I:

• Imported some Cambridge demo db data (substations, cables, supply points)
• Imported a Google Static Map based on the bounds of the Smallworld data
• Exported to Geospatial PDF format which is readable with Adobe Reader

I apologize for not providing the actual files, but it seems that my online file repository is no longer available. I will include screen shots instead. You can click on the image to get the full size.

FME workspace...

Adobe Reader. Note the ability to show attributes for selected objects. You can also do "goto selected object".

This was just a very simple example of mashing together data from multiple sources (including Smallworld) and exporting to a format that even "Ken's grandpa can read*".

Adobe Reader is free and widely available on most computers. It seems to me that this would be one format that would lend itself well to an inexpensive mobile field solution.

* inside joke from the FMEOdyssey 2010 meeting in Denver in March.

Converting Coordinate System From Smallworld to FME

When setting up an FME Workspace to interact with Smallworld, you always want to be sure that your SWORLDSWAF reader/writer coordinate system is the same as your Smallworld session's Application Coordinate System (see discussion here). The post "Smallworld/FME Configuration Tips and Tricks" has a link to a slideshow that describes how to make sure that your Smallworld Application Coordinate System and FME SWORLDSWAF reader/writer coordinate systems match . The relevant information starts on slide 31.

I recently discovered another way of figuring out the FME COORDINATE_SYSTEM_DEF that corresponds to a Smallworld coordinate_system.

The FME Workbench's CoordinateSystemDescriptioConvertor Transformer (see example here) can take a Proj.4 coordinate system definition string and convert it into a suitable FME definition.

And how do you get the Proj.4 definition string? Easy... just use the following method.

_pragma(classify_level=basic)
_method coordinate_system_mixin.proj4_string
 ## proj4_string : string
 ##
 ## returns a string that represents self in the Proj.4 format.
 ## This can be used by coordinate system converters to convert
 ## self into other representations.

 # according to the comments in
 # coordinate_system_mixin.new_proj():  "PROJ_TYPE is a symbol
 # specifying the projection type. The naming convention is
 # similar, but not identical, to that used in the USGS Proj4
 # library. See below for details."
 #
 # because the Magik names are not always identical to the Proj4
 # names, we provide a lookup table here.
 _local projection_name_lookup << hash_table.new_with(:accurate_tmerc,:tmerc,
            :accurate_utm,:utm)
 
 _local (proj,params) << _self.proj_type_and_params

 _local info << rope.new()
 info.add_last(write_string("+proj=",projection_name_lookup[proj].default(proj)))

 _for k,v _over params.fast_keys_and_elements()
 _loop
  info.add_last(write_string("+",k,"=",v))
 _endloop

 _local int_str << internal_text_output_stream.new()

 int_str.write_list_with_separator(info,space_char)

 _return int_str.string
 
_endmethod
$

The next step will be to write a Magik method that actually converts the Smallworld coordinate_system definition directly into the FME COORDINATE_SYSTEM_DEF string all within Magik. That should allow for some greater flexibility in automating various Smallworld/FME processes.

Enjoy!

FME Odyssey Slides Now Available

Safe Software staff recently did their '2010: An FME Odyssey' tour of select North American cities. I was fortunate to attend the one in Denver. This was my first time at a FME User Group meeting and found it very helpful and informative. The interaction with the Safe staff was great. I also found very helpful the presentations given by real FME users explaining how they were using FME to solve a particular problem. You can find all the presentation slides here.

While this blog is focussed on Smallworld, I would still like to suggest that even the presentations that do not cover Smallworld would provide benefit to Smallworld users because they describe how problems in various industry segments were solved using FME. The Safe folks liked to say "data format is irrelevant" when it comes to FME. I agree heartily and look forward to discovering ways that FME can provide business benefit to customers that store their data in Smallworld.

When a foot is not a foot

When integrating FME input/output with Smallworld there is one unit of measure that you need to be particularly wary about... the "foot". Depending on whom you are talking to a foot can be different things.

Consider this...

	When people talk about	FME calls it	Smallworld calls it
	International Foot	IFOOT	:feet
	US Survey Foot	FOOT	:usfeet

This is important in two places when setting up your FME mapping files:

1. Finding the correct FME coordinate system for the non-Smallworld end of the mapping file
   • If a customer tells you that they are giving you data to import into Smallworld and its units is "feet" make sure that you clarify whether it is in International Feet or US Survey Feet.
   • Looking at the FME coordinate system browser, I notice that many of the State Plane Coordinate Systems use the US Survey Foot.
   • You will need to clarify the same question when a customer asks you to export data from Smallworld. Typically, though, they will ask you for the data in a particular coordinate system and FME will know what units comprise that coordinate system.
2. Finding the correct FME coordinate system to match your current Smallworld application coordinate system
• Probably the biggest confusion of feet/usfeet/internationalFeet comes when you are trying to configure an FME coordinate system to match your Smallworld application coordinate system. You need to be absolutely sure that you are using the same "feet" unit in the FME version of the Smallworld application coordinate system.
• Most application coordinate systems use "cm" or "mm" for units so you won't run into this confusion. Even in US installations where units are measured in feet and inches, the application coordinate system and database coordinate systems are often set to metric units. (Remember that what a user sees on the GIS can be set to show Imperial units even if the underlying coordinate systems use metric units).

A classic sign that you are dealing with the Foot/USFoot issue is if you try loading in data from an external source and it looks like it "almost but not quite" lines up with your existing data. In that case you should carefully review all the coordinate system projections in your data flow process to ensure that the correct "feet" units are used throughout.

'2010: An FME Odyssey' coming to a North American city soon

Safe Software is promoting the recent release of FME 2010 with a series of one-day complimentary events in cities throughout Canada and the USA. Check out the full-day program overview here. I plan to attend the session in Denver on March 24. I hope to see some of you there!

How do interact with FME in Smallworld?

I was asked a question yesterday at the conference about how I interact with FME from Smallworld and thought I would start an informal survey of readers to see what your responses are.

The question is: How do you interact with FME from Smallworld?

1) Do you see FME as an application that is launched from a Smallworld session?

2) Or do you prefer to use FME more as a standalone tool that connects to a Smallworld server process?

My own preference is to see FME as a standalone tool that makes a connection to a Smallworld server. That way any business analyst that does not have a Smallworld session running could still make a connection to a central Smallworld server port and run the workspace. Having this configuration would also facilitate use of FME Server with Smallworld. This method of interaction would be great for reading from Smallworld. But I suppose that writing to a Smallworld database might make one lean towards option#2.

With option #2, you as a Smallworld user could control which alternative the data was being written to and then you could do a final check of the data in your alternative before you committed it.

So I suppose that my preferred mode of operation includes both above-mentioned options. A "FME connects to Smallworld server" option for reading data and for selected controlled write operations. And a "Smallworld launches FME" option for the more ad hoc write operations.

I'm curious what your thoughts are.

Exposing hidden data from Smallworld to FME

A member of the sw-gis group recently posted a question asking how to export join attribute information from Smallworld via FME. I responded to the group but thought the answer might be worth sharing here as well.

Question:

I'm using FME with Smallworld 4.0. How can I export join attributes (e.g., export transformer bank id with the transformer)? I coudn't find it in the FME workbench.


Answer:

I think the easiest way to do this is to use the fme_pseudo_field_factory.

For example...

If you’re your record exemplar is :transformer and the transformer_bank ID attribute is called :transformer_bank_id, then you can define a new FME Pseudo Field...

_pragma(classify_level=basic, usage={redefinable})
## used by FME SWAF interface to allow a customizer to add
## pseudo fields to the schema for Smallworld tables to be
## passed to FME.
transformer.define_shared_constant(:fme_pseudo_fields,
                                 {fme_pseudo_field_factory.new_derived_field(:transformer_bank_id,:extdb_string)},
                                 _false)
$

You can actually define more than one fme_pseudo_field for a given record exemplar if you want to.

Declaring :transformer_bank_id as a new derived field will make this "field" available to the FME feature. When FME asks a "transformer" for its :transformer_bank_id, it will simply call transformer_bank_id and use that value in the workspace.

There is nothing special about the naming convention of the pseudo field. The only thing to keep in mind is that the class on which you define :fme_pseudo_fields must respond (either as a method or field call) to the name of the derived field.

So you could actually accomplish the same thing as above with...

_pragma(classify_level=basic, usage={redefinable})
## used by FME SWAF interface to allow a customizer to add
## pseudo fields to the schema for Smallworld tables to be
## passed to FME.
transformer.define_shared_constant(:fme_pseudo_fields,
                                 {fme_pseudo_field_factory.new_derived_field(:bank_id,:extdb_string)},
                                 _false)
$

As long as you also have defined a method...

_method transformer.bank_id
      ## bank_id : integer
      ##
      ## returns self’s transformer_bank_id

      _return _self.transformer_bank_id
_endmethod
$

The FME fme_pseudo_fields concept is very powerful because it allows you to send data to FME that is not stored in a single attribute in Smallworld.

I have used it in the past for exporting data from Physical Network Inventory (PNI). If you are familiar with that datamodel you will know that there are many layers of relationships. Some of the records have geometries and some do not. A customer wanted to export all underground and aerial routes (these are the features with geometry) but wanted to label the features in the exported format with data from a related non-RWO record. Using the fme_pseudo_fields approach made it very easy to create a Magik method that created a suitable labelling string based on the related records but expose the results to FME as a feature attribute.

Configuring the GE Smallworld FME Plugin as a Server

Anyone that has done any work with GE's version of the FME Smallworld Reader/Writer will know that it does not behave well as a stand-alone server.  So you are faced with one of two options:

- purchase SpatialBiz' FME Plug-in.  It has been designed as a server process.

- modify your existing (i.e., already paid-for) GE Smallworld Reader/Writer using a combination of Python, TICs and a few Magik enhancements to allow the FME Workspace to dynamically control the fme_tics_client class.

You can find a more detailed description by look at the "Behind the Scenes" section at http://evangelism.safe.com/smallworld2008.

Here are the steps you would need to take:

- create a Python Module that exposes Tics32.dll functionality

- extend the Python Module to expose various Magik methods such as setCollections(), rollback(), rollforward(), gotoAlternative(), etc.

- modify fme_tics_client.private_run() to listen for TICS messages from the new Python module.

- configure your FME Workspace to use the new Python Module.

I have prepared a package (licensed under LGPL) of Python and Magik code (along with a workspace sample) that I am happy to send out upon request.  Thanks to my employer (Red Planet Consulting, Inc.) for permitting me to make this available to the Smallworld community.

Send me a note at... 

...to request a free copy of the Python, Magik and FME Workspace files I used to integrate FME Server with the GE Smallworld FME Reader/Writer.