Tuesday, November 11, 2008

Dialog Designer v1.8 now available

Graham Garlick over at iFactor Consulting has uploaded v1.8 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:
  • autosave a copy of current XML dialog description to the generated module.
  • dt_dt language added (thank you Frank Wolff @ GE)
  • added button to refresh of icon list, to capture newly loaded modules
  • expose the double_click_notifier attribute for tabular_list and tree_list

Thursday, October 30, 2008

Empty iterator still runs _finally

In my years of doing code reviews I have always gotten on people's cases for writing...

_if _not some_set.empty?
_for a_value _over some_set.fast_elements()

... what's the point, I asked, of asking SOME_SET if it is empty? If it is empty, then the loop won't be entered anyways and now you are just cluttering up the file with redundant code. It seems that the following should be equivalent...

_for a_value _over some_set.fast_elements()

Well, today I learned that those two statements are not equivalent. If you use the _finally keyword, its scope will always be processed, even if the iterator has not even run a single loop.

MagikSF> _for a_value _over {:a,1,"bb",300}.fast_elements()
MagikSF> _for a_value _over {}.fast_elements()

So lesson learned... if your _finally block of code expects some value to be set in the main _loop block, be sure to put error handling in there for the case where the loop had no iterations to process.

Monday, October 6, 2008

Configuring the GE Smallworld FME Plugin as a Server

Anyone that has done any work with GE's version of the FME Smallworld Reader/Writer will know that it does not behave well as a stand-alone server.  So you are faced with one of two options:
- purchase SpatialBiz' FME Plug-in.  It has been designed as a server process.
- modify your existing (i.e., already paid-for) GE Smallworld Reader/Writer using a combination of Python, TICs and a few Magik enhancements to allow the FME Workspace to dynamically control the fme_tics_client class.

You can find a more detailed description by look at the "Behind the Scenes" section at http://evangelism.safe.com/smallworld2008.

Here are the steps you would need to take:
- create a Python Module that exposes Tics32.dll functionality
- extend the Python Module to expose various Magik methods such as setCollections(), rollback(), rollforward(), gotoAlternative(), etc.
- modify fme_tics_client.private_run() to listen for TICS messages from the new Python module.
- configure your FME Workspace to use the new Python Module.

I have prepared a package (licensed under LGPL) of Python and Magik code (along with a workspace sample) that I am happy to send out upon request.  Thanks to my employer (Red Planet Consulting, Inc.) for permitting me to make this available to the Smallworld community.

Send me a note at... 

...to request a free copy of the Python, Magik and FME Workspace files I used to integrate FME Server with the GE Smallworld FME Reader/Writer.

Sunday, September 28, 2008

FME Evangelist features Smallworld and FME Server

So this is what humidity feels like.  I just arrived in Hollywood, FL (USA) an hour ago having left Boulder, CO a few hours before.  Going from 1 mile (and dry) to sea level and humid is quite the change.  

I'm looking forward to seeing many from the Smallworld community at the conference on Monday and Tuesday.   Please stop by at the Red Planet Consulting booth and say "hi".  Me and John will be the two guys dressed a in mechanics shirts.

Mark Ireland the "FME Evangelist" has a Smallworld User's Conference Special Edition article giving some added detail about the FME Server on Smallworld that Mark Stoakes and Michael Weisman from Safe Software and I will be demonstrating this week.  In addition to the links to the on-line demo videos from a few weeks ago, Mark also provides a bit more detail about the Python functionality that communicates with the Smallworld server image.

Friday, September 12, 2008

Viewing Hurricane Track in Smallworld using FME and SOMs

I realized today that I have customers and friends in the track of Hurricane Ike.

View Larger Map

So I was thinking... what would it take to view hurricane track information in Smallworld?

Here is what I came up with...
  1. Figure out which Smallworld SOMs (Spatial Object Managers) you have licensed.
  2. Create a FME workspace that reads hurricane data from a provider (eg., Google has http://mw1.google.com/mw-weather/tropical/kml/all_hurricanes.kmz) and converts it into a format that your SOMs are licensed for.   You can download the workspace file here (right-click and Save As... the link)
  3. Create a .bat file that calls your FME workspace and can be scheduled using a task scheduler.
    fme.exe %~dp0Hurricane_KML2MapInfoTab.fmw --SourceDataset_KML21 http://mw1.google.com/mw-weather/tropical/kml/all_hurricanes.kmz --DestDataset_MITAB %~dp0
  4. Create Smallworld SOM configuration that reads the regularly-updated SOM data.
  5. Create Smallworld ACE and Style changes that render the storm data on Smallworld.
I have tried this preliminarily with MapInfo and it seems to work.  What is not clear to me yet is if the MapInfo SOM will refresh the data on Smallworld each time the underlying MapInfo data is refreshed.  Anyone interested in trying this?

[UPDATE: It seems that once a MapInfo SOM is connected to a MapInfo file, then that file is locked for other processes.  Ie., FME won't be able to update it.  So that is likely not a very good idea.  We could use the WMS SOM but then we would need to set up a WMS server.  You can definitely do that with FME Server but I am willing to bet that most Smallworld customers do not have FME Server set up yet.  Maybe another option would be to modify the FME Workspace to write out to a text (or XML) file that could then be read by a Smallworld post-render-set plugin that would then dynamically draw the storm track information each time the map is refreshed.]

Anyone doing something else with tracking storms in Smallworld?

Stay safe and dry.

Monday, September 8, 2008

FME Server on Smallworld Demo

As mentioned in a previous post, Safe Software will be presenting a hands-on FME Server workshop at the Smallworld 2008 Americas Users Conference.

In anticipation of that workshop I have created a number of screen casts of a demonstration of how we at Red Planet Consulting [NOTE: I no longer work for Red Planet. I now work for iFactor Consulting] put together Smallworld with FME Server to download and upload data using web clients. I would be happy to discuss any questions you might have about the contents of this demonstration.

This configuration uses GE's Smallworld FME Plugin. Out of the box the plugin does not exhibit "server"behaviour. But with a few Magik and Python tweaks we got it to work.

Click on the following links to launch Flash videos of the various parts of the demo:

  1. Start SIAS Server
  2. Start SIAS Client
  3. Download without Bounding Box
  4. Download with Bounding Box
  5. Upload Isle of Wight

Smallworld 2008 Americas Users Conference

I am looking forward to the upcoming Smallworld 2008 Americas Users Conference.  I will be there presenting and helping with a FME workshop and, of course, staffing the Red Planet Consulting booth.

I will be giving a presentation (Using FME as a Smallworld Spatial Analysis Tool) on Monday afternoon from 4:30-5:15.

I also wanted to mention that the author of Dialog Designer (now at v 1.7), Graham Garlick, will be presenting his very cool open-source Magik GUI development tool.  Unfortunately the agenda does not explicitly mention his name nor his topic, but you will find him presenting in the "Taming Graphics: Improving yourMagik draw, drop and drag" hands-on workshop on Tuesday morning from 8:30am-10:00am.

On Tuesday late morning (10:30am-noon) I will be in the "FME and Smallworld: Integrating and Distributing Spatial Data with New Server-based Spatial ETL" workshop.  Mark Stoakes from Safe Software will be leading the hands-on workshop allowing users to try their hand with the new FME Server product connected to the Smallworld Cambridge database.  I will be starting off the workshop by demonstrating using SIAS to download/upload data via the FME Server interface.

So, I hope I get to meet many of you at the conference.  Look for me at the Red Planet booth or in the conference sessions.  You will find me wearing one of those trademark Red Planet mechanics shirts.

Friday, August 1, 2008

Another Magik Blog

Pedro Miranda has started a new blog...

dedicated to Smallworld magik, it will be mainly a place to post assorted bits of useful code, targeting developers....and information regarding SW.

You can find it at http://magikbites.blogspot.com/

Friday, May 16, 2008

Dialog Designer v1.6 now available

Graham Garlick over at iFactor Consulting has uploaded v1.6 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:
  • added outlookbar widget
  • support progress_bar pane type in statusbar
  • support dropdown image button groups within a toolbar
  • click on rendered menubar or statusbar to jump to its editing tab

Sunday, March 9, 2008

OLE_Controller Monitor

Have you ever written Magik functionality that uses OLE/COM and wondered if you are sending :release_object() to all the ole_controller objects? Wonder no longer. Have a look at the ole_controller_monitor class I created recently.

This class allows you to start/stop/reset monitoring on the ole_controller class and report to you how many un-released ole_controller objects still exist in your monitored environment.

This is a great help in cleaning up OLE-based reports (eg., Excel, Visio, Word) and preventing the eventual OLE/memory-related errors that force a user to restart their Magik session.

API and steps to use the class are included in the file header.

Tuesday, March 4, 2008

Smallworld FME 4.1 Interface takes data Back to the Future

I had a colleague point out to me yesterday that if you are using the Smallworld FME interface for CST 4.1 you need to verify that your date (and date_time I suppose) attributes are being exported correctly. After further analysis both I and he discovered that dates prior to 01Jan1970 and after 07Feb2106 will not be copied across correctly. The 07Feb2106 threshold may not be problematic for most Smallworld users but the 01Jan1970 threshold definitely is.

For example, if you have an "installed date" attribute of 04Dec1969 and you extracted it using the Smallworld FME 4.1 interface you would notice that FME recognizes it as 10Jan2106. So your feature ends up going back to the future but your users of the exported data come back to you.

I have raised a case with GE Help Desk on this issue and they are looking into it.

(I have not tested this on the Smallworld FME interface for CST 4.0.)

Friday, February 29, 2008

Magik FME Interface can introduce unwanted data on export

I have always been very interested in using Safe Software's Feature Manipulation Engine (FME) with Smallworld. I think FME is a very cool and powerful tool. Occasionally, though, I encounter problems with the FME Magik interface that I think are worth sharing with the Smallworld/FME community.

The current issue has to do with the FME-Smallworld interface introducing data to FME features that was never part of the Smallworld data to begin with. In the Smallworld VMDS (Version Managed Data Store) it is possible to define a field as "non mandatory". As the mode implies, that means that the data validation code will not require that you populate that field with an attribute value. VMDS does not have a data type called "NULL" like other databases do. So to allow a "non mandatory" attribute value to be represented as "empty" VMDS requires that the data dictionary be defined with an "unset_value" for that attribute. An "unset_value" is a value that will represent "no value" (or NULL; or _unset) for that attribute.

So, for example, if you defined an attribute called "installation_date" as data type :ds_date and you want to enable this attribute as "non mandatory", the CASE tool will prompt you for an "unset value". Actually, it will suggest that you use the "date" Magik class as the unset value. The "date" Magik class happens to be 01Jan1970. This means that whenever you enter 01Jan1970 into the "installation_date" attribute VMDS will interpret that you mean that you are actually assigning _unset to that attribute.

So, getting to the problem with the Smallworld-FME interface. A client of mine discovered some time ago that if you use the FME 3.3 interface and you have one of these non-mandatory date fields and the value of the attribute is _unset, then the Smallworld-FME interface will actually send the unset_value to the export format. That seems very wrong to me. If your Smallworld database states that the "installation_date" is _unset, I do not want my exported-via-FME format to now show that the "installation_date" is 01Jan1970.

Luckily for us, that bad use of unset_value was all coded in Magik and could be patched by us. In FME330 method dd_record_mixin.fme_export() there is a block of code...

#--- Use the default value if available
_if (fdata << _self.perform(fld_name)) _is _unset _andif
fdata << fdesc.unset_value

... that basically says: if an attribute value (fdata) is _unset, then try to send that attribute's field_descriptor's (fdesc) unset_value instead. Wrong! If an attribute value is _unset then the code should send _unset to FME as well. I think it is dangerous for a data mapping interface to introduce data to an export format that does not exist in the source data. If you want to introduce data to the export format that does not exist in the source data then modify your .fme or .fmw files or use the fme_pseudo_field functionality. A user should have the confidence that the source data will be faithfully transmitted to FME without any data massaging.

If you are still with me to this point, you will be happy to know that I am finally getting to the point of this post. The Smallworld FME interface was completely rewritten at CST 4 and I assumed that the problem with ds_date (and ds_time) unset_values had been resolved. It has not! Instead, the problem has been made worse. The logic for deciding whether to default an _unset non-mandatory ds_date/ds_time value to its unset_value has been taken out of Magik and moved into the ACP/DLL files. And the logic no longer even tries to access the field descriptor's unset_value. Now the logic always sends "19700101" as the date string to FME regardless of what your data dictionary specifies for an unset_value.

Hopefully it is clear what this issue means for you. If you have non-mandatory date and time fields in your VMDS database, then all blank values will show up as 01Jan1970 as far as FME is concerned. I suppose you could work around this by modifying your .fme or .fmw files to test for "19700101" and convert it to a blank value. That could work some of the time but what if 01Jan1970 was a valid date in your business. You would never be able to know whether the "19700101" represented 01Jan1970 or _unset.

I have submitted a case report to GE on this issue yesterday and look forward to hearing their response.

If you are using FME at CST 4 (or higher), you can run this block of Magik code to identify a blank non-mandatory ds_date/ds_time attribute in your VMDS database and then send it to FME so that you can use one of the FME tools (I recommend FME Viewer) to see that your blank date/time values in Smallworld have been converted to a non-blank date in FME.

# this block of code works at CST 4.x

# start the fme_tics_server if not already running
_local current_server
_if (current_server << fme_tics_server.current_server) _is _unset _orif
_not current_server.acp_running?
current_server << fme_tics_server.current_server

_local field_type , field_name , a_table , test_record
_constant TYPES << {:ds_date,:ds_time}

# make sure your VMDS dataset is correctly named here.
_constant V << gis_program_manager.cached_dataset(:gis)

# you can restrict the list of table names here to the tables
# that you regularly export to FME...
#_constant TABLE_NAMES << {}
# ... or you can choose to scan all user tables in the current
# collection.
_constant TABLE_NAMES << V.user_table_names

write("looking for a record that has _unset as a value for a non-mandatory date or time field...")

_for table_name _over TABLE_NAMES.fast_elements()
_loop @outside
a_table << V.collections[table_name]

_for fd _over a_table.physical_fields()
field_name << fd.name

_if (field_type << fd.type) _isnt _unset _andif
TYPES.includes?(field_type.name) _andif
_not fd.mandatory?

_for a_rec _over a_table.fast_elements()
_if a_rec.perform(field_name) _is _unset
test_record << a_rec

write("found that '",test_record,"' has a blank value in non-mandatory field '",field_name,"'")
_leave @outside

_if test_record _isnt _unset
write("sending the record to the FME server. Please view the results using FME Viewer. Specifically compare the value of the blank date/time attribute in Smallworld to the value in FME.")
write("Could not find a record with a blank non-mandatory date or time field.")

In my example, the Smallworld record looked like...
:id 1237639
:date_field unset
:time_field unset

But the FME Viewer showed the attributes as...

Note the different values for the date attributes.

Good luck!

Announcing "Programar en Magik de SmallWorld" blog

For Magik practitioners wishing to read about Smallworld Magik in Spanish, Joan in Barcelona has created Programar en Magik de SmallWorld.

Congratulations Joan!

Wednesday, February 6, 2008

CASE Tool out of synch with your actual datamodel

It is possible that your CASE Tool representation of your datamodel might not accurately reflect the actual datamodel you have in your database. This happened to me recently when I discovered that some CASE work I had done in my own development environment had touched more objects than I was working with when I did a CASE Archive. When I loaded the CASE archive into the production database I had not noticed that it also updated some other objects with an older version of the CASE definition.

This was not a problem until someone tried to apply changes for the "unknowingly-touched" CASE object and it introduced some confusion into the production environment. Well, we resolved the issue but then had the question... How do we ensure that the CASE Tool accurately represents the state of the datamodel?

The answer to that question: "Reverse Engineer"

Here are some tips that will help you to reverse engineer a CASE model.
  • read the Smallworld Case Tutorial help "Reverse engineering from the Gis dataset"
  • an important thing to note is that you need to do the reverse engineer action into an empty CASE partition. Use a procedure such as create_database() to create new "empty" case.ds and case_dd.ds files and then update your Case SOC to point to these new .ds files
  • you should also temporarily keep your "old" case.ds and case_dd.ds files. Do this by creating a new SOC (:old_case) that has a :case dataset that points to these old ds files.
  • when you perform the reverse engineer action you will see that the graphics for relationships and drafting_* objects are not visible in the new CASE. You will also see that the CASE objects are not grouped together the way you had them in the original CASE tool. That is because the Gis dataset does not keep information about CASE graphics in its data dictionary.
  • I use a script like the following one to "recover" some of my description geometries from the "old" CASE tool into the new recently-reverse-engineered CASE tool. (Be sure that you have a handle to both the old and new CASE tools for this).

_local new_case_v << gis_program_manager.cached_dataset(:case)
_local old_case_v << gis_program_manager.cached_dataset(:old_case)

_local old_drafting_coll, new_drafting_coll , geometries , properties , new_rec

write("creating CASE drafting_* objects...")
_for drafting_table_name _over {:drafting_lines,:drafting_areas,:drafting_points,:drafting_texts}.fast_elements()
old_drafting_coll << old_case_v.collections[drafting_table_name]
new_drafting_coll << new_case_v.collections[drafting_table_name]

_for old_rec _over old_drafting_coll.fast_elements()
properties << property_list.new()

new_rec << new_drafting_coll.insert(old_rec)

_for a_geom _over old_rec.geometries().fast_elements()
properties[a_geom.app_type] << a_geom

_if properties.empty?


_local new_case_coll << new_case_v.collections[:sw_gis!case_object]
_local old_case_coll << old_case_v.collections[:sw_gis!case_object]

write("updating CASE Object geometries...")
_for new_co _over new_case_coll.fast_elements()
_if (old_co << old_case_coll.select(predicate.eq(:name,new_co.name)).an_element()) _is _unset
show("skipping ",new_co.name)


_local new_case_coll << new_case_v.collections[:sw_gis!case_relationship]
_local old_case_coll << old_case_v.collections[:sw_gis!case_relationship]

write("redrawing CASE relationship geometries...")
_for new_co _over new_case_coll.fast_elements()
_if (old_co << old_case_coll.select(predicate.eq(:object_info,new_co.object_info)).an_element()) _is _unset
show("skipping ",new_co.object_info)


Dialog Designer v1.5 now available

Graham Garlick over at iFactor Consulting has uploaded v1.5 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:
  • updated Documentation file.
  • fixed issue with hanging plugin action selection gui when a raster image was missing in CORE.
  • fixed issue with non-refresh of Magik/XML code when first going to the Magik/XML tab
  • added input mechanism to add extra slots to dialog Base Class
  • added input mechanism for setting pragma statement code settings
  • added input mechanism for adding additional slots to the base gui class
  • added input mechanism for setting the package for the dialog
  • es_es language edited (thank you Alejandro CaƱas [alexcaas at gmail dot com])
  • added unit_text_item widget
  • fixed :select_directory (open_file_dir widget) incompatibility with SW4.
  • fixed internationalization issue with the :top_dock,:bottom_dock (etc.) messages
  • added date_time_input widget

Friday, January 25, 2008

Smallworld/Oracle InSync and point/text orientation

I recently had the opportunity to set up a Smallworld->Oracle InSync process with an Oracle 10 XE Locator (subset of Oracle Spatial) database. After the process had been running successfully for some time, I discovered that the point and text geometries were not retaining their orientation values during the InSync run.

It turns out that you have to explicitly tell InSync (via the oracle_objects_dataset.dataset_instance_metadata shared constant) that you want text and/or points to retain their orientation.

An example of the parameters in the context of the dataset_instance_metadata...

property_list.new_with( :customer_oracle,
property_list.new_with( <other_properties>,
) ),
:private )

I would have thought that the default should be to always include oriented points and texts. But short of patching the InSync code, you will need to be aware of this unintuitive assumption of orientations and explicitly set the property to _true.

One other note about text geometry. While InSync allows you to set up a field mapping between a Smallworld text geometry and an Oracle geometry field, there is no functionality to store the actual Smallworld text geometry character string in the Oracle geometry field. All that gets copied into Oracle is the text location (and possibly orientation). If you want to get an actual character string to appear on the map in Oracle you will need to make use of Oracle text labelling at that "text" location. You would typically use the Oracle Map Builder application to configure what string is drawn at the "text" location.

Friday, January 4, 2008

Released: Smallworld Electric Office powered by Oracle

At GITA 30 last March, GE Energy announced that they were developing an application layer on Oracle using Java tools to target medium-sized Electric utilities in North America.

Yesterday GE Energy issued a press release (Google search here) announcing Smallworld Electric Office* powered by Oracle.

And now the fun begins. I hope that we start getting early feedback from users posting to the sw-gis Yahoo! Group.