Friday, February 29, 2008

Magik FME Interface can introduce unwanted data on export

I have always been very interested in using Safe Software's Feature Manipulation Engine (FME) with Smallworld. I think FME is a very cool and powerful tool. Occasionally, though, I encounter problems with the FME Magik interface that I think are worth sharing with the Smallworld/FME community.

The current issue has to do with the FME-Smallworld interface introducing data to FME features that was never part of the Smallworld data to begin with. In the Smallworld VMDS (Version Managed Data Store) it is possible to define a field as "non mandatory". As the mode implies, that means that the data validation code will not require that you populate that field with an attribute value. VMDS does not have a data type called "NULL" like other databases do. So to allow a "non mandatory" attribute value to be represented as "empty" VMDS requires that the data dictionary be defined with an "unset_value" for that attribute. An "unset_value" is a value that will represent "no value" (or NULL; or _unset) for that attribute.

So, for example, if you defined an attribute called "installation_date" as data type :ds_date and you want to enable this attribute as "non mandatory", the CASE tool will prompt you for an "unset value". Actually, it will suggest that you use the "date" Magik class as the unset value. The "date" Magik class happens to be 01Jan1970. This means that whenever you enter 01Jan1970 into the "installation_date" attribute VMDS will interpret that you mean that you are actually assigning _unset to that attribute.

So, getting to the problem with the Smallworld-FME interface. A client of mine discovered some time ago that if you use the FME 3.3 interface and you have one of these non-mandatory date fields and the value of the attribute is _unset, then the Smallworld-FME interface will actually send the unset_value to the export format. That seems very wrong to me. If your Smallworld database states that the "installation_date" is _unset, I do not want my exported-via-FME format to now show that the "installation_date" is 01Jan1970.

Luckily for us, that bad use of unset_value was all coded in Magik and could be patched by us. In FME330 method dd_record_mixin.fme_export() there is a block of code...

#--- Use the default value if available
_if (fdata << _self.perform(fld_name)) _is _unset _andif
fdata << fdesc.unset_value

... that basically says: if an attribute value (fdata) is _unset, then try to send that attribute's field_descriptor's (fdesc) unset_value instead. Wrong! If an attribute value is _unset then the code should send _unset to FME as well. I think it is dangerous for a data mapping interface to introduce data to an export format that does not exist in the source data. If you want to introduce data to the export format that does not exist in the source data then modify your .fme or .fmw files or use the fme_pseudo_field functionality. A user should have the confidence that the source data will be faithfully transmitted to FME without any data massaging.

If you are still with me to this point, you will be happy to know that I am finally getting to the point of this post. The Smallworld FME interface was completely rewritten at CST 4 and I assumed that the problem with ds_date (and ds_time) unset_values had been resolved. It has not! Instead, the problem has been made worse. The logic for deciding whether to default an _unset non-mandatory ds_date/ds_time value to its unset_value has been taken out of Magik and moved into the ACP/DLL files. And the logic no longer even tries to access the field descriptor's unset_value. Now the logic always sends "19700101" as the date string to FME regardless of what your data dictionary specifies for an unset_value.

Hopefully it is clear what this issue means for you. If you have non-mandatory date and time fields in your VMDS database, then all blank values will show up as 01Jan1970 as far as FME is concerned. I suppose you could work around this by modifying your .fme or .fmw files to test for "19700101" and convert it to a blank value. That could work some of the time but what if 01Jan1970 was a valid date in your business. You would never be able to know whether the "19700101" represented 01Jan1970 or _unset.

I have submitted a case report to GE on this issue yesterday and look forward to hearing their response.

If you are using FME at CST 4 (or higher), you can run this block of Magik code to identify a blank non-mandatory ds_date/ds_time attribute in your VMDS database and then send it to FME so that you can use one of the FME tools (I recommend FME Viewer) to see that your blank date/time values in Smallworld have been converted to a non-blank date in FME.

# this block of code works at CST 4.x

# start the fme_tics_server if not already running
_local current_server
_if (current_server << fme_tics_server.current_server) _is _unset _orif
_not current_server.acp_running?
current_server << fme_tics_server.current_server

_local field_type , field_name , a_table , test_record
_constant TYPES << {:ds_date,:ds_time}

# make sure your VMDS dataset is correctly named here.
_constant V << gis_program_manager.cached_dataset(:gis)

# you can restrict the list of table names here to the tables
# that you regularly export to FME...
#_constant TABLE_NAMES << {}
# ... or you can choose to scan all user tables in the current
# collection.
_constant TABLE_NAMES << V.user_table_names

write("looking for a record that has _unset as a value for a non-mandatory date or time field...")

_for table_name _over TABLE_NAMES.fast_elements()
_loop @outside
a_table << V.collections[table_name]

_for fd _over a_table.physical_fields()
field_name <<

_if (field_type << fd.type) _isnt _unset _andif
TYPES.includes?( _andif
_not fd.mandatory?

_for a_rec _over a_table.fast_elements()
_if a_rec.perform(field_name) _is _unset
test_record << a_rec

write("found that '",test_record,"' has a blank value in non-mandatory field '",field_name,"'")
_leave @outside

_if test_record _isnt _unset
write("sending the record to the FME server. Please view the results using FME Viewer. Specifically compare the value of the blank date/time attribute in Smallworld to the value in FME.")
write("Could not find a record with a blank non-mandatory date or time field.")

In my example, the Smallworld record looked like...
:id 1237639
:date_field unset
:time_field unset

But the FME Viewer showed the attributes as...

Note the different values for the date attributes.

Good luck!

Announcing "Programar en Magik de SmallWorld" blog

For Magik practitioners wishing to read about Smallworld Magik in Spanish, Joan in Barcelona has created Programar en Magik de SmallWorld.

Congratulations Joan!

Wednesday, February 6, 2008

CASE Tool out of synch with your actual datamodel

It is possible that your CASE Tool representation of your datamodel might not accurately reflect the actual datamodel you have in your database. This happened to me recently when I discovered that some CASE work I had done in my own development environment had touched more objects than I was working with when I did a CASE Archive. When I loaded the CASE archive into the production database I had not noticed that it also updated some other objects with an older version of the CASE definition.

This was not a problem until someone tried to apply changes for the "unknowingly-touched" CASE object and it introduced some confusion into the production environment. Well, we resolved the issue but then had the question... How do we ensure that the CASE Tool accurately represents the state of the datamodel?

The answer to that question: "Reverse Engineer"

Here are some tips that will help you to reverse engineer a CASE model.
  • read the Smallworld Case Tutorial help "Reverse engineering from the Gis dataset"
  • an important thing to note is that you need to do the reverse engineer action into an empty CASE partition. Use a procedure such as create_database() to create new "empty" case.ds and case_dd.ds files and then update your Case SOC to point to these new .ds files
  • you should also temporarily keep your "old" case.ds and case_dd.ds files. Do this by creating a new SOC (:old_case) that has a :case dataset that points to these old ds files.
  • when you perform the reverse engineer action you will see that the graphics for relationships and drafting_* objects are not visible in the new CASE. You will also see that the CASE objects are not grouped together the way you had them in the original CASE tool. That is because the Gis dataset does not keep information about CASE graphics in its data dictionary.
  • I use a script like the following one to "recover" some of my description geometries from the "old" CASE tool into the new recently-reverse-engineered CASE tool. (Be sure that you have a handle to both the old and new CASE tools for this).

_local new_case_v << gis_program_manager.cached_dataset(:case)
_local old_case_v << gis_program_manager.cached_dataset(:old_case)

_local old_drafting_coll, new_drafting_coll , geometries , properties , new_rec

write("creating CASE drafting_* objects...")
_for drafting_table_name _over {:drafting_lines,:drafting_areas,:drafting_points,:drafting_texts}.fast_elements()
old_drafting_coll << old_case_v.collections[drafting_table_name]
new_drafting_coll << new_case_v.collections[drafting_table_name]

_for old_rec _over old_drafting_coll.fast_elements()
properties <<

new_rec << new_drafting_coll.insert(old_rec)

_for a_geom _over old_rec.geometries().fast_elements()
properties[a_geom.app_type] << a_geom

_if properties.empty?


_local new_case_coll << new_case_v.collections[:sw_gis!case_object]
_local old_case_coll << old_case_v.collections[:sw_gis!case_object]

write("updating CASE Object geometries...")
_for new_co _over new_case_coll.fast_elements()
_if (old_co <<, _is _unset
show("skipping ",


_local new_case_coll << new_case_v.collections[:sw_gis!case_relationship]
_local old_case_coll << old_case_v.collections[:sw_gis!case_relationship]

write("redrawing CASE relationship geometries...")
_for new_co _over new_case_coll.fast_elements()
_if (old_co <<,new_co.object_info)).an_element()) _is _unset
show("skipping ",new_co.object_info)


Dialog Designer v1.5 now available

Graham Garlick over at iFactor Consulting has uploaded v1.5 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:
  • updated Documentation file.
  • fixed issue with hanging plugin action selection gui when a raster image was missing in CORE.
  • fixed issue with non-refresh of Magik/XML code when first going to the Magik/XML tab
  • added input mechanism to add extra slots to dialog Base Class
  • added input mechanism for setting pragma statement code settings
  • added input mechanism for adding additional slots to the base gui class
  • added input mechanism for setting the package for the dialog
  • es_es language edited (thank you Alejandro CaƱas [alexcaas at gmail dot com])
  • added unit_text_item widget
  • fixed :select_directory (open_file_dir widget) incompatibility with SW4.
  • fixed internationalization issue with the :top_dock,:bottom_dock (etc.) messages
  • added date_time_input widget