Exporting Smallworld Data to KML

There was a recent question on the sw-gis Yahoo group asking about how to export a Smallworld trail to KML for use in other applications. It turns out that can be done easily using the open source Magik Components Library (mclib). The following video demonstrates how this is done. Important information:
- Magik Components Library link is here
- Thanks to Brad Sileo (iFactor Consulting) for contributing this code to the MCLIB project.



Give it a try and let me know what kind of cool export applications you are using. And if you have any improvements to make to the functionality, please feel free to contribute to MCLIB.

Smallworld Magik JSON Parser

I added a Magik JSON Parser to the Magik Components Library today.

Here is a demo of how it works...

mclib_case_html_reporter gives erroneous Mandatory? results

The mclib_case_html_reporter is used by Smallworld clients to create an HTML-based report of the CASE tool. The HTML-based format is very helpful for deploying the datamodel information to users who either don't have access to a Smallworld license or do not have the skills to navigate throught the CASE tool.

A customer recently pointed out to me that the Mandatory? flag reported in the HTML report is not always the same as the value reported in the CASE tool. That is problematic in certain situations where users rely on the accuracy of the data in the HTML report to make decisions about how to migrate data in and out of data models. Having good information about the Mandatory property on a field is important for making good decisions about how to default values, etc.

Here is some background on how the pieces work and why the Mandatory? result was inconsistent.

A review of method case_field.mandatory? indicates that there are actually two different ways that a field can be defined as mandatory. If a user explicitly modifies a case_field to be Mandatory=Yes or Mandatory=No, that explicit action will create a join between the sw_gis!case_field object and a sw_gis!case_field_mandatory? record. That sw_gis!case_field_mandatory? record will hold the explicit information about whether the sw_gis!case_field is mandatory or not. But... there are also situations where no such explicit sw_gis!case_field_mandatory? record exists. In that case, method case_field.mandatory? says that as long as the field is a physical field, it should be considered Mandatory.

The mclib_case_html_reporter works by first making an XML dump of all the relevant CASE tables into a large XML file. Then a series of XSLT transformations are done to morph the relevant XML data into suitable HTML representation. It was in this XSLT transformation that I discovered that the XSL file did not take into consideration both "mandatory" scenarios described in case_field.mandatory?.

The fix was to extend the relevant XSL files to include the second "mandatory?" scenario. You can see for yourselves the changes that were made by looking at the SVN changeset. There weren't that many lines to change, but because it was so deep down in the XSL definitions, it took a while to sort out.

This code is freely available to anyone and I encourage you to download and contribute to the Magik Code Library.

Dialog Designer 2.0 now available

For some time I have been posting an update each time a new version of Dialog Designer has been released. Version 2.0 has now been released. You can find the announcement at the iFactor Consulting blog. From now on, if you want to be notified of Dialog Designer updates, please read or subscribe to the iFactor Consulting blog.

Happy New Year!

porting sw_dotnet_acp to a TICS framework

Pedro Miranda (http://magikbites.blogspot.com) has contributed the very cool sw_dotnet_acp code to the Magik Components SourceForge community (aka mclib). Recently someone asked me if I had used this library with the TICS framework instead of with the ACP framework. I had not done that but thought that would be an interesting question to ask my readers.

If anyone is interested in collaborating on a sw_dotnet_acp port to something like sw_dotnet_tics in the context of the mclib community, please post a comment to this post or send me a note to the contact information on the sidebar of this blog.

Dialog Designer v1.8 now available

Graham Garlick over at iFactor Consulting has uploaded v1.8 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:

• autosave a copy of current XML dialog description to the generated module.
  
• dt_dt language added (thank you Frank Wolff @ GE)
  
• added button to refresh of icon list, to capture newly loaded modules
  
• expose the double_click_notifier attribute for tabular_list and tree_list
  

Dialog Designer v1.6 now available

Graham Garlick over at iFactor Consulting has uploaded v1.6 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:

• added outlookbar widget
• support progress_bar pane type in statusbar
• support dropdown image button groups within a toolbar
• click on rendered menubar or statusbar to jump to its editing tab

Magik FME Interface can introduce unwanted data on export

I have always been very interested in using Safe Software's Feature Manipulation Engine (FME) with Smallworld. I think FME is a very cool and powerful tool. Occasionally, though, I encounter problems with the FME Magik interface that I think are worth sharing with the Smallworld/FME community.

The current issue has to do with the FME-Smallworld interface introducing data to FME features that was never part of the Smallworld data to begin with. In the Smallworld VMDS (Version Managed Data Store) it is possible to define a field as "non mandatory". As the mode implies, that means that the data validation code will not require that you populate that field with an attribute value. VMDS does not have a data type called "NULL" like other databases do. So to allow a "non mandatory" attribute value to be represented as "empty" VMDS requires that the data dictionary be defined with an "unset_value" for that attribute. An "unset_value" is a value that will represent "no value" (or NULL; or _unset) for that attribute.

So, for example, if you defined an attribute called "installation_date" as data type :ds_date and you want to enable this attribute as "non mandatory", the CASE tool will prompt you for an "unset value". Actually, it will suggest that you use the "date" Magik class as the unset value. The "date" Magik class happens to be 01Jan1970. This means that whenever you enter 01Jan1970 into the "installation_date" attribute VMDS will interpret that you mean that you are actually assigning _unset to that attribute.

So, getting to the problem with the Smallworld-FME interface. A client of mine discovered some time ago that if you use the FME 3.3 interface and you have one of these non-mandatory date fields and the value of the attribute is _unset, then the Smallworld-FME interface will actually send the unset_value to the export format. That seems very wrong to me. If your Smallworld database states that the "installation_date" is _unset, I do not want my exported-via-FME format to now show that the "installation_date" is 01Jan1970.

Luckily for us, that bad use of unset_value was all coded in Magik and could be patched by us. In FME330 method dd_record_mixin.fme_export() there is a block of code...

#--- Use the default value if available
_if (fdata << _self.perform(fld_name)) _is _unset _andif
fdesc.responds_to?(:unset_value)
_then
 fdata << fdesc.unset_value
_endif

... that basically says: if an attribute value (fdata) is _unset, then try to send that attribute's field_descriptor's (fdesc) unset_value instead. Wrong! If an attribute value is _unset then the code should send _unset to FME as well. I think it is dangerous for a data mapping interface to introduce data to an export format that does not exist in the source data. If you want to introduce data to the export format that does not exist in the source data then modify your .fme or .fmw files or use the fme_pseudo_field functionality. A user should have the confidence that the source data will be faithfully transmitted to FME without any data massaging.

If you are still with me to this point, you will be happy to know that I am finally getting to the point of this post. The Smallworld FME interface was completely rewritten at CST 4 and I assumed that the problem with ds_date (and ds_time) unset_values had been resolved. It has not! Instead, the problem has been made worse. The logic for deciding whether to default an _unset non-mandatory ds_date/ds_time value to its unset_value has been taken out of Magik and moved into the ACP/DLL files. And the logic no longer even tries to access the field descriptor's unset_value. Now the logic always sends "19700101" as the date string to FME regardless of what your data dictionary specifies for an unset_value.

Hopefully it is clear what this issue means for you. If you have non-mandatory date and time fields in your VMDS database, then all blank values will show up as 01Jan1970 as far as FME is concerned. I suppose you could work around this by modifying your .fme or .fmw files to test for "19700101" and convert it to a blank value. That could work some of the time but what if 01Jan1970 was a valid date in your business. You would never be able to know whether the "19700101" represented 01Jan1970 or _unset.

I have submitted a case report to GE on this issue yesterday and look forward to hearing their response.

If you are using FME at CST 4 (or higher), you can run this block of Magik code to identify a blank non-mandatory ds_date/ds_time attribute in your VMDS database and then send it to FME so that you can use one of the FME tools (I recommend FME Viewer) to see that your blank date/time values in Smallworld have been converted to a non-blank date in FME.

_block
# this block of code works at CST 4.x
sw_module_manager.load_module(:fme_engine)

# start the fme_tics_server if not already running
_local current_server
_if (current_server << fme_tics_server.current_server) _is _unset _orif
 _not current_server.acp_running?
_then
  fme_tics_server.start(_unset,_unset,:gis)
  current_server << fme_tics_server.current_server
_endif

_local field_type , field_name , a_table , test_record
_constant TYPES << {:ds_date,:ds_time}

# make sure your VMDS dataset is correctly named here.
_constant V << gis_program_manager.cached_dataset(:gis)

# you can restrict the list of table names here to the tables
# that you regularly export to FME...
#_constant TABLE_NAMES << {}
# ... or you can choose to scan all user tables in the current
# collection.
_constant TABLE_NAMES << V.user_table_names

write("looking for a record that has _unset as a value for a non-mandatory date or time field...")

_for table_name _over TABLE_NAMES.fast_elements()
_loop @outside
a_table << V.collections[table_name]
 
_for fd _over a_table.physical_fields()
_loop
   field_name << fd.name

   _if (field_type << fd.type) _isnt _unset _andif
    TYPES.includes?(field_type.name) _andif
    _not fd.mandatory?
   _then

  _for a_rec _over a_table.fast_elements()
  _loop
     _if a_rec.perform(field_name) _is _unset
     _then
        test_record << a_rec

        write("found that '",test_record,"' has a blank value in non-mandatory field '",field_name,"'")
        print(test_record)
        _leave @outside
     _endif
  _endloop
   _endif
_endloop
_endloop

_if test_record _isnt _unset
_then
write("sending the record to the FME server.  Please view the results using FME Viewer.  Specifically compare the value of the blank date/time attribute in Smallworld to the value in FME.")
current_server.set_output_rwo_set(rwo_set.new_from({test_record}))
_else
write("Could not find a record with a blank non-mandatory date or time field.")
_endif
_endblock
$

In my example, the Smallworld record looked like...

fme_unset_date_tester8672:
:id  1237639
:date_field  unset
:time_field  unset

But the FME Viewer showed the attributes as...



Note the different values for the date attributes.

Good luck!

Dialog Designer v1.5 now available

Graham Garlick over at iFactor Consulting has uploaded v1.5 of the open source Dialog Designer. You can get the latest version from the SourceForge Magik Components community or from a zip file at iFactor Consulting.

Here are the release notes:

• updated Documentation file.
• fixed issue with hanging plugin action selection gui when a raster image was missing in CORE.
• fixed issue with non-refresh of Magik/XML code when first going to the Magik/XML tab
• added input mechanism to add extra slots to dialog Base Class
• added input mechanism for setting pragma statement code settings
• added input mechanism for adding additional slots to the base gui class
• added input mechanism for setting the package for the dialog
• es_es language edited (thank you Alejandro Cañas [alexcaas at gmail dot com])
• added unit_text_item widget
• fixed :select_directory (open_file_dir widget) incompatibility with SW4.
• fixed internationalization issue with the :top_dock,:bottom_dock (etc.) messages
• added date_time_input widget

Dialog Designer v1.4 available

Graham Garlick's latest update to Dialog Designer is now available as v1.4. You can visit iFactor Consulting's web site to download the code here.

This version has the following additional features

• bug fixes (tabular_list, element_editor, tree_list)
• modified button_item icons/cursor
• fr_ca language added (internet translation ... corrections likely)
• es_es language added (internet translation ... corrections likely)
• pt_pt language added (internet translation ... corrections likely)
• ko_kr language added (thank you Heewook Park [heewook.park@ge.com] )

Dialog Designer v1.3 available

Graham Garlick's latest update to Dialog Designer is now available as v1.3. You can find it on the Magik Components SourceForge project. Based on the dates of the releases, v1.3 is not yet in the packaged mclib release. But if you have a CVS connection to SourceForge you can update your source tree with his latest code. You can also visit iFactor Consulting's web site to download the code here.

This version has the following additional features

• added style selection widget (place a gui element for selecting a style)
• removed dependency on the path_select plugin
• added file/directory selection widget (place a button that launches the file/directory selection dialog)
• added plugin widget (embed an action from some other plugin or an entire other plugin)
• added auto-generation of supplementary gui/framework & plugin files so developers can expand/customize produced GUI functionality in a place that the Dialog Designer won’t overwrite if the GUI layout is tweaked and re-built.
• added FAQ section to the main documentation
• de_de messages updated
• fr_ca messages created

Magik HTTP requests; Open Source Magik; Reading RSS

Brad packaged a new release (in .zip file) of the Magik Components source tree here. Be sure to choose the latest version of "mclib CST 4.0 SWAF". This will include the dialog designer that I mentioned yesterday.

I wanted to highlight one of the cool components that I have found in this package. The module is called :mclib_http_interface. It provides a Magik interface to making HTTP requests to a server and then return the results back to you as a simple_xml_document. You can then use the API on simple_xml_document to parse information out.

To highlight the ease and power of this capability, I quickly hacked together some code that allows you to connect to the Yahoo!Weather RSS feed and get a weather forecast for your location. This is a simple example and you can clearly extend it to your heart's content.

Download the Magik Components product and register the accompanying product.def file in your curent image.

Load the :mclib_http_interface module (eg., sw_module_manager.load_module(:mclib_http_interface)

Now run the following block of code. As written, it will give you a weather forecast for Boulder, CO, USA in degrees Celsius. The example also shows how you can glean lat/long information from a feed and convert it to a Magik coordinate.

_block
 # see http://developer.yahoo.com/weather/ for more information
 # about the Weather RSS feed
 _constant BASE_URL   << "http://weather.yahooapis.com/forecastrss"
 _constant QUERY_CHAR << %?
 _constant QUERY      << "p=80305&u=c" # forecast for USA ZipCode 80305 in Celsius
 _constant URL        << write_string(BASE_URL,QUERY_CHAR,QUERY)
 
 _local hr        << mclib:http_request.new_for_url(_unset,_unset,URL)
 _local response  << hr.connect()
 _local a_xml_doc << response.smart_result()

 _local rss_channel << a_doc.top_element.element_matching_name(:channel)

 _local item << rss_channel.element_matching_name(:item)

 _local long << item.element_matching_name("geo:long").elements[1].as_number()
 _local lat  << item.element_matching_name("geo:lat").elements[1].as_number()

 write()
 write(item.element_matching_name(:title).elements[1],"  at ",coordinate.new(long,lat))

 
 _local forecast_attributes
 
 _for a_forecast _over item.elements_matching_name("yweather:forecast")
 _loop
  forecast_attributes << a_forecast.attributes

  write(forecast_attributes[:day]," (",forecast_attributes[:date],") ",forecast_attributes[:text]," low=",forecast_attributes[:low]," high=",forecast_attributes[:high]," ")
 _endloop
_endblock
$

Incidentally, you can get more information about the Yahoo!Weather API at http://developer.yahoo.com/weather

Where I think this http_interface could come in really handy is for you to write GeoRSS parsers that would allow you to show location events on a Smallworld map (maybe as a SOM or a post_render_set). Have a look at Direction Magazine's Fun with GeoRSS article if you are interested in other ideas for interfacing with other data sources in real time from Smallworld.

Magik Dialog Designer now available

Some time ago I posted about the Magik Dialog Designer and that it would be made freely available soon. Well I am happy to say that that day has come (actually was a couple of weeks ago) and it is now available on the Magik Components SourceForge page. Thank you to Graham Garlick for putting together such a useful tool!

This tool essentially lets you create GUIs in SWAF using drag-and-drop and then creates all the relevant Magik code (including module structures and XML files) to include that GUI in your product. It also stubs out any of the action methods associated with buttons.

For now you will have to browse the Magik Components source code tree yourself and find the module. You can follow the CVS connection instructions to setup CVS on your machine to check out the latest Magik Components code. If you are not interested in installing CVS you could e-mail Graham at graham [at] ifactorconsulting [dot] com and ask him for the Dialog Designer zip file. Tell him Alfred sent you :)

(The module should eventually make its way into the Download Magik Components section.)

Since our time together at Navigant/Smallworld/GE, I have always been a fan of Graham and his creative innovations with Magik. In my opinion, this Magik Dialog Designer is one of his greatest Magik achievements. Although his Raster Calculator was pretty cool too. But I ramble on... so please go download the Dialog Designer yourself and have fun.

Magik Dialog Designer

Under the heading of “cool and useful Magik stuff”…

I recently had the opportunity to meet with a friend of mine who showed me his latest creation -- Dialog Designer

It is a tool that allows you to graphically layout your new Magik GUI with many of the commonly-used widgets, menubar, statusbar, and docks. Once you have layed out the GUI graphically and applied appropriate properties to the components, the Dialog Designer will stub out all the required Magik files, classes, methods, shared constants and XML config/gui files and place them in a new module directory structure.

While it will not write the Magik behaviour behind the widgets – you still need to write Magik code that does something when the button is clicked – it goes a very long way to building your :build_gui() method for you. And the nice thing is that tricky tasks like resizing rowcols or other widgets is now done automatically for you.

As a long-time Magik developer, I have never been too excited about GUI design because it can be a bit complicated (and time consuming) in Magik. Using the Dialog Designer will make the GUI design considerably easier and definitely save you a number of hours per GUI in development time.

The company that my friend works for has said that they will release the Dialog Designer to the SourceForge MagikComponents community. Until they have done that, I will keep the name of the developer and his company to myself. Stay tuned for a press release from them (hopefully soon) announcing the availability of this very cool and useful tool for Magik developers.

writing Smallworld symbols/styles to JPG images

Recently a question appeared on sw-gis asking how to output your Smallworld styles to some external format.

If you download the SourceForge MagikComponents (http://sourceforge.net/projects/magikcmpnts) [MagikComponents has been moved to https://www.assembla.com/code/magik-components/subversion/nodes/mclib/trunk ] you should see a number of modules that provide results similar to what you are asking or provide examples of how you might render styles to JPG files.

• modules\mclib_style_images provides code to create jpg files from the various style records.
• modules\mclib_case_html_reporter provides code that creates an HTML output for your CASE tool. Part of this code (see case_html_report_plugin.output_style_info_for() ) creates jpeg files of the various geometry field styles.

Smallworld Open Source Software

The proprietary nature of the Smallworld suite of products is sometimes used as a negative point against a potential customer choosing Smallworld as their GIS system. One argument is that the programming language (Magik) is not a standard and open language and therefore it is expensive to train (and retain) good Magik developers. A group of developers in the Smallworld community are attempting to address that concern and are using Open Source Software concepts as their guiding principles. I recently had the opportunity to interview Brad Sileo of Ten Sails Consulting about his thoughts on Open Source as it applies to Smallworld.

I have noticed that SourceForge has a MagikComponents project. Is that the main focus of the current open-source "movement" in the Smallworld community? Are there other open source initiatives going on with the Smallworld product.

This project is the main area I am aware of, though there are several packages currently hosted there. In addition to the Magik Component library, the Minto Spatial Workbench tools are hosted on the same Sourceforge site. These separate sets of software are both offered under different public licenses.

What is the SourceForge MagikComponents project all about?

I think this project is about lowering the cost of ownership for Smallworld software, and allowing the Smallworld community to build towards bigger, more complex solutions through reuse. Of course, that's a pretty grand vision, and it would require lots of contributors to really make any impact. A less idealistic goal is to provide a simple framework to share some nice bits of code that make the system easier. I find that I always want the suite of tools from that mclib layered product on any system I work on because they save me from reinventing things over and over.

Why do you think the Smallworld community should consider an open source paradigm? What benefits do you see to customers, GE partners and GE itself?

On the grand scale, moving the entire Magik platform to open source would provide a silver bullet to the age-old "proprietary" claims of Smallworld competitors. In reality, the majority of the Smallworld platform actually is open source, with the exception of the virtual machine and the sys_core code. But, the distinction between open source and free software is a critical one - without the open nature of Smallworld source, programming would be very difficult in the system. If it was taken further and made available as free software, the growth in Magik development could certainly provide direct benefit to the existing customer through additional bug fixes, code libraries, and interfaces.
At about that same time that Smallworld was first putting out Magik, Sun Microsystems decided to release its internal, proprietary, cross-platform language and make it available as free software. Java took off like a rocket into many different application spaces on the strength of being open and available.

What kind of Licensing concerns are you aware of with respect to open-sourcing Smallworld? How do(es) the licensing scheme(s) of SourceForge address those concerns?

There are two licenses being used for software in the Mclib library right now - The GNU Public License(GPL) and the Lesser GNU Public License(LGPL). The main difference is that the lesser license is intended to be used with libraries, and as such does not require that all code built from it be released for free access. I would encourage folks publishing items through the library to understand these licenses and consider which is appropriate for their software. I have heard some developers express concern about using modules in the library because they fear this will require them to publish all of the code in their systems for open use. As a practical matter, in most cases, the majority of code built on top of tools from this library is not reusable in a new context. The use of the LGPL mitigates this concern, which is why I have been working to move modules to that license. Of course, it's worth a read of the software license agreement delivered with Smallworld Core Spatial technology, which actually supersedes all of this discussion with its stance on derivative works.

What kind of actions from the Smallworld community (vendors, partners and users) would help further the open source cause?

There would be some benefit in more organizations recognizing that they get value from open source software and thus aiming to return reusable items to the library. There will always be a battle between closed/commercial software makers and the open source choice, which is no different in the Smallworld space than in operating systems, word processors, or web browsers. It takes a critical mass of contributors to push an open source initiative to the point where it can produce its own momentum. When that happens, the benefits start to magnify for the existing users and you can get an exponential return for the overall community in available functionally and software quality. This momentum also pushes the commercial software to compete with the open solutions, which further benefits the consumers.

*I also conducted the same interview with a colleague at GE Energy. He was able to provide insightful answers to the interview questions. But when he sought approval from his managers he was prohibited from publishing his comments.

Thank you to Brad and my unnamed colleague for their contributions (published and not) to this article.

I'm interested to hear if there are other Open Source initiatives in the Smallworld community. If anyone has information about this, please post a comment to this blog or send me an e-mail.

I wonder if we could get an official response from GE Energy to the question: "How do you see Smallworld working together with the Open Source paradigm?" Maybe it is something to ask them at the upcoming GITA Annual Conference. I would welcome that dialog.