writing Smallworld symbols/styles to JPG images

Recently a question appeared on sw-gis asking how to output your Smallworld styles to some external format.

If you download the SourceForge MagikComponents (http://sourceforge.net/projects/magikcmpnts) [MagikComponents has been moved to https://www.assembla.com/code/magik-components/subversion/nodes/mclib/trunk ] you should see a number of modules that provide results similar to what you are asking or provide examples of how you might render styles to JPG files.

• modules\mclib_style_images provides code to create jpg files from the various style records.
• modules\mclib_case_html_reporter provides code that creates an HTML output for your CASE tool. Part of this code (see case_html_report_plugin.output_style_info_for() ) creates jpeg files of the various geometry field styles.

ds_transfer might be corrupting your backups

If you are using the ds_transfer function in "hot" mode to create backups from your production VMDS, you run the risk of corrupting your backup files so that they cannot be properly restored. I had originally published a post about this to the sw-gis group in July of last year. Recently I was at a client site where the backup scheme used ds_transfer in "hot" mode and I realized that it would be helpful to repost this information in this blog.

If you are using PowerOn, it is possible that the original implementors set up your backup scheme to use ds_transfer in "hot" mode. You may want to read the following article and see if your backups are in jeopardy.

Original Article follows....

Since there is a discussion about backups and the ds_transfer mechanism was mentioned as a solution I thought I would throw out a warning about that approach. Simply put, when using ds_transfer.new() in "hot" mode, you should ensure that no one is writing to the database while you are performing the ds_transfer.

The typical ds_transfer scenario involves ds_transferring one DS file at a time. Imagine that you ds_transfer gdb.ds and it takes a total of 15 minutes. 10 minutes into the ds_transfer a user writes a fiber(1234) RWO and geometry to that partition. But the alternative that was being written to had already been processed at minute 2 of the 15 minute process. After the 15 minutes are done, the ds_transfer starts working on the rwo.ds file.

You will not see a problem in the live production dataset, but you will now have a situation where the "ds_transferred" dataset has an inconsistency between the rwo.ds and the gdb.ds. This is a simple scenario, but imagine that changes are at a larger scale or possibly involve adding/removing alternatives while a hot ds_transfer is in progress. These will all be permitted actions but may cause unintended results in the "ds_transferred" files. If you are using the ds_transfer.new() with "hot" mode you should try opening a copy of your recently ds-transferred files to see if they are consistent with each other.

If you want to continue using the ds_transfer functionality to perform "hot" backups, refer to ds_transfer.transfer_partition() to see how it might be beneficial to you. This method has the advantage of allowing you to snapshot all DS files in a partition at the same time so you will not encounter any of these data consistency issues.

controlling runaway procedures at the Magik prompt

Have you ever run a procedure at the Magik prompt and it kept running and the only way you could terminate it was to close the Magik session? There is another way!

I was working on a procedure today that parses the results of a Relational Integrity Check and summarizes the errors by rwo_type. I wasn't sure how long it would take to run, so I forked the procedure in a new thread.

For illustration here, I have created a procedure that runs an endless loop. If you ran this at the Magik prompt, you would need to terminate the session in order to end the loop. To avoid this hassle, I have forked the procedure in a new thread using fork_at(). You can review the public comments for this method in the class browser to learn about the valid arguments.

MagikSF> th << _proc()
_loop

_endloop 
_endproc.fork_at(5)
$

Now it is easy to query the thread to see if it is running or not...

MagikSF> th.status_description
$
"runnable"

You can pause the thread...

MagikSF> th.suspend()
$
thread(unnamed suspended 5)
MagikSF> th.status_description
$
"suspended"

You can resume a paused thread...

MagikSF> th.resume()
$
thread(unnamed runnable 5)
MagikSF> th.status_description
$
"runnable"

Or you can finally terminate the thread...

MagikSF> th.kill()
$
thread(unnamed runnable 5)
MagikSF> th.status_description
$
"terminated"

IAS, SOA and other TLAs

Much has been said about Smallworld moving to a Service Oriented Architecture (SOA). But what does it mean to you or your users. How can it benefit you and them?

The O'Reilly Radar Blog today posted a number of articles dealing with Yahoo! Pipes. The introductory article is Pipes and Filters for the Internet. It gives a very good description of what pipes are in terms of the web and RSS (another TLA).

What I found interesting was that the example used by the blog to demonstrate the power of pipes, GeoRSS and filters on the web was a classic GIS one:

A good example Pipe is "Apartment Near Something". It allows you to input what you would like to be near (for example: "parks"), what city (for example: "Palo Alto, CA"), and how far (for example: 2 miles). It outputs GeoRSS of available apartments in a Palo Alto, CA that are near parks.

The beauty of a tool like this is that your business analysts can put together dynamic queries of multiple data sources including your Smallworld data. You might also call this "roll your own dashboard". This might include applications that present power outage information, gas leaks or unserviced customers near your services taps.


If Smallworld has not included GeoRSS with their next version of IAS then I think it would be easy for someone else to write a GeoRSS service quite quickly to provide this service.

The Yahoo! Pipes graphical tool actually reminds me a bit about Safe Software's FME Workbench tool. It seems to me that if Safe would include a GeoRSS reader and various RSS manipulation transformers in its toolkit, then could provide a service that Yahoo! Pipes cannot: a RSS manipulation toolkit that resides behind a corporate firewall. A typical Smallworld customer has data that cannot or should not be exposed to the 'net. That means the Yahoo! Pipes tool cannot access it. But if FME had this capability, your business analysts could do the same thing in a secure environment.

In short, SOA is one way to get your data out to non-programmers and others that are more interested in the data than they are in the mechanisms/protocols for accessing that data. Yahoo! Pipes is a useful tool for that. If Safe could add this functionality to their toolkit, I would be even more excited!

Developers: patch it yourself

(Note: this discussion is also useful to non-developers. If you are responsible for managing developers to provide customizations for your Smallworld environment, you may want to ensure that they are providing core code changes to you in a way that simplifies future upgrades. Using the core patch mechanism is one way to manage custom changes to core code.)

In the "old days" many of us used to include basic_additions and basic_changes in our custom Magik code to indicate core classes and methods that we were enhancing or changing. Sometimes we needed to change or patch core methods because we needed to do it quickly to meet a customer business requirement. The problem with this approach is that it these basic_* folders were scattered all over the source tree and made it more difficult than necessary to upgrade the core code at a later time.

I would like to suggest a different methodology that uses the core patching mechanism. It is one that I have seen used at some customer sites and have adopted for my ongoing work for other customers.

Many methods in the Smallworld core product are classified as not redefinable or even restricted. According to protocol, customizers may not modify restricted methods or really any method that is not classified as redefinable. The reality is that often times to meet a customer business need we do need to modify such methods. This adds to future code upgrade efforts because we now have introduced code that we must manually inspect for changes at each upgrade by comparing it to the core-upgraded version of that code.

It can be argued that changing a non-redefinable method is actually fixing a defect in the code. If the code does not do what a customer expects it to do, that is a defect. And if it is a defect, then it should be reported to the GE Smallworld Helpdesk. The helpdesk may not be able to produce a patch for you in the time period your project needs it for. In that case, you need to patch the defect yourself and have some kind of mechanism in place to revisit that patch once the next TSB is released for that product.


The following steps outline a process that could be followed. We refer to an example of an actual Smallworld-FME defect.

identify the defect

e.g., FME zero-length sectors incorrectly identified as donut holes. The problem was found in fme_tics_client.geom_field_for()

report the defect to GE Smallworld Helpdesk

report at http://supportcentral.ge.com

if Helpdesk can supply patch

install it per their instructions. In this example they were unable to so we created our own patch

if Helpdesk cannot supply patch

determine what module (and module version) your magik class is part of

MagikSF> sw_module_manager.module(fme_tics_client.module_name)
$
sw_module(fme_engine,2)

determine what product your code is part of and where that product directory exists

MagikSF> sw_module_manager.module(fme_tics_client.module_name).product
$
sw_product(sw_fme_400)

MagikSF> sw_module_manager.module(fme_tics_client.module_name).product.directory
$
"E:\projects\customer\fme400"

Ensure that that product has a source/patches_z_proposed folder

NOTE: the "z" in z_proposed is not a spelling mistake. All the "patches_*" folders in a product's source folder are loaded alphabetically. We need the "z" in z_proposed in order to have our patches loaded last. Because the core SW products already have patches, it is possible that we may be "patching a patch". The only way for this to be effective is to load our patches last.

get new patch number
Unless you have been assigned a range of patch numbers by Smallworld, the patch number you use will possibly overlap one that was already used by Smallworld. You can get around this by assigning a version to it that has a string in it unique to your organization.

For example, at Red Planet Consulting (RPC), we started our patches at 1. Therefore, the patch file might have been labeled as p1_1.magik. This patch probably existed for Smallworld at one time, so to make our patch unique, we change the file to p1_1rpc.magik. A corresponding entry in the file itself declares the patch as unique, too...

sw!declare_patch(1,"1rpc","description")
$

You will want to create a patch list or log for your organization so that you do not get duplicate patch numbers with your unique organization identifier.

create patch file

using the newly-logged patch number, create a patch file in the product's source/patches_z_proposed folder.

e.g., create E:\projects\customer\fme400\source\patches_z_proposed\p1_1rpc.magik.

be sure to add a reference to this file in the patch_list.txt for that folder.

You can look at any existing patch file for a template of how to create a patch file. The important parts to include are:

# at the top of the file, declare which module this patch fixes.
# This is important because the patch mechanism attempts to load ALL
# patch files regardless of the relevance of their code to the current
# image.  Putting a sw!patch_software() call at the top of the file will
# prevent your patch from loading if it is not relevant to the current image.
sw!patch_software(:fme_engine,2)
$

# the very last line of the patch file should be the patch declaration.
sw!declare_patch(1,"1rpc","FME zero-length sectors incorrectly identified as donut holes")
$

The benefits of this approach include:

• Smallworld is notified of "defects" to the code. This feedback loop is important to improve the product
• Customers are given a more robust solution to introducing core fixes identified by the customizer even before the next TSB. An added benefit of a patch-based approach is that patches can be easily "backed out" of an environment if they are deemed to be causing problems in that environment.
• At TSB/upgrade time, there are fewer folders to review for changes. Each time a TSB or upgrade is released, someone will need to look at all the patches_z_proposed folders and files to either remove or modify the files depending on what core code has been update by that TSB or upgrade. It is easier to maintain core patches in one folder per product as opposed to changes in one folder (eg., basic_changes) per module.
  

Smallworld Open Source Software

The proprietary nature of the Smallworld suite of products is sometimes used as a negative point against a potential customer choosing Smallworld as their GIS system. One argument is that the programming language (Magik) is not a standard and open language and therefore it is expensive to train (and retain) good Magik developers. A group of developers in the Smallworld community are attempting to address that concern and are using Open Source Software concepts as their guiding principles. I recently had the opportunity to interview Brad Sileo of Ten Sails Consulting about his thoughts on Open Source as it applies to Smallworld.

I have noticed that SourceForge has a MagikComponents project. Is that the main focus of the current open-source "movement" in the Smallworld community? Are there other open source initiatives going on with the Smallworld product.

This project is the main area I am aware of, though there are several packages currently hosted there. In addition to the Magik Component library, the Minto Spatial Workbench tools are hosted on the same Sourceforge site. These separate sets of software are both offered under different public licenses.

What is the SourceForge MagikComponents project all about?

I think this project is about lowering the cost of ownership for Smallworld software, and allowing the Smallworld community to build towards bigger, more complex solutions through reuse. Of course, that's a pretty grand vision, and it would require lots of contributors to really make any impact. A less idealistic goal is to provide a simple framework to share some nice bits of code that make the system easier. I find that I always want the suite of tools from that mclib layered product on any system I work on because they save me from reinventing things over and over.

Why do you think the Smallworld community should consider an open source paradigm? What benefits do you see to customers, GE partners and GE itself?

On the grand scale, moving the entire Magik platform to open source would provide a silver bullet to the age-old "proprietary" claims of Smallworld competitors. In reality, the majority of the Smallworld platform actually is open source, with the exception of the virtual machine and the sys_core code. But, the distinction between open source and free software is a critical one - without the open nature of Smallworld source, programming would be very difficult in the system. If it was taken further and made available as free software, the growth in Magik development could certainly provide direct benefit to the existing customer through additional bug fixes, code libraries, and interfaces.
At about that same time that Smallworld was first putting out Magik, Sun Microsystems decided to release its internal, proprietary, cross-platform language and make it available as free software. Java took off like a rocket into many different application spaces on the strength of being open and available.

What kind of Licensing concerns are you aware of with respect to open-sourcing Smallworld? How do(es) the licensing scheme(s) of SourceForge address those concerns?

There are two licenses being used for software in the Mclib library right now - The GNU Public License(GPL) and the Lesser GNU Public License(LGPL). The main difference is that the lesser license is intended to be used with libraries, and as such does not require that all code built from it be released for free access. I would encourage folks publishing items through the library to understand these licenses and consider which is appropriate for their software. I have heard some developers express concern about using modules in the library because they fear this will require them to publish all of the code in their systems for open use. As a practical matter, in most cases, the majority of code built on top of tools from this library is not reusable in a new context. The use of the LGPL mitigates this concern, which is why I have been working to move modules to that license. Of course, it's worth a read of the software license agreement delivered with Smallworld Core Spatial technology, which actually supersedes all of this discussion with its stance on derivative works.

What kind of actions from the Smallworld community (vendors, partners and users) would help further the open source cause?

There would be some benefit in more organizations recognizing that they get value from open source software and thus aiming to return reusable items to the library. There will always be a battle between closed/commercial software makers and the open source choice, which is no different in the Smallworld space than in operating systems, word processors, or web browsers. It takes a critical mass of contributors to push an open source initiative to the point where it can produce its own momentum. When that happens, the benefits start to magnify for the existing users and you can get an exponential return for the overall community in available functionally and software quality. This momentum also pushes the commercial software to compete with the open solutions, which further benefits the consumers.

*I also conducted the same interview with a colleague at GE Energy. He was able to provide insightful answers to the interview questions. But when he sought approval from his managers he was prohibited from publishing his comments.

Thank you to Brad and my unnamed colleague for their contributions (published and not) to this article.

I'm interested to hear if there are other Open Source initiatives in the Smallworld community. If anyone has information about this, please post a comment to this blog or send me an e-mail.

I wonder if we could get an official response from GE Energy to the question: "How do you see Smallworld working together with the Open Source paradigm?" Maybe it is something to ask them at the upcoming GITA Annual Conference. I would welcome that dialog.