Friday, December 14, 2007

Dialog Designer v1.4 available

Graham Garlick's latest update to Dialog Designer is now available as v1.4. You can visit iFactor Consulting's web site to download the code here.

This version has the following additional features
  • bug fixes (tabular_list, element_editor, tree_list)
  • modified button_item icons/cursor
  • fr_ca language added (internet translation ... corrections likely)
  • es_es language added (internet translation ... corrections likely)
  • pt_pt language added (internet translation ... corrections likely)
  • ko_kr language added (thank you Heewook Park [heewook.park@ge.com] )

Tuesday, November 20, 2007

FME File Association Updater - FME Talk | Google Groups

The topic FME File Association Updater - FME Talk | Google Groups: in the FME Talk Google Groups provides a nice solution for the case where you have two versions of FME installed on your workstation and the version you want to use is not the one associated with the .fme and .fmw files. This has been an issue for me when doing a Smallworld upgrade from pre-4.x to 4.0. These versions of Smallworld support different versions of FME. If you want to continue running pre-4.x Smallworld/FME functionality and 4.x Smallworld/FME functionality then you either need to do something clever with the FME_HOME environment variable in your Magik code or you can use the script described below...

Hi FME'ers, If - like me - you constantly install FME beta versions for testing purposes, you may be wishing that there were a way to revert the file associations back to the official release version for production work.

Well now there is a way.

See: http://www.fmepedia.com/index.php/FME_File_Type_Association_Updater

New Safe support dude Aaron Koning has come up with a neat little script that will let you choose which fme.exe file to associate all your FME files with. Now you can ensure that when you double-click an fmw file, it opens the version of FME that you want, and not an arbitrary beta version.

Hope this is useful (it is for me). Thanks Aaron,

Mark

Wednesday, October 31, 2007

Waterfall software development is back with a vengeance

There are some great articles on Waterfall development process at http://www.waterfall2006.com. (You may need to read between the lines to understand what the authors are getting at. Note the conference date in the top left corner of the home page.)

My personal favourites:

Enjoy!

Identify and Recover Deleted Data without a database restore

Customers will come to us and say... "someone deleted/updated some data a while back and we only discovered it now. We don't want to restore a backup because that will undo the work from the last week. Help!"

It may be possible to "rescue" the old data if you still have a suitably older checkpoint somewhere in your database.

First, you need to get a full inventory of all the checkpoints available to you. It is possible that there exist checkpoints in your database that you do not even know about. The following code will give you a full inventory and list the checkpoints (and alternative that they are in) sorted from most recent to oldest creation times.

_pragma(classify_level=debug)
_method ds_version_view.debug!all_checkpoints()
 ## debug!all_checkpoints() : sorted_collection
 ##
 ## returns a sorted_collection of all checkpoints in self and
 ## the tree beneath self sorted from most recent to oldest.
 ## The elements returned are actually simple_vectors of the
 ## form:
 ##       {checkpoint A_CHECKPOINT,string ALTERNATIVE_NAME}
 ##
 _local checkpoints << sorted_collection.new(_unset,
          _proc(a,b)
           >> a[1].time_created > b[1].time_created
          _endproc)
 
 _for a_view _over _self.actual_alternatives()
 _loop
  checkpoints.add_all(a_view.checkpoints.map(_proc(a_cpt)
           _import a_view
           >> {a_cpt,a_view.alternative_path_name()}
          _endproc)) 
 _endloop


 _return checkpoints
_endmethod
$
_pragma(classify_level=debug)
_global show_all_checkpoints <<
_proc@show_all_checkpoints(a_view)
 ## show_all_checkpoints(ds_version_view A_VIEW) : _unset
 ##
 ## shows all checkpoints and related information for A_VIEW and
 ## tree.

 _for cpt_info _over a_view.debug!all_checkpoints().fast_elements()
 _loop
  show(cpt_info[1].time_created,cpt_info[1].checkpoint_name,cpt_info[2])
 _endloop
_endproc
$


To run this code, do something like...

MagikSF> show_all_checkpoints(gis_program_manager.cached_dataset(:gis))
$


... which will give you output like...

date_time(10/30/07 17:34:44) "is!insync_1193790807" :||oracle_sync_parent|
date_time(10/30/07 17:34:44) "is!insync_1193790807" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 16:32:54) "is!insync_1193787116" :||oracle_sync_parent|
date_time(10/30/07 16:32:53) "is!insync_1193787116" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 15:33:09) "is!insync_1193783570" :||oracle_sync_parent|
date_time(10/30/07 15:33:08) "is!insync_1193783570" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 14:34:02) "is!insync_1193780009" :||oracle_sync_parent|
date_time(10/30/07 14:34:02) "is!insync_1193780009" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 13:33:52) "is!insync_1193776407" :||oracle_sync_parent|


Once you have identified a checkpoint that is old enough to contain your old data, you take one of the two following two approaches (comments in the code explain the differences).

_block
 # if you want to use your GIS tools to see what the data looked
 # like at a previous checkpoint, then get a handle on the
 # current view and go that checkpoint in READONLY mode.

 # DO NOT switch to :writable while at this alternative unless
 # you are ABSOLUTELY SURE of what you are doing.  The safest
 # action to take once you are done viewing data at the old
 # checkpoint is to switch to the "***disk version***"
 # checkpoint in that alternative before continuing.
 _local v << gis_program_manager.cached_dataset(:gis)
 
 v.switch(:readonly)
 v.go_to_alternative("|oracle_sync_parent",:readonly)
 v.go_to_checkpoint("is!insync_1192840224",:readonly)
_endblock
$

... OR ...
_block
 # if you want to keep the main dataset at the current view and
 # then get a second handle to the old_view so that you can copy
 # old data (eg., incorrectly deleted/updated data) into the
 # current view, then get a handle on a REPLICA of the current
 # view and go that previous checkpoint in READONLY mode.

 # DO NOT switch to :writable while at this alternative unless
 # you are ABSOLUTELY SURE of what you are doing.  The safest
 # action to take once you are done viewing data at the old
 # checkpoint is to switch to the "***disk version***"
 # checkpoint in that alternative before continuing.
 _global old_view

 # once you have finished using OLD_VIEW, be sure to :discard()
 # it.
 old_view << gis_program_manager.cached_dataset(:gis).replicate()
 
 old_view.switch(:readonly)
 old_view.go_to_alternative("|oracle_sync_parent",:readonly)
 old_view.go_to_checkpoint("is!insync_1192840224",:readonly)

_endblock
$


If you are going to use data in the old_view to update/recreate data in the current view, then you might want to explore using classes/methods such as
record_transaction.new_update()
record_transaction.new_insert()
ds_collection.at()
ds_collection.insert()
ds_collection.clone_record()
ds_collection.update()
ds_collection.insert_or_update()



[NOTE 2014 June 02: Reinhard Hahn sent me some new code that supercedes the code he originally posted in the comment to this post]

_pragma(classify_level=debug)
_iter _method ds_version_view.all_available_versions(_optional top?)
  ## yields all available Versions of a replicate of
  ## _self in the following order:
  ##
  ## alternatives (if TOP? isnt _false: beginning with ***top***)
  ## -- checkpoints (beginning with :current)
  ## -- base versions for checkpoints (only for alternative_level
  ##    > 0)
  ##
  ## for each version yields
  ## A_VIEW,ALTERNATIVE_PATH_NAME,CHECKPOINT_NAME,BASE_LEVEL, CHECKPOINTS_TIME_CREATED
  ## where A_VIEW is always a replicate, which will be discarded
  ## outside this method. The order of the CHECKPOINTS will be in
  ## reversed chronological order (i.e. beginning with the most
  ## recent one)
  ##
  ## BASE_LEVEL is 0 for the version itself, -1 for the immediate
  ## base version, -2 for the base version of the base version
  ## and so on 
  ##
  ## example:
  ## v,:|***top***|,:current,0,v.time_last_updated()
  ## v,:|***top***|,"cp_1",0,cp_1.time_created
  ## v,:|***top***|,"cp_2",0,cp_2.time_created
  ## v,:||alt1|,:current,0,v.time_last_updated()
  ## v,:||alt1|,:current,-1,v.time_last_updated()
  ## v,:||alt1|,"cp11",0,cp11.time_created
  ## v,:||alt1|,"cp11",-1,v.time_last_updated()
  ## v,:||alt2|,":current",0,v.time_last_updated()
  ## ...
  ##
  ## NB:
  ## If the different versions of _self have different
  ## datamodels, you might be annoyed by messages like the following:
  ##
  ##  compiliing my_table.geometry_mapping
  ##  fixing up details for dd_field_type(my_field_type) : Version 12345, transition to 54321
  ##  Defining my_other_table
  ##
  ## To avoid this, you might want to suppress any output by
  ## redefining the dynamics !output! and suppressing the output
  ## of the information conditions db_synchronisation and
  ## pragma_monitor_info. Of course you have then to put any
  ## wanted output to other output targets like !terminal! or a
  ## locally defined external_text_output_stream. So your code
  ## might look like the following: 
  ## MagikSF> t << proc(a_database_view)
  ##  _handling db_synchronisation, pragma_monitor_info _with procedure
  ##  _dynamic !output!  << null_text_output_stream
  ##
  ##  # ... any more preparation code ...
  ##
  ##  _for r,alt_pname,cp,base_level,cp_time _over a_database_view.all_available_versions()
  ##  _loop
  ##
  ##    # ... my code running in the loop, for example:
  ##    !terminal!.show(alt_pname,cp,base_level,cp_time)
  ##    !terminal!.write(%newline)
  ##    !terminal!.flush()
  ##    # ...
  ##
  ##  _endloop
  ##
  ##  # ... any summary output, for example:
  ##  !terminal!.write("Ready.",%newline)
  ##  !terminal!.flush()
  ##  # ...
  ##
  ## _endproc.fork_at(thread.low_background_priority)
  ##
  ## If you don't want to use these !terminal! calls, you could
  ## also dynamicall redefine !output!:
  ## MagikSF> t << proc(a_database_view)
  ##  _handling db_synchronisation, pragma_monitor_info _with procedure
  ##  _dynamic !output!  << null_text_output_stream
  ##
  ##  # ... any more preparation code ...
  ##
  ##  _for r,alt_pname,cp,base_level,cp_time _over a_database_view.all_available_versions()
  ##  _loop
  ##    _block
  ##      _dynamic !output! << !terminal!
  ##      # ... my code running in the loop, for example:
  ##      show(alt_pname,cp,base_level,cp_time)
  ##      # ...
  ##    _endblock
  ##  _endloop
  ##
  ##  !output! << !terminal!
  ##  # ... any summary output, for example:
  ##  write("Ready.")
  ##
  ## _endproc.fork_at(thread.low_background_priority)

  _local change_alt_level <<
    _if top? _isnt _false 
    _then
      >> - _self.alternative_level
    _else
      >> 0
    _endif
  _local r << _self.replicate(change_alt_level,_true)
  _protect
    _for rr _over r.actual_alternatives(:same_view?,_true)
    _loop
      _local alt_name << rr.alternative_path_name()
#      _if alt_name.write_string.matches?("*{consistency_validation}") _then _continue _endif
      _local cps << rr.checkpoints.as_sorted_collection(_proc(lhs,rhs)
                      _if lhs _is :current
                      _then
                        _if rhs _is :current
                        _then
                          _return _maybe
                        _else
                          _return _true
                        _endif
                      _elif rhs _is :current
                      _then
                        _return _false
                      _endif 
                      _local res <<  rhs.time_created _cf lhs.time_created
                      _if res _isnt _maybe
                      _then
                        _return res
                      _endif
                      _return lhs.checkpoint_name _cf rhs.checkpoint_name
                    _endproc)
      cps.add(:current)
      _for cp _over cps.fast_elements()
      _loop
        _local cp_name
        _local cp_time 
        _if cp _is :current
        _then 
          rr.rollforward()
          cp_name << :current
          cp_time << rr.time_last_updated()
        _else
          cp_name << cp.checkpoint_name
          rr.go_to_checkpoint(cp_name)
          cp_time << cp.time_created
        _endif 
        _loopbody(rr,alt_name,cp_name,0,cp_time)
        _if rr.alternative_level > 0
        _then
          _local rrr << rr.replicate(:base,_true)
          _protect
            _local level_diff << -1
            _loopbody(rrr,alt_name,cp_name,level_diff,rrr.time_last_updated())
            _loop
              _if rrr.alternative_level = 0 _then _leave _endif
              rrr.up_base()
              level_diff -<< 1
              _loopbody(rrr,alt_name,cp_name,level_diff,rrr.time_last_updated())
            _endloop 
          _protection
            rrr.discard()
          _endprotect
        _endif 
      _endloop
    _endloop
  _protection
    r.discard()
  _endprotect
_endmethod
$

Tuesday, October 30, 2007

Smallworld/FME Configuration Tips and Tricks (part 2)

I recently received a number of questions from a colleague that had recently upgraded to FME 2007 as part of the CST 4.1 upgrade. It seemed to me that the questions and answers were actually a continuation of the original Smallworld/FME Configuration Tips and Tricks so I present them here as "part 2". By the way, there is a growing list of Smallworld/FME FAQs at http://www.fmepedia.com/index.php/Category:GE_Smallworld_Format_FAQ

Question: I previously used the Scale function in my fme files to adjust coordinate values between database units (mm) and coordinate system units (m). The factor would be 0.001 or 1000 depending on if I was exporting or importing.

FACTORY_DEF SWORLD SamplingFactory \
INPUT FEATURE_TYPE * \
@Scale(0.001) \
SAMPLE_RATE 1
I used the Workbench for the first time to create an fmw file for exporting objects. I used the Scale Transformer which appears to work. Is this the right approach or should I create an FME coordinate system that matches our Smallworld database coordinate system? This must be a common problem for Smallworld/FME users.

Answer: I typically create a custom FME coordinate system that matches the Smallworld application coordinate system and then let FME do all the unit conversions. That way you don’t need to mess around with the scale transformer.



Question: Below is code based on the example in your FME presentation. Would it be better to rewrite this to include a loop and to execute the fme command for each table or run it as it is?

_block
_global fme_tics_server
_local current_server

# start the fme_tics_server if not already running
_if (current_server << fme_tics_server.current_server) _is _unset _orif
_not current_server.acp_running?
_then
fme_tics_server.start(_unset,_unset,:gis)
current_server << fme_tics_server.current_server
_endif

# indicate which records to prepare for export
_local vc << gis_program_manager.cached_dataset(:gis).collections
_local export_records << rwo_set.new_from(vc[:table1])
export_records.add_all(vc[:table2])
export_records.add_all(vc[:table3])

current_server.set_output_rwo_set(export_records)
# run the command line FME
_local command << {"\\gis\FME\fme","\\server\mapping\FME\sworld2mapinfo.fmw",
"--DestDataset_MAPINFO",'"\\server\mapping\FME\fme2007"',
"--PORT_SWORLDSWAF", current_server.port_number.write_string}

system.do_command(command)
_endblock
$
Answer: If all of your export collections are processed by the same FMW file (which I see it is; which I also think is a good thing) then I do not see a reason to have to put the code in a loop.



Question: Is there a way to create a log file that can be examined during and/or after the translation? I run the code above and have no feedback whether the translation is running or not.

Answer: Probably the best way to get feedback from FME about the progress of the migration is to modify your FMW file to specify a Log File:



If you want to be more flexible, then Publish the Parameter…



... and then you can modify your Magik code to dynamically set the LOG_FILE value at export time ...
   _local command << {"\\gis\FME\fme","\\server\mapping\FME\sworld2mapinfo.fmw",
"--DestDataset_MAPINFO",'"\\server\mapping\FME\fme2007"',
"--PORT_SWORLDSWAF", current_server.port_number.write_string,
"--LOG_FILE","\\server\mapping\FME\fme2007\mapinfo.log"}


Question: I could previously export objects with mapped geometries by only specifying the mapped geometry field in the fme file. I found that nothing gets written in the new version unless I add each geometry field into the Workbench. Is this correct?

Answer: You are correct that each mapped geometry must be accounted for in your FMW file. The way you have shown your data features...



... is valid. If you want to make your workspace a bit cleaner-looking, you can use a “merge filter” ...



... and then you can flow all your data from that single “Merged” feature to your transformers ...

Tuesday, October 23, 2007

Dialog Designer v1.3 available

Graham Garlick's latest update to Dialog Designer is now available as v1.3. You can find it on the Magik Components SourceForge project. Based on the dates of the releases, v1.3 is not yet in the packaged mclib release. But if you have a CVS connection to SourceForge you can update your source tree with his latest code. You can also visit iFactor Consulting's web site to download the code here.

This version has the following additional features
  • added style selection widget (place a gui element for selecting a style)
  • removed dependency on the path_select plugin
  • added file/directory selection widget (place a button that launches the file/directory selection dialog)
  • added plugin widget (embed an action from some other plugin or an entire other plugin)
  • added auto-generation of supplementary gui/framework & plugin files so developers can expand/customize produced GUI functionality in a place that the Dialog Designer won’t overwrite if the GUI layout is tweaked and re-built.
  • added FAQ section to the main documentation
  • de_de messages updated
  • fr_ca messages created

Friday, October 12, 2007

discovering which SWAF config.xml file to change

Have you ever wondered exactly where in the XML config hierarchy a particular plugin is defined? With XML file inheritance in SWAF, it can sometimes get very complicated.

I have created some enhancements to the XML parsing and plugin creation methods that allow you to track the one or many XML FILENAMES where a plugin is defined.

Usage:
  1. load this file
  2. start the application whose configurations you want to trace.
  3. get a handle on some plugin and then ask it for its debug!source_files. It will return a rope of XML file names that contain that plugin's definition in the order that they were encountered. I.E., the last XML file in the rope is the one that the plugin used for its definition.
For example:
MagikSF> smallworld_product.applications.an_element().plugin(:map_plugin).debug!source_files
$
sw:rope:[1-2]
MagikSF> print(!)
$
rope(1,2):
1 ".....\sw_user_custom\pni_gui\resources\base\data\config.xml"
2 ".....\sw_user_custom\custom_application\resources\base\data\config.xml"


so, this tells you that the "custom_application" config.xml file overrides the same plugin settings in PNI_GUI config.xml. If you want to make a change to the :map_plugin you will need to make it in CUSTOM_APPLICATION for the changes to take hold.

DISCLAIMER: DO NOT PUT THIS IN YOUR PRODUCTION CODE. This script is intended only for debugging your XML configuration files and should not be in any image that is writing data to a production database.

What is RSS?

I noticed today that about 45% of subscribers to this blog use the e-mail subscription option instead of RSS. While e-mail is definitely a good way to be notified of updates to the blog I would suggest that you learn about RSS and how it might lighten your inbox load.

The best explanation I have found so far is a 3 minute video called RSS in Plain English (the website also has links to information in other languages).

One advantage of RSS over e-mail notification is that your inbox is not filled with update notifications at inconvenient times. Using RSS you go to your RSS page (or reader application) when you want to.

Wednesday, October 10, 2007

SECO Energy gives customers access to outage data

There is a good article in the September issue of Transmission & Distribution World titled Customers Access Online Outage Data. It was written by Barry Owens of SECO Energy. In case you missed it, Barry also gave the featured Customer Presentation at the recent Smallworld 2007 Americas Users Conference (presentations are here).

This is a project that I had heard about before and one of the things I had wondered was: "in a power outage situation, how will affected customers have the power to run their computers to check their outage status online?". Barry answers that question (and many others) in a well written presentation of their project.

Congratulations also to Peter Manskopf at GE for leading this successful project.

Wednesday, October 3, 2007

MUnit Testing Framework

I recently received a request from a reader wondering where the unit test application was on my blog. I don't have a post on that yet so I thought that now was as good a time as any to make one.

I'm not going to go into the details here. Suffice it to say, that the MUnit framework is based on the original JUnit framework but for Magik instead of Java.

I have found it helpful when developing code to create some kind of automated test scripts. There is much that can be debated about how much or how little test scripting to do.

Mark Field had a detailed presentation at the Smallworld 2007 Americas Users Conference this year titled "Implementing MUnit for Smallworld". I imagine you could contact him for the presentation or maybe we can get him to post more information on his blog. If you have the username/password, you can download the presentation from https://www.kinsleyassociates.com/ge/smallworld/presentations.asp

[NOTE: The above links are old. You can find the presentation slides and download zip file at http://sw-gis.wikidot.com/magik-munit]

GE also had/has a version of MUnit but from what I understand they are reluctant to release it to the community. I found that there was a version of MUnit created by Jan Vorel at http://www.janvorel.com. Unfortunately the current version of that site is not very user friendly like it used to be and you can no longer download Jan's code from there. But if you look at http://web.archive.org/web/20030406030601/www.janvorel.com/ you will find an archived version of the web site that still has a working link to his version of MUnit. It only works on CST 3.3 but I found that with a few modifications it works in CST 4.x as well.

The implementations of MUnit that I have seen do not allow you to test mouse/keyboard input. Mark's presentation describes how he enhanced MUnit to include those features.

One enhancement I did for the MUnit framework some time ago was to create a new test listener that exported the test results as XML and/or HTML. An example of those results is here.

You can click here for a Yahoo Groups search result on MUnit.

FME Workbench Design Patterns

You could consider the FME Workbench to be a graphical IDE (integrated development environment). Therefore it makes sense to employ the concept of Design Patterns to FME Workspaces. For anyone that has worked in an object-oriented language (like Magik) Design Patterns are not a foreign concept.

There is a presentation on FMEpedia called Introduction to Design Patterns that is worth reading by anyone wanting to make more efficient use of their FME Workspaces. The included PowerPoint presentation has a number of design patterns. Note that this presentation was inspired by a discussion the FME people had with someone from the Smallworld community and the examples in the presentation are all Smallworld-FME ones.

Wednesday, September 12, 2007

Profiling a dynamic enumerator problem with XPRF

A recent post on the sw-gis Yahoo! Group made me think that this might be a good example to demonstrate how to use the XPRF Magik class to profile your Magik code and help you find potential bottle necks in it.

Because the post did not say so explicitly, for the purposes of this exercise I am assuming the "dynamic enumerator" referred to is the one that makes use of the methods :can_match?() and :match_string(). (Sometimes people use the term "dynamic enumerator" to refer to the core extensible enumerator functionality or the PNI "dynamic enumerator" functionality.)

The first thing I would do would be to do a profile analysis on the :match_string() method on that particular object. An analysis will tell you if you have an issue with database access or maybe some code inefficiencies in your match_string() method.

You could do that as follows...

temporarily rename your subclassed some_object_name.match_string() method to some_object_name.debug!match_string()

create a new method...

_method some_object_name.match_string(_gather args)
_global x_profile
_local ok?, result
(x_profile,ok?,result) << xprf.new_from_send(:debug!match_string|()|,_scatter args)

_return result
_endmethod
$


Then try to open the some_object_name's editor again in such a way that your match_string() code is called. Likely it will take longer now than before because all the match_string() code is being run through the XPRF profiler. When the list is populated in the editor you will be ready to investigate the performance profile.

First, make sure that the global x_profile was populated...


MagikSF> x_profile
$
a sw:xprf


If it is populated, you can use various interfaces to review your profile. The least user-friendly is to call x_profile.basic_report(). This will give you potentially hundreds of lines of output at the Magik prompt... a bit overwhelming if this is the first time you have done this.

Another option is to use the :xprf_dialog class. By default it is not loaded into your image, so you need to load the :profiler_magik_gui module into your current image. Use the sw_module_dialog.open() to load the module or...


MagikSF> sw_module_manager.load_module(:profiler_magik_gui)
$


Once the module is loaded you can use the following Magik command to give you a more user-friendly GUI that will show you expandable lists of all the methods that were called within the context of your profiler.


MagikSF> xprf_dialog.open(x_profile)
$


Use that tool to see where the bulk of your time on the match_string() is being spent.

Possible problems you might have in your match_string() code...
- inefficient code in loops
- select predicates on non-indexed fields

Things that you could investigate to improve performance include:
- make your code more efficient
- index any columns that you are writing a predicate against.
- if the table contents are not changing very often you might consider caching the strings you get from the table in a reusable rope (eg., shared_variable) and only read the data from the table once per session in order to populate the cache

If all those things do not improve the performance to the degree that you would like, then you may want to consider using the CASE tool to configure that field as a "catalogue mapping". That will provide a different field editor but still give you the control over what goes into that field.

FME SOM Demo is now online

A Flash video of the Smallworld FME SOM Demonstration is now online. Click here to view it. The flash file size is about 9MB so it might take a few minutes to download.

(Unlike my attempts at the demo at the Smallworld Users Conference this presentation does not include any blue screens :) )

Smallworld/FME Configuration Tips and Tricks

The Smallworld 2007 Americas Users Conference came to an official end yesterday evening. I had the good fortune of being able to present a paper at the conference entitled "Smallworld Core Spatial Technology and FME: Configuration Tips and Tricks". The PowerPoint slides are available for download here. If you do not have the PowerPoint application then you can view an HTML version of it that requires Internet Explorer and ActiveX by clicking here.

[NOTE: the contact information at the end of the presentation is now outdated. I now work for iFactor Consulting. You can find my new contact information in the Contact Information section on the side of this blog]

I had also agreed to help Mark Stoakes from Safe Software at his FME hands-on workshops by demonstrating the use of the Smallworld FME SOM. At both workshops my computer blue-screened just as I started the demo. That was very unfortunate because there was not enough time to debug the problem during the session. So I will be putting together a description of what I did and posting that here and on FMEpedia. (I discovered later that the problem had to do with my laptop being connected to a docking station which received intermittent power and therefore somehow messed up the parallel port on which my Smallworld license dongle was connected.)

If you received a GE USB Stick in your swag bag and, like me, were not at the session where the password was revealed, try '1111'.


Thursday, September 6, 2007

ACEs, ACEs, everywhere

I recently tried to make a copy of an ACE in the Physical Network Inventory (PNI) product. The out-of-the-box ACE is called "Physical NI User". So I thought that all I needed to do was make a copy of that ACE, call it something like "Physical NI User New", and then modify the application registration code to include...

:ace_name,"Physical NI User New"


Apparently that was not enough. If you want to have two user groups use different ACEs, you definitely need to change the application registration :ace_name. But you also need to change the ACE name in the following XML configuration files:

in a suitable application config.xml file
  • verify config/framework_data/map_type[@ace_name] has the name you are expecting
  • verify config/plugins/plugin[@class_name="pni_editor_manager"]/property[@name="ace_name"]["value"] has the name you are expecting
  • verify config/plugins/plugin[@class_name="document_manager"]/document/document_properties/property[@name="ace_name"]["value"] has the name you are expecting
In addition, if you have specified any bookmarks for an ACE, they are "hardcoded" for the ACE in which they were originally created.

The typical symptom which indicates that some of the above-mentioned ACE settings are incorrect is the one where you look at the object control plugin and notice that the v/h/s settings are not what you thought you had set them to in the current ACE. Most likely you did set the v/h/s settings correctly but one or more of your application configurations are still pointing to an ACE that you are not expecting it to.

Tuesday, September 4, 2007

See you at the Americas Users conference

The Smallworld 2007 Americas Users Conference is coming up in less than one week near Denver, Colorado. I look forward to seeing many of you there. Hopefully we will learn a lot about the new Smallworld/Oracle initiatives.

I will be presenting a paper on Tuesday entitled "Smallworld and FME: Configuration Tips and Tricks". Safe Software will be presenting some good hands-on workshops about how to use their FME tool so I won't be covering that territory again. But I will be presenting how to get your Smallworld system set up with the FME interface so that you can do the cool things that Safe Software will show you in the workshops. I also plan to give a short FME SOM (Spatial Object Manager) demonstration during one of the FME workshops.

If I don't see you at the presentation or workshop, please feel free to come by and say "hello" at the Red Planet Consulting booth where I will be during the exhibition times.

See you there!

Sunday, August 12, 2007

another Smallworld/Magik blog

It seems that another Smallworld/Magik blog has entered the blogosphere. Welcome to Mark at Smallworld Technical Blog!

Thursday, August 2, 2007

Magik HTTP requests; Open Source Magik; Reading RSS

Brad packaged a new release (in .zip file) of the Magik Components source tree here. Be sure to choose the latest version of "mclib CST 4.0 SWAF". This will include the dialog designer that I mentioned yesterday.

I wanted to highlight one of the cool components that I have found in this package. The module is called :mclib_http_interface. It provides a Magik interface to making HTTP requests to a server and then return the results back to you as a simple_xml_document. You can then use the API on simple_xml_document to parse information out.

To highlight the ease and power of this capability, I quickly hacked together some code that allows you to connect to the Yahoo!Weather RSS feed and get a weather forecast for your location. This is a simple example and you can clearly extend it to your heart's content.

Download the Magik Components product and register the accompanying product.def file in your curent image.

Load the :mclib_http_interface module (eg., sw_module_manager.load_module(:mclib_http_interface)

Now run the following block of code. As written, it will give you a weather forecast for Boulder, CO, USA in degrees Celsius. The example also shows how you can glean lat/long information from a feed and convert it to a Magik coordinate.

_block
# see http://developer.yahoo.com/weather/ for more information
# about the Weather RSS feed
_constant BASE_URL << "http://weather.yahooapis.com/forecastrss"
_constant QUERY_CHAR << %?
_constant QUERY << "p=80305&u=c" # forecast for USA ZipCode 80305 in Celsius
_constant URL << write_string(BASE_URL,QUERY_CHAR,QUERY)

_local hr << mclib:http_request.new_for_url(_unset,_unset,URL)
_local response << hr.connect()
_local a_xml_doc << response.smart_result()

_local rss_channel << a_doc.top_element.element_matching_name(:channel)

_local item << rss_channel.element_matching_name(:item)

_local long << item.element_matching_name("geo:long").elements[1].as_number()
_local lat << item.element_matching_name("geo:lat").elements[1].as_number()

write()
write(item.element_matching_name(:title).elements[1]," at ",coordinate.new(long,lat))


_local forecast_attributes

_for a_forecast _over item.elements_matching_name("yweather:forecast")
_loop
forecast_attributes << a_forecast.attributes

write(forecast_attributes[:day]," (",forecast_attributes[:date],") ",forecast_attributes[:text]," low=",forecast_attributes[:low]," high=",forecast_attributes[:high]," ")
_endloop
_endblock
$


Incidentally, you can get more information about the Yahoo!Weather API at http://developer.yahoo.com/weather

Where I think this http_interface could come in really handy is for you to write GeoRSS parsers that would allow you to show location events on a Smallworld map (maybe as a SOM or a post_render_set). Have a look at Direction Magazine's Fun with GeoRSS article if you are interested in other ideas for interfacing with other data sources in real time from Smallworld.

Tuesday, July 31, 2007

Magik Dialog Designer now available

Some time ago I posted about the Magik Dialog Designer and that it would be made freely available soon. Well I am happy to say that that day has come (actually was a couple of weeks ago) and it is now available on the Magik Components SourceForge page. Thank you to Graham Garlick for putting together such a useful tool!

This tool essentially lets you create GUIs in SWAF using drag-and-drop and then creates all the relevant Magik code (including module structures and XML files) to include that GUI in your product. It also stubs out any of the action methods associated with buttons.

For now you will have to browse the Magik Components source code tree yourself and find the module. You can follow the CVS connection instructions to setup CVS on your machine to check out the latest Magik Components code. If you are not interested in installing CVS you could e-mail Graham at graham [at] ifactorconsulting [dot] com and ask him for the Dialog Designer zip file. Tell him Alfred sent you :)

(The module should eventually make its way into the Download Magik Components section.)

Since our time together at Navigant/Smallworld/GE, I have always been a fan of Graham and his creative innovations with Magik. In my opinion, this Magik Dialog Designer is one of his greatest Magik achievements. Although his Raster Calculator was pretty cool too. But I ramble on... so please go download the Dialog Designer yourself and have fun.

Magik plugin for Eclipse

If you are a Magik developer that prefers to use Eclipse over Emacs you should know that ASTEC today released version 0.5.1 of their free Magik Development Tools (MDT). To read more, visit http://www.mdt-project.com. There is also a white paper that gives a good overview of what MDT together with Eclipse can do for you.

Monday, July 30, 2007

FME 4.1 Translator DLL problem

[Update: 07Sep2007 - see the comments below about additional information about where to find sworldswaf.dll files and which one to use where.]

I tried to use the FME 4.1 SWAF interface today and ran into a problem where the FME clients (eg., Universal Viewer) could not recognize any of the data dictionary information queued up in Smallworld for export to FME. I contacted GE Help Desk and they were very quick (on the order of minutes) to send me a new sworldswaf.dll file that must be copied into the FME root folder (eg., c:\program files\fme). This fixed the problem with the FME 4.1 interface but then introduced the problem for the FME 4.0 SWAF interface. Not that big a deal unless you are running two versions of SWAF with the same FME clients.

Interesting notes about the DLL versions. The FME 4.1 documentation says that you need to be running FME 2006 GB build 2651 at a minimum with version 1.1.0.0 of sworldswaf.dll. The version of the DLL that shipped with my version of FME build 2651 was 1.0.1.0. Either the documentation has a typo or Safe Software does not have the latest version of the DLL. Anyways, the version of the DLL that the GE Help Desk sent me was 1.0.0.11 (that's right, it seems like an earlier version number). Apparently this DLL is still too new to have made it onto either GE or Safe Software's web sites. You will need to log a case with the GE Help Desk to get the correct DLL.

On a related note: One of my Smallworld-FME presentations was accepted to the 2007 Americas Smallworld Users Conference in Colorado this year. I hope to see alot of you there. More about that later.

Friday, July 20, 2007

Where did Geography go?

One thing I regret about my undergraduate studies is that I took no geography courses. I only discovered GIS after I graduated. I enjoy the technical nature of GIS very much but have always been intrigued by the geographic questions that the GIS technologies strive to address.

So you might ask: "if you are intrigued by Geography and GIS why are you working with Smallworld?". Good question... GE seems to have made a decision not to market the Smallworld Core Technology as a GIS but rather to present it as a network management suite of tools. I know that Smallworld can be used for GIS analysis but the product development has not emphasized that at all. (I would be interested to read how you are using the Smallworld tools to understand non-utility geographic issues.)

So because I am still interested in GIS and Geography, Jerome E. Dobson's article "Bring Back Geography!" resonated with me. If you are at all interested in Geography (or wonder what it is) this article is definitely worth reading. (Yes, it is an ESRI site. But if you are interested in Geography and GIS in general, you go to where the information is)

Dobson writes of the role of Geography largely in the United States of America. He writes about how during the 1st and 2nd World Wars, geographers were an important part of preparing for peace after the war. And then he points out that Geography was purged from the major USA Universities after World War II. His call is for geographers to reassert themselves into the public consciousness so that they can make important contributions for the common good.

Because of my daily work with GIS technologies, I always fancied myself a bit of a geographer. Dobson writing about GIS says...

In contrast, advances in GIS alone are likely to cast us as clerks handling data for the ecologists, political scientists, economists, and other current leaders in these topics


I do actually seem myself as a bit of a clerk (or consultant to clerks) that manages data for others. Not that being a clerk is a bad thing... it puts food on my family's table and is something that is also technically challenging and rewarding. But this article has got me thinking about how I can take the next step beyond technology and start learning more about Geography to understand my world better and then also contribute to its common good.

Wednesday, July 11, 2007

how to exhaust your UVAs without knowing it

Unique Value Allocators (UVAs) in Smallworld are special objects that return a unique integer value for system IDs only once. Theoretically you could exhaust your UVA bucket at some point and then you will not be able to insert any more RWOs, geometries, links, nodes and other system objects.

There are a number of ways to exhaust your UVAs but one that came to my attention recently is the method :copy_transform() on various geometry classes. I have used this method in the past for exporting geometries to some other file format. What was strange at the time was that when I used this method I received a "database not writable" error. That raised some red flags in my mind: why does the database need to be writable if all I intend to do is read data from it and export it to another file format?

It turns out in that case that I was using :copy_transform() to convert the coordinates of geometries from one coordinate system to another. Unbeknown to me, the :copy_transform() method actually makes a copy of your geometry before transforming it. So you are left with two copies of the same geometry in the database after the transform. And creating a copy of the geometry uses UVAs (which cannot be recovered) and involves more system resources than you would likely need for your intended purpose -- simply to transform the coordinates of a geometry from one coordinate system to another.

I am writing all this because we have come across a situation where an installation used export code from another vendor that apparently makes frequent use of the :copy_transform() method to export geometry data into a 3rd party format. Recently that installation's main UVA was exhausted. There is nothing wrong with transforming your coordinates before exporting to a 3rd party but there are simpler ways of doing transforms using !current_coordinate_system! that do not gobble up UVA resources (maybe I will post about that some day).

I would recommend that you run the following command at the Magik prompt:

MagikSF> print_users_of(:copy_transform|()|)
$

The output of this procedure will list all methods that call :copy_transform(). Scan the list and if any of the method names (or classes that they are written on) look as though they are not core Smallworld methods/classes, please confirm that that code cannot be done some other way without using up your precious UVAs.


(There are other ways to exhaust your UVAs prematurely, but I wanted to focus on this one approach today)

Wednesday, June 27, 2007

Peter Batty's thoughts on GE's spatial offerings on Oracle

For a former-insider's point of view on General Electric's announcement to create new GIS applications on the Oracle stack, see Peter Batty 's Thoughts on GE's next generation system based on Oracle - part 1.

I plan to write some more about this myself in the coming weeks, particularly as it relates to a recent demo I saw of the system running on Oracle.

Friday, June 22, 2007

progress counters and dialogs

I recently had a colleague ask me how to pop up a progress bar dialog to indicate to a user the percentage complete of a process. I did not know how to answer that at first... Smallworld has historically not used progress bar dialogs as much as some users would expect. So I will present two options for showing progress status.

Text-based process

If you have some kind of text based process use progress_counter...


_block
_local some_collection << rope.new(20)

# create a progress_counter class that reports every 5
# increments. Because we know some_collection.size we can also
# estimate completion time
_local pc << progress_counter.new_on_count(5,some_collection.size)

_for idx,an_el _over some_collection.fast_keys_and_elements()
_loop
# the write() and sleep() statements are here to make a simple
# example...
write(an_el)
_thisthread.sleep(1000)

# ... but the :next() method is important to let the
# progress_counter (aka PC) know that you have completed one
# iteration of the loop.
pc.next()
_endloop
_endblock
$
unset
unset
unset
unset
unset
Done 5 (15 left) time taken 5 seconds 297 milliseconds (ETC. 15 seconds 891 milliseconds ) average rate 0.9439305267
unset
unset
unset
unset
unset
Done 10 (10 left) time taken 10 seconds 625 milliseconds (ETC. 10 seconds 625 milliseconds ) average rate 0.9411844826
unset
unset
unset
unset
unset
Done 15 (5 left) time taken 15 seconds 812 milliseconds (ETC. 5 seconds 270 milliseconds ) average rate 0.9487724325
unset
unset
unset
unset
unset
Done 20 (0 left) time taken 21 seconds 15 milliseconds (ETC. 0 milliseconds ) average rate 0.9562876356



... you can also ask the progress_counter to report at a given time interval...


_block
_local some_collection << rope.new(20)

# create a progress_counter class that reports every 2
# seconds. Because we know some_collection.size we can also
# estimate completion time
_local pc << progress_counter.new_on_time(3,some_collection.size)

_for idx,an_el _over some_collection.fast_keys_and_elements()
_loop
# the write() and sleep() statements are here to make a simple
# example...
write(an_el)
_thisthread.sleep(1000)

# ... but the :next() method is important to let the
# progress_counter (aka PC) know that you have completed one
# iteration of the loop.
pc.next()
_endloop
_endblock
$
unset
unset
unset
Done 3 (17 left) time taken 3 seconds 125 milliseconds (ETC. 17 seconds 708 milliseconds ) average rate 0.9600000000
unset
unset
unset
Done 6 (14 left) time taken 6 seconds 250 milliseconds (ETC. 14 seconds 583 milliseconds ) average rate 0.9600000000
unset
unset
unset
Done 9 (11 left) time taken 9 seconds 359 milliseconds (ETC. 11 seconds 439 milliseconds ) average rate 0.9616468318
unset
unset
unset
Done 12 (8 left) time taken 12 seconds 453 milliseconds (ETC. 8 seconds 291 milliseconds ) average rate 0.9648530373
unset
unset
unset
Done 15 (5 left) time taken 15 seconds 578 milliseconds (ETC. 5 seconds 199 milliseconds ) average rate 0.9616468318
unset
unset
unset
Done 18 (2 left) time taken 18 seconds 703 milliseconds (ETC. 2 seconds 81 milliseconds ) average rate 0.9612351238
unset
unset
Done 20 (0 left) time taken 20 seconds 765 milliseconds (ETC. 0 milliseconds ) average rate 0.9612351238



If you want to use a more user-friendly indicator, use the progress_indicator_dialog. It is loaded into the core 4.1 image and you can find details about it in the header of product/modules/sw_core/progress_manager/source/progress_indicator_dialog.magik


_block
_local q << progress_indicator_dialog.new ( "Generating" )
q.interrupt_handler << handler.new ( _unset, _unset )
q.info_string << "The current series will contain approx 10 frames."
q.bar_label << "Generating frames..."
q.image << { :help, _unset }
q.activate ()

q.max_count << 10

_for n _over range(1,10,1)
_loop
_thisthread.sleep(1000)

q.progress_changed(n)
_endloop
_endblock
$


And there you have it... easy ways to let your users know the progress of a process.

Tuesday, May 15, 2007

Smallworld 2007 Americas Users Conference dates

The Smallworld 2007 Americas Users Conference dates have been set for September 9-12. You can read the announcement here.

Monday, May 14, 2007

Magik OLE Excel Reporting Performance Enhancements

I recently received a question about how to speed up Magik reporting to Excel using the OLE functionality. I you have ever used OLE from Magik to access other APIs you will know that it is a powerful but not overly fast way of connecting to 3rd party applications.

The questioner had indicated that they were writing data from Magik to Excel one cell at a time and then formatting each cell individually, too. While this is a logical way of doing a report it was taking a very long time to create a typical customer report.

My suggestions to improve data transfer and formatting performance were:

  • If your report is basically a row report, then you can copy one row at a time onto the Windows text clipboard in Magik. Then you can use an OLE call on Excel to paste the copied text into a range of cells. Be sure that the copied text has a delimiter that Excel can parse on. That way you can pass a large amount of data from Magik to Excel with a single OLE call.


_block
_constant delim_char << tab_char
_local a_frame << smallworld_product.applications.an_element().top_frame

# using a text output stream will let you cleanly create
# a delimited string later on.
_local temp_stream << internal_text_output_stream.new()

_for a_record _over records.fast_elements()
_loop
# using OLE, set your cell range...

data_list {a_record.attr1,a_record.attr2,a_record.attr3}
temp_stream.reset()
temp_stream.write_list_with_separator(data_list,delim_char)

a_frame.clipboard_text << temp_stream.string

# using OLE, do a "Paste" into the current cell
_endloop
_endblock
$
  • As for formatting: is there similar formatting for a row or column of cells? If there is, then keep track of how many rows or columns you have as you are copying/pasting the data into Excel. Then, once you have finished getting the data into Excel, format a range of columns or a range of rows or a range of cells that require common formatting. Again, that way you do the formatting with a minimum number of OLE calls
  • Have you looked at the Smallworld Explorer “Export to Excel” functionality? I think that core code (4.0 SWAF) might make use of an Excel template.
    • see collection_export_engine.export_to_excel

Monday, May 7, 2007

[slightly off topic] Google Maps of Boulder Bike Paths

This is not Smallworld-related but is of interest to me. Every since I moved to Boulder (Colorado, USA) in March, I have been amazed by the number of bicycle paths in the city. Apparently the city will even plow the snow off the paths after a snow storm.

As a new resident of this fair city, I have often wondered how I can get to my destination using the available bike path network. The city provides some great paper and pdf network maps and even plans to create a bike-route-finding website (aka on-the-fly directions for cyclists) later this year. But I think there is still a need for integrating the network maps with a web search engine like Google Maps. Well, Google has recently implemented Google My Maps and I have started using that freely-available tool to start building an overlay on Google Map search results. You can read about it and see the current state of the project at the Cycling in Boulder blog.

If you are interested in contributing to the data in Boulder, please let me know. And if you are interested in doing something for a bike path network in your town that would be great, too! I have two ultimate goals:
  • get people onto their bikes
  • have a large enough interest in something like this that Google takes note and introduces searchable/traceable bike path networks into their Google Maps data.

Tuesday, May 1, 2007

Magik Dialog Designer

Under the heading of “cool and useful Magik stuff”…


I recently had the opportunity to meet with a friend of mine who showed me his latest creation -- Dialog Designer



It is a tool that allows you to graphically layout your new Magik GUI with many of the commonly-used widgets, menubar, statusbar, and docks. Once you have layed out the GUI graphically and applied appropriate properties to the components, the Dialog Designer will stub out all the required Magik files, classes, methods, shared constants and XML config/gui files and place them in a new module directory structure.


While it will not write the Magik behaviour behind the widgets – you still need to write Magik code that does something when the button is clicked – it goes a very long way to building your :build_gui() method for you. And the nice thing is that tricky tasks like resizing rowcols or other widgets is now done automatically for you.


As a long-time Magik developer, I have never been too excited about GUI design because it can be a bit complicated (and time consuming) in Magik. Using the Dialog Designer will make the GUI design considerably easier and definitely save you a number of hours per GUI in development time.


The company that my friend works for has said that they will release the Dialog Designer to the SourceForge MagikComponents community. Until they have done that, I will keep the name of the developer and his company to myself. Stay tuned for a press release from them (hopefully soon) announcing the availability of this very cool and useful tool for Magik developers.

Tuesday, April 17, 2007

Using FME to report lines lengths per grid

I recently received a request from a reader asking me to give some more detail about how I accomplished the grid/valve KML report in "FME as a Spatial Intelligence tool for Smallworld". He wanted to do a slightly different report that summarized length of lines per grid.

Instead of explaining how I did that original report, I tried to see for myself if what he was asking was in itself an easy query in FME. I created a similar report using FME Workbench. You can see the FMW file here.

In less than an hour I was able to take as input Smallworld area and line features, and then compute the total length of all lines for a given area and then create both a CSV and KML report. CSV is a good and simple tabular report format. KML provides a nice visual representation that anyone using Google Earth can use to see your query results.

If you have FME Workbench, you can open the report_pipe_lengths_by_grid.fmw file and view the comments I put in there describing how the data/transformer flows work. If you do not have FME, you can download a fully functional trial version from http://www.safe.com.

The basic steps included:
  • read area and line feature sets from Smallworld into FME Workbench
  • use the LineOnAreaOverlayer transformer to split the line features at the area boundaries
  • use the LengthCalculator transformer to calculate the length of each line feature inside a given area boundary
  • use the Aggregator transformer to sum up the length of all lines per area feature
  • use the FeatureMerger transformer to assign the aggregate line length to each area feature
  • write results out to a KML file (click here for image of result)
  • write results out to a CSV file (click here for file)
The steps were fairly easy and demonstrate again the power of the FME tool for reporting and data manipulation applications.

[Shameless commercial plug: if you would like to explore how you could use FME to get meaningful reports out of your data, please contact me at the e-mail addresses listed on the "Contact Information" in the side bar of this blog]

Tuesday, April 10, 2007

getting a list of all users currently connected to Smallworld

I learned a new aspect of Windows command-line scripting the other day that I thought might be worth sharing here.

The requirement was to get a list of all users connected to a particular swmfs server and notify them that the server was about to go down.

I discovered that the FOR command is just what the doctor ordered for this task. The basic form is as follows...

FOR /F "skip=5 delims=@" %%i IN ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds') DO net send %%i Smallworld server going down in 2 minutes.


Where...

  • FOR /F ... IN ...
    • Command key words that tell the system that you want to iteratively process the results of some file or command
  • ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds')
    • the command that will have each line processed individually for some information. This particular command provides a list of all the user@computername that are connected to message.ds in the specified folder on %COMPUTERNAME%
  • %%i
    • the internal variable representing each line of the file/command
  • "skip=5 delims=@"
    • instructions to the command to skip the first 5 lines of swmfs_list and then also split the processed lines by the @ character.
  • DO
    • a command key word that indicates that what follows next is the command to be processed on each line of iterator
  • net send %%i Smallworld server going down in 2 minutes.
    • the NET SEND command is used to send the message "Smallworld server going down in 2 minutes" to the user specified in variable %%i. Remember that the variable %%i represents the first element of each line after it has been split by the @ character.

Special note about the variables. If you are running this script from within a BAT file, you need to reference the variable with two percent signs: eg., %%i. If you want to test this functionality at the command line, you need to reference the variable with only one percent sign: eg., %i.

I hope you find this useful.

getting a list of all users currently connected to Smallworld

I learned a new aspect of Windows command-line scripting the other day that I thought might be worth sharing here.

The requirement was to get a list of all users connected to a particular swmfs server and notify them that the server was about to go down.

I discovered that the FOR command is just what the doctor ordered for this task. The basic form is as follows...

FOR /F "skip=5 delims=@" %%i IN ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds') DO net send %%i Smallworld server going down in 2 minutes.


Where...

  • FOR /F ... IN ...
    • Command key words that tell the system that you want to iteratively process the results of some file or command
  • ('swmfs_list -full \\%COMPUTERNAME%\sw400\product\data message.ds')
    • the command that will have each line processed individually for some information. This particular command provides a list of all the user@computername that are connected to message.ds in the specified folder on %COMPUTERNAME%
  • %%i
    • the internal variable representing each line of the file/command
  • "skip=5 delims=@"
    • instructions to the command to skip the first 5 lines of swmfs_list and then also split the processed lines by the @ character.
  • DO
    • a command key word that indicates that what follows next is the command to be processed on each line of iterator
  • net send %%i Smallworld server going down in 2 minutes.
    • the NET SEND command is used to send the message "Smallworld server going down in 2 minutes" to the user specified in variable %%i. Remember that the variable %%i represents the first element of each line after it has been split by the @ character.
I hope you find this useful.

Tuesday, April 3, 2007

CST 4.1 allows single-quote strings

CST version 4.1 now allows single-quote strings.

Here's an example. Let's say you wanted to put in the short-hand notation for inch in a string. So you might try...


MagikSF> length << "5""
$
**** Error (parser_error): on line 1
length << "5""
^
Missing end of statement


The Magik compiler gets confused by your use of the double-quote character as a string delimiter and an inch unit.

To get around that in Magik, you would have to do something like...


MagikSF> length << write_string(5,%")
$
"5""


But as of version 4.1 you can use single quotes for strings...


MagikSF> length << '5"'
$
"5""


And if you wanted to do something similar with the short hand for feet units, you could do...


MagikSF> length << "5'"
$
"5'"



Incidentally...

MagikSF> '5' = "5"
$
True

Thursday, March 29, 2007

Gartner: GE Energy Plans Applications on Oracle Spatial Instead of on Smallworld GIS

The IT analysis and consulting firm Gartner has recently published a report titled GE Energy Plans Applications on Oracle Spatial Instead of on Smallworld GIS. You can find it here.

Accessing Smallworld data in Oracle without the Official GE Oracle Solution Suite

Through this article I hope to describe a way in which you can access Smallworld data in Oracle without using any of the components of the official GE Oracle Solution Suite.

A comment by Anonymous got me thinking about Smallworld SQL Server. Not to be confused with Microsoft's SQL Server database, Smallworld SQL Server is Magik code that creates a Magik server image that provides access for external applications into the Smallworld VMDS database.

If you open the Smallworld Help Documentation and search for...
  • "sql server" (in SW 4.0 help)
  • "22 sql server" (in SW 3.21 help)

... you will be taken to a link that provides answers to Frequently Asked Questions.

A description of SQL Server in the 4.0 help documentation states...
Essentially the Smallworld SQL Server provides remote access to a Smallworld database from Windows-based clients. Using the SQL Server and a suitable ODBC-enabled application, you can quickly and easily access all the data within a Smallworld database on a remote machine from a Windows-based PC.
Once you have set up a SQL Server image on a machine, you can install the Smallworld-supplied ODBC client driver on any machine in your network. As part of the configuration for that ODBC driver, you specify the host name and port of the SQL server image's machine. Then, you can view Smallworld data using any 3rd party tool that can read ODBC data sources.

That means that you can use other programming languages to get at Smallworld data and you can also use applications like Excel to read Smallworld data.

A couple of months ago I helped a client set up SQL Server and the way we tested it initially was to use Excel to read in data from Smallworld VMDS using the Excel "Import External Data" feature. Very cool, in my opinion.

And that leads me into the next piece of the "Smallworld Data in Oracle" puzzle. Did you know that you can configure Oracle to read external data as though that data actually resided in Oracle? That functionality is called Oracle Heterogeneous Services. Basically you can use that to access non-Oracle systems from the Oracle database server. One of the non-Oracle systems that are supported by this feature ODBC data sources. You can probably see where I am going with this:
  • use Smallworld's SQL Server to serve up data readable by an ODBC client driver
  • use Oracle Heterogeneous Services to make it seem to Oracle and your other applications that the Smallworld data resides in Oracle

Once you have the Oracle Heterogeneous Services configured, then all the tools you have available to you to process Oracle data are automatically available to you to process Smallworld VMDS data.

Performance issues
It is likely that anything served up by a Magik server image may be slower than your external applications expect. One way around this is for your Oracle techies to create their own VMDS/Oracle table synchronization application. Now that they have all the VMDS data available to Oracle tools, they can use the programming tools they are familiar with to update native Oracle tables with data from the ODBC-sourced Oracle tables.

Ideally you would not want to have two tables with the same data, but you might need to do some kind of synchronization to improve performance. The good news is that whoever writes the synchronization function will not need any Magik knowledge.

Geometries
Smallworld SQL Server does not serve up geometry data types. But if it is configured correctly it can serve up geometry coordinates which could be passed through the ODBC mechanism to Oracle. The Smallworld help documentation describes how to expose coordinates to the ODBC client.


Based on my customer interaction over the years, this technology does not seem widely known or used. I am not sure about the licensing requirements, but Anonymous suggested that the price might be considered expensive. I would suggest that if you are interested in this technology you could probably arrange a "try-before-you-buy" license with GE before paying anything for it.

I have limited experience with this configuration so I am curious to hear from anyone that has tried (or would be willing to try) this configuration. It seems that it holds some promise. I am interested to hear about performance issues and how easy it is to set up this scenario.

Tuesday, March 27, 2007

questions on the limitations of new GE Product on Oracle/Java

(For some reason the RSS feed does not show comment updates, so I wanted to respond to a comment in a new post.)

ninjacodemonkey had some comments on my original article about VMDS/Magik versus Oracle/Java that beg some questions. The comment ends with "Of course, this does depend on GE's willingness and ability to support two competing product lines. Obsolescence is a two-way street my friend."

Questions I have for GE:
  • do you intend to support both Oracle and VMDS product lines into the future? I have heard some unofficial comments about continued investment in the VMDS product line into the future. Could GE communicate how much they plan to invest in Magik/VMDS in the next 5-10 years. Clearly the level of investment would indicate GE's commitment to both both product lines.
  • I am at a bit of a disadvantage in that I was not at GITA 30. Were any readers of this blog at the conference and did any of you hear any communication from GE about investment in existing and new products on VMDS/Magik? I am interested in your comments.

Questions I have for ninjacodemonkey:
  • you state "It [GE's Oracle technology] appears to be a far less functional, less flexible and lower performing application." At the risk of making this sound like a college exam question...
    • can you give an example of less functional
    • can you give an example of less flexible
    • can you give an example of lower performing
  • "mediocrity of Java"
    • Why should customers choose the "proprietariness of Magik" over the "mediocrity of Java"? I think Pedro and Tom have some valid concerns about the cost of Magik developers compared to Java developers. Do you think overcoming the "mediocrity of Java" is worth the extra cost of acquiring Magik skills?
  • It seems to me, though, that some customers might actually benefit from the Oracle/Java product. Can you think of any customer profile that would realize benefits with the new product line?
  • Finally, what do you mean by "Obsolescence is a two-way street my friend."?
Those are my questions for the day. As always I welcome all opinions whether from ninjacodemonkey, stylechimp, casechimp, or any other form of primate.

Monday, March 26, 2007

talk to me GE

If you were able to attend GITA 30, you would have likely heard the announcement "GE Energy to Introduce New Oracle-Based Product Portfolio for Electric Disturbition Utilities" (be sure to read the comments for this article). It is also likely that if you had any questions and concerns at that meeting that there would have been plenty of GE representatives there to answer your questions and assuage your concerns. It sounds to me that GE had a consistent message that they were able to get out to the attendees.

If you were not able to attend GITA 30 it is likely that you now have more questions than answers about this strategic announcement by GE. I was unable to attend GITA so I count myself in that group of under informed Smallworld users/partners. If this year's attendance was any indication, then I imagine that many others in the Smallworld community were also not able to make use of the direct interaction with GE representatives to have their questions/concerns heard. And GITA attendance is largely from the Western Hemisphere. European, Asian and African Smallworld community members would have had even less chance to get their questions answered.

I suspect that this year's Smallworld Users Conferences will be dominated by presentations on this new architecture (and hopefully demonstrations), but it seems like a long time to wait until September/October to have our questions answered.

Maybe GE would consider some of the following communication options:
  • press release
  • send a more detailed note out to all Smallworld users/partners
    • Smallworld Helpdesk has a fairly efficient mechanism in the SupportCentral site to send out blanket e-mails to all users. Use this feature to give a more-detailed-than-the-press-release announcement to the Smallworld community
  • Online forum
    • I understand that there might be concerns about making information available on an uncontrolled list, so extend the Global Customer Forums (see On-line Smallworld resources (closed and open)) to include a forum where customers/partners are invited to post any questions they have about the new product.
    • Commit to having GE management/technical staff answer each post in a timely manner.
  • FAQ
    • As the questions start coming in to GE as phone calls and e-mails, compile a list of Frequently Asked Questions and post them in the forum.
  • online demonstrations
    • provide WebEx demonstrations of a more technical nature where we can see how the new tools are supposed to work.
Most importantly, please talk to us sooner than later. It would be helpful even to send out a note (another press release, maybe) that indicates dates and types of information will be made available. For example "GE will have a new customer forum opened by April 1", etc...

As always, I welcome readers' comments. In particular, I am interested in what kind of communication you would find helpful to address your questions or concerns about this new "non-GIS" spatial product announcement by GE.

Tuesday, March 20, 2007

How to create Smallworld Competitors

I think I might hammer the open source / public license idea a bit more. Hopefully the more we do this the more likely that we might actually get a response from GE. I have tried to get some responses from GE about their official view of open source or public license as it pertains to Smallworld but my requests have vanished into the e-mail ether.

Here are some ideas that I would like to see GE address:

  • open source any modules that will provide easier access for external systems to query/manipulate data in the VMDS database.

    A recent comment to one of my articles suggested doing this for Smallworld's SQL Server product. This is a tool that allows Windows applications to see Smallworld VMDS data as ODBC data sources. I think this functionality would be a great candidate for open source by GE. I think taking that action could reenergize the Smallworld user base and show real commitment by GE to address customer concerns about lack of access to VMDS data.

    The pertinent questions I think are: what would it hurt GE to open source SQL Server? What would it benefit GE to open source SQL Server? I cannot think of any real negative impacts to these actions. As the Anonymous commenter suggested, not many customers use the Smallworld SQL Server product anyways.

    Opening/Freeing that product could spur innovation in open access to VMDS and may actually provide a positive benefit to customers in a short time frame.
  • There is a great article at Ask The Wizard called Creating Competitors. "Ask the Wizard" is a blog run by FeedBurner CEO Dick Costolo.

    GE's CEO Jeff Immelt stresses learning as a key to success in today's dynamic world. I would recommend the Creating Competitors article to the Smallworld managers as an article that at least gives them some ideas about how someone has built a successful software organization in today's dynamic world.

    Here are some key points from that article:
    • Provide an open API
      • even if you control your core code, providing an open API into your system will spur innovation.
        • I think that Smallworld is already doing this with products such as Smallworld Automation and the new Service Oriented Architecture and Internet Application Server. But it would not hurt to review whether these are the types of APIs that will allow others to build on your product and not on your competitors product. I would also not hurt to make these open API mechanisms more widely known to current and potential Smallworld customers.
    • Provide a free version of your software
      • people will always want a free version of something. If you don't have one, they will get it from your competitor. Who do you think they will look to when they are ready for an upgrade?
        • I do not know of a free version for Smallworld, but it might be worth pursuing something like that. I seem to remember a few years ago GE handed out demo CD's of Smallworld that actually installed Smallworld on your computer with an easy install wizard interface and then let you work on the Cambridge demo database.
        • Maybe once GE officially moves to Oracle/Java architecture, they could make the VMDS/Magik architecture the free version. VMDS is very portable and could be easily licensed for free use. Compare this to trying to license Oracle for free use.
These are my thoughts. I acknowledge that I do not have the responsibility nor knowledge to run a large organization such as GE Energy's Smallworld group. I am sure that the people that do that are making the best decisions they can in each situation. What I am trying to initiate with this article is a public discussion about how to continue to make Smallworld thrive in a changing market place. Until GE enters this public discussion, I am afraid these posts may sound more like monologues than the multilateral conversations that I hope for.

As always I would love to hear reader comments to anything I post.

Monday, March 19, 2007

Recent Smallworld Highlights

I found some interesting links on the GE Energy Geospatial Asset Management page.

  • An official view from GE on the outlook of the GIS industry in 2007.
  • Cambridge office holding an Open Day. I am curious about what an "Open Day" is. The flyer does not give too many details. Anyone in the UK or Ireland attending and willing to report on it afterwards?
  • I'm always interested in hearing about large companies that see Smallworld as the best solution for them. Find out more by reading a GE Energy executive interview with Dr.-Ing. Andreas Bordonetti, Head of Power Distribution, Azienda Energetica S.p.A.