Waterfall software development is back with a vengeance

There are some great articles on Waterfall development process at http://www.waterfall2006.com. (You may need to read between the lines to understand what the authors are getting at. Note the conference date in the top left corner of the home page.)

My personal favourites:

• wordUnit: A Document Testing Framework by Kent Beck
• Refuctoring by Jason Gorman
• Defect-full Code: Ensuring Future Income with Maintenance Contracts by Kay Pentecost
• Testing: Saving the Best for Last by Lisa Crispin

Enjoy!

Identify and Recover Deleted Data without a database restore

Customers will come to us and say... "someone deleted/updated some data a while back and we only discovered it now. We don't want to restore a backup because that will undo the work from the last week. Help!"

It may be possible to "rescue" the old data if you still have a suitably older checkpoint somewhere in your database.

First, you need to get a full inventory of all the checkpoints available to you. It is possible that there exist checkpoints in your database that you do not even know about. The following code will give you a full inventory and list the checkpoints (and alternative that they are in) sorted from most recent to oldest creation times.

_pragma(classify_level=debug)
_method ds_version_view.debug!all_checkpoints()
 ## debug!all_checkpoints() : sorted_collection
 ##
 ## returns a sorted_collection of all checkpoints in self and
 ## the tree beneath self sorted from most recent to oldest.
 ## The elements returned are actually simple_vectors of the
 ## form:
 ##       {checkpoint A_CHECKPOINT,string ALTERNATIVE_NAME}
 ##
 _local checkpoints << sorted_collection.new(_unset,
          _proc(a,b)
           >> a[1].time_created > b[1].time_created
          _endproc)
 
 _for a_view _over _self.actual_alternatives()
 _loop
  checkpoints.add_all(a_view.checkpoints.map(_proc(a_cpt)
           _import a_view
           >> {a_cpt,a_view.alternative_path_name()}
          _endproc)) 
 _endloop


 _return checkpoints
_endmethod
$
_pragma(classify_level=debug)
_global show_all_checkpoints <<
_proc@show_all_checkpoints(a_view)
 ## show_all_checkpoints(ds_version_view A_VIEW) : _unset
 ##
 ## shows all checkpoints and related information for A_VIEW and
 ## tree.

 _for cpt_info _over a_view.debug!all_checkpoints().fast_elements()
 _loop
  show(cpt_info[1].time_created,cpt_info[1].checkpoint_name,cpt_info[2])
 _endloop
_endproc
$

To run this code, do something like...

MagikSF> show_all_checkpoints(gis_program_manager.cached_dataset(:gis))
$

... which will give you output like...

date_time(10/30/07 17:34:44) "is!insync_1193790807" :||oracle_sync_parent|
date_time(10/30/07 17:34:44) "is!insync_1193790807" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 16:32:54) "is!insync_1193787116" :||oracle_sync_parent|
date_time(10/30/07 16:32:53) "is!insync_1193787116" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 15:33:09) "is!insync_1193783570" :||oracle_sync_parent|
date_time(10/30/07 15:33:08) "is!insync_1193783570" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 14:34:02) "is!insync_1193780009" :||oracle_sync_parent|
date_time(10/30/07 14:34:02) "is!insync_1193780009" :||oracle_sync_parent|oracle_sync|
date_time(10/30/07 13:33:52) "is!insync_1193776407" :||oracle_sync_parent|

Once you have identified a checkpoint that is old enough to contain your old data, you take one of the two following two approaches (comments in the code explain the differences).

_block
 # if you want to use your GIS tools to see what the data looked
 # like at a previous checkpoint, then get a handle on the
 # current view and go that checkpoint in READONLY mode.

 # DO NOT switch to :writable while at this alternative unless
 # you are ABSOLUTELY SURE of what you are doing.  The safest
 # action to take once you are done viewing data at the old
 # checkpoint is to switch to the "***disk version***"
 # checkpoint in that alternative before continuing.
 _local v << gis_program_manager.cached_dataset(:gis)
 
 v.switch(:readonly)
 v.go_to_alternative("|oracle_sync_parent",:readonly)
 v.go_to_checkpoint("is!insync_1192840224",:readonly)
_endblock
$

... OR ...

_block
 # if you want to keep the main dataset at the current view and
 # then get a second handle to the old_view so that you can copy
 # old data (eg., incorrectly deleted/updated data) into the
 # current view, then get a handle on a REPLICA of the current
 # view and go that previous checkpoint in READONLY mode.

 # DO NOT switch to :writable while at this alternative unless
 # you are ABSOLUTELY SURE of what you are doing.  The safest
 # action to take once you are done viewing data at the old
 # checkpoint is to switch to the "***disk version***"
 # checkpoint in that alternative before continuing.
 _global old_view

 # once you have finished using OLD_VIEW, be sure to :discard()
 # it.
 old_view << gis_program_manager.cached_dataset(:gis).replicate()
 
 old_view.switch(:readonly)
 old_view.go_to_alternative("|oracle_sync_parent",:readonly)
 old_view.go_to_checkpoint("is!insync_1192840224",:readonly)

_endblock
$

If you are going to use data in the old_view to update/recreate data in the current view, then you might want to explore using classes/methods such as

record_transaction.new_update()
record_transaction.new_insert()
ds_collection.at()
ds_collection.insert()
ds_collection.clone_record()
ds_collection.update()
ds_collection.insert_or_update()

[NOTE 2014 June 02: Reinhard Hahn sent me some new code that supercedes the code he originally posted in the comment to this post]

_pragma(classify_level=debug)
_iter _method ds_version_view.all_available_versions(_optional top?)
  ## yields all available Versions of a replicate of
  ## _self in the following order:
  ##
  ## alternatives (if TOP? isnt _false: beginning with ***top***)
  ## -- checkpoints (beginning with :current)
  ## -- base versions for checkpoints (only for alternative_level
  ##    > 0)
  ##
  ## for each version yields
  ## A_VIEW,ALTERNATIVE_PATH_NAME,CHECKPOINT_NAME,BASE_LEVEL, CHECKPOINTS_TIME_CREATED
  ## where A_VIEW is always a replicate, which will be discarded
  ## outside this method. The order of the CHECKPOINTS will be in
  ## reversed chronological order (i.e. beginning with the most
  ## recent one)
  ##
  ## BASE_LEVEL is 0 for the version itself, -1 for the immediate
  ## base version, -2 for the base version of the base version
  ## and so on 
  ##
  ## example:
  ## v,:|***top***|,:current,0,v.time_last_updated()
  ## v,:|***top***|,"cp_1",0,cp_1.time_created
  ## v,:|***top***|,"cp_2",0,cp_2.time_created
  ## v,:||alt1|,:current,0,v.time_last_updated()
  ## v,:||alt1|,:current,-1,v.time_last_updated()
  ## v,:||alt1|,"cp11",0,cp11.time_created
  ## v,:||alt1|,"cp11",-1,v.time_last_updated()
  ## v,:||alt2|,":current",0,v.time_last_updated()
  ## ...
  ##
  ## NB:
  ## If the different versions of _self have different
  ## datamodels, you might be annoyed by messages like the following:
  ##
  ##  compiliing my_table.geometry_mapping
  ##  fixing up details for dd_field_type(my_field_type) : Version 12345, transition to 54321
  ##  Defining my_other_table
  ##
  ## To avoid this, you might want to suppress any output by
  ## redefining the dynamics !output! and suppressing the output
  ## of the information conditions db_synchronisation and
  ## pragma_monitor_info. Of course you have then to put any
  ## wanted output to other output targets like !terminal! or a
  ## locally defined external_text_output_stream. So your code
  ## might look like the following: 
  ## MagikSF> t << proc(a_database_view)
  ##  _handling db_synchronisation, pragma_monitor_info _with procedure
  ##  _dynamic !output!  << null_text_output_stream
  ##
  ##  # ... any more preparation code ...
  ##
  ##  _for r,alt_pname,cp,base_level,cp_time _over a_database_view.all_available_versions()
  ##  _loop
  ##
  ##    # ... my code running in the loop, for example:
  ##    !terminal!.show(alt_pname,cp,base_level,cp_time)
  ##    !terminal!.write(%newline)
  ##    !terminal!.flush()
  ##    # ...
  ##
  ##  _endloop
  ##
  ##  # ... any summary output, for example:
  ##  !terminal!.write("Ready.",%newline)
  ##  !terminal!.flush()
  ##  # ...
  ##
  ## _endproc.fork_at(thread.low_background_priority)
  ##
  ## If you don't want to use these !terminal! calls, you could
  ## also dynamicall redefine !output!:
  ## MagikSF> t << proc(a_database_view)
  ##  _handling db_synchronisation, pragma_monitor_info _with procedure
  ##  _dynamic !output!  << null_text_output_stream
  ##
  ##  # ... any more preparation code ...
  ##
  ##  _for r,alt_pname,cp,base_level,cp_time _over a_database_view.all_available_versions()
  ##  _loop
  ##    _block
  ##      _dynamic !output! << !terminal!
  ##      # ... my code running in the loop, for example:
  ##      show(alt_pname,cp,base_level,cp_time)
  ##      # ...
  ##    _endblock
  ##  _endloop
  ##
  ##  !output! << !terminal!
  ##  # ... any summary output, for example:
  ##  write("Ready.")
  ##
  ## _endproc.fork_at(thread.low_background_priority)

  _local change_alt_level <<
    _if top? _isnt _false 
    _then
      >> - _self.alternative_level
    _else
      >> 0
    _endif
  _local r << _self.replicate(change_alt_level,_true)
  _protect
    _for rr _over r.actual_alternatives(:same_view?,_true)
    _loop
      _local alt_name << rr.alternative_path_name()
#      _if alt_name.write_string.matches?("*{consistency_validation}") _then _continue _endif
      _local cps << rr.checkpoints.as_sorted_collection(_proc(lhs,rhs)
                      _if lhs _is :current
                      _then
                        _if rhs _is :current
                        _then
                          _return _maybe
                        _else
                          _return _true
                        _endif
                      _elif rhs _is :current
                      _then
                        _return _false
                      _endif 
                      _local res <<  rhs.time_created _cf lhs.time_created
                      _if res _isnt _maybe
                      _then
                        _return res
                      _endif
                      _return lhs.checkpoint_name _cf rhs.checkpoint_name
                    _endproc)
      cps.add(:current)
      _for cp _over cps.fast_elements()
      _loop
        _local cp_name
        _local cp_time 
        _if cp _is :current
        _then 
          rr.rollforward()
          cp_name << :current
          cp_time << rr.time_last_updated()
        _else
          cp_name << cp.checkpoint_name
          rr.go_to_checkpoint(cp_name)
          cp_time << cp.time_created
        _endif 
        _loopbody(rr,alt_name,cp_name,0,cp_time)
        _if rr.alternative_level > 0
        _then
          _local rrr << rr.replicate(:base,_true)
          _protect
            _local level_diff << -1
            _loopbody(rrr,alt_name,cp_name,level_diff,rrr.time_last_updated())
            _loop
              _if rrr.alternative_level = 0 _then _leave _endif
              rrr.up_base()
              level_diff -<< 1
              _loopbody(rrr,alt_name,cp_name,level_diff,rrr.time_last_updated())
            _endloop 
          _protection
            rrr.discard()
          _endprotect
        _endif 
      _endloop
    _endloop
  _protection
    r.discard()
  _endprotect
_endmethod
$

Smallworld/FME Configuration Tips and Tricks (part 2)

I recently received a number of questions from a colleague that had recently upgraded to FME 2007 as part of the CST 4.1 upgrade. It seemed to me that the questions and answers were actually a continuation of the original Smallworld/FME Configuration Tips and Tricks so I present them here as "part 2". By the way, there is a growing list of Smallworld/FME FAQs at http://www.fmepedia.com/index.php/Category:GE_Smallworld_Format_FAQ

Question: I previously used the Scale function in my fme files to adjust coordinate values between database units (mm) and coordinate system units (m). The factor would be 0.001 or 1000 depending on if I was exporting or importing.

FACTORY_DEF SWORLD SamplingFactory     \
INPUT FEATURE_TYPE *                   \
@Scale(0.001)                          \
SAMPLE_RATE 1 

I used the Workbench for the first time to create an fmw file for exporting objects. I used the Scale Transformer which appears to work. Is this the right approach or should I create an FME coordinate system that matches our Smallworld database coordinate system? This must be a common problem for Smallworld/FME users.

Answer: I typically create a custom FME coordinate system that matches the Smallworld application coordinate system and then let FME do all the unit conversions. That way you don’t need to mess around with the scale transformer.



Question: Below is code based on the example in your FME presentation. Would it be better to rewrite this to include a loop and to execute the fme command for each table or run it as it is?

_block
_global fme_tics_server
_local current_server

# start the fme_tics_server if not already running
_if (current_server << fme_tics_server.current_server) _is _unset _orif
 _not current_server.acp_running?
_then
fme_tics_server.start(_unset,_unset,:gis)
current_server << fme_tics_server.current_server
_endif

# indicate which records to prepare for export
_local vc << gis_program_manager.cached_dataset(:gis).collections
_local export_records << rwo_set.new_from(vc[:table1])
export_records.add_all(vc[:table2])
export_records.add_all(vc[:table3])

current_server.set_output_rwo_set(export_records)
# run the command line FME
_local command << {"\\gis\FME\fme","\\server\mapping\FME\sworld2mapinfo.fmw",
  "--DestDataset_MAPINFO",'"\\server\mapping\FME\fme2007"',
  "--PORT_SWORLDSWAF", current_server.port_number.write_string}

system.do_command(command)
_endblock
$

Answer: If all of your export collections are processed by the same FMW file (which I see it is; which I also think is a good thing) then I do not see a reason to have to put the code in a loop.



Question: Is there a way to create a log file that can be examined during and/or after the translation? I run the code above and have no feedback whether the translation is running or not.

Answer: Probably the best way to get feedback from FME about the progress of the migration is to modify your FMW file to specify a Log File:



If you want to be more flexible, then Publish the Parameter…



... and then you can modify your Magik code to dynamically set the LOG_FILE value at export time ...

   _local command << {"\\gis\FME\fme","\\server\mapping\FME\sworld2mapinfo.fmw",
  "--DestDataset_MAPINFO",'"\\server\mapping\FME\fme2007"',
  "--PORT_SWORLDSWAF", current_server.port_number.write_string,
"--LOG_FILE","\\server\mapping\FME\fme2007\mapinfo.log"}

Question: I could previously export objects with mapped geometries by only specifying the mapped geometry field in the fme file. I found that nothing gets written in the new version unless I add each geometry field into the Workbench. Is this correct?

Answer: You are correct that each mapped geometry must be accounted for in your FMW file. The way you have shown your data features...



... is valid. If you want to make your workspace a bit cleaner-looking, you can use a “merge filter” ...



... and then you can flow all your data from that single “Merged” feature to your transformers ...

Dialog Designer v1.3 available

Graham Garlick's latest update to Dialog Designer is now available as v1.3. You can find it on the Magik Components SourceForge project. Based on the dates of the releases, v1.3 is not yet in the packaged mclib release. But if you have a CVS connection to SourceForge you can update your source tree with his latest code. You can also visit iFactor Consulting's web site to download the code here.

This version has the following additional features

• added style selection widget (place a gui element for selecting a style)
• removed dependency on the path_select plugin
• added file/directory selection widget (place a button that launches the file/directory selection dialog)
• added plugin widget (embed an action from some other plugin or an entire other plugin)
• added auto-generation of supplementary gui/framework & plugin files so developers can expand/customize produced GUI functionality in a place that the Dialog Designer won’t overwrite if the GUI layout is tweaked and re-built.
• added FAQ section to the main documentation
• de_de messages updated
• fr_ca messages created

discovering which SWAF config.xml file to change

Have you ever wondered exactly where in the XML config hierarchy a particular plugin is defined? With XML file inheritance in SWAF, it can sometimes get very complicated.

I have created some enhancements to the XML parsing and plugin creation methods that allow you to track the one or many XML FILENAMES where a plugin is defined.

Usage:

1. load this file
2. start the application whose configurations you want to trace.
3. get a handle on some plugin and then ask it for its debug!source_files. It will return a rope of XML file names that contain that plugin's definition in the order that they were encountered. I.E., the last XML file in the rope is the one that the plugin used for its definition.

For example:

MagikSF> smallworld_product.applications.an_element().plugin(:map_plugin).debug!source_files
$
sw:rope:[1-2]
MagikSF> print(!)
$
rope(1,2):
1     ".....\sw_user_custom\pni_gui\resources\base\data\config.xml"
2     ".....\sw_user_custom\custom_application\resources\base\data\config.xml"

so, this tells you that the "custom_application" config.xml file overrides the same plugin settings in PNI_GUI config.xml. If you want to make a change to the :map_plugin you will need to make it in CUSTOM_APPLICATION for the changes to take hold.

DISCLAIMER: DO NOT PUT THIS IN YOUR PRODUCTION CODE. This script is intended only for debugging your XML configuration files and should not be in any image that is writing data to a production database.

What is RSS?

I noticed today that about 45% of subscribers to this blog use the e-mail subscription option instead of RSS. While e-mail is definitely a good way to be notified of updates to the blog I would suggest that you learn about RSS and how it might lighten your inbox load.

The best explanation I have found so far is a 3 minute video called RSS in Plain English (the website also has links to information in other languages).

One advantage of RSS over e-mail notification is that your inbox is not filled with update notifications at inconvenient times. Using RSS you go to your RSS page (or reader application) when you want to.

SECO Energy gives customers access to outage data

There is a good article in the September issue of Transmission & Distribution World titled Customers Access Online Outage Data. It was written by Barry Owens of SECO Energy. In case you missed it, Barry also gave the featured Customer Presentation at the recent Smallworld 2007 Americas Users Conference (presentations are here).

This is a project that I had heard about before and one of the things I had wondered was: "in a power outage situation, how will affected customers have the power to run their computers to check their outage status online?". Barry answers that question (and many others) in a well written presentation of their project.

Congratulations also to Peter Manskopf at GE for leading this successful project.

MUnit Testing Framework

I recently received a request from a reader wondering where the unit test application was on my blog. I don't have a post on that yet so I thought that now was as good a time as any to make one.

I'm not going to go into the details here. Suffice it to say, that the MUnit framework is based on the original JUnit framework but for Magik instead of Java.

I have found it helpful when developing code to create some kind of automated test scripts. There is much that can be debated about how much or how little test scripting to do.

Mark Field had a detailed presentation at the Smallworld 2007 Americas Users Conference this year titled "Implementing MUnit for Smallworld". I imagine you could contact him for the presentation or maybe we can get him to post more information on his blog. If you have the username/password, you can download the presentation from https://www.kinsleyassociates.com/ge/smallworld/presentations.asp

[NOTE: The above links are old. You can find the presentation slides and download zip file at http://sw-gis.wikidot.com/magik-munit]

GE also had/has a version of MUnit but from what I understand they are reluctant to release it to the community. I found that there was a version of MUnit created by Jan Vorel at http://www.janvorel.com. Unfortunately the current version of that site is not very user friendly like it used to be and you can no longer download Jan's code from there. But if you look at http://web.archive.org/web/20030406030601/www.janvorel.com/ you will find an archived version of the web site that still has a working link to his version of MUnit. It only works on CST 3.3 but I found that with a few modifications it works in CST 4.x as well.

The implementations of MUnit that I have seen do not allow you to test mouse/keyboard input. Mark's presentation describes how he enhanced MUnit to include those features.

One enhancement I did for the MUnit framework some time ago was to create a new test listener that exported the test results as XML and/or HTML. An example of those results is here.

You can click here for a Yahoo Groups search result on MUnit.

FME Workbench Design Patterns

You could consider the FME Workbench to be a graphical IDE (integrated development environment). Therefore it makes sense to employ the concept of Design Patterns to FME Workspaces. For anyone that has worked in an object-oriented language (like Magik) Design Patterns are not a foreign concept.

There is a presentation on FMEpedia called Introduction to Design Patterns that is worth reading by anyone wanting to make more efficient use of their FME Workspaces. The included PowerPoint presentation has a number of design patterns. Note that this presentation was inspired by a discussion the FME people had with someone from the Smallworld community and the examples in the presentation are all Smallworld-FME ones.