Monday, January 18, 2010

Visualization in Smallworld

(the cool stuff/demo is at the end of this longer-than-normal post)

Lately I have taken an interest in Visualization techniques within the Smallworld context. I am using the term "Visualization" in the sense of using visual representations to simplify complex data relationships. The term has various other related meanings as well.

In a sense, we work with Visualization techniques in Smallworld every day... we deal with a visualization of geographic relationships between objects that are of interest to us. When you create a report using the Schematics Generator you are using visualization techniques to help you comprehend complex relationships by keeping only the salient information. Any Thematic Mapping tool that you might use also lets you visualize complex data relationships easier.

I have been intrigued by the use of heat maps to provide a way of showing intensity of some attribute spatially. I still haven't got my mind around all the applications that might be used in the utility space for heat maps. Some that come to mind are: heat map of all the transformers that are about to blow up; heat map of all the outages in a storm. While the pictures look cool, I'm trying to better understand what value a heatmap image would bring to GIS users that cannot be already provided by other kinds of thematic mapping applications.

I have a feeling that the full potential for visualizations such as heat maps and density maps has not been realized within the Smallworld realm but I sure could use some ideas from readers about what possible applications might be.

My colleague Graham Garlick has done alot of work with generating heat maps, density maps and TIN density maps within Smallworld and I want to put a plug in for him in this post. If you are interested in knowing more about how to generate these kinds of maps in Smallworld, please contact iFactor Consulting and ask for Graham's heat map demo.

So now to the final point of this post. The visualizations that I have already referred to are typically georeferenced rasters that overlay on a map and give you a sense of the spatial relationships of your data. I have been exploring another area that involves network visualizations.

My first foray into this realm was by making use of a software package called Graphviz to visually show the results of network follower trace results in PNI. This was really helpful in debugging network follower results. Typically, a Magik developer debugs network follower trace results by querying the nf_links of a network follower and trying to understand how they are linked. Using a visualization tool like Graphviz allowed me to easily create a .jpg file that I could view in any image viewer. By looking at the image output, I could easily spot places where the trace stopped unexpectedly or went down an unexpected branch. Sometimes a picture really is worth a thousand words (or a thousand minutes of Magik debugging!!!)

My second attempt at network visualization came from a slightly different tack. If visualization techniques are meant to represent complex interdependencies in an easy-to-understand manner, what is a really complex difficult-to-understand scenario in Smallworld that I could try to visualize? The answer to that question is: XML configuration file dependencies. Anyone that has ever tried to modify config.xml, gui.xml or a host of other XML configuration files in Smallworld will tell you that it can be somewhat mystifying as to which version of the file to change. Any given XML file can inherit from another XML file and its elements can supersede the same elements of a "superclass" XML file. In addition to that, XML files are considered module resource files. That means that they can be placed in customisation products which have a different pathname than the same module found in the core product. If this all sounds confusing, rest assured it is. I have seen many instances (and caused some myself) where I have changed the wrong XML file or changed too many "inheriting" XML files when it would have been more efficient to change a "superclass" XML file. But it was always difficult to get my mind wrapped around the big picture of XML file dependencies. Until now...

I have take a bit of time to write a Magik wrapper around the Graphviz engine. That in itself is not that remarkable and anyone with interest in that could do it. What is of more interest to me is the result that I was able to create with this Graphviz tool showing the interdependencies of XML files. As an example, click here to see the dependencies of the Electric Office config.xml files for applications.

  • If you look at the image, you will see information about which XML files inherit from others.
  • You will see the full pathname of each file. This will help you know whether to look in the actual module resource file or in a corresponding customization product file
  • You will see some cells that are green. These are the cells that represent XML file elements that have a material change relative to a "superclass" file.
  • You will see some cells that are red. These are cells that represent XML elements that exist in a configuration file but whose contents are no different than the same element in the inherited XML file. Many times it can be confusing to have the exact same XML element contents in two files, so looking at this visualization gives you an easy-to-understand representation of which duplicate XML elements you can remove.
  • You will see some cells that are yellow. These represent XML elements that require further investigation. They indicate that the current XML element is an exact duplicate of another XML element somewhere in the graph. If you notice yellow cells in two XML files that share the same parent, you might want to consider moving those XML elements up the inheritance ladder.
  • And finally, just seeing the overall flow of XML inheritance can give you a quick indication of whether you have too many dependencies or maybe where it would make the most sense to insert a suitable XML file "superclass" that would serve your own customizations most efficiently.
This was just a quick demonstration of how a network visualization tool can be used to simplify complex interdependencies in the Smallworld environment. I would love to hear from readers about other complex scenarios that might lend themselves to this sort of network visualization.


Anonymous said...

Visualization is, I concur, a grossly underdeveloped aspect of GIS. Smallworld has produced some pretty good results at visualization: the GIS itself, CASE tool, Schematics Generator, Spatial Intelligence, etc. My observation about all of these is that they require a fairly high level of training to operate, and are therefore not available to "the masses", and are, by now, “old school”.

I think Google Maps has done a wonderful job of lowering the training required to do some very sophisticated analysis. Leveraging the work that Google has done, there’s a local guy who’s taken public domain data and done some really exiting analysis and presentation of what’s visible to a person from any point on the terrestrial earth: [not so exciting if the vantage point is from the flat, plain states though--the interface could use some polishing, but for a privately developed site, it’s impressive]. The data were all preexisting; he’s done the analysis and created a graphically descriptive display of the metadata that is novel and accessible to the lay person.

The part of the human brain dealing with visual input is relatively large and well developed. With a couple of million years’ practice, we’ve become good at recognizing visual relationships. Let’s better mash-up that engine with the GIS. How can we engineers, with all the tools we have at our disposal, make visible, for a greater population, the very expensive data that our clients have painstakingly gathered? The data are there. We need to break out of old thought processes and develop some new ones, as was done at the above website. I suspect it may be easier for those just entering the work force to come up with new visualizations (OTOH, was developed by a veteran of the work force).

I like your posts. They keep me thinking.

Bruce Morehouse

Your XML visualization tool is great. It begins to address Smallworld’s functionality hole of missing XML tools.

Anonymous said...

Hi Alfred,

I poked around with Graphviz many months ago thinking there would be some way to create a free alternative to the Schematics Generator. Other tasks pulled me away but I think your visualization comments have re-energized me a bit. I can now see some other uses that I haven't before.

Quince Lunde

Reinhard said...


about one year ago, I've written a magik procedure producing input code for graphviz to display module dependencies within one layered product, which we use to plot on a wall paper.

Your XML-dependencies plot is very nice.