Tuesday, March 6, 2007

goodbye VMDS/Magik, hello Oracle/Java?

Well, not quite.

Earlier this week, GE issued a press release "GE Energy to Introduce New Oracle-Based Product Portfolio for Electric Disturbition Utilities". I had heard rumblings of such an announcement in the days leading up to the release and was very anxious to understand what GE Energy would be announcing.

At first glance, this announcement seems to suggest another significant paradigm shift in the Smallworld application. The most recent significant paradigm shift was the move to the Smallworld Application Framework (SWAF). Some customers are still preparing for the SWAF upgrade. To be faced with an announcement like this (Smallworld on Oracle/Java) might come as a bit of a shock. But if you read the press release carefully, you will see that this announcement is geared to small- and mid-sized electric utility companies.

If you are a larger utility company that has invested considerably in VMDS and Magik technology, Bob Gilligan states that this new product line will complement the current Smallworld offerings. I find it significant that Mr. Gilligan used "complement" and not "replace".

Clearly GE Energy needs to grow more license sales and it seems to me that offering Smallworld in an Oracle/Java environment will make the sale more appealing to smaller users who do not have the resources to commit to learning another database and programming language. Having said that, I suspect that as some of the mid-sized utilities start asking the Oracle/Java application to do greater and greater things (especially around long transaction management and customising the data model), it might be a more natural step to direct them to VMDS/Magik. I think VMDS/Magik will go away at GE Energy's peril. I am of the opinion that the current VMDS/Magik package provides better solutions for some of the Smallworld user's business requirements than Oracle/Java is capable of.

GE Energy may spend more energy developing the Oracle/Java side of things because that is where the revenue growth may be, but I would strongly urge them to continue to support VMDS/Magik either directly or via some Open Source framework so that the Smallworld user community can service it.

I have included some highlights (with my comments) of the press release.
I have formatted direct quotes from the press release in Georgia font. All other comments are mine.

The highlights of the announcement include:
  • GE Energy is collaborating with Oracle to expand GE’s geospatial product portfolio by developing a network design and maintenance application based on Oracle® Database 10g, Oracle Spatial 10g and Oracle Fusion Middleware.
  • This solution is designed specifically to meet the needs of the small- to mid-size North American electric distribution market.
    • One can take a half-full or half-empty view of this press release. The statement above makes me lean towards the half-full view. It seems that GE Energy is trying to increase its market share and trying to offer something that the smaller utilities are asking for. My half-full view continues when I realize that there are opportunities in this approach for current Smallworld customers. Let's say your large utility acquires a small- to mid-size utility. Would you rather integrate your GIS system with another Smallworld platform or with an entirely different GIS vendor? I think that any move that keeps Smallworld viable in the market place and does not take away the product that already works for current customers can only be good for the entire Smallworld user community.
  • GE Energy has been working with selected customers in the development of the application and expects beta testing to commence later this year.
    • I'm looking forward to seeing how beta testing goes.
  • As a result, companies increasingly seek products to meet their needs with minimum customization, ease of install/upgrade and low maintenance.
  • said Bob Gilligan, general manager of GE Energy’s transmission and distribution business. “Our new approach will complement our current offerings and result in products that leverage Oracle’s spatial software."
  • GE Energy’s Smallworld Core Spatial Technology* remains a key technology for large utilities and telecommunications customers with complex solution needs. No existing products will be retired as a result of this new offering. The addition of applications based on Oracle Spatial will provide more options for companies with specific IT requirements.
    • This is an important statement that no existing products will be retired. I think it is very important that GE Energy repeat that statement until it becomes very clear to all of its current customers. I have frozen this statement in time as a PDF file. In my view this is a very import statement and I will do what I can to hold GE Energy to it.

What is not included in the announcement:
  • VMDS/Magik will be retired.
    • Actually, the press release states the current CST will continue to exist for customers with complex networks.
      • GE Energy’s Smallworld Core Spatial Technology* remains a key technology for large utilities and telecommunications customers with complex solution needs. No existing products will be retired as a result of this new offering.


I would be interested in your comments.

Cheers,

Alfred

13 comments:

Anonymous said...

Hi Alfred - great update.

In your blog you said:

"I am of the opinion that the current VMDS/Magik package provides better solutions for some of the Smallworld user's business requirements than Oracle/Java is capable of."

I'd be interested in your reasons behind this statement. Is this based on current limitations of Oracle in terms of long transactions?

Alfred Sawatzky said...

Hello Anonymous.

I have three reasons for making that statement.

1) emotional. clearly not a good reason for making a business case, but I think the VMDS solves some long transaction issues very cleverly and I have a gut feeling that it still works better than Oracle with respect to long transaction. Smallworld has had 10+ years experience supporting a long transaction solution whereas other vendors are still working at getting a good solution. (Have a look at Smallworld Technical Paper #9) The gut feeling is likely based on the next two points.

2) anecdotal. When I was still working for GE Energy last year, a colleague of mine was working on a project where he was working with Oracle Workspace Manager. He described to me how Workspace manager used tables to keep track of table differences between versions. Creating throw-away versions (ie., "what if" scenarios) was a suitable use for this, but when you needed to post any changes to the parent version, the posting and conflict resolution took longer to do than in Smallworld.

3) related experience. Last year I worked on a project for a customer that was using ESRI's ArcGIS product on top of Oracle and using Oracle's version management facilities. I'm not sure if the problem was with ArcGIS or the Oracle version management facilities, but that client only posted data to the parent alternative once per week because of the time and complexities involved wiht posting data up. We wanted to do something as "simple" as getting a difference stream between two versions to figure out what had changed. In Magik/VMDS we could have done it in 5 lines of code. In the ArcGIS/Oracle environment we had to do it with some complex SQL queries coupled together with custom code from a 3rd party ESRI vendor to accomplish the same thing.

Having said all this, I would be interested to hear from anyone that has more details about how Oracle handles its long transactions. Is it disk block based? Table based? Other?

Nothing beats anecdotal evidence than cold hard facts.

Pedro Miranda said...

Hi Alfred,

I did not understood well the issue here. I know that smallworld VMDS could be implemented in oracle instead of the normal ds files, is it a new kind of implementation based on the Oracle spatial?
I think that there are two different issues here, one which I agree with you is that the Smallworld VMDS is, up until now, the best GIS database for medium to large Smallworld implementations, we have used it in all our projects, and I think we will continue to use it for a long time.
The other is the Magik issue, very controversial among us, with those who love it, and other who don't.
I think Smallworld would benefit a lot with a change from magik to java or ruby, one of the major drawbacks in using smallworld is the cost if maintaining and changing source code, there a few resources with knowledge of Magik, the learning time is large and costly.
Although the implementation of the numerous classes in magik in another language would be tremendous, not to mention what would clients do with their millions of source code in magik if that change was made.

I would like to ear some opinion about the magik issue here.

Regards

Alfred Sawatzky said...

Hello Pedro,

The issue as it relates to VMDS/Oracle is that for the new product, GE Energy is proposing to use Oracle's own version management functionality (Workspace Manager) instead of the version management inherent in VMDS. The unknown is how does Oracle's Workspace Manager handle hundreds/thousands of versions compared to VMDS? How does Workspace Manager handling merging/posting versions compare to VMDS? My limited experience with Oracle Workspace Management (at a single ArcGIS implementation) is that VMDS handles version management in a simpler yet powerful manner.

For years, when customers asked Smallworld representatives why Smallworld was not on Oracle, the standard answer was that VMDS does version management better than Oracle. There was also always a little bit thrown in about how Smallworld was working together with Oracle to help Oracle develop their long transaction functionality. The sense of that statement was that Smallworld does version management better than Oracle. Now, apparently, Oracle does it just as well? What I would be interested in is some kind of benchmarking that shows how these two version management paradigms compare against each other.

There is some interesting data on Oracle's web site related to Workspace Manager. This includes a slide show. Slide 26 shows an example where 75 workspaces are in use. I'm wondering if that means that there are 75 alternatives active? Or if that means that at any time there are 75 writers each writing to their own alternative?

The Oracle Workspace Manager White Paper provides some great insight into how Workspace Manager works. There are alot of similarities with VMDS and maybe a few improvements (eg., audit history).

Anyways, that is what I see as the issue between VMDS and Workspace Manager.

My preference for Magik has been formed by using it for over 10 years. But I can see how it might be more expensive for a Smallworld customer to find a Magik resource compared to finding a Java resource simply based on the laws of supply and demand. I would be interested to hear from customers about how much more expensive it really is to have a Java resource compared to having a Magik resource.

Regards,

Alfred

Anonymous said...

Hi Alfred,

I have a few comments from the perspective of a small to medium sized utility:

We have one Magik developer and really need two. Within the constraints for hiring that we have we are not even going to try to get an experienced Magik programmer. At this point we are looking at getting one experienced in web technologies, particualarly Java, and train in Magik, if necessary.

I liked your idea about taking Magik/VMDS open source. The reality is that many of GE's competitors have more $$ to spend on development than GE has for the whole Smallworld product line. I think that this makes it tough to keep up. Taking Magik/VMDS open source could help with that problem.

As far as Oracle, one thing is for sure, a lot of the backdrop data needed to provide a reference for a utilities facilities data does not need to be version managed.

I would guess that one of the big issues for many smaller utiltiies is that other "GIS" products are already well entrenched ( I know that this is the case where I work). We are finding that the use of Oracle as the database provides us an easier path to work within those communities rather than being separate and isolated.

The reality is that Smallworld is at great risk to becoming irrelevant in the GIS market even though it has some superior technology because it does not fit well in a multi-vendor environment. It needs to do something and I think that Oracle and Java are a good choice. However, GE should support VMDS as long as existing customers want it.

Anonymous said...

None of the issues mentioned for a new Java/Oracle Spatial utility solution justify a new product to compete with Smallworld. GE could also develop a standard solution for small utilities with Smallworld. This would probably be easier and faster than doing it with Java and Oracle.

I believe the real reason for the new solution is that GE no longer believes in the future of Smallworld. I do not know many Magik programmers that expect to be doing Magik 10 years from now.

The problem with Smallworld is still that SWMFS is a closed database that is difficult to access with external applications and this is the gap that an Oracle installation fills.

As for the idea of making (at least part of) Smallworld open source; I agree that will add to the expected lifetime and market share of Smallworld. I take it that you mean making it available not only as open source but also as public licence.

Certainly if GE makes the SQL server bit open source (and free) there will be a lot to gain for GE. I have never used this part of Smallworld because customers consider it to be to expensive. We have customers that synchronize their Smallworld data to Oracle purely to make it available to external applications.

Spatial Intelligence is another product that could be made available as open source.

With regard to solutions on Oracle; I am also working on a project now where a client is migrating to ESRI ArcFM UT and from what I have experienced so far I must say that Smallworld is still the better solution. By the way the merge/post performance problem with ESRI on Oracle that you mentioned in your blog is indeed not an Oracle issue but related to ArcSDE.

There is also a client in Europe that chose Smallworld after a merger and thereby dumped a G/Technology-Oracle spatial installation.

Alfred Sawatzky said...

Hello Anonymous,

Thank you for your comments. I appreciate your suggestion to make something like the SQL Server bit open source. And yes, I did mean open source and public license. I will see if GE can give us some response to limited open source and public license for some of its components. Particularly, one would think that any component that eases access to VMDS would be a good candidate for a public license.

Regards,

Alfred

Anonymous said...

Hi Alfred,

I totally agreed with U on the idea of magik/VMDS open source. If GE can provide this facilities, many.. many.. students like me can learn and used smw gis & magik programming in their research and will graduates with at least some basic knowledge (if not expert) in magik and smw gis. At least it will increase the number of professionals with smw/magik skill. In my country, because of this constraint we are opt for a cheaper or other open systems. I just wondering why there are not many (or none?) Uni in North America & Europe have smw licenses and doing research using magik/smw compared to ESRI product? In my opinion, if smw still remain as a closed proprietary system, it will be soon (next 5 years?) be remembered as one of the good gis product... but not many (or nobody?) used it anymore....

Alfred Sawatzky said...

Hello Student.

Thank you for your comments. I agree with your concerns over the lack of Smallworld in educational institutions. I know that even before Smallworld was acquired by GE, Smallworld had tried to institute and university program with varying success. If you are comparing Smallworld to ESRI, I suspect that it is easier to teach and learn basic GIS concepts on an ESRI system than on an out-of-the-box Smallworld system.

In my 10 years of Smallworld consulting, I have often been asked by a customer: "can you make the Smallworld system do [fill in the blank]?" Invariably the answer was/is "Yes ... with a little bit of Magik". And that is still true... you can make Smallworld do pretty much anything that ESRI systems can do, but the fact is that often times someone needs to do somee coding first.

If I were an educator faced with a choice between a very powerful toolkit and a package that provides many out-of-the-box GIS functions, I would probably choose the out-of-the-box GIS package. If I, after 10 years of Smallworld work, would enter the education field and were faced with the same decision, I might actually include both Smallworld and ESRI in my offerings (as long as the Smallworld license fees were comparable to the ESRI license fees). (I don't plan to become an instructor because I know that field is much more difficult than anything I have done as a Smallworld consultant)

I think another contributor to the difference in university presence between Smallworld and ESRI is money.

When I read about ESRI and Jack Dangermond, I get the sense that Mr. Dangermond is trying to make the world a better place through GIS. Add that motivation together with the fact that ESRI is privately held and you get a company that has a strong push to get its GIS message out regardless of the cost. If you look at the http://www.esri.com/about_esri.html you will see that "ESRI designs and develops the world's leading geographic information system (GIS) technology. GIS is an important tool... one that helps shape the world around us."

I believe GE has a different challenge. They are trying to market Smallworld as a tool to design and manage complex networks. I believe that Smallworld does have some of the best technology for managing networks. They are not trying to solve many of the world's problems using GIS technology. I believe that Smallworld has all the supporting functionality to compete in other more traditional GIS spaces, but it seems that GE has decided to focus on one of the strengths of Smallworld... managing networks. And I believe they are taking this focussed approach because they need to answer to shareholders who are apparently demanding good value for their own hard-earned investment money.

So, if you are ESRI and do not answer to a vast class of shareholders, you can take a long view of things and give your software away below cost in order to further your cause. If youare GE, giving your software away rarely furthers your cause (ie., "making the numbers" for the next quarter).

As for universities using Smallworld for research... have a look at these links:
http://www.geography.hunter.cuny.edu/~abala/gis2/tigmod_paper.pdf
http://www.geo.hunter.cuny.edu/~carsi/INDEX/CURRENTAFFAIRS/in_update/updates.htm

Regards,

Alfred

Anonymous said...

I find this whole dialog quite intriguing. To be honest it never occurred to me that GE's OAI announcement would be taken as the death knell for VMDS/Magik technology.

Some things to consider are that any product, application or technology that is designed to fundamentally replace it's precursor must, at a minimum, provide the same level of functionality as its forbear. I think this is not the case with GE's Oracle technology. It appears to be a far less functional, less flexible and lower performing application.

Second, the majority of customers who use Smallworld technology today are unfortunately quite hooked on its most "archaic" features. These are it's light-speed retyping development language called Magik and its equally blazing fast database VMDS. It seems quite unreasonable that many people will trade these for the mediocrity of Java and the complexity of Oracle. In short, those who have it and understand it, love it.

I should think that it will be long time before GE's largest customers will drop the old technology in favor of another. Of course, this does depend on GE's willingness and ability to support two competing product lines. Obsolescence is a two-way street my friend.

Unknown said...

Here is some more info on the subject:
An interview with Robert Laudati, marketing program manager, electric software products for GE Energy:
GE Energy Embraces a Bigger World with Oracle Collaboration:
http://www10.giscafe.com/nbc/articles/view_weekly.php?articleid=372944&page_no=1

Anonymous said...

I like Oracle and Oracle Spatial for somethings, but here's my issue with SmallWorld: Lack of progress.

SWAF is merely ok in my books. Magik an irritating (but I have a bias towards strongly-typed languages). But what's worse is I haven't found SW to have made real changes in the things we need the most.

Mobile.. Magik is a VM like Java and .Net, yet GE has not spent a dime making Magik and SW more mobile friendly. Their current mobile offering is written using entirely different technologies (Java & XML). Is it really that hard to create a mobile Magik/SW??

Performance. Magik is still pretty slow, and I haven't noticed any changes in VMDS performance in the 7 years I've worked with SW. I think VMDS has the most awesome version management I've ever worked with (I've worked with Oracle's and ESRI's SDE and both are immature implementations compared to VMDS).

Interoperability. It's 2007 and we're still using ACP. I just have a really hard time understanding why SW can't use any of the newer inter-process communication protocols directly in core. Even a SOAP interface would be great. Instead, the only alternative to ACP is the Automation interfaces which has proven finicky for us, especially on Citrix.

SIAS. If one uses 4.11's JSF you're OK, but if you want to add SIAS features to an existing JSF application, you have to use the web interfaces, and there are no support classes for that. You have to write almost everything yourself. Not even their javascript is reusable. It would be nice to have a web interface that wasn't trying so hard for vendor lock-in.

Just my $0.02 worth.

sbm said...

Hello Alfread,

I am not sure whether this comment blog is still active as the last comment posted is on July 19, 2007. I hope it is active. I would appreciate if somebody is able to suggest whether the performance of Oracle (12G spatial) Workspace manager has improved over Smallworld VMDS version management functionality, in last couple of years. This is in connection with issues like long transaction, posting and conflict resolution simplicity etc. We are evaluating proposals for our new Enterprise GIS project (for medium size electric utility). Is it worth going on complete Oracle (i.e. without VMDS) with smallworld? How is the performance of “Smallworld on Oracle solution”?. There is also one proposal for using VMDS along with Oracle spatial. The suggested method is to use “ In-sync” which will enable us to synchronize the Smallworld VMDS data to Oracle purely to make it available to external enterprise applications.