Exporting Smallworld Data to KML

There was a recent question on the sw-gis Yahoo group asking about how to export a Smallworld trail to KML for use in other applications. It turns out that can be done easily using the open source Magik Components Library (mclib). The following video demonstrates how this is done. Important information:
- Magik Components Library link is here
- Thanks to Brad Sileo (iFactor Consulting) for contributing this code to the MCLIB project.



Give it a try and let me know what kind of cool export applications you are using. And if you have any improvements to make to the functionality, please feel free to contribute to MCLIB.

Smallworld Technical Paper No. 14 - GIS in the Cable Market

by John Rand MSCTE, Design Manager, Cambridge Cable Ltd, Cambridge U.K.

John Rand has considerable experience in the cable TV and Telecommunications industry, originally specialising in the Local Network with British Telecom. He joined Cambridge Cable on formation and was instrumental in the development of the integrated cable TV and telecommunications network design. Responsible for selecting and implementing GIS including advising on development. He studied Telecommunications at Southgate College, London and cable television at Atlanta Georgia. Currently studying Management at Anglia University, Cambridge, specialising in Operations and Project Management.

Abstract

This paper will look at the cable industry's requirements for a Geographical Information System (GIS) and the contribution a GIS will make to the cable TV telecommunications business. Analysis will be made of existing CAD systems and the GIS capabilities applicable to the unique UK cable industry and the reasons for changing systems. This industry is expanding at a rapid rate and I have illustrated how GIS can be used to meet the business plan goals. The areas of implementation, finance and marketing are also discussed.

The paper concludes that to reap the full business opportunities presented to this unique industry a GIS is the only credible system available for utilising the company's resources to their maximum benefit.

Introduction

GIS is only just starting to emerge as a useful tool within the cable industry. Its presence has been known for many years and although a few vendors have tried to develop a successful product, none until recently appear to have achieved this. However that scene now appears to be changing and we are at a point now where before us lies a seemingly endless vista of possibilities for the use of the system within the industry. Though the nature of the system is far-reaching in every aspect of our business, the implementation and development of our requirements are by no means straightforward and indeed are proving to be quite complex. This paper will look at how Cambridge Cable intends to use this "State of the art" technology through its benefits, to reach our vision of becoming the premier provider of entertainment, information and communications services for the benefit of the community and our customers, employees and shareholders and how this technology can benefit the whole industry. The presence of GIS will eventually be experienced in just about every department of the company providing the core information to drive the company forward. It will eventually be as commonplace as any other information system only more crucial.

The leading cable companies in the UK are beginning to implement GIS. This paper will look at the benefits of GIS to the cable industry, the newest of the utility companies.

The UK Cable Market

In order to appreciate fully the importance of the GIS industry within the cable market it is essential to understand the unique nature of the cable industry within the UK and the implications. A brief illustration of the industry follows.

UK Cable Market

Within the UK, 139 cable television franchise areas have been created by the Department of Trade and Industry (DTI). Franchise areas cover areas of dense population therefore only 70% of the UK is covered, however, expansion and creation of other franchise areas are possible. The first 12 franchises were awarded in the mid 80s and the remaining 127 franchises areas in the last five years. Cable television companies holding franchisee area licences can also apply for "Public Telecommunication Operator Licences" (PTO) for their areas, therefore creating a dual service industry, cable TV and telecommunications. The regulatory bodies for cable television and telecommunications are the Independent Television Committee (ITC) and Office for Telecommunications (OFTEL) respectively. The provision of two major service products by one company makes us unlike any other utility company.

[ Figure 1 not available ]

Cambridge Cable's Position within the UK Market

Cambridge Cable Limited (CCL) was formed in July 1988. It was awarded the Cambridge franchise in June 1990 and started constructing the network in June of 1991. The Anglia franchise was acquired in December 1992 thus making a total of approximately 200,000 homes covered by our operation. CCL is jointly owned by Comcast Communications of Philadelphia, USA and Singapore Telecom International.

How the Industry Works

The measure of the size of a franchise or company is how many homes fall within the franchise boundary; the penetration of our services into this number is one of the core statistics to watch. This represents expected income revenue with which to repay investment. The income from the two services is subtly different in that from cable TV it is a set flat monthly rate, depending on the chosen package, whereas telecommunications revenue is dependent upon usage. As owners of a cable TV franchise, there is no competition for broadband services, (British Telecom cannot operate cable television services on its network until 1997); however, British Telecom is our main competitor for "Local Loop" services. This is where the cable companies must use all their resources to succeed and gain the upper hand. The leading UK cable companies pride themselves in using "state of the art" technologies and practices to achieve this and thus the importance of using GIS becomes abundantly clear.

Economics of the Industry

The basic economics of the industry are similar to those of any other; finance is raised and used to construct an infrastructure network over which our services can be carried. Both services will be constructed as one network. The incremental costs for the second network are minimal as the greatest investment lies within the civil construction costs.

Cambridge Cable has four main strategic goals. With the assistance of GIS all these objectives can be achieved and maintained efficiently.

1. To create and develop profitable market opportunities. Through geographical market analysis, correct products and services can be determined and potential opportunities exploited.

2. To provide a wide range of differentiated quality services and products at competitive prices. Again GIS will be invaluable as a tool for market analysis.

3. To ensure the network is "future-proof", user-friendly and cost-effective. GIS will be used to simulate different architectural models and assess new technologies, giving us the required information to build an economical and reliable network.

4. To hire, develop, and retain the right people at the right time. The implementation of GIS as "state of the art" equipment demonstrates commitment by Cambridge Cable to new technology and to providing people with the right tools and information to develop careers.

[ Figure 2 not available ]

GIS will be a significant force in achieving these goals and contributing to the company's success.

The obstacle to this is that the existing situation relies on manual interaction between the different departments. Currently, design is drafted on a CAD system and from there on is printed and used in paper format. Other systems exist within the cable TV operation, Subscriber Management System, Network Management Systems and the Telecom Network Circuit Assignment Systems, however none of these interact leaving numerous opportunities for miscommunication and "information-error".

The supra-system of the business and the requirement for return on investment and instant current information on network and customers is causing stress on the sub-systems of:-

Subscriber Management System

Network Management System

Telecom Circuit Assignment System

plus various manual systems.This creates the need for a global system. The answer is the implementation of a GIS which can facilitate the interaction of all these sub-systems.

The combination of return on investment, instant access to current information on the network and customer information is essential; the supra-system is causing stress on all three of these sub-systems, creating the need for a global system. The answer lies in implementing GIS which can facilitate the interaction of these sub-systems.

The Cable Industry and GIS

Cable Requirements of a GIS

The process of obtaining customers to bring in revenue begins with constructing the network, therefore, design is required. The principal requirement is for a system able to produce comprehensive designs, information on the areas already constructed and on the network status which will be readily available to those requiring it at any time. An ideal example of how a complete system would work is as follows:

1. Survey information would be collated from the field on a portable PC and input directly on to the digital Ordnance Survey map. This information would then be downloaded into the GIS and the design created thereon.

2. On completion of the design, customer addresses would be transferred to the Subscriber Management System and automatically "populated". Telecom assignment information would also be transferred to the Circuit Assignment System and purchasing would also automatically receive Bill Of Information (BOM) information.

3. The GIS information would be linked to the cable television and Telecommunication Network Management Systems. Should there be a network performance problem or outage, instant geographical information/reports can be generated alongside instantaneous information for customer services.

4. Black spot analysis for maintenance purposes becomes effortless and marketing can identify meaningful information.

[ Figure 3 not available ]

History of GIS in the Cable Industry

As mentioned earlier GIS has never quite found its feet within the cable industry. The majority of cable companies have either used pen and paper or CAD systems which are extremely stylised (Newell and Sancha 1990). A few GIS vendors have tried to develop GIS systems for the cable market. Cambridge Cable purchased a CAD system early 1991 and had been using it as a successful tool until recently. Many functions available on GIS were not available on CAD and this was compounded by the lack of support given to the product. The CAD system imposes severe limitations on effective use within this rapidly growing industry, Newell and Sancha (1990) commented "Several of the established CAD vendors tried to adapt their CAD systems for GIS applications. This resulted in most unsatisfactory compromises"; "CAD vendors continued to try to convince the industry that they had a viable product by integrating their databases with the CAD function" and item referred to "marrying together two inadequate systems" (Newell and Sancha 1990). The two technologies of database and CAD do not integrate easily. In late 1992 the industry started to talk more about GIS. No single GIS system proved to be totally reliable and no single vendor stood out. Cambridge Cable were approached by Smallworld - a company well established in GIS and based locally in Cambridge - to work with Smallworld to develop the combined cable TV and telecommunications model. Also to establish an unrivalled product for the industry. Fundamental requirements included; data capture, performance, customisation and integration (Newell and Theriault 1989). These aspects were severely limited or non-existent with Cambridge Cable's current system.

[ Figure 4 not available ]

Implementation of GIS

It is clear now that no improvements to the CAD system could have provided our business with its requirements. The GIS is now installed within the design department and is already proving beneficial through its ability to calculate system performance thus helping to obviate the need to use design contractors.

The next stage of implementation is to integrate the Network Management Systems and Subscriber Management System etc.

The final and more idealistic stage of implementation is that to integrate GIS throughout the company to provide full, up to date network information to everyone. "As the number of users sharing information in this way increases, the system will constitute a continually improving Geographic Information System for the benefit of all" (Bernhardsen and Tveitdal 1986). This will also improve the "work conditions for the specific personnel groups" Kubik, Merchant et al 1987) in that having relevant current information immediately at hand will be of enormous benefit for optimum performance.

Why Invest in GIS

Should a cable company invest in GIS? Considering the magnitude of the initial investment, should they stay with their manual paper or CAD methods? When also is the best time to invest? The analogy I put forward is that of comparing GIS to that of computing in the 60s. The first generation have high purchase costs, high maintenance and few benefits but as things have progressed no business would be without one. GIS is now sufficiently developed to be useful to the cable industry. Moving eventually, like computing in the 60s, no cable operation will exist without a GIS. With networks growing at a tremendous rate any wise company would invest in GIS. The thought of transferring enormous amounts of data at an advanced stage of the build is horrifying and expensive.

The investment in a GIS system as a proportion of the total investment in constructing the total network is only a fraction of the costs. Considering this will be controlling the network assets and providing the benefits described later, it can be seen as an essential long-term investment.

What are the Benefits to the Cable Business

• Improve the quality of network design by enforcing engineering rules and standards which can be preset and fine tuned.
• Facilitate the achievement of cost reduction and quality improvements in passing addresses to the Subscriber Management System with implementation of an automated interface.
• Improve repairs to the network through the provision of visual aids on fault investigation.
• Simulation of different architectural network models to evaluate the most cost-effective solution to design scenarios, e.g. "Fibre to the Feeder" architecture versus "Fibre to the Kerb".
• A GIS is a very useful tool in evaluating new technologies and their impact on existing networks, e.g., PDH Versus SDH
• Analysis of financial comparisons of percentage turnover ploughed in against speculation.
• Having advanced equipment attracts the right calibre of staff and enables them to advance their careers with current technology.
• Attraction of investment within the company by being seen as innovative and conscious of the need to have accessibility to vital information.
• Accurate inventory of assets and asset management for capital accounts. Also analysis of potential acquisitions including identification of existing or potential plant within those areas.
• Interactive queries for precise retrieval of information concerning the network.
• Quicker response times to customer orders, due to readily available information, especially regarding telecom enquiries and indication of likely installation dates.
• Substantial savings can also be made through integrating GIS with purchasing, warehousing, and the construction programme. Bill of Materials (BOM) created by GIS can be transferred to the purchasing computer system where they can be ordered on minimum lead time in relation to the construction schedule and received in the warehouse for "just in time" materials management.
• Geographic survey information captured on GIS can be sold commercially to any other parties interested in such data.
• It can also query information without the need to survey.

Economic Benefits

Many of the above points can represent very tangible cost savings and through collation of data this can be proved. However with GIS there are considerable intangible cost benefit savings that only GIS can give, as opposed to improving existing systems. An example of this is that, after the initial investment in GIS, savings in staff can be made without the usual element of human error.

Intangible benefits:-

More information (marketing, customer service, fault locates etc) Better analysis with less labour time (marketing, new technologies) Ability to do analysis not possible before (new RF and telecom technologies) Better decisions (build areas, new technologies) Better planning (network design, business plan) Better understanding and analysis of highly complicated systems

[ Figure 5 not available ]

Return on Investment

Deciding where to build currently concentrates on areas of highest density. This is desirable because eventually we want to provide service to every home in the franchise area. The denser areas are typically those with the best demographics although, currently, no marketing analysis is done to determine if any of these areas are better than others. Through using GIS as a sophisticated marketing tool and analysing areas, the best potential dense areas can be built first. Early high penetration will be achieved and high revenue will be received, yielding a high return on investment.

Cable Marketing and GIS

Marketing

It is essential to hit the right potential customer base with the right services. Traditionally lower socio-economic groups are better target groups for cable TV while the higher groups are more likely to be interested in telecommunication services. However it is being found now that the types of socio-economic groups mean slightly less than customer "Lifestyles" which concentrate on the use of disposable income. If this is now to be used, data set analysis can be done prior to design in order to identify the correct market and concentrate in building in that area first. Without GIS the process would be a very lengthy and laborious task.

Since telecommunications is regarded as an essential service, churn is not experienced to the same extent as with cable TV customers. If market identification and "right sizing" can be advised prior to a sale, enormous savings can be made in the areas of abortive sales calls, installation and equipment retrieval. Sophisticated marketing analysis can be done on the remaining potential customer base to determine the required product.

Conclusion

Clear benefits can be seen in implementing a GIS system within a cable business and the advantages are clearly defined. There is also confidence within the industry that there are credible vendors with a tremendously useful product of enormous value to a company's operation.

The near future for GIS looks exiting and in the long term there will be far reaching effects on our business. It is an essential tool in effective competition. A culture change in the working environment will be required to make acceptable this prolification of invaluable information. Precise marketing is that key and, by using GIS to interact and analyse all available information, cable companies will be able to achieve greater success within their market.

References

Bernhardsen,T and Tveitdal,S. 1986. Community Benefit of Digital Spacial Information. VIAK A/S - Auto Carto London, Vol.2.

Dickinson,H.J. and Calkins,H.W., 1988. The Economic Evaluation of Implementing a GIS. International Journal of Geographical Information Systems, Vol.2, No.4, pp307-327.

Joint Nordic Project, 1987, Digital Map Data Bases, Economics and User Experiences in North America (Helsinki, Finland: Publications Division of the National Board of Survey, Finland).

Kubik,K., Merchant,D. and Schenk,A. 1987. Design Considerations for Urban Information Systems. A-ASPRS- ACSM, Vol.5.

Marble,D.F. and Peuquet,D.J., 1983. Geographic Information Systems and Remote Sensing. Manual of Remote sensing, 2nd ed, American Society of Photogrammetry, Vol. 1.

Newell,R.G. and Sancha,T.L., April 1990. The Difference Between CAD and GIS. Computer Aided Design magazine.

Newell,R.G. and Theriault,D.G. September 1989. Ten Difficult Problems in Building a GIS. Presented at British Cartographic Society Symposium, Cambridge.

Theriault,D.G. April 1989. An overview of Geographical Information - the technology and its users. Presented at conference, Geographic Information Systems.

Acknowledgements

David Theriault, Smallworld Systems Ltd. Keith New, Cambridge Cable Ltd.

Glossary CHURN - Turnover of customers, disconnections after connection. OUTAGE - Complete loss of service. PDH - Presynchronous Digital Hierarchy. SDH - Synchronous Digital Hierarchy.

Smallworld Technical Paper No. 13 - The Wide Area Connection

by John Rowland, Grampian Regional Council

Abstract

GIS data is voluminous, demanding upon bandwidth and therefore normally requires high speed network links. This has served to constrain "real time" wide area distribution of GIS data. In conjunction with British Telecom, Gandalf Digital Communications Ltd and Smallworld Systems Ltd, Grampian Regional Council believes it has been able to implement a realistic solution to this problem using Smallworld's recently developed "Persistent Cache" functionality running over British Telecom "Kilostream" links.

This paper:

• briefly explains Grampian Regional Council's requirement for wide area GIS;
• overviews wide area communication options;
• explains the basic concept of intelligent bridging;
• describes the key features of the Smallworld System which have been used to implement wide area connections;
• reviews experience to date;
• briefly considers what the future may hold.

Grampian Regional Council & its Corporate GIS

Grampian Regional Council administers a land area of approximately 8,000km² which is home to a population of 530,000, half of whom live in the City of Aberdeen. As with other Scottish Regional Councils, its responsibilities include the provision of water, drainage, roads, economic development, strategic planning, fire, police, education and social services.

The Council's main headquarters is Woodhill House in Aberdeen, some departments also operate from a number of divisional and other offices located throughout the Region.

In 1992 the Council commenced implementation of a Corporate GIS which was installed in Woodhill House for use by four departments (Economic Development and Planning, Property, Roads and Water Services). The Council selected Smallworld GIS running under UNIX as its core system. At present all departments share a single corporate GIS database which is managed by a Sun MP630 file server. With ongoing data capture this database continues to increase in size; at the time of writing it held 4GB (Giga bytes) of GIS data.

Responsibility for maintaining this database and ongoing implementation of the system on behalf of user departments is vested in a six person team called the "GIS Unit". To date the Council has acquired a total of twenty nine GIS "seats" from Smallworld with more on the way. Ten of these seats have recently been acquired by the Department of Water Services for installation at six different office locations remote from Woodhill House (see figure 1).

Until recently it had not been viable for the Council to operate their Corporate GIS over a wide area network. However, Smallworld's recently developed Persistent Cache database management software combined with "state of the art" network bridge technology has enabled the Council to implement wide area connections using leased British Telecom Kilostream lines. At the time of writing two of the Department of Water Services' remote offices have been connected to the main file server in Woodhill House.

[ Figure 1 not available ]

Wide Area Connection Components

The wide area connection has four key components: a physical communication link (British Telecom 64Kbps Kilostream in the first instance), intelligent bridging (Gandalf LANLine), Smallworld version managed GIS database and Smallworld Persistent Cache software (2).

Physical communication links

A 500m x 500m Ordnance Survey vector tile of an urban area typically contains around 250Kbytes of uncompressed data. In order to pass such a tile over a network and display it in a total elapsed time of less than 45 seconds the network has to pass data at a speed of in excess of 44Kbps (Kilo bits per second). In order to view 1km² of similar data in the same time the speed would have to increase to in excess of 180Kbps.

This should not be a problem over a local area networks with a bandwidth of10Mbps (Mega bits per second). However, if all that there is between office locations is a public telephone network, a couple of high speed modems operating at 14.4Kbps and the inherent "dial up" delay of analogue communications, then there clearly is a problem.

There is no alternative but to seek a digital communications link . Depending upon what you are prepared to pay, digital links can provide effective line speeds of 64Kbps up to in excess of 8Mbps with minimal "dial up" delay. They can either be ISDN ("pay when you use") dial up links or dedicated "Kilostream" or "Megastream" leased lines.

ISDN

ISDN In United Kingdom ISDN (Integrated Services Digital Network) is available either as ISDN2 providing an effective 128Kbps line speed using two 64Kbps channels or ISDN30 providing an effective 1.92Mbps using thirty 64Kbps channels. At the time of writing British Telecom ISDN2 socket installations were being charged at approx £400 per site, line rent at £84 per quarter and transmission at normal telephone call rate charges.

Leased Lines

Leased lines normally incur an initial installation charge and a subsequent annual rental charge which varies according to distance from the nearest digital exchange. At the time of writing British Telecom were charging £900 per site to install 64Kbps "Kilostream". The annual line rent of a link between two sites varies according to distance and proximity to BT exchanges, some indicative figures are quoted in the ISDN2 v Kilostream comparison below.

In contrast 2Mbps "Megastream2" currently costs £6,200 per site plus £750 per link for a first installation and 8Mbps "Megastream8" £9,734 per site plus £2,625 per link. Line rents vary according to distance between BT exchanges, for example if two exchanges were 50km apart, Megastream2 would currently cost £15,740 per annum to rent and Megastream8 £55,108 per annum. Even the most optimistic GIS cost benefit analysis may have difficulty in justifying expenditure of this magnitude!

Despite current talk of information super highways it is of little surprise that many multi site GIS installations are still reliant upon using tapes, discs and couriers to transfer data between individual sites.

The wide area connections to Grampian Regional Council's six Water Service remote offices are being implemented using a single 64Kbps Kilostream channel to each office.

[ Figure 2 not available ]

Intelligent Bridging

Bridge or gateway devices are needed to connect the physical wide area communication link between two remote sites to the local area networks (LANs) at those sites.

A bridge is effectively a filter which joins two network segments such that data will only pass through the bridge to a second segment if it is destined for a device connected to it. Bridges are commonly used to segment local area ethernets so that unwanted data packets are not allowed to flow along segments where they are not needed.

In a UNIX environment bridging is achieved using the IP (Internet Protocol) part of the TCP/IP protocol (1). Every device connected to an ethernet has its own unique IP address. A data packet being transmitted from one device to another always carries with it the IP address of the device to which it is being sent. In the case of a data packet which is broadcast to all devices on a network the IP address is coded so as to indicate that it needs to be delivered to every device.

Gateways are special devices for transferring data between two different networks which adhere to different network protocols. As such they actually have to restructure the data packets which pass through them and are therefore inherently slower than bridges.

Wide area physical communication links between sites are nearly always slower than the local area networks they connect together, hence bridge or gateway devices are needed to prevent unwanted local area traffic from escaping to and causing congestion on the physical wide area link. Bridges supplied by Gandalf and other vendors for this purpose incorporate a number of intelligent features to enhance their performance.

Data Compression

Data Compression algorithms are used to compress transferred data, so as to achieve actual throughput which exceeds the quoted bandwidth of the physical wide area communication link. The degree of compression depending upon the extent that data is already compressed. For example tests at Grampian Regional Council indicate that their Gandalf "LANLine" bridges operating over 64Kbps Kilostream are able to compress raw NTF files by ratios in excess of 3:1 and already compressed TIFF files by ratios of around 2:1, thus achieving effective throughput of data in excess of 192Kbps for raw NTF and 128Kbps for TIFF. Even higher compression ratios of up to 8:1 can be achieved with these devices.

[ Figure 3 not available ]

Transparent Automatic Dial Up

Transparent Automatic Dial Up Bridges built specifically for connecting local area networks to "dial up" links such as ISDN embody an "automatic dial up facility whereby (for UNIX networking) the bridge is configured with a table which maps different network IP addresses to the phone numbers to which they are connected. Thus packets emanating from a "departure" site will cause their interconnecting bridge to automatically dial up the phone number of the "destination" site.

ISDN bridges will normally also have a configurable "time out" connection period which specifies how long an ISDN connection should remain connected for after a packet has been transmitted. For example if the time out were set to 30 seconds then the connection will close every time there is a break of 30 seconds between transmitted packets. Given that ISDN connection dial up can be made in as little as 5 seconds it is quite feasible to make several very short connections during the course of the working day and only incur a relatively small phone bill.

Automatic dial up and subsequent timed out disconnection is totally transparent to the user thus the ISDN bridge provides a virtual permanent connection.

Bandwidth On Demand

Bandwidth On Demand ISDN2 incorporates two individual 64Kbps channels which can either be used in parallel to achieve an effective 128Kbps bandwidth (with compression actual throughput will be even faster), or separately to send data to two different destinations at the same time. Similarly Kilostream can be installed in multiples of 64Kbps channels and used in much the same way.

"Bandwidth on demand" characteristics of local to wide area bridges enable individual ISDN and Kilostream channels to be automatically opened and closed to different destinations according to actual traffic volumes. With the Gandalf "LANLine" bridges it is also possible to mix and match Kilostream and ISDN together such that an ISDN connection can be opened when a single permanent Kilostream channel becomes overloaded.

Virtual Extended Local Area Networks

The net effect of state of the art intelligent bridging used in conjunction with digital wide area communication links such as ISDN and Kilostream is to create a virtual extended local area network. In a UNIX environment client workstations located at one site can access server devices at another site several kilometres away as if both devices were connected to the same local area network. Albeit with degraded performance if the volume of data being transferred between sites exceeds the available bandwidth of the physical wide area link.

Not only does this permit remote offices to access main office data, but also to output data to peripheral devices, such as expensive large format electrostatic plotters, located in the main office.

Database Version Management

In order to understand how Persistent Cache is being used to provide Grampian Regional Council's "wide area connection" it is first necessary to provide a brief explanation of their implemen-tation of Smallworld's version managed database.

Smallworld Version Management permits several versions of the database to exist simultaneously. In Grampian's case these versions are organised hierarchically as illustrated by figure 2. There is a single definitive top alternative" which is normally never written to directly. Each department is then provided with its own version of the "top alternative" which again are normally never written to directly, instead all users who are required to write to the database are each provided with their own "personal writable alternative".

For routine data capture work users are usually asked to update their departmental alternative on a daily basis by "posting up" their own personal alternative to it. This has to be preceded by a "merge down" of all changes which have already been posted to their departmental alternative. Once all personal versions have been "merged and posted" a departmental administrator then ensures that their own department's alternative is "merged and posted" to the "top" definitive alternative. Thereby inheriting changes and updates made by other departments.

Grampian Regional Council's GIS Unit is responsible for maintaining the Ordnance Survey map base and other shared corporate datasets such as a number of different gazetteers. Within the alternative structure the GIS Unit is treated as another department thus departments, and in turn end users, have their map base maintained for them by virtue of the "post and merge" procedures.

Within the UNIX file system the GIS database is held in a set of files storing different types of data (eg geometrical points, lines, areas, associated attributes etc). Database alternatives can be created so as to either be located totally within a file set held in a single directory or, created so as to reside in a separate sub directory with the same file structure. Thus the UNIX file system can if desired be configured so as to totally or partially mirror the database alternative structure (figure 3). This in turn implies that different alternative versions of the database can be stored on different storage devices on the same network.

Persistent Cache

Smallworld's Persistent Cache software (2) enables all or a subset of a GIS database to be cached to a local disc attached to a client workstation which is in turn configured to be a local cache server to both itself and other clients. By maintaining a copy of frequently accessed data in the local cache, it is an elegant and transparent way of providing large systems with high performance over low speed communication links.

In figure 4, workstation A is a local cache server located at a remote site along with client workstation B. GIS read transactions generated by workstations A and B look first to the local cache to retrieve data. If the requested data has not been cached it is retrieved from the main file server via the wide area connection and then cached.

The local cache has a configurable operating capacity, once this capacity has been filled old cached data is deleted from the cache on a "least recently used" basis. The cache capacity can be set to be large or small depending upon the size of the required database subset. If need be (local disc space permitting) it could be set to be large enough to replicate the original database.

When using Persistent Cache, remote site users are able to retrieve cached data very quickly and uncached data at the speed of the wide area connection. Hence if a subset of the main database is cached there will be occasions when read transactions may suddenly appear to slow down as data is retrieved over the wide area connection.

Write transactions write directly to the user's alternative every time a database record is inserted, updated or deleted and then subsequently copied back to the local cache.

At appropriate periods of time, remote site users initiate merging and posting of their changed data with higher order alternative versions. The merge and post processes are run on whichever machine the various alternatives are held. The local cache being updated where new "merged down" change data is located in a geographical area that is already held in cache.

By virtue of the ability of being able to map alternative versions of the database onto different UNIX directories (see figures 2 and 3) user's alternatives can either be held on the main server back at headquarters or somewhere locally at the remote site. This provides organisations with a high degree of flexibility as to how they operate over wide area connections.

Holding Remote Site Alternatives on Main Server

If users' alternatives are located at headquarters then all write data is passed over the wide area connection whenever a database record is inserted or updated. In a data capture environment this implies that relatively small amounts of data are passed frequently over the wide area connection.

Database commits and alternative version posting are processed back on the main server and therefore no data is passed over the wide area connection. Similarly the merge process (merging down of changed data from higher order alternatives) is also undertaken back on the main server, however the amount of changed data passed back across the wide area connection will depend upon the volume of merged down changed data which maps onto currently cached "geography". By holding all remote site alternative change data on the main server the remote site users do not need to be concerned with data backup and other routine system administration tasks which can all be undertaken back at headquarters.

[ Figure (diagram) not available ]

Holding Remote Site Alternatives Locally

By holding user's alternative change data locally no write data is passed over the wide area connection until the locally held alternative versions are merged and posted with and to higher order versions located back on the main file server. If daily posting and merging is undertaken then this implies a daily transfer of a larger volume of change data over the wide area connection.

The volume of changed data merged back down to the locally held alternatives is entirely dependent upon the amount of data which has been recently posted to the top (definitive) version of the database by other users. This could be considerable if say a new batch of Ordnance Survey maps had been recently loaded.

Populating the Local Cache

The local cache is essentially an extended reflection of the data which a local client work-station holds in memory. It is therefore composed of a subset of object class layers for "blocks" of geographical extent. For example Grampian's Water Service divisional offices cache background map and water supply object class layers for all or part of their divisional areas of operation.

Upon initial creation the local cache is "empty" and must be populated. Users can be left to do this during the course of natural usage, upon first access all data is "hauled" over the wide area connection and then cached. This could be a little tedious if two or more users at the local site are simultaneously hauling data over a 64Kbps line. They could therefore instead organise to "zoom out" to a large extent of geography as they leave for home so that the area in which they wish to work the following day has been cached upon return to work the following morning.

Alternatively initial cache data can be written to tape by staff back at headquarters and then copied into the local cache in order to "kick start" it.

Grampian Regional Council's Wide Area Connection

Kilostream v. ISDN2

Although the Council already had some operational wide area communication links it was decided that the Corporate GIS would have its own dedicated links because of difficulties in extending heavily subscribed existing facilities to sites where GIS was required. The lowest cost option able to provide acceptable performance was therefore sought. This turned out to be a choice between ISDN2 and single channel Kilostream. Capital installation costs were very similar for both (approx £2,500 per site) however, in the case of ISDN2 ongoing running costs varied considerably according to degree of use.

For total daily connection times of less than about four hours per working day ISDN2 is cheaper to operate than fixed fee Kilostream as illustrated below for a notional 247 working days per year at current British Telecom day rate call charges:

[ Figure (cost notes) not available ]

The above costings indicate that the most cost effective option is dependent upon the nature of GIS use at the remote site. If there is a low level of write transaction at a site where a significant proportion of the database is held on the local cache then ISDN2 provides a very flexible and potentially inexpensive wide area link. However, if there is a high level of regular write transaction or considerable regular "hauling" of uncached data throughout the working day then Kilostream is going to be the more viable.

Because it was known that the first two Water Service offices to be connected were "heavy" GIS users (they had been previously using GIS in a standalone capacity) and there still appeared to be technical problems handling broadcast messages over ISDN it was decided to adopt Kilostream for the first wide area connections.

Experience to date

Initial use indicates that the successful operation of the wide area links is more dependent upon operational management than technical factors. The two remote sites connected to date comprise of two locally networked GIS workstations currently used for data capture work. By its very nature data capture work does not involve frequent extended panning across the map base, hence "hauling" of uncached data has not been a problem with a relatively large capacity cache which was pre-populated prior to installation.

Data transfer across the wide area connection performs rather like a motorway contraflow, in so much that if there is very little traffic on the motorway then, ignoring speed limits, traffic flow is virtually as quick as if there were no contra flow. However as the volume of traffic increases the actual throughput speed decreases in almost exponential proportion.

1km² of inner city water data takes only slightly longer to display when retrieved over the wide area connection as when retrieved straight from cache. However 1km² of inner city water data plus all Landline OS data takes significantly longer to display.

Grampian's two Water Service offices have been configured so that local user's alternative versions are stored back on the main server, consequently data is passed over the Kilostream every time a record is inserted or updated. Users have noticed a degradation of write transaction time when they both write simultaneously. The degree of degradation is acceptable but does indicate that sites with a number of writing users may need to either store their alternative versions locally or be provided with access to additional communication channels over the wide area link.

The conclusion to date is that the nature of GIS usage needs to be understood in order to specify and configure a wide area connection for optimum performance.

[ Figure 4 not available ]

What Of The Future

Grampian Regional Council believes that it has been able to implement wide area networked GIS at realistic cost using technology which is available today. It has been proven that a single channel Kilostream link operating at 64Kbps is adequate for the scale of present implementation. Furthermore this has been achieved with a great deal of "behind the scenes" activity which is totally transparent to the user.

The computer press makes great play of cheap high speed local and wide area ATM (Asynchronous Transfer Mode) networks being the way of the future (3), however the technology is not yet available and until it is, it is difficult to see how GIS data can be viably transferred between different systems in anything like real time.

In the longer term the Council is keen to reduce the cost of providing wide area connections to more marginal GIS users by using ISDN2 instead of Kilostream. It is also keen to exploit the potential for transfer of data between different organisations using ISDN. The cost of operating ISDN2 between locations over 35 miles apart is the same no matter whether they are 36 or 500 miles apart. Unlike "fixed" Kilostream links, ISDN connections can be made between any two locations which can dial to one another.

Persistent Cache has also been seen as a way of relieving congestion on heavily used local area networks. The Council is currently planning a 6 seat GIS sub network in its headquarters which will use Persistent Cache to reduce the volume of GIS data over the building's main backbone LAN.

Acknowledgements

The authors wish to thank British Telecom, Gandalf Digital Communications Limited, Grampian Regional Council and Smallworld Systems Limited for their support and assistance in compiling this paper. Particular thanks go to Alistair Reid, Andrew Swanson and George Wallace of Grampian Regional Council for their part in installing wide area connection components and Andrew Reid of Gandalf for his enthusiastic support, also to the staff of the Department of Water Services for acting as "test drivers".

References

1. SOUTHERTON A. Modern UNIX, Chapter 4, Wiley 1992.

2. NEWELL R.G. BATTY P.M. GIS databases are different. Proceedings of the AGI 93 Conference Part 3.

3. UNIX NEWS No 56 October 1993, ATM is the wave of the future p63-65.

Copyright © 1996 Smallworld Systems, Inc. All rights reserved