This is the nature of human endeavor. The world used to be flat. The earth used to be the center of the universe.

Discovery and development is an iterative process. What we do today will likely be replaced tomorrow. Just because we can’t be perfect at the outset doesn’t mean we shouldn’t try.

Just because our data doesn’t represent the “truth” doesn’t mean we shouldn’t use it. We take our best shot at it and refine it as we better understand the problem. Otherwise we sit in an empty room holding a blank slate…

        

‘…the geospatial industry almost completely left behind support for UNIX-like OSes’

It is true that the proprietary GIS vendors have largely abandoned Unix and Unix-like operating systems and continue to do so. However the open source GIS community has embraced all major operating systems with software that runs on Linux, *BSD, Mac OS X, and yes, Windows. This support includes everything from the desktop to the server.

For those of us that prefer to operate in a Un*x environment, that’s good news.

        

Update: It’s now shipping. See the announcement.

        

For example, here is a simple shell script to convert shapefiles in a directory from their current (in this case projected) coordinate system to WGS 84:

#!/bin/sh
for shp in *.shp; do
ogr2ogr -t_srs EPSG:4326 wgs84_shapes $shp
done


This simply takes every shapefile in the current directory and converts it to WGS 84 and places it in the wgs84_shapes subdirectory. If you don’t know what an EPSG:4326 is or need help with projections and coordinate systems, check out the Spatial Reference website.

Total runtime for converting 91 shapefiles was about 47 seconds—way faster than you could do it pointing and clicking.

If you are a Linux or a Mac user, you can take that little shell script and use it directly. If you are a Windows user, you can use the batch language to write a similar script, or better yet install MSYS or Cygwin. With MSYS or Cygwin you can run the script above as-is on Windows.

The quickest way to get the GDAL/OGR utilities for Linux and Windows is to install FWTools. For Mac, download the GDAL framework and required dependencies from Kyng Chaos.

Don’t get me wrong—using a GUI is great, however when you have huge quantities of data to convert there is no better way than the command line.

        

It’s an interesting read with some good points until I got to this:

Our current favorite example of this approach is Ruby-on-Rails. This system is the little language, Ruby, extended with integrated support for database access and manipulation through the “model-view-controller” programming pattern. Ruby-on-Rails compiles into standard JDBC, but hides all the complexity of that interface.

Rails compiles to JDBC? Now I’m questioning the the whole paper…

        

Part of this trend stems from a desire to deliver mapping to the masses. If you’re a consultant or service provider, it’s a good thing. Being able to provide your “customers” with a toolset over the web makes sense.

I don’t think anybody in the know seriously thinks that GIS on the desktop is on the verge of extinction or irrelevance. For each and every slick GIS application on the web, there is a backend somewhere. And that backend uses data that came from somewhere. I would guess that in most cases, it was touched by something on the desktop.

Data creation and analysis is in large part the domain of the desktop. Sure there are automated acquisition methods untouched by human hands. But for a lot of projects, you simply have to create the data yourself, whether from scratch or by manipulating an existing data set.

Your view of desktop GIS depends on your mapping culture. Those of you coming from the “neo” side of things may see little use for it. If you are a data creator or analyst, you need something more. Will the web someday provide those tools? Perhaps. One example is the PyWPS project which is already providing a web interface to several “pure” GIS analysis tools. Of course these run against the data on the backend, not your data. Efforts in this arena will ultimately provide a tool set that can be used to create a web interface to your data as part of a custom solution.

In the Open Source realm, we have an impressive set of tools such as GRASS, OSSIM, Quantum GIS, uDig, GDAL/OGR, PostGIS. Working together these applications can satisfy the majority, if not all, of your GIS requirements.

Does desktop GIS still have wheels? I think so…

        

He (or she?) seems to have disappeared off the face of the blogosphere. After stirring the pot with several posts (see example), he is gone.

I wonder, did the lawyers from some unnamed giant take him down? Or did he say the wrong thing about the wrong person/group/software/topic and disappear on a lonely road late at night?

Perhaps he retired and I missed the announcement. In any event it all seems mysterious and to my suspicious mind, perhaps even sinister…

        

Brittle

Solid, but liable to break or shatter

In other words, we can design solid systems that serve us well, but they can be brittle. We can push on them a bit and they perform well, but push too hard and the whole thing shatters.

Lets take a look at an example of a potentially brittle system. This system is designed to serve GIS data over the web and is fairly typical. The components of this system might be:

• Database server
• Database software
• Web server
• Web server software
• GIS middleware
• Application framework software
• Map server software
• WMS data sources from one or more Internet servers

This system should be fairly stable, so where are the potential failures? Ever had someone move, rename or worse yet delete a layer or data source? How about someone changing the password to your GIS data stored in your spatial database?

These are certainly things that contribute to brittle systems. Perhaps the biggest problem (especially in larger organizations) is the dreaded Dependency Dragon. Components are built to work together but they have version dependencies. You can’t run a modern word processor on your operating system from 1986. Likewise, you have to match your server and GIS software to the appropriate versions. This is overstating the obvious, however it is one of the major problems in maintaining a stable non-brittle system.

If we never had to upgrade anything, we would be in pretty good shape, assuming the power stays on and we can afford to pay the ISP. Then a critical security patch comes along for one of the system components and the dreaded upgrade cycle begins. This can be mitigated if we have complete control over all the servers and software that make up our system. If not, we may find that an operating system upgrade over the weekend has reduced our shiny mapping application to brittle pieces scattered over the digital landscape.

Much has been written about systems reliability and I don’t express even a hint of expertise in this area. What follows are observations that may help to reduce the “brittleness” of a GIS system. This applies to both web mapping and desktop/enterprise systems. To make a system more stable and less brittle, we must:

Minimize the number of potential failure points

This can be accomplished by consolidating services where possible and simplifying the ways in which data are generated, updated, and maintained.

Manage change from a holistic perspective

Systems are just that. They interoperate and therefore must be managed as a whole. This can help avoid disconnects and changes that contribute to failure. To do this, we must involve the system administrators, database administrators, application developers, and data administrators. Often changes that break a system are made by a person that has no clue about the consequences of a seemingly minor change.

Plan for change

Don’t let it just happen to you. Plan and coordinate changes to operating systems, databases, data, and applications. This of course requires communication with all involved.

Document the Dependencies

Creating dependency diagrams and/or documents and using them as a reference point for change can help you avoid the dragon and eliminate failures related to upgrades, enhancements, and security patches. By identifying the components in a system and documenting how they interact we can prevent mishaps.

No system can be 100% reliable, but brittle systems are like a millstone around your neck. They are time consuming, costly, and can have direct economic impact if not brought under control.

        

Well maybe its not that bad but organizing and managing your GIS data is difficult. If you need to maintain canonical datasets, the spread of “temporary” and/or “working” copies is your enemy. The problem seems to be directly related to the size of the organization. It becomes particularly grievous when dealing with enterprise data in a large organization.

Even when controls or policies are in place that mandate the use of only the authoritative copy of a dataset, the nature of GIS practically encourages the duplication of data. Here are some scenarios that lead to the duplication of data:

• Perception that local data is better/faster/more available that than on the server. The answer, make a copy! Oh, and while we’re at it, tweak it a bit.
• Resistance to change in technology. You move all your vector or raster data to a relational database for centralized management and administration and folks still use the client tools to make a local copy in a format they are more comfortable with.
• An error is found in a dataset, someone makes copy, corrects it, and then uses it in their work rather than pushing the fix back through the organization to update the canonical copy.
• A customization of the data is needed and through evolution of versions, you end up with a dozen copies, typically labeled boundary, boundary_old, boundary_good, boundary_2k, boundary_best, and so on. Which one do you use? Better yet, which one does the new guy that comes on 2 years from now use?

The crux of the problem is that you can’t stop people from creating local datasets. I can hear the GIS freedom fighters howling now. Don’t get me wrong, I firmly believe in doing what it takes to get the job done and I’m not a command and control type. I’ve witnessed this from both sides of the fence and am convinced that in the long run, something has to be done to get control of GIS data without impeding the workflow and ability to complete projects in a timely fashion.

Perhaps its just a management problem.

        

Case in point – I recently needed to create two seamless regional layers from over 100 individual shapefiles. The source shapefiles were stored in individual subdirectories two levels deep. This translates into a lot of pointing and clicking to add each shapefile to the appropriate list in a GUI prior to appending them all together. To make matters worse, I had two shapefiles in each subdirectory that needed to be merged, resulting in two layers. This would mean even more pointing and clicking.

Enter a humble Ruby script (about 15 lines with comments) mixed in with a little bit of ogr2ogr and the process takes no time at all. Of course you can substitute Ruby with Perl/Python/Shell or whatever. Fortunately the individual files were consistently named in each directory. The script just gets the list of top-level directories, creates a command to pass to ogr2ogr and executes it, successively appending the individual shapefiles to create the regional shapefile. This takes like 10 seconds to run. Creating the second shapefile was a matter of changing a line or two in the script (I didn’t take time to make it robust with argument handling) and in another 10 seconds the job is done. In the GUI world, I’d still be clicking around (Ok, I’m exaggerating but you get the idea).
Of course most of the GUI Desktop GIS applications provide some sort of scripting language to do this sort of job, but often navigating the complicated API associated with an embedded scripting language takes longer to do than the actual job. If I’m in a hurry, I just want to get the job done, not navigate some complex object model to work out how to do a simple task.

There seems to be a trend in the commercial GIS world away from command line interfaces in favor of heavy GUI toolboxes and interfaces. I often find these limiting and extremely slow, if they operate at all. Mixing a scripting language executed from the command line with the many excellent Open Source GIS tools (GDAL/OGR included) often provides a fast and powerful way to get the job done.