Readers following this blog have seen my occasional references to geocaching – a sport/hobbby/pastime that Miles and I do quite a bit of, which involves using a hand-held GPS to place and find hidden treasures – either in the woods or in the city.

One of the many unusual aspects of geocaching is the fact that it relies completely on the existence of a single web-based database, represented by the site geocaching.com. As web-based database applications go, the site is a modern marvel. The database represents hides, finds, people and their discovery logs, travel bugs (ID’d items that travel the world, hopping from container to container), and more, all sliced and diced a million ways to Sunday. The site is deeply geo-enabled, letting users hone in on hides near them, along a route, or near arbitrary destination locations. It’s also one of the best examples I’ve seen of useful Google Maps mashups, relying heavily on the open APIs provided by Google to integrate its cache database with Google’s map database. This is what map mashups are all about, and geocaching.com has done an amazing job with them.

As the popularity of personal GPSs rises, so does the game’s popularity. But when geocaching.com goes down (or slows down), so does the game, which involves more than half a million hides world-wide, and many millions of players. The site, which is, sadly, based on Microsoft database technology and ASP, does go down from time to time (big surprise); it’s a “single point of failure” in bit-space for the entire meat-space game – a precarious position.

We could – and probably should – have a separate discussion about ways to distribute the load and eliminate that single point of failure, either by replicating / load-balancing to other servers elsewhere in the world, or by coming up with a protocol and distributed architecture so the game isn’t in the hands of a single group to begin with. Discard the dependency on a single organization and open source the whole concept.

Lots of difficult problems to solve there, but save that thought for another day. These notes are about Web 1.0 vs. Web 2.0 cultures. Yes, I know those terms are vague and scattered, but for these purposes I’m thinking about one key ingredient of Web 2.0: Open-ness, manifested in technology as interoperability between servers and clients via published APIs.

The ability for people to do cool things with data living on someone else’s server is what has enabled the rapid growth of cottage industries surrounding the most popular web 2.0 sites. There are dozens of external web sites and desktop/phone clients doing amazing stuff with the data living on Twitter. Facebook’s API is credited with the huge ramp-up in that site’s popularity over the past couple of years, as thousands of developers wrote applications to interoperate with the site. RSS/Atom have enabled countless opportunities for interoperability between sites. XML-RPC lets us create excellent desktop publishing tools for posting to blogs of all kinds, and to get our data into and out of web 2.0 sites. Google’s maps API has opened up a universe of possibilities for creative developers working on other sites. Flickr’s open API has created a vast cottage industry of external sites grabbing, slicing, and dicing data on Flickr in creative ways. When a site is built on top of structured data, that data should be available in programmatic ways. Open that gate and let the building begin. It’s not just about technology – it’s a mindset that opens doors.

Compare: In 2008, I can’t even get an RSS feed of my recent finds from geocaching.com. In fact, even though I can see my recent finds through the site, I can’t even create a distinct URL for that view of their data to give you here. Nor can I get an RSS feed of caches recently published in my area. In fact, geocaching.com doesn’t even seem to know that RSS exists – one of the most fundamental technologies on the web in the past eight years, completely missing.

Similarly, there is apparently no way for external sites or clients to programmatically retrieve data from the site. Since the day we first heard that 2nd-generation iPhones would come with a built-in GPS, many of us thought the iPhone would become the ultimate geocaching device, allowing us to go “paperless” from anywhere in the world, without loading up our GPSs with waypoint data before leaving the house. Instead, what we ended up with was a well-intentioned but anemic client called Geopher Lite – a noble attempt to create a geocaching application for the iPhone, but which fails spectacularly for one simple reason: While Geopher can easily determine your current location, it can’t pass that information to geocaching.com and get back a list of nearby caches. And if you select a cache with its built-in browser, it can’t get that cache’s coordinates into its own dataset. Geocaching.com is so closed down that even the most basic level of interoperability is impossible. It’s just sad.

I’m currently in the process of building a geocaching satellite site in Django (more details on that in the future). Not having an open API at geocaching.com is a major pain in the butt, and has put the kibosh on many of my plans. Shortly after getting started, I realized that if I had something as simple as a cache ID, such as GCK6F2, there was no way for me to construct an automated link to that cache’s page at geocaching.com — the cache ID isn’t even present in the unreadable hairball URL (geocaching.com apparently never got the memo that “URLs are architecture, and should be readable / elegant / meaningful). So I asked in the forums whether there was some kind of shortcut URL I could use to redirect from a known cache ID to a cache’s page. I did get a useful answer, but I also had not one, but two very experienced community members insinuate that I was a bad guy, probably intending on scraping the entire database for my nefarious purposes.

This blew my mind. The culture of the site is so web 1.0 that a basic question about interoperability was met with distrust. Not only is geocaching.com lacking the technology it needs to enter the web 2.0 world, it’s lacking the culture needed to support it. In 2008, interoperability between sites needs to be encouraged, not discouraged. Sad that geocaching.com’s traditional closed-ness has created this kind of culture.

There are many things I’d like to do with my project that I won’t be able to do as a result. But I do plan to respect the geocaching.com terms of service, even if I don’t agree with all of them.

The irony is that geocaching.com relies so heavily on the open APIs provided by Google and other mapping services, but provides no open-ness back to the web in return. Imagine using geocaching.com without the map mashups integration – it would be nearly impossible. One would think that the folks at geocaching.com would see their own mashups as an example of the great ideas that bloom when datasets and APIs are open and shared.

I don’t want to give the wrong impression – again, geocaching.com is an absolute marvel, and one of my favorite web database applications in the world. Hats off to everyone who’s labored on the site over the years; you’ve built something really incredible. I really do appreciate your work. But it’s time for change.

In a perfect world, geocaching.com would ditch the Microsoft technologies it’s sitting on and re-write the entire system in Django, being sure to build open, published APIs into every imaginable corner of the system. Then, to solve the reliability problems of the site, move it all into Google App Engine, solving the scaling problems for good (App Engine happens to love Django, but that’s coincidental). Finally, sit down with all of the geocaching.com employees and explain to them that it’s time for a culture shift — that it’s time to enter the world of open-ness and interoperability that transforms sites from walled gardens to thriving platforms. Then just sit back and watch as hundreds or thousands of add-on sites and services bloom, possibly leading to entirely new modes of geocaching.

I know, pie-in-the-sky stuff, not likely to happen. And I don’t like to come off as an armchair analyst, pretending to know what’s best for a site I don’t own or control. Re-building / rethinking geocaching.com would be a massive undertaking. I don’t want people telling me how I should throw away my labors of love and start over, so I’m loathe to suggest same for someone else’s project. On the other hand, geocaching.com is a resource for the web community, and it’s not keeping up with the technologies that drive modern web communities forward. I’m just dreaming aloud here – take it as such.

Filed under Geo, Web Dev | 10 Comments »