Passenger to driver: Where are we going?
Driver to passenger: I have no idea, but we're making great time!

Are We There Yet?

I have a four-year-old child. Like most modern, suburban families, we periodically pile into the car and go visit the grandparents. It's a six and a half hour drive. As you might imagine, I hear that question a lot. In fact, I've started hearing it in my sleep. After one particularly tough trip, I started mumbling it to myself. At work. Out loud. In public. When I realized that everyone at the table was looking at me, and all other conversation had stopped, I figured it was time to reformulate that question into something lucid. In a quick moment of introspection, I realized why that question had been burrowing into my brain.

For all the motion and commotion in this field, we don't really seem to be getting very far. I mean, we keep covering the same ground over and over again. Let's have a show of hands. How many of you wrote a new "Address" class in 1999? (Not to mention "Person", "PhoneNumber", or "Employee.") Why do we keep doing this to ourselves?

There are a lot of reasons for this repetition. Among these are the desire to correct mistakes of the past, the necessity of rebuilding the world in each new technology, and sometimes, the legal or cultural prohibition against using what has been done before. Most of the time, this kind of repetition comes from a sort of "architectural impedance mismatch." (One form of bad AIM is the "Wrong Base Class" syndrome. Think about how many BusinessObject classes you have dealt with. How interchangeable are they?) This mismatch is what makes us keep reinventing the wheel every two to five years. After all, if you can't reuse it, you must reinvent it.

That cynical voice in the back of my head calls this "job security," but I find that answer unsatisfying. I cannot help but think about all the cool and amazing things I could be spending my time on, instead of coding yet another database access layer. Let's dig deeper.

In the entire space of all current and possible technologies, there are many forces at work. Market forces, ideological forces, regulatory forces, to name a few. We are tempted to look for simple dichotomies like innovation versus standardization, but these are both the outgrowths of a larger, dynamic struggle, just like the mass of dead remains that makes a coral reef. We talk about these various forces and influences with the metaphors of conflict. "Browser wars." "Desktop wars." We even have our Geneva conventions, in RFCs, ISO standards, ANSI standards, ad infinitum. The interplay of these forces is like a game of Othello, with many players maneuvering to capture and lock down territory. One region might be stable for a long time, with all the action and excitement at the periphery, until a sudden reversal revives long dormant interest. For example, the balance of market and ideological forces changed, bringing Linux to new attention and revitalizing interest in UNIX and X. The front lines of the battle move and shift, not across two-dimensional territory, but in many dimensions. For that matter, the most skillful players create new dimensions to rout a particularly intransigent opponent. You can see this maneuver exemplified in the positioning of Java.

One of the results of this struggle is constantly shifting "geography" of technology space. In a real sense, we are not getting anywhere because the ground keeps transforming beneath us. We aren't there yet, not because our destination keeps moving, but because the terrain between here and there keeps changing. These changes, along with the fierce competition for territory forces the inhabitants of the niches in technology space to evolve. More evolved forms tend to be more diverse. Forms diversify to occupy every available niche. In the process, they become more increasingly specialized and their relationships become more complex. The net result is an ever-increasing number of niches with an ever-increasing number of inhabitants.

Take XML, for example. I have used XML extensively on several projects. I'm running the risk of being called a heretic, but I think XML has some big problems. For instance, XML really defines a meta-standard. That is, it creates a standard framework for expressing other standards. So, instead of thousands of different, incompatible communication protocols, you end up with thousands of different, incompatible XML schemas. The end result is the same. Babel. The aim of XML and its complementary standards was to provide interoperability between diverse applications from many sources. The net effect is to standardize one niche and create dozens of new ones, in which the competition is just as fierce as ever. It isn't quite a case of two steps forward and one step back.  It's more like one step forward, two sideways, and half a step up.

Or, take a look at Jini, from Sun. Jini is a very cool technology. I can add services and resources to a network just by plugging them in? Cool! I can let my applications discover what capabilities are available? Excellent! But wait a minute. So, my application can discover that there is a FooGrommet on the network, but what does that mean? How can my application use one of those? Well, the answer is that FooGrommet should implement some "well-known" interface. So who defines the interfaces? Ah, I see. More dynamic forces, in the form of working groups and committees.

This pattern shows up time and time again, everywhere. SQL. HTML. WML. Directory services. These competing forces are always at work. The natural consequence is of these forces is complexity.  Standardization and differentiation are only side effects.

And of course, who gets to manage all that complexity? Who has to create architectures that use enough standards to minimize risk, while going outside the standards to deliver value? Who gets to play "mix-n-match" with vendors, tools, and technologies? Who indeed.

So, what do we do about this? How do we make genuine forward progress in the evolving, changing environment of technology space? Which way is forward, anyway? Well, I am going to ask a lot of questions with this column. I don't expect to answer all of them. Sometimes, the value comes just from asking the question, whether it gets answered or not. By all means, send me your questions, too! We'll explore this space together.