Between producers and consumers you need to insert some sort of exchange, a trusted intermediary. I’ve always liked the way that people draw this as a cloud. It suggests angels are at work, or possibly that if you look closely things will only get blurry and your glasses will get wet. Even without getting your self wet things aren’t clear even from the outside. For example what do we mean by trust? Does it mean low-latency, reliability, robust governance, low barriers to entry, competitive markets, minimal concentration of power – who knows?

There are some leading design patterns for working on these problems. Sometimes the cloud condenses into a single hub. For example one technique is to introduce a central hub, or a monopoly. The US Postal system, the Federal Reserve’s check clearing houses, AT&T’s long distance business are old examples of that. Of course none of those were ever absolute monopolies; you could always find examples of some amount of exchange that took place by going around the hub – if you want to split hairs. Google in ‘findablity’, eBay in auctions, Amazon in the online book business are more modern examples.

In the blog update notification space the hub originally was weblogs.com, but for a number of reasons it didn’t retain that role. Today the cloud’s structure is kind of lumpy. There are lumps on the producer side and the consumer side. On the producing side all the majors have aggregated a supply of pings; for example Typepad presumably has a huge supply of pings, WordPress comes out of the box pinging Ping-o-matic. On the consuming side big players aggregate pings as well, some of these are evolving toward or are explicitly in the role of intermediaries. The blog search sites are a good example (Pubsub, Feedster, Technorati). They all labor to discover fresh content. The trends seems to be toward condensation. Industries in this situation often, given time, create some sort of federated approach.

Federated approachs are clearly more social than hubs. In markets where the participants are naturally noncompetitive they can be quite social. For example when national phone companies federate to exchange traffic, or small banks federate to clear checks or credit card payments. That can change over time, of course. The nice feature of a federated architecture is that it helps create clarity about where the rules of the game are being blocked out. How and who rules the cloud is part of the mystery of trust.

There is a third school of design for the cloud, what might be called “look mom no hands” or maybe technology magic. The first time I encountered that one was the routing algorithms for the Arpanet; but that’s another story. These techniques go by various names; just to pick a few P2P, distributed hashing, multicast. The general idea for these is that the N participants all install a clever lump of code; these clever bits then collaborate (using elegant ideas) and a routing network emerges that makes the problem go away.

The magic based solutions have great stories or illustrations to go with them. For example the early Arpanet architecture had each node in the net act advertise it’s service to it’s neighbors who would then shop for the best route – boy did that not work! I worked on on three multiprocessors in the 1970s that had varing designs for how to route data between CPU and memory. One of these, the Butterfly, had a switched who’s topology was based on the Fast Fourier Transform.

Recently I’ve been reading some papers about a system that uses what you might call fountain routing. All the nodes are arranged as peers around a circle. Packets bounce thru a few nodes to get where they are going. Each node has a few paths to distant parts of the circle and more paths to it’s neighbors. The illustration shows one node’s routes. A packet entering at that node would get thrown toward a node closer to it’s destination that node will have more routes to even closer nodes.

The monopoly/hub designs are social only to the extent that a monarchy a social structure. The federated designs are social in the manner of a town meeting or the roman senate. The technical magic solutions are social in the sense the sense that they take as a given that everybody will sign up to the same standard social contract. Each of these social architectures has it’s blind spots. For example the federated design can easily lock out small players because they can’t get a seat at the table. Magic technical solutions seem to break down when the traffic patterns stop behaving what ever their designers presumed. E.g. as the blogging example demonstrates the real world is rarely high trusting, random, uniform, collaborative, and peer to peer.