Writing about Amazon: “Organizing into services taught teams not to trust each other in most of the same ways they’re not supposed to trust external developers”

Or writing about a fundamental truth: “Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there’s more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.”

But then getting to Google’s deepest fundamental tragic flaw: “one last thing that Google doesn’t do well is Platforms” … “product is useless without a platform”   …  ”.Google+ team took a look at the aftermarket and said: “Gosh, it looks like we need some … Let’s go contract someone to, um, write some… for us.”  … “Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs” … “The problem is that we’re a Product Company…”

ok, nevermind.  But,  I think it’s shockingly exactly right and in the long run it’s not a bad reason to short Google.

If I’ve peak’d your interest read the whole thing.

I suspect this says something about Hal Varian’s status in the senior leadership.  I presume he understands this.  But maybe not how fundamental it is.

“Starting times were shifted from 8 to 8:30. All class times were cut 5 to 10 minutes to avoid a longer school day that would interfere with after-school activities. Moss said improvements in student alertness made up for that lost instruction time.

The portion of students reporting at least eight hours of sleep on school nights jumped from about 16 percent to almost 55 percent. Reports of daytime sleepiness dropped substantially, from 49 percent to 20 percent.

First-period tardies fell by almost half, students reported feeling less depressed or irritated during the day, health center rest visits dropped substantially; and the number of hot breakfasts served more than doubled. Moss said the healthier breakfast probably aided classtime alertness….”

But it’s not impossible, since a few school districts have managed to make the switch.

I have since been reminded of another cartoon troupe.  When I was young, to illustrate what a rube some character was, he would sign a contract by making an X on the dotted line. And so I was pleased to observe a gentleman signing his credit card purchase by swiping the pen on the terminal screen from left to right making only a horizontal line. As I like to see if I can get a reaction out of the retail clerk I’ve taken to drawing a little smillie face when I sign on those terminals.

Signatures have legal standing, and some years ago the Feds authorized industry to go forth and invent some standards for electronic signatures. We now have a multitude of such systems. I was greatly amused by this one:

In that example the user signs by typing into the top box. In the lower box there then appears a faux signature. It’s a cartoon signature! Giving the impression that somebody actually signed this using a pen.

Almost immediately upon the wide spread adoption of patents industries fell into gridlock.   In a classic game of mutually assured destruction the individual firms in the industry would gather up patents.  Sooner or later somebody pulled the trigger.  If you lack a license to a patent the judge can shut down your entire business.

I think the first historical example of an entire industry shutting down was back in the late 19th century and involved the sewing machine makers.  That was in the days before antitrust laws.  So the captains of the sewing machine industry all got on the train and gathered in a smoky hotel room in Florida; where they invented the patent pool, cross licensing, etc.

An interesting side effect of a good patent pool is that it keeps other players out of your industry.  They only way they can get in is to accumulate enough patents to demand entry.

The anti-trust laws don’t prevent patent pools from arising.  They are common around industrial standards.  The players contribute, often under surprisingly murky terms, patents to a pool.  In this scenario the patents often play an interesting disciplinary role, if you try to implement the standard in a way the standards body considers offensive they remove your access to the patent pool.

Using IP rights in a disciplinary, or exclusionary, manner is also key to the solidarity of the FSF community.  Microsoft plays similar games as well.

For example Microsoft has a short list of psuedo-standards – i.e. standards they either wrote up entirely in-house or were the work of captive standards bodies.  You can see that list here.   In entirely an entirely unexceptional manner they then grant a license to practice a limited amount of the IP on some of their patents.   Just what you need to implement the standards, nothing more, nothing less.  Of course a naive journalist might think these licenses are more generous then that.

The IP that Microsoft captured two years ago is incredibly useful for solving internet identity problems.  In particular it could be used to allow users to reveal only exactly the minimum amount of information required to do this or that.  If the site wants to know if your over 18, you can let them know yes, you are.  Proving it.  And revealing nothing else.  If the site wants to know if your credit score is over 550; it can do that that.  It’s incredibly clever tech.  It’s incredibly useful.  If your the kind of person who thinks patents should exist, but only for really really innovative ideas – well this is it.

Sadly though.  This tech could be solving a vast range of real world problems today.  It could have been solving them for years and year now.  So if you looking for an example of IP rights undermining innovation and real world problem solving – well this is it.

If you are looking for interesting examples of a company using patent rights to shape the market toward standards that they own and operate then – well this is it.

Of course nothing is to stop foolish journalists from writing junk like: “This is a irrevocable promise by Microsoft that the company will not assert any claims against anyone using the technology that relate to any patents covering the technology.”  Which is bull.   You can use this IP only in implementing that short list of standards.  Only two on that list actually.  And I suspect only one of them, e.g. the standard that implies you have adopted Microsoft’s in house solution to the identity problem.

He argues that we have settled into usage patterns that are at odds with what TCP/IP was designed for.  This is obvious, of course.  TCP/IP was designed for long lived connects between peers; but what we use it for today is very short connections were one side says “yeah?” and the other side replies with, say, the front page of the New York Times.  I.e. we use it to distribute content.

And so he argues for a new architecture.  Something like a glorious p2p proxy server system.  You might say “yeah?” unto your local area and then one or more agents in your local area would reply with, say, the front page of the New York Times.

The talk is a bit over an hour and fun to listen to.  There is much to chew on, and like he says, it’s a hard talk to give.  In a sense he’s trying to tempt his listeners into heading out into a wilderness.  I’m not sure on the one hand he appreciates how much activity is already out there, in that wilderness.  On the other hand switching to a system like this requires getting servers to sign a significant portion of their content to guard against untrusted intermediaries.  There are reasons why that hasn’t happened.   That he never mentions push bothers me.  He points to a few systems that he finds interesting in this space, but I don’t think the ones he mentions are particularly interesting systems.

These are provocative ideas.  Very analogous to the ideas found in the ping hub discussions and the peer to peer discussions.   It would be fun to try and build a heuristic prefeching/pushing privacy respecting http proxy server swarm along these lines.  No doubt somebody already has.

No doubt that widget has just horrible engineering.  Terrible security for example.  Sloppy RF design which no doubt ruins your WIFI whenever you touch the toggle.  I’m assuming that widget is based on a bawky old industrial standard.

It’s an interesting puzzle when we will finally make the transition.  Today the vast majority of the controls that sit around the house have a wire behind them rather than a radio.  Someday the majority will have a radio behind them.  And just as interesting is how the market will shake out.   One thing that frustrates the transition is how fast the technology moves.  If we standardized on anything the tech would move on before we got it deployed.

Maybe these control signals will move thru the cellular network?  Push the doorbell and an SMS message gets sent to it’s cellular service provider who then routes the message back to your household’s high fi who’s speakers then discretely make the announcement that somebody is at the door.   A similar, but different, story for the household thermostat.  In that case the house would have a dozen tiny temperature sensing faceless cell phones scattered about the house, move them if you like, who’s only purpose is to pass occasionally information about the temperature in their environment.

All this could be done instead over bluetooth or wifi or yet-to-be-named.  And, I guess, these days the marginal cost of the widget/phone supporting lots of different schemes for connecting to the net.  But, cell phones are amazingly cheap.   I have a suspicion that the cost advantages of the cellular system’s widget production and the associated tangle of players is going to pull things down that path.

This is, of course, another part of the puzzle around Android et. al. and the puzzle of where in the stack things are more or less commoditized, and the eternal flame of who will own what in the network operating system.

Let’s start with that picture.  Google has revealed some of the plumbing that might form a standard on the boundary between Wave Service providers and any Wave Federations that may emerge.  Lower in the drawing we have some hints about how the traffic between members of these federations might move.  The term XMPP is mentioned.  But that is really not enough.  I’m not bothered by that, it’s early days and rough specs are to be expected.

Let’s move up in drawing, into the Wave Service Providers box.  It would have been a more credible industrial standard move if Google had one or two other players willing to run up on stage and signal their intent to become Wave Service Providers.   Alternately they might have released a complete reference implementation for a Wave Service Provider and, of course, they should place it under an open source license.   The word providers is plural, but at the moment we can only be confident that Google will deploy.  Until some other names signal that they are, at minimum, seriously considering wave I think it is fair to say: Google’s open platform signal isn’t really credible.  It’s not a party if your peers don’t come.  It’s a certain kind of federation if all your partners are tiny and you are huge.  But, again, it’s early days and it’s a big world out there so I’m sure Google can find somebody to come on board.

All the cut points in that layer cake is full of make or break options for Google.   Take a historcial example up at the top, whe the Macintosh shipped in 1984 they set a stellar example of getting that bit right.  They provided beautiful example applications and they provided clear and concise user interface guidelines.  Right now all we have is an example application, one which almost a research proof of concept.  It certainly isn’t as elegant a user experiance as the other Google web apps.

Much as my layer cake implies we will see multiple wave service providers it implies we will see multiple wave aware applications.  How many?  I think, and hope, it’s many.  But were is the signal from Google about this?  One wonders where the Google Search, Maps, Mail, Calendar, Docs, Voice etc. teams stand on all this?   Now, I think it would be insane for Google to take the risk of forcing all and sundry through out the company into a forced march to adopt wave.  But it’s very odd that you could make the argument that there will be only one wave aware application.   We need a much clearer signal about this.

Say you wanted to build an application for collaborative bill payment.  Questions start popping up fast.  Do you design it as plugin to some master wave application’s UI?  There was a period in the 80s and early 90s when a lot of the desktop OS vendors tried to create unified application frameworks; these didn’t work out.  Often it didn’t work out due to market power dynamics between the app vendors and the OS vendors and to a lesser degree it didn’t work out due to execution issues; but it looks to me like the same questions arise here.

Say I’m a vendor of web forum software, my customers install it on their systems.  Say I’m contemplating building a next generation version that’s wave aware, can it be installed on any of N wave service providers?   Does that sentence even meaningful?   This looks like a rerun.  We spend a lot of time in the 80s building software so it could run across multiple platforms.  These wave service providers look to me very similar to cloud computing vendors, very similar to platform vendors, and very similar to desk top OS vendors with their UI minimally interoperable user interface conventions.

Google has revealed a bit of the API provided by their sandbox Wave Service Provider.  How to build automated participants (aka robots), and how to build smallish widgets that provide little visualizations and games.  There is bit of hint that it will be possible how to build entirely unique kinds of wave documents.  That is one of the signs that it would be possible to build, say, a wave aware accounting application, a wave aware college admissions system, a wave aware travel agency.

It’s early days and none of the above should be taken as critical.  It is my intent to see if I can block out what I’d like to see happen.  Or maybe to just start to block out where to ask questions about what’s to be desired.

Wave provides the tools to do realtime distributed coordination of a complex activity.  For example that activity might be a game of checkers, or developing a product plan, a conversation, the distribution of a todo list.  So Wave provides tools to solve the coordination problems that arise when you have a data structure distributed around and multiple parties all modifying it.  Wave adopts the same technique we use for source control, optimistic concurrency.  Everybody edits their local copy.  These edits may turn out to conflict with each other.  The resulting conflicts are resolved by some mechanism, which in the Wave terminology are given the math like name operational transforms.  In source control systems I’ve always called that conflict resolution.

A Wave document is said to consist of a set of wavelets which in turn contain one or more XML documents.  For example a Wave document representing a game might have wavelets for all the players, spectators, officials, the score board, game state, the moves, advertising, discussion threads, and individual comments, etc. etc.  Nothing in the Wave specification blocks out how all those wavelets manage to relate to each other.  Different activities will, of course have different kinds of constitute parts.  Nothing I’ve read yet specifies even the building blocks for things like users, bits of HTML text, etc.

But the spec does block out what the primitives are for editing the XML documents that constitute the atomic elements of the Wave.  Those operations are a small set of editing operations: move pointer, insert text, insert XML tag, split XML element.  It reminds you of using a text editor.

These are the operations which might give rise to conflict.  If Alice and Bob are working on the plan for a party and both change the budget for snacks those edits might both be represented by a series of operations (move to char 120, delete 5 characters, insert “25.00″); with Alice entering “25.00″ and Bob entering “45.00″.  The protocol has a scheme for resolving this conflict.  It does not average the two!  It just picks one, deterministically, and move on.

That’s about it.  But there are some entertaining bits piled on top that are fun, and necessary.   I’ll mention three of these: historical background, how this all gets federated, how access rights are managed.

Optimistic concurrency goes back at least into the 1970s, at least that’s the first time I saw it.  I think the first time I saw it used to for a realtime application with human users was drawing system out of Parc in the 1990s – and one of Google’s whitepapers on Wave mentions that.  These days there are two very nice applications that I’ve used to coordinate activities Subethaedit and Etherpad.  I highly recommend Etherpad to anybody who’s working on an agenda or meeting notes jointly with other people – it’s fun.

While it is possible to imagine implementing Wave entirely as a peer to peer system with no central coordination, Subethaedit actually does that.    Wave implementors are all going to have a server that labors on behalf of the users participating in the activity the Wave represents by storing the Wave document, and orchestrating the on going edits and naively resolving conflicts as they arise.   The plan is to allow a user’s wave server to collaborate with other such servers.   That works by having one server act as master for each wavelet.  It’s worth noting that every participant in a Wave document is not necessarily a participant in every wavelet of that document.  In our example game, two spectators at a game can have a private chat within the game’s Wave document.  To be responsive each server caches copies of the wavelets his users are participating in, and to for reasons of  thrift and privacy these are limited to just those.  The authoritative server is responsible for retaining the master copy of the wavelet and for resolving conflicts.  So every edit flows to this master and then back out to the other participating servers.   There is a bit of crypto complexity layered on on top of that to assure that that bad actors can’t masquerade as another participant.

It is very unclear at this point how access rights to waves are managed.  Obviously wavelets will have participating users.  The participation will be asserted along with some rights; for example the right to view but not the right to modify.  In addition there will be groups of users, and these too will be asserted as participating in a wavelet.  If that’s the case then determining if a user has the right to do something will involve searching the user and group assertions on the wavelet.  Remember above that a Wave Document consists of just a set of Wavelets.  Obviously for any given kind of Wave Document there will be more structure than just a set.  For example our spectators’ private conversation would consist of the conversation and then a thread of comments.  It is totally unclear if how access rights are propagated or computed across those structures.

Everything is amazingly raw at this point.  Which signals virgin territory.   It’s unclear how good a landlord Google will be, but no doubt a lot of people are going to go forth and attempt to stake some claims on this landscape.

There are some bright spots in the specs.  Turtle (v.s. n3) and RDFa are both pretty nice.    They increase the chances that tools will rendezvous and thus decrease the frustrations around interoperability.  I think it is nice that Google’s giving RDFa some love.

The tough part of making any successful standard is getting a large volume of transactions taking place over it.  After years of effort, to first order, no data moves over RDF.  In other words the Semantic Web has been a non-starter.  Maybe it has a long runways but few Internet success stories do.

So one of the things that angered me: people at the core of the Semantic Web standardization movement seemed largely disinterested in working on getting a large volume of data to flow.  And, wanting to work on that I rarely found people to join in common cause with.

That is a classic problem in making standards.  The story is often the same.   The shape of the exchange network that emerges almost always the same.   The distribution of  producers, by volume, shakes out into a highly skewed distribution, a power-law curve.   It is the same story with the consumers.  You wee that pattern everywhere you look from seen in phone call traffic to web traffic.   For example on the consuming side Google and a dozen other entities try to consumer everything.

The lesson in this, if your advocating a standard, is that you need to be consciously puzzling out how to bringing both consumers and producers onto the standard; and you need to be conscious about bringing both the big players and the vast long tail of small players on was well.   You always have these four audiences to bring on board.  And of course there are others, i.e. experts, tool makers, etc. etc.   Everybody has to understand the value proposition, and that story is always the same.  You get on board because it substantially improves the transactions with the other guys.  The producers desire the consumers and visa versa.

This is the classic chicken and egg problem.  Why would anyone waste scarce capital to produce RDF is there is nobody interesting in consuming it, and visa versa?  Sometimes you can solve this problem bottom up; you seed some exchange between small producers and consumers and then other see what fun they are having and join in.  Given the vast number of little producers and consumers it is typically easy to get this started, but getting it to spread requires a lot of tinder and it’s best if there aren’t alternate ways to do these exchanges.  The trouble for the Semantic Web has always been XML.  XML provides a well supported means for the little guys to do their exchanges.

A while back somebody at Google asked me: “What can we do to help the Semantic Web along?”   I recall thinking, but not saying, that they should loudly signal that they are going to consume RDF, that search results would depend on that.  That would be enough, thought I, to trigger the entire search engine optimization industry into frenzy.  RDF would start popping out all over in the hope it would improve one’s Google juice.  If that actually happened then they could decide if the resulting data was useful.  But, I’m embarrassed to say I kept my mouth shut.   It didn’t seem to me like my interlocutor was likely to take kindly to the idea.   I doubted he’d cotton to the idea of Google actually playing hardball with their market power to shape standards.  Thinking back I discover I’m embarrassed.  I froze like a deer in the headlights. That makes me smile.

I’m happy to see that Google appears to have done exactly that.

So, what I find most encouraging about the Google announcement is that it might actually do something to draw out more semantic goodness.   With luck we might start to observe some RDFa in the wild now.  No doubt, at this moment, hordes of SEO consultants are reading the RDFa spec.  I wonder how long before all the big vending sites (Amazon, eBay, Walmart, Etsy, etc. etc.)  are too.

There are at least three aspects of that metaphor that I find lacking.  You need a metaphor that gives equal weight to both sides of the equation.  The services a platform provides are just as important as the applications it enables.  You need a metaphor that gives more weight to life-cycle of platforms; that in each round we experience a race to see who will own the platform.  The platform as land metaphor is far to lead us to ‘pay no attention to the man behind the curtain.’  You need a metaphor that embraces how important the network effects are.  All these can be see thru the lens of each other; particularly in the early days of as a platform emerges.

Consider the current state of play.  Developers seek out fresh real estate to build on and these days they appear to be gathering in two regions: smart phones and cloud computers.   So there are two species of platforms, two competitive games in play, two industrial standards battles.  In the life cycle of these platforms both horse races are well out of the gate.  Apple and Amazon respectively have grabbed substantial early leads.

Picking the right metaphor helps to assure you stay focused the right things and that you have the right expectations.  For example some applications, payments for example, are probably destined to become key platform features.  That in turn informs the question of who’s in the game.  For example is eBay/Paypal a cloud computing OS waiting to happen?  I think so.  It also helps to explain why Google or Amazon have a payments offering.

We know to expect an operating system to provide a file-system and a GUI.  We know to expect that a local government will provide public schools and some regulation of the sewage.  So, presumably we should be forming expectations about what features a cloud computer offers, or a smart phone.  Here’s a nice long list for cloud computing.  Here’s a shorter list for smart phones.  When the column fodder charts are that messy you can be sure of a lot of condensation and turbulence ahead.

The early days in the life-cycle of a platform are interesting in part because where the lines are drawn is under discussion.  Things settle down during the midlife.  I can recall the heady early days of the Mac when every release of the OS brought with it new extremely exciting APIs.  But also how each of these APIs was actually prototyped by somebody else, often an application builder.  There is always a tension between what will be owned by the platform and what will be owned by those around him.  This is a bit like how some wags like to complain that the town’s public produce markets or schools competes unfairly with private enterprise.   Right now, for example, there is a firm that is dominate in the geo-location via wifi market and there are three clear ways that might go.  They might be rolled up into one of the platform players.  They might be displaced by an open substitute (based on say open street maps), or of course they might survive as a vendor.

There is one place where I seem to most often run into confusion caused by the platform as land metaphor and that is with websites that are playing the open API card.  The metaphor causes them to focus primarily on getting developers to adopt their API.   That’s an unalloyed good thing, not just because it is actionable.  But it tends to make them blind to the dynamics of the battle unfolding all around them.  For example; for various reasons a service offered inside of EC2 is preferable to a service offered outside, at minimum it will be more performance and the bandwidth costs will be zero.  So I suspect we will see a trend toward all firms offering a open API moving some or all their offering inside of EC2.  More generally, and presuming that real competitors to EC2 emerge, they will have to build the same kind of branch offices inside each EC2 competitor.   That in turn is exactly like the “Render unto Caesar the things which are Caesar’s” dynamic seen around older operating system platforms.  Where, for example, a hardware maker or application maker has to carefully assure his offerings are supported by the OS vendor.