The intellectual property discussion is a tarpit. I’m not about to dive into that! I just want to stand on the shore for a moment though before saying something about the power of optimistic concurrency to enable higher rates of creativity.

The arguments for intellectual property are assorted. For example one argument that’s made is that the creator of new ideas has a moral right to a portion of the value engendered by those ideas. Another popular argument is that the creators of ideas will horde them and the property rights are a bargain society makes to encourage the revealing of the ideas. Some folks have suggested that an alternative to intellectual property rights might would be to offer prizes; sometimes these are called tournaments.

While any discussion of either property rights and or tournaments ought to rapidly be converted into a discussion of who gets to decide the rules of the game, I want to take a turn off this well trod path and point out something I’d not noticed until the other day.

Optimistic concurrency provides one potent benefit for the participants – they are not forced to commit their reputation to the game up front. Consider a simple scenario.

I decide that I’d like to make my open source project faster. So I get to work on that and after a while I fail. Meanwhile, as a side effect of that effort, I manage to make the code cleaner. So I take my improvements and I submit them back to the central project authorities. They smile and say thank you and I have a little win. I smile.

Notice that my failure to make the code faster is never a matter of public record. If it had been a matter of public record I would have suffered a loss. Taken a few points of damage to my reputation. These damage points would almost certainly outweigh any positive points I might have garnered from making the code cleaner.

Economists looking at open source have a reflex. They immediately start looking for a market and the currency used to clear the transactions. Many of them have decided that the currency is reputation. (I won’t even begin to get into how reputation is not necessarily a scarce commodity!) When they gleefully announce this to me and some large proportion of my friends in open source we look at them with a kind of bewildered stare. “Ah… no?” we think. One reason for that reaction is that the work is not driven by a bet on one’s reputation, in fact if you thought your reputation was a risk as you proceeded to investigate some creative improvement to the code you would likely significantly reduce the scale of the things you attempt. We work very hard in some of open source communities to avoid reputation games – we strive to create an abundance of reputation. How’s that for optimistic!

I carry a lot of negative baggage in and around the idea of tournaments and games. One way to sum up my feelings about them is that they frame up a situation where N individuals enter a room and 1 winner emerges. Which is all well and good until you notice that N-1 losers emerge as well. It confuses me why people of free will would enter such a room.

The advantage of optimistic concurrency as an organizing principle is that is that it allows the individuals to enter the room without taking as high or certain a risk that they will exit as losers. Heaven only knows how many people are at this moment attempting to make a better Apache HTTP server. That’s a good thing!

A second thing that confuses me about the culture’s affection for tournaments is how naive people seem to be about the framing issues. You can’t have a tournament unless somebody establishes a set of rules for how the game will be played. That individual, the rule maker, is the one with the power. So if I establish a prize for making Apache faster then I am almost certainly displacing efforts to make it cleaner.

I have heard the arguments made in favor of creating a frame around the work. One of the presumed benefits is that it creates a motive force for the participants – a carrot (the prize) and the stick (the risk to their reputation of failure). One of the presumed benefits is that it gives the organization (the rule maker, the hierarchy, what ever) a way to measure the quality of the work being done. These are, as far as they go, probably all true.

That model is at least problematic in creative problem solving work. Consider a typical example.

To frame the pending work you and your organization decide that you’ll write a plan. In software these are called by many names. For example FRD or Functional Requirements Document. This plan becomes the rules for the next stage of the game, i.e. implementation. This plan becomes input to the judges of the game; i.e. the QA department. It’s very easy to fall into the trap of allowing this process to shift a cooperative creative enterprise into a in-house competitive one.

Two obvious problems here. The FRD is written at point in the project life cycle when the absolutely minimum amount of information is available about what’s is going to be possible to achieve. I.e. it is written at the point when the creative labor has the least knowledge available to make a good bet. Second, the FRD raises the barriers to capturing other opportunities that might arise during the implementation phase. For example if an implementor encounters an opportunity to make the code cleaner during the implementation phase, but the FRD doesn’t license such work, he is faced with ethical and organizational coordination costs that reduce the chance of capturing that value. It is impossible to enumerate the set of all possible dimensions such chance opportunities might arise along.

In discussing this with various people I was surprised by how many people noticed one additional thing I’d not noticed. Creative labor will seek the place in the problem space where the work is coordinated optimistically. For example there is an entire syndrome in the art world where some artists have become specialists in the art of writing the proposal. The proposal becomes the art. The actual implementation of the proposal is a minor epilog. I’ve observed that in any number of domains including FRD writing.

That’s a weird unintended side effect of attempting to tackle the problem of coordinating a scarce resource. That the art of coordination begins to become the principle attractor of creative energy. Talk about your unintended consequences!

In brief, optimistic concurrency is a technique for solving coordination problems. It presumes there will rarely be a problem. Should a problem arises you then fix it after the fact. Contrast that with a conservative or pessimistic approach that sets up rules and procedures once set in motion don’t suffer from concurrency problems. This tool, optimistic concurrence, is found in the database community’s workshop.

Recently I’ve become increasingly interested in how this approach is scale free. You can use it to organize database updates or entire social systems.

Consider an example: you have a database, of bank accounts say, and a horde of folks are updating that database. They access account balances and add or subtract amounts from them. A problem can arise: what happens if two people both pull the account balance, do their bit of arithmetic and put back their answers – at “the same time.” One of them goes $100+$10; the other one goes $100+$20, the first one puts back $110; then the second one puts back $120. This is bad, the account now says $120. The correct answer is $130. Trouble! Conflict! Rivalry! Call the police. Write some laws!

The classic solution to this problem is to “lock” the account during the update. This forces the second guy to waits while the first guy finishes. Problem solved – but for one problem. The coordination cost (i.e. managing the locks) is more costly than the work were doing. I’m sure we have all experienced bureaucracies like that! Optimistic concurrency can be a fix for this class of problems.

If we assume that the chance of this kind of collision, e.g. two updates to the same account at the same instant, is amazingly small then we can design an approach that shifts the cost of resolving the conflict caused by the coordination. Instead of bracketing the transaction we can arrange for the expensive part to happen only if a conflict occurs. There are a few ways to do this but the general idea is that each account update checks before it commits it’s change to see if the account was updated while it was working on it’s update. If so then it has to redo the update.

Resolving the conflict after the fact is a pain, but overall it is less painful than taxing every single edit.

Which approach is preferable depends on the chance and cost of a conflict. If the chance/cost of conflict is high then you probably want to go with a more regulated design. If the chance is low then optimistic concurrency is preferable.

One reason I’m thinking about this a lot these days is that this is a question about scarcity, abundance and collegiality. It is a tiny example of the design problems around a public good. The membrane design problem. If conflict is likely then you need a more tightly regulated membrane; if the conflict is rare then you can design a system were typical actions unfold at low cost but you’ll need something to resolve the disputes when conflict arises.

Optimistic concurrency is a scale free tool. You can use it for tiny things like the account balance in the example above. You can use it for big things like the entire source code of a large software project.

We use it extensively in open source projects. Consider a typical transaction that might be made to project P’s code. “Make P faster.” We rarely hand out ownership (i.e. a lock) for a project like that. We rarely create plans (i.e. a lock) for a project like that. Rather we allow any number of contributors to attempt that goal. We resolve conflicts late in the transaction, when they return with the code that achieves the goal. That’s sometimes called “code talks.” But the important thing is not that we shun talk, what’s important is that we resist the lock grabbing. Lock grabbing presumes that the options for what to do with the code are scarce – they aren’t.

I’ve convinced myself that open systems and optimistic concurrency are very similar ideas. Open systems make a choice to be optimistic about the coordination problem. They assume that the system will rarely suffer from rivalry or conflict. Open systems presume that when it does arise, the conflict can be resolved after the fact. I’m very amused by a feedback loop this reveals. When conflict rises, over some threshold, then it appropriate to move to a less optimistic and more regulated system design.

It amuses me is how often some people, usually those familiar with highly regulated systems, will advocate the introduction of a bucket of regulations when ever a conflict occurs. The large angular size of a current conflict often causes even those with the most affection for an open system design to consider switching to a less optimistic design. “Redoing the transaction” is deeply embedded in the optimistic concurrency design. That cost is high – particularly if you personally are stuck doing the reimplementation.

In optimistic systems the cost of freedom is cleaning up the mess when your optimistic presumption is proven wrong. That’s a statistical certainty.

Optimistic concurrency works better in situations where the options for action are abundant and the probability of collision and conflict are low. It’s worth noting that modularity helps to enable just that. Modularity makes the system more granular. The chance that two updates will conflict around one bank account is much lower than the chance that two updates at same bank.