Here is a nice example of reaching a bogus conclusion. I’m a huge fan of positive reenforcement. Animal trainers know that a smart animal will withdraw and become devious if you if you try to train it using negative reinforcement. But it is trivial to prove that negative reinforcement is more highly correlated with improvement than positive.

Let’s train a pile of pennies to behave. Good pennies come up heads. Flip the pennies ten times. Divide them into two piles, the good ones and the bad ones. Now punish the bad ones. Beat them with stick. Repeat the experiment. Notice how their behavior improves! Now reward the good ones. Kiss them. Repeat the experiment. Notice how they appear to be slacking off! Clearly negative reinforcement works and positive reinforcement doesn’t.

The technical term for this is regression toward the mean. If the behavior is random the past performance doesn’t tell you anything about the future. But if you have a high performing penny you can be reasonably confident it’s performance in the future will be only average. In the stock market of pennies should bad pennies have depressed prices invest in them. That is counter intuitive, but only a variation on how we can reach the bogus conclusion that negative reinforcement improves performance.

Here’s an example of the bogus conclusion in real life. At one point the Israeli Air Force concluded that they should stop complementing the pilots that did well. They had noticed that afterward they would perform less well. They concluded they should continue scolding the poor performing pilots. They tended to do better. It’s an interesting example of thinking that the causality is due to what ever you did most recently.

In risky, aka random, situations – such as a warfare, experimentation, entrepreneurial activities, learning, etc. – your very likely to have an extremely strong component of random luck. In that case you can trivially fall into this bogus trap. In this environment you need only find the loosers, and beat them. Later they try some other random schemes to make things work and sure enough some will regress toward the mean. You pat yourself on the back for your deep understanding of behaviorism.

This isn’t an abstract issue. It goes straight to the question of how we design the systems we work inside. Let’s look at Brad DeLong’s recent posting on the tournament based architecture of the academic.

The process of climbing to the top of the professoriate is structured as a tournament, in which the big prizes go to those willing to work the hardest and the smartest from their mid-twenties to their late thirties.

I have a very poor opinion of tournament based systems design, particularly when rounds in the game are a huge chunk of a human life span. I think it’s pop-Darwinism. I think it leads to abuse of power by those who set the rules. I think it makes those who play in the tournament devious because their most highly leveraged approach is to work on gaming the system. I think this design drives out skill diversity and that has fatal consequences on the quality of problem solving. I could go on, but let me draw attention to something else Brad said.

Brad is primarily talking about how the tournament architecture of academia has some exclusionary effects. Not about all the reasons tournaments might be a lame design. He uses as a launching board the blood sport going on around Larry Summer’s recent comments about the women. Because Brad really wants to talk about something else he buries the sentence that clears the air so he can talk about what he’s trying to puzzle out.

And I say this as someone who thinks that Summers’s views on gender, genetics, and math achievement are almost certainly wrong, are unsupported, and should not be pushed forward by somebody who is twenty years beyond the stage of his career where you throw out lots of unfiltered ideas in the belief that what matters is the quality of your best one.

Right on bro.

But I notice how that sentence speaks to the question of tournament design viewed thru the lens of regression to the mean. It appears that there is a forgiving phase in the game when quantity of ideas is treated as a good thing. The phase is highly random. Presumably all your competitors in the game have very similar skills or even extremely similar in skill. How might you get that? Well, design the game design with a recipe that includes very standardized filtering schemes and then add a good dose of group think. Now, given a very uniform pool of players the winners of this tournament will almost surely very be positioned to undergo regression toward after they get the prize. Opps. They should become extremely careful to husband their high rank after they capture it. I.e. they probably ought to shut up; or at least they should stick to what has worked well in prior rounds. If the game is set up with lots of negative reinforcement and that is extra true for those of high rank then this design leads to an extremely conservative behavior being your best option – if your a winner in early rounds. Blogging bad; since it breaks the mouth shut rule. Tenure good, since it tempers at least a little the risk of idea generation in a highly negative reinforcement based system.

Since they generate far more losers as output than winners, systems based on tournaments are fundamentally about negative reinforcement

Now look at high tech. Open systems work to create a large surface where random experiments can take place. Some of these work out, most don’t. Notice how given that if you manage to have a success you ought to fear of regression toward the mean.

I’ve spent a half day twice at LinuxWorld here in Boston.

There are some traits that partition the universe of exhibitors. Big firms, little firms, and “dot orgs.” Hardware, Software. Developer facing, enterprise facing. The Developer facing fall into two camps; hardware and software. There are folks in booths, folks in stations inside somebody else’s booth, and then guys wandering around on the floor.

Generally I find the big firm booths lacking in authenticity; over designed booth furniture; bevies of bitter beautiful booth barista; huge lcd screens owned by power point; presenters who’s talk is feed into their ear from a tape player. Sometimes a gimick; for example Sun and AMD have a lan party setup. Well, it’s not a party it’s a contest and it’s obnoxiously loud. All it’s neighbors are pissed.

Redemption for a big firm booth is possible. Some of the firms have let out a big chunk of their real estate to stations occupied by their “developer partners.” These are typically just like little the small firms; except they are sitting at a station that lacks the authentic signs of a real firm – i.e. somebody eating in the back of the booth, the luggage, or better yet a child. This lack of authenticity meant I didn’t actually realize they were there for a while.

In a strange demonstration of some kind of emerging hierarchy of nobility I spent a while talking to a guy in a station at a huge firm who runs the developer network for mediums sized firm and then later in the space of the medium size firm I spoke to guys at stations he had provided them.

There is a lot of stuff going on around ‘thin client.’

I think my favorite booth was LTSP.org’s. They guys seem to mostly work in schools. Their problem; a huge pipe of misc. computers – old, donated, random – that they can’t bring them selves to give away. So they have given life to the thin-client network computer fantasy. Except this time it works. You get the feeling that if it has a CPU, a screen, and a keyboard these guys can make something useful out it. Disk? no way! I’d never notice etherboot.org before. This is such a hack! They replace the ROM on the ethernet card so that you can bootstrap off it, and then off the net. The traffic at both these booths was huge.

I had fun talking to buys who’s companies sell thin client hardware. They all have really fun surprising customer stories. Like the schools making useful stuff out of old junk these guys are really making people’s lives better in little insurance offices, auto dealers, etc. etc.

Lots of people say document management in their booth signage. I don’t think two of them are doing the same thing! I had a lot of fun talking to a guy who’s document management system has mostly gotten used to capture faxes, and scans of paper documents in small to medium sized offices.

I’m mostly not that interested in the enterprise facing vendors; but there is an trend there. When I used to go to Mac World you would see trends. Sometimes they were huge – like one year there were a dozen disk drive expansion vendors. The next year there would be two. The following year there would be none, or one. The trend I think you can see here is tools from small vendors to help solve what could be called the migration problem (moving from Windows to Linux for example) or it could be called the heterogeneous systems problem. I like the second term better. I think we are moving into a much more complex and confusing IT world. Lots of junk of various kinds. A dozen databases where somebody obsessive might prefer one. A hundred systems all jockey for screen space in front of the person getting the work done. Of course if there was nothing but time then we could integrate those hundred systems into one rational unified UI; but who’s got time? It’s much more fun to solve problems and move on to the next problem. Getting a pretty unified UI or master database schema is navel gazing.

So we get the hacks to assure documents of a given account number to pop up in the web browser when ever the call handling software receives a call. We get hacks to keep N different employee account databases in synch. Why N? Mergers, politics, expedience. Surely your not going to tell one division that they can’t install a system that gets them a huge productivity improvement just because it happens to integrate poorly with the LDAP directory you the parent company’s obsessive compulsive IT group has wedded themselves to?

The outlook substitutes market is looking healthy. Someday that’s going to totally disrupt the outlook/exchange network hub. But it’s only motivation right now is cost. CIO’s do not care about cost. They care about risk. They care about game changing advantage. They care about keeping a lid on diversity, aka chaos. Offensively, they are typical monopolists. So these offerings remain poorly impedance matched to the customers that have huge outlook/exchange installations.

There are a lot of offerings in the network monitoring space. Like all of software these show little correlation between features and price.

When I browse at these open source shows I’m always struct by how rarely my “man that’s cool” reaction is. During the first years of the Mac it was a very very common experience. Rich geographic database apps on the mac in 1986, for example. Or the constraint based drawing programs. Or the way that you could drop an object into a pallet and the pallet would learn that subset of the attributes of the object which were relevant to the pallet and create a new cell in the pallet.

I expected I’d see cool innovations in the network monitoring booths. But mostly not. There were some clever tricks for packing more info in the trend graphs; but not really surprising. Nobody seems to be making any progress on the hard problem – managing the alert population given the human failings of the recipients of the alert stream.

I did like the system that observes the OS call patterns of your application and then generates alerts when new things happen.

Nobody seems to be doing anything that creates collaboratively authored rule sets.

I’m always amused by how some booths are vicious in their lead qualification. They have one or maybe two questions they ask and if you don’t happen to give the right answer they ignore you and go back to talking to themselves. They have got that evil beast serendipity in a cage.