Here’s some more fun stuff gleaned from the book I’m reading.

One of the canonical reasons for poor judgement is known as availablity; i.e. we think and work using the tools that are easily at hand. If a tool is beyond reach we are far less likely to apply that tool. There are lot of examples. The folks that plan for floods, for example, rarely plan for floods other than those they have experienced. If your asked a mess of people to estimate how many words start with the letter r, like road, v.s. how many end with the letter r, like tour they will get the wrong answer, begins with, because your noodle is set up to recall words by their first rather than their last letter. I love the variant of that example; people will guess that abstract nouns, like love, hate, truth, are occur more commonly than concrete knowns, like door, street, bicycle. Apparently people have an easier time recalling abstract nouns v.s. concrete ones. Who knows why, possibly because the abstract nouns cover broad swaths of experiance.

In reading about availablity, and the other insights outlined in this book, I find myself thinking about how they get mixed into the ways people market products, debate issues, manage planning, and even how you could create carnival games out of them. One reason a component manufature has to give way a huge number of free samples is to assure that his components are highly available at the moment when some random designer is reaching out to pick something up to solve his problem. That’s one reason Open Source works; it adopts a r-Strategy.

This example of availably is particularly thought provoking. A reasonably standard formal way to estimate risk is to create a fault tree. You can view lots of examples at Google Images. The process of making a fault tree is lot like the process people use in making any plan. You attempt to collect all the contengencies and then sort out their likelyhood and dependencies. For example in thinking about why the car might not start you might include dead battery and left headlights on by mistake. Professional planning and fault tree work involves getting experts to help. You might do that by doing surveys and investigations of various car starting falures; or you might interview experts (say are mechanics) and have them help generate contengencies and probablities of those contengencies.

Because of availablity asking the experts does not work.

Say you have methodically assembled a large and nominally complete fault tree and because you doubt you’ve thought of everything you include a node for “other causes.” At that point your fault tree covers a 100 percent of the failure scenerios. You go to the experts and ask them to help by assigning 100 points (i.e. percentage chance) across the perimeter of the tree.

Why doesn’t that work?

It doesn’t work because studies show that you get radically different answers from the experts if you check their work by varying the fault tree you present. If you have a tree with 10 nodes and you collapse 2 or 3 into the “other causes” node the experts will then raise their estimates for all the remaining causes. Because the nodes you merged into “other causes” are no longer easilly available.

Of course what’s in that “other causes” node is the long tail of contengencies.

Cool! Hans Rosling has a blog. Hans is one of the principles behind Gapminder which I posted about a long time ago.

Reading Charles Tilly’s book on how we explain stuff has lead to buzzing in my head about conventional wisdom. We all have lots of these conventional bits of wisdom we use to explain things. For example: Question:”Why do they do that?” Awnser: “For the money.” We collect these into little internal FAQs. That Q&A is taken from the booklet labeled ‘rational economic-man.’ Each of the booklet we collect is self consistent.

Together these Catechisms let us move quickly through our days. It’s a very effective heuristic. It’s always comforting to be able to explain the world. It makes it seem safer.

Trouble arises when these rules of thumb are wrong in material ways. That “in it for the money” rule is unlikely to be useful in social spheres, for example. Distance helps to reduce the chance that a mistake will have consequences. If you’re a Fortune 500 company you may not occupy any social sphere and the rule above will do just fine. Living in the US a poor explaination for why HIV infection rates are 27% in South Africa is less likely to materially effect your life.

These days people like to go on about how the world is getting smaller and how the boundaries between spheres, particularly geographic ones, are getting thinner and thinner. In so far as that is true the risk of living with a consistent but flawed Catachism is increasing. There is a rising tension between a Reality Based v.s. Faith Based scheme for moving through your days. Which is a huge pain in the neck! Nobody, and I mean nobody, has the time to get through a day deriving each action from first principles.

Hans Rosling knows stuff, based on facts, that won’t fit your default rule book. For example how do explain the 2% HIV infection rate in Sudan v.s. the 27% rate in South Africa? Did you know that electricity is good for infant survival rates independent of income?

An evidence base world view requires five thoughts at the same time:
1. World is getting better and better,
2. but at the cost of climate change,
3. and billions still live miserable lives in poverty
4. and in the last decade life got worse for 100 of millions,
5. but as the world is stupidly managed, we have many opportunities to fix the world for the grandkids!

Go forth, read and subscribe.

If you visit my blog you might notice that at the bottom of each posting there is a link for reporting typos. Since I installed that I get about one report for each posting I make, and about one a week from some random visit to an old posting. From time to time somebody reports two or three typos in a batch. I also get torrent of spam thru that UI, but that’s no big deal.

Recently some kind soul has been working submitting a lot of typos. Here’s the list of posting he (or she) has happened to report typos on so far.

• That was Quick
• Universal Demand for Quality
• Cones
• Town Cable Contract
• Who’s the Pitcher
• del.icio.us
• Stuff Addiction II
• Design Traps
• Building Houses on Sand
• What is Social Software
• Trustable Tagging Agents
• Just Push the Button!
• Intergeneration Income Patterns
• R scoping, assignment, confusion
• Imagine
• Love it or Leave it
• How to Get Rich

Who ever you are, thank you very much!

Back in the dark ages, before web sites had feeds, I’d visit the website of a Bay area consulting firm once or twice a month. It was a fun place were the nutty ideas of the sixty’s counter culture would get framed up for sale to moguls of silicon valley. Never mind that Apple sues Apple; it was all very chummy; these folks all went to school together during the 60s.

One of the meta-meme’s I captured from that venue was the trick of importing ideas from the natural world – e.g. ecology, biology, agriculture, sociology, evolution – into my own problem spaces – i.e. software architecture, project management, business architecture, etc. Though fraught with opportunities for error, this is an extremely fecund technique.

Here’s a nice example, pointed out by Sam of the technique.

Which brings our attention to the term apotheosis.

A mechanism by which one cell dies if it becomes severely mutated as a means of protecting the entire organism.

Traditional Apache HTTPD has a very very primitive example of apotheosis. There you have a swarm of child processes that handle the incoming HTTP requests. You can configure these children to commit suicide after N requests.

I have built systems where processes commit suicide for assorted reasons. Lack of customer is one. Memory bloat is a common one. It is often a lot easier to have them die than fix all the legacy problems that cause them to leak this or that resource. And then I have built systems where suicide and murder are used when various handshaking patterns fail. These are useful when you can’t control some of the components.

But to tell the truth I find the apotheosis idea much more interesting at the social network level. Since it connects to the whole suite of puzzles that are discussed in Group Thing, Exit, Voice, and Loyality, and go by various cliches: “Seppuku,” “Fall on Your Sword,” “Spend more time with my family.” And in turn reminds me of this marvelous quote.

To the untraveled, territory other than their own familiar heath is invariably fascinating. Next to love, it is the one thing which solaces and delights. Things new are too important to be neglected, and mind, which is a mere reflection of sensory impressions, succumbs to the flood of objects. Thus lovers are forgotten, sorrows laid aside, death hidden from view. There is a world of accumulated feeling back of the trite dramatic expression–“I am going away.” – Theodore Dreiser in “Sister Carrie”

This is the kind of blog posting that requires a disclaimer. No this is not about any current situation in any portion of my many lives. If it were I would not have posted it!

Consistently following the advice to “Buy cheap, sell dear” requires that you know something the market doesn’t. Consider a very simple example. Two towns one has abundant amounts of oysters, so abundant that the local population is sick of them; meanwhile in the second town oysters are a delicacy. A trader can make a good living moving oysters between the two; but only so long as he can keep his sources secret. The lack of transparency enables the profit; or more generally the information asymmetries between the two towns.

I was very amused by a story that appears in the early pages of Frank Partnoy’s book Infectious Greed. In this story a trader discovered way to trade between two markets; rather than moving oysters he moved information. The markets were currency markets; a private market and a public market. So in one market his actions were visible, i.e. they generated information, while in the second market they were largely invisible.

While all commerce, markets, etc. are, to a degree, about risk it’s particularly helpful at this point to introduce a point about betting. If you wish to place a $100 bet on your home team there are an infinite number of ways you might do that. For example you might place a $300 dollar bet on the home team and a compensating $200 bet on the opposing team. That may seem like an odd choice but notice that it allows you, with nearly total honesty, to go around telling everybody you bet $200 dollars on the opposing team. It lets you signal the opposite of your true intentions. That creates an information asymmetry, one that you control.

The public/private markets enable the same pattern. If you can trade in both a public and a private market for the same good; but only one of these trades will generate an signal about your intentions. If your trades are large enough you can move the market with that signal.

In the story the trader played this game with the international currency markets. He actually had two pairs of markets he could play this game in. First he had the traditional currency market and the currency options markets which were at that time not well connected. Secondly he had public exchanges and private, so called over the counter, deals he could make. There is a short paragraph in the midst of the story about how his boss got a call from New Zealand’s central banker demanding that he stop toying with their currency.

Because benefits can flow to market actors from information asymmetries most commercial dialog is permeated by a subtext of information hoarding. In some scenarios, like the one above, the appearance of an abundance of information might be a signal that information is scarce in an adjacent market. I think this is a rarely realized element of why open source appears so suspicious to some commercial observers.

Appearing in Tilly’s book “Why”:

Effective commons goverance is easier to achive when (i) the resources and the use of the resources by humans can be monitored, and the information can be verified and understood at relatively low cost (e.g. trees are easier to monitor fish, and lakes are easier to monitor than rivers); (ii) rates of change in resources, resource-user populations, technology, and economic and social conditions are moderate; (iii) communities maintain frequent face-to-face communication and dense social networks – sometimes called social capital – that increase the potential fo trust, allow people to express and see emotional reactions to distrust, and lower the cost of monitoring behavior and inducing rule compliance; (iv) outsiders can be excluded at relatively low cost from using the resources (new entrants add to the havested pressure and typically lack understanding of the rules); and (v) users support effective monitoring and rule enforcement.

That sentences is from “The Struggle to Govern the Commons” (pdf). Very interesting how poor the match is between that list and the situation with Open Source.

Excessive government regulation may prevent you from convinently selling your vote but recent developments in Redmond assure that you may more conviently sell your free speech rights. Should be good for the Herbalife folks (HLF).

Kieran Healy mentions at the end of this post about identity theft that the rumor has reached him that call center workers have a minted a new noun: irate; as in the contraction of irate customer.

I’ve spent much of my career dealing with post start up code bases. Along the way I’ve picked up a strategies for approaching the beasts. For example always pick apart the garbage collection, i.e. those portions of the system where they built the code to clean up afterward. They never get that right. For example how do they close accounts; particularly accounts where multiple parties are involved.

An even more productive strategy is to look at the exceptions; they are always a very generative area to dig into. Irates are a class of exceptions; the customers who fail to conform to the call center’s scripts. These folks reside in the the long tail of the cases the call center is expected to handle. The common cases all have well worn and efficient scripts. When these customers discover that the center’s scripts don’t resolve their problems they either give up and move on, or they keep pushing. Their persistence gets them labeled irate. Calling them exceptions might be a better choice; but in point of fact the call center wouldn’t mine “firing those customers.”

Long time ago, when bank machines first came on the scene, I once asked the teller if the bank machine meant that the only people who still came to her desk were the incompetent customers. I suspect that call centers have a number of neologisms for labeling the stream of customers they deal with; since to a degree all these customers are exceptions. Interesting place for a case study on tagging; since I bet their software includes the tools to label a customer: malingerer, idiot, irate, etc. E.g. all the words used to label a person as failing to conform to some norm.

I’ve been thinking a lot recently about a dialectic I first encountered explicitly named in the semantic web community: open v.s. closed world. It is, apparently, a term stolen from the world of logic and automated proof systems; but that doesn’t actually interest me about it. Inside the closed world you can achieve the illusion of being in control. All you data will be will formed and fully populated. The exceptions won’t happen. The data will respect your designs. When these things turn out not to be true you diagnositc gaze can move backward in time to seeking the place where things went astray.

Most of my rules for how to think about legacy software involve a generalization of an old rule I learned about hardware: “It’s always the connectors.” The slight generalization of that is that it’s all about the border of the closed world.

In contrast to the closed world are the worlds where you have progressively less and less ablity to predict the shape of the data. The cases to be handled become more and more diverse. You get a lot of that in real time control systems. No two pumps in the plant are the same, they were all bought at different times, they all have different quirks, and differing maintainance histories. All the components are irate, passive agressive, moody, and certainly they are all idiots. I’ve got a problem these days with my VOIP phone number; it’s taken 10 emails to slowly convince their customer support organization that I am none of those and that my problem demands handing off to more expensive labor.

The real world is open. The code and data in real world system should be expected to be mostly about the boundaries, the exceptions, the startup, the shutdowns. When things go wrong in these systems the you tend to look forward to how to bake in the handlers for this new case. This suggests that this is a fundimentally different kind of programming – an insight that makes me recall how some of the early languages back in the 1960 were structured so that each statement consisted of three parts: what to do, what to do next if that worked out, what do do next if it didn’t.

When one of the parties in a multi-player relationship announces the identity of a shared client has been stolen what does your software do? You, of course, are welcome to become irate when that happens.

One of the delights of growing up in New York and going to school in Pittsburg was access to a number of authentic ethnic enclaves. Little communities that hadn’t melted into the gruel of american culture, and which, better yet, had prospered. Boston, where I now live, has some ethnic enclaves too: Chinatown, the North End, etc. Of course, our most largest segregated community is black and poor. Most people though think of Boston as the home of those smaller segregated communities, those where we house displaced populations of high school students. We keep the first segregated with the shape of the public transportation system. The second problem is kept in control by carefully walling them off in the universities.

Emegrants into a new land always try to reproduce their native homeland. I’ve read that the Spanish methodically leveled the tropical ecology in Mexico until they had reproduced the arid plains of Spain. At regular intervals some new emigrant to my town opens a small grocery store selling an assortement of foods identical that of an analagous store back in his homeland. A trunk from New York or Montreal visits once a weak so he can restock. I love these stores, but sadly they are apparently incompatible with the local social and economic climate and six to eighteen months later they close down. If we are lucky they evolve quickly into something else.

But really! I am rolling on the floor laughing to see what the lawyers at the Berkman center did when they got off the boat on the shores of the new virtual world!