Monthly Archives: August 2003

Process vs Stupidity

Clay Shriky writes a delightful rant on the role of process as response to stupidity. …”Process is an embedded reaction to prior stupidity. When I was CTO of a web design firm, I noticed in staff meetings that we only ever talked about process when we were avoiding talking about people. ‘We need a process to ensure that the client does not get half-finished design sketches’ is code for ‘Greg fucked up.'”

He is wrong.

Healthy communities are full of diverse people of radically different skills. The
challenge in making a community that functions well is creating something out
of those talents that is closer to the maximum over the diverse talents rather
then the maximum of their lack of skills.

Did Greg fuck up? Maybe Greg’s the only guy in the group that actually ever
gets anything done. Maybe everybody knows that. Maybe the group is seeking a way to keep empower that talent while assuring that the occational errors that Greg makes stand a better chance of getting caught before they go out the door. So the community seeks a process.

Open source is full of processes to solving this problem – the problem of how
to get the max(talent) rather than the max(idiocy). We have numerious devices that are “biased-to-action” but enable the many eyes of different talents to keep a bit-o-quality-assurance on the risk implicit in that.

What pisses me off about Clay’s note is that he’s playing to people’s most base instincts. First he’s encouraging people to assume that process is a reaction to other people’s stupidity. That’s kind of thinking is toxic to community; it encourages people to label others rather than strive to find more functional processes.

Secondly he’s encouraging people to assume whenever they see a process to leap to the conclusion that was in reaction to some idiot’s incompetence. That reminds me of Mao’s advice to begin each negotiation by calling your opponent a “running dog.” Again it’s toxic to the community. What do you gain by accusing the people that crafted the process of “not trusting people.” I suspect that they were thinking that they had discovered a clever way to get the max of talent and speed in the face of a background noise of inaction, errors, and varying ablities.

This kind of thing spins people off your the merry-go-round; if your going to do this you better have some pretty delightful ponys on the ride to compensate.

None of this is to say that institutions don’t accumulate vistigal organs who’s cost totally overwhelms their benefit. I suspect that once Greg, aka mr. action man, leaves your probably need a process that adds a little action and a little less process that’s focus’d on quality assurance.

This is why it’s a healthy thing to keep at hand some suspision of vested interests, and institutional decay. But it is also health to have some respect for effective institutions, processes, and standards. A good dose of doubt is always best.

OK – now that I’ve gotten that rant out of my system.

(There was an only very marginally unrelated second half to this posting which I finally chopped off and posted separately.)

Fellow Travelers

Tim O’Reilly tries to figure out where open source and the internet are going. “I think there’s a paradigm shift going on right now, and it’s really around both open source and the Internet, and it’s not entirely clear which one is the passenger, but at least they are fellow travellers.”

I agree with much of what he says, but I’m not so sure I think it’s a paradigm shift. Same elephant from a different perspective.


  – thanks Kimbo

Feeding the Link Parasites is a Sin

forest.jpg
Your invisible if your art doll site doesn’t have anybody linking to it!

I know! Let’s make some links! Hmm… blog comments?

We have a problem here. Blog comments are a platform for link parasites just as Microsoft Outlook as a platform for spammers.

That is a problem with the current architecture of the blogging universe.

By virtue of how search engines work web sites accumulate quality ratings from the incoming links they attract. Meanwhile they accumulate reputation by their content, and the out-bound links they create. That reputation gives weight to the links.

Designers should accept some responsibility for creating systems that nurture this. Their designs should help to create good links. At a minimum they should not encourage the creation of links by bad actors.

Good people making lots of good links is a public good. Delightfully it also benefits both the source and the destination of the link. It creates a tiny bit of reputation for the source and a tiny bit of quality rating for the destination.

What’s excellent about the blogging ecology is how it has helped to generate a huge increase in the number of the best kind of links. Links generated by good actors; links that raise the reputation of the source site and raise the quality ranking of the links destination.

Link parasites create links that aim to aid only one side of the link, and manufactured blog comment links tend to drag down the reputation of their hosting blog.

Hacking the search engines with manufactured links is nothing new. Political parities, activist groups, marketing firms, and artists all do it all the time both in the real and the virtual world.

Should one of these link hackers chooses to manufacture a thousand links from art doll blogs to my site, hence slandering my site as being a high quality art doll site, then there isn’t much I can do about that.

But I can complain to the art doll blog
owners, and in turn I can complain about the blog authoring tools that enabled it.

The blog comment mechanisms are a dish of agar for bad actors to manufacture bad links the same way that Microsoft’s mail programs. Just as Microsoft Outlook is a platform for mail virus the blog comment system is a platform for link parasites. That’s a sin!

The good news is that link parasites damage the reputation of the hosting site. Good news? Yes, because it creates an incentive to get the problem resolved. Bad links mislead the search engines. They make the comment pages almost impossible to assign a usable reputation to, and that bleds over to the rest of the site.

The site owner desires a means to protect his reputation and the search engine wants a hint how to treat the links it finds.

A simple solution is to mark the link using attributes in the link. “Please consider this link to be the responsibility of an unknown third party. Your’s sincerely: site owner”

A site authoring tool that fails to do this is doing a disservice both to the public good of the web and to the reputation of the site author. Bad tool!

It has long been a fantasy by hypertext geeks that links would have bundles of meta-data on them. Today you can annotate a link to indicate that following it will take you to the “next page” and most browsers have a keystroke equivalent that will follow that link. This is rarely used for the usual reasons a standard fails to get adopted, i.e. the chicken and egg problem. Chicken: why learn the next-page keystroke if nobody annotates their pages. Egg: why bother to annotate if nobody knows the keystroke.

It seems hopeful that both the search engine and the blog authoring tool have their incentives line up. If they both adopt the standard then the link parasites will have to find someplace else to play their games.

Additionally it appears that we have some hope that mechanism to help is already there, say by adding something like author=”unknown-3rd-party” to the links in comments.

I think it’s neat that while historically putting meta-data onto links hasn’t created much return on the investment in this situation the benefit flows right back to the site author. Now he can defend his reputation and the authoring tools gets to avoid being bad.

If this was a discussion at an international industrial standards body then we would call this annotation a ‘pedigree’ and we would want to use something like SAML to create the assertion. That starts to drag us into the whole identity rat hole, among others.

So while we wait for those guys to get back from their meetings maybe we could just start putting author=”unknown” into blog comment links. If a handful of the big blogging tools and one or two of the search engine leaders indicated that they would get with the program the problem would be solved.

This problem goes by many names in the real world: astroturf, whisper campaign, etc. and you can hire firms to dis-intermediate the bad acting for you. Of course for others it’s just called mobilizing your base.

So! Anybody who’s got this far I want to encourage you to link to this art doll site. It will accrue to your reputation, I’m sure!

Network Maintainance

I’ve been thinking a bit recently about how accounting principles just don’t seem to work when applied to social network, and possibly not even to networks in general.

For example over at unfogged.com Ogged writes a nice little post where in he declines to take too seriously a bit-o-publish/perish research showing that women are generally less effective negotiators than men.

Meanwhile over at Crooked Timber we find Chris talking about the thousands of deaths during the recent heat-wave in France. He quotes Eric Klinenberg’s fascinating work on an analogous situation that happened in Chicago a few years back. Apparently in that disaster men tended to die in much greater numbers than women, in-spite of the data’s suggestion that pool of women was both larger and more vulnerable. Why? Because women have stronger social networks.

Clearly these two are related. You can optimize for the easy to measure short term benefit of optimal negotiated outcome, or you can optimize for a strong social network – which will deliver value in the long run. Of course that’s not black and white.

Finally I’ve been nurturing a model of why the electric grid in the Northeastern US fell apart the other day. A model that works from the assumption that the guys running elements of the grid all knew that there was good chance that it would collapse and were running the business on that basis – creating a feedback loop which accelerates the chance of a collapse. Each owner of an element in the network keeps lowering his trigger points for when he will exit the system because he’s thinking that when the collapse happens he better protect his capital investment. That makes the system increasingly susceptible to cascading failures.

There is another side to the story which is nicely summed up in this article by John Kay. The scenario he outlines involves the complex trade-off a firm makes between short-term income and long-term capital investments. Regulation and standard (ethics, professional standards, inspection regimes, etc.) all work to temper an a cyclic instability in that game.

Consider two identical power generators A and B are selling to the same customers in a competitive market. To first order their costs are identical. Mr. A comes in one morning and discovers that Mr. B is selling power for less than he is. Puzzled he drives by Mr. B’s plant and discovers he’s stopped trimming the trees under his power lines and repairing the roof. Incompetent he thinks. When he get’s back to the plant the next day the accountant comes into his office and explains that due to lost revenue they need to borrow money to meet payroll. The next day Mr. A fires the crew that trims the trees under the power lines and lowers his prices.

Both Mr. A and Mr. B are now caught in a rush to the bottom. If one of them can survive and the other one doesn’t the winner will be sitting pretty. The one with the most stored up fat in their business will survive.

While, there is plenty that’s different between a collapsing social network; and the collapsing electrical grid. In fact all the networks have important subtle differences. For example the triggers in the electrical grid are obviously very sharp, while social networks and the Internets packet networks are breakdown more gracefully, so the nature of the cascade failures are very different. Some networks are more brittle than others.

Another difference is the nature of what happens when the network does fail. Some network communities have what you might call social will to rebuild while others don’t. I suspect that the social network in authoritarian nations have a lot more trouble rebuilding than those in more liberal nations.

The players in the industry around the electric grid probably assume that after the collapse political will emerge to assure a stronger regulator hand and that will allow them to turn again toward investing in maintenance, capital equipment, talented staff, etc.

This faith that the social contract will save us is a beautiful thing. That people believe these things will heal themselves testifies to the faith people have in the social contract. That collectively people will work to create standards of behavior that serve the public good.

Of course the players might be wrong. For example in this article here we see President Bush deciding that upgrading standards for the distribution grid is really not that urgent a priority. Presumably that provides more time for the guys to get together and negotiate an optimal outcome. Meanwhile the old ladies always have their social networks.

Some people don’t believe in the public good, the social contract, need to regulate the networks, the markets, etc… Men!

Mailing List Metrics

I don’t know how many times I’ve watched somebody put up a slide showing how the distribution of posts in a mailing list, or commits to source code, or what ever are power-law distributed. So it’s nice to see somebody trying to drag some other statistics out of the posting data.

Who’s a regular? Who’s a flamer? Who’s an information provider?

A person with a high posts/thread ratio would seem to be a conversationalist. A person that tends to respond, but rarely post would seem to be a reactor – maybe even a flamer. A person who starts threads but rarely responds in threads would might well be a spammer.

You can read more in this news article about Marc Smith, Sociologist at Microsoft which appears to be connected to the PR around the release of NetScan .

Netscan’s pretty neat too. It shows that alt.sci.sociology‘s near neighbors include alt.binaries.pictures.erotica.high-heels and alt.comedy.standup. Who knew?

Simple Measures Kill Diversity

Power-law distributions emerge when new nodes attach to the existing network with some preference for the current winners. You can increase the severity of the distribution by increasing the preferential attachment, and you can temper it by decreasing it.

For example to increase the severity the trick is to encourage the new comers in the belief that current leaders are a good proxy for their quality measures – it’s a simple message: ‘Go with the winner!’ To decrease the severity you need to provide to new commers with a more complex message: “Pick what’s best for you, here’s a bucket of information to help you do just that.”

Improving the information available to newly arriving members of the network, providing them with more complex subjective information, enables them to make their decision more rationally. That will make for a more diverse network and a less severe power-law distribution.

A system like Epinions.com manages this by creating a bit of a feed back loop that allows existing members to report to new members what they now know. Word of mouth can also help, since a new comer

Information & Power-laws

The simplest model of how a power-law distribution emerges presumes both urgency and ignorance. New players enter the network with an urgent need to connect and no time to evaluate their choices. To solve this problem they thrash around for something to substitute for knowledge, a proxy metric. If the proxy they pick leads to their preferentially attaching to already popular nodes the power-law emerges.

This is why advertising can be so powerful in an emerging market.

Bearing that in mind this posting from Clay is amusing. Nightmares are interupting the sleep of marketing executives at major motion picture studios! They wake up in a sweat imagining
that the audience at the first showing of BogusTeenMutantSpacemen IV are
fidgiting in their seats sending instant messages on their phones “mv scks! run awy!”

Always nice to see the vested interests disrupted by the increased ablity of the unwashed masses to coordinate their activities. But…

My nightmare is that this will mean they have to carefully craft movies so they
will get high ratings from the kind of people who watch the first screening at
4:30pm on a Friday afternoon and think that it’s ok to use your phone to IM during the movie. Teenagers!