This morning’s paper triggers the realization that the Broken Window’s Theory is the conservative backlash to Alice’s Restaurant.

“… And I, I walked over to the, to the bench there, and there is, Group W’s where they put you if you may not be moral enough to join the army after committing your special crime, and there was all kinds of mean nasty ugly looking people on the bench there. Mother rapers. Father stabbers. Father rapers! Father rapers sitting right there on the bench next to me! And they was mean and nasty and ugly and horrible crime-type guys sitting on the bench next to me. And the meanest, ugliest, nastiest one, the meanest father raper of them all, was coming over to me and he was mean ‘n’ ugly ‘n’ nasty ‘n’ horrible and all kind of things and he sat down next to me and said, “Kid, whad’ya get?” I said, “I didn’t get nothing, I had to pay $50 and pick up the garbage.” He said, “What were you arrested for, kid?” And I said, “Littering.” And they all moved away from me on the bench there, and the hairy eyeball and all kinds of mean nasty things, till I said, “And creating a nuisance.” And they all came back, shook my hand, and we had a great time on the bench, talkin about crime, mother stabbing, father raping, all kinds of groovy things that we was talking about on the bench…. “

I’m too proud of having collected this newspaper clipping:

StimulusWatch.Org is very thought provoking.  Some folks really had their act together, Brockton Ma, and Blue Island, IL are two random examples; while some places go unmentioned Lowell Ma for example (our fourth largest city).  Since I presume the Senators from Maine are now nearly infinitely powerful I’m suprised how little money Maine appears to have gotten.  $/citizen by various voting areas would be a fun.  Then it would be easier to see which cities got nothing.  They have a ranking for most discussed, but I think it would be fun to have a score for most stupid discussion – i.e. most coments per dollars spent created.  There are plenty of errors in the data.  For example double entries or projects that create 1 job for every five thousand dollars.

“My mistake is perhaps instructive, because it exemplifies some cognitive biases that bedevil all of us: wishful thinking, overvaluing the causal impact of character traits (and individual agency, generally) relative to structural tendencies, and an unfortunate tendency to take sides” — Books Do Furnish a Room

Why does the culture so love to overvalue individual agency relative to structural tendencies?

This is incompetent public relations.  It is childish blame shifting.  Google slandered every single web site on the planet, and where do they place the responsibility?

… What happened? Very simply, human error. Google flags search results with the message “This site may harm your computer” if the site is known to install malicious software in the background or otherwise surreptitiously. We do this to protect our users against visiting sites that could harm their computers. We maintain a list of such sites through both manual and automated methods. We work with a non-profit called StopBadware.org to come up with criteria for maintaining this list, and to provide simple processes for webmasters to remove their site from the list. … we push these updates in a staggered and rolling fashion

On a non-profit supplier, a human, and a nameless process.  That is bull.  Google is a large complex mind bogglingly rich and powerful institution.  That institution needs to take and act responsibility.  Distracting the public by attempts to shift the blame is dishonorable and unprofessional.  Now institutions don’t have honor.  I hope the PR department and the Vice President who signed that posting can do better.

I have a rule of thumb about what to do in a crisis.  Fight the fire, but do not every do the postmortem during the fire fighting.  After it’s over it is best if you can get somebody uninvolved to do the diagnosis and block out the lessons learned.  For example there is no way that the Vice President who signed the blog posting is likely to fully block out what’s so wrong with it.  It is unlikely that the people on either side of the vendor relationship were the bit-o-data was handed off that triggered the fire are likely to avoid finger pointing; which is frankly a stupid waste of breadth.  This single sentence: “Since we push these updates in a staggered and rolling fashion, the errors began appearing between 6:27 a.m. and 6:40 a.m. and began disappearing between 7:10 and 7:25 a.m., so the duration of the problem for any particular user was approximately 40 minutes.” suggests a huge amount of work is going to be required to preempt this kind of foul up in the future.”

None of that will happen if the firm embraces the dodge of blaming human error, low quality supplier, and blind process.  Good PR is not excuse making.  Google needs to wake up the fact that people are now afraid of them.  Cute dog ate my homework dodges don’t cut it.  People need to see both transparency and professionalism.

Wife and I are off to NYC.  Taking the bus down tommorrow (Ok, I’m an idiot since there is a snow storm predicted), and then returning late in the day on Saturday.  It would be fun to met up with people.  If anybody would like to get together; please get in touch!  There’s a Korean place on 36th near 5th that I love and hopefully we will have dinner on Friday night.  We are also hoping to get to see Dan Hurlin’s Disfarmer, tommorrow night if the God’s cooperate.  We have a foolish plan involving ferryboats!

I seem to be building logging infrastructure today.  I keep recalling one or another of the rules for playing this game.  Might as well try to put them down.

• Who? – The speaker’s unique ID and type should be in each log line.
• Transcript – The speaker’s utterances should have a serial number, so you can notice gaps.
• Checksum – A running check sum is a big help in proving things.
• When – The utterances should have a time stamp (daemontools multilog t is good)
• Synchronize our watches – NTP is a must everywhere.
• Breadcrumbs – Jobs/tasks/work-items/requests should have a unique ID that is threaded these across to the logs and across process/module/machines
• Health – All processes (machines, threads …) should emit a heart beat; heart beats should include some health indicators so other parties can notice when they expire or get sick
• Replay – Logs that enable a rebuild from last snapshot will save your butt.  Often your close and only some minor optimization (truncating output, discarding binary info, say) is preventing it.  I once rebuilt an entire source repository from years of mail to prove an intrusion had not touched the sources.
• Syntax – It’s good if the logs are well tokenized, i.e. embedded strings are escaped; and character encodings are worked out.
• Standardized – It’s good, but it’s hopeless. This is the worst case of the 2nd part of “”Be strict in what you send, but generous in what you receive”
• Innummerable – You can lay an ontology over the space of exceptions.  Accept that, and then proceed as usual.
• Now – The sooner the log analysis takes place the better.  Don’t wait until your patient is in intensive care.  Analogy: test driven development.
• Email – The accumulated headers in modern email are full of lessons learned
• FSEvents – the asynchronous file system journaling/notifications of (BeOS, et. al.) are worth looking at closely.
• Fast – I tend to embrace that writing the log is not transactional or even particularly reliable so I can have volume instead.

This is great, and my first reaction is yes, yes.

For many cultural and artistic movements there is a “counter-cultural moment” when a previously obscure and out-of-the-way tendency suddenly gains prominence. It may be the surrealists in the ’30s, Bauhaus in the ’20s, the hippy movement in the late ’60s, punk in the late ’70s – each had a few years in which it was both important and yet still opposed to the mainstream. But that counter-cultural moment is short, often just a handful of years, and once it is over, the movement either becomes the mainstream or fades away. Only nine years after the summer of ’67 the hippies were a thing of the past and punk set itself up in direct opposition to the pompous and irrelevant dinosaurs that “progressive rock” bands had become.

“Geek culture” has had its counter-cultural moment. It’s over.

It is now eight years today since Wikipedia started, four years since blogging, ten years of Google. These formerly “hungry upstarts” are now the establishment. Netflix partners with Wal-Mart, Google partners with CBS, Amazon is bigger than any of those “establishment” bookstores it challenged and pushes around publishers with impunity. Geek culture is now mainstream. Google, Amazon, Netflix have passed their counterculture sell-by date.

But yet again.  There are some distinctions to be teased out.  For example, some of what is “geek culture” is indistinquishable from the techno-scientific-industrial revolution; and that firestorm just keeps going and going.  Some of what is “geek culture” is the rise of groups (scale free) condensing in the network; and that just keeps happening.  And while the cultural bit is all true there it’s bigger than that – touching all the social sciences.

On balance, and I’ve felt for quite a while now,  we went thru a real phase change when the Internet went mainstream.  At the moment it transititioned it was a real oh-dear/oh-my-goodness moment.  We are on the otherside of that.  There is plenty of remaining turbulance yet to be played out.  Plenty of solid bits waiting to be be liquidated.  Some, like the newspaper industry or walmart, are plenty big.    But the big-bang appears to be behind us.

I’ve been meaning to write this up for the last few months but I wanted to get the code into a shape I liked first.  The way EC2 works you spin up fresh virtual machines by doing:  ec2-run-instance AMI-123 … and so the natural thing to do is if you need machines configured for different roles you take the time to stamp out a different AMI (aka Amazon Machine Image) for each role.  That’s tedious.    Alternately you can gin up an image which learns what role it should fill as it starts up.  Some folks already build their AMI to let you do that.  For example the kind folks at Alestic provide assorted AMI that include this feature: “On first boot, runs instance user-data script if it starts with #!”.

So here’s what I do.  First I arrange to have the custom personality packed up in what I call a start-up-bundle.  I put that someplace.  In the example below it’s at: <http://example.s3.amazonaws.com/starters/start-up-example>.  I prefer to have the file publicly available, so that fetching it trivial.  But since it’s content is private I encrypt the file.  In the example below the password for decrypting the file is “stirringdullroots”.

To start an machine up and have it take on this personality I then do: “ec2-run-instance ami-115db978 -f start-up-example-userdata …”

This is what you’d find in the file start-up-example-userdata, i.e. the script which grabs the starter bundle, decrypts it, and then recurs into the bundle to let the personality take shape.

#!/bin/bash
trap 'echo ERROR Bootstrapping failed; exit 1' ERR
echo "Bootstrapping from <http://example.s3.amazonaws.com/starters/start-up-example>"
set -o pipefail
cd /tmp
curl -s "http://example.s3.amazonaws.com/starters/start-up-example" | openssl enc -d -aes-256-cbc -k stirringdullroots | tar x
echo "Unpacked: `find start-up-bundle | wc -l` files"
cd start-up-bundle
echo "Run start-up scripts for various modules"
for i in */start-up.sh ; do
MODULE=`dirname $i`
echo "Starting module $MODULE: via $i"
(cd $MODULE; bash start-up.sh)
echo "Finished starting module $MODULE"
done

So each startup bundle has two parts the userdata script, and the encrypted lump-o-personality.

Of course there are scripts for building start-up-bundles and other orchustration of the scheme, but that’s the heart of the story.

As you can see each machine’s personality consists of a set of modules.  I have a collection of standard modules for things like java, perl, erlang, tinydns, rabbitmq, sbcl, etc. etc.  One nice feature of this design is that it should be possible to make and share these with other folks.

Economists will appreciate that anthropologists have noted that scarcity is one the reasons why old ladies sometimes transition from Nana, to  crone, to shunned, to  witch, and are then murdered. Food gets scarce and somebody’s got to go.  I see that Google let go “about a 100”  recruiters.

bAnd, I see that Google is setting some products aside. I’ll miss one of these: catalog search. It was fun, take a peek before it’s gone.

Those that live on the bottom lands, out on the long tail, suffer the most when the storms come or times get tough. Easier on the conscious if you label them as unworthy.  Of course declaring them witches is a bit over the top. There are other moves. For example you can set them free. Sort of like how polygamist societies shun their young men due to a shortage of wives you can always donate your weaker products to the open source community  (don’t forget the patents)

Some of those kids will thrive.  Sometimes they get adopted by a stronger family.  I wonder if the mashup editor team is getting folded into the Google App Engine team.  But then I also wonder about the health of Google App Engine.  I recently paged thru their example application catalog and it’s a lot thinner than I would have expected.  There was almost nothing in there that I thought: “Yeah!  I should blog this.”  That is  not good.

Kremlin watching.

Idiots.  Comcast upgraded their email system over night and now I have thousands of duplicate messages in dozens of mail folders.    And, so does everybody in my household.  Guess who get’s to clean up this mess?

The email announcing this wonder (Smartzone(tm) Communications Center) includes the phrase: “You Asked for It!”    It would have been more polite of course if they had given us all fair warning, before throwing this punch.  You know as in: “Ok buddy, you asked for it.”    This is has an abusive relationship.  Their product managers must be really really happy about their switching costs.