Traffic on my sites is highly skewed. Hour by hour I don’t get much traffic; but just occationally a flash-crowd shows up at my door. The distribution would be power-law; except I truncate them. By the way; “truncate the flash-crowd” is a nice way to say the machine crashes. Don’t say “Damn the machine crashed!” instead you say “Oh boy! A flash crowd!”

I’d get around 50% more traffic if I could deal with the flash-crowds successfully. But I’m a cheap bastard and I’m not willing to buy a bigger pipe, a bigger web server, and spend the time to stress test my site regularly to assure it can deal with these very rare very huge bursts of traffic.

Last time the flash crowd showed up I rebooted the poor machine and used the Coral Distribution Network to solve the problem. Coral’s cool. It’s research project built on the planet net distributed computing research platform. So they let people use it for free. Which is very nice of them; though it makes me feel like a freeloader.

That all leads to two desires (or ideas).

First off I wish I had a means to configure my web server so it would automatically shed load to Coral when the flash crowd show up. I’d prefer not freeload during the 95% of the time when my site is doing it’s usual thing out on the heavy tail of the power-law distribution. This might be easy with one of the Apache modules that does load monitoring and shaping; but I haven’t found the right mix of stuff to make it easy for me to set up.

Secondly, it would be way cool to architect a means that sites with similar load distributions could form cooperative peering relationships and then when one of them get’s struck by a flash crowd the other members of the consortium would step up to deal. This one would make a sweet topic for somebodies doctoral thesis (particularly the design of the goverance and it’s enforcement). Or, the other kind of somebody could ‘just’ would make a weekend hack our of these two ideas.

Ok, I admit it, I’m trying to bait the lazy web here. I forecast sunny weather!