Ping/Poll why they don’t scale

Off an on for the last few months I’ve been playing with the problem of how web sites notify their reader producers-exchange-consumerswhen they have fresh content. This is sometimes called the Ping problem. Sometimes it’s called push, to contrast it with the typical way that content is discovered on the net i.e. pulling it from it’s source. The problem interests me for an assortment of reasons, not the least of which is that it’s a two sided network effect. On the one side you have writers and on the other side you have readers. I draw a drawing like the one on the right for situations like that. Two fluffy clouds for the producers and the consumers connected thru some sort of exchange.

The design problem that I find fascinating is the exchange. For example you could build a single hub, or a peer to peer system, or possibly something based on standards. Lots of choices both technical, social, and economic.

The set of producers has some very exceptional members. For example if you limit the problem to just blogs then Blogger and Typepad together probably account for a huge slice of the pie. Similarly some consumers have a huge appetite for content. Google, Technorati, Feedster, etc. want to read everything! These two populations are skewed, i.e. their power-laws. I draw a picture like this to illustrate that.

The blogging example provides a nice case study for the more general problem. In that community we currently have two standard solutions to the problem. Both don’t scale. Ping, which let’s producers notify a handful of sites when they update, was originally designed to notify a single site (and now a handful of sites) of updates. While it serves the voracious readers well it doesn’t scale up to help the rest of the readers. So the rest of the readers use polling to check if sites have changed. It’s a curious counter point that polling is a pain in the neck for the elite producers; in effect it enlists the long tail of readers into a denial of service attack on them.
Illustration showing how Ping serves high-volume consumers while poll stresses high volume producers.

While it’s not hard to construct conspiracy theories when working on problems like this, it is also foolish to ignore that powerful winner take all games are in the room. The shape of the distributions creates that high stakes game. It’s critical that the design take the distributions and traffic patterns into account. The Ping/Poll design is that it’s elites on the consuming side benefit while the elites on the producer side suffer. I’d be amazed if that was intentional!

Fascinating problem.

9 thoughts on “Ping/Poll why they don’t scale

  1. Julien Couvreur

    Interesting analysis. Do you know of other problems that present similar characteristics?
    Why not have high traffic sites use push and high volume readers use pull?

  2. Elliot Metsger

    Sam – I love it. I’ve discovered your xml attractive nusiance (how else? Manufactured Serendipity :) ) and I’ve been reading non-stop for the past three days and eathing this stuff up.

  3. Pingback: Ascription is an Anathema to any Enthusiasm » Blog Archive » Firms in Space - Relationship Space

  4. Pingback: Ascription is an Anathema to any Enthusiasm » Blog Archive » Long Tail - Wrong Drawing

  5. Pingback: Ascription is an Anathema to any Enthusiasm » Blog Archive » Ping Crisis?

  6. Pingback: Ascription is an Anathema to any Enthusiasm » Blog Archive » Evernote

  7. Pingback: Ascription is an Anathema to any Enthusiasm › frackim

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>