Ascription is an Anathema to any Enthusiasm

Exchange networks are full of trust and reliability issues. How do you know that the intermediaries aren’t modifying the content? How do you know that swarm members in a p2p distribution scheme aren’t freeloading?

You can tackle the freeloading problem with a reputation scheme. The participants in the swarm can report on how helpful their peers are. The system can create statistics on those reports. The statistic for a given peer become his reputation. In networks where peers have durable identities and participate over long periods this is probably sufficient to reduce freeloading to reasonable levels. Though you would still need generous participants to enable new participants to enter the swarms.

That participants can collaborate to keep an eye on each other. These models are statistical models of the behavior reports. Successful collaborative efforts demand a lot forgiveness and generosity, so for many measures it’s not a problem that the models are only estimates. What you sum up into the participants reputations will define what the participants aspire to. For example if you model how long a participant stays in the swarm and you can create incentives to stay longer.

The standard solution to middleman problems is to insist on validation end-to-end. That’s one reason the .torrent files in BitTorrent have checksums in them. An intermediary can’t modify the distributed file without the end user noticing.

I’m interested in how to use swarming approaches on streaming data; for example video or audio broadcast, blog pings, weather or other sensor networks. Harder to get an end to end solution for these streams. The broadcaster could sign every packet, I guess, but that would put significantly add to the costs.

Another possible approach is to assure that the checksums and the packets they validate travel by different and hard to predict routes. Participants can report on their peers adherence to a rule that checksum’s should never travel with their associated packets.

I’ve been reading the BitTorrent code. Which lead to discovering this nice tool for drawing python module dependency graphs.

The drawing shows the modules of a swarm member in the current beta version (4.1.2) of BitTorrent. In the middle, the brown tangle, is the new DHT (or distributed hash table). Using that a swarm implement their own tracker. The two purple bits on the lower right implement the http server for the tracker. The purple bits on the left include the BitTorrent encoder, or bencode, as well as configuration and argument parsing. The stuff at the top implements the swarm’s collaborative file exchange; e.g. is orchestrating the exchange of pieces over connections with peers at reasonably rates. Here is a pdf version of that drawing.

If you want to use the module drawing tool you really need to subclass it’s main class and fine tune what’s shown. That’s not hard but you probably have to climb the foot hills of both the graphviz and python learning curves. If you enjoy playing with graphviz, as I do, then you can edit’s it’s output before handing it off to the graph layout program. The Macintosh application that wraps up graphviz in a pretty Mac GUI is sweet. (Or, LOL, you could paypal me $998.35, email me your code and I’ll make one for you! Fourth of July sale, mention this ad before the 10th of July and get 10% off!)

The second data set I encountered with power-law behavior was while attempting to improve the performance of a huge software system, circa 1979. But I haven’t managed to collect an example of that kind for my collection so far.

Meanwhile this example shows power-law distributions in the inter-object references for in core data structures for a range of programs.

Back in the day when I’d read articles about GC algorithum design I never saw these graphs. Odd, don’t you think, designing a GC algorithm with out a model of this distribution?

via Rainer Wasserfuhr via his del.icio.us tagging.

I just wanted to post that.

The survey site doesn’t work in Safari, but it’s fine in Firefox. It also seems to think that I “posted” a lot of the links that WordPress bundled with my blog.

The nice thing about P2P content distribution systems is how they lower the barriers to entry for content producers. When things are working then the cost of distributing content to N consumers drops for the producer from N to 1, and the cost for the consumers rises from 1 to 2 (see here). In theory this enables content production to move much further down the long tail. It empowers the smallest players.

The design of these systems is a beautiful example of the design issues around a collaborative system. The consumers need to collaborate. If the total contributions of the consumers don’t amount to 2N then things fall apart. I’m finding it interesting to kick the tires on this problem. You can design systems to temper the freeloading by having the consumers accumulate a reputation. You can base the reputation on reports provided by other consumers. So if A provides content to B, then B can add to A’s reputation as a good actor in the system. If peer to peer exchanges happen in largely random patterns then A’s reputation will be assembled from a diffuse set of partners; making it harder to forge.

I assume it’s possible to design such a scheme. One that would allow peers in the system to know the contribution level of their partners with a reasonable degree of confidence. I haven’t looked very hard. I assume there are some papers on designing such diffuse reputation systems.

Ok, so I take it as a given that I can design a system where the participation demands a uniformity of contribution. But wait, I don’t want that! Look at the real world. Systems in the real world have multiple actors contributing to their total energy; and the distribution of their contributions is usually highly skewed. If the real world is that way, then is it a good idea to design P2P systems that effectively outlaw that distribution? What consequences would follow from that?

One thing’s clear. If you enforced uniformity you’d get class stratification. Participants of similar reputations would tend to flock together.

I’m reading a book about Common Interest Developments, i.e. the semi-walled garden highly homogenous communities created by developers. Their enthusiasts buy into a utopian fantasy. One who’s “overvalued idea” is the maintenance of property values. Other values tend to be displaced.

If we force uniformity of contribution into the architecture of P2P system, then that becomes it’s overvalued idea. Getting all fixated on the prevention of freeloading displaces other values? Why does the real world not work that way.

The delicate trick in getting the P2P reputation system may well be finding ways to encourage a diversity of participants. A means that it doesn’t lead to stratification, with that stratification and progressively tightening regulation of each group’s wall and internal norms. Very interesting tangle of problems.