I note two other processes that grow a power-law distribution in Newman's survey paper. The first is a variation of the preferential attachment model. I think of that as a shopping model. New nodes shop for what to connect to. The first of these two models has a different method of shopping.
The alternate shopping model fixes a problem with the preferential attachment model. If you look at the simulation for the preferential growth model it works by drawing a random existing connection and them mimicking that. This is not really credible. The new nodes don't have a view of the set of all the existing connections to draw upon. Newly arriving nodes presumably can only see some local region of the network.
The alternate shopping model replaces the random draw with a bit of search. The new node starting from a random existing node proceeds to either continue searching (shopping) at a neighbor of that node or it stops shopping and mimics a connection on at that node. Each round of the search has some chance of terminating v.s. iterating. Like the preferential attachment model this shares a random draw from the universe of the whole graph. I'm particularly pleased because simulations of this model can easily be modified to include other attributes of the nodes the shopping visits - i.e. merit.
The second model might be called the acquisition model. Newman reports that if create a network by randomly adding edges between random nodes there comes a time when the network very quickly becomes entirely connected. They call that a phase transition. As we approach the phase transition the nodes form a slurry of components of various sizes. The distribution of these sizes is power-law. I think of that as an acquisition model because it mimics to a degree what happens in as an industry matures. At first there are numerous small entrants into the industry each solving local problems. Later as the industry becomes more standardized these firms begin to merge. This model helps to suggest why we see a power-law distribution of firm sizes in many industries. Industries that complete the phase transition become monopolies.
I particularly like this model because it helps to inform my thinking about what happens when one of the Porter's barriers to a industry consolidationis eliminated.
This is a wonderful paper. Your one stop shopping for all things network-science. Seventy five pages of overview! Dozens of pages of bibliography! Enjoy!
1 Introduction 1.1 Types of Networks 1.2 Other Resources 1.3 Outline of the Review 2 Networks in the Real World 2.1 Social Networks 2.2 Information Networks 2.3 Technological Networks 2.4 Biological Networks 3 Properties of Networks 3.1 The Small-World Effect 3.2 Transitivity or Clustering 3.3 Degree Distributions 3.3.1 Scale-Free Networks 3.3.2 Maximum Degree 3.4 Network Resilience 3.5 Mixing Patterns 3.6 Degree Correlations 3.7 Community Structure 3.8 Network Navigation 3.9 Other Network Properties 4 Random Graphs 4.1 Poisson Random Graphs 4.2 Generalized Random Graphs 4.2.1 The Configuration Model 4.2.2 Example: Power-Law Degree Distribution 4.2.3 Directed Graphs 4.2.4 Bipartite Graphs 4.2.5 Degree Correlations 5 Exponential Random Graphs and Markov Graphs 6 The Small-World Model 6.1 Clustering Coefficient 6.2 Degree Distribution 6.3 Average Path Length 7 Models of Network Growth 7.1 Price’s Model 7.2 The Model of Barabasi and Albert 7.3 Generalizations of the Model of Barabasi and Albert 7.4 Other Growth Models 7.5 Vertex Copying Models 8 Processes Taking Place on Networks 8.1 Percolation Theory and Network Resilience 8.2 Epidemiological Processes 8.2.1 The SIR Model 8.2.2 The SIS Model 8.3 Search on Networks 8.3.1 Exhaustive Network Search 8.3.2 Guided Network Search 8.3.3 Network Navigation 8.4 Phase Transitions on Networks 8.5 Other Processes on Networks 9 Summary and Directions for Future Research
In my last posting on power-laws (this posting) I illustrated a simple simulation. It grows a network by having each node connect to the network by randomly mimicking the connections made by so far in the network. That model's important because it gives rise to power-law distributions in the connection wealth of individual nodes. Important because it under cuts the argument that the success of a node is based on it's quality. It's not a totally unreasonable model for what happens in markets where the supply chain relationships are very sticky (i.e. we don't simulate yet any breaking of connections) and the new entrants select their supply chain associations based by modeling existing associations. Such markets aren't that exceptional in the wake of Moore's law and his friends.
In this posting I want to look at the advantage of being an early mover into such a market. Older nodes in such a networks tend to be richer, but how much more? The advantage is exponential because the advantages of early entry tend to multiple over time.
Older nodes grow wealth for two reasons. First since the game (the simulation if you prefer) lacks any means for a node to lose wealth the longer you play the greater you chance of getting randomly selected for new connections as the game proceeds. Each connections a node garners becomes an additional generator of luck in future rounds of the game. So connections caught early in the game keep returning value thru-out the game. Secondly the effect is multiplies. Secondly early players are situated in a less competitive game. They have a higher change in each round of winning.
Both of these effects compound so that we can expect that the advantages of getting into the network early to be exponential. An in this graph showing which round a node is born in and how many links it manages to accumulate illustrates that. The horizontal axis is linear. The vertical axis is log base two; thus nodes with only two connections get a score of one.
The roughly double expodential distribution of this advantage is clear; but since the fates (i.e. my random number generator) are indeed fickle getting on the bandwagon early doesn't assure success. One round of the exponential is absorbed in the log vertical scale while the other can be seen if you visualize a fitted line to that noisy data. As you can see, some of the early players gained no long term benefit. I often go to work for those companies.
This graph is only for one run of the simulation but other runs are similar. That graph ought to be a scatter plot, but my graphing tool only does line plots and I'm too lazy to bother to fix it at this point.
All this reminds me of a meeting many years ago where 30 or so very senior technologies sat in a room grilling the senior vice president of a famous desk top software company. The usual problem was a hand. The schedule was slipping, promises had been made, PR as embarrassed, and we had gathered to decide how draconian the cutting would be to get the damn product out.
The question: "Just how important is, time to market?"
The senior vice president allowed as in his experienced it had always appeared to very important, if not more important than all other aspects of the product. Some of us in the room found that to be a less than compelling answer.
He was right. I think we now know why. I'm a believer now!
It's tended to make less enthusiastic for traditional enthusiasm for process, elegance, care and more enthusiastic for a bias-for-action and a generosity with options that might reveal new networks of oportunity.
I believe the earliest paper that outlines the process that gives rise to a power-law distribution is Herb Simon's paper from the 1950s that explains that the distribution of words found in human language texts (pdf). He explained them by modeling the process of word selection in those texts by proposing that the next word is selected based on the popularity of the previous words that have been attached to the text.
Barbasi and Albert's revisit this model (pdf) in the context of growing a network. Their model can be illustrated by this simple function that builds a graph. As we will see the resulting graph will exhibit a power-law distribution in the node connection counts.
(defun make-pgraph (n)
(loop
with g = (make-initial-pgraph n)
finally (return g)
for i from 1 below n
as node1 = (find-prefered-node g)
as node2 = (find-prefered-node g)
do (add-node g node1 node2)))
This function grows a graph where each node adds two edges to the network. The function proceeds by creating an initial node, who's two edges are self connected. It then proceeds to add the remaining N-1 nodes with two edges connected to two nodes, node1 and node2. We select which node to connect to via find-prefered-node.
All the debates about what gives rise to power-law distributions arise from arguments about the behavior of the find-prefered-node function. Here's the one used here. It randomly picks one of the existing edges and then randomly selects the node at one end of that edge or the other.
(defmethod find-prefered-node ((g pgraph))
(let ((e (svref (edges g) (random (edge-count g)))))
(if (eq 0 (random 2))
(left-node e)
(right-node e))))
Here is a graph showing the wealth distribution of the nodes in five graphs grown by this code. Wealth is measured by the number of connections a node manages to aggregate.
I don't see much merit in that function that guides the preferential attachment.
More code below the fold.
This is ANSI standard Common Lisp.
(in-package "COMMON-LISP-USER")
(defclass pgraph ()
((nodes :type simple-vector :initarg :nodes :accessor nodes)
(edges :type simple-vector :initarg :edges :accessor edges)
(node-count :type fixnum :accessor node-count :initform 0)
(edge-count :type fixnum :accessor edge-count :initform 0)))
(defclass pgraph-element ()
((pgraph :type pgraph :initarg :pgraph :accessor pgraph)))
(defclass node (pgraph-element)
((edges :type list :initform () :accessor edges)
(edge-count :type fixnum :initform 0 :accessor edge-count)))
(defclass edge (pgraph-element)
((left-node :type node :initarg :left-node :accessor left-node)
(right-node :type node :initarg :right-node :accessor right-node)))
(defmethod add-node-1 ((g pgraph))
(let ((n (make-instance 'node :pgraph g)))
(setf (svref (nodes g) (node-count g)) n)
(incf (node-count g))
n))
(defmethod make-edge ((g pgraph) (left-node node) (right-node node))
(let ((e (make-instance 'edge
:pgraph g
:left-node left-node
:right-node right-node)))
(flet ((add-edge-to-node (n)
(push e (edges n))
(incf (edge-count n))))
(add-edge-to-node left-node)
(add-edge-to-node right-node))
(setf (svref (edges g) (edge-count g)) e)
(incf (edge-count g))
e))
(defmethod add-node ((g pgraph) (node1 node) (node2 node))
(let ((n (add-node-1 g)))
(make-edge g n node1)
(make-edge g n node2)
n))
(defun make-initial-pgraph (n)
(let* ((g (make-instance 'pgraph
:nodes (make-array n)
:edges (make-array (* 2 n))))
(n (add-node-1 g)))
(make-edge g n n)
(make-edge g n n)
g))
(defmethod find-prefered-node ((g pgraph))
(let ((e (svref (edges g) (random (edge-count g)))))
(if (eq 0 (random 2))
(left-node e)
(right-node e))))
(defun make-pgraph (n)
(loop
with g = (make-initial-pgraph n)
finally (return g)
for i from 1 below n
as node1 = (find-prefered-node g)
as node2 = (find-prefered-node g)
do (add-node g node1 node2)))
The graphing is done via this package which in turn assumes you are using one other package by the same author and Macintosh Common Lisp.
(defmethod graph-pgraph ((g pgraph))
(log-log-graph (sort (map 'list #'edge-count (nodes g)) #'>) "P-Graph"))
(defun log-log-graph (data &optional name)
(let* (max-x
max-y
(data-points (loop
for tentitive-x from 1
for pt in data
as x = (if (consp pt) (first pt) tentitive-x)
as y = (if (consp pt) (second pt) pt)
collect (list (log x 10) (log y 10)) into collected-data
maximize x into mx
maximize y into my
finally
(setf max-x mx
max-y my)
(return collected-data))))
(if name
(graph:graph data-points name)
(graph:graph data-points))
(flet ((make-tick-marks (max)
(nreverse
(loop named accumulate-ticks
with tick-marks = ()
for i = 1 then (* 10 i)
do
(labels ((finish ()
(return-from accumulate-ticks tick-marks))
(push-tick (tick)
(push (list (log tick 10) (format nil "~D" tick)) tick-marks))
(add-tick (tick)
(push-tick tick)
(when (>= tick max) (finish))))
(add-tick i)
(add-tick (* 2 i))
(add-tick (* 3 i))
(add-tick (* 5 i))
(add-tick (* 7 i)))))))
(apply #'graph:x-tick-marks (make-tick-marks max-x))
(apply #'graph:y-tick-marks (make-tick-marks max-y)))
(graph:grid-graph)))
(defun display-a-few (n m)
(graph-pgraph (make-pgraph m))
(loop for i from 1 below n
do
(graph:graph+
(loop for i from 1
for c in (sort (map 'list #'edge-count (nodes (make-pgraph 1000))) '>)
collect (list (log i 10) (log c 10))) "P-Graph")))
Joi Ito takes a stab at getting his head around what he thinks about the power-law puzzle. Dave Winer comments that he can't think about the issue because he's decided he doesn't like the messenger's credentials; the messenger in this case being Clay.
The power-law puzzle: We carefully designed a set of networks with the expectation that they would be highly egalitarian, we carefully avoided distribution bottlenecks, we were careful to commoditized the middleman; i.e. end-to-end networks. Much to our surprise when you look at the statistics of the communities that emerge on top of these platforms what you discover is power-law distributions. I.e. distributions of wealth that are frightenly analagous of the worst periods of human history.
For example, Dave Winer thought he was giving everybody their own radio broadcast station when he named his product Radio. Surprisingly the outcome was a lot of radio stations; but only a very small handful captured the vast majority of the listeners. Rather than a peer to peer outcome we got a centralized broadcast outcome.
The same story is true about web servers. We gave everybody the ablity to setup a web server at very low cost. Hundreds of millions took up the opportunity. Today the traffic distributions have settled into a extremely severe power-law curve.
The designers didn't see this comming. Those same designers need to understand what happen. They might want to consider adding some additional design principles to the next round of systems.
Sticky connections and lousy information are the two things give rise to a power-law distribution.
Lousy information: is the eye of the beholder. Vendors and sellers rarely agree. Newcommers to a network rarely have the right skills to evaluate their choices. Popularity of existing nodes is a poor proxy for real information about what to connect to.
Sticky, sometimes called loyality, is what limits the ablity for a late entrant to displace an early entrant. It's what gives the wealth time to adapt. Yahoo had to be asleep at the wheel for a very very long time to allow google to even have a chance at displacing them and inspite of that it hasn't happened. Microsoft could fumble the GUI ball for a decade and still Apple couldn't displace them.
Joi Ito run at this puzzle is to suggest that the power-law syndrome maybe a useful amplifier for good ideas. That's true, but only if the system tempers the above two tendencies and if the platform where ideas reside is egalitarian. None of those three is true. Ideas thrive, in a power-law sense, if they spread fast, if they are sticky, and if they can manage to leverage the powerful gatekeepers in the knowledge networks. Those knowledge networks might be the blogsphere, FCC guarded communication networks, the elite acedemic journal networks, or your community's social network - but you need to get past one of them.
There does appear to be another scenario where good ideas can leverage the power-law to aid their survival. That's by creating a new network, a new platform. Which remindes me of the old cliche about startups these either go public or get acquired by Microsoft.
That new platforms is another way to work around the powerlaw puzzle in a sense is what Joi is getting at when he writes: "All disruptive technologies and innovations break power law curves by exhibiting exceptional fitness."
I don't see that it has anything to do with "exceptional fitness." Creating a new landscape on which new enterprises rise up isn't about fitting a niche; it's about creating a new universe of niches. What grows first on newly exposed land is weeds. They are fit only in the sense that they spread fast. They persist if they manage to keep prevent other more interesting things from displacing them.
Lazy web query...
I'm looking for work about how members of a network build models of the network. In particular how do members of power-law distributed networks build their model of the network around them. For example if a member can sample the network randomly how long does it take for him to form an accurate model? What if his ablity to sample is limited the further he moves away from his home base in the network? My intuition says that the models built using either of these methods are amazingly poor; but I'm curious to read something that talks about this question more formally.
Over the last few years it has become clear that if you begin with a homogenous substrate, like the Internet with it's peer to peer architecture, what emerges is a network with a power-law distribution. I.e. there is an entirely natural mindless meritless mathematics that drives level playing fields toward terrible unequal outcomes.
In turn that makes it clear that thinking about the topology of the networks that a platform enable is the the key challenge facing designers in this generation. That if we want to maintain some degree of equality as the world captures the benefits of one vast uniform platform for communications and trade we are going to have to mighty careful.
One extreme topological form in this design space is the bridge, or two sided network. For example eBay that bridges from buyers to sellers (the marketplace was the old example of this). For example google that bridges from searchers to those that wish to be found (the yellow pages was the old example of this). For example platform vendors that bridge from diverse hardware to diverse applications.
The power-law distributions emerge because some players in the network become hubs; and a few of these hubs become very very large. I suspect that the bridge topology is only one way that a hub can function - it's a design that helps the hub manage both to scale and to remain durable. I yearn to see if there are others kinds hubs - for example is there one that is analagous to the train yard.
In anycase I was amuzed by this paragraph in this cool tool (yet another visualization of social networking googa).
LJNet puts social networking on crack - but this might actually have some negative side effects. My primary concern is worsening the potential social side-effects of 'bridge-collapsing' - allowing my friends to connect to other friends of mine without going through me as a bridge, a moderator, a contextualizer, etc can have some unpleasant side effects. Thus I'm considering a password-protected version of the LJNet system that only allows users to see their own networks, and to anonymize the 2nd degree friend nodes so that if I see that my friend has another friend with many interests in common with me, I'll have to ask my friend to meet this person. Of course this wouldn't be fool-proof, but this is my other concern before releasing it publicly.
Most people think the middleman is a somewhat dubious role; i.e. the whole problem of agency. But, what about the risks of "bridge collapse?"
There are only a very very few works I'm aware of that talk directly about how to control the slope of the power law curve. Clay Shirky's essay on inequality for example. Michael Porter's list of things that keep an industry fragmented is another. Both of those enumerate tools that create niches, barriers and membranes. Each one of those tools deserves it's own book - or at least a page in a wiki.
Rereading Clay's essay I notice that one of the techniques he mentions is to reduce the amount of heterogenity in the system; the example he gives is hierarchy - i.e. the millitary. I'm reminded that hierarchy is one of the common solutions to the coordination problem around collective action.
It's a cultural truism that centralized hierarchtical system often run amuck and become the prime source of abusive of power and severe inequality. The lession embedded in the power law story (i.e. appreciating that homogenous networks spontanously create power-law distributions) is: "Hold on homogenous creates hubs, and those hubs have power, and that power is just as susceptible to abuse as any centralized hierarchtical design.
Rereading Clays essay this week, this week, I'm struck by the way that firms often attempt to temper the problems hierarchy creates by creating multiple overlapping hierarchies; e.g. functional organization/proffesional hierarchies for that overlap project/product-line organization.
As a move in the game of systems design it provides a kind of check and balance. A means of auditing and oversight. Additionally it helps in creating safe niches for key activities. Inside those niches various kinds of skills, resources, public-goods can emerge.
I have model, totally ungrounded by the facts, that as the population grew the three spheres of commercial, civic, and religious activities broke apart. When people talk about the seperation of church and state they are speaking of that historical event - but just as critical was the way that commerce began to become distinct from the creating of public goods that is the work of the civic sphere.
Governance of such tangles of overlapping groups is a facinating mess. The coordination problems run deep; as they must if the groups are to remain distinctive.
A friend and I got to wondering yesterday if there are any examples of systems were tribe A elects the elites of tribe B and visa versa? You can see plenty of cases where a dominate tribe appoints the elites of a subordinate tribe. You can seem plenty of places where one tribe has numerous moves where it can check and balance the moves of other tribes.
It would be a very interesting university where tenure choices in each department were always made by other departments, for example.
Wouldn't it be simpler if we just standardized on one ranking scale?
No, it wouldn't.
Update: but wait there is more! I guess if your going to have empire your really get to know your assorted elites.
David Chess writes about categories.
Some categories exist. For example there really is something called an elephant. You can decide if X is an elephant. Elephant category exists because there is some continuity in the genetic thread that runs thru members of the species.
I think a lot of people presume that a similar continutity runs thru other systems where it doesn't. Other systems that appear to have continuty are for every undergoing frankinstein like hybridization. It's all very confusing.
Similarly lots of systems don't have any useful sharp boundries. There just isn't a useful sharp boundry between a blog, a newspaper, or a firm's updates to it's product catalog.
One of the aspects of the power-law discussion is how the elite catagory in a network create the impression of a catagory. For example are the top 5000 firms on the planet a good sample for informing your model of what a firm is? Would a sample of the bottom 5000 or the middle 5000 firms be better?
No.
A conversation with Brian reveals the interesting insight that many products emerge first as situated software; to use Clay's delightful term.
Consider for example MovableType. It emerged situated in the intersection of the set of people that could install Perl and the set of people who manifested a desire to rant (or reveal). For the longest time that community of people both limited it's growth but at the same time forgave it any number of short comings. Finally the folks who wrote it budded off TypePad which eliminated a the Perl expertise portion from the embedded situation.
If you want to get dragged into conversation about membrane design again then you can turn up the knob and rant about how that's another example of how public goods can be captured by private entities. But that's somebody else's rant.
All this is analogous to Clayton Christensen's rants about how disruptive products seem to always emerge serving unserved customers. Such customers are both forgiving of the products short comings and they provide a strong demand signal that helps to shape the product that emerges. Both of these are necessary preconditions for creating something new.
Oh no, there are now two dudes named Clay in this posting; what's up with that!
In any case.
Another point to make about situated software is this balance between a forgiving environment and a strong signal that helps the software to adapt.
The challenge in making a thing survive over time is getting it to adapt.
So among the benefits that a piece of situated software draws from it's environment is this adaptive advantage. More forgiveness. Better feedback.
One reason that real world software is so much harder is that it's unforgiving. One reason that Windows has thrived is that they have demanded a high level of forgiveness from their users. They get to do that because of the monopoly.
Economists should stop being so fixated on the pricing advantages a monopoly captures and turn their attention to the adaptive advantages. Oh wait, I'm getting dragged back into the membrane design discussion; I hate that discussion.
Clay's new essay about what he calls situated software is very important. He is saying something new about software. I don't think he's run the idea to ground yet, which makes it even more fun.
He says the idea arose out of observing how his students executed an assignment - build a web app useful to our local community. The key insight was that some of these applications leveraged the complement of that assignment - build a web app that makes use of the local community.
Clay is close to asking a question that I've been puzzling about in various forms for the last few years. If we embrace that group membranes are a good thing then what kinds of things are possible inside of the membrane? It is hard to get to this question since so much energy is devoted to worrying over the membranes.
Clay frames that question by introducing the idea that there is a class of software that we could call "situated software." Software that's inside the membrane. Software that is dependent on the social contract or physical reality it runs inside of. He contrasts this with what he names "web school" software; i.e. software designed to run in cold cruel world, the globalized, firewall free world. That other world is, of course, interesting because the huge audiences create the chance at minimum of drawing on a huge sample space of users, and at the maximum of spinning up huge network effects. But yeah! Ignore that. What if I want to create software that empowers huge numbers of groups.
You can reduce what Clay's saying in this essay; should you want to, by could mapping it into various conventional frameworks. Nothing wrong with that, as long as we take care to avoid killing the baby. Clay even does some of that in the tail end of the essay.
To some extent the idea is like personalization; i.e. software that adapts to it's users. To some extent it's like localization; i.e. software that adapts to the culture of it's users. To some extent it's like the things that we too casually repeat about the democratization of software development; i.e. that exciting things happen when users are empowered to author their own solutions.
I have gotten a degree of milage out of that last one. It gives you a long series of historical events you can pick apart. In each you can look at the social unfolding of what happens when software becomes more local. Lots of examples: minicomputers, PCs, spreadsheets, desktop publishing, drum machines, Hypercard.... Clay's essay makes me tempted to go back and think about each of them again as a "localization event," as enabling situated software to emerge. That's more inward looking, which is what I want, than my usual model for these, (i.e. undermine the ivory towers).
He talks about how one of the student applications, a market making app, could skip having to build the module we find in market-makers like eBay which try to reify the buyer/seller reputations. His student's applications could gloss over that because the local community's reputation system was already available. It was in the platform so to speak.
That reminded me of how often I've encountered in-house one of a kind systems with no training materials what so ever. You do occasionally hear complaints about the lack of training materials (or doc); but generally the social networks of the organization are entirely sufficient to make up for the lack of doc. In fact I'm reasonably confident that the lack of doc often turns out to be something of a positive for both the local social network's health and the applications in question. At minimum it makes adapting the applications easier. Documentation has a tendency to be like a coating of varnish making an application shinny and immovable.
Similarly he talks about another sample application where the problem of how to manage the dialog with your users is solved by physical presence. In what he calls "web school" or real internet application that problem bleeds into the entire can of worm around email preference management and privacy policies. Who in their right mind asks their bank to send them more email?
In the example he gives the students built kiosks put them in a public space, devices with minimal UI. Both the minimal UI and the use of public space are things I've noticed too. The first time I noticed it was with spreadsheets - an early success at software democratization. You would not believe the percentage of spreadsheets that contain no calculation what so ever; instead they live their lives as ritual artifacts placed on a table in a meeting. (information radiator, is a similar story).
Thanks Clay!
A couple notes on choice.
This is a nice review of a book I need to read on the excess of choice in modern life. One story from the review: many young people arrive in their thirties having failed to make any choice about their line of work. The modern world both does not constrain their choices. They are taught to value above all else a diverse portfolio of options, i.e. freedom. They haven't specialized. They have no depth of expertise.
Then on NPR I hear this story. This guy took an oath to surf every single day. He'd taken the oath last time there was a February 29th on a Sunday and he swore to continue until the next time it happened again. Seems like a reasonable work around for the problem of excessive choice.
This is one of my primary interests, i.e. the question of loyalty. Consistent behavior is one of ways we provide a model for others that they can react to. Consistent behavior is one of the ways we simplify the near infinite cognitive load of moving thru day to day life. Consistent behavior is one of the corner stones of durable collective action. Consistent behavior is what gives us culture, structures, standards. Consistency is the means we use to lower negotiation costs. It's just too bizarre to pretend that the collective society could be renegotiating the entire social contract every few moments.
So I'm more than a little surprised that apparently philosophers have settled into the presumption that to "honor sunk costs" is a fallacy. Seems to me that the philosophers have been become a bit too loyal to the catechism of the church of portfolio theory.
Finally I recall that when I visited Ireland one of the citizens cornered me wanting to know what I thought of the high divorce rate in the US. Much later I learned that the percentage of folks who are married in the US is substantially higher than that found in Ireland. (You may consider all this to be hearsay.) A fact that reminded me then and now of how the absence lack of a dominate church in the US seems to create a higher level of church membership than that found in countries with a more centralized church.
My take on all this is that clearly if you limit choice substantially (demanding very high loyalty) you get rigid cartoons of real groups. If you create extremely high levels of choice you get very transitory groups that never achieve any depth to their collective activities. It's one of those not to hot not to cold problems.
Bonus link. It appears that German has a word for Slovenly Peter.
Via Sam we find this site that is attempting to collect a set of fundimental information axioms. A very impressive set of people are involved. This enterprise is reminisant of the wonderful book Information Rules as well as a number of other efforts such as those mentioned here.
I'm finding it fun to treat these axioms as a list of homework assignments. I.e. "Critique axiom #N".
For example here is their first axiom:
Axiom 1 - Metcalf's Law
If there are n people in a network, and the value of the network to each of them is proportional to the number of other users, then the total value of the network (to all users) is proportional to n X (n-1) = n2 - n (Shapiro and Varian, 184).
Member aggregation is more important than the type or amount of resources owned (Hagel and Armstrong, 14).
Recognizing that a network's value might in fact rise O(n2) is another way of saying that there is a scale advantage to any system that has network nature; and it tries to give the reader some intuition about the magnitude of that advantage. N2 seems like a lot of advantage.
Most of use encounter scale advantages first on the production side of things. That a large car manufacture is more efficient than a small one because he can share certain fixed costs (R&D, administration, accounting, whatever) more widely. Of course that kind of scale advantage isn't N2; in fact it tends to drop off so that after a while the benefits that acrue from merging to huge automakers are pretty minimal.
What has come to bother me about Metcalf's law is that I suspect there are similar limits to it's power; limits that arise from the limited attention and limited flexiblity of the parties around the edge of the network. Each time you double the population on the edge of the network it will take me some time to find out that some of the new members should replace my current correspondents - since they are more valuable correspondents than my olde ones. The new participants are, in effect, a sample of the universe of all possible correspondents. Additionally is there an arguement along these lines: a some point the sample becomes large enough that new samples do little to increase the value of my total set of correspondents.
My second concern about Metcalf's law is revealed by Reed's Law - i.e. that the issue isn't how many correspondent pairs are enabled by the network but rather the number of groups that the network enables to form. That number is truely mind boggling large. So large that if these groups are actually able to form it suggests that the network is assured of triggering a total denial of service attack on our attention. A syndrome I'm sure some of use have noticed from time to time.
I find Reed's law a much more convincing model than Metcalf's. I find it a much more compeling model because I'm convinced that groups (i.e. graphs) are a more useful unit of conceptualization rather than links. Of course it too must have scale limiting syndromes that need to be understood.
My third problem with both Metcalf's law and Reed's law is the manner in which they gloss over the issue of what the hell is do you mean "value." Value doesn't exist in the abstract. Value only exists in terms of some constituency.
If a conversation takes place, intermediated by the network, between P1 and P2; the value created there may accrue to P1, P2 or it may accrue to any of the entities that deploy the network N1..Nn; or it may accrue to the groups G1..Gn that enclose those players. For the economist gazing down from his ivory tower it maybe all well and good to just say "the economy" nameing thus one of these groups; or maybe te union of these groups; but down in the trenchs these issues are central to problem at hand.
For example you can design a network that encourages value generation for different ones of those entities. It's not good enought to just say the network will have value and there fore we should go get one. You really have to make choices about how to encourage or channel the value that emerges.
In the end that's what's wrong with Metcalf's law; important as it is. It implies that the value arises in the pairs - rather than say the groups. The model of value colors how think about the network; and if you get it wrong you'll be surprised in unfortunate ways. For example I think Metcalf's law is about right for the telecom network, while Reed's law is closer to right for the Internet. That one key reason why the telecom industry is burnt toast.
But, not to put too fine a point on it and remembering this is Axiom #1 for God sake, I think that Metcalf's law get's the order of the overall value too low, and that Reed's law get's the order of the value too high. That the right model is one revealed in the power-law distribution that emerges from networks; and that model's controling terms are those revealed in the slope and bounding box around that curve. Somebody more mathematicly inclined than I will have to frame that in a manner that competes with Reed's law and Metcalf's law in on their own modeling terms.
Ah, that was fun ... if you can't rant in your own blog and all that...
I do love an ellipsis...
When I first started thinking deeply about network effects (i.e. things who's value rise in the number of participants) I was very pleased to realize that there are some wonderful small scale examples; i.e. pot luck dinners, flea markets. One of those is that bit of play ground equipment the turntable.
These days I'm more interested in how you get these network effect things to start, or how you create new turntables. How you draw off something from the resulting energy that assures they can be self sustaining, maintained.
So I just can't believe how wonderful this example is. Via Rebecca Blood; a turntable that taps the energy in children's play to pump water. What a hack! Listen to very affecting audio here. There are 500 of these in South Africa.
Gee wiz! boffens reveal that the little people create and the hubs in the power-law network just move those creations around!
Say it's not true!
Pretty soon they will discover that Google didn't author all those pages? The folks at Penguin didn't write those classics? eBay actually doesn't have _any_ of that cool junk their selling? Disney didn't invent those wonderful stories? The librarian didn't bring me those books?
The leaves of the network dig the potatoes, the hubs shuffle them around. Damn middlemen!
Tim Bray posts an interesting comment.
I was talking today to this really smart guy named Jonathan Leblang who works for A9, and he said “You know, Google’s success may conceal a death warrant.” I said “Huh?” He said “Well, the most useful Web pages used to be the ones that aggregated a bunch of useful links, and so people would point to those and Google would find them. Nowadays, why would anyone go to the work to put a page like that together if you can just rely on Google to find stuff?” Hadn’t thought about it that way.
Let me pick that apart because it's both deeply true, but in this case it's false.
First off it catches my interest because I'm a deep believer in the hypothisis that good things happen in the presence of a rich, deep, complex supply chain of stuff you can cobble together. That a house should be full of stuff, that a brain needs a lot of ideas in it, that a city is better than the country because it's full of all kinds of stuff. You could say I'm more into the fecundity of clutter than efficency.
One example of this struck me when I went to Paris the first time: the incredible diversity of food products in the markets. How could you possible create a culture of food without that wealth of ingredients? That a rich network of supply of a given kind creates a culture for that kind of activity. This is why cities tend to specialize.
The first half of the insight that Jonathan Leblang is noticing is that to solve a hard problem - like the "find what I'm looking for" problem that Googles is working to solve - you first have to accumulate a huge assortment of ideas for how to solve it. I'm confident that Google and eBay are in the Bay area because the rich culture of ideas, supplies, and other stuff needed to nurture firms of that kind.
Jonathan notes that Google depends on a particular kind of supply; i.e. the supply of high quality pages full of links on a given topic.
The second half of the insight, the bleak half, is that as you solve the problem sufficently then your reward will be to capture more and more of the users. You will aggregate all the demand and this will lead to starving out the supply. As the network effects begin to kick-in the users will start defaulting to your solution and alternatives will die out.
That is what's happened to food in the US. Industrial franchised food consumes so much of the demand for food that there remains only a very small niche for food that's not highly standardized. Eatting in the rural regions of the US can be an amazingly consistent dull experiance. You have to work hard to avoid it.
This is just another way to look at what happens as the slope of the power-law curve begins to steepen and standards fall into place. Or to recall the cliche good-enough drives out the better.
Now this is the interesting thing. I think this model doesn't apply in the case of Google. That in this case we are looking a very common kind of confusion about the nature of "stuff." Physical-stuff (grapefruits, car parts, computers) is different from information-stuff. The linkages of supply and demand are different. So the arguement he's making assumes that the supply of pages of aggregated links is somehow connected to the demand for such pages. That is just not true, or at least it's not true in the same naked darwinian scarcity of niche sense that it's true for the food vendors.
People are packrats; they create a supply of those pages that aggregate stuff about what ever they are interested in independent of the demand for those pages. So even if Google captures more and more of the traffic of folks looking for stuff; people will still make collections of info-stuff. The specialist blogging pages are a fine example of that. I don't write this blog for my audience, I write it for me. (At this point we could run off on a subplot that about Google acquiring blogger to incourage supply.)
That's entirely different than the food markets of Paris. If demand for passion fruit wains the supplier of passion fruit will too. But for info-stuff, a Martin Geddes recently said over at Telepocalypse: "No scarcity, no market, no problem."
That said, there is a scenario where info-stuff begins to look like physical-stuff; i.e. when it begins to spin up in a strong network effect. Thus Apache's httpd server (who's network effect of mindless adoption coupled with it's rich community of complementary products and services) does drive out other web servers. It drifts toward becoming a physical object because it's network effect has some physical qualities to it.
Open standards can help temper that effect; the HTTP standard does help avoid the entire web server market collapsing into a singularity. The vicious competitive attempts to make the HTML standard more proprietary go a long way toward explaining the near singularity that developed around Microsoft's Internet Explorer. If your attempting to create public goods around info-stuff - closed standards enable scarcity, market, problem.
Since the links in social networks are power-law distributed some people have mind numbingly large numbers of friends while most of us have many many fewer. Your much more likely to know these highly friendly people compared regular folks. Should you count the number of friends of your friends and compute an average you'll almost certain to discover that average is greater than the number of friends you have. So generally speaking your friends are just plan friendly than you are.
That presumably leads to everybody feeling as if they don't have as many friends as they ought to. Everybody's impression is that they are a wall flower.
This insight was gleaned from M. E. J. Newman's paper "Ego-centered networks and the ripple effect - or - Why all your friends are weird" found here.
I've never been to a rave and I don't really want to but I saw one in that horrible movie. Who's in charge? It is a question that comes up.
When people ask that about open source I like to tease them. I say it's kind of like what happens if you put a bunch of people together in any group. Pretty soon they all start spontanously dressing a like, using the same cliches, making fun of people outside. It's in the nature of things, like the rythmic clapping of an audience. I gather that rythmic clapping is somewhat more common in European audiences than American ones.
Standard behaviors can emerge entirely bottom up. Fact is, given the way that power-law network tend to emerge out of all kinds of unregulated linking up, I'm beginning to think it's more the rule than the exception.
It's actually kind of amazing the way a lot of open source projects seem to spontanously organize. You create a body of code; you add a half a dozen interested parties; after a bit they all start rattling around in something that approaches - just a bit - a synchronized manner. Who's in charge? The code repository?
In the fractal nature of these kinds of discussions I got to noticing this at three scales all at the same time. First you have the ongoing to-and-fro-ing about "what is a blog." Which is of course the conversation about what is the emerging standard blog. Second you have the question as it applies to an indivdual and his blog. There you might call it "finding your voice." Some people's voice is long tedious essays; others are tightly written humorous observations; some like to write little provocative Zen koans; while yet others have found a voice that consists of just revealing a stream of URLs they find interesting. Part of the tension in the "what is a blog" conversation arises from the way it does the violence of catagorization to individual voices. Who in charge? Where to you get off announcing that the annual christmas letter isn't a form of blogging.
Group blogs are an fun kind of intermediate level. It's the middle class! Sometimes easy going, tollerant, urban. Sometimes uptight, gated community, suburban. I'd not noticed the way that if you look at a few group blogs; like many-to-many or crooked timber you can clearly see that the particpants have begun to adopt a similar voice. Comming to them fresh you might assume that they gathered together because they shared a common voice; but if you read some of their individual writing from years past you notice that they had either wider ranges or even entirely different centers of mass. I suspect if you just measured the size of the postings you's see a kind of learning to clap in unison begins to emerge.
Over on the brand spanking new Planet Apache this process is particularly stark. First off we have a mess-o-people posting how have already developed a voice over the last few years. Second we are aggregating those entirely automaticly from their individual blogs; so to first order there is no reason to expect these voices to begin to standardize. Third the Apache communities I've particpated in have been particularly good at remaining both tolerant and diverse - which would suggest there is less social pressure toward a common voice. It's very much in the best interest of a healthy open source project to remain like that; otherwise you make it a lot harder to bring on new blood. But then on the otherhand the Apache communities I've particpated in have been extremely narrowly focused, very much communities of limited liablity. That's because they focus down onto the working code; often have no other scope. That suggests two things; that these folks have a strong expectation that planet apache will focus down on something, probably the work of the various projects, and secondly that these folks aren't fluent in what happens if you don't limit the liablity.
I have good friend who drew my attention to a behavior he calls "monkey see monkey do;" i.e. that we primates like to try things. We watch the other monkeys and then we go "Oh, that looks like fun. I think I'll try that!" And, it is fun. I certainly tried blogging for much those same reasons.
It will be interesting to see what happens at planet apache. Maybe we will wander into a common voice there. Maybe we will remain "just a bunch a guys." Maybe some people will choose to submit a partial feed of their own blog not because there are guidelines for the planet's content but because they decide they want to avoid the subtle temptation to conform any conventions that might begin to emerge there. Certainly some people like to settle into a framework. Who knows?
Hopefully we won't be tempted to answer the question: "Who's in charge!" Or was Mark Slemko once so wisely said: "That would be wrong, except when it's not."
Clay writes a excellent overview of some of the means for engineering the shape of the power-law curve. It's nice to see another voice talking about that problem rather than just railing about the meer existance of the curve.
While I have a few minor quibbles with Clay's posting I'm so pleased!. For example I'm pleased by his collection of examples of mechanisms people use to try and reshape the curve. A large collection of such mechanisms is key to informing a strong intuition about what you can do to fix extreme cases or inequality, failed growth, polarization, etc. One scale free network is not like another.
He misses the issue of how network design biases who will capture the new growth in the network - some designs encourage the emergence of a more egalitarian distribution.
He touches on the issue of volitility. This is a two edged sword; you want stablity - since network participants pay a cost to reshuffle the network - and you want moblity/oportunity. I believe, but I don't have enough data or a reasonably model that the distribution of volitility in most of these networks is similar to that found in the distribution of firm sizes from year to year. Small firms change size a _lot_ more than large firms - it's a double expodential. If that's the right distribution for the volitility then the design problem is to manage the constants in that distirbution.
He provocatively introduces the idea that conservatives have a tolerance for inequality. I think that's far too generous. I think that conservatives have an enthusiasm for inequality. That they believe that elite status is the rightous reward rather than a happen stance of system design. That a more severe slope to your power-law curve will drive people to increased striving and they are blind to the extent that a more egalitarian slope enables innovation, creation, diversity, and reduced social tension. Since what happens if you encourage diversity is the emergence of many many loosely joined power-law networks sorted out by different arts there is a deadly tendency of conservatives to encourage competititon between these arts that leads to a monotheistic world with a single dominate network and ranking.
That in turn brings me to the information issue. I wish Clay had mentioned that one way to reduce the slope of the curve is to improve the information available to the network members. That encourages members to link to things that are more diverse. I.e. the habit of linking to the "more popular blogs" is less egalitarian than the habit of linking to the "most popular blogs that discuss my interests." You can't do the latter if you don't have good information.
There is a meta issue, like the one about how many networks you think exist about the size of your network/community. If you regulate your network; by increasing the information, innovation, limiting it's upper or lower bounds, etc. etc. that implies you have drawn a boundry around it. That you have converted it from a public-good into a club-good. That you have given it a cell membrane. You can do that with pricing, certification, etc. etc. You can't ignore these membranes.
I was surprised that he doesn't use the word innovation, he has before, to talk about one of the goals of your network design.
One of the textbook examples of a network effect is the ATM network. The more ATMs in your network the more value your customers will perceive in opening a bank account with your bank. All else being equal customers will pick a bank with a larger network of free ATMs. For very small banks, those with only a hand full of offices, this became a serious competitive problem about a decade ago. To solve the problem these banks formed branded ATM networks. In my region, for example, a large number of very small banks formed a network known as SUM. The SUM network allows me to bank with a small bank but get the advantage of a large free ATM network.
So yesterday I stopped to get money and the ATM machine announced that the bank I was in had withdrawn from the SUM network. My first reaction to the bank’s withdrawal was ‘traitor.’ The question arises why would a bank decide to “go it alone.” The small banks are locked into the alliance in common cause against the large banks. Presumably Citizen’s Bank, the one that withdrew has decided it’s now a large bank and can compete own it’s own. I presume that the costs of participating in the alliance of small banks have now risen above the benefits of remaining in the alliance.
This is, of course, the same question as why California or New York remains part of the federal union. California, like all the industrialized states, sends more resources into the federal union than it get’s back.
This is also much like the problem that faces the very rich. Since they can “go it alone” they have less incentive to continue to support the common cause of public schools, public transportation, public recreation, public policing. As they come to have their own schools, transportation, recreation, police the incentive to contribute to the commonwealth declines.
The choice to “go it alone” does not arise because one is rich in absolute terms, but because one is rich in relative terms. For example, they have very luxurious school buses in the town a friend of my resides in. In that down the citizens can maintain their commitment to public transportation and education because their wealth is relatively uniformly distributed. Common cause is much easier when the parties are homogenous.
Meanwhile I have a friend who lives in a large cooperative apartment building. They have trouble agreeing to spend the money to spruce up the lobby because some owners purchased their units at a time when real estate prices were extremely depressed (these folks are merely middleclass) while other people purchased their units when prices were extremely inflated (these folks are very upper-class). The result is that for year the lobby stagnated until a tenant showed up who could negotiate a solution to the resulting coordination problem, i.e. had political skills.
In the case of California, or the citizens of a town, or the residents of a Coop; leaving the alliance is hard. You can call that lack of mobility, sticky, or loyalty. In the bank example I find myself wondering how the SUM network’s governance documents were framed. Should they have been written to make it harder for Citizen’s Bank to exit?
Oh darn! I'm going to be out of town much of January. Well, I can read the papers I guess.
A good Paul Krugman article about the shifting distribution of wealth and more importantly a nice summary of how to shift it.
Suppose that you actually liked a caste society, and you were seeking ways to use your control of the government to further entrench the advantages of the haves against the have-nots. What would you do?
One thing you would definitely do is get rid of the estate tax, so that large fortunes can be passed on to the next generation. More broadly, you would seek to reduce tax rates both on corporate profits and on unearned income such as dividends and capital gains, so that those with large accumulated or inherited wealth could more easily accumulate even more. You'd also try to create tax shelters mainly useful for the rich. And more broadly still, you'd try to reduce tax rates on people with high incomes, shifting the burden to the payroll tax and other revenue sources that bear most heavily on people with lower incomes.
Meanwhile, on the spending side, you'd cut back on healthcare for the poor, on the quality of public education and on state aid for higher education. This would make it more difficult for people with low incomes to climb out of their difficulties and acquire the education essential to upward mobility in the modern economy.
And just to close off as many routes to upward mobility as possible, you'd do everything possible to break the power of unions, and you'd privatize government functions so that well-paid civil servants could be replaced with poorly paid private employees.
Variations on these are exactly the same as what you do if you want to increase the slope of market concentration or the dominance of a given standard.
For example if your trying to increase the concentration in a market you want to make it harder for the small players to get access to supply (that's analagous to cutting off health care to the poor). Or you might want to raise regulatory barriers such that strong players have a scale advantage in meeting those new requirements (standardized testing for example).
The article that Krugman cites refers to "Wal-Martization" as causal factor in the increasing disparity of wealth and the falling mobility between classes. That's exactly right, as Wal-Mart has consolidated the retailing industry they have created increasing barriers to the ablity of small firms to reach consumers. So, for example, when Wal-Mart requires RFID tags or EDI support from their suppliers they raise barriers that assure only larger suppliers can survive. It's a commercial twist on the old chestnut of excessive goverment regulation.
One aspect of the whole distribution of wealth debate that goes under reported is, I believe, a blindness to the effect on small businesses. I'd be very surprised if it weren't true that what ever is true about the distribution of individual wealth wasn't exactly true about the distribution of firm wealth. I just don't see why there would be a meaningful difference between these two kinds of economic entities - in a macro economic sense.
It is conventional wisdom in the scale free networking crowd that highly connected hubs are the key to assuring that everybody is only a few hops away from everybody else. These hubs, ranked by the connectivity, are also at the top of the power-law distribution that is characteristic of scale free networks.
Marketing people know this - if you can get the hubs to recommend your product then your golden. Epidemiologists know this - if you can vacinate the hubs you can slow the spread of the disease.
One aspect of that which I don't think is well appreciated is that if your a hub you need to have a strong immune system. It's easy for the marketing guys to target hubs, it's much harder for them to convince them. It strikes me as obvious that if being a highly social person is geneticly linked then it would be extremely advantagous - from a survial point of view - if that trait was linked to a strong immune system.
So I didn't find this report surprising. Shyness can be deadly. That holds that shy people have weaker immune systems. I'm peeved by the value judgement there, i.e. that shy is bad. Social is only one of many many attributes that people might specialize in.
Much the same way that a plant pays a high cost having thorns or an exagerated sexual display (in energy that might be expended on other goals for example) I very much doubt that having a strong immune system is a low cost trait for a animal to carry. Same thing for risk adversion.
Meanwhile, I think there some subtle trade off lurking here between attractive and social that I'm having trouble framing up just right. Warning insta-Theory! But I suspect that attractive people then to be shy.
A delightfully visual posting about how to visualise the substructure of the power-law of attention. It is a little misleading. I don't know that one could turn the gravity down low enough for any landscape to have escarpments as brutal as those in power-law land. But, it does help to shake loose people's blindness to the way that the power-law of attention is the sum of many many power-law of attention to X, Y and Z.
Seems reasonable that gravity would lead to perferential attachement and hence to a power-law distribution. Here we have the top ten bodies in the solar system plotted in rank order. As usual around here this is a log-log chart. The earth's mass is one on the Y axis.
I was reminded of Ivan Illich the other morning. One of his rants was to frame the problem of how to raise the average volocity of the Mexican population as two choices. Choice A: build a jet airport in Mexico city. Choice B: build a large number of small bridges all over the nation. It was clear which raised the maximum speed; but it was also clear the B raised the average speed substantially more. It's a good example of the trade-offs you make deciding to aid the power-law's elite vs. the long tail.
What triggered this reminding? The Chinese space shot. First I smiled because that's such an extreme case of choice A. Such a fine example of societies tendency to pointless monumentalism. Such an odd thing for a socialist state to engage in; not that anybody really harbors many illusions of the Chinese as particularly socialist anymore. But then I laughed out loud, because at least A and B acutally create linkages. Space shots don't go nowhere. That's really the extreme of monumentalism.
The "klaws" at Venture blog writes:
...John Maynard Keynes called a "beauty contest".
He named it after beauty contests that ran in newspapers of his day. A selection of women appeared in the paper, and the idea was to pick the prettiest one (nobody ever said Keynes was politically correct). The paper would award a prize to the people who picked the winner -- the one with the most votes. In other words, if you are trying to win the prize, you don't pick the one you think is prettiest -- you pick the one other people will pick ...
The venture business is driven by the same logic when setting valuations.
Nice. I like to find examples of simple systems with network effect. So that newspaper contest is a perfect example.
Another example of a game with network effect is the that bit of playground exquipment the turn-table. You can tell it has network effect because mothers stand on the periphery and say "somebodies' going to get hurt."
The newspaper contest example seems particularly relevant today when I'm surrounded by people who are being punished by their membership (loyality) to the community that goes by the name Red Soxs fans.Meanwhile, I guess, I should toss in this quote by Adam Smith:
Are you in adversity? Do not mourn in the darkness of solitude, do not regulate your sorrow according to the indulgent sympathy of your intimate friends; return, as soon as possible, to the day-light of the world and of society. Live with strangers, with those who know nothing, or care nothing about your misfortune; do not even shun the company of enemies; but give yourself the pleasure of mortifying their malignant joy, by making them feel how little you are affected by your calamity, and how much you are above it.
Difficult advice to take when the entire region is in a funk. I gleaned that from Daniel Drezner
Your invisible if your art doll site doesn't have anybody linking to it!
I know! Let's make some links! Hmm... blog comments?
We have a problem here. Blog comments are a platform for link parasites just as Microsoft Outlook as a platform for spammers.
That is a problem with the current architecture of the blogging universe.
By virtue of how search engines work web sites accumulate quality ratings from the incoming links they attract. Meanwhile they accumulate reputation by their content, and the out-bound links they create. That reputation gives weight to the links.
Designers should accept some responsibility for creating systems that nurture this. Their designs should help to create good links. At a minimum they should not encourage the creation of links by bad actors.
Good people making lots of good links is a public good. Delightfully it also benefits both the source and the destination of the link. It creates a tiny bit of reputation for the source and a tiny bit of quality rating for the destination.
What's excellent about the blogging ecology is how it has helped to generate a huge increase in the number of the best kind of links. Links generated by good actors; links that raise the reputation of the source site and raise the quality ranking of the links destination.
Link parasites create links that aim to aid only one side of the link, and manufactured blog comment links tend to drag down the reputation of their hosting blog.
Hacking the search engines with manufactured links is nothing new. Political parities, activist groups, marketing firms, and artists all do it all the time both in the real and the virtual world.
Should one of these link hackers chooses to manufacture a thousand links from art doll blogs to my site, hence slandering my site as being a high quality art doll site, then there isn't much I can do about that.
But I can complain to the art doll blog owners, and in turn I can complain about the blog authoring tools that enabled it.
The blog comment mechanisms are a dish of agar for bad actors to manufacture bad links the same way that Microsoft's mail programs. Just as Microsoft Outlook is a platform for mail virus the blog comment system is a platform for link parasites. That's a sin!
The good news is that link parasites damage the reputation of the hosting site. Good news? Yes, because it creates an incentive to get the problem resolved. Bad links mislead the search engines. They make the comment pages almost impossible to assign a usable reputation to, and that bleds over to the rest of the site.
The site owner desires a means to protect his reputation and the search engine wants a hint how to treat the links it finds.
A simple solution is to mark the link using attributes in the link. "Please consider this link to be the responsibility of an unknown third party. Your's sincerely: site owner"
A site authoring tool that fails to do this is doing a disservice both to the public good of the web and to the reputation of the site author. Bad tool!
It has long been a fantasy by hypertext geeks that links would have bundles of meta-data on them. Today you can annotate a link to indicate that following it will take you to the "next page" and most browsers have a keystroke equivalent that will follow that link. This is rarely used for the usual reasons a standard fails to get adopted, i.e. the chicken and egg problem. Chicken: why learn the next-page keystroke if nobody annotates their pages. Egg: why bother to annotate if nobody knows the keystroke.
It seems hopeful that both the search engine and the blog authoring tool have their incentives line up. If they both adopt the standard then the link parasites will have to find someplace else to play their games.
Additionally it appears that we have some hope that mechanism to help is already there, say by adding something like author="unknown-3rd-party" to the links in comments.
I think it's neat that while historically putting meta-data onto links hasn't created much return on the investment in this situation the benefit flows right back to the site author. Now he can defend his reputation and the authoring tools gets to avoid being bad.
If this was a discussion at an international industrial standards body then we would call this annotation a 'pedigree' and we would want to use something like SAML to create the assertion. That starts to drag us into the whole identity rat hole, among others.
So while we wait for those guys to get back from their meetings maybe we could just start putting author="unknown" into blog comment links. If a handful of the big blogging tools and one or two of the search engine leaders indicated that they would get with the program the problem would be solved.
This problem goes by many names in the real world: astroturf, whisper campaign, etc. and you can hire firms to dis-intermediate the bad acting for you. Of course for others it's just called mobilizing your base.
So! Anybody who's got this far I want to encourage you to link to this art doll site. It will accrue to your reputation, I'm sure!
I've been thinking a bit recently about how accounting principles just don't seem to work when applied to social network, and possibly not even to networks in general.
For example over at unfogged.com Ogged writes a nice little post where in he declines to take too seriously a bit-o-publish/perish research showing that women are generally less effective negotiators than men.
Meanwhile over at Crooked Timber we find Chris talking about the thousands of deaths during the recent heat-wave in France. He quotes Eric Klinenberg's fascinating work on an analogous situation that happened in Chicago a few years back. Apparently in that disaster men tended to die in much greater numbers than women, in-spite of the data's suggestion that pool of women was both larger and more vulnerable. Why? Because women have stronger social networks.
Clearly these two are related. You can optimize for the easy to measure short term benefit of optimal negotiated outcome, or you can optimize for a strong social network - which will deliver value in the long run. Of course that's not black and white.
Finally I've been nurturing a model of why the electric grid in the Northeastern US fell apart the other day. A model that works from the assumption that the guys running elements of the grid all knew that there was good chance that it would collapse and were running the business on that basis - creating a feedback loop which accelerates the chance of a collapse. Each owner of an element in the network keeps lowering his trigger points for when he will exit the system because he's thinking that when the collapse happens he better protect his capital investment. That makes the system increasingly susceptible to cascading failures.
There is another side to the story which is nicely summed up in this article by John Kay. The scenario he outlines involves the complex trade-off a firm makes between short-term income and long-term capital investments. Regulation and standard (ethics, professional standards, inspection regimes, etc.) all work to temper an a cyclic instability in that game.
Consider two identical power generators A and B are selling to the same customers in a competitive market. To first order their costs are identical. Mr. A comes in one morning and discovers that Mr. B is selling power for less than he is. Puzzled he drives by Mr. B's plant and discovers he's stopped trimming the trees under his power lines and repairing the roof. Incompetent he thinks. When he get's back to the plant the next day the accountant comes into his office and explains that due to lost revenue they need to borrow money to meet payroll. The next day Mr. A fires the crew that trims the trees under the power lines and lowers his prices.
Both Mr. A and Mr. B are now caught in a rush to the bottom. If one of them can survive and the other one doesn't the winner will be sitting pretty. The one with the most stored up fat in their business will survive.
While, there is plenty that's different between a collapsing social network; and the collapsing electrical grid. In fact all the networks have important subtle differences. For example the triggers in the electrical grid are obviously very sharp, while social networks and the Internets packet networks are breakdown more gracefully, so the nature of the cascade failures are very different. Some networks are more brittle than others.
Another difference is the nature of what happens when the network does fail. Some network communities have what you might call social will to rebuild while others don't. I suspect that the social network in authoritarian nations have a lot more trouble rebuilding than those in more liberal nations.
The players in the industry around the electric grid probably assume that after the collapse political will emerge to assure a stronger regulator hand and that will allow them to turn again toward investing in maintenance, capital equipment, talented staff, etc.
This faith that the social contract will save us is a beautiful thing. That people believe these things will heal themselves testifies to the faith people have in the social contract. That collectively people will work to create standards of behavior that serve the public good.
Of course the players might be wrong. For example in this article here we see President Bush deciding that upgrading standards for the distribution grid is really not that urgent a priority. Presumably that provides more time for the guys to get together and negotiate an optimal outcome. Meanwhile the old ladies always have their social networks.
Some people don't believe in the public good, the social contract, need to regulate the networks, the markets, etc... Men!
Power-law distributions emerge when new nodes attach to the existing network with some preference for the current winners. You can increase the severity of the distribution by increasing the preferential attachment, and you can temper it by decreasing it.
For example to increase the severity the trick is to encourage the new comers in the belief that current leaders are a good proxy for their quality measures - it's a simple message: 'Go with the winner!' To decrease the severity you need to provide to new commers with a more complex message: "Pick what's best for you, here's a bucket of information to help you do just that."
Improving the information available to newly arriving members of the network, providing them with more complex subjective information, enables them to make their decision more rationally. That will make for a more diverse network and a less severe power-law distribution.
A system like Epinions.com manages this by creating a bit of a feed back loop that allows existing members to report to new members what they now know. Word of mouth can also help, since a new comer’s acquaintances maybe better able to model the what qualities he's looking for - i.e. they bring local knowledge to the problem.
Rich information is what tempers the severity of the power-law. Simple quality metrics increase the severity.
The right slope on your power-law curve will depend on your goals. If you want it severe, i.e. a few highly standardized solutions that everybody uses, then you want to labor to get people to rendezvous around a single set of simple numeric quality measures. Price, speed, etc. If you want to encourage a lot of diversity then you want to encourage a large number of measures; preferably subjective measures.
Putting these ideas into practice consider the blogging universe. We have lots of sites that survey the ecology and will tell you who the leaders are. That's a simple measure, and a proxy for quality. If we wanted to get a better system for new comers to use to select which bogs to subscribe to we would need a much more complex description of the ecology. One that allowed the new comer to say I'm interested in - oh - winter mountain biking, mushroom collecting, and the economics of small businesses and it would present a collection of blogs for him to shop among. Google could do that.
Another example of these rules unfolding in practice is everything2.com (a wiki/encylopedia kind of thing) where members are encouraged to rate individual nodes. The word they selected for rating was "cool", that resulted in a everthing2 becoming a site with a lot of cool nodes; as versus "interesting" or "funny" or "wise" or "sexy" or "useful" ...
Simple measures kill diversity. But, I suspect, that like an invasive weed they they build out faster.
The simplest model of how a power-law distribution emerges presumes both urgency and ignorance. New players enter the network with an urgent need to connect and no time to evaluate their choices. To solve this problem they thrash around for something to substitute for knowledge, a proxy metric. If the proxy they pick leads to their preferentially attaching to already popular nodes the power-law emerges.
This is why advertising can be so powerful in an emerging market.
Bearing that in mind this posting from Clay is amusing. Nightmares are interupting the sleep of marketing executives at major motion picture studios! They wake up in a sweat imagining that the audience at the first showing of BogusTeenMutantSpacemen IV are fidgiting in their seats sending instant messages on their phones "mv scks! run awy!"
Always nice to see the vested interests disrupted by the increased ablity of the unwashed masses to coordinate their activities. But...
My nightmare is that this will mean they have to carefully craft movies so they will get high ratings from the kind of people who watch the first screening at 4:30pm on a Friday afternoon and think that it's ok to use your phone to IM during the movie. Teenagers!
I'm very much enjoying reading Chuq Von Roshpach chew his cud on the issues around running online groups. He is saying a lot of things that people who do this know. But which I don't think are written down much. This is the kind of social knowledge that Phil Agre complains that smart people are often sadly entirely unaware of.
For example I like this peice about a technique he's tried for drawing forth the folks on the long tail of the power-curve.
It reminded me of a story told to me by a friend about the community theater group that one lucky day managed have a famous actor offer to join the company. But, when it came to perform the play the famous actor's presense on the stage was so powerful that everybody else seemed to disappear into the back ground.
That's not good for you community.
I don't think I can recall any group I've ever been in were this kind of imbalance didn't begin to be a problem at some point. Where the slope on the power-law curve grew to severe. Sometimes that lead to the long tail just drying up and the group evaporates. Sometimes a way around the problem is found. I'm surprised how often it resolves by the sad device that the 'famous actor' get's increasingly peeved that people are not respecting his role as leader dude or he complains that they are falling further and further into the background of the stage set. Finally he resolves the problem by walking off in a huff.
The famous actor problem is yet more subtle when you have a cabal of famous actors at the head of your group.
Nice paper sketching an straight forward algorithum to layout a graph informed by the assumption that the graph's connectivity exhibits a power-law distribution. The general idea is to sort the graph into rank order and then layout the upper-classes first and pack the lower-classes in latter.
I love the way that a even a graph layout problem becomes a statement on strategy, social engineering, governance, etc. when your working on things like this.
"The highest layer possesses a small number of highly influential nodes. Consequently, it requires the highest temperature so that the nodes are forced to spread out from one another, and explore the global search space. This is essential as these highly connected nodes define the final structure of the graph."
A simple chart showing the income distribution and an another attempt to give a feel for it via by spreading it across the US (and on into space). These are great. It's hard to visualize power-law distributions it's even harder to think straight about how the evolve. Income distribution is always power-law distributed but it's more severe in some places and eras than others. Much society has come to admit that we must try to manage the interest rate we need to admit that we must manage this.
Clay Shirky chimes in on the backchannel discussion (i.e. just how rude is it that in modern meetings people are passing notes in quantity all the time with others both inside and outside the meeting?). He does his bit to move the thread along and rounds up one of the usual suspect, i.e. "Yo vested-interests! Get over it! Too late now. Cat's out of the bag. Deal." He throws in the usual story about a previous debate of this kind. The one about calculators in school.
I recall the calculator debate. I was in college. Some students could afford a potent calculator, most couldn't, I couldn't Some people had access to symbolic math software and others didn't. I did. Matlab over the internet could do some of my homework almost with nearly zero effort. My perception at the time was about fairness in the face of widely disparate computational power.
My son recently took a math SAT II test with an approved calculator that could do an awful lot of symbolic math. I wondered how many of the students knew that. How many had been prep'd on the calculator's capablities? How advantagous to the calculator vendor if he could get an entired generation skilled in his calculator's UI.
The oldest story of this kind - i.e. stories about out cries over the disruption to existing structures triggered by new technology - I know is about silent reading. Apparently there was a time when men only read outloud, or at least most of them did. About the time the printing press appeared the idea that you could read silently, not even moving your lips, spread across Europe. This created all all kinds of concern amoung those who felt that they were responsible for looking after the thoughts of the population.
That's of course all entangled in the story of the Protestant revolution. Boy is that a classic story of innovation leading to the displacement of vested interests (aka the hub, aka the intermediary) in that case the role of the chuch as the intermediary between man and God.
There is a maple tree in my backyard. Each year it drops a carpet of winged seeds on the ground. Tens of thousands of seeds, every year, for 30 to 60 years. One, just one, of those seeds needs to take for this tree to fulfill it's Darwinian mandate and create the next generation. One survives out of million? The ecologists call this a r-Selected strategy.
I, on the other hand, have only a few children, and unlike the maple tree I am expending vast resources on each one of them. The ecologists call this a K-Selected strategy.
Species with an r-Selected architecture (weeds, annuals, insect pests,bacteria) tend to be opportunistic. They spread fast. The "r" stands for resources. If storm clears a section of forest the nearby maple is ready to seed that clearing. These species quickly cover new territory and quickly compose the dead. They tend to fluctuate quickly with the weather.
Species with K-Selected strategy (coconuts, apples, birds, most mammals) expend more on each offspring. Their populations tend to be more constant and self regulating
I assume that some business models are more r-Selected, with others are more K-Selected. If the weather is good and new markets are emerging fast then one should invest in the r-Selected businesses. For the last hundred years technology has created new markets in quantity - a good time for r-Selected businesses. Ones were it's best to encourage a large population of lightly funded experiments. Now if you think that's easy I'd recommend a careful dissection of a maple seed. Conversely I'd assume that if your operating in a very mature market with volatile weather then larger enterprises more careful planning and husbanding of your resources if a better tactic.
All this give rise to my new cartoon of open source. That those tens of thousands of projects at source forge are like maple seeds. That Open Source is a species that has adopted a r-Selected strategy. It assumes that that the problem at hand is not husbanding ideas, but covering the fresh earth of possibilities. That the problem isn't finding options worth executing upon, but finding which of a ten thousand possible options will actually take root.
Now one thing that's curious about this is that if you look at large Open Source projects you might not see that. Those projects look more like whales. They tend to have a certain overhead where in their resources more carefully managed so the baby doesn't die.
Of course all this is in total contrast to closed source projects. Every one of those I've worked on has expended tremendous resources attempting to manage scarce resources. Those projects seem much more like K-Selected species, carefully hording their fatty reserves so they can survive the next round of lousy weather that the market or management blows down the hallway.
Finally I think this has something critical to say about what goes on as you move along the power-law curve. That the population that resides on the long tail tends to be more r-Selected (limited by resources and driven by opportunism) while the upper classes that reside in the towering heights tend to be more K-selected (durable in the face of bad weather, fat, enthusiastic about property rights that guard their inheritance.).
Tim Oren's blog is rising fast in my news reader, where I keep the blogs sorted in a rough rank order.
I highly recomend his recent essay No eBay of Blogs with which I largely agree.
But, on the otherhand... I think many people are missing out on how strong the synergies are between the client and the server in blogdom. That comments, trackbacks, identity, reputation, and moderating will all strengthen that entanglement. If that grows stronger it will be harder and harder to avoid a potent network effect emerging - one that appears to me to be much stronger to me than the document exchange network effect that makes the Microsoft Office so strong a monopoly. If you owned that, you'd get the hub, you'd be the eBay, you'd be the landlord. An embrace and extend strategy looks quite plausible here.
I don't know if that's any different an insight than saying the same thing about the client/server synergies around HTTP. Nobody grabbed that hub. Many tried though. The build out was so extremely fast, making hubs easier to grab since that reenforces the power-law. But the early winners were open, so at least the server side has managed to remain reasonably open. The client side remains in contest, but some poeple think the good fairy Ms. Open is starting to gain the upper hand there too.
One reason the landlord of HTTP was hard to grab is that powerlaws are more likely to emerge when new entrants lack the knowledge to make informed choices so they just pick the market leader. In the HTTP case folks installing servers tended to be highly knowledgable about the lock-in risk and hence valued open. I mention this because I don't think that's true about the majority of folks that will be adopting a blogging solution. If the client and server become one, as I suspect is likely, then that will further reenforce the emergance of a landlord of blogdom.
I'm a fan of a design pattern for software where in a very large, complex, rich data-type creates the meat of the application and then over that you pour a gravy of scripting.
In the most delightful manifestations of this pattern the languages carry a deep and complementary flavor over from the meat. Examples of that happening are cases like SQL query languages, Hypercard, APL, Perl's string handling syntax, Awk, Erlang, Mathmatica. One of the sweeter aspects of AppleScript is how that language is flavored by the application/document model that informs the user experience of the Mac.
A scripting layer creates a huge range of large range of affordances for the the users (or more likely their agents the developers) to leverage the underlying data-type. The scripting lets me, the software designer, delegate searching the space of opportunities that data type presents. In the limit the scripting creates an infinity of possibly extensions while the universe users/developers provides an infinity of search agents. Additionally as developers become fluent in the scripting language (so they become more loyal) and the population of users grows (so the audience for developers grows) a network effect can emerge.
You can see this same design pattern cast in a slightly different light has emerged around Internet middle-ware. A middle-ware programming language like PHP, Perl, etc. etc. provides the gray on top of the rich data stored in locally and in the Web.
There is a competing design pattern were we focus not on the synergy between the data and the scripting language but instead on the boundary between them. Focusing on that boundary leads to talking about APIs or protocols. This is not without it's benefits, for example it can leave the choice of scripting language unbound. That's good for at least two reasons. First it enlarges the pool of developers you get to draw upon. Second it lightens the design load; both in the short-term (you don't have to design a scripting language to get going) and the longterm (you don't have to evolve your scripting language to keep going).
I got to discussing this with a colleague the other day. He was advocating that we play of the scripting card in a certain context. Surprisingly, my intuition was opposed.
Like most people working on a problem I reach into my bag of tricks pulling out various tools and seeing if they help. These days I'm in a bit of a rut. Every time I reach in the bag I seem to pull out this damn power-law model.
So I notice that on the Unix platform there is a huge diversity of scripting languages; it's a part of the Unix story going way back. I'd assume if you collected some statistics about the popularity of various languages in the Unix you'll find a power-law curve who's parameters are reasonably egalitarian. Of course it's a stretch to claim that any power-law curve is egalitarian but some are more monotheistic than others!
Over on the PC platform the story is quite different. There the one true scripting language is Basic (which marketing in a classic 'new and improved' upgrade ploy decided to call Visual Basic); it's a part of the DOS story going right back to the beginning. So it is a huge uphill fight to introduce an application specific language into that climate. Where in the Unix culture a learning a bit of a new scripting language is a common experience in the PC culture it's an unusual one. If we had statistics about the scripting community scale on the PC, Then, like most everything else on the PC, it wouldn't be particularly egalitarian.
It's interesting to recall that when the Mac was first introduced Gates made a point of horse trading with Apple so that the Basic implementation Apple had lined up, finished, and ready-to-go was never shipped.
People tend to get all fixated on the role of protocols (XML et. al.) as the tool for creating openness and interoperability across the Internet. That maybe a mistake. It maybe we should be a lot more conscience about the role of a diverse population of languages in this game. That a world of one language is, in time, a monotheistic world. That world, while possibly somewhat more efficient, encourages a monopoly.
Unix is more a world of small gods. Let's hope we can keep the Internet that way.
Good article showing that all of the metrics of firm scale are power-law distributed: Zipf Distribution of US Firm Sizes (pdf) by Robert L. Axtel. I betcha that the same shift in exponent found in the wealth distribution has been taking place in the firm size distribution. If true that would shine an interesting light on the 'small business friendly' retoric of various parties.
All of Axtel's work is interesting. He has a agent theory based explaination for why firms scale is that way, along with a simulation. I don't find the theory all that convincing, but it is a very nice peice of work.
He also has an interesting graph in one of the presentations out on the web that shows that only in Russia are the city sizes not distributed "right". Presumably that's telling a strong story about planned economies. It got me wondering if there is a similar story inside firms about the distribution of the size of their various divisions.
My logs recently have been getting a lot of traffic from TurnItIn.com. They sell a plagarism detection service. Here is an example report from their system. Presumably they are scraping my site to populate their databases.
Interesting business, lots of nice aspects. First notice how they can sell their service to both the creators and the consumers of written works, i.e. they are middlemen. Nothing like a protection racket: sin, crime, fraud, paranoia, etc. Finally you have a great little network effect; only the most a paranoid customer is going to check for plagarism at more than one such site but you certainly are going to be drawn to the #1 site.
In time they could manage to become a standard; any work submitted to anyone might require an originality score from these folks. Maybe then can get some state legislators to make this a requirement for all the state schools. When all is said and done they can turn around and start collecting licencing fees for IP rights holders.
But, they better move fast. Lots of companies have a big pool of data they can use to check for plagarism: Google, Northlight, or even one of the big 80s online info services. They better move fast, get that brand advantage going.
Of course a system like this isn't much help if your problem is with a journalists who's just making it up. In fact it could be that what you want your writing to aspire the Goldilocks plagarism score. "Not too hot, not too cold."
It's amusing to notice that the model that power-law distributions are more likely to arise in emerging markets, ones where user's are reduced to follow the leader behavior, is similar to Gossling's phase model of standardization.
One of the roles of standards making is to provide the market with metrics they can use to measure product quality. That's obviously the function of standards like weights and measures. I suspect that many buyers hold back from entering a market until they see the emergance of open standards. They then use that event as a signal that the market has matured sufficently that they can rest assured that the knowledge institutions have appeared. Such buyers are, presumably, not intending to reap the advantage of being an early adopter, i.e. the chance that by moving early they can trigger a transformation in their own upstream markets.
Meanwhile you can see other analogies in this story about how fast a rumor/panic spread in Hong Kong regarding SARS. Here the prankster managed to yell "fire" in the crowded theater by fradulently claiming to be a reputable local news paper. Individuals who heard the rumor all try to assess it's quality. After a while they are reduced to looking around and noting that other people seem to be acting on the rumor. That's all you need, ignorance and users who are ready to act. Ready to link into the graph. In this case the nodes in the graph are behaviors to take regarding SARS rather than vendors in a marketplace.
Many years ago I was in one of those perennial meetings where the senior vice president stands up before the entire division and takes questions. These meetings often are nothing more than a study in power politics. Young turks with nothing to loose ask questions that bait the leader. Sycophants ask questions in support of what ever this weeks messaging seems to be.
Occationally someone naive about the stress in the room will ask one of those questions that goes right to the heart of the problem. In this meeting a young engineer asked "How important is time to market?" This was on all of our minds. The product we were working to release was more than a year behind schedule. We were all having a tough time making trade-offs between schedule and creating a virtuous product.
At that time in my life I was working with a very witty coworker who was full of very amusing one liners. For example he'd say "Here at YoYoDyne our leadership toggles betwix avuncular and asshole." During this interval we were in an avuncular phase.
The senior VP tackled the question: time to market. He stated that he really didn't know, but that in all his experience first to market was worth a substantial amount. I remember thinking, hm that's not a very strong answer to what's our #1 ethical issue around here, but I certainly respected that it was an honest answer.
Today I think we know the answer. It's not a pretty picture. Capturing users is much more important than creating a virtuous product.
We know this because we know that power-law distributions will arise whenever the new users arriving in a marketplace exhibit a preference for the market leader.
We also know this behavior is extremely likely in an emerging market. Consider this scenario. New users show up in the new marketplace. They thrash around trying to decide which vendor to hook up with. They thrash seeking some reasonably rational basis for making their decision. Now recall that this is an emerging market. Almost by definition that means that expertise is very thin on the ground. Users have a very hard time finding any source of expert advice. The user looking for some measure of product quality is left in a bit of bind. The market, because it is new, doesn't have any mature sources of information about product quality.
At that point the user has to fall back on proxy measures of quality. That users then select the product to buy entirely on the basis of what other users already selected. It's a fall-back onto almost the only information that user has available. The user is behaving rationally. Meanwhile, the market rapidly locks into a power-law distribution. The few early winners gain huge market share.
Hopefully, over time, markets will mature institutions, people, and collective knowledge to measure product quality in more accurate ways. At that point the early market leaders might start to be displaced. Of course how quickly that happens depends on how sticky the product offerings are. Software products are particularly sticky because of the training costs, and the way they capture the user's data in ways that, by default, make it hard to switch.
Some markets never get the chance to mature. They evolve too fast. The entire stack of industries that stand on top of electronics revolution are like this. As the cost of computation, storage, and communications all fall, at large multiples each year, new markets keep emerging. Each time one of these markets emerges we get the same story repeated.
Large numbers of new users enter the new market, they thrash around for a measure of quality, they settle for using the behavior of others as a proxy for quality, and that assures the entire market grows up with a power-law distribution of vendors.
So the answer to the question? Time to market is everything when you are dealing with a newly emerging market that lacks matured ways to measure quality. This is doubly true if the customer relationships your capturing are highly sticky.
In my wallet right now there is a little war going on. Representatives of various armies are fighting it out. Let me introduce them.
I have two kinds of Government currencies.
There is a nice 10 thousand yen note in there left over from a trip to Japan I took almost a year ago. I was forced to use Japanese currency when in Japan. They don't use credit cards or checks much - in fact you could always tell that a restaurant was going to be amazingly expensive if they displayed the 'flag' of visa/master-card on their threshold.
There is some US legal tender - "This note is legal tender for all debts, public and private". It says "In God we trust" while Japanese note has a picture of the emperor on it and I've no idea what it says.
I have some private currency.
There's a gift certificate from a huge bookstore in a distant part of town. It's worth $10.33 cents.
I have a gift card that came via a rebate from the purchase of a cell phone.
I have quite a few forms of plastic based currency.
There is a credit card provided by my company that I'm coerced into using when I travel. That let's them capture a number of benefits. They get the discount points on the transactions. The card allows them to prevent me from shopping in certain venues. It lowers their book keeping costs.
There is a credit card from from a small bank in the Midwest. I'm convinced to use this card because I'm bribed with a 1% cash back program. It's very complex. I have to accumulate points, and then once over a certain threshold I get 1% back. If I remember to request it. Of course I run the risk they will change the terms or go out of business before I get my money back. They already changed the terms once, I used to get 2% back.
There is a credit card from co-marketed by a major credit card processing bank and Beans. I was convinced to get that because it came with 2% cash back. They changed that to half a percent after six months. I'd get rid of it but the news paper subscription is tied to it.
There is a bank card. I had to ask them to send me one that didn't have a debit card tied to the bank card. I sometimes use this card to buy groceries. They let me have cash back.
Then I have a bunch of cards that let me do transactions at semi-private clubs. All the places I can say one way or another "put it on my account".
I have my health insurance 'club' card.
I have the health insurance 'club' card of my previous employer, since showing it gets me certain discounts.
I have my library 'club' card. It's interesting because it gets me into two library networks and a few hundred individual libraries. They have linked all those accounts together.
I have the card that denotes my membership in the car driving club, aka my driver's license. I need that to be allowed to take cars onto the highways. These days it's the only card that let's me fly on commercial airplanes. The phone company also demanded it when I got a cell phone.
I have three cards for gaining access to my job. One gets me onto the landlord's premises. One let's me get into various properties around the country my employer does business in. One let's me get into the garage at work.
I have a card that let's me enter the building were my son goes on Saturday mornings.
Then there are my charity club memberships, but I don't carry those; except my ACLU membership card.
My absolutely favorite club card is the one that let's me enter the Library of Congress.
All these things are there to let me do transactions. All of these are forms of currency, currency substitutes, or representatives of account relationships.
Currency has lots of network effects. Transactions are simpler if the parties have a currency they both agree to accept. Transactions are simpler if each party doesn't have to include a phase in where they negotiate the means of payment. For example merchants are required by the credit card companies not to offer a discount for cash, but most will if you can deal with somebody in authority. Transactions are cheaper if we all don't have to run different balance sheets for each kind of currency and then try to reconcile them once a month.
All these are competing forms of money are all trying to balance out transaction costs, bookkeeping costs, relationship stability, loyalties, trust, etc. etc.
Some are just trying to get a share of that market so they can take a bit of each