Category Archives: power-laws and networks

JMESPath is sweet

Leave a reply

Back in 60s card images were the canonical format for data.  Unix inherited that and tools like, sed, awk, and even perl all have carry forward that legacy, though they call ’em lines.

There have been many attempts to get something more meaty to pump thru our pipe fittings. For example: Erlang’s got something, Lisp’s got something,  XML, Protocol buffers, JSON, and many many others.  JSON is quite popular this morning.

JMESPath is analogous to XML’s XPaths, but for JSON.  I first encountered JMESPath in the AWS CLI tool’s –query arguments.  But it is totally a stand alone thing.  There are implementations in many popular languages and the licenses are quite generous.   I like using it to pluck data out of API end-points.

One aspect of network effects is that when you capture one your rewards (or possibly your curse) is an immovable installed base.   JSON (and XML) have succeeded where better schemes designed in the past have failed is because they both now have an installed base, i.e. the vast number of HTTP based API end-points.  Getting those to switch to something better would be a PIA.  Worse is better and all that.

While JMESPath is targeted at JSON, you can often use it to dig data out of the native data structures in what ever language you’re using today.

It was a nerd prank when I was a freshman in college to blow large quantities punch card chad under another’s dorm room door.

Like any good DSL JMESPath take a bit of getting used to.  The interactive tutorial is sweet.

The Poor? The pathologist report: it’s chronic, hopeless, and their own damn fault. Might be genetic too.

Leave a reply

Steve Randy Waldman has another awesome post, and this case he tackles the mystery of how you can have a reasonably well functioning wealthy liberal democracy at the same time as a huge segment of the population is shockingly poor.  Wealth inequality is a simple answer, but then why doesn’t the democratic process work to fix that?  So you get a “trilema.”  I love triangles.

His names for the three sides of this triangle are: Liberal, Equality, and Nonpathology.  Clearly this idea is going to have trouble getting traction if only because that last one is so odd.  And that’s the key idea.  You can have a functioning liberal democracy along with extreme inequality if you can get everybody to flesh out the bible’s “For you always have the poor with you” sufficiently.  If the majority of the population accepts that the root cause of both is that the poor are afflicted with some pathological flaw – genetic say, or bad maybe bad fashion sense.  This is amusingly covered in the in Westside Story’s “Officer Krupke.”

This technique for suppressing the natural feedback loop you’d expect in a democracy is.  This isn’t just the usual technique of reactionaries to say that it would be futile to try and fix a problem they don’t care much about.

Once you decide that the problem is that the poor are suffering from the disease state – which is only true to the extent that they are poor – you can call in various quacks to prescribe their favorite prescription.  Interview training say.  Or better impulse control.  Or more entrepreneurship risk taking.  Or scolding that they should study harder.  You know: the things that the well off struggle to improve in their own lives.  This is totally a win for the elites because the prescriptions just happen to server their goals.  Tax cuts!

It’s a very good essay, particularly the tail end where he addresses some of the stories elites tell, and the poor often accept, about the pathological behaviors of the poor.

 

Graphical Programming and yEd

Leave a reply

Graphical programming languages are like red sports cars.  They have lots of curb appeal, but they are rarely safe and reliable.

I long worked for a company whose product featured a very rich graphic programming. It allowed an extremely effective sales process.  The salesman would visit the customer who would sketch a picture of his problem on the whiteboard, and the salesman would enquire about how bad things would get if the problem didn’t get solved.

Meanwhile in the corner the sales engineer would copy the drawing into his notebook.  That night he would create an app in our product who’s front page looked as much like that drawing as possible.  It didn’t really matter if it did anything, but it usually did a little simulation and some icons would animate and some charts’ would scroll.  The customers would be very excited by these little demos.

I consider those last two paragraphs a delightful bit of sardonic humor.  But such products do sell well.   Customers like how pretty they look.  Sales likes them.  Engineering gets to have mixed feelings.  The maintenance contracts can be lucrative.  Thathelps with buisness model volatility.  So yeah, there is plenty of value in graphical programming.

So one of the lightning talks at ILC 2014 caught my attention.  The speaker, Paul Tarvydas, mentioned in passing that he had a little hack based on a free drawing application called yEd.  That evening I wrote a similar little hack.

Using yEd you can make an illustrations, like this one showing the software release process for most startups.

My few lines of code will extract the topology from the drawing, at which point you can build whatever strikes your fancy: code, ontologies, data structures.  (Have I mentioned how much fun it is to use Optima to digest into a glob of XML?  Why yes I have.)

I was also provoked by Fare Rideaus‘ talk.  Fare is evangelizing the idea that we ought to start using Lisp for scripting.   He has a package, cl-launch, intended to support this.  Here’s an example script.   Let’s dump the edges in that drawing:

bash-3.2$ ./topology.sh abc.graphml
Alpha -> Beta
Beta -> Cancel
Beta -> Beta
Beta -> Beta
bash-3.2$

I’ve noticed, dear Reader, that you are very observant.  It’s one of the things I admire about you.  So you wondering: “Yeah Ben, you found too many edges!”   Well, I warned you that these sports cars are rarely safe.  Didn’t I?

Welfare Economics

1 Reply

MBA types like to talk about “your business model,” and less so they like to talk about “their business model.”   I like to ask about the model’s effect on the wealth distribution.   It’s a hard question, but generally few businesses actually shift wealth and income in what I’d see as the desirable directions.

With that said here’s a cute B-School chart:

For my purposes think of these two technologies as two business models; i.e. ways of organising the world to create goods for sale to the public.  And for my purposes we can think of the two axis as being rich and poor.   It helps illustrate how the technology has consequences.

That drawing is taken from an interesting post by Steve Randy Waldman, who’s coming at the question I’m interested in from what might be a quite productive angle.  But one way or another this kind of modeling helps to illuminate what I mean when I try to highlight how your business model, standard, technology, ontology, etc. shape in interesting and oft powerful ways the resulting distribution.

Continuous v.s. Batch: The Census

Leave a reply
Log, from Blamo: Civil War Reenactor

Log, from Blamo: Civil War Reenactor

I am enjoying this extremely long blog post about how logs can form the hub for a distributed system, by Jay Kreps from Linked-in.  It’s TLMR “too long, must read?”  It reminds me of my post about listening to the system, but more so.

He has a wonderful example of batch v.s. continuous processing.  A dialectic worthy of its own post at some point.

The US census provides a good example of batch data collection. The census periodically kicks off and does a brute force discovery and enumeration of US citizens by having people walking around door-to-door. This made a lot of sense in 1790 when the census was first begun. Data collection at the time was inherently batch oriented, it involved riding around on horseback and writing down records on paper, then transporting this batch of records to a central location where humans added up all the counts. These days, when you describe the census process one immediately wonders why we don’t keep a journal of births and deaths and produce population counts either continuously or with whatever granularity is needed.

Cute.  My goto example has always been the difference between the annual cycle(s) that arises from agriculture and tax law revisions v.s. the newspaper’s daily cycle in service of the demand for fish wrapping.

jobscalculatedriskBut of course that’s not really continuous, it’s just batch with different cycle times.  And yet I once encountered a continuous system that involved a pipeline across a desert.  Each time the sun would emerge from behind the clouds the pipe would warm up and a vast slug of material would be ejected out the far end into a hastily build holding pit at the refinery.  Maybe slug processing would be a good fall back term for the inevitable emergence of  batches in continuous systems.  Blame the clouds.

 

Smeed’s Law

1 Reply

In the 1930s a traffic engineer in England noticed a curious pattern in the data about highway deaths.  Here is the chart from the article he published.

The vertical axis shows deaths/car and the horizontal shows cars/person with one dot for each country.  That’s for 1938.   In 1938 few people in Spain(19) owned a car, but those that did were causing a lot of deaths.   Switzerland(2) wasn’t fitting the model very well.   You can make up your own insta-theory for why countries with few cars/person kill more people with each car.

Here’s a chart from 1980.  More countries, more years, more confirmation of the model.  The data are shown twice, the second time is a log-log graph.

Note that there are lots of things you might think would affect the numbers for a given country.  For example: seat belts, population density, driver median age, safety regulations, insurance, policing, road quality, dash-board cams…  But those aren’t part of this simple curve and so can only effect the residuals.

I stole these charts from J.G.U. Adams short article “Smeed’s Law: some further thoughts” in Traffic Engineering and Control, 1987

I find this all weird.   You would think the traffic engineers would have a polished consensus by now of what this is saying.  Adams’ article has some interesting things to say.  For example societies learn to manage the cars as their numbers increase.   But I don’t sense there is a consensus in the profession.  Even now, 80+ years after the pattern was first noticed.

Selling out your Friends

Leave a reply

Robert Shiller: “It’s not the financial crisis per se, but the most important problem we are facing now, today, I think, is rising inequality in the United States and elsewhere in the world.”  And he won a Nobel Prize.

I have a theory about this problem.  Think of the set of all the world’s supply chains as a network.  I think we need to grow this graph so it’s a lot more bushy at the low-end.  Shrubbery!   I guess this theory shares a lot with Bill McKibbon’s ideas in Deep Economy; or the Prahalad’s ideas in Fortune at the Bottom of the Pyramid.

‘I don’t keer w’at you do wid me, Brer Fox,’ sezee, ‘so you don’t fling me in dat brier-patch. Roas’ me, Brer Fox,’ sezee, ‘but don’t fling me in dat brier-patch,’ …

I continue to harbor great optimism about the Internet,  It can help us with this.  The Internet has an amazing power to enable communities of common interest to form.  These communities are great of shubbery.  Precursors of commerce?  Maybe.

But, it’s worth chewing on the ideas in “how to lose friends and family via mult-level marketing” a posting that Andrew highlights.  Andrew introduces the idea that MLM schemes provide a way for people to liquidate (e.g. convert to cash) their social networks.  Liquidate is what you get when your done the monetizing a social network.  Lots of people are into that.  Monetize – what a word!  What can’t we monetize, my cat?

So while I love the Internet’s power as a host of community forming I must say I’m taken aback by how rapidly capitalism has evolved businesses models that feed on these tender shrubs.

Ironically my social network got infected by one of these parasites just today.   A friend signed up for Venmo, a p2p payment company, and they posted this exciting fact to Facebook on his behalf.  I admit to an unhealthy curiosity about these emerging currency systems.  For example, I think Bluebird is very interesting.  So I went and signed up for Venmo and installed the app.  A few moments later I was distressed to discover it was scanning the entire address book on my phone, maybe a few thousand entries.  If you want to use thier payment network you have to hand over your contacts.  No way to void it.  So I uninstalled, etc.  Who knows if that helped?

I totally get that building out “the network” is an existential issue for companies like Venmo.  Desperate need is an excuse in a starving man, is it an excuse for a start up?  Not that you need to worry about Venmo.  Venmo got bought, and the buyer then got bought by Paypal.  So they captured and sold a network.  That this is what most internet startups need to do worries me.

Returning to shrubbery as a tool to work inequality problem.  No doubt there are many much more ethical ways to convert the small communities into engines of economic activity.  It would be great to have a list.  No doubt looking at MLM business models would inform that search.

Ray Dolby’s business model

Leave a reply

8trackI read Ray Dolby’s obituary in the New York Times because the Dolby noise reduction system is a textbook example of a two sided network business model.  Invented in the early 60’s the system enabled you to get better audio quality out of recorded sound.  It transformed the audio signal to route around flaws in the tape, tape heads, and transport mechanisms.  The problem it solved grew quite severe when cassette tapes became popular.  To get the benefit a lot of parties along the supply chain needed to play along.  Two in particular.  The companies that manufactured cassette players and the companies the manufactured the cassettes containing the entertainment.

DOLBY_Product_header-NOKIA-N8The obituary get’s it wrong.  Dolby’s achievement wasn’t the signal processing algorithms; his achievement was getting all the players to sign onto his system.  Two-sided networks (standards) are all about the difficulty of attracting, coordinating, and locking-in two diffuse groups.  Dolby managed to own a standard.  And so he got to charge a toll for his small part in intermediating between sound producers and consumers.  .  He them managed to steward that role so that even today his company (DLB) stands at the center of the standardization of sound.  Next time your watching a DVD notice how right there in the credits the Dolby name will appear.  Think about how much time and space that credit takes v.s. other contributors.  And today, it’s all digital!

I wonder if any of the New York Time’s obits talk about the deceased’s business model.

Open Reader API

Leave a reply

I use Vienna as my RSS feed reading.  The new beta release is interesting.  A while back Vienna added support to pull your feeds from Google Reader.  I never used that, i pull each feed directly from the source.  I didn’t use it for three reasons: 1) while I read blogs on multiple devices I partitioned the blogs per-device; 2) I never got around to it; and 3) I don’t really like the idea of an intermediary knowing all the blogs I read.

The new release has support for pulling feeds from other sources.  And I’m delighted to see that there is some hope that we will see an open standard emerge for the aggregation service’s API; along with open implementations of that.

In turn this would help to allow more privacy around the aggregation service.  That’s a hard problem, but I have a sketch of a possible solution.