I must be wrong, but apparently there is no well tooled standard way to manage trust for digital artifacts.  Consider an example: the instructions for installing Ruby’s rvm tool look like this:

curl -sSL https://get.rvm.io | bash

That’s wonderfully simple, although it implies a lot of trust in get.rvm.io!

I run into this problem a lot. For example I have scripts that help me configure virtual machines. Here’s one that installs a hyperdex that I run as I’m setting up a new machine.

cat <<'EOF' > /etc/yum.repos.d//hyperdex.repo
[hyperdex]
name=hyperdex
baseurl=http://centos.hyperdex.org/base/$basearch/$releasever
enabled=1
gpgcheck=0
EOF
yum clean all
ls -l /etc/yum.repos.d/
yum --assumeyes install hyperdex
ls -l /etc/yum.repos.d/

I then do something like this on a fresh machine:

curl http://example.com/install-hyperdex.sh | bash -

I can share these scripts, but I’d be loath to entice people into the bad habit of running such things.

And this bad habit is becoming very common.  For example Continuum Analytic’s amazingly cool python data tool suite and it’s extremely useful tools for managing python versions and packages is conveniently installed by downloading a 16 megabyte shell script which you then cheerfully hand off to bash.

Of course there are systems that help with this mess.  Yum will check gpg signatures, and it’s lame that says “gpgheck=0”.   But why am I unaware of any tools that make it straight forward to check signatures on scripts like my install-hyperdex.sh script, or Ruby’s script for installing rvm. What I want is a tools that lets me tell users something like this:

To install Awesome-Software do this:

run-remote-script http://example.org/install-awesome-software

If you don’t have run-remote-script see how to install it by visiting …

Obviously we could ask users to step thru longer instructions.

To install our awesome software first be sure you have installed gnupg. Then download our signing key into your keyring.

curl https://example.org/our-signing-key.asc | gpg --import

Now you can download our install script and check that we signed off on it.

# Get the script.
curl -o /tmp/foo http://example.com/install-thing.sh.asc
# Verify it's signature looks ok.
gpg --verify $/tmp/foo

If that looks ok, then extract the script and run it.

sed -e '1,3d' -e '/-----BEGIN PGP SIGNATURE-----/,$d' /tmp/foo | bash -

That will work … if you don’t want users.

Why is this so damn hard?  As I said at the start: we have “no well tooled standard way to manage trust for digital artifacts”.  For heaven’s sake I had to use sed to extract the script!

One part of the answer would seem to be that systems like homebrew, ports, rpm, yum, etc. etc. are all trying to solve a larger problems.  Which is fine, but they fail to address my problem, or the problem the rvm team has, or the problem that the python data guys have.

I have tooled up some of this, for my own needs when building virtual machines. But, it’s hardly a useful tool for others.  And, it’s very much a work in progress.

Gosh, I feel like somebody must have already written a tool analogous to “run-remote-script.”

This talk, which I only listened to, about the Forbin Project’s Google’s systems for back up and restore was fun.  It just lets you glimpse little bits of what is obviously an elephant.

It is unsurprising but sad how extremely proprietary the computer industry has become.

Here are the things I enjoyed in the talk, particularly the first two.

• He hinted that they can use encryption key management to delete customer data without the bother of erasing all the backup fragments.
• It’s all about the restore.  That’s when the system comes under intense scrutiny.  So it must be fast and automated.   So much of the design evolved after that pressure became clear to them.
• They replicate, ala RAID 4, the restores across tapes.  Bad blocks identified (or suspected) are repaired
• They do regular restore testing  (5% ?).
• Replication/redundancy is necessary over many dimensions, not just copies.  Geography, mechanism, … but he didn’t enumerate the dimensions.
• He hinted that to speed restores they might use only half tapes, since that reduces seek time. Though if you think about it you’ll see that you can sort the tapes so redundancy blocks are in the 2nd half.
• They do ship data physically.
• Logistics planning is obviously a thing.

I would have loved to get a brief overview of the API they deliver to the systems that utilizes their services.  Particularly the introductory material that outlines the contracts you can negotiate via configuration and what then are your responsibilities as a user of the services.  There are some very slight hints about that, but not much.

The multi-dimensional aspect to redundancy got me to wondering if they have a backup exchange agreement with other big Sky Net operators.

The talk and questions is an hour and fifteen minutes.

A long time ago I was bewildered by a chart at the Oil Drum which showed that miles driven was basically perfectly correlated with GDP.  Here’s that chart.  In the years since I’ve occasionally thought that maybe it’s flat because Y axis is lousy.

This topic came up again.  Andrew, who is particularly interested in the inability of various actors to accept that they got it wrong, pointed out that the traffic planning folks have got their projections wrong for a while.  He reposts this damning “fan chart.”

Andrew’s post lead to an interesting, if cynical, conversation in the comments, which in turn triggered Raghuveer Parthasarathy to revisit that the correlation; updating the range, tidy up the axis, etc.  He posted these three charts.

First we have total miles driven.  Clearly something happened, i.e. this awful recession.  And maybe something happened to create a slight bend in the trend from the range between, say, 1995 and 2006.  I think that’s what the inset chart is intended to help clarify.  It’s odd that miles traveled appears to rise around the dotcom bubble burst.   That was not the case where I live!  What ever that four years is odd.   The lack of any recovery after the recession is a puzzle too.

The second chart is miles/person.  Now the lack of recovery post 2008 is even more striking.  Those last four dots seem to suggest that drivers are becoming dispirited.  Let’s blame Facebook?

And now the miles/$-gdp.  This is the oil drum chart updated with 10 new years of data.  But, yeah, the overlapping portions of the two charts do not agree with each other.  Weird.

Again we can see the odd four years around the internet bubble.  And, curiously this chart seems to shows that miles/gdp rises a bit around a recession   It’s a lagging indicator?

But of course the most fascinating thing is that there is a twenty year trend of less driving per GDP dollar.  I have a sickening feeling that’s the rise of the bank’s share of the GDP, but who knows?

Like my facebook or banking suggestions it’s not hard to find people making up other insta-theories.   Aging population.  Or: Have you tried to get a drives license recently, it’s a PIA!  Youth unemployment.  Student loan debt.  I don’t doubt there are professionals that think about this much more carefully than I can.  I’d love to know what they think.

This essay on the recent unemployment numbers is a pretty reasonable attempt to walk the line between the two “consensus” opinions about the economy.  I.e. “It’s could be better but there is steady improvement.  Oh yeah, inflation is concern.”  v.s. “Seriously stagnant man!  Oh yeah, that people are suffering is a concern!”

I found this chart particularly interesting.