Category Archives: programming

Typing Injury.

I’d not seen this before.

JWZ’s essay on RSI, or typing injury. “… it terrified me. … my career being over”

I have one of my own written years and years ago. “… a friend who lost the ability to pick up a piece of paper …”

And I see that Bill Clementson recently joined this miserable club.

It’s a puzzle how until it happens one isn’t particularly interested; and even if you were interested getting advise isn’t straight forward. The advice is largely the wisdom of crowds. I.e. it’s hearsay, rumor, and stories like the ones above. The best you can hope for is to pick out the better of the old wive’s tails. It’s not often you get to refer to JWZ as an old wife! There is very little hard science and what exists seems to me to be very lame and often self serving.

It amazes me that an industry that has generated so much wealth hasn’t found a way to fund some substantial research into the affliction that forces the retirement of it’s most productive labor. Of course all minority groups have trouble getting attention for their problems. But in this case the minority group has actually got money. Still, it says something about who captures the wealth.

defun-with-cache

I don’t often have this kind of fun.

(defmacro defun-with-cache (name args &body body)
  (multiple-value-bind (binding call apply?)
      (rip-apart-arglist args)
    `(let ((#1=#:cache (make-hash-table :test #'equal)))
      (setf (cache-of-function ',name) #1#)
      (defun ,name ,args
	(flet ((,name ,args ,@body))
	  (let ((#2=#:key (list ,@binding)))
	    (values-list
	     (or (gethash #2# #1#)
		 (setf (gethash #2# #1#)
		       (multiple-value-list
			   ,@(if apply?
				 `((apply #',name ,@call))
				 `((,name ,@call)))))))))))))

As I like to say: “Lisp has plenty of syntax, it just spends it’s syntax dollars to buy different things.”

Just look at the nesting: defmacro, multiple-value-bind, let, setf, defun, flet, let, values-list, or, multiple-value-list, if, and apply; it’s like the parade of the wild and crazy programming constructs. Then you have the the fun going on with punctuation characters: #=:,@’ … and people complain about Perl.

Update:

Ah, the power of many eyes. It’s got a bug. Lambda lists in Common Lisp include the not unusual feature that you can write something like this pattern (x y &optional z) and then the caller can do (f 1 2) or (f 1 2 3). And of course you can specify an default value for the optional arguements i.e. (x y &optional (z -1)). There is an unusual feature in this vicinity though; you can write (x y &optional (z -1 z-provided?)) and then your function will actually have four arguments. The fourth one, z-provided? is bound by the language as a flag telling you if z was defaulted or not.

I like that the code above used an flet to create a version of the function which as it would have been without the caching wrapped around it. But I don’t see a straight forward way to do that if I’m going to handle things like z-provided?. So I got rid of the flet.

(defmacro defun-with-cache (name args &body body)
  "Like defun, but this memoizes the function into a cache that maybe
   latter cleared."
  (multiple-value-bind (binding call)
      (rip-apart-lambda-list args)
    `(let ((#1=#:cache (make-hash-table :test #'equal)))
      (setf (cache-of-function ',name) #1#)
      (defun ,name ,args
	(let ((#2=#:key (list ,@binding)))
	  (values-list
	   (or (gethash #2# #1#)
	       (setf (gethash #2# #1#)
		     (multiple-value-list
			 (progn
			   ,@body))))))))))

That I can’t use an flet in a straight forward way is an unusual breakdown of language symmetry. The need for the apply? flag in the original is analogous, but different.

Traditional Values

I love the way given a heterogeneous audience a single bit of humor can be at once all kinds of humor.

Fun Fact about Common Lisp: standard Common Lisp has no networking libraries at all, but it does have a built in function to print integers as Roman numerals — using either the new style (14=XIV) or the old-style (14=XIIII) Roman numerals.

Dave Roberts) choosing to pass over it’s author’s fun but sarcastic invective and labels it sardonic irony.

In the good old days audiences were more homogenous. It was easier then for the humorist to know his target audience. That informs the question of why it is so widely presumed that nice people don’t do humor.

DNS Cache Snooping

If you do sysadmin tasks for a group of people one of the challenges of the job is that your log files are full of bits of info that should be kept private. Here’s an example I’d not thought through sufficently: the DNS (Domain Name Server), actually the DNS cache. When one of your users visits a site (for example the url: http://cult-of-the-vile-snail.com/) the DNS server helps them.

Most DNS servers keeps a cache of the names looked up. That makes later attempt to lookup the cult-of-the-vile-snail faster and reduces upstream load. Here’s a curiosity. The protocol for talking to the DNS server has ways that allow you to ask just for immediately availably entries; and not look further upstream.

Thinking about starting an inquisition? Worried about your colleagues falling for the snail cult? Ask the DNS server. In some scenarios the DNS server can leak information about what the other users at your site are doing. By asking the DNS server if it has cult-of-the-vile-snail in it’s cache you can discover if anybody local has visited that site.

I had known for a long time about this privacy leak around local DNS caches.

I hadn’t realized what this paper points out. You can use these tricks on DNS servers out in the wild. For example say you wanted to squat on some domain names and you wish to know which misspelt domain names would be good choices. By poking a few thousand public DNS servers you can get a good estimate of which misspellings are common. Or say you wanted to do market research on how popular a competitor’s domain name is; same trick.

Here is a guy using this trick to get a picture, and amazing pictures they are too, of how wide spread the Sony rootkit is.

I certainly hadn’t realized how subtle the privacy implications of getting a good configuration for your DNS server is.

Niagria

Sun’s new processor offering is kind of interesting. It appears that they decided that the right processor architecture for servers (i.e. machines used by thousands of people at a time) is just different than the right one for more machines used more personally.

If you freeze the machine your using right at this moment and ask it how many threads are standing by waiting to get some processor cycles, i.e. not blocked for any reason other than access to the processor, the answer is almost certainly none, or one. If you designing the chip for that machine it’s worth spending transistors to make one processor run really fast.

If you freeze a reasonably loaded server, say one serving up web pages to a thousand users, with a few middleware applications behind the server, and a database or two behind those. The answer is may well be a few dozen, or possibly even a few hundred. Because this work load is more naturally fragmented your better off spending your transistors on multiple processors.

You might say they asked to question: if we could put a room full of servers on a chip what would that look like? Or maybe it’s the return of Connection Machine. I wonder if Sun’s got a *Lisp implementation?

I spent a lot of my early career working on multiprocess architecture both hardware and software. I became disillusioned. Given Moore’s law they were always only a few years ahead of the curve, which limited your markets and meant once you won customers you lost them a short while later. The action shifted to personal computing and away from servers, where the problems were easier to map onto the hardware. Mapping customer problems onto parallel architectures was sometimes easy; but more typically it tended to require too many clever engineering hours to be worth it. The programming languages, compilers, and operating system architectures that would reduce the how clever your engineers needed to be tended to be immature and the market wasn’t deep enough to spin up the network effects to fix that. That was the first time I began to appreciate the value of moving where the network effect current is fast.

Two things really triggered my exit from the multiprocessor industry. One was the Macintosh, it was just so cool and it had so many obvious opportunities to do neat things. The other was the death of the line printer. We used to have these line printers, and they would have one print head for every column on the page. And then we bought a new printer and it only had one print head that went back and forth. It was much faster. I thought – Yeah, if you can’t even make the case for multiprocessing in a task so transparently parallel in the physical world how likely is it that this whole multiprocessor thing is going to work out?

I wonder if it will be different this time? Three things have changed. Much of the action has shifted back to the server side. The total size of the market is just amazingly larger – which can feed the network effects needed to build the tools. Our talent at this kind of stuff is much deeper and broader. So it seems like it might.

But it’s daring to decide that the processor industry has a viable niche where a new species can thrive.

Building Houses on Sand

Philip Jacob worries, and rightly so, if this current enthusiasm for building websites that rollup bits and pieces isn’t, well, insanely risky.

“…would you incorporate a library in your application that was licensed under terms like this?

  • Required you to read and understand several pages of legalese
  • Is free right now although you might be charged for it at some point in the future at a price that you cannot negotiate in advance
  • Can stop working at any time for any reason without notice
  • Can undergo functional changes at any time without notice
  • Can be rate-limited at any time without notice
  • Does not have any service level agreement
  • Places restrictions on the data you generate using the library, some of which are bizarrely techno-legalistic and open to interpretation”

The licenses of all these fun web APIs are mavalously diverse. It is clear that each firm’s lawyer plays the role of genetic engineer gene splicing together a handful of licenses to create yet another mutation. While in long run (after you’re dead) the ecology will put most of these out of their misery. License interop is hard. This is another example of “standards as a substitute for lawyers.” Phil would like to have some standard licenses; but

We know how this plays out. First we get a bloom of diversity. Then we get consolidation. In the end we get a power-law distribution across some number of licenses. In the final end game the elite members of that distribution engage in a long standards battle who’s outcome can not be predicted – they might make peace, they might split into seperate markets, one might win, etc.

In early phases the smart players move quickly to capture market share that will provide key negotiating power in the later stages. Some of the early movers have to be profitable, others don’t. Some players value a strong position in the future more highly than others. Some players discount risk of having to switching later.

There are adjacent disputed territories to consider. Consider four of the larger players. The communication companies (though they haven’t fully accepted their fate yet) lock in users with physical networks and charge for bandwidth by the month. Companies like AOL, Yahoo, Google charge via advertising and lock-in by capturing a large distribution bottleneck. Platform vendors charge for a stream of updates and lock in via APIs and licenses. Players like the FSF or the ASF are sort of inside out with their focus on freeing developers (their principle consituency) from the licensing/API lock-in points.

All this is just part of the passion play around the emerging Internet OS, and to think I used to believe this was all kind of simple.

What is Social Software

There is a thread unfolding over here about this one liner:

“The whole point of social software is to replace the social with software”

But the thread has descended into a who said exactly what discussion that avoids the provocative nature of the standalone statement.

The statement is obviously true in for some situations.

For example consider a bug database. These tools allows a group of people to engage in the work of resolving bugs more effectively by draining most of social interaction out of the work. They enable the bug to found by one actor and resolved by another without the two interacting socially at all. Much the same way that doctors can treat desease without engaging in any particularly social interaction with their patients.

In a second example look at a problem that arises in source control systems. Two individuals are hacking away and their chanages happen to overlap. Mr. Speedy gets his changes into the source control system first. Mr. Methodical shows up later and discovers that his changes conflict with Speedy’s. There is nothing more commonly used as an example of social than conflict resolution and here we have exactly that problem a tiny dishpan model. The conflict resolution that happens at this point might demand a social interaction, but we have discovered that in a surprisingly large number of cases it works out well to just dump the whole problem into Mr. Methodical’s lap and let him puzzle out a solution. In this case the software design has done exactly what the quote suggests; replaced the social with software.

Or consider the wiki. I stuff some useful content into the wiki. Another actor dives in and rephrases it into grammatical english. A third actors repairs a date I got wrong. That process is hyper-effective because none of the actors need to engage in a social interactions. Each actor bears only the cost of his contribution, but none of use have to orchestrate a social relationship with each other. Most people’s initial reaction to wiki’s is bewilderment because they are so extremely a-social. It takes a while before you discover that can be a positive.

Social relationship creation and maintenance is costly. In numerous situations it is absolutely worth those costs. But that does not mean it should be automatically tacked onto every interaction. Fixing a bug, resolving a source code conflict, touching up a wiki entry, can be an oportunity to met new people and make new friends; but they do not need to be forced to serve that function.

I was quite conscious of this when I added the “report a typo” link that appears below all my blog postings. I know that most people don’t complain about my many typos because to do creates a delicate social dynamic. The form found under that link therefore doesn’t even prompt for an email address. I made the choice that I was more likely to get useful typo reports without the social aspect and that was a better balance of design than improving my ablity to say thank you to people do provide typos.

That tiny example shows the kind of tuning about social that systems of this kind enable.

A more accurate statement might say:

One exciting aspect of social software is the option of removing social aspects from the interactions.

If you want to get all big picture-ish … the whole point of the scientific revolution was the discovery that you could making amazing progress on some problems if you discarded all the important stuff. That how fast an object falls is not related to how much you love it; and the weather tomorrow is not related to your attitude about the rain gods. Autism can be surprisingly useful.

One way to frame the problems social software is dealing with is to label them as coordination problems. The bug fixing, wiki refining, and source control conflict resolution are all coordination problems at their core. They all run the risk of reaching bogus outcomes if you drain off the social elements entirely. The system failures that arise when that happens are well known. For example there are libraries full of books on what happens when product development becomes divorced from the end user’s needs and situations.

What’s exciting about open source is that it lets you experiment with exactly how to set the knob on how much social you leave in the coordination scheme you deploy.


Via the typo link, this extremely insightful addition: “And social interactions often draw from a limited pool, so by removing the need for them with software, this pool can be conserved and applied to actions with a greater possible return.”

Egg Drop Programming

… asked us if we ever intentionally got lost in a town, perhaps a town new to us, so that we had to learn the place in order to get back to a place we knew. Several people nodded vigorous agreement, and one guy noted that he and his colleagues use a similar technique to learn a new legacy code base. They call this air-drop programming. This is a colorful analogy for a common pattern among software developers. Sometimes the best way to learn a new framework or programming language is to parachute behind enemy lines, surrender connection to any safety nets outside, and fight our way out. …

Took that from here.

There are varients of this technique, walk-about, year abroad, home visit.

My techniques of this kind aren’t as random as that sounds. For example in a city I will find the old train station, park and then walk in the direction that seems to go down hill economicly. Or in code I will find the error logging or memory management and then work backward up the call tree, at first breadth first and then selectively depth first. Sometimes I look for the chinatown, or the I the center of what was the city’s primary industry 30 years ago. Often it’s good to look for the place where a bloom of new code occured, try to figure when it happened, and then look at the borders between it and the older code.

Now: Get lost!

Barndoor standards

Javascript is a perfect example of a syndrome in standardization that keeps CIO’s up at night. Let’s call it the barndoor syndrome; after closing folksy saying: “Shutting the barndoor after the horse escapes.”

Javascript escaped into the wild before it was standardized; it then underwent very rapid mutation in the installed base. Three forces drove this rapid emergance of new species: the security nightmare, the fun everbody was having, and high stakes competition. The last is particularly corrosive to the collaboration necessary for standards making. This family of species are now all over the installed base, and as we all know installed bases are very hard to move.

The poor web site designer is stuck with a miserable choice. He can antagonize large numbers of users; or he can make himself miserable. It’s a kind of quanity/quality trade off.

The standard(s) for Javascript aren’t a foundation for innovation; they are more like a beacon in the night toward which their authors hope the installed base slowly migrates.

When learning the language the standard is only a point against which you can measure the distance, in units of exceptions possibly, you must travel to reach this or that subpopulation of the installed base.

Driving the horse out of the barn is very tempting, since it builds momenteum and helps you search for the best design informed by actual use rather than ivory tower mumbling (i.e. security architectures). So we could rename this syndrome entrepeurial standards making rather than barndoor standards making.

When small entrepeurial firms do this it’s reasonably ethical; how else are they going to get traction in the market. When large monopolist firms do it the ethics are much more muddy. Which is something to think about when reading people’s critiques of Microsoft’s infocard. It is of course irrelevant to Microsoft if their designs go thru a legitimate standards process; just as long as it wins in the marketplace. Microsoft has cleverly attempted to substitute for a real standard process a conversation in among bloggers. The technorati are one audience you need to convince before a standard will gain great  momentum, but they are not a substitute for real legitimate standards making.  Assuming that you lack  sufficient  market power to just command it’s success.

Prophesy AI

“If only AI had worked out better.” is one of the favorite sayings of a contempary and fellow traveler thru the 80s AI boom. There are so many problems in life that a bit of effective AI ought to be able to solve. Coordinating the time of a meeting, or calling up the cable company and squeezing the next six month discount out of them

I’m currently reading “When time shall be no more. Prophecy Beliefs in Modern American Culture.” I haven’t gotten to the modern part yet, I’m still working thru the history of apocolypticism prophecy; three meaty chapters worth! The pattern the author sees in the historical record is that when ever western societies undergo trama a bloom of apocolyptic prophecy sweeps thru a segment of the population. He reports that each time things settle down again the bloom dries up, quickly. Each time this happens the various characters and events in the book of revelations are methodically mapped onto the recent history. This allows the narators to show thier enemies are the agents of satan. To hear him tell it these blooms seem more organic rather than planned. The forshadowing (teehee) is that later in the book modern media empires and PR based political movements will discover and exploit this market. A clear sign of the end times!

The laundry list of historical figures who’s names have been proven to sum up to 666 is amazing! Every pope, of course, and most historical figures.

Mapping names into a calculation that reaches a particular value is a simple search problem. The kind of thing that early AI tech is very good at. So it’s not suprising that you can get software to find these calculations. What I found amusing though was that you can find message boards that outline how to hack that software, so you can use it for free. Surely we need an open source version.

Now if only AI had worked out we would have software that could read any conflict narative and map it’s participants and events into the book of revelations allowing us to deamonize and polarize with much greater efficency.