Category Archives: stories

Displacement and Common Lisp

Displacement is an economic or cultural process where by a community of practice wakes up one morning to discover that the tide of history has left it high and dry. The displaced community is not, necessarily, at fault in these stories. The archetypical example of displacement was the introduction of a new technology into northern england that displaced the tenant farmers from their land holds. In that case the displacement unfolded pretty quickly because their legal claim to their land was based on leases; so when the leases came due and the landlord’s (who wanted to convert to the new paradigm) displaced the residents. The technical innovation that displaced the farmers was a more robust breed of sheep coupled with a business fad for sheep.

In the software industry displacement happens when an existing language community – Cobol, Fortran, whatever – wakes up one morning to discover that the industries current fast growing network effect is taking place outside their community. Some communities manage to catch up by quickly piling on the tools, design patterns, etc. etc. required to play in the new world.

The Common Lisp made one mistake back in the 1980s that helped with it’s displacement. This mistake was around graphic user interfaces. When the Mac came out it redefined how graphic user interface interaction would take place. It set a standard for the interaction. This standard featured the idea of a current selection. First the user would use his mouse to accumulate the selection. He’d select the window/document to work on. He’d adjust the view to bring the thing he wished to modify into view. He’d then select that object. Only then would he browse a selection of commands to affect the object. Who knows if that’s the best design; but it certainly became the standard approach.

Meanwhile over in the older graphic user interface communities the command loop worked quite differently. For example selection was often entangled with mouse location. For example if you moved the mouse over an window or an object the object was automatically selected – but that’s only an example.

In the Common Lisp community a really unbelievably elegant user interface tool kit emerged known as CLIM (or Common Lisp Interface Manager). But this beautiful elegant thing had no concept of “the selection.” As a result it was totally irrelevant to the building of the kinds of user interface that were demanded by those working where the action was. Great ideas displaced by no particular fault of it’s own.

Microsoft had an analogous brush with displacement when the Internet broke out and desk top suddenly became marginalized. Microsoft has, historically, been better at mustering the sense of fear and panic necessary to respond to displacement events. So when it became clear they were at risk they reacted.

The Common Lisp community endures. I still use and prefer it for all kinds of tasks.

But, recently I’ve been concerned by what looks to me like a another displacement threat – character sets.

Emacs is a key complement to the Lisp community; and quite a few others. Emacs has amazing character set support, both the major variants (GNU Emacs, and XEmacs). The support is uniquely powerful. The approach emerged in a branch of emacs known as Mule and more recently has getting folded back into Emacs. All the input output streams can be configured to declare their encoding schemes. The internal strings and buffers are especially clever. The usual trick for systems having to tackle this problem is to normalize all the characters into one standard format, typically unicode. Mule’s approach is different; buffers in a mule enhanced emacs retain their character encoding. Load a file of Big5 characters and point at an arbitrary character in the string and mule emacs knows that’s a big5 character; paste a string of unicode characters into that buffer and now you have a buffer who’s characters are in assorted character set encoding in a way analogous (but with a different implementation) the way that you can have a buffer with characters in assorted fonts. That I find the mule design so cool reminds a bit of how cool I found the CLIM design; but at this point I’m feeling a bit paranoid.

So on the emacs front things are in in pretty good, even very good, shape. It’s all a bit rough around the edges though. The emac communities are still holding mule emacs at arm’s length so you often have to build the mule variant by hand. You often need to get the version of emacs that’s ahead of the stable release curve to capture the features you need. Font support is both amazing and frustrating. If your running under X then you can get a large set of international fonts and after a mess of suffering you can get your GNU emacs or xemacs to use them. One curiosity of the mule buffer design is that a character encoded in one character set may not have a font to render it only because the only font you have installed able to render that cute character happens to be laid out using a different character set. That’s a big pain on the Mac which has beautiful fonts but I can’t see how to get to them from unicode characters sets in emacs.

Over on the Lisp side of things the story is slowly resolving it’s self. There are a _lot_ of really fine commercial and open Common Lisp implementations. Each one has a slightly different story about how and when the unicode problem is getting addressed. The best unicode support in an open implementation happens to be in the slowest implementation. The implementation I’m using today (Steel Bank Common Lisp, or SBCL) has very fresh unicode support.

It’s taken me almost two weeks to get a working tool chain for this stuff. I have tried a lot of combinations and experianced a lot of crashes where both Lisp and Emacs die horrible recursive deaths choking as tried to display or transport characters down pipelines. Currently I’m running the lastest released beta version of XEmacs, build from scratch to get mule support. I’m running that under X on my Mac; so I’m using the open international X fonts. I’m running the bleeding edge versions (CVS Head) of both SBCL and Slime (the emacs< ->lisp interaction mode). [Hint: (setf slime-net-coding-system 'utf-8-unix)]

I’m happy to report that I can now stream unpredictable UTF-8 streams thru reasonably long chains of tools and it all works. Everything in my tool chain except the fonts is beta or bleeding. I’ll be really happy when I’ve got the database linkages working.

If this was 1995 I’d be less concerned about displacement; but it’s 2004. The good news is that the problem is getting solved.

Cutting the Resistor

Way way back in the 1960s the university where I used to punch cards and hand them thru the little window to run my batch jobs bought a new computer. The disk drive was so massive that when they brought it into the building the drive grazed the edge of the door way knocking off the frame and a few rows of concrete block. The drive was undamaged.

Shortly after that machine arrived a guy showed up. An hour later the machine ran much faster. Apparently the salesman had recommended him. The machine was the low end model and if you wanted it to behave like the high end model all you needed to do was cut a resistor or something.

Apple segments their market for the MacOS into three bundles: Darwin, MacOS (client), and MacOS (server). The function of segmented marketing for a vendor is to capture more of the revenue/benefit available below the willingness to pay curve. Given that most markets have a few customers who will pay a lot (desperate customers, rich customers, etc.) you want to have a product variant that you can sell them.

At the other end of the spectrum you get customers who have little willingness to pay; a cheap product offering may capture some value even from those guys. Network effects make things more complex. The customers willing to pay nothing may still add value for the vendor; by increasing his network and thus attracting other customers with deeper pockets. This creates the curious effect that some markets the right price point for the low end product is free – it’s not actually free the customer is bringing himself to the party as payment.

The practical advice given to vendors about segmenting a product line is to be very careful to watch costs. If your lucky the expensive product should cost little more to make and distribute than the mid-priced product. That advise leads to the common stories folks tell about computers who’s speed could be doubled by cutting a resistor. Back at the vendor the discussions about what to bundle with each version of a segmented product are painful for people with ethics. Once you work thru the ethical puzzles there is actually a very interesting design puzzle in designing segmented products – i.e. how to assure that you capture the maximum network effects across the whole product line. If you get it wrong the segmented products start evolve into distinct specie and you get three separate markets. That’s already happened to Apple between Darwin and MacOS(client).

Another problem, though generally a small one, is that some clever user discovers how to cut the resistor. Suddenly their expensive product is just as good as the mid-priced offering. This tends to be more an embarrassment than a revenue problem. I own a big fat phone, a Treo, and the geek community around it keeps working around all the functionally limits that it’s vendor was forced to stick into because his real customer are the phone companies, not the end users.

The story from my youth has another chapter. At the time I was told that the reason for resistor cutting guy was that early customers of that line of machines had written into their purchase contracts a clause along these lines. Sure we will pay a huge amount for the first few of this machine, but latter if you succeed in selling a large number and your prices fall we require you to give us a refund of the price difference. The vendor was dodging the clause in the contract by creating a lower price segment in his market and then leaking the recipe to enhance the model back toward the original model.

Recent events in the around the Treo (a hack to enable bluetooth data, and a hack to enable wifi support) reminded me of that story; and then this morning I say this neat trick to let me get some of the features of MacOS(server) to work on my MacOS(client).

When these hacks for breaking down the barriers between a vendor’s segmented market emerge it’s always interesting to think thru what might be the motivation. Is it just some hacker having a good time? Maybe it’s the vendor trying trying to assure he gets the maximum network to emerge around his product. Maybe it arose out of the vendor’s cost saving. Maybe the vendor is trying to work around customers who’s goals are not in synch with your best interests (as in the two examples above). All these and other reasons are likely in play which makes the stories more interesting, but harder to tell.


From the amazingly affecting blog of a first year teacher in NYC (in this episode the school system hadn’t managed to pay her yet).

“Allegedly, I’ll be getting an “emergency” check on Tuesday (Monday is Yom Kippur and the schools are closed), but I’m living through this three day weekend on the “State Quarters” that I’ve been saving since I applied to be a Teaching Fellow last October. At the grocery store this morning, the man behind me in line wanted to know if I’d been to Atlantic City, since I was paying from a paper cup full of quarters. I had saved those quarters to use for teaching; I didn’t spend a single one that came into my hands in 10 months. Good thing, too, I guess. After making sure that I saved two of each in the holders I have (a map and a folder), I had a bit more than $20 which will hold me through the weekend and get me to work on Tuesday.”

Start at the beginning or start with the latest but do read it. It will be interesting to see how long she can keep it up before the institutions around her find a reason to shut her down.

Note: I deleted this posting from 2002 by mistake; spam! So I’ve reposted it. I apparently don’t understand the “edit timestamp” feature in Word Press; so it looks brand new.

Vernal Pool

The vernal pool has appeared at the bottom of the yard. Walking along the avenue this morning water was gushing. Out of manhole covers and hoses from basement sump pumps. The valley the avenue runs along was the site of the first fresh water mills in this area, possibly the first in north america. The mill stream drew it’s water from the surrounding hills. The dozens of mills along that stream ran up until the 1930 or 40s. At that point the produce farms that served the city planted a final crop of housing. The network of roads and storm sewers pulled all the moisture out of the land and the mill stream dried up. Now each spring the snow cover melts, the string rains fall, and what used to be years worth of water comes down off the hills into the mill stream. Roads close. Basements flood. A pool forms in my backyard where the old maps show a small stream passing thru on it’s way to a small swamp and pond were now in the parking lot of a grocery story a large puddle lays.

What if the Turing Test has a social network?


Via David Chess we learn that evil web spidering robots have found a way around yet another security device. The CAPTCHA system works by showing the visitor a picture, see at right, and then asking them to type in the word hidden in that picture. It’s a Turing test of that the visitor has at hand a human visual system.

The robot’s authors hacked around this by having the robot turn around and ask it’s human friends to answer the question for it.

But at least one potential spammer managed to crack the CAPTCHA test. Someone designed a software robot that would fill out a registration form and, when confronted with a CAPTCHA test, would post it on a free porn site. Visitors to the porn site would be asked to complete the test before they could view more pornography, and the software robot would use their answer to complete the e-mail registration.

This reminds me of a story I heard a long time ago about how the SAT folks became curious about the high scores at one school. They traveled out there to discover that they class was solving the test as they did all other problems in their community, cooperatively.

Out of Office

I sent email to a large private email list recently. I have gotten 20 “out of office” replies. This is dumb. The out of office robot ought not reply if the sender isn’t one of your correspondents for some reasonably definition of corresponent. How hard is that? For example if your have never sent X email and never replied to X’s email, and never even participated in an email thread with X, and X isn’t in my address book – what does this robot think it’s doing send X details of your personal life? There must be some really amusing stories of email robots revealing personal info.

New York Metro Region

It was experiances like this that most loved when I lived in NYC

“…Upstairs (and to the side…it’s weird, hard to visualize; it is not completely of this Earth) you’ll find International Food Warehouse (370 Essex St, Lodi, NJ; 201-368-9511). It seems to be squatting in an extra wing of warehouse space.