Displacement is an economic or cultural process where by a community of practice wakes up one morning to discover that the tide of history has left it high and dry. The displaced community is not, necessarily, at fault in these stories. The archetypical example of displacement was the introduction of a new technology into northern england that displaced the tenant farmers from their land holds. In that case the displacement unfolded pretty quickly because their legal claim to their land was based on leases; so when the leases came due and the landlord’s (who wanted to convert to the new paradigm) displaced the residents. The technical innovation that displaced the farmers was a more robust breed of sheep coupled with a business fad for sheep.

In the software industry displacement happens when an existing language community – Cobol, Fortran, whatever – wakes up one morning to discover that the industries current fast growing network effect is taking place outside their community. Some communities manage to catch up by quickly piling on the tools, design patterns, etc. etc. required to play in the new world.

The Common Lisp made one mistake back in the 1980s that helped with it’s displacement. This mistake was around graphic user interfaces. When the Mac came out it redefined how graphic user interface interaction would take place. It set a standard for the interaction. This standard featured the idea of a current selection. First the user would use his mouse to accumulate the selection. He’d select the window/document to work on. He’d adjust the view to bring the thing he wished to modify into view. He’d then select that object. Only then would he browse a selection of commands to affect the object. Who knows if that’s the best design; but it certainly became the standard approach.

Meanwhile over in the older graphic user interface communities the command loop worked quite differently. For example selection was often entangled with mouse location. For example if you moved the mouse over an window or an object the object was automatically selected – but that’s only an example.

In the Common Lisp community a really unbelievably elegant user interface tool kit emerged known as CLIM (or Common Lisp Interface Manager). But this beautiful elegant thing had no concept of “the selection.” As a result it was totally irrelevant to the building of the kinds of user interface that were demanded by those working where the action was. Great ideas displaced by no particular fault of it’s own.

Microsoft had an analogous brush with displacement when the Internet broke out and desk top suddenly became marginalized. Microsoft has, historically, been better at mustering the sense of fear and panic necessary to respond to displacement events. So when it became clear they were at risk they reacted.

The Common Lisp community endures. I still use and prefer it for all kinds of tasks.

But, recently I’ve been concerned by what looks to me like a another displacement threat – character sets.

Emacs is a key complement to the Lisp community; and quite a few others. Emacs has amazing character set support, both the major variants (GNU Emacs, and XEmacs). The support is uniquely powerful. The approach emerged in a branch of emacs known as Mule and more recently has getting folded back into Emacs. All the input output streams can be configured to declare their encoding schemes. The internal strings and buffers are especially clever. The usual trick for systems having to tackle this problem is to normalize all the characters into one standard format, typically unicode. Mule’s approach is different; buffers in a mule enhanced emacs retain their character encoding. Load a file of Big5 characters and point at an arbitrary character in the string and mule emacs knows that’s a big5 character; paste a string of unicode characters into that buffer and now you have a buffer who’s characters are in assorted character set encoding in a way analogous (but with a different implementation) the way that you can have a buffer with characters in assorted fonts. That I find the mule design so cool reminds a bit of how cool I found the CLIM design; but at this point I’m feeling a bit paranoid.

So on the emacs front things are in in pretty good, even very good, shape. It’s all a bit rough around the edges though. The emac communities are still holding mule emacs at arm’s length so you often have to build the mule variant by hand. You often need to get the version of emacs that’s ahead of the stable release curve to capture the features you need. Font support is both amazing and frustrating. If your running under X then you can get a large set of international fonts and after a mess of suffering you can get your GNU emacs or xemacs to use them. One curiosity of the mule buffer design is that a character encoded in one character set may not have a font to render it only because the only font you have installed able to render that cute character happens to be laid out using a different character set. That’s a big pain on the Mac which has beautiful fonts but I can’t see how to get to them from unicode characters sets in emacs.

Over on the Lisp side of things the story is slowly resolving it’s self. There are a _lot_ of really fine commercial and open Common Lisp implementations. Each one has a slightly different story about how and when the unicode problem is getting addressed. The best unicode support in an open implementation happens to be in the slowest implementation. The implementation I’m using today (Steel Bank Common Lisp, or SBCL) has very fresh unicode support.

It’s taken me almost two weeks to get a working tool chain for this stuff. I have tried a lot of combinations and experianced a lot of crashes where both Lisp and Emacs die horrible recursive deaths choking as tried to display or transport characters down pipelines. Currently I’m running the lastest released beta version of XEmacs, build from scratch to get mule support. I’m running that under X on my Mac; so I’m using the open international X fonts. I’m running the bleeding edge versions (CVS Head) of both SBCL and Slime (the emacs< ->lisp interaction mode). [Hint: (setf slime-net-coding-system 'utf-8-unix)]

I’m happy to report that I can now stream unpredictable UTF-8 streams thru reasonably long chains of tools and it all works. Everything in my tool chain except the fonts is beta or bleeding. I’ll be really happy when I’ve got the database linkages working.

If this was 1995 I’d be less concerned about displacement; but it’s 2004. The good news is that the problem is getting solved.