Category Archives: common-lisp

clbuild cl-xmpp ejabberd

1 Reply

Assorted brush clearing.

Adding additional libraries to a clbuild setup is easy. Just add lines to your wnpp-projects file. For example:

cl-json get_darcs http://common-lisp.net/project/cl-json/darcs/cl-json

Give it a try and add additional things until everything you need is included. I’ll admit to editing the dependencies file by hand.

Sooner or later you get to things for which this doesn’t work. You can teach clbuild about additional projects by just adding them to the source directory and then using the register-asd command on that project. Given that your stuck doing that you will need to tell clbuild not to attempt to download them. You the command called skip marks the project to prevent any attempt to download.

That scenario arises if you want to use cl-xmpp, which depends on cl-sasl. Cl-sasl uses yet another source control system, GNU arch, which clbuild doesn’t handle yet. Clbuild is really pretty awesome at dealing with the excessive diversity of source control systems members of the Lisp community have embraced. We do love our exceptions.

I did try using get_tarball, which is deprecated, but that didn’t work out. Possibly because the tarball filename is cl-sasl_0.3; and that code appears to want it to be cl-sasl. I didn’t look further.

Cl-xmpp did not work, out of the box, with my ejabberd. That problem appears to be explained in this archive of the development mailing list. The resolution involves passing :mechanism :sasl-digest-md5 to xmpp:auth, but it also requires a patch to ejabberd to improve it’s xml conformance. I haven’t made the ejabberd patch yet, or maybe I need to upgrade ejabberd.

Ben Laurie recently observed “people don’t really talk much about the experience of writing and debugging code.” I suspect if that void were filled we would see many dull postings like the above; or more likely postings like the ones Jason S. Cornez makes in the archive above.

This posting is, in part, triggered by Ben’s comment. But it’s also provides a checkpoint. I need to context switch to other activities at this point. I certainly didn’t expect to devote quite so much time to this when I added this to my wnpp-projects file Saturday afternoon.

cl-xmpp get_cvs_clnet

Update: ejabberd is probably doing the right thing (see here)… Hm, looks like the code forked and a better version is to be had as so:

cl-xmpp get_git http://www.lichteblau.com/git/cl-xmpp.git/

Maybe I should always read the entire transcript of the development mailing list before I start anything.

A Doctoral Thesis is not a Standards Specification, but…

Leave a reply

I’ve greatly enjoyed much of Richard Gabriel’s writing over the years.  Though I’ll admit I haven’t read anything he’s done in the past few year.  In any case I happened to I listened to this interview he gave at OOPSLA to Software Engineering Radio.  The interviewer wanted to learn about this thing, Lisp, and he asks a series of questions to dig into the matter.  While for me this was pretty dull Richard does tell retell a story I’d not heard in recent years.  That got me to thinking about a model of how ideas used to flow from the academic research labs into programming community at large; and in particular how the Lisp community didn’t use standards in quite the same way as other language communities.

Lisp is a great foundation for programming language research.  It is not just easy to create new programming frameworks in Lisp.  The pie chart of where you spend time building systems has a slice for framework architecting and engineering.  Lisp programmers spend a huge portion of thier time in that slice compared to folks working in other languages.  In Lisp this process is language design, where as in other languages it’s forced into libaries.  There is a tendency in other languages for the libraries to be high cost, which makes them more naturally suited for a standardization gauntlet.  In Lisp it’s trivial to create new frameworks and they are less likely to suffer the cost and benefits of becoming standardized.

You get a lot more short term benefit in Lisp, and you pay latter as sweet frameworks fail to survive.  They don’t achieve some level sustianance because they don’t garner a community of users too look after them.

Back in the day this was less of a problem.  And thereby hangs the tail that Richard casually mentioned.  He was sketching out how a pattern that was common during the AI’s early golden age.  Graduate students would aim high, as is their job, and attempt to create a peice of software that would simulate some aspect of intelegence – vision, speech, learning, walking, etc. etc. – what aspect doesn’t really matter.  In service of this they would create a fresh programming language that manifested their hypothisis about how the behavior in question could be manifested.  This was extremely risky work with a very low chance of success.  It’s taken more then fifty years to begin to get traction in all those problems, and back in the day computers were – ah – smaller.

Enticing graduate students into taking huge risks is good, but if you punish them for failing then pretty soon they stop showing up at your door.  So you want to find an escape route.  In the story that Richard sites, and which I’d heard before, the solution was to give them a degree for the framework.

Which was great.  At least for me.  All thru that era I used to entertain myself by reading these doctorial thesis outlining one clever programming framework after another.

What’s facinating is that each of those acted as a substitute for a more formal kind of library standardization.  They filled a role in the Lisp community that standardized libraries played today in more mainstream programming communities.  This worked in part because individual developers could implement these frameworks, in part or if they were in the mood in their entirety, surprisingly quickly.  These AI languages provided a set of what we might call programming patterns today.  Each doctoral thesis sketched out huge amount of detail, but each instance of the ideas found there tended to diverge under the adaptive presure of that developer unique problem.

So while a doctoral thesis isn’t a standards specification it can act, like margarine for butter, as a substitute.  Particularly if the consumers can stomach it.  Lisp programmers like to eat whole frameworks.

Fun and Lazy.

Leave a reply

I love reading Lisp blogs; for example in this posting we have a sketch of how to define a function G that you can use when the debugger decides you neglected to write a function F. You then bind G to the F’s function definition and tell the debugger to try again. I can’t begin to imagine how you’d do that in most languages. In Common Lisp is not even that unusual; I’ve certainly done things along these lines.

Going back into the late 1970s I’ve had a joke about how lazy evaluation can be taken to extreme lengths. You build your operating system and wait until one of your developers actually writes some code. That code invokes a operating system call. You discover you haven’t written that call yet (in fact you haven’t written anything yet). So you call the routine who’s job it is to see that code get’s written. Oh my that’s not written yet either. So the error handler for that invokes the get programmer assigned handler … which invokes the project manager handler … which invokes the HR hiring manager handler … Later when this all unwinds the program just works. So damn it, just ship it!

VMWare, SBCL, Mac

Leave a reply

There are a few ways to get a good Lisp environment under Mac OS X (including OpenMCL, SBCL, Lispworks, etc.). This is a note about yet another approach. We use VMWare to create a virtual machine, a guest. VMWare is currently free, since it’s in beta. In that guest machine we run Linux. We do that to get access to the network effect around the Linux Lisp community.

I tried doing this using FreeBSD. I like FreeBSD. But I took the immaturity of the threading on FreeBSD as a signal that the Lisp network on FreeBSD appears is weak.

I somewhat randomly picked the Gentoo variant of Linux because the Gentoo Lisp “portage” appears to be well populated. You can snarf a VMWare “virtual appliance” for Gentoo (400 megabytes). Which is good because the Gentoo folks seem to think that installing Gentoo should be a hazing ritual where in you prove you love the hardware more than your family.

All I wanted here was a Lisp process; i.e. a server. So I didn’t X, et. al. So I enabled ssh and set host only networking.

Once you have your guest Gentoo you need to bring it up to date. There are some rough instructions here. Portage is the gentoo community’s name for the infrastructure used to manage all the fun software you can install on your machine. Emerge is gentoo’s cute word for installing software and world is their word for the software on your machine. Getting your machine to be current is a two step process; which you do with the emerge command. First you need to get your portage system current, i.e. synchronize with the main line (emerge sync), and then you need to let emerge bring the world up to date (emerge world). That should be enough of a model to be able to understand what they are talking about in the gentoo wiki and doc.

It might be you could skip bringing it up to date, but you can’t skip installing the VMWare tools into the guest OS. There is doc in the VMWare manual (which you can skim), and doc on the gentoo wiki (which you must follow). You have to do this, otherwise the guest clock won’t be in sync with the host’s; and that will cause ASDF to recurse to death.

One clever feature of portage is that you can set some flags to modify the personality of the software that it installs. For example to indicate that you want doc, or gui support, etc. These flags are set in /etc/make.conf. Before installing any lisp software I set some of these to get the kind of lisp I needed. I added “threads unicode doc source” to the setting for USE in that file.

At this point you can install sbcl by doing ’emerge sbcl’. You can install slime/swank by doing ’emerge slime-cvs’.  emerge -p slime-cvs will not notice that the cvs repository has changed, so you may need to do ’emerge slime-cvs’ from time to time; to avoid having slime complain about protocol mismatching.  If you put doc in your make.conf USE, then the line ‘app-emacs/slime-cvs -doc’ in your /etc/portage/package.use; otherwise it will start huge tool chains to generate the doc.  I also have ‘dev-lisp/sbcl threads unicode -ldb’ in /etc/portage/package.use.
Presuming you want it to default to UTF-8 for sbcl external format on streams; do have ‘export LC_CTYPE=en_US.UTF-8′ in the environment before you run sbcl.
You’ll need a .sbclrc file; and it will likely want to include the usual gentoo initialization: ’emerge dev-lisp/gentoo-init’. My .sbclrc fires up swank; so I can connect to it from Aquamacs, running on the host mac. This is approximately what’s in my .sbclrc file. Note that by slamming the *loopback-interface* in swank it becomes possible for the host machine’s slime to connect into the guest machine’s swank listener; that’s safe because the vmware networking keeps the guest machine inaccessible from outside the host machine.

;;;; -*- mode: lisp -*-
(in-package "CL-USER")

;;; This is ~/.sbcl loaded by SBCL upon startup

;;; These make things nicer when debugging.
(declaim (optimize (safety 1)))
(declaim (optimize (debug 3)))

;;; Let gentoo's dev-lisp do it's thing.
(load "/etc/gentoo-init.lisp")

;;; Load swank and enable remote access.
(asdf:operate 'asdf:load-op 'swank)
(setf swank::*loopback-interface* "192.168.89.128")
(setf swank:*use-dedicated-output-stream* nil)
(setf swank:*communication-style* :fd-handler)
(swank:create-server :dont-close t :coding-system "utf-8-unix")

Once that's working you can proceed to install all the lisp packages you like; either via asdf-install or via emerge.

Note that the server is creted with a utf-8 coding system, that presumes that your emacs has utf support, fonts, and both (setf slime-net-coding-system 'utf-8-unix) (set-language-environment "UTF-8") are set before you connect.
I just use tramp, and sshfs via fuse.
All this was a lot more tedious than I'd imagined; but it works nicely now. And I have sbcl, threads, and unicode - which is a combination you can't get natively on the Mac right now. If I was a better person, if VMWare on the mac wasn't beta, and if I thought this set up had stabilized I would make a virtual appliance, a virtual lisp machine :), and post it.

Personal Note from Richard Stallman

Leave a reply

June 1984:

The Lisp Machine is a product of the efforts of many people too numerous to list here and of the former unique unbureaucratic, free-wheeling and cooperative environment of the M.I.T. Artificial Intelligence Laboratory. I believe that the commercialization of computer software has harmed the spirit which enabled such systems to be developed. Now I am attempting to build a software-sharing movement to revive that spirit from near oblivion.

Since January 1984 I have been working primarily on the development of GNU, a complete Unix-compatible software system for standard hardware architectures, to be shared freely with everyone just like EMACS. This will enable people to use computers and be good neighbors legally (a good neighbor allows his neighbors to copy any generally useful software he has a copy of). This project has inspired a growing movement of enthusiastic supporters. Just recently the first free portable C compiler compiled itself. If you would like to contribute to GNU, write to me at the address above. Restrain social decay–help get programmers sharing again

From the preface of the Lisp Machine Manual.

defun-with-cache

Leave a reply

I don’t often have this kind of fun.

(defmacro defun-with-cache (name args &body body)
  (multiple-value-bind (binding call apply?)
      (rip-apart-arglist args)
    `(let ((#1=#:cache (make-hash-table :test #'equal)))
      (setf (cache-of-function ',name) #1#)
      (defun ,name ,args
	(flet ((,name ,args ,@body))
	  (let ((#2=#:key (list ,@binding)))
	    (values-list
	     (or (gethash #2# #1#)
		 (setf (gethash #2# #1#)
		       (multiple-value-list
			   ,@(if apply?
				 `((apply #',name ,@call))
				 `((,name ,@call)))))))))))))

As I like to say: “Lisp has plenty of syntax, it just spends it’s syntax dollars to buy different things.”

Just look at the nesting: defmacro, multiple-value-bind, let, setf, defun, flet, let, values-list, or, multiple-value-list, if, and apply; it’s like the parade of the wild and crazy programming constructs. Then you have the the fun going on with punctuation characters: #=:,@’ … and people complain about Perl.

Update:

Ah, the power of many eyes. It’s got a bug. Lambda lists in Common Lisp include the not unusual feature that you can write something like this pattern (x y &optional z) and then the caller can do (f 1 2) or (f 1 2 3). And of course you can specify an default value for the optional arguements i.e. (x y &optional (z -1)). There is an unusual feature in this vicinity though; you can write (x y &optional (z -1 z-provided?)) and then your function will actually have four arguments. The fourth one, z-provided? is bound by the language as a flag telling you if z was defaulted or not.

I like that the code above used an flet to create a version of the function which as it would have been without the caching wrapped around it. But I don’t see a straight forward way to do that if I’m going to handle things like z-provided?. So I got rid of the flet.

(defmacro defun-with-cache (name args &body body)
  "Like defun, but this memoizes the function into a cache that maybe
   latter cleared."
  (multiple-value-bind (binding call)
      (rip-apart-lambda-list args)
    `(let ((#1=#:cache (make-hash-table :test #'equal)))
      (setf (cache-of-function ',name) #1#)
      (defun ,name ,args
	(let ((#2=#:key (list ,@binding)))
	  (values-list
	   (or (gethash #2# #1#)
	       (setf (gethash #2# #1#)
		     (multiple-value-list
			 (progn
			   ,@body))))))))))

That I can’t use an flet in a straight forward way is an unusual breakdown of language symmetry. The need for the apply? flag in the original is analogous, but different.

Traditional Values

Leave a reply

I love the way given a heterogeneous audience a single bit of humor can be at once all kinds of humor.

Fun Fact about Common Lisp: standard Common Lisp has no networking libraries at all, but it does have a built in function to print integers as Roman numerals — using either the new style (14=XIV) or the old-style (14=XIIII) Roman numerals.

Dave Roberts) choosing to pass over it’s author’s fun but sarcastic invective and labels it sardonic irony.

In the good old days audiences were more homogenous. It was easier then for the humorist to know his target audience. That informs the question of why it is so widely presumed that nice people don’t do humor.

Reading Old Code

3 Replies

About 25 years ago Symbolics and Lisp Machines spun off from the MIT AI lab. MIT licensed the intelectual property around a machine known as the CADR, or MIT CADR, to these companies. This was a landmark event in the history of open source. This event made it clarified a lot of people’s thinking, particularly Richard Stallman.

Licensing the CADR community’s code to private interests shifted the incentives around that Lisp community. In effect MIT transfered ownership of the community, which it did not own, to private interests. Where previously the motivations for community member were intrinsic, social, and in the final analysis about common cause afterward the motivations were explicit, commercial, and foremost monetary. The resulting boom and bust of the community was quite a ride; and just short of fatal.

Again and again over the last 25 years various parties have made a run at getting the MIT intelectual property licencing office to relinquish the rights to the CADR, to move them into the commons. At long last Brad Parker has succeed. Go Brad!

Looking at the code from today’s vantage point is kind of recreational. Most of the files are from 1980. There is one file dated 1973; making it the oldest file, who’s data seems accurate, on my machine.

Lisp machines instruction sets were sympathetic to the implementation of dynamically typed languages. With support for ephemerial garbage collection, exception handling, dynamic binding, etc. etc. The CADR was microcoded. The sources include all the microcode.

The 1970s marked the acceptance that people would get their own computer with a big display and related software. The code contains a complete window system.

While object oriented programming runs back at least as far as Simula circa 1968; it didn’t really begin to win the day over efficency until people found they were racing each other to see how fast they could build a window system. We still see people searching for the right cut point between class system and primitive types. It’s interesting to see that the class system in the CADR appears to have treating all the types all the way down as first class instances. But it looks like this code doesn’t use multiple inheritance!

When MIT did this, the larger Lisp community had lots of branch offices, each with it’s own dialect of Lisp. Common Lisp the ANSI standard dialect that attempted to reduce that diversity and create some standard around which people could rendezvous came latter. The elegance of the Common Lisp dialect, particularly the beauty of it’s original specification, was an important part of what triggered my decision to switch over to Lisp.

The dialect of Lisp used for the CADR is interesting. Consider this routine that returns a tree of cons cells representing the class hierarchy:


;Returns a tree whose leaves are instances of CLASS-CLASS
(DEFMETHOD (CLASS-CLASS :CLASS-CLASS-HIERARCHY) ()
  (CONS SELF (COND ((EQ CLASS-SYMBOL 'OBJECT-CLASS)
                    NIL)
		   ((ENTITYP SUPERCLASS)
		    (< - SUPERCLASS ':CLASS-CLASS-HIERARCHY))
		   (T (MAPCAR (FUNCTION (LAMBDA (X) (<- X ':CLASS-CLASS-HIERARCHY)))
			      SUPERCLASS)))))

Methods were apparently invoked by the function “< -" and named using symbols from the keywords package, i.e. :CLASS-CLASS-HIERARCHY. In common lisp methods are simply functions so where this code has (< - SUPERCLASS ':CLASS-CLASS-HIERARCHY) in Common Lisp you’d write (CLASS-CLASS-HIERARCHY SUPERCLASS).

In that routine the object the method is working on is lexially bound to the variable SELF as it’s fields (or slots) of the class instance. That kind of syntatic sugar has fallen out of favor. These days you’d have to explicitly bind those.

I wonder about the quote on ':CLASS-CLASS-HIERARCHY; was it already vestigial?

The sources include some amusing litter. At the time in the MIT used a networking design known as Chaosnet; a variant of Ethernet. Apparently the physical address of hosts on that net were selected so they revealed the machine’s physical location along the cable. Sort of how many nanoseconds your machine is from the terminator on the cable. I’d totally forgotten how fickle those early networks were.

I actually doubt that history would have unfolded very differently if MIT had relinquished the license back in 1980 rather than in 2005; but it’s a debatable point.

Update: John Wiseman write: “I worry that the Lisp community’s fascination with the past is mostly pathology at this point.”

Absolutely true. All tiny ethnic enclaves have that problem. Members of a diaspora can and ought to say such things. Outsiders are best advised to mind their own business.

Life Lessons

Leave a reply

Leo Simons seems to be having exactly the experiance that pushed me out of Ada and into Lisp 22 years ago.

Java in many ways is kind-of a joke (yes yes its great for some stuff). On the surface its this really type-safe, compiled, predictable language everyone is using for everything. When you dig a little deeper and look at what actually is going on in “real life”, you’ll see that there’s usually some hack to get rid of all that type safety and predictability. For example, you generate source code based on XML, or you generate object code based on XML. And of course we don’t stop there. Since its kind-of hard to sensibly specify chunks of code that are bigger than a “class” (hint: other languages have things like “modules”), we use some huge application library, which we of course configure using more XML. Nevermind that we need to load 500 megs of jars into memory to make all that happen.

Programming languages ought to allow the designer to craft his notations, type models, and execution models so they fit the problem. Not demand that the problem be force fed into the mold handed down by somebody who hadn’t a clue what your problem requires.

Problem is that if you decide to make a switch your forced to write off a vast sunk cost, a network of relationships and a fluent skill set. At the same time your stuck deciding where to jump; and you can not know the color of the grass until your living right on top of it.

domain specific languages

Leave a reply

Back in the 1970s when I first learned Unix the key idea was that you built lots of clever tools which you’d then hook up in pipelines and shell scripts to solve larger problems. Tools like grep, sed, awk where exemplars of this approach. Tools like lex and yacc (now flex and bison) made it trivial to invent new micro-languages for the problem at hand. It was common to work on systems with a dozen lexer’s and parsers. The unix macro language m4 provides another scheme for getting a micro language.

One reason I got into Lisp was that after building dozens of systems using those design patterns I began to yearn for something that would unify it all. Perl, which came later, is one example of that urge comming to bearing fruit. Two things really got me interested in Lisp; symbols and macros.

Rainer Joswig put up a video (Quicktime via BitTorrent) showing a simple example of how you can use macros in Lisp to build a micro-language. It looks like he stumbled across an article illustrating the micro-language approach and it made him sad to see how horribly complex this approach is in other languages.

The example sets up a micro-language for parsing for call records dumped from a phone switch.
You get to watch a not a-typical Lisp coding session. He grabs some example data to play with. Then he grabs the text from the article that outlines the record formats. Thru the demo that text will get reformated into the statements in the new micro-language. The largest change that happens to that text happens early because he changes all the tokens so they conform to lisp coding conventions.

He then writes a bit of code to parse one line. After testing that code he will reformat it into a Lisp macro so he can stamp out parsers for each flavor of record. This is cool part. For a nonLisp programmer the going gets a bit rough at this point. The tricks of the trade for writting macros aren’t explained and so it’s probably hard to see how the macros work. For example he uses something called “backquote” to create templates that guide the expansions. The backquote templates are denoted with a single character almost the smallest character that could be used, i.e. the single backquote. When the templates are expanded they are copied and as they are copied bits are filled into them were ever comma (and some other syntax sugar) appear.

Lots of other things are glossed over, for example that when you compile a thing like this the macros are all expanded at compile time and don’t necessarily even survive into the runtime environment.

But the final finished micro language is very small and concise so it makes a nice introduction into the kind of programming that I miss when programming in other languages.

Thanks to Zach Beane for the putting together the BitTorrent, it only took 7 minutes to download from 20+ peers in the swarm. Rainer Joswig original post is here.

Meanwhile. Can it really be true that there isn’t a version of lex/flex that handles utf-8 or 16 bit characters? That is too bizzare. It’s as if the corner stone of an entire culture of programming hasn’t made the transition to the world of international data handling.