Category Archives: common-lisp

Quicklisp new packages, August 2013

warren-teitelmanBut first, I was sad to see that Warren Teitelman passed away.  I first encountered the idea that all the basic operations could record how to reverse their actions, and hence you could easily implement undo, in his Programmer’s Assistant.  I’ve wondered if he actually invented undo.

Programmer’s Assistant first in BBN Lisp and then in Interlisp made a huge impression on me.  It is curious how his invention of DWIM became something of a joke over the years.

There are videos of Warren and his dog competing on his Google+ page only weeks before his heart attack.

Now back to our regularly scheduled programming…

cl-autowrap 23K -- BSD-2-Clause
  Import c2ffi specs and generate CFFI wrappers
  author: Ryan Pavlik

cl-date-time-parser 10K -- MIT License
  Parse date-time-string, and return (values universal-time fraction). Parsable date-time-format: ISO8601, 
     W3CDTF, RFC3339, RFC822, RFC2822, RFC5322, asctime, RFC850, RFC1036.
  author: Takaya OCHIAI
  git: git://

cl-ledger 253K -- No license specified?
  Port of the Ledger accounting system to Common Lisp.
  git: git://

cl-paymill 7K -- BSD, 2 clause.
  CL-PAYMILL is a common lisp interface to the Paymill
    payment service API.  See
  author: Peter Wood, email: pete_wood at

cl-string-match 106K -- BSD
  Provides implementations of the standard sub-string search (string
   matching) algorithms: brute-force, Boyer-Moore, Rabin-Karp, etc.

epigraph 18K -- BSD
  A library for representing and processing graphs (nodes and edges)
  author: Cyrus Harmon

trivial-tco 3K -- MIT
  A Common Lisp library to assist in ensuring certain code is executed with tail
    call optimizations enabled.
  author: Ralph Möritz

vgplot 9K -- GPL
  Interface to gnuplot
  author: Volker Sarodnick

Quicklisp new packages, July 2013

bt-semaphore -- MIT
  A simple semaphore class for bordeaux-threads inspired by SBCL's semaphore.
  author: Ralph Möritz

cl-dsl -- GPL
  Easily define domain specific languages
  author: Alexander Popolitov
  git: git://

cl-glfw3 -- BSD-2
  Bindings for GLFW 3.x
  author: Alex Charlton

cl-hctsmsl -- GPL v3
  Library to generate human-readable HTML and CSS forms.
  author: Mark Fedurin

cl-indeterminism -- GPL
  Codewalk the form and find variables and functions that are undefined.
  author: Alexander Popolitov
  git: git://

consix -- BSD
  Lispy Qix-like game in 7 days
  author: death
  git: git://

fs-watcher -- MIT License
  Filesystem watcher for changes
  author: Florian Margaine

lquery -- Artistic
  A library to allow jQuery-like HTML/DOM manipulation.
  author: Nicolas Hafner
  git: git://

quicksearch -- MIT License
  Quicksearch searches CL library, then outputs results at REPL.
  author: Takaya OCHIAI
  git: git://

quickutil -- BSD 3-clause
  Client for Quickutils, a new way to think about utilities.

towers -- Public Domain
  Silly geodefense clone wannabe
  author: death
  git: git://

weblocks-tree-widget -- LLGPL
  A tree widget for weblocks
  author: Olexiy Zamkoviy
  git: git://

emacs w3m about pages

If you use w3m within emacs you might be interested to know you can do this:

(defun w3m-about-foobar (&rest args)
  "Hang a emacs generated page on this url: about://foobar/"
    (format "<p>This is so exciting, you have %d buffers!</p>"
            (length (buffer-list))))
  (insert "<img src=\"\"/>")

Which means you can build entire pseudo web sites that are local to your emacs :).

I got to wondering about how I might do this because I use w3m to view Lisp documentation[1], and I have this little slime contribution (slime-documentation-search.el) that adds an easy way to search  As you can see if you look there are at least four alternatives to; and I’ve been wondering if I could puzzle out a way to create little website that would let you browse between them.

The trick above looks like it could help, but the obvious approach doesn’t work because w3m doesn’t support iframes.  Exercise for the reader, I guess.

Plot Window #3

Another batch of improvements in plot-window.  Plot-window is not, as yet, intended for other people’s use.    It provides a bridge between Lisp and assorted Javascript widgets.  The original idea was to allow a web page to be used as a display from the Lisp REPL.  So, for example, to enable you to plot data using a Javascript plotting widget.

Most of the changes are internal.  For example I’m toying with a way to manage Parenscript modules (example module here).  I’m also using the LOG4CL, which I highly recommend.

All the widgets have now been factored out into individual examples.  Previously using a widget rewrote the display page entirely.  Now they are inserted at an insertion point.  Here’s a little screen capture…



One project I’ve been doing while unemployeed is to review how the Common Lisp landscape has changed since I was last paying attention. And the big change is the emergance of a rapidly growing pool of libraries (900+) all distributed via Quicklisp.

My current favorite is a library called Optima. Optima adds constructs for doing rich pattern matching. It can match lists, arrays, objects, structs, strings; and if you load fare-quasiquote you can write patterns using backquote.

Let’s build a JavaScript to Parenscript translator. First we can use a Javascript parsing library.

> (ql:quickload '("optima" "fare-quasiquote" "parse-js" "parenscript"))
> (defpackage #:js2ps (:use #:cl #:optima #:parenscript #:parse-js))
> (in-package #:js2ps)
> (pprint (parse-js "a[3] += f(x,y)")))
   (:assign :+ 
      (:sub (:name "a") (:num 3))
      (:call (:name "f")
       ((:name "x") (:name "y")))))))

Then we can write a recursive function using Optima to translate that into parenscript like so:

(defun js2ps (txt)
      ((r (x)
         (match x
           (`(:toplevel ,stms)         `(progn ,@(mapcar #'r stms)))
           (`(:stat ,s)                `(progn ,(r s)))
           (`(:assign :+ ,left ,right) `(incf ,(r left) ,(r right)))
           (`(:call ,func ,args)       `(,(r func) ,@(mapcar #'r args)))
           (`(:name ,name)             (js2ps-name name))
           (`(:num  ,n)                n)
           (`(:sub ,array ,index)      `(aref ,(r array) ,(r index)))
           (eck (error "TBD ~S" eck)))))
    (r (parse-js txt))))

Let’s try it, and then using ps* we can have parenscript translate it back into Javascript.

js2ps> (js2ps "a[3] += f(x,y)")
(progn (progn (incf (aref a 3) (f x y))))
js2ps> (ps* *)
"a[3] += f(x, y);"

You have already noticed that the patterns look just like the code for building the result. Pretty eh?

A complete translator is easy. The fiddly bits arise around handling variable bindings and around what in Parenscript are called @ and chain. There code here that does much of that.

I’ve also used Optima is to scrap web pages. It’s trivial to load and parse an page into an s-expression, just quickload “drakma” and “closure-html”, and then do (parse (http-request url) (closure-html:make-lhtml-builder)). Then let’s say you wanted to know the prices that macbook airs have sold for on ebay.

 (defun snarf-prices (page-sxpr)
  ;; for example: (snarf-prices (fetch-price-page "macbook air 13" 3))
  (labels ((recure (x)
             (match x
               (`(:div ((:class "g-b bidsold") (:itemprop "price"))
                       ,(ppcre "\\$([\\d.]+)" price))
                 (push (parse-number price) result))
               (`(,(satisfies keywordp) _ ,@children)
                (map nil #'recure children)))))
     (recure page-sxpr)
     (nreverse result))))

The second pattern match `(,(satisfies keywordp) _ ,@children) uses two new tricks. The statisfies shows how we can put arbitrary predicates into your patterns, and the _ is a wildcard.

The first pattern shows how to match strings to regular expressions. The pattern (ppcre “\\w*\\$([\\d.]+)” price) would match the string
” $647.10″ and binds price to the string “647.10”.

The code for an earlier version of this page scraping example can be seen here.

One problem with Optima is that it’s a bit addictive. Once you start using it you stop using lots of other techniques. For example I hardly ever use cl-ppcre directly anymore. For example if I want to pluck the host out of some URLs I’ll write: (match url ((ppcre "//([-_.\\w]+)/" host) host)). It also lets me write code in the style of awk or perl, i.e. with a set of pattern matches.

And I should have mentioned, it expands into pretty good raw code. I say pretty good rather than great because it doesn’t currently factor out common subexpressions.

So try it out, you’ll like it.

Quicklisp new packages, June 2013

Summary of the new packages in the new release of Quicklisp.

big-string -- BSD 3-clause (see LICENSE)
  Big strings, similar to Java's StringBuilder.
  author: Robert Smith

cl-css -- MIT-style
  Simple inline CSS generator
  author: Leo Zovic
  git: git://

cl-curlex -- GPL
  Leak *LEXENV* variable from compilation into runtime
  git: git://

enchant -- Public Domain
  Bindings for Enchant spell-checker library

cl-geoip -- WTFPL 2.0
  Wrapper around libGeoIP
  git: git://

cl-performance-tuning-helper -- MIT
  A simple performance tuning helper tool box for Common Lisp
  author: SUZUKI Shingo
  git: git://

tcod -- No license specified?
  Common Lisp bindings for libtcod, a truecolour terminal-emulation
  library written in C.

cl-template -- MIT
  A simple output-agnostic templating system for Common Lisp.
  git: git://

clache -- LLGPL
  A general caching facility for Common Lisp with an API is similar
  to a hash-table.
  author: Tomohiro Matsuyama
  git: git://

clinch -- BSD
  Simple 3d graphics engine for Lisp.
  author: Brad Beer (WarWeasle)
  git: git://

clite -- ISC
  Lite weight testing framework
  git: git://

clobber -- No license specified?
  A alternate approach to persistance based on transaction logging.
  git: git://

delorean -- No license specified?
  Delorean is a time machine for unit tests
  author: Andy Chambers
  git: git://

generators -- BSD
  A common lisp package providing python style generators based
  on delimited continuations
  git: git://

gettext -- GNU Lesser General Public Licence 3.0
  A port of gettext runtime to Common Lisp
  git: git://

inner-conditional -- LLGPL
  Series of macros which optimizes out the inner conditional jumping
  author: Masataro Asai
  git: git://

lowlight -- MIT
  A simple and flexible syntax highlighter
  git: git://

new-op -- No license specified?
  Provides (new <type>) -> new instance of type, and more.
  git: git://

petit.package-utils -- MIT
  petit tool box for packaging
  author: SUZUKI Shingo
  git: git://

policy-cond -- Public Domain
  A macro to insert code based on compiler policy.
  author: Robert Smith

pretty-function -- No license specified?
  Scheme to enable nicer printing of closures and other pure functions.
  git: git://

rectangle-packing -- LLGPL (but he's flexible, so ask)
  Code to pack rectangles into a bigger rectangle.  Useful for
  texture packing for OpenGL.
  git: git://

stmx -- LLGPL
  Composable Software Transactional Memory
  author: Massimiliano Ghilardi
  git: git://

track-best -- Free
  Macros/functions for tracking the best items.

treedb -- MIT
  A hierarchical key-value-database
  git: git://

utilities.print-items -- LLGPLv3; see COPYING file for details.
  This system provides some generic condition classes in
  conjunction with support functions and macros.
  git: git://

weblocks-stores -- LLGPL
  A base for weblocks stores
  author: Olexiy Zamkoviy
  git: git://

weblocks-utils -- Public Domain
  Utils for weblocks framework
  author: Olexiy Zamkoviy
  git: git://

cl-interpol and parenscript

Cl-interpol adds Perl style interpolated strings to Common Lisp. For example:

#?"The result is ${(let ((y 2)) (+ x y))}"
"The result is 42"

Cl-interpol works by defining a reader macro, i.e. #?”…”. For personal reasons I don’t like reader macros so I wrote a quick macro to avoid them.

(defun interp% (str)
  "Avoid the need to use a unique read table."
  (assert (stringp str))
  (with-input-from-string (s str)
    (cl-interpol::interpol-reader s #\? nil)))

(defmacro interpolate (str)
  (interp% str))

Which lets you write:

 (interpolate "{This string appeared in package: ${(package-name *package*)}.}")

Like any self respecting Lisp code cl-interpol avoids interpreting the string at runtime, instead it converts the interpolated string into code. For example that last example expands into:

(with-output-to-string (#:g13131)
  (write-string "This string appeared in package: " #:g13131)
  (princ (progn (package-name *package*)) #:g13131)
  (write-string "." #:g13131))

I started using cl-interpol because somebody suggested it as yet another way to generate HTML. The Lisp community as a few templating languages for this kind of thing.  Who doesn’t?  The more popular of these have mimics in Parenscript.

Parenscript, you will recall, is a thin gloss over Javascript that enables you to write your Javascript using s-expressions which are then converted into Javascript before delivery to the browser.

So I wanted that for cl-interpol as well. So I wrote something that translates the expanded code into parenscript.

As in the example above the output of cl-interpol’s expansion is always a with-output-to-string form, so my hack consists of a parenscript macro for with-output-to-string which then walks the enclosing form converting it into parenscript and then into javascript. For example:

This parenscript:

(let ((x 40)) (interpolate "{How about ${(+ x 2)}.}")))

becomes this javascript:

(function () {
    var x = 40;
    return ['How about ', String(x + 2), '.'].join('');

Cl-interpol has lots of features, and I certainly do not handle things it can do. I’ve only covered cases as I need them. But I’ve found it useful.

More widgets in plot-window.

Here’s another video showing a bit more progress on my hack that allows you to use a browser as a place to display output from your Common Lisp REPL.  I’ve been playing with various javascript widgets.  Here the demo routine shows:

  1. Using JQuery to animate the color of a small rectangle.
  2. A rich WYSIWYG text editor.  This one’s called nicedit.   There are dozens of these.
  3. Mapping via mapstraction, which provides a provider independent mapping widget, in this case I’m using open street maps as a provider.
  4. A tool that does syntax highlighting, i.e. it displays source code with pretty colors.

The initial plot is unchanged from my earlier posting.  You might want to watch this full screen…

Getting these to work isn’t terribly hard, but it’s slightly harder than I’d expected since most of them don’t expect to be loaded dynamically as I’m doing here.

The code running in this demo is in the dev branch over at github, but there is an odd bug I haven’t tracked down yet. Pretty soon I’ll need to change the name 🙂


I’ve been playing with Parenscript and Websockets.  I’ve made a small useful thing.  Plot-window can be used to plot data from your Common Lisp repl.  It displays the plot into a web page; you leave this web page up as you work, the plot function revises the web page on demand.

Here’s a little screen cast. That shows how to clone it from github, load it up, start a little embedded webserver, open the display page in the browser, and then finally we make a makes a few plots from the REPL.

The actual charts are rendered by Flot, one of many Javascript charting libraries.  So you can actually make many many different kinds of charts.  (FYI Liam Healy has a posting about a more traditional approach to Lisp charting.)

Finally, this short video is a preview of how this might be extended to use a web browser as a generalized display for your lisp process, in this case a Parenscript form is evaluated in emacs which builds and animates a page (using D3JS and SVG).

Old Habits

I’m enjoying Planet Lisp’s feed of new Lisp projects at github:

Long long time ago I fell into a coding convention.  Most of my little projects have a file where I define the package (or packages); and this file is loaded first.  Most projects I see at github follow this style.  But a few don’t.  Further I never ever switch packages inside of a given source file.

I think it’s time to set aside these habits.

I’m reasonably confident that both these habits arose because of Emacs limitations.  Back in the day it wasn’t particularly clever about handling the package.   I don’t think I ever worked in a version that was so limited that it required the package to be asserted in the mode line, but I certainly have worked on code bases were every file asserted the package twice, at the top, once in the mode line and once via in-package.

I certainly worked in variants of emacs that had firm limits on the in-package form; i.e. that it appear early in the file, and that you not switch packages.

The only reason this worth stating out loud is because I see a lot of little projects that consist of three files:   my-project.asd, package.lisp, and my-project.lisp.   That pattern is, I think, obsolete.  There really isn’t a good reason anymore for the package.lisp file for simple little things.

  (in-package #:cl-user)
  (defpackage #:my-little-package
    (:uses #:common-lisp))
  (in-package #:my-little-package)

  (defun my-awesome-hack ()
    ; ...

While I am waffling about the value of including a mode line at this point, as shown I’m leaning toward eliminating it too.