One project I’ve been doing while unemployeed is to review how the Common Lisp landscape has changed since I was last paying attention. And the big change is the emergance of a rapidly growing pool of libraries (900+) all distributed via Quicklisp.

My current favorite is a library called Optima. Optima adds constructs for doing rich pattern matching. It can match lists, arrays, objects, structs, strings; and if you load fare-quasiquote you can write patterns using backquote.

Let’s build a JavaScript to Parenscript translator. First we can use a Javascript parsing library.

> (ql:quickload '("optima" "fare-quasiquote" "parse-js" "parenscript"))
...
> (defpackage #:js2ps (:use #:cl #:optima #:parenscript #:parse-js))
> (in-package #:js2ps)
> (pprint (parse-js "a[3] += f(x,y)")))
(:toplevel
 ((:stat
   (:assign :+ 
      (:sub (:name "a") (:num 3))
      (:call (:name "f")
       ((:name "x") (:name "y")))))))

Then we can write a recursive function using Optima to translate that into parenscript like so:

(defun js2ps (txt)
  (labels
      ((r (x)
         (match x
           (`(:toplevel ,stms)         `(progn ,@(mapcar #'r stms)))
           (`(:stat ,s)                `(progn ,(r s)))
           (`(:assign :+ ,left ,right) `(incf ,(r left) ,(r right)))
           (`(:call ,func ,args)       `(,(r func) ,@(mapcar #'r args)))
           (`(:name ,name)             (js2ps-name name))
           (`(:num  ,n)                n)
           (`(:sub ,array ,index)      `(aref ,(r array) ,(r index)))
           (eck (error "TBD ~S" eck)))))
    (r (parse-js txt))))

Let’s try it, and then using ps* we can have parenscript translate it back into Javascript.

js2ps> (js2ps "a[3] += f(x,y)")
(progn (progn (incf (aref a 3) (f x y))))
js2ps> (ps* *)
"a[3] += f(x, y);"

You have already noticed that the patterns look just like the code for building the result. Pretty eh?

A complete translator is easy. The fiddly bits arise around handling variable bindings and around what in Parenscript are called @ and chain. There code here that does much of that.

I’ve also used Optima is to scrap web pages. It’s trivial to load and parse an page into an s-expression, just quickload “drakma” and “closure-html”, and then do (parse (http-request url) (closure-html:make-lhtml-builder)). Then let’s say you wanted to know the prices that macbook airs have sold for on ebay.

 (defun snarf-prices (page-sxpr)
  ;; for example: (snarf-prices (fetch-price-page "macbook air 13" 3))
  (labels ((recure (x)
             (match x
               (`(:div ((:class "g-b bidsold") (:itemprop "price"))
                       ,(ppcre "\\$([\\d.]+)" price))
                 (push (parse-number price) result))
               (`(,(satisfies keywordp) _ ,@children)
                (map nil #'recure children)))))
     (recure page-sxpr)
     (nreverse result))))

The second pattern match `(,(satisfies keywordp) _ ,@children) uses two new tricks. The statisfies shows how we can put arbitrary predicates into your patterns, and the _ is a wildcard.

The first pattern shows how to match strings to regular expressions. The pattern (ppcre “\\w*\\$([\\d.]+)” price) would match the string
” $647.10″ and binds price to the string “647.10”.

The code for an earlier version of this page scraping example can be seen here.

One problem with Optima is that it’s a bit addictive. Once you start using it you stop using lots of other techniques. For example I hardly ever use cl-ppcre directly anymore. For example if I want to pluck the host out of some URLs I’ll write: (match url ((ppcre "//([-_.\\w]+)/" host) host)). It also lets me write code in the style of awk or perl, i.e. with a set of pattern matches.

And I should have mentioned, it expands into pretty good raw code. I say pretty good rather than great because it doesn’t currently factor out common subexpressions.

So try it out, you’ll like it.

Summary of the new packages in the new release of Quicklisp.

big-string -- BSD 3-clause (see LICENSE)
  Big strings, similar to Java's StringBuilder.
  author: Robert Smith
  mercurial: https://bitbucket.org/tarballs_are_good/big-string
  more: https://bitbucket.org/tarballs_are_good/big-string

cl-css -- MIT-style
  Simple inline CSS generator
  author: Leo Zovic
  git: git://github.com/Inaimathi/cl-css.git
  more: https://github.com/Inaimathi/cl-css#readme

cl-curlex -- GPL
  Leak *LEXENV* variable from compilation into runtime
  git: git://github.com/mabragor/cl-curlex.git
  more: https://github.com/mabragor/cl-curlex#readme

enchant -- Public Domain
  Bindings for Enchant spell-checker library

cl-geoip -- WTFPL 2.0
  Wrapper around libGeoIP
  git: git://github.com/dasuxullebt/cl-geoip.git
  more: https://github.com/dasuxullebt/cl-geoip#readme

cl-performance-tuning-helper -- MIT
  A simple performance tuning helper tool box for Common Lisp
  author: SUZUKI Shingo
  git: git://github.com/ichimal/cl-performance-tuning-helper.git
  more: https://github.com/ichimal/cl-performance-tuning-helper#readme

tcod -- No license specified?
  Common Lisp bindings for libtcod, a truecolour terminal-emulation
  library written in C.

cl-template -- MIT
  A simple output-agnostic templating system for Common Lisp.
  git: git://github.com/alpha123/cl-template.git
  more: https://github.com/alpha123/cl-template#readme

clache -- LLGPL
  A general caching facility for Common Lisp with an API is similar
  to a hash-table.
  author: Tomohiro Matsuyama
  git: git://github.com/html/clache.git
  more: https://github.com/html/clache#readme

clinch -- BSD
  Simple 3d graphics engine for Lisp.
  author: Brad Beer (WarWeasle)
  git: git://github.com/BradWBeer/CLinch.git
  more: https://github.com/BradWBeer/CLinch#readme

clite -- ISC
  Lite weight testing framework
  git: git://github.com/lispy-stuff/clite.git
  more: https://github.com/lispy-stuff/clite#readme

clobber -- No license specified?
  A alternate approach to persistance based on transaction logging.
  git: git://github.com/robert-strandh/Clobber.git
  more: https://github.com/robert-strandh/Clobber#readme

delorean -- No license specified?
  Delorean is a time machine for unit tests
  author: Andy Chambers
  git: git://github.com/cddr/delorean.git
  more: https://github.com/cddr/delorean#readme

generators -- BSD
  A common lisp package providing python style generators based
  on delimited continuations
  git: git://github.com/AccelerationNet/generators.git
  more: https://github.com/AccelerationNet/generators#readme

gettext -- GNU Lesser General Public Licence 3.0
  A port of gettext runtime to Common Lisp
  git: git://github.com/copyleft/gettext.git
  more: https://github.com/copyleft/gettext#readme

inner-conditional -- LLGPL
  Series of macros which optimizes out the inner conditional jumping
  author: Masataro Asai
  git: git://github.com/guicho271828/inner-conditional.git
  more: https://github.com/guicho271828/inner-conditional#readme

lowlight -- MIT
  A simple and flexible syntax highlighter
  git: git://github.com/chfin/lowlight.git
  more: https://github.com/chfin/lowlight#readme

new-op -- No license specified?
  Provides (new <type>) -> new instance of type, and more.
  git: git://common-lisp.net/projects/new-op/new-op.git
  more: http://common-lisp.net/project/new-op

petit.package-utils -- MIT
  petit tool box for packaging
  author: SUZUKI Shingo
  git: git://github.com/ichimal/petit.package-utils.git
  more: https://github.com/ichimal/petit.package-utils#readme

policy-cond -- Public Domain
  A macro to insert code based on compiler policy.
  author: Robert Smith
  mercurial: https://bitbucket.org/tarballs_are_good/policy-cond
  more: https://bitbucket.org/tarballs_are_good/policy-cond

pretty-function -- No license specified?
  Scheme to enable nicer printing of closures and other pure functions.
  git: git://github.com/nallen05/pretty-function.git
  more: https://github.com/nallen05/pretty-function#readme

rectangle-packing -- LLGPL (but he's flexible, so ask)
  Code to pack rectangles into a bigger rectangle.  Useful for
  texture packing for OpenGL.
  git: git://github.com/woudshoo/rectangle-packing.git
  more: https://github.com/woudshoo/rectangle-packing#readme

stmx -- LLGPL
  Composable Software Transactional Memory
  author: Massimiliano Ghilardi
  git: git://github.com/cosmos72/stmx.git

track-best -- Free
  Macros/functions for tracking the best items.
  git: http://git.nklein.com/lisp/libs/track-best.git/

treedb -- MIT
  A hierarchical key-value-database
  git: git://github.com/chfin/treedb.git
  more: https://github.com/chfin/treedb#readme

utilities.print-items -- LLGPLv3; see COPYING file for details.
  This system provides some generic condition classes in
  conjunction with support functions and macros.
  git: git://github.com/scymtym/utilities.print-items.git
  more: https://github.com/scymtym/utilities.print-items#readme

weblocks-stores -- LLGPL
  A base for weblocks stores
  author: Olexiy Zamkoviy
  git: git://github.com/html/weblocks-stores.git
  more: https://github.com/html/weblocks-stores#readme

weblocks-utils -- Public Domain
  Utils for weblocks framework
  author: Olexiy Zamkoviy
  git: git://github.com/html/weblocks-utils.git
  more: https://github.com/html/weblocks-utils#readme

I’m enjoying Planet Lisp’s feed of new Lisp projects at github: http://planet.lisp.org/github.atom.

Long long time ago I fell into a coding convention.  Most of my little projects have a file where I define the package (or packages); and this file is loaded first.  Most projects I see at github follow this style.  But a few don’t.  Further I never ever switch packages inside of a given source file.

I think it’s time to set aside these habits.

I’m reasonably confident that both these habits arose because of Emacs limitations.  Back in the day it wasn’t particularly clever about handling the package.   I don’t think I ever worked in a version that was so limited that it required the package to be asserted in the mode line, but I certainly have worked on code bases were every file asserted the package twice, at the top, once in the mode line and once via in-package.

I certainly worked in variants of emacs that had firm limits on the in-package form; i.e. that it appear early in the file, and that you not switch packages.

The only reason this worth stating out loud is because I see a lot of little projects that consist of three files:   my-project.asd, package.lisp, and my-project.lisp.   That pattern is, I think, obsolete.  There really isn’t a good reason anymore for the package.lisp file for simple little things.

  (in-package #:cl-user)
  (defpackage #:my-little-package
    (:uses #:common-lisp))
  (in-package #:my-little-package)

  (defun my-awesome-hack ()
    ; ...
  )

While I am waffling about the value of including a mode line at this point, as shown I’m leaning toward eliminating it too.

WDYT?