One of the reasons Common Lisp is a marvel is because it allows the programmer to manipulate the program after parsing and before falling into the compiler. This allows the designer to create craft the programming language best suited to his problem domain and then write the program in that language. The program he writes in his custom language is then converted into a program that can be compiled into something with all the efficency of a program written in at a lower semantic level.

This is note is kind of a complementary note to the one about the value of knowing assembler language :-).

The tool provided in the language for this is called a macro. That name tends to mislead programers that are familiar with macros written at the lexical or textual level. Lisp macros get to chew on the whole program, though usually they just chew on sections of the parse tree.

When you start to write really complex lisp macros you need to walk the entire program tree. To make that possible the language, which is huge, has a very small kernel of what are known as “special forms.” By using macros to convert the entire program into these special forms we are able to let any author that comes along latter play with the code, all the way down. Thus for example if you want you could write macros that find all the asignments in a function and keep a journal of them so that the function’s execution can be reversed.

You can read more about this kind of thing in the second chapter of this three chapter paper by Richard C. Walters.

Meanwhile the first chapter is a nice example of the kind of rich package you can build on this foundation. A pretty printer so powerful that I once used it to convert Lisp forms into C code.

More about code walking here.