Lisp at Work

Lisp is not a good fit for most of the projects I'm doing at work. Our company has few Lisp programmers, so there would be no one to help maintain any code I wrote, and no one has a Lisp environment set up on their machine. I'll sometimes do a quick prototype in Lisp, and I keep a REPL open on my machine, but I don't develop code I expect to share or deploy. Once I have a prototype or proof of concept, I'll write the real thing in a language that is more commonly used at work.

But sometimes, Lisp is such a good fit for a problem that it overrides the considerations of how well it fits into your company's ecosystem. Not often, but sometimes.

I had to migrate some builds from CircleCI 2 to CircleCI 4. We had tried (twice) and failed (twice) to do an in-place upgrade of the server, so we instead brought up CircleCI 4 in parallel and migrated the builds over. To migrate a build, we had to extract the build information from the old server and import it into the new server. The API to the old CircleCI server did not expose all the data we needed, and there was no API on the new server that let you import everything we needed.

But CircleCI is written in clojure. So you can connect to a running instance of CircleCI and open a REPL. The REPL has access to all the internal data structures. It is an easy matter to write some lisp code to extract the necessary data and print it to stdout. It is also an easy matter to write some lisp code to read some data from stdin and import it into the new server.

The migration code could be written in any language, but it had to talk to a clojure REPL, format data in such a way that the clojure REPL could read it, and parse the output from the clojure REPL, which was going to be a Lisp object. Any language with a Common Lisp reader and printer would do, but you sort of get these for free if you actually use Lisp. I knew that the migration code could be written in Lisp because I had already prototyped it.

So I wrote a migration function that would take the name of the project to be migrated. The I created a Dockerfile that would boot an instance of sbcl on a core file. I preloaded the migration code and dumped the core. I ran the Dockerfile and got an image that, when run, would migrate a single project read from the command line. I then created a server that a user could visit, enter the name of his project, and it would run the Docker image as a kubernetes job.

We migrated most of the projects this way. At one point, I wrote an additional script that would read the entire list of projects in the old server and simply print them to stdout. After we shut down the migration server, I'd get requests from people that didn't seem to understand what a deadline was. I could prime the migration script from the file and it would initialize a project on the new server with the old state that I dumped. Migration stragglers could often recover this way.

Using Lisp for the migration tool did not add risk to the company. Just using the tool did not require Lisp knowledge. Anyone that needed to understand the migration process in detail had to understand clojure anyway. The migration took place over a period of weeks, and the migration tool was shut down at the end of this period. Long term maintenance was not a concern. Users of the migration tool did not have to understand Lisp, or even know what language was being used. It was a black box kubernetes job. I got buy-in from management because the tool simply worked and solved a problem that had twice been been unsuccessfully attempted before.