Monday, March 31, 2025

Avoiding Stringly Typed Code

It can be tempting to implement certain objects by their printed representation. This is especially true when you call out to other programs and pass the parameters in command line arguments and get a result back through the stdout stream. If an object is implemented by its printed representation, then serialization and deserialization of the object across program boundaries is trivial.

Objects implemented by their printed representation are jokingly referred to as “stringly typed”. The type information is lost so it is possible to pass strings representing objects of the wrong type and get nonsense answers. There are no useful predicates on arbitrary strings, so you cannot do type checking or type dispatch. This becomes a big problem for objects created from other utilities. When you call out to a bash script, you usually get the response as stream or string.

The solution? Slap a type on it right away. For any kind of string we get back from another program, we at least define a CLOS class with a single slot that holds a string. I define two Lisp bindings for any program implemented by a shell script. The one with a % prefix is the program that takes and returns strings. Without the % it takes and returns Lisp objects that are marshaled to and from strings before the % version is called. The % version obviously cannot do type checking, but the non-% entry point can and does enforce the runtime type.

Sunday, March 30, 2025

Keep a REPL Open

I keep a REPL open at all times whether or not I’m actually doing Lisp development. It’s my assistant that evaluates random expressions for me. I’ll script up little tasks in Common Lisp rather than in Bash. When I need to rapidly prototype something larger, I’ll switch to the REPL and do it there.

At work, my REPL has accumulated a bunch of code for talking to systems like GitHub, CircleCI, and LDAP as well as our in-house tools. These are utilities for my use only. I don’t write mission critical apps in Common Lis. No one else in the company uses it, and it is more important that the code be maintainable by the rest of the team than that it be written in a language I like. So I write the mission critical code in Python, or Golang, or Java, or whatever the rest of the team is using. I keep my Common Lisp to myself. I have, however, used it to protype code that evetually ends up ported to Python or Golang.

On occasion, I’ve wanted to quickly share some functionality before I have taken the time to port it. I’ve found two ways to do this. The first is to slap a web server on it. I use Hunchentoot for this. I translate JSON to Lisp coming in to the web server and Lisp back to JSON going out. This is all you effectively need for a black-box microservice. There have been a couple of transient projects where the whole thing was not expected to be maintained for a long time and by anyone other than me, so I can just throw up a microservice and tell my colleagues to hit it with a curl command.

The second way is to create a docker image that contains the Common Lisp code and all of its dependencies. It can take a bit of work to configure a lisp setup in your environment, so having it hiding inside a docker image allows me to correctly set up the Lisp environment along with the Lisp interpreter and the rest of the code. My colleagues can just pull and run the container and it will work. Again, this is only for small, throwaway projects that no one else is expected to modify or maintain. For anything that is mission critical or is expected to be shared at some point, I write it in Python or Golang or Java, etc.

I could have written these as a series of Bash scripts or Python programs, but when you start connecting a series of these together, you quickly run into the limitations of using a pipe to talk between programs. My Lisp scripts all reside in the same address space, so they can share structured data without any fancy marshaling protocol.

Saturday, March 29, 2025

Angry Fruit Salad

I like to program in living color. My color scheme is definitely “angry fruit salad”, but there is a method to my madness.

My eyeglasses have a very strong prescription. Chromatic aberration is significant at the edges of my field of vision, so it is important that text be mostly monochromatic, or it will split into tiny glyph-shaped spectra. So my main text color is green on a black background, like a terminal from the 1970s. From there, I chose cyan for comments in the code because it is easy to read. I generally favor the warmer colors for the more “active” elements and the cooler colors for the more “passive” ones, but there are many exceptions.

I have found that my brain gets used to the colors. When something shows up in an unexpected color, it immediately look wrong, even if I don’t know why. I can leverge this effect by using a very wide variety of colors for different semantic elements. I’m not consciously aware of the semantic meaning, I can just tell if the code looks the wrong color.

So my code looks like the Vegas strip: gaudy, neon colors fighting for attention. I’m sure it would drive many people up the wall. A VSCode theme sort of based on this is available at https://github.com/jrm-code-project/VSCode-Theme.

Friday, March 28, 2025

Vibed into non-functioning

Continue vibing? Well, why not go all the way.

The AI wasn’t doing so well with the main game loop, so I gave it enough help that a blank window would come up. The window would respond to the X key being pressed in order to exit, but you could also close the window to exit as well.

I told the AI that I wanted a grid of tiles. Some tiles had mines. The remaining tiles had an integer which was the number of mines in adjacent squares. The AI wanted to load some textures from files 0.png through 8.png. I asked it to generate those files, but it didn’t want to. So I broke out Paint and generated some crude 32x32 png images of numbers, a mine, a blank, and a flag.

The AI tried to load these images directly, so I had to instruct it that you need a dependency on SDL2-image and that you can load the image on to a surface, and then you can load a texture from the surface (think of a texture as a bitmap on the GPU and a surface as a bitmap on the CPU). There were several rounds of trying the code, getting an error, and pasting the error in to the AI. As per the philosophy of vibe coding, I just accepted the suggested changes without looking at them. I did have to direct it to not to try to “use” packages because that simply introduced name conflicts.

I got to the point where I could compile and load the game so far with no errors. I was testing the code at each step. It wasn’t making much progress in so far as displaying anything, but it at least didn’t regress.

Until it did. I had vibed to the point where I got a nice black rectangle on the screen that did not display anything or respond to any input. No errors were printed. Time to debug. The problem is that I only had a vague idea of what it was doing. I wasn’t paying much attention to changes being made. I dove into the code that had been generated.

What a mess. I had my suspicions as to what was wrong. Some of the newly added code needed to use the SDL2 image library. It needs to initialize the SDL2 image library, load the surfaces, and load the textures in that order. When it exits, it has to unload things in reverse order. When I wrote my Platformer Tutorial, I wrote a set of with-... macros that would pair up loading/unloading and initialize/uninitialize steps with an unwind-protect. If you use the with-... macros, you automatically get the LIFO order of operation that you need for the code to function, and the unwind-protects make sure that the uninitialization occurs even if you error out or abort the application.

The vibed code had none of this. It didn’t know about unwind-protect. It didn’t even know about nesting. It simply tried to run the initialization code first, the inner code next, and the cleanup code after that. But it combined the code through concatenation, not composition, so the necessary LIFO properties were absent. In addition, the initialization code was not paired with the cleanup code. It was pure coincidence that a cleanup happened after an initialization. The initialization code was spread about several functions in an ad hoc manner and the cleanup code was clumped in different sections. It was spaghetti code, and you needed to analyze it carefully to determine if the code initialized things in the right order or cleaned up correctly. One obvious bug was the code destroying the surfaces while the textures were still in use.

I poked at it a little bit, but there was no easy way to massage the code into a working state. It was just too disjoint. I eventually just deleted the vibed code. Firt of all, it didn’t work. Second of all, when I removed it, I regained the lost functionality of the close box and the X key for exit. It is a bad sign when removing code increases functionality.

Vibing is 0 for 2 at this point. If I have time in the next few days, I may revisit this once again and see how much hand-holding I have to give the AI to generate a working display.

Thursday, March 27, 2025

AI Silliness

In The Cyberiad, Stanislaw Lem tells the story of two robots, Trurl and Klapaucius, who are master constructors and are quibbling friends. One day, Trurl builds an artifical intelligence. He shows it off to Klapaucius, who is unimpressed. Klapaucius gives it increasingly absurd tasks, irritating Trurl. One of the challenges is to write a poem, lofty and tragic, and full of pathos, six lines long. About a haircut. And every word beginning with the letter S. Trurl objects, but the AI is up to the task

Samson snored.
She scissored short.
Sorely shorn, Soon shackled slave, Samson sighed,
Silently scheming,
Sightlessly seeking
Some savage, spectacular suicide.

It was science fiction 60 years ago, but our current LLMs can handle this easily. I think you can guess the author and prompt I put in for this:

The Unspeakable Syntax: A Tale of Lispian Horror

It was on a night fraught with cyclopean terrors that I first delved into the accursed tomes of programming lore known to the initiated as the Common Lisp Hyperspec. I had been warned by my colleagues — wizened wretches whose trembling fingers bore the mark of years lost to the parentheses — that this was no ordinary language. But my hubris, my insatiable curiosity, drove me to confront the dark corners of syntax mankind was not meant to know.

Upon the screen unfolded an incomprehensible vista of nested parentheses — lines of code whose curvatures undulated like the tentacles of some primordial beast. They defied reason and mocked the laws of Euclidean space. The parentheses seemed to breathe, to writhe, to multiply in grotesque fractal patterns that scraped against the outer edges of my sanity.

Each ‘car’ and ‘cdr’ was not merely a construct of the language, but an invocation — arcane words that seemed to echo from the abyssal void, summoning constructs whose purposes were shrouded in madness. The recursive forms spiraled infinitely inward, a pitiless vortex from which there could be no escape. My attempts to comprehend the structures were as futile as a mortal gazing upon the boundless chaos of Azathoth, the blind idiot god at the center of all things.

The culmination of my horror came as I confronted a macro. It was a labyrinthine incantation, a concatenation of symbols that seemed to weep with ancient and forbidden knowledge. It whispered to me secrets of the universe I dared not fathom, and though I averted my gaze, its presence imprinted itself upon the fragile canvas of my mind. I knew then that this language was not designed by mere mortals. No, it was the work of entities beyond comprehension—beings for whom the manipulation of symbols is as natural as the shifting of tectonic plates.

The syntax of Common Lisp was not a tool for mankind, but a trap — an eldritch puzzle whose solution would unravel the fabric of reality itself. And now, as I type these words, I can feel the parentheses closing in, their curved forms encircling me, threatening to enclose me in an eternal recursion from which there can be no escape. May the gods have mercy on my soul, for I have gazed into the syntax, and the syntax has gazed back into me.

Wednesday, March 26, 2025

Hard Won Esoteric Knowledge

I do my development in Ubuntu-22.04 Linux running on the Windows Subshell for Linux. I recently got a laptop refresh and the latest software doesn’t run. The fix is obscure, so I thought I’d document it.

sbcl runs fine out of the box in WSL2, but I’m encountering a bug where TCP connections to one particular server are being left in the CLOSE_WAIT state indefinitely. After several minutes, I hit the limit on the number of open files.

The “right thing” would be to track down who isn’t closing the connection properly, but it’s only a few hundred connections. It appears that ulimit is set to 1024, which is pretty easy to hit with this bug. Bumping ulimit to something more reasonable is a lazy workaround. It isn’t a solution — I’m still leaking open files — but I’ll be able to leak thousands of them without having problems.

But increasing nofiles turned out to be a problem. I edited all the magic files in /etc until they all said I could have 131071 open files. When I re-started WSL, all the ways I could start a shell agreed that the ulimit was 131071, yet when I started sbcl and ran this:

(uiop:run-program "prlimit" :output *standard-output*)

RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space  67108864  67108864 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024   1048576 files
NPROC      max number of processes                62828     62828 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals          62828     62828 signals
STACK      max stack size                       8388608 unlimited bytes
NIL
NIL
0 (0 bits, #x0, #o0, #b0)

The limit was at the old value of 1024.

WSL launched sbcl without a shell, so the ulimit setting was not being run.

The solution is easy, but it took me a long time to figure it out. Not only do you need to edit all the magic in /etc, and add ulimit statements to your .bashrc, you should also add ulimit statements to your .profile, and then instruct wsl to launch your program under a login shell:

(require ’sly)
(setq sly-lisp-implementations
      ’((sbcl  ("C:\\Program Files\\WSL\\wsl.exe"
                      "--distribution-id" "{df4f07a6-2142-405c-8a6a-63f1ca3a7e8d}"
                      "--cd" "~"
                      "--shell-type" "login"
                      "/usr/local/bin/sbcl")
               )))

This bit of insanity allows me to run sbcl with 131071 open files in Linux as my inferior lisp program in a Windows Emacs running SLY. (Running Emacs under Windows gives me a way to use a modified Dvorak keyboard. I could run Emacs in the Linux subsystem, but the Wayland server is in a container and doesn’t let you modify the keyboard.)

Tuesday, March 25, 2025

Vibe Coding in Common Lisp, continued

I unwedged the AI with regard to the package system, so I asked the AI to write the game loop.

Now there are a number of ways to approach a game loop, but there is one strategy that really wins: a nested set of with-... macros. These win because they tie resource management to the dynamic execution scope. You write with-window, and upon entry to the form, a window is allocated and initialized and comes into scope during the body of the code. When the body returns, the window is destroyed and deallocated.

These with-... macros are built around allocation/deallocation pairs of primitives that are called from an unwind protect. The abstraction is the inverse of a function call: instead using a function to hide a body of code, you use a function to wrap a body of code. The body of code doesn’t hide in the callee, but is passed as an argument from the caller. One important feature of programming in this way is that resources are never returned from a function, but are only passed downwards from the allocation point. This keeps objects from escaping their dynamic scope.

The entry to the game loop consists of a few nested with-... macros that initialize the library, allocate a window, allocate a drawing context, and enter a event loop. When the event loop exits, the resources are torn down in reverse order of allocation leaving the system in a clean state.

But the AI did not use with-... macros. The code it generated had subroutines that would create a window or allocate a drawing context, but it would assign the created objects into a global variable. This means that the object is set to escape its dynamic scope when it is created. There is nothing to prevent (or even discourage) access to the object after it has been deallocated. There were no unwind-protects anywhere, so objects, once allocated, were eternal — you could never close a window.

In the large, the code was built to fail. In the small, it immediately failed. Calling conventions were not even followed. Keyword agument functions were called with positional arguments, or with an odd number of arguments, irrelevant extra arguments were passed in, the AI would pass in flags that didn’t exist. We’ll grant that the AI does not ultimately understand what it is doing, but it should at least make the argument lists superficially line up. That doesn’t require AI, a simple pattern match can detect this.

The event loop did not load, let alone compile. It referred to symbols that did not exist. We’ll we can expect this, but it needs to be corrected. When I pointed out that the symbol didn’t exist, the AI began to thrash. It would try the same symbol, but with asterisks around it. It would try a variant of the same symbol. Then it would go back and try the original symbol again, try the asterisks again, try the same variant name again, etc. There is nothing to be done here but manual intervention.

There are some macros that set up an event loop, poll for an event, disptach to some code for that event while extracting the event particulars. You can roll your own event loop, or you can just use one of pre-built macros. When the AI began to thrash on the event loop, I intervened, deleted the code it was thrashing on and put in the event loop macro. The AI immediately put back in the code I had removed and started thrashing on it again.

Again, it is clear that the AI has no knowledge at all of what it is doing. It doesn’t understand syntax or the simplest of semantics. It cannot even tell if a symbol is bound to a value. Even the most junior developer won’t just make up functions that are not in the library. The AI doesn’t consult the documentation to validate if the generated code even remotely passes a sniff test.

You cannot “vibe code” Common Lisp. The AI begins to thrash and you simply must step in to get it unwedged. It doesn’t converge to any solution whatsoever. I suspect that this is because there is simply not enough training data. Common Lisp would appear to need some semantic understanding in order to write plausibly working code. Just mimicking some syntax you found on the web (which is ultimately what the AI is doing) will not get you very far at all.

Monday, March 24, 2025

Vibe coding in Common Lisp

Can you “vibe code” in Common Lisp?

Short answer, no.

I set out to give it a try. The idea behind “vibe coding” is to use an AI to generate the code and to blindly accept everything it generates. You offer some feedback about error messages and whether the code is doing something close to what you want, but you specifically don’t try to analyze and understand the code being generated. You certainly don’t try to debug it and fix it.

A.I. is trained on a lot of open source code. If your language is popular, there is a lot of code to train on. Chances are, if you have a problem, then not only has someone attempted to code it up in Python, but several people have given it a try and someone has ported it to JavaScript. Someone has solved your problem on StackOverflow.

Common Lisp is not a popular language. There is not a lot of code to train on, and most of it is someone’s homework. So for any particular problem, the A.I. doesn’t have a lot to go on. It becomes clear pretty quickly that the A.I. has no understanding of what it is doing, it is just trying to synthesize something that is similiar to what it has seen before, and if it hasn’t seem much, you don’t get much.

So I tried to vibe code in Common Lisp. I decided to try to write a Minesweeper game. That seemed like it had probably been done enough times before that the A.I. might be able to generate some vibe code.

I told it that we were going to write a video game and that it should generate an asd file for the game, and some skeleton code that would be a good start. It generated an asd file and four small files: main.lisp, game.lisp, input.lisp, and graphics.lisp. There was little actual code in the files, just some function stubs, but you could see where the cross-file dependencies were going to be.

The asd file wouldn’t load. The generated files had some minor dependencies upon each other and imported the required symbols from the other files. This imposed an implicit load order because a file couldn’t be loaded until the file it depended on had created the package for the symbols that it referenced. This is an old problem in Common Lisp, and the solution is to set up all the packages before loading anything. But I was vibing, so I told the AI that the forward references of the symbols were causing problems.

The AI added require statements into the files to try get them to load in a working order. It didn’t help. require statements have a whole bunch of issues and aren’t used very much thes days. The modern solution is to make the dependencies explicit in the system definition file. I gave the AI a direct instruction to make sure that the package.lisp file loaded before any other. Rather than edit the asd file, it decided to add even more require statements.

I declared failure at this point and manually edited the package.lisp file to create the packages and import the inital symbols, I added a dependecy on the package.lisp file to every other file, and I removed all the spurious require statements. It was clear the AI was not going to hit upon this solution.

The AI obviously has no model of what the package system is. It doesn’t reason that you need to load it first. It simply knows that it can insert require statements to express a dependency. So it thrashes around added require statements in the hope that it will converge to a solution. It converged to a circular dependency instead.

Sunday, March 23, 2025

The Obarray

Early Lisps interned all their symbols in a single symbol table called the obarray. Every program you loaded into your Lisp image would share the obarray. Memory was limited, so you usually only ran one program (like Macsyma) at a time.

But as memory got larger and cheaper, people started to want to run multiple Lisp programs, like Macsyma and Emacs, at the same time. The problem was they would collide over the use of the symbols. (In particular, over the property lists.) Someone — I’m not sure who — came up with a hack that would swap out the obarray depending on which program you were loading.

The origin of the package system was "the obarray hack". Packages are first-class named obarrays, with some support for controlling the sharing of symbols among the obarrays, a limited form of inheritance and some code that maintains consistency.

In any but the smallest Lisp project, you need to decide on a symbol and package strategy. Some people keep with the original spirit of the package system and create just a handful of coarse-grained packages that each encompass a logical program. Other people use packages as modules, which gives you a set of many fine-grained packages, one to each module in your system.

I favor the former approach. I either use one package for my entire program, or I break it into just a couple of main packages. I don’t try for a finer grained approched. The package system wasn’t really designed for a fine-grained appoach.

Screwing up your packages can easily make your system unusable. If you dynamically create and link packages as you load your code, you have to be careful about the order in which you load your files. Load a file out of order and you’ll end up with dozens of symbols interned in the wrong package.

When I’m working on a project, I always have a file called package.lisp which defines all the packages in the project in one place. The package.lisp is always the first file loaded. Once I’ve done this, then the order in which the other files are loaded becomes far less important. It saves a lot of headaches.

Friday, March 21, 2025

CLOS vs. message passing

When I was in high school and first getting interested in computers, I heard about the concept of message passing. This struck me as a good idea, although I couldn't articulate why at the time. Now I know a bit more about computers, and I often find that message passing is a good way to structure certain kinds of programs. In particular, it really works well with client/server architectures.

The idea behing message passing is that you have active agents that communicate by sending passive messages amongst themselves. A message is a fairly simple piece of data. It is basically a symbolic token and maybe some parameters. The recipient of the message interprets the meaning of the token and acts accordingly. Conceptually, the interface is narrow: an agent exposes one endpoint and all messages come through that endpoint. This facilitates the creation of strong abstraction barriers.

The standard way of implementing this is to have a reverse proxy within the agent that disatches messages to the appropriate handler within the object. The object has a table of message handlers and it looks up the appropriate handler in the table and calls it.

I wanted to use this paradigm for talking to several heterogeneous servers — a GitHub server, an LDAP server, a CircleCI server, etc. But I got bogged down in the details of how to implement this. It was proving difficult to map the reverse proxy implementation on to CLOS. But then I remembered something: synchronous message passing is isomorphic to simple function calls. I didn't want to implement a message dispatcher in CLOS, I could just use CLOS's built-in method dispatch.

Messages are just the names of generic functions, and parameterized messages are just generic functions that take arguments. The method dispatch table doesn't reside in the object but in the generic function. In fact, very little is left of the object itself. It can often be instance with no slots that only has an identity.

Once I got my head straightened out, the code came together quickly.

Wednesday, March 19, 2025

Object Last or First

In Java or C#, the first argument to a method call is the object that contains the method. This is built in to the language and it is to some extent an artifact of the mechanism of method dispatch. So when you add a type (class) to the language, you will have a set of methods where the first argument is the object of that type.

In Common Lisp, the first argument to a function is not special. When you add a type, the functions that operate on that type can place the object anywhere in the argument list. The convention in Common Lisp is (mostly) for the object to be the last argument in the argument list. Look at the list and sequence functions for examples of this convention.

The Common Lisp convention reads more like English. (subst new old sequence) reads directly as "substitute new for old in sequence". In Java, the same method would be called like this sequence.subst(old, new), which would have to read as "In sequence, substitute for old, new", which is a bit more awkward.

But I think I prefer the Java convention. In Lisp, this would be (substitute sequence old new). There is no implementation need for the object to be the first argument, but I think there is an advantage to the regularity of the convention. It places the object in the same place in the argument list for all the functions that operate on the object.

The argument that has persuaded me the most is that if the return type of the function is the same type as the object, as it often is, then you can elegantly chain the method calls together with a fold-left. So consider a table object. It might have an insert method that takes a key and value and returns a new table. If the insert method is like this: (insert table key value), then you can insert a bunch of keys and values with a fold-left like this: (fold-left #’insert table ’(key1 key2 ...) ’(value1 value2 ...)).

Note how order of arguments is analagous between the fold-left and the insert method. When the object is the last argument, then you have to insert an intermediate lambda expression to shuffle the arguments around, and the table argument moves from being after the key and value in the insert method to being before the key list and value list in the fold-left method. It is a small thing, but I find it very appealing that in moving from the single argument case to the argument list case we don’t have random changes.

Of course I don’t think we should change Common Lisp to conform to a different convention, but I tend to write my own functions with the object as the first argument rather than the last.

Tuesday, March 18, 2025

Just Keep Consing

Lisp was the first garbage collected language. But early garbage collectors were painful to use. They had significant overhead and would pause your program for several seconds or even minutes at the worst possible times. People tried to avoid garbage collection by re-using objects, using allocation pools, etc. Many people would run their Lisp programs with the garbage collection turned off. They would reboot their Lisp machines when they ran out of memory. Lisp Machine Inc. had a product called "Picon" which was carefully crafted to avoid any runtime allocation.

Generational garbage collectors began to be adopted in the early 80s. Generational collectors have much less overhead than the earlier "Stop the world" collectors. Memory has gotten much cheaper, so larger heaps are practical. Large heaps have two benefits: garbage collection becomes less frequent, and objects have time to "age" and perhaps become garbage before the next generational collection. Some garbage collection algoriths have no cost overhead for very short-lived objects.

It is no longer necessary to re-use objects or try to avoid allocating memory. Garbage collection pauses are usually short enough to be unnoticeable. You can typically set the heap size nice and large and forget about it. It is certainly possible to encounter a program that has a pathological memory usage pattern, but it is much less common than it used to be.

Because of the way linked lists work, the result of walking down a list usually comes out in the reverse order. In the old days, you would make the effort of trying to accumulate the result in the forward direction by keeping track of the last cell in the answer and mutating it to accumulate the next cell. This is a pain. These days, it you can just accumulate the result in the reverse order and then call reverse when you are done. In practice, this is no slower than accumulating the result in the forward direction, but certainly a lot simpler. It generates more garbage, but it is short-lived garbage with little or no overhead.

Monday, March 17, 2025

Series vs. streams

A generator is an abstraction of a sequence of values. It is a procedure that returns the next value in the sequence each time it is invoked. The generator can run out of items to return at some point if the sequence is finite, or it can keep generating values if the sequence is ininite.

A generator is decidely non-functional. Each time it is called it has the potential to return a different value. But let's make it functional. Instead of returning a single value, let's return two values: the next value in the sequence and the next state of the generator. The generator can now be pure functional and return the exact same two values each time. The caller will keep track of the current generator will replace the current generator with the next one returned by the call.

We implement generators as a promise that returns a pair of the next value and the next generator. The returned pair is what S&ICP call a stream. In other words, a stream is output of a functional generator that is 180 degrees out of phase of the generator.

Streams are similar to series in that you can write computations that operate on the aggregate stream, but it will be piplined to operate one element at time. But rather than having the compiler perform a code walk to set up an explicit pipeline, the runtime system sets up an implicit pipeline through the constraints of the promises. This makes streams a bit more flexible than series.

Series are more efficient than streams because the compiler can turn the implicit pipeline into an explicit one that is easy to optimize. Streams turn into a series of nested lexical closures with the attendant overhead.

One of the difficulties in using streams is that you often have to pay very careful attention to avoid fencepost errors and generating elements one beyond what is necessary. This isn't just a matter of using up a tad more storage, but it can lead to unexpected infinite loops because you attempt to reach one beyond the base case. Very often you find that you need two versions of each function: one that takes a stream argument, and one that takes a generator argument that you are careful to avoid calling unless necessary.

Streams are lazy by nature. Laziness introduces a need for static types. If you have a computed value, you can examine it to find out its type, but if you have a promise, you cannot tell what type of object it will return without forcing the promise. You cannot do a type dispatch on a promise because you don't know what it will return. A static type would indicate the type of the returned value without forcing the promise.

Series requires that the entire pipeline from source to sink be visible to the compiler. Streams do not have this requirement.

Despite their drawbacks, I rather like streams. I use them in my linear-fractional-transformations package to represent exact real numbers as streams of linear fractional transformations. I also use streams of integers to represent the continued fraction expansion of exact real numbers.

Sunday, March 16, 2025

Universal Function

Lisp was an early language. These days everyone and his brother has a new language, but Lisp was the first of its kind. John McCarthy, mathematician that he was, wanted to prove that his new language was universal. He broke this down into two steps.

First, he showed that S-expressions — the list structure representation of Lisp — could faithfully represent Church’s lambda expressions. This is kind of taken for granted now, but McCarthy made the effort to prove it. Church had already proven that lambda expressions could represent any computable function, so McCarthy had a proof that S-expressions, too, could represent any computable function.

Then, he showed that his language could implement a universal function. A universal function is a function that can emulate any other function. If you have a universal function, you can emulate any other function, so you can compute any computable function. A universal function takes two arguments, a specification of what function to emulate and (a list of) some inputs. It returns the same value as if the function had been called with those inputs.

McCarthy’s universal function took a function specification in the form of a lambda expression and a list of arguments. It binds the arguments to the formal parameters of the lambda expression, the performs a recursive descent evaluation of the body of the body of the lambda expression. McCarthy called his universal function APPLY. By writing APPLY in Lisp, McCarthy showed that Lisp was universal. (EVAL began its existance as a helper function for APPLY).

To tell the truth, this is pretty studly: McCarthy proved that his new language was universal by writing the first meta-circular evaluator in it. These days, people invent languages by throwing together enough features until they have something that looks like a language. It’ll probably be universal — universality turns out to be fairly easy to achieve — but how do you know? If you can write a Lisp interpreter in your language, it’s universal.

Saturday, March 15, 2025

Obscure suggestions

Suppose you have come up with an elegant recursive algorithm that is easy to understand and implement. This will not do. A true mathematician is judged by how clever he must be to understand his algorithm. To that end, you must make your algorithm as difficult to understand as possible. This is how you prove that you are smarter than your readers. Here are some suggestions:

  • Instead of giving the next state as function of the current state, give the current state as a function of the next state and let your audience invert the function.
  • Split your recursion into two parts, but give one part recursively and the other co-recursively. Your readers will enjoy the fun puzzle of figuring out how to stitch the parts back together.
  • Remove the recursion by replacing it with re-assignment and explicit stack manipulation.
  • Avoid motivating examples.
  • Omit all unnecessary details, and a few of the necessary ones as well.
  • Unicode gives you thousands of single character variable names.
  • Use existance proofs rather than constructive ones. You can prove there is a base case without explicitly stating what it is.
  • Let X refer to a set or an element of a set, depending on context.
  • Depend on the context. A lot.
  • There is no rule that says variable names must be unique.

Take and apply some of these ideas and you can turn your elegant algorithm into something that will humiliate the smartest of your readers.

Friday, March 14, 2025

Defclass vs. defstruct

Common Lisp provides two ways to create new compound data types: defstruct and defclass. Defstruct creates simple cartesian record types, while defclass is part of a full object-oriented programming system. How do you decide which one to use?

It’s easy. Unless you have a compelling reason to use defstruct, just use defclass. Even if you don’t use any other features of CLOS, defclass better supports class redefinition, and this just makes life easier.

If you modify a defstruct and recompile it, the old instances of that struct type become obsolete. They probably won’t work with the new definition. You’ll most likely have to rebuild them. If things get too screwed up, you’ll end up having to restart your Lisp image.

CLOS, on the othe hard, is designed to be dynamic. You can redefine and recompile a class on the fly. You can change the class of an instance. As you develop your code, you’ll be adding and removing slots and changing the class hierarchy. defclass usually handles these sorts of dynamic changes transparently, without having to restart your Lisp image.

CLOS achieves this by adding an extra level of indirection, and perhaps you cannot tolerate the extra overhead. Then by all means use defstruct. But if you are indifferent, defclass is a better choice.

Thursday, March 13, 2025

Tip: Alphabetize arbitrary lists

Whenever I have a list of items, if there is no other better order for them, I arrange them in alphabetical order. Arbitrary lists have a way of getting large and unweildy over time, but if they are kept in alphabetical order, you can find the entries and spot omissions easier.

If there is a better ordering, then certainly use it. But keeping arbitrary lists alphabetized has two advantages: first, they are easier to use because you can find entries quicker. Second, it is a signal to the reader that the list is in fact in an arbitrary order.

Wednesday, March 12, 2025

with-slots vs. with-accessors

Most CLOS objects are implemented as standard-instances. A standard-instance is a collection of storage cells called slots, and the slots are addressed by name. You could imagine an alternative implementation where an instance is a vector that is addressed by an integer, but named slots are more flexible and abstract.

Many object systems map the named fields of an instance into lexically scoped variables. Within a method body, you can just refer to the slot as if it were a variable. Assignment to the variable effectively updates the slot. There are pros and cons to this. On the plus side, it is very convenient to refer to slots as if they were variables. On the minus side, it is difficult to rename a slot, because you have to rename all the references to it, and slot names can collide with lexical variables. It can make the code brittle with regard to slot naming. But CLOS lets you choose if you want to do this or not. The with-slots macro installs a set of symbol macros that let you refer to each slot as if it were a variable.

But the slots of an instance are an implementation detail. You really want an abstract API for your objects. You want logical fields to be accessed by getter and setter functions. The logical field will typically be backed by a slot, but it could be a computed value. Logical fields are more flexible and abstract than slots.

When you define a slot, you can specify a :reader and :accessor function for that slot. This covers the very common use case of a getter/setter pair that is backed by a slot in the instance.

You can also map the logical fields of an instance into lexical variables. The with-accessors macro installs a set of symbol macros that let you refer to each logical field as if it were a lexical varible.

I often see with-slots used where with-accessors would be more appropriate. If you find yourself wanting to use with-slots, consider if you should be using with-accessors instead.

Personally, I prefer to avoid both with-slots and with-accessors. This makes CLOS objects act more like structs. Structs are easier for me to understand than magic lexical variables.

Tip

The accessors for slots are generic. You therefore want them to have generic names. For example, suppose you have a point class with an x and y slot. You don't want to call your accessors point-x and point-y because the names would be inappropriate for subclasses. You want to have names something like get-x and get-y. These functions would naturally work on subclasses of points, but because get-x and get-y are generic, you could also extend them to work on any class that has a meaningful x and y.

Tuesday, March 11, 2025

Symbol macros

A symbol macro is a symbol that macroexpands into a lisp form. It is similar to a preprocessor macro in C, but it must expand into a syntactically complete expression. Symbol macros are the underlying technology behind the with-slots and with-accessors macros. They allow you to introduce an identifier that appears to be a lexical variable, but actually executes some arbitrary code to compute a value. So we can place the storage for a variable in a vector or in an instance, and use a symbol macro to make it appear to be an ordinary variable.

Gerry Sussman doesn't like symbol macros. They are a lie. It appears that you are just doing an ordinary variable access, which should be a quick and simple operation, but in fact you could be executing arbitrary code. This can lead to some nasty suprises.

But in my opinion, you shouldn't discard a useful tool just because there is a way to misuse it. If your symbol macro is just redirecting a variable to a slot in an instance, there is little harm in that.

Monday, March 10, 2025

Advanced CLOS — update-instance-for-changed-class

Like most object systems, instances in CLOS have a reference to their class. Unlike most most object systems, CLOS provides a protocol for changing that reference. Normally, this is a pretty insane thing to want to do. It effectively changes the class of the instance and it is pretty unlikely that the instance structure will be compatible with the new class. But there are two situations where you might want to do it anyway:

  • When you edit the class definition, you can arrange for the system to dynamically upgrade existing instances to the new class definition. This means you won't have to restart your lisp and rebuild all the instances from scratch. You can just reload the class definition and the instances will be seamlessly upgraded on the fly. This is much more pleasant experience for the developer.
  • While you normally don't want to change the class of an instance at runtime, there are some rare situations where it can make sense. A good example is the unified table interface. Instances are thin wrappers around a concrete table implementation. It makes sense to change a table instance from one concrete implementation to another. For instance, you might want to change a hash table to a assocation list. You can simply call change-class on the instance.

When the class changes, the representation will be wrong. This is where we add an :after method to update-instance-for-different-class:

(defmethod update-instance-for-different-class :after ((previous alist-table) (current plist-table) &rest initargs)
  (declare (ignore initargs))
  (setf (representation current) (alist-plist (representation previous))))
  
  ...etc...
> (defvar *foo* (make-instance 'alist-table :initial-contents '((:a . 420) (:b . 69))))
#<ALIST-TABLE 2 EQL>

> (representation *foo*)
((:A . 420) (:B . 69))

;; But I'd rather have a plist-table
  
> (change-class *foo* 'plist-table)
#<PLIST-TABLE 2 EQL>

> (representation *foo*)
(:a 420 :b 69)

;; And now I'd like a wttree-table

> (change-class *foo* 'wttree-table)
#<WTTREE-TABLE 2 EQUAL>

> (representation *foo*)
#(2 NIL :A #(1 NIL :B NIL 69) 420)

Naturally, you have to be judicious in your use of this feature of CLOS. You can easily construct nonsense objects. But some times it makes perfect sense,

Sunday, March 9, 2025

Unified table interface

On day 16 of the Advent of Code, I make use of a priority queue for Dijkstra's algorithm. I ported Stephen Adams's weight-balanced binary tree implementation from MIT Scheme to Common Lisp. Stephen Adams's implementation (and therefore my port of it) has the MIT license. Weight-balanced binary trees are a way to implement key-value maps with these properties:

  • The trees are immutable. This means that when you add or remove a key, you get a new tree with the change. The old tree is unchanged. This makes the trees easier to reason about and suitable for functional programming. For example, you can iterate over the tree without having to worry about mutating the tree during the iteration.
  • Most operations on the tree, and insertion, lookup, and deletion in particular, are O(log n). While theoretically not as fast as a hash table, n has to be quite large before log n becomes a big factor. In practice, a weight balanced binary tree is competitive with a hash table for any reasonably sized table.
  • Weight-balanced binary trees support set operations such as union, intersection, and difference. These operations run in O(log n) time as well.
  • Keys are stored in sorted order. This makes it easy to iterate from smallest to largest key (or in the direction).

But it occurred to me that I wanted a unified abstract interface to all the various table-like data structures that Common Lisp provides. You should be able to call a generic table/lookup on a property list, association list, hash table, or weight-balanced binary tree and have it do the right thing. I wrote a simple table package that provides this.

https://github.com/jrm-code-project/table

The package is documented in the `README.md` fie.

Saturday, March 8, 2025

Advent of Code 2024: Day 25

On day 25, we are given a set of locks and keys as ascii art. A typical lock looks like this:

.....
.#...
.##.#
.##.#
###.#
#####
#####

and a typical key looks like this:

#####
#####
##.#.
##.#.
##.#.
#..#.
.....

We read the input file with a little state machine that accumulates lines until a blank line or end of file is reached. It decides whether what it read was a lock or a key by looking to see if the first row is all #'s or not. If it is, it's a key, otherwise it's lock.

(defun read-input (pathname)
  (let ((package (find-package "ADVENT2024/DAY25")))
    (with-open-file (stream pathname)
      (let iter ((line (read-line stream nil))
                 (accum '())
                 (locks '())
                 (keys '()))
        (if line
            (let ((char-list (map 'list (lambda (c) (intern (string c) package)) line)))
              (if (null char-list)
                  (let ((item (make-grid (length accum) (length (first accum))
                                         :initial-contents (reverse accum))))
                    (if (every (lambda (s) (eq s '\#)) (first accum))
                        (iter (read-line stream nil)
                              '()
                              locks
                              (cons item keys))
                        (iter (read-line stream nil)
                              '()
                              (cons item locks)
                              keys)))
                  (iter (read-line stream nil)
                        (cons char-list accum)
                        locks
                        keys)))
            (let ((item (make-grid (length accum) (length (first accum))
                                   :initial-contents (reverse accum))))
              (if (every (lambda (s) (eq s '\#)) (first accum))
                  (values (reverse locks) (reverse (cons item keys)))
                  (values (reverse (cons item locks)) (reverse keys)))))))))

A key fits into a lock (but doesn't necessarily open it) if none of the '#'s in the key overlap with the '#'s in the lock. This is easily checked by iterating over the key and lock in parallel and ensuring that at least one of the characters is '.'.

(defun fits? (key lock)
  (collect-and (#M(lambda (k l)
                    (or (eql k '|.|) (eql l '|.|)))
                  (scan 'array key)
                  (scan 'array lock))))

For part 1, we are asked to find the number of key/lock combinations that result in a fit. We use map-product from the alexandria library to map the fits? predicate over the cartesian product of keys and locks. We then count the number of fits.

(defun part-1 ()
  (multiple-value-bind (locks keys) (read-input (input-pathname))
    (count t (map-product #'fits? keys locks))))

There is no part 2 for this problem.


We've arrived at the end of the 2024 Advent of Code. I started this series with two intents: to demonstrate an approach to solving the problems that is more idiomatic to Common Lisp, and to learn more about the series library. I don't claim my solutions are the best. They could all use some improvement, and I'm sure you code golfers can find numerous ways to shave strokes. But I think each solution is fairly reasonable and tries to show off how to effectively use Common Lisp in a number of simple prolems.

For these problems I purposefully avoided the loop macro and tried to use the series library as much as possible. I used named-let for the more complex iterations.

I was ultimately disappointed in series. I like the idea of automatically generating pipelines from a more functional style, but it simply hits the complexity wall far too quickly. For simple iterations, it's great, but for anything even slightly more complex, it becomes difficult to use.

The full source code I wrote is available on GitHub at https://github.com/jrm-code-project/Advent2024 Be aware that I have not included the puzzle input files. The code will not run without them. You can download the puzzle inputs from the Advent of Code website and put them in the appropriate directories, each in a file called input.txt

I'm curious to hear what you think of my solutions. If you have any comments or suggestions, please feel free to contact me via email or by leaving a comment.

Friday, March 7, 2025

Advent of Code 2024: Day 24

In day 24, we are given a set of equations that decribe some combinatorical logic. The first task is to read the input and parse out the combinatoric circuit and simulate it. To do this, I hijack the lisp reader. I create a readtable this is just like the standard Lisp readtable, but with these differences:

  • Case is not folded.
  • The colon character is no longer a package prefix marker, but rather a terminating macro character that inserts the token :colon into the stream.
  • The newline character is no longer a whitespace character, but rather a terminating macro character that inserts the token :newline into the stream.

These changes to the reader make it esay to parse the input file. We build a labels expression where each named quantity in the circuit (the wires) is a function of zero arguments. Simulating the solution is then just a matter of calling eval on the resulting expression.

(defun get-input (swaps input-pathname)
  (flet ((maybe-swap (symbol)
           (cond ((assoc symbol swaps) (cdr (assoc symbol swaps)))
                 ((rassoc symbol swaps) (car (rassoc symbol swaps)))
                 (t symbol))))

    (let ((*readtable* (copy-readtable nil)))
      (setf (readtable-case *readtable*) :preserve)
      (set-syntax-from-char #\: #\;)
      (set-macro-character #\: (lambda (stream char) (declare (ignore stream char)) :colon))
      (set-macro-character #\newline (lambda (stream char) (declare (ignore stream char)) :newline))

      (with-open-file (stream input-pathname :direction :input)
        (let iter ((token (read stream nil :eof))
                   (line '())
                   (gates '())
                   (wires '())
                   (outputs '()))
        
          (multiple-value-bind (line* gates* wires* outputs*)
              (if (or (eq token :eof) (eq token :newline))
                  (if line
                      (if (member :colon line)
                          (values '()
                                  gates
                                  (cons `(,(third line) () ,(first line)) wires)
                                  outputs)
                          (values '()
                                  (cons `(,(maybe-swap (first line)) ()
                                          (,(ecase (fourth line)
                                              (XOR 'logxor)
                                              (OR 'logior)
                                              (AND 'logand))
                                           ,@(list (list (third line)) (list (fifth line)))))
                                        gates)
                                  wires
                                  (if (and (symbolp token)
                                           (char= (char (symbol-name token) 0) #\z))
                                      (cons `(list ,(list token)) outputs)
                                      outputs)
                                  ))
                      (values '() gates wires outputs))
                  (values (cons token line) gates wires (if (and (symbolp token)
                                                                 (char= (char (symbol-name token) 0) #\z))
                                                            (cons (list token) outputs)
                                                            outputs)))
            (if (eq token :eof)
                `(labels (,@wires*
                          ,@gates*)
                   (fold-left (lambda (acc bit)
                                (+ (* 2 acc) bit))
                              0  (list ,@(sort outputs* #'string-greaterp :key (lambda (term) (symbol-name (car term)))))))
                (iter (read stream nil :eof) line* gates* wires* outputs*))))))))

For part 2, we are told that the circuit is supposed to add two binary numbers. We are also told that the circuit the circuit has four of its wires swapped. We are asked to find the swapped wires.

It is hard to understand what is going on because almost all the wires have random three-letter names. We start by renaming the wires so that they have a bit number prefixed to with them. If a gate has two numbered inputs where the numbers are equal, we propagate the number to the output of the gate.

Once the wires are numbered, we sort the wires by their numbers and print the wire list. The regular pattern of gates is instantly obvious, and the swapped wires are easy to spot. It isn't obvious how to find the swapped wires in the general case, but it is unnecessary to solve the puzzle, so there is no code for this.

Thursday, March 6, 2025

Advent of Code 2024: Day 23

For day 23 we’re going to look for cliques in a graph. A clique is a subset of vertices in a graph such that every pair of vertices in the clique is connected by an edge. In other words, a clique is a complete subgraph of the graph.

The graph is given as a list of edges. The graph is undirected, so the edge (a, b) is the same as the edge (b, a). We represent the graph as a hash table mapping vertices to a list of adjacent vertices.

;;; -*- Lisp -*-

(in-package "ADVENT2024/DAY23")

(defun get-input (input-pathname)
  (let ((neighbor-table (make-hash-table :test #’eql))
        (package (find-package "ADVENT2024/DAY23")))
    (iterate (((left right) (#2M(lambda (line) (values-list (str:split #\- line)))
                                (scan-file input-pathname #’read-line))))
      (let ((left*  (intern (string-upcase left)  package))
            (right* (intern (string-upcase right) package)))
        (push right* (gethash left* neighbor-table ’()))
        (push left* (gethash right* neighbor-table ’()))))
  neighbor-table))

Given a neighbor table, we can get a list of the two vertex cliques by looking at the keys and values of the hash table.

(defun two-vertex-cliques (neighbor-table)
  (collect-append
   (mapping (((vertex neighbors) (scan-hash neighbor-table)))
     (mappend (lambda (neighbor)
                (when (string-lessp (symbol-name vertex) (symbol-name neighbor))
                  (list (list vertex neighbor))))
              neighbors))))

Given a two vertex clique, we can find a three vertex clique by looking for a vertex that is connected to both vertices in the two vertex clique. We find the neighbors of each vertex in the clique and then take the intersection of the two lists of neighbors. We distribute this intersection over the two vertex clique to get the list of three vertex cliques. Note that each three vertex clique will appear three times in the list in different orders.

In Part 1, we count the number of three vertex cliques in the graph where one of the vertices begins with the letter ‘T’. We divide by three because we generate three vertex cliques in triplicate.

(defun part-1 ()
  (/ (count-if (lambda (clique)
                 (find-if (lambda (sym)
                            (char= #\T (char (symbol-name sym) 0)))
                          clique))
               (let ((neighbor-table (get-input (input-pathname))))
                 (mappend (lambda (clique)
                            (let ((left-neighbors (gethash (first clique) neighbor-table))
                                  (right-neighbors (gethash (second clique) neighbor-table)))
                              (map ’list (lambda (common-neighbor) (list* common-neighbor clique))
                                   (intersection left-neighbors right-neighbors))))
                          (two-vertex-cliques neighbor-table))))
     3))

For Part 2, we are to find the largest maximal clique. We use the Bron-Kerbosch algorithm to find the maximal cliques.

(defun bron-kerbosch (graph-vertices clique more-vertices excluded-vertices)
  (if (and (null more-vertices) (null excluded-vertices))
      (list clique)
      (let iter ((answer '())
                 (excluded-vertices excluded-vertices)
                 (more-vertices more-vertices))
        (if (null more-vertices)
            answer
            (let* ((this-vertex (car more-vertices))
                   (more-vertices* (cdr more-vertices))
                   (neighbors (gethash this-vertex graph-vertices)))
              (iter (append (bron-kerbosch graph-vertices
                                           (adjoin this-vertex clique)
                                           (intersection more-vertices* neighbors)
                                           (intersection excluded-vertices neighbors))
                            answer)
                (adjoin this-vertex excluded-vertices)
                more-vertices*))))))

(defun maximal-cliques (graph-vertices)
  (bron-kerbosch graph-vertices ’() (hash-table-keys graph-vertices) ’()))

Once we have found the maximal cliques, we can find the largest clique by sorting the cliques by length and taking the first one. We sort the vertices in the clique and print as a comma separated list.

(defun part-2 ()
  (format
   nil "~{~a~^,~}"
   (sort
    (first
     (sort
      (maximal-cliques (get-input (input-pathname)))
      #’> :key #’length))
    #’string-lessp :key #’symbol-name)))

Wednesday, March 5, 2025

Advent of Code 2024: Day 22

On Day 22 we are introduced to a simple pseudo-random number generator (PRNG) that uses this recurrance to generate pseudo-random numbers:

S1 = ((Xn << 6) ⊕ Xn) mod 224
S2 = ((S1 >> 5) ⊕ S1) mod 224
Xn+1 = ((S2 << 11) ⊕ S2) mod 224

We just define this as a simple function, but we are carful to put a check-type on the input to make sure it is a number in the correct range. This gives the compiler enough information to optimize the body of the generator to a sequence of inline fixed-point operations, avoid the overhead of a function call out to the generic arithmetic.

(defun next-pseudorandom (pseudorandom)
  (check-type pseudorandom (integer 0 (16777216)))
  (macrolet ((mix (a b) ‘(logxor ,a ,b))
             (prune (x) ‘(mod ,x 16777216)))
    (let* ((s1 (prune (mix (* pseudorandom 64) pseudorandom)))
           (s2 (prune (mix (floor s1 32) s1)))
           (s3 (prune (mix (* s2 2048) s2))))
      s3)))

We can generate a series of random numbers from a given seed:

(defun scan-pseudorandom (seed)
  (declare (optimizable-series-function))
  (scan-fn '(integer 0 (16777216))
           (lambda () seed)
           #'next-pseudorandom))

The nth pseudorandom number is the nth element in the series, i.e. the result of applying the next-pseudorandom function n times to the seed:

(defun nth-pseudorandom (seed n)
  (collect-nth n (scan-pseudorandom seed)))

Part 1 of the problem is to sum the 2000th pseudorandom numbers generated from seeds given in a file.

(defun part-1 ()
  (collect-sum (#Mnth-pseudorandom (scan-file (input-pathname)) (series 2000))))

For part 2, we're going to be simulating a market. The prices are single digit pseudorandom numbers:

(defun scan-prices (seed)
  (declare (optimizable-series-function))
  (#Mmod (scan-pseudorandom seed) (series 10)))

The bidders in our market are monkeys, and we read them from our input file:

(defun scan-monkeys (input-pathname)
  (declare (optimizable-series-function 2))
  (cotruncate (scan-range :from 0)
              (scan-file input-pathname)))

The seed that we read from the input pathname will be used to create a price series for each monkey.

Each monkey looks for trends in the market by looking at the last four price changes. If the last four prices changes match the trend the monkey looks for, the monkey will make a trade and get a profit of the current price.

For part 2, we assume all the monkeys look for the same trend. Some trend will maximize the total profit of all the monkeys. We want to know what that maximum profit is.

We'll proceed in two steps. First, we make a table that maps trends to profits for each monkey. We'll start with an empty table, then we'll iterate over the monkeys, adding the trend info for that monkey. Once we have the table, we'll iterate over all the possible trends and find the one that maximizes the total profit.

price-deltas is a series of the differences between the prices in the price series. We'll use this to determine the trend.

(defun price-deltas (price-series)
  (declare (optimizable-series-function)
           (off-line-port price-series))
  (mapping (((before after) (chunk 2 1 price-series)))
     (- after before)))

price-trends is a series of trends. The trend is simply a list of the last four price deltas.

(defun price-trends (price-series)
  (declare (optimizable-series-function)
           (off-line-port price-series))
  (mapping (((d1 d2 d3 d4) (chunk 4 1 (price-deltas price-series))))
           (list d1 d2 d3 d4)))

add-trend-info! adds the trend info for a monkey to the table. We'll look at a count of 2000 prices (minus the first four because there aren't enough to establish a trend). The key to an entry in the table will be taken from the price-trends. The value for an entry is the price after that trend. The table maps a trend to an alist that maps monkeys to profits, so once we know the trend, we look to see if an entry for the monkey already exists in the value. If it does, we're done. But if it doesn't, we add an entry for the monkey with the profit.

(defun add-trend-info! (table monkeyid seed)
  (iterate ((count (scan-range :from 4 :below 2001))
            (trend (price-trends (scan-prices seed)))
            (value (subseries (scan-prices seed) 4)))
    (declare (ignore count))
    (unless (assoc monkeyid (gethash trend table '()))
      (push (cons monkeyid value) (gethash trend table '())))))

Once we have added the trend info for all the monkeys, we find the entry in the table that maximizes the total profit.

(defun trend-table-maximum (table)
  (let ((best-score 0)
        (best-key nil))
    (maphash (lambda (key value)
               (let ((score (reduce #'+ (map 'list #'cdr value))))
                 (when (> score best-score)
                   (setq best-key key)
                   (setq best-score score))))
             table)
    (values best-key best-score)))

Finally, we can put it all together in the part-2 function:

(defun part-2 ()
  (multiple-value-bind (best-key best-value)
      (let ((table (make-hash-table :test #'equal)))
        (iterate (((monkeyid seed) (scan-monkeys (input-pathname))))
          (add-trend-info! table monkeyid seed))
        (trend-table-maximum table))
    (declare (ignore best-key))
    best-value))

Tuesday, March 4, 2025

Collate / index-list

I was talking to Arthur Gleckler last night and he mentioned that he had been making good use of a function he called index-list. This function takes two selector functions and a list of objects. The first selector extracts a key from each object, and the second selector extracts a value. A table is returned that maps the keys to a list of all the values that were associated with that key.

I had to laugh. I had written the same function a few month back. I called it collate.

Here is Arthur’s version in Scheme:

(define (index-list elements table choose-data choose-key)
  "Given a hash table ‘table’, walk a list of ‘elements’ E, using
‘choose-key’ to extract the key K from each E and ‘choose-data’ to
extract a list of data D from each E.  Store each K in ‘table’ along
with a list of all the elements of all the D for that K."
  (do-list (e elements)
    (hash-table-update!/default
     table
     (choose-key e)
     (lambda (previous) (append (choose-data e) previous))
     ’()))
  table)

And here is my version in Common Lisp:

(defun collate (list &key (key #’car) (test #’eql)
                               (merger (merge-adjoin :test #’eql)) (default nil))
  (let ((table (make-hash-table :test test)))
    (dolist (element list table)
      (let ((key (funcall key element)))
        (setf (gethash key table)
              (funcall merger (gethash key table default) element))))))

So how do they differ?

  • Arthur’s version takes the hash table as a parameter. This allows the caller to control the hash table’s properties. My version creates a hash table using the test parameter, which defaults to eql.
  • Arthur’s version uses choose-key to extract the key from each element. My version uses key, which is a keyword parameter defaulting to car. My choice was driven by the convention of Common Lisp sequence functions to take a :key parameter.
  • Arthur’s version uses a default value of ’() for the entries in the hash table. My version uses the :default keyword argument that defaults to ’().
  • Arthur’s version uses choose-data to extract the datum in each element. My version uses the :merger keyword argument to specify how to merge the entire element into the table. If you only want a subfield of the element, you can compose a selector function with a merger function.
  • Arthur’s version uses append to collect the data associated with each element. My version uses a merger function to merge the element into the entry in the hash table. The default merger is merge-adjoin, which uses adjoin to add the element to the list of elements associated with the key. merge-adjoin is paramterized by a test function that defaults to eql. If the test is true, the new data is not merged, so the result of (merge-adjoin #’eql) is a list with no duplicates.
  • If you instead specify a default of 0 and a merger of (lambda (existing new) (+ existing 1)) you get a histogram.
  • Another merger I make use of is merge-unique, which ensures that all copies of the data being merged are the same, raising a warning if they are not.
  • Finally, I occasionally make use of a higher-order merger called merge-list that takes a list of mergers and applies them elementwise to two lists to be merged. This allows you to create a singleton aggregate merged element where the subfields are merged using different strategies.

Like Arthur, I found this to be a very useful function. I was auditing a data set obtained from GitHub. It came in as a flat list of records of users. Each record was a list of GitHub org, GitHub ID, and SAML/SSO login. Many of our users inadvertently have multiple GitHub IDs associated with their accounts. I used my collate function to create a table that mapped SAML/SSO login to a list of all the GitHub IDs associated with that login, and the list of orgs where that mapping applies.

Advent of Code 2024: Day 21

For day 20, we are entering a combination on a numeric keypad. But we cannot just enter the combination, we have to direct a robot to enter the combination by entering the directions to move the robot. But we cannot enter the directions directly, we have to get another robot to enter the directions to move the first robot. Part 1 of the problem has two layers of robots, but part 2 has a cascade of 25 layers of robots.

The door we need to unlock has a numeric keypad, but each robot has a directional keypad. The A key is an ’enter’ key.

;;; -*- Lisp -*-

(in-package "ADVENT2024/DAY21")

(defparameter *numeric-keypad* #2a(( 7  8  9)
                                   ( 4  5  6)
                                   ( 1  2  3)
                                   (nil 0  A)))

(defparameter *directional-keypad* #2a((nil |^| A)
                                       ( <  |v| >)))

(defun read-input (input-pathname)
  (collect ’list
    (#M(lambda (line)
         (collect ’list
           (#M(lambda (c)
                (or (digit-char-p c)
                    (intern (string c) (find-package "ADVENT2024/DAY21"))))
              (scan ’string line))))
       (scan-file input-pathname #’read-line))))

Given a keypad, we can find the coordinates of a key by scanning for it.

(defun key-coords (keypad key)
  (let ((coords (scan-grid-coords keypad)))
    (collect-first
     (choose
      (#Meql
       (#Mgrid-ref (series keypad) coords)
       (series key))
      coords))))

To move the robot arm, we’ll jog it vertically or horizontally by pressing keys on the directional keypad.

(defun jog-x (dx)
  (make-list (abs dx) :initial-element (if (minusp dx) ’< ’>)))

(defun jog-y (dy)
  (make-list (abs dy) :initial-element (if (minusp dy) ’|^| ’|v|)))

A valid two-dimensional jog must never go over the dead key.

(defun valid-jog? (keypad from jog)
  (let iter ((current from)
             (jog jog))
    (cond ((null (grid-ref keypad current)) nil)
          ((null jog) t)
          (t (iter (ecase (car jog)
                     (|^| (coord-north current))
                     (|v| (coord-south current))
                     (>   (coord-east  current))
                     (<   (coord-west  current)))
               (cdr jog))))))

Given the coords of a from key and a to key on a keypad, we can compute the ways to jog the arm from to to. There may be more than one way, so we return a list of the ways to jog the arm. Zig-zag jogging is never going to be optimal, so we omit that option.

(defun jog-xy (keypad from to)
  (let ((dx (jog-x (- (column to) (column from))))
        (dy (jog-y (- (row to) (row from)))))
    (cond ((null dx) (list dy))
          ((null dy) (list dx))
          (t (let ((column-first (append dx dy))
                   (row-first    (append dy dx)))
               (cond ((and (valid-jog? keypad from column-first)
                           (valid-jog? keypad from row-first))
                      (list column-first row-first))
                     ((valid-jog? keypad from column-first)
                      (list column-first))
                     (t (list row-first))))))))

In the general case, we’ll get a list of two possibilities. Either we move vertically first or we move horizontally first. One of these possibilities will lead to the shortest sequence of inputs. Oftentimes we can prune this to one possibility, e.g. we are keeping in the same row or column, or one possibility would take us over the dead key.

Instead of using coords, we would like to specify the key names.

(defun step-paths (keypad start-key end-key)
  (jog-xy keypad (key-coords keypad start-key) (key-coords keypad end-key)))

Given a target sequence we want a robot to enter into a keypad, we want to compute sequences on the robots directional keypad that we can enter to cause the robot to enter the target sequence. There will be multiple possibilities, and we want any of the shortest ones. Notice that last thing entered in a sequence is the A key, so we can assume the robot is starting from that key having pressed A in the prior sequence.

This is where we insert a memoization cache to control the combinatoric explosion that will occur when we cascade robots.

(defparameter seq-paths-cache (make-hash-table :test #’equal))

(defun seq-paths (keypad sequence)
  (if (eql keypad *numeric-keypad*)
      (seq-paths-1 keypad sequence)
      (let* ((key sequence)
             (probe (gethash sequence seq-paths-cache :not-found)))
        (if (eq probe :not-found)
            (let ((answer (seq-paths-1 keypad sequence)))
              (setf (gethash key seq-paths-cache answer) answer)
              answer)
            probe))))

(defun seq-paths-1 (keypad sequence)
  (cartesian-product-list
   (butlast (maplist (lambda (tail)
                       (cond ((null tail) nil)
                             ((null (cdr tail)) nil)
                             (t (revmap (lambda (jog)
                                          (append jog (list ’a)))
                                        (jog-xy keypad
                                                (key-coords keypad (first tail))
                                                (key-coords keypad (second tail)))))))
                     (cons ’a sequence)))))

Given the ultimate sequence we want to end up typing on the ultimate keypad, we want to move up through the cascade of robots generating meta sequences that drive the robot on the next level down. This produces a combinatoric explosion. But the puzzle doesn’t care about the actual sequence of keys, only that the number of keystrokes, is minimal, so we keep at each level the keystrokes for each target key, but we can ignore the order in which the robot presses the target keys. At each level of the robot cascade, we will know, for example, that we have to enter "move up, press A" some thirty-two times in total. This means that the robot one level up will have thirty-two copies of the "move left, press A, move right, press A" meta-sequence.

The meta sequences can be fragmented at each press of the A key and then we can count each fragment individually. So we only need to know the meta sequence for a handful of fragments to determine the number of keystrokes needed to enter a sequence. This is kept in our memoization table.

But there are multiple meta-sequences that can be expanded from a sequence. If they have different lengths, we want one of the shortest ones, but even among the shortest ones of the same length, the next level of expansion may produce meta-meta-sequences of different lengths. We can use a clever trick to prune the longer meta-meta-sequences. We pre-load the memoization cache to avoid returning alternatives that create large expansions two level up in the cascade. So now when we compute the meta-sequence we won’t compute so many alternative possibilities, but only possibilites that do not expand to longer solutions if run through the computation twice. There are eleven of these:

(defun preload-cache ()
  (clrhash seq-paths-cache)
  (setf 
   (gethash ’(|v| A)     seq-paths-cache) ’(((<  |v| A)   (^ > A)))
   (gethash ’( <  A)     seq-paths-cache) ’(((|v| < < A) (> > ^ A)))

   (gethash ’(|^|  >  A) seq-paths-cache) ’(((< A)     (|v| > A) (^ A)))
   (gethash ’(|v|  >  A) seq-paths-cache) ’(((< |v| A)     (> A) (^ A)))
   (gethash ’( >  |^| A) seq-paths-cache) ’(((|v| A)     (< ^ A) (> A)))
   (gethash ’( <  |^| A) seq-paths-cache) ’(((|v| < < A) (> ^ A) (> A)))
   (gethash ’( <  |v| A) seq-paths-cache) ’(((|v| < < A)   (> A) (^ > A)))

   (gethash ’(|v|  <   <  A) seq-paths-cache) ’(((< |v| A)     (< A)       (A) (> > ^ A)))
   (gethash ’( >   >  |^| A) seq-paths-cache) ’(((|v| A)         (A)   (< ^ A) (> A)))

   (gethash ’( >  |v| |v| |v| A) seq-paths-cache) ’(((|v| A)   (< A) (A)   (A) (^ > A)))
   (gethash ’(|v| |v| |v|  >  A) seq-paths-cache) ’(((< |v| A)   (A) (A) (> A) (^ A)))))

With the cache preloaded with these values, we always generate meta-sequences that have minimal keystrokes, but furthermore, the meta-meta-sequences will also have minimal keystrokes.

The rest of the file generates meta sequences up the cascade of robots.

(defun next-seq-tables (seq-table)
  (remove-duplicates (collapse-seq-tables (next-seq-tables-1 seq-table)) :test #’equal))

(defun collapse-seq-tables (seq-tables)
  (revmap #’collapse-seq-table seq-tables))

(defun symbol-lessp (left right)
  (string-lessp (symbol-name left) (symbol-name right)))

(defun term-lessp (left right)
  (or (and (null left) right)
      (and (null right) nil)
      (symbol-lessp (car left) (car right))
      (and (eql (car left) (car right))
           (term-lessp (cdr left) (cdr right)))))

(defun collapse-seq-table (seq-table)
  (let ((table (make-hash-table :test #’equal)))
    (dolist (entry seq-table)
      (let ((key (car entry))
            (count (cdr entry)))
        (incf (gethash key table 0) count)))
    (sort (hash-table-alist table) #’term-lessp :key #’car)))

(defun next-seq-tables-1 (seq-table)
  (if (null seq-table)
      (list (list))
      (let ((tail-tables (next-seq-tables-1 (cdr seq-table))))
        (extend-seq-tables (car seq-table) tail-tables))))

(defun extend-seq-tables (entry tail-tables)
  (revmappend (lambda (tail-table)
             (extend-seq-table entry tail-table))
           tail-tables))

(defun extend-seq-table (entry tail-table)
  (revmap (lambda (path)
            (extend-with-path path (cdr entry) tail-table))
          (seq-paths *directional-keypad* (car entry))))

(defun extend-with-path (path count tail-table)
  (append (revmap (lambda (term) (cons term count)) path)
          tail-table))

(defun seq-table-length (seq-table)
  (reduce #’+ (map ’list (lambda (entry) (* (length (car entry)) (cdr entry))) seq-table)))

The initial-paths-table takes the target numeric sequence and produces a table of the sequence fragments to enter that sequence. Order is not presevered.

(defun initial-paths-table (numeric-seq)
  (map ’list (lambda (path)
                (let ((table (make-hash-table :test #’equal)))
                  (dolist (term path (hash-table-alist table))
                    (incf (gethash term table 0)))))
       (seq-paths *numeric-keypad* numeric-seq)))

We generate the table for a generation by iteratively calling next-seq-tables until we reach the number of robots in the cascade.

(defun generation-table (n numeric-seq)
  (if (zerop n)
      (initial-paths-table numeric-seq)
      (revmappend #’next-seq-tables (generation-table (1- n) numeric-seq))))

(defun shortest-table (sequence-tables)
  (car (sort sequence-tables #’< :key #’seq-table-length)))

Finally, we can compute the complexity of the sequence by counting the number of keypresses in the shortest sequence and multiplying by the code in the sequence.

(defun complexity (code n-generations)
    (* (seq-table-length (shortest-table (generation-table n-generations code)))
       (fold-left (lambda (acc digit)
                    (if (eql digit ’a)
                        acc
                        (+ (* acc 10) digit)))
                  0
                  code)))

And we can compute the answer to part 1 and part 2 with a cascade of two robots and a cascade of twenty-five robots respectively.

(defun part-1 ()
  (reduce #’+ (map ’list (lambda (input) (complexity input 2)) (read-input (input-pathname)))))

(defun part-2 ()
  (reduce #’+ (map ’list (lambda (input) (complexity input 25)) (read-input (input-pathname)))))

Monday, March 3, 2025

Advent of Code 2024: Day 20

For day 20, we return to a maze problem. The maze involved, however, is trivial — there are no decision points, it is just a convoluted path.

;;; -*- Lisp -*-

(in-package "ADVENT2024/DAY20")

(defun read-input (input-pathname)
  (read-file-into-grid
    (char-interner #’identity (find-package "ADVENT2024/DAY20"))
     input-pathname))

(defun find-start-and-goal (maze)
  (let ((inverse (invert-grid maze ’|.|)))
    (values (car (gethash ’S inverse))
            (car (gethash ’E inverse)))))

We compute the distance to the goal at all points along the path by walking the path backwards.

(defun compute-distances (maze)
  (let ((distances (make-grid (grid-height maze) (grid-width maze)
                              :initial-element nil)))
    (multiple-value-bind (start goal) (find-start-and-goal maze)
      (declare (ignore start))
      (let iter ((current goal)
                 (distance 0))
        (when current
          (setf (grid-ref distances current) distance)
          (iter (let* ((neighbors (#M2v+ (scan ’list (list +north+ +south+ +east+ +west+))
                                     (series current)))
                       (fill? (#M(lambda (maze neighbor)
                                   (and (on-grid? maze neighbor)
                                        (not (eql (grid-ref maze neighbor) ’\#))
                                        (null (grid-ref distances neighbor))))
                                 (series maze)
                                 neighbors)))
                  (collect-first (choose fill? neighbors)))
                (1+ distance))))
      distances)))

When we run through the maze we are allowed to cheat just once by walking through a wall. For part 1, we can walk just one step through a wall, but for part 2, we can walk up to 20 steps ignoring the walls. We might as well combine the two solutions into a single parameterized function. We will be asked to count the number of cheats that shorten the path by at least 100 steps.

I tried for quite some time to come up with a series oriented way to solve this, but it turned out to be much easier to just write a named-let iterative loop. So much for series.

First, we have a function that finds the cheats for a specific location. We are given a grid of distances to the goal, a coord that we start from, the current distance to the goal, the number of steps we can take through the walls, and the number of steps we have to shave off to count this cheat.

We iterate in a square grid centered at the current location and twice as wide plus one as the cheat steps. Check the locations in the distance grid that fall within the square and this tells us how much closer to the goal we can get by cheating to that location. We have to add in the manhattan distance from the current location to the cheat location to get the total distance. Subtract that from the original distance to the goal and we have the number of steps we save by using this cheat. If it exceeds our threshold, we count it.

(defun scan-square-coords (size)
  (declare (optimizable-series-function))
  (let ((displacement (coord size size)))
    (#M2v- (scan-coords (1+ (* size 2)) (1+ (* size 2)))
           (series displacement))))

(defun count-location-cheats (distances coord distance cheat-steps threshold)
  (collect-sum
   (choose
    (mapping (((cheat-vec) (scan-square-coords cheat-steps)))
      (let ((manhattan-distance (+ (abs (column cheat-vec)) (abs (row cheat-vec))))
            (cheat-coord (2v+ coord cheat-vec)))
        (and (<= manhattan-distance cheat-steps)
             (on-grid? distances cheat-coord)
             (let ((cheat-distance (grid-ref distances cheat-coord)))
               (and cheat-distance
                    (let* ((distance-if-cheating (+ manhattan-distance cheat-distance))
                           (savings (- distance distance-if-cheating)))
                      (and (>= savings threshold)
                           1))))))))))

So then we just iterate over the locations in the distance grid and call this function for each location, summing the results.

(defun count-cheats (distances-grid cheat-steps threshold)
  (collect-sum
   (choose
    (mapping (((coord distance) (scan-grid distances-grid)))
      (and distance
           (count-location-cheats distances-grid coord distance cheat-steps threshold))))))

For part 1, we can only take two steps through a wall.

(defun part-1 ()
  (count-cheats (compute-distances (read-input (input-pathname))) 2 100))

For part 2, we can take up to 20 steps through a wall.

(defun part-2 ()
  (count-cheats (compute-distances (read-input (input-pathname))) 20 100))

Sunday, March 2, 2025

Advent of Code 2024: Day 19

For day 19, we are constructing sequences from fragments. We are first given a list of fragments, separated by commas. For example:

r, wr, b, g, bwu, rb, gb, br

The we are given a series of sequences that we need to construct by concatenating the fragments. For example:

brwrr  = br + wr + r
bggr   = b + g + g + r
;;; -*- Lisp -*-

(in-package "ADVENT2024/DAY19")

(defun read-input (input-pathname)
  (let ((parsed
          (collect 'list
            (#M(lambda (line)
                 (map 'list #'str:trim (str:split #\, line)))
               (scan-file input-pathname #'read-line)))))
    (values (first parsed) (map 'list #'first (rest (rest parsed))))))

Our job is to determine if the sequences can be constructed from the fragments. This is an easy recursive predicate:

(defun can-make-sequence? (fragments sequence)
  (or (zerop (length sequence))
      (some
       (lambda (fragment)
         (multiple-value-bind (prefix? suffix)
             (starts-with-subseq fragment sequence :return-suffix t)
           (and prefix?
                (can-make-sequence? fragments suffix))))
      fragments)))

Part 1 is to determine how many of the sequences can be constructed from the fragments.

(defun part-1 ()
  (multiple-value-bind (fragments sequences) (read-input (input-pathname))
    (count-if (lambda (sequence)
                  (can-make-sequence? fragments sequence))
              sequences)))

Part 2 is to count the number of ways we can construct the sequences from the fragments. Naively, we would just count the number of ways we can construct each sequence using each of the fragments as the first fragment and then sum them.

(defun count-solutions (fragments sequence)
  (if (zerop (length sequence))
      1
      (collect-sum
        (#M(lambda (fragment)
             (multiple-value-bind (prefix? suffix)
                 (starts-with-subseq fragment sequence :return-suffix t)
               (if prefix?
                   (count-solutions fragments suffix)
                   0)))
          (scan 'lists fragments)))))

But the naive approach won’t work for the larger input. The combinatorics grow far too quickly, so we need to be more clever. One possible way to do this is with “dynamic programming”, but most of the times I've seen this used, it involved a table of values and you had to invert your solution to fill in the table from the bottom up. But this is unnecessarily complicated. It turns out that “dynamic programming” is isomorphic to simple memoized recursive calls. So we won't bother with the table and inverting our solution. We'll just add some ad hoc memoization to our recursive count-solutions:

(defparameter *count-solutions-cache* (make-hash-table :test 'equal))

(defun count-solutions (fragments sequence)
  (let ((key (cons fragments sequence)))
    (or (gethash key *count-solutions-cache*)
        (setf (gethash key *count-solutions-cache*)
              (if (zerop (length sequence))
                  1
                  (collect-sum
                    (#M(lambda (fragment)
                         (multiple-value-bind (prefix? suffix)
                             (starts-with-subseq fragment sequence :return-suffix t)
                           (if prefix?
                               (count-solutions fragments suffix)
                               0)))
                      (scan 'list fragments))))))))

(defun part-2 ()
  (multiple-value-bind (fragments sequences) (read-input (input-pathname))
    (collect-sum
     (#M(lambda (sequence)
          (count-solutions fragments sequence))
        (scan ’list sequences)))))

This runs at quite a reasonable speed.

Saturday, March 1, 2025

Advent of Code 2024: Day 18

For day 18, we have a maze again, but this time the input is given as coordinate pairs of where the walls go. The start and goal are the upper left and lower right respectively.

(in-package "ADVENT2024/DAY18")

(defun read-input (file grid n-bytes)
  (iterate ((coord (#M(lambda (line)
                       (apply #’coord (map ’list #’parse-integer (str:split #\, line))))
                      (cotruncate (scan-file file #’read-line)
                                  (scan-range :below n-bytes)))))
    (setf (grid-ref grid coord) ’\#))
  (setf (grid-ref grid (coord 0 0)) ’|S|)
  (setf (grid-ref grid (coord (1- (grid-height grid)) (1- (grid-width grid)))) ’|E|))

(defun sample-input ()
  (let ((grid (make-array (list 7 7) :initial-element ’|.|)))
    (read-input (sample-input-pathname) grid 12)
    grid))

(defun input (n-bytes)
  (let ((grid (make-grid 71 71 :initial-element ’|.|)))
    (read-input (input-pathname) grid n-bytes)
    grid))

The bulk of the solution simply reuses the Dijkstra’s algorithm from day 16. I won’t reproduce the code here. We just adjust the path scorer to not penalize for turns.

For part 1, we load the first 1024 walls and find a shortest path.

(defun part-1 ()
  (let* ((grid (input 1024))
         (solutions (solve-maze grid)))
    (score-path (car solutions))))

For part 2, we want to find the first wall in the list of walls that prevents us from reaching the goal. Binary search time.

(defun total-walls ()
  (collect-length (scan-file (input-pathname) #’read-line)))

(defun binary-search (pass fail)
  (if (= (1+ pass) fail)
      (list pass fail)
      (let* ((mid (floor (+ pass fail) 2))
             (grid (input mid)))
        (let ((solutions (solve-maze grid)))
          (if (null solutions)
              (binary-search pass mid)
              (binary-search mid fail))))))

(defun get-coord (n)
  (collect-nth n (scan-file (input-pathname) #’read-line)))

(defun part-2 ()
  (collect-nth (car (binary-search 1024 (total-walls)))
  (scan-file (input-pathname) #’read-line)))