REPL + Prompt

I'm interested in LLMs and I'm into Lisp, so I wanted to explore ways to combine the two. Earlier, I wrote a pseudocode macro that uses an LLM to expand pseudocode into Common Lisp code. I also wrote an autodoc feature that uses an LLM to generate docstrings if you don't provide them yourself. These are two examples of Lisp calling into the LLM. Naturally, I wanted to see what would happen if we let the LLM call into Lisp.

We can provide Lisp “tools” to the LLM so that it can have an API to the client Lisp. Some of these tools are simply extensions to the LLM that happen to written in Lisp. For example, the random number generator. We can expose user interaction tools such as y-or-n-p to allow the LLM to ask simple y/n questions. But it is interesting to add Lisp introspection tools to the LLM so it can probe the Lisp runtime.

There is an impedance mismatch between Lisp and the LLM. In Lisp, data is typed. In the LLM, the tools are typed. In Lisp, a function can return whatever object it pleases and the object carries its own type. An LLM tool, however, must be declared to return a particular type of object and must return an object of that type. We cannot expose functions with polymorphic retun values to the LLM because we would have to declare the return type prior to calling the function. Furthermore, Lisp has a rich type system compared to that of the LLM. The types of many Lisp objects cannot easily be declared to the LLM.

We're going to live dangerously and attempt to give the LLM the ability to call eval. eval is the ultimate polymorphic function in that it can return any first-class object. There is no way to declare a return type for eval. There is also no way to declare the argument type of s-expression. Instead, we declare eval to operate on string representations. We provide a tool that takes a string and calls read-from-string on it, calls eval on the resulting s-expression, and calls print on the return value(s). The LLM can then call this tool to evaluate Lisp expressions. Since I'm not completely insane, I put in a belt and suspenders check to make sure that the LLM does not do something I might regret. First, the LLM is instructed to get positive confirmation from the user before evaluating anything that might have a permanent effect. Second, the tool makes a call to yes-or-no-p before the actual call to eval. You can omit this call to yes-or-no-p by setting *enable-eval* to :YOLO.

It was probably obvious to everyone else, but it took me a bit to figure out that maybe it would be interesting to have a modified REPL integrated with the LLM. If you enter a symbol or list at the REPL prompt, it would be sent to the evaluator as usual, but if you entered free-form text, it could be sent to the LLM. When the LLM has the tools capable of evaluating Lisp expressions, it can call back into Lisp, so you can type things like “what package am I in?” to the modified REPL, and it will generate a call to evaluate “(print *package*)”. Since it has access to your Lisp runtime, the LLM can be a Lisp programming assistant.

The modified REPL has a trick up its sleeve. When it gets a lisp expression to evaluate, it calls eval, but it pushes a fake call from the LLM to eval on to the LLM history. If a later prompt to is given to the LLM it can see this call in its history — it appears to the LLM as if it had made the call. This allows the LLM to refer to the user's interactions with Lisp. For example,

;; Normal evaluation
> (+ 2 3)
5

;; LLM command
> add seven to that
12

The LLM sees a call to eval("(+ 2 3)") resulting in "5" in its history, so it is able to determine that the pronoun “that” in the second call refers to that result.

Integrating the LLM with the REPL means you don't have to switch windows or lose context when you want to switch between the tools. This streamlines your workflow.