LLM in the Debugger

There is one more place I thought I could integrate the LLM with Lisp and that is in the debugger. The idea is to have the LLM have gander at the error and suggest a fix before you get dropped into the debugger as usual. The mechanism is pretty straightforward. You use the debugger hook to call the LLM with the error message and a backtrace and tell it you'd like a diagnosis and a suggested fix. You also tell it that it can use its tools to inspect the Lisp environment. Then you cross your fingers and hope that the LLM has some luck. At the very least you get a second opinion on your error.

I had some difficulty with this because SLIME smashes the *debugger-hook* on each interaction. Eventually I settled for a macro that you can wrap your code with. with-llm-debugger binds *debugger-hook* to the LLM debugger before running the body of the macro. Let's see it in action:

(defun my-average (list) (/ (reduce #'+ list) (length list)))

Now let's make the mistake of thinking it takes a &rest argument:

> (with-llm-debugger (my-average 2 3 4))
; in: GEMINI::WITH-LLM-DEBUGGER (MY-AVERAGE 2 3 4)
;     (MY-AVERAGE 2 3 4)
; 
; caught STYLE-WARNING:
;   The function MY-AVERAGE is called with three arguments, but wants exactly one.
; 
; compilation unit finished
;   caught 1 STYLE-WARNING condition
An unhandled error condition has been signalled: invalid number of arguments: 3

;;
;; **Troubleshooting a Lisp Argument Error**
;;
;; Okay, so I've got this Common Lisp error message: "invalid number of arguments:
;; 3".  And the backtrace pinpoints `MY-AVERAGE` as the culprit, called with the arguments
;; `2 3 4`. Hmm, that's pretty straightforward. The function `MY-AVERAGE` was called
;; with three arguments, but it wasn't expecting them.
;;
;; The next step, naturally, is to figure out what `MY-AVERAGE` *does* expect.  Since
;; I don't have direct access to the user's code, I'll need to query the Lisp environment
;; itself. The best way to do that is to use the `default_api.describe` function on
;; the symbol `MY-AVERAGE`. This should give me its definition, which will include
;; its expected argument list.
;;
;; Once I have that argument list, the fix should be obvious.  Either the arguments
;; being passed are wrong, or the `MY-AVERAGE` function needs a definition adjustment.
;;  I'm ready to proceed, query the environment, and get to the root of this.
;;
;; Processing function call: describe

The LLM has decided to call into Lisp. The describe function is a tool that is just a wrapper around the Lisp describe function. It takes a symbol and returns whatever the Lisp describe function returns.

;; **Understanding the `MY-AVERAGE` Function Call Error**
;;
;; Okay, so looking at this, it seems the problem is pretty straightforward. `describe`
;; clearly tells me that `MY-AVERAGE` is a compiled function and that it's expecting
;; a single argument, which it's calling `LIST`. I can see from the backtrace that
;; I attempted to call it with the individual numbers 2, 3, and 4. The error message
;; makes perfect sense now. I provided three separate arguments, but the function's
;; definition only allows for a single argument: a list.
;;
;; Essentially, I need to wrap those numbers in a list before passing them to the
;; function. Instead of calling it with `(MY-AVERAGE 2 3 4)`, I should have been using
;; `(MY-AVERAGE '(2 3 4))`. That way, the function receives a single list containing
;; the elements it expects. A classic case of mismatched expectations between function
;; definition and invocation! I'll update the calling code and that should resolve
;; the issue.
;;

So those are the LLM's “thoughts” on the error. Here is its diagnosis and conclusion:

The error "invalid number of arguments: 3" for `MY-AVERAGE`
occurred because the `MY-AVERAGE` function expects a single argument,
which should be a list, as indicated by its lambda list
`(LIST)`. However, it was called with three separate arguments: `2`,
`3`, and `4`.

**Diagnosis:**
The function `MY-AVERAGE` is defined as:

```commonlisp
(LAMBDA (LIST)
  (BLOCK MY-AVERAGE (/ (REDUCE #'+ LIST) (LENGTH LIST))))
```
;; Prompt Tokens:       4316
;; Thoughts Tokens:      227
;; Candidate Tokens:     223
;; Total Tokens:        4766
;; Prompt Tokens:       3959
;; Thoughts Tokens:      209
;; Candidate Tokens:      16
;; Total Tokens:        4184
LLM Debugger Response: 
This definition clearly shows that `MY-AVERAGE` is intended to take a single argument named `LIST`.

**Suggested Fix:**
To correctly use `MY-AVERAGE` to calculate the
average of 2, 3, and 4, you need to pass these numbers as a single
list argument.

Change the call from:
`(MY-AVERAGE 2 3 4)`

To:
`(MY-AVERAGE '(2 3 4))`

Not too bad, I guess. But I'd have to put it through its paces on real errors before I could say how useful it is.