Hard Won Esoteric Knowledge

I do my development in Ubuntu-22.04 Linux running on the Windows Subshell for Linux. I recently got a laptop refresh and the latest software doesn’t run. The fix is obscure, so I thought I’d document it.

sbcl runs fine out of the box in WSL2, but I’m encountering a bug where TCP connections to one particular server are being left in the CLOSE_WAIT state indefinitely. After several minutes, I hit the limit on the number of open files.

The “right thing” would be to track down who isn’t closing the connection properly, but it’s only a few hundred connections. It appears that ulimit is set to 1024, which is pretty easy to hit with this bug. Bumping ulimit to something more reasonable is a lazy workaround. It isn’t a solution — I’m still leaking open files — but I’ll be able to leak thousands of them without having problems.

But increasing nofiles turned out to be a problem. I edited all the magic files in /etc until they all said I could have 131071 open files. When I re-started WSL, all the ways I could start a shell agreed that the ulimit was 131071, yet when I started sbcl and ran this:

(uiop:run-program "prlimit" :output *standard-output*)

RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space  67108864  67108864 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files                1024   1048576 files
NPROC      max number of processes                62828     62828 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals          62828     62828 signals
STACK      max stack size                       8388608 unlimited bytes
NIL
NIL
0 (0 bits, #x0, #o0, #b0)

The limit was at the old value of 1024.

WSL launched sbcl without a shell, so the ulimit setting was not being run.

The solution is easy, but it took me a long time to figure it out. Not only do you need to edit all the magic in /etc, and add ulimit statements to your .bashrc, you should also add ulimit statements to your .profile, and then instruct wsl to launch your program under a login shell:

(require ’sly)
(setq sly-lisp-implementations
      ’((sbcl  ("C:\\Program Files\\WSL\\wsl.exe"
                      "--distribution-id" "{df4f07a6-2142-405c-8a6a-63f1ca3a7e8d}"
                      "--cd" "~"
                      "--shell-type" "login"
                      "/usr/local/bin/sbcl")
               )))

This bit of insanity allows me to run sbcl with 131071 open files in Linux as my inferior lisp program in a Windows Emacs running SLY. (Running Emacs under Windows gives me a way to use a modified Dvorak keyboard. I could run Emacs in the Linux subsystem, but the Wayland server is in a container and doesn’t let you modify the keyboard.)

Weekend update

Over the weekend I changed the script project to launch the script within a browser rather than from the command line. This allows me to debug from the IDE directly. I spent a fair amount of time trying to get the browser to stop complaining that my script language was an untrusted ActiveX control, but I had no luck. I implemented IObjectSafety, but then the browser stopped trying to run my script altogether. I added a reader to the Lisp engine so now the script can be turned into list structure. I'm adding a minimal evaluator so I can test out function calling. I haven't quite figured out how to integrate the package system with everything, but there is something minimal in place.

Past that hurdle

The problem wasn't necessarily an out-of-memory issue, it was a mismatch in how native COM objects are marshaled to managed CLR objects. I wrote a custom marshaler that emits debugging messages and used it to figure out which argument was having problems. A little fussing around and I got it working just fine. At this point, the scripting host starts the scripting engine, calls `AddNamedObject' a couple of times, calls `ParseScriptText' with the source code of the script, and tells the engine to go. I need to suck some meta-data out of the named objects so that I can use them. That seems a little clunky, and I'm surprised that it isn't quite built in to the interop code. A little poking around in how VisualBasic and JScript do it lead me to believe that there isn't a good canonical mechanism. It is amazing this stuff works so well. It's impressive that it works at all. The code is on my code site at http://jrm-code-project.googlecode.com/svn/trunk/LScript/