Package system horrors

Arcane Sentiment mentioned how the Lisp Machine used to handle package prefixes in this post, and it reminded me of an interesting problem.


When we were building the LMI K-machine we had to cross-compile from the existing Lisp Machine. Naturally, the cross-compiler would read the source code and intern the source code symbols as part of the process. But the K-machine had a different package setup. New packages weren't a particular problem, but the locations of some symbols were. In addition, some things that were implemented as macros in the Lisp Machine were implemented as compiler intrinsic special forms on the K-machine, and vice versa.


The compiler naturally uses symbol identity to determine how to handle a special form. If you want to compile a conditional, for example, the CAR of the form had better be the symbol CL:IF. A symbol named "IF" in a different package won't generally work.


Well, that's not exactly correct... When the Lisp Machine was built, there was no CL package. There was no Common Lisp. So when the CL package was added, it was necessary to make sure that the symbol named "IF" in the CL package was the same EQ symbol as the one the compiler used for conditionals. There are a couple of ways to arrange for this. The symbol could be explicitly interned in the CL package, or it could be visible to the CL package via inheritance.


We wanted to change some of these things on the K-machine, but the cross-compiler was running on the LMI Lambda and when it read the source code it would intern the symbols in the Lambda's packages. Munging the existing Lambda packages was out of the question. The ‘solution’ was to play some nasty tricks with the entire package system. We created two different packages, both named "COMMON-LISP". Some of the symbols were common between them, but others were not. The package inheritance was different, too. The trick was to dynamically switch how package names were resolved to package objects.


So if you have two different “package systems”, how do you refer explicitly to “the symbol IF in the other "COMMON-LISP" package? With a reader hack, of course. We made the triple colon ::: prefix indicate which package system to use to resolve the package name.


This kind of worked, but it was a horror. Most of the system didn't care, the rest didn't know, but if you accidentally had the wrong package system in place when you read a file, you would definitely trash the entire machine and need to reboot. We had a few helper functions that would try to ensure that the correct package system was selected before doing anything (for instance, when calling the cross-compiler, we'd switch out, and when it returned we'd switch back. Works great until you enter a debug repl...). The big problem was that it was impossible to ‘close’ over the correct context, so you could not in general guarantee seamless operation. (For example, if you referred to the K-machine's CL:READ symbol, you probably wanted to switch to the K-machine package set, but there was no way to do this automatically.)