Consider this problem. You are running a high school and you suspect some gang activities, so you want to be on the lookout for people wearing gang `uniforms'. But you don't know what any of the uniforms might be. So you watch the students and take note of what they are wearing: some have puffy jackets, some have hoodies, some have t-shirts, some have baggy pants, some have hats that they wear backwards, etc. etc. etc. Over the course of the day, you see a lot of different clothing, a lot of different combinations, and all in different colors. You note, however, that you did see five students wearing purple puffy jackets, baggy pants, and red baseball caps. What is the liklihood that this is a gang uniform? Obviously, it will depend on the size of the student population and the prevalence of certain kinds of clothing, but in the extreme case where you see, say, fifteen students with the *exact* same outfits that match in twenty-seven different random qualities, you can be pretty sure that is a gang. On the other hand, if you see two students that happen to wear the same color puffy jacket and otherwise have nothing in common, you can write that one off as coincidence. So given the number of students and the relative prevalence of various clothing elements, how many points of coincidence do you need to be fairly certain you have found a gang uniform?

Signs of life

It is showing signs of life. I have gotten to the bootstrap step where we replace the bootstrap version of MAKE-INSTANCE with the generic version. On the next call to MAKE-INSTANCE a lot of stuff happens that eventually lands us in the yet-to-be-implemented primary method of MAKE-INSTANCE.

At this point, I need to clean up a few of the kludges and finish implementing a few of the important functions (for example, my wimpy sort routine only works on lists of a single element so far).

Function Cells in C#

I found this interesting syntactic hack to mimic function cells in C#. Suppose you know the signature and name of a function at compile time, but the value will be computed at runtime and may change over time. First, define a delegate type for the signature of the function:

delegate object FooFunction (string s, int i);

In the class where the function Foo will be defined, we create a static variable to hold the delegate:

static FooFunction currentFooFunction;

and a Property to read it:

public static FooFunction Foo
{
    get
    {
        return currentFooFunction;
    }
}

You can now write code like this:
object bar = Foo("test", 4);
which appears to be an ordinary static function call, but in actuality is a call that indirects through the currentFooFunction.

The basic trick here is to have a Property that returns a delegate. The Property code can perform an arbitrary computation to return the delegate, and the compiler knows that a delegate returned from a Property call can be used in a syntactic function position.

That is my final answer

The IL version had a problem that I didn't notice before. Generic functions need to be closed over themselves. This allows the implementation to store a trampoline function when the generic is created and/or modified. The trampoline function recomputes what code is run when the generic is run, and then smashes itself out of the code path. Subsequent runs of the generic function don't go through the trampoline.

The problem is that the method object held by the .NET delegate is a read-only object. It cannot be overwritten. No problem, we'll just indirect through the field we can write. But we need to pass in the delegate object itself as an argument so we can modify it. In order to do that the delegate method needs access to the this pointer in the delegate. But the point of a delegate is to provide the delegated method with a this pointer. That is the Target field in the delegate. But the target object held by the .NET delegate is also read-only. It isn't possible to construct a delegate that delegates to itself in .NET (at least not without major black magic). We can create a delegate that delegates to a static method within the delegate class itself, but then we no longer have access to the instance object.

The solution is another level of indirection. We create a ManifestInstance class that has mutable state we need and closes over itself. The delegate simply defers to the ManifestInstance. This is the way delegates are intended to be used, so there is no need to use IL to add extra fields or anything. We can do this with standard, simple C#.

But I still want my StandardObjects to be printed with some sort of useful debugging info rather than {Method = {System.Object uninitializedFuncallableInstanceMethod(System.Object[])}}. The StandardObject delegate needs a DebuggerTypeProxy. A DebuggerTypeProxy is a helper object that tells the debugger how to render objects when the standard mechanisms aren't sufficient. But for reasons I don't understand, you can't associate a DebuggerTypeProxy with a .NET delegate! Argh!!!!

But I found a trick. You can associate a DebuggerDisplay attribute with a delegate. This is a string attribute that is interpreted in the context of the object being debugged. You can put things like "Object {Name}" in the DebuggerDisplay attribute and it will print the literal string `Object' followed by the Name field of the object. For our StandardObject delegate, we tell it to print the Target field. This sort of works: "{Target}", but then the debugger uses the ToStringmethod of the ManifestInstance. It makes it hard to visually distinguish the ManifestInstance from the delegate that refers to it. Instead, we want to get a computed property of the ManifestInstance. Unfortunately, this doesn't work: "{Target.ObjectDebuggerDisplay}", but this does: "{((ManifestInstance)Target).ObjectDebuggerDisplay}". Now when the debugger sees a StandardObject, it calls a method defined in ManifestInstance that has access to the object's state.

Since all this can be done in simple C# code, there is no reason to hack IL for StandardObjects. Back to the hacking....

Trying again

Having standard-object implemented in a separate assembly is getting unweildy. I think I'm going to return to the manifest-instance technique.

The Visual Studio debugger can use certain annotations on your code to make debugging more pleasant. For example, I have annotated my Keyword and Symbol classes so that the watch window can print them correctly (not quite exactly, yet --- no escaping). When I hit a breakpoint, I can get a display like this:

Name	Value	Type
foo	:NAME	object{Lisp.Keyword}
bar	CLOS::STANDARD-GENERIC-FUNCTION	object{Lisp.Symbol}

Right now, however, my StandardObject delegates are printing like this:

Name	Value	Type
generic	{Method = {System.Object uninitializedFuncallableInstanceMethod(System.Object[])}}	object{Lisp.StandardObject}

That's not so useful. I can add the debugging annotations to the StandardObject, but since it is in a different assembly, it creates a circular dependency. If I go back to the somewhat more clunky, but portable representation, I'll find it a lot easier to debug.

Later that same day....
Argh. You can't add a debugger type proxy to a delegate in C#. Back to IL.

Standard Object

I'm a fan of CLOS. When I wrote the .NET interface to Larceny, I based it on Eli Barzilay's Swindle, which is based on Tiny CLOS. Naturally, if I'm writing scripting code in Lisp, I want to have the same sort of interface to the underlying object layer.

CLOS has a couple of interesting requirements that can be tricky to implement. The first is the ability to construct funcallable-standard-instances, that is, first-class procedure objects that can have additional data associated with them. The second is the ability to call change-class on an instance to transform it in place to an instance of another class. (Why would I want to do that?! Normally, a user of CLOS wouldn't want to or need to do something so horrendous to an object, but if you are using a REPL and change a class definition by reloading a file, you want the system to update existing instances to the new schema. Rather than forcing the user to exit the REPL, recompile, reload, and then reconstruct his state, we tweak the objects behind the scenes and keep going.) If we combine these features we can potentially turn any instance into a procedure. (No, we don't want to design features that deliberately use this ability, we want to allow the user to type defclass..., say `Oops! I meant defgeneric...,' and do the fixup on the fly.)

The CLR has first-class functions called delegates. A delegate is basically an object that contains an object and a reference to a method implemented by the object. When the delegate is invoked, the method within the object is invoked and the answer is returned. If we consider the method to be a piece of code and the object to be an environment, then we can see that a CLR delegate is simply a closure. If we use delegates to implement funcallable-standard-instances, then our CLOS generic functions can be used anywhere a delegate can (modulo type co- and contra- variance). A non-funcallable standard-instance can simply be a delegate that refers to a method that raises an exception.

We need to associate some additional data with our standard-instances. First, an instance has an associated class object. While it is true that our delegates have an associated .NET type, it is always going to be the System.Delegate type and the .NET types are insufficient for our purposes. Second, an instance has an associated set of `slots'. We need the ability to both refer to these elements and to replace them in order to implement change-class. There are a couple of strategies that will work here. One is to hold the additional data in a weak hash table keyed by the delegate. This technique was used in the original version of Swindle, but the hash lookup will incur a fair amount of overhead. A portable technique is to define a helper class with the necessary fields:

class ManifestInstance
{
    StandardObject class;
    object [] slots;
    function onFuncall;
}


A StandardObject would be a delegate to a trampoline function in ManifestInstance. This technique works, but it seems kind of a kludge. We'd like to have our own special subclass of delegate that simply has two additional fields. (We'll have an additional benefit: the debugger will show us the class and slots as immediate subvalues in the delegate. If we indirect through a ManifestInstance, we have to manually chase down the pointer in the debugger.) We can create subclasses of delegates by using the delegate declaration in C#, but we can't add any new fields.

We can, however, do this trick in IL and assemble it into a signed assembly that can be placed in the GAC. We can then just add a reference to it in VC#.

I've added some code for this in http://jrm-code-project.googlecode.com/svn/trunk/Jrm.StandardObject/. I'm still tweaking it a bit, but it seems to be working.

I'm trying to figure out how the CLR type system will work with Lisp. My script does a simple thing of calling .write on the document object. It is currently erroring out because `.write' is undefined. Fair enough, I didn't define it. But it ought to be defined when I try to invoke it on the document object. Hmm.... maybe that's the key....

This just keeps getting better and better! Now I can't debug because something is timing out the script and aborting the process before I can look at what is going on.