I’m able to build and run some simple scripts, but I don’t have all the language features implemented yet, and I’m waiting on a bugfix for an issue in Collections that’s causing trouble for me.  (A big thanks to Alex for putting up with all my bug reports and feature requests, BTW!)

RTTI generation probably sounds intimidating, but the framework for it all condenses down to about 1200 lines of code spread across 4 units.  There are a couple more minor things I still need to add to it, but it’s not all that complicated when you get down to it.  Here’s a quick overview of the four units and what they do:

vmtStructure: A very simple unit that redefines the VMT of a class and several of the RTTI structures from TypInfo.pas in ways that are easier to work with.  A lot of it is based on work by Hallvard Vassbotn.

vmtBuilder: This unit has four routines for creating parts of the VMT and RTTI tables.  The RTTI structures can be very fiddly; they pack data in ways that is not actually supported by the Object Pascal language, such as inline variable-length arrays.  Reading them (or writing them) requires a lot of pointer-based math, so I wrote up these routines to encapsulate the scary work.  Just pass in the data that the structures are supposed to contain, and they’ll arrange it for you.

The vmtBuilder unit also introduces an interface called IBuffer, which provides a simple way to pack arbitrary data together.  It’s mostly just for internal use by the other RTTI-creation routines.

newClass: Here’s where the real fun begins.  The TNewClass class contains a bunch of methods for defining your new class itself.  You create the TNewClass, call the Add*Field methods (AddIntegerField, AddObjectField, etc.) in order, call AddMethod for each method, and then call the ClassPtr method, which takes all the information you’ve fed it and turns it into a new VMT and accompanying RTTI tables.  You can optionally pass a TPrivateHeap instance to ClassPtr, which will place the new data into the private heap instead of the standard application memory.  (I put that in because it’s very useful for dealing with an arcane but important detail of the RTTI system.)

TNewClass currently does not support creating properties, because they’re just syntactic sugar and I was focused on real functionality.  This will change once my script compiler gets to the point that it requires them.  It also doesn’t support creating classes that implement interfaces.  That would add a lot of complexity and I don’t think it’s worth the effort.  But the whole thing’s open source, so if someone needs it and feels like creating a patch for it, they’d be welcome to.

rttiPackage: Here’s the part where I stop drawing on Hallvard’s work and instead owe a big debt of gratitude to Barry Kelly for explaining some of the trickier details.  If you only want to create a new class, you could stop with the three units above.  But if you want to integrate it into the RTTI system so you can get at its members, (which is basically the only way to use it since they aren’t available at compile time,) you need a way to feed all this new RTTI you’ve created into RTTI.pas.  That’s what the rttiPackage unit does.

It contains two new interfaces, INewUnit and IRttiPackage, and classes that implement them.  INewUnit can contain any number of TNewClass instances, and it organizes them so the RTTI system thinks they were all declared in the same unit.  Then IRttiPackage is the construct that pulls everything together.  It collects INewUnit instances and creates and registers a new TLibModule record, which makes RTTI.pas think you’ve loaded a new runtime package, so it loads the RTTI tables for the (fake) new package, containing the data for your new classes.

IRttiPackage.Install, the method that sets up all the data, can take an optional reference to a procedure that allows you to alter the code address of your new classes’ methods at install time.  This was to support the script engine.  When the script compiler builds the metadata for the methods, the actual code address doesn’t exist yet because the RTTI hasn’t been loaded, so it sets the code address as the offset within the script.  (This will also make it a lot easier to save the script and load it later.)  Then when the RTTI table gets loaded as the script engine is starting up, it can use TRttiMethod.CreateImplementation to build a bridge between native Delphi land and the script engine, and set the implementation stub’s address to the official code address for the new method.  It’s kind of an ugly trick, but it works, and it’s the only way I could find to make it work.

I mentioned in my last post that the compiler will require Delphi XE because it uses this trick. I’ll have to revise that further.  There’s a glitch in the setup code for Method Implementations in XE RTM that can cause bogus FPU exceptions if you use it enough. This was fixed in XE Update 1, so that will be the minimum requirements for the script compiler when it’s ready.  The RTTI generation code itself, though, will work under D2010, if anyone has other uses for it.

More to come soon…

On a more technical level it suffices to say that they use custom and extremely compact and fast data structures, tricks and hacks to be able to represent millions and millions of objects within the constraints of a 32-bit Windows system. Throw in the use of Physical Address Extensions, storing per-class information in “virtual” class vars to reduce object instance size, creation of classes and their VMTs dynamically at runtime (!!), pointer packing, multithreading, the list just goes on and on.

I figured if Hallvard’s friend can create classes and their VMTs dynamically at runtime, then anyone could, with the right level of technical knowledge about how a VMT works.  My CodeRage presentation in 2009 was about theoretical research in that area, demonstrating that it could be done.  But then I just sorta sat on it.  It was a cool trick, but I didn’t have anything useful I could do with it, and  I had more important things to work on, like my game engine.

But a few months ago I started ran into some serious issues with making the scripting for my game engine work.  I’ve been using PascalScript, but its support for units is badly broken (and there’s no good way to fix it without basically rearchitecting the whole system) and its GOTO support is just broken enough to cause lots of trouble for me.  (I need to support GOTOs properly.  Don’t ask.)

So I looked at DWS, which has been getting a lot of hype lately with all the recent development.  But it also has bad unit support, no GOTOs at all, and no way to save a compiled script.  I raised each of these issues with Eric and he gave reasons why he has no intention to fix each of them.

There’s one interesting thing DWS can do, though: you can define new classes and use them in the script engine.  Being able to do this in my game engine would allow for a lot of customization opportunities.  But a bit of testing demonstrated that, like the holographic Professor Moriarty on Star Trek: The Next Generation, these newly-defined objects are purely virtual constructs that can’t step out into the real world.  If you try to bring a reference to one of these script objects into Delphi-land, you get a nil instead.  This greatly reduces the opportunity for customization.

So I figured I may as well try my hand at my own version, drawing on my class-creation research.  For the last few months, with a lot of hard work (and a fair bit of technical assistance from Barry Kelly on the arcane details of the RTTI system) I’ve been putting together a new Object Pascal-based script compiler.  It’s not completely finished yet, but the features that will set it apart are up and running:

• Unit-based compilation.  All your code doesn’t have to be in one unit.  It also doesn’t expect that you’re building a single script to be executed directly.  You can do that, but you can also run a valid compile cycle made up entirely of one or more units, with no program unit and no main routine, and use it as a library, calling individual routines from the script engine.  Units can be compiled and saved individually, similar to Delphi’s DCU system, and linked with other units to form a script program.
• Dynamic, RTTI-based creation of new classes.  You can define a new class in the script engine and the compiler will generate a new VMT and all the extended RTTI necessary to use this class from native Delphi code just like any other.  For example, you can create a class that descends from a native type, override a virtual method, pass an instance out of the script engine to a variable reference, call the virtual method, and the script engine will be invoked to run it.  (This particular trick require the TMethodImplementation class to set up, so Delphi XE or later will be required.)

The plan is to use the RTTI capabilities to achieve the most seamless integration possible between scripts and native code.  In my next few posts I’ll be going over some details as to how I made this all work.

I mentioned in a comment that I wouldn’t like to see much change in the RTTI data structures because it would make dynamic class creation code difficult to maintain, and I’ve been doing some serious work in that area recently. Barry responded with the rationale:

It’s less a case of completely changing, than having more data appended on the end. It’s a real PITA to jam in extra data without breaking code. As it is, the RTTI format contributes a fairly substantial size penalty largely because its encoding can’t be optimized much. It would be nice to consider something like e.g. .NET metadata tables, which use a conciser representation. Instead, Delphi RTTI is left with every xref taking up a minimum of 4 bytes, and frequently with redundant strings for names that could be pooled. But that can’t happen within the current framework.

He mentions two things here: changing the way references work to make the RTTI take up less space, and making it less difficult to update the RTTI structures with new information.  I could get behind something like that, especially since the update would almost by necessity involve cleaning up the type definitions in the process.  It’s a real pain to work with some of the existing RTTI structures, because you get stuff like this:

{ vmtFieldTable entry in VMT }
  PVmtFieldTable = ^TVmtFieldTable;
  TVmtFieldTable = packed record
    Count: Word; // Published fields
    ClassTab: PVmtFieldClassTab;
   {Entry: array[1..Count] of TVmtFieldEntry;
    ExCount: Word;
    ExEntry: array[1..ExCount] of TVmtFieldExEntry;}
  end;

Why are those last three fields commented out?  Because they’re declared as static arrays, but the upper bound isn’t known at compile time and changes from one instance to another.  Basically, they contain dynamic arrays inserted inline into the record, which is something that Delphi’s type system doesn’t support.  And since the type system doesn’t support it and the compiler can’t locate the members of the records, extracting data from a TVmtFieldTable (or several other structures that do similar things) is a real pain.

RTTI that’s easier to work with and takes up less space in the EXE?  If they can pull it off, I’d love to see it!

A couple weeks ago, I gave a presentation at the CodeRage 4 online conference about dynamic class creation at runtime in Delphi.  The replays were just posted yesterday at Embarcadero’s CodeRage site.  My session can be downloaded here or viewed online here.  The sound quality was terrible, unfortunately.  I’m not sure what was going on with my headset the evening I recorded this, but it sounds like I was recording inside a giant water pipe, instead of sitting in my quiet, ordinary apartment room.  (And this was after editing.  It sounded even worse at first!)  Despite this, it was received fairly well, with a bunch of good questions and comments.

I showed a demo of a tool I’d written to explore the structure of a class, plus a code library for creating new classes at runtime.  I just put it up for download if anyone wants to play around with it.  You can find it here.  The dproj file is in Delphi 2010 format, and I haven’t tested it with any earlier versions, but I’ve tried to use IFDEFs to define out code that wouldn’t run on earlier versions, and this ought to work on any version from D5 on.  Take a look, and feel free to use it in your own projects if you can find some use for it.  And of course, if there’s any way you find to improve on it, please let me know.
Mason Wheeler