RTTI Generation code now available

Over the past couple weeks, I’ve been working on refining and testing my RTTI generation and the scripting system I’ve been building on top of it, which I’ve decided to call RTTI Script.  I think I’m finally starting to get something ready for public consumption.  I set up a Google Code repository this morning for RTTI Script, but so far all it contains is the RTTI generation code.  The actual compiler and script executor still need some work.

I’m able to build and run some simple scripts, but I don’t have all the language features implemented yet, and I’m waiting on a bugfix for an issue in Collections that’s causing trouble for me.  (A big thanks to Alex for putting up with all my bug reports and feature requests, BTW!)

RTTI generation probably sounds intimidating, but the framework for it all condenses down to about 1200 lines of code spread across 4 units.  There are a couple more minor things I still need to add to it, but it’s not all that complicated when you get down to it.  Here’s a quick overview of the four units and what they do:

vmtStructure: A very simple unit that redefines the VMT of a class and several of the RTTI structures from TypInfo.pas in ways that are easier to work with.  A lot of it is based on work by Hallvard Vassbotn.

vmtBuilder: This unit has four routines for creating parts of the VMT and RTTI tables.  The RTTI structures can be very fiddly; they pack data in ways that is not actually supported by the Object Pascal language, such as inline variable-length arrays.  Reading them (or writing them) requires a lot of pointer-based math, so I wrote up these routines to encapsulate the scary work.  Just pass in the data that the structures are supposed to contain, and they’ll arrange it for you.

The vmtBuilder unit also introduces an interface called IBuffer, which provides a simple way to pack arbitrary data together.  It’s mostly just for internal use by the other RTTI-creation routines.

newClass: Here’s where the real fun begins.  The TNewClass class contains a bunch of methods for defining your new class itself.  You create the TNewClass, call the Add*Field methods (AddIntegerField, AddObjectField, etc.) in order, call AddMethod for each method, and then call the ClassPtr method, which takes all the information you’ve fed it and turns it into a new VMT and accompanying RTTI tables.  You can optionally pass a TPrivateHeap instance to ClassPtr, which will place the new data into the private heap instead of the standard application memory.  (I put that in because it’s very useful for dealing with an arcane but important detail of the RTTI system.)

TNewClass currently does not support creating properties, because they’re just syntactic sugar and I was focused on real functionality.  This will change once my script compiler gets to the point that it requires them.  It also doesn’t support creating classes that implement interfaces.  That would add a lot of complexity and I don’t think it’s worth the effort.  But the whole thing’s open source, so if someone needs it and feels like creating a patch for it, they’d be welcome to.

rttiPackage: Here’s the part where I stop drawing on Hallvard’s work and instead owe a big debt of gratitude to Barry Kelly for explaining some of the trickier details.  If you only want to create a new class, you could stop with the three units above.  But if you want to integrate it into the RTTI system so you can get at its members, (which is basically the only way to use it since they aren’t available at compile time,) you need a way to feed all this new RTTI you’ve created into RTTI.pas.  That’s what the rttiPackage unit does.

It contains two new interfaces, INewUnit and IRttiPackage, and classes that implement them.  INewUnit can contain any number of TNewClass instances, and it organizes them so the RTTI system thinks they were all declared in the same unit.  Then IRttiPackage is the construct that pulls everything together.  It collects INewUnit instances and creates and registers a new TLibModule record, which makes RTTI.pas think you’ve loaded a new runtime package, so it loads the RTTI tables for the (fake) new package, containing the data for your new classes.

IRttiPackage.Install, the method that sets up all the data, can take an optional reference to a procedure that allows you to alter the code address of your new classes’ methods at install time.  This was to support the script engine.  When the script compiler builds the metadata for the methods, the actual code address doesn’t exist yet because the RTTI hasn’t been loaded, so it sets the code address as the offset within the script.  (This will also make it a lot easier to save the script and load it later.)  Then when the RTTI table gets loaded as the script engine is starting up, it can use TRttiMethod.CreateImplementation to build a bridge between native Delphi land and the script engine, and set the implementation stub’s address to the official code address for the new method.  It’s kind of an ugly trick, but it works, and it’s the only way I could find to make it work.

I mentioned in my last post that the compiler will require Delphi XE because it uses this trick. I’ll have to revise that further.  There’s a glitch in the setup code for Method Implementations in XE RTM that can cause bogus FPU exceptions if you use it enough. This was fixed in XE Update 1, so that will be the minimum requirements for the script compiler when it’s ready.  The RTTI generation code itself, though, will work under D2010, if anyone has other uses for it.

More to come soon…


  1. Thorsten says:

    One of the major (but by far not the only) shortcomings of the extended RTTI the last time I checked was the total lack of support for array properties. That makes things like TStringList or TFields more or less unusable through RTTI. Or even TComponent for that matter.

    I was looking into the possibility of implementing IDispatch dynamically for any delphi class, but had to give up after a while as there were simply way too many holes still in the RTTI.

    How are you planing of solving this for your scripting engine?

    • Mason Wheeler says:

      I’ll handle array properties the same way PascalScript does, with accessor functions, for XE at least. Hopefully XE2 will include proper RTTI support for array properties. (And, if I’m particularly lucky, RTTI for standalone functions too, and an Invoke method for function and method pointer typed fields/properties…)

  2. Thorsten says:

    Well, once you have to start defining imports manually for some things the whole RTTI becomes pretty useless. If you already have to define imports for some things you could just as well define imports for others and there isn’t much point in having the RTTI.

    It’s one of these things where 90% is just as good as 0%….

    I did file a whole bunch of QCs against the RTTI in D2010 when I did my investigation if it could be usable to make any .bpl transparently accessible via IDispatch (and/or the dynamic support in .net 4). Unfortunately not a single one of the major issues I raised got fixed in XE, and based on past experience I’m holding much hope for XE2.

    Naturally, I would love to be positively surprised here…

  3. Thorsten already mentioned about the RTTI issues, I would like to point out just one thing: even if you, Mason, reach a point where you find that most of your efforts are “useless” don’t give up, there are a lot of NEW things that you’re learning!! just my two cents…

  4. John Jacobson says:

    Interesting. I just wrote an object persistence framework (sorry, not publicly available). I started it off using RTTI but ended up using attributes instead.

  5. So you were faster than me. 🙂

    I believe we’ve been talking about that topic at Delphi Live. As I mentioned I’ve built a library for similar purposes. I didn’t look at your code yet (not much time), but perhaps we could try to combine our efforts. I’ll have to investigate the differences in the design of our libraries when I have more time…

    TRttiMethod.CreateImplementation() is a comfortable way of doing things, but I consider it a transitional stage, mostly due to speed concerns. (Boy do I wish we had access to a JIT compiler in native Delphi 🙂 ) Having a way of constructing methods at runtime, based on a directed acyclic flow graph to provided by the user of the library, was the ultimate goal of my efforts. But that’s going to take some more time and effort.

    • Mason Wheeler says:

      No, this isn’t the same stuff we were talking about at Delphi Live, but it’s kinda related.

      And I know what you mean about having access to a compiler. It would make a lot of things a lot easier (and basically remove the need for this entire project) if there was some way to include DCC as a unit in your Delphi program. But I may as well wish for rainbows and unicorns while I’m at it… 😛

  6. Paoti Rios says:

    Some thing like a LibJit?


    I’m not into low level stuff… so i wonder if the GPL its the problem

  7. […] zur Laufzeit erstellen?   Heute, 19:21 Vielleicht ist dies im Zusammenhang hiermit ja hilfreich? Uwe RaabeCertified Delphi Master Developer Blog: The Art of Delphi […]

Leave a Reply to Moritz Beutel