Note: This is an alpha version.  There are probably plenty of bugs in it.  There are still features I haven’t gotten around to implementing.  The sight of the code would probably send Barry Kelly fleeing from the room, screaming in horror.  So use at your own risk, and I take no responsibility if this causes your system to crash, your data to come out all wrong, or demons to fly out your nose.  Specifically, the program keyword is not implemented yet, (only units,) var parameters don’t work yet, and a handful of features aren’t implemented in the script executor.

Having said that, here’s a few cool features I’ve come up with:

• Class creation:  Defining a new class inside the script code is not supported yet, but the basic framework is there, and script units implement new classes with their routines as methods, accessible to the script executor through the RTTI system.  I’ll add class creation pretty soon.
• Automatic registration: Ever try to register a class with the script engine in PascalScript?  It’s a painful process involving generating a registration unit that basically redeclares your entire class, and then has to be kept in sync or things break.  (And if you get the dependency order wrong, it fails silently,  leading to weird errors further down the line.)  I figured that was way too complicated, so for RTTIScript, all you have to do is register a class reference, and it uses the RTTI system to figure out the internal members and dependencies for you.
• Execution environment: You’ll often have a bunch of little standalone routines and global variables that you want to import into a script to have available at all times, to provide an interface between the script and your program.  So I introduced the concept of the “environment:” You create a class and register it with the compiler, and it treats all the class’s public methods and properties as globals for the script.  (This also means that you can use property getters and setters to have the script’s interaction with its “global variables” invoke side effects.)
• Script saving: I know Eric Grange thinks that a fast compiler is good enough and a bytecode version of a script is not necessary, but I don’t agree.  By his own tests, even a very fast script compiler can take 2 seconds to compile a 100,000 line script.  I’ve actually got an example of one that big in a project that the author wants to convert over once the TURBU project is ready.  Adding a two-second lag into the loading process and making the player wait while a level loads, which should feel instantaneous, is unacceptable.  I don’t have script saving implemented yet, but it’s definitely on the to-do list.

Anyway, the alpha version is available for any guinea pigs curious developers out there who’d like to take a look at it.  It requires Delphi XE, it’s pretty fast, and when I get the rest of the features set up it’s going to be awesome!  Feel free to check it out, and to submit a bug report if you find some errors. (Which I’m sure plenty of people will.)

I expanded on his idea and sent a suggested implementation to Barry Kelly.  Unfortunately, it didn’t make it into XE, but it’s in XE2 now.  It’s not exactly the code I suggested–a few things have changed in the way TValue works internally since D2010, and there’s at least one bugfix that I can see–but the basic idea is there.  When storing values into a TValue and retrieving them again, there’s special code to set and retrieve the most common types directly, without having to go into the generic RTTI-based (much slower) conversion routines.

I reran the timing test under XE2, and the results were much better this time:

Variants: 2140
TValues: 2895

Still about 30% slower than Variants, but at least it’s not 2000% slower anymore.  Thanks, Barry, and thanks to Robert too for the original suggestion!

Both Firebird DBX drivers blew up on me when I tried to do this, in different ways.  The Embarcadero driver may or may not have had a problem with the booleans, but I never really got the chance to test that out because the object I was trying to write contained a blob field, and I got an exception from Firebird: “Incorrect values within sqlda structure.”  A bit of Googling revealed that this is a known issue with writing blobs to Firebird that’s been in the Embarcadero driver for several years now. Apparently it’s still around.  Not much I can do about that without the source code.

So I tried Chau Chee Yang’s driver.  It didn’t have any trouble with the blobs, but it did choke on writing a Boolean field.  I traced into the call into the database driver and looked at the assembly, and as near as I can tell, his implementation of the DBXWritableRow_SetBoolean function is:

begin 
 assert(false); 
 result := 0; 
end;

Not hard to guess why.  Firebird doesn’t support booleans, so no need to waste time writing a method for it.  However, this is something I can fix without access to the driver source, with a little bit more RTTI surgery.

The standard version of the BOOLEAN domain is implemented as a special-case version of a smallint.  So what we need is a way to write the value back out as a smallint instead of a boolean.  When the driver DLL gets loaded by the DBExpress engine, it builds a method table object containing a bunch of function pointers to various functions in the DLL.  One of them is the aforementioned DBXWritableRow_SetBoolean.  There’s also a DBXWritableRow_SetInt16 that deals with writing out smallints.  So a redirector function is easy enough to write:

var
  LSetSmallint: TDBXWritableRow_SetInt16;

function Passthrough_SetBoolean(Handle: TDBXWritableRowHandle;
  Ordinal: TInt32; Value: LongBool): TDBXErrorCode; stdcall;
begin
  if value then
     result := LSetSmallint(handle, ordinal, 1)
  else result := LSetSmallint(handle, ordinal, 0);
end;

(Yes, I need that if statement in there instead of simply calling Ord(value).  As a LongBool, any nonzero value is considered True, and sometimes you’ll get things other than 1, for whatever reason.)  LSetSmallint represents the current driver’s DBXWritableRow_SetInt16 function.  Now you’ve just got to insert it into the method table.  It’s buried under several inaccessible layers of encapsulation, of course, but the RTTI objects can dig it out.

procedure FixConnection(connection: TSqlConnection);
var
   ctx: TRttiContext;
   cls: TRttitype;
   fld: TRttiField;
   driver: TDBXDriver;
   table: TDBXMethodTable;
begin
   cls := ctx.GetType(connection.DBXConnection.ClassType);
   driver := cls.GetField('FDriverDelegate').GetValue(connection.DBXConnection).AsType<TDBXDriver>;

   cls := ctx.GetType(driver.classtype);
   driver := cls.GetField('FDriver').GetValue(driver).AsType<TDBXDynalinkDriver>;

   fld := ctx.GetType(TDBXDynalinkDriver).GetField('FMethodTable');
   table := fld.GetValue(driver).AsType<TDBXMethodTable>;
   table.FDBXWritableRow_SetBoolean := Passthrough_SetBoolean;
   LSetSmallint := table.FDBXWritableRow_SetInt16;
end;

This has to be run after you open the connection but before you try to write any booleans out.

BTW I emailed Chau Chee Yang about all this.  He wrote back that he plans to support booleans in his driver “as soon as possible.”  So let’s hope that he finds the time soon, and that Embarcadero is able to free up some of their limited development resources to fix up their driver issues.  Tricks like this may be fun to dig around and build workarounds for, but in the end RTTI surgery is kind of an ugly hack that violates encapsulation, and it would be nice to not have to use it.

The first part is easy.  The second, not so much, if you want to use DBExpress.  This really should be handled internally as a special case inside the DBX driver.  Unfortunately neither Embarcadero nor Chau Chee Yang, maker of the alternative dbExpress Firebird driver, has released the source to their drivers, neither driver handles the BOOLEAN domain, and neither driver has any sort of callback/event handler that you can set up to intercept and modify the schema of a query result.  But I’m not gonna let a little thing like that stop me!

To know how to fix this, we first need to know what’s going wrong.  Here’s the basic overview:

To access database data through DBExpress, you generally use TSimpleDataset.  This is a TCustomClientDataset descendant that contains an internal dataset for DB access called TInternalSQLDataSet.  The outer layer has all the functionality that makes TClientDataset wonderful to work with.  You run a query, and it has the internal dataset access the database and pull back data, which gets stuffed into the client dataset for you to manipulate.

When the internal dataset talks to the database, it runs the query and gets back a record set, which includes data and a schema.  It iterates over the schema and prepares field defs for each column, then creates fields based on the field defs.  If you could catch and modify the field defs before it builds fields from them, you’d be good, but the whole process happens inside the InternalOpen method, with no conveniently-placed event handlers for you to interpose any code.

However, the InternalInitFieldDefs method, which sets up the field defs, is virtual, so it can be overridden in a descendant class.  Only problem is, we’re talking about the internal dataset here.  It’s kind of difficult to replace the internal dataset of your TSimpleDataset, because it’s a private field.  But the AllocDataSet method, which sets up the internal dataset, is also virtual.  With a little bit of RTTI surgery, we can set a value to a private field.

Things could get really messy at this point.  I could create a new dataset that descends from TSimpleDataset, but then I’d have to put it in a package, register it on the component palette, etc… unless I use an interposer class, a component with the same name that’s hidden from the form designer but gets instantiated instead of the original class at runtime.  So that’s our starting point.

Here’s my unit, weighing in at just under 100 lines of code plus the MPL header.  I’ll explain how it works at the bottom.  This does require Delphi 2010 or greater.

{*****************************************************************************
* The contents of this file are used with permission, subject to
* the Mozilla Public License Version 1.1 (the "License"); you may
* not use this file except in compliance with the License. You may
* obtain a copy of the License at
* http://www.mozilla.org/MPL/MPL-1.1.html
*
* Software distributed under the License is distributed on an
* "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
* implied. See the License for the specific language governing
* rights and limitations under the License.
*
*****************************************************************************
*
* This file was created by Mason Wheeler.  He can be reached for support at
* tech.turbu-rpg.com.
*****************************************************************************}
unit FirebirdDataset;

interface
uses
   SimpleDS;

type
  TSimpleDataset = class(SimpleDS.TSimpleDataset)
  protected
    procedure AllocDataSet; override;
  end;

implementation
uses
  Classes, SysUtils, RTTI, DB, Provider, SqlExpr,
  DBCommon;

type
  TInternalSQLDataSet = class(SimpleDS.TInternalSQLDataSet)
  private
    FQuery: TSqlQuery;
  protected
    procedure InternalInitFieldDefs; override;
  public
    constructor Create(AOwner: TComponent); override;
  end;

{ TSimpleDataset }

procedure TSimpleDataset.AllocDataSet;
var
  ctx: TRttiContext;
  cls: TRttiType;
  fld: TRttiField;
  FDataset: TInternalSQLDataSet;
  FProvider: TDataSetProvider;
begin
  cls := ctx.GetType(self.ClassType);

  fld := cls.GetField('FDataset');
  FDataSet := TInternalSQLDataSet.Create(Self);
  FDataSet.Name := 'InternalDataSet';
  FDataSet.SQLConnection := Connection;
  FDataSet.SetSubComponent(True);
  fld.SetValue(self, FDataset);

  fld := cls.GetField('FProvider');
  FProvider := fld.GetValue(self).AsType<TDataSetProvider>;
  FProvider.DataSet := FDataSet;
end;

{ TInternalSQLDataSet }

constructor TInternalSQLDataSet.Create(AOwner: TComponent);
const
  QUERY = 'select rFields.RDB$FIELD_NAME NAME ' +
          'from  RDB$RELATION_FIELDS rFields ' +
          'join RDB$FIELDS fields ON (rFields.RDB$FIELD_SOURCE = fields.RDB$FIELD_NAME) ' +
          'where  (rFields.RDB$RELATION_NAME=:tablename) and (fields.RDB$FIELD_TYPE = 7) ' +
          '  and (rFields.RDB$FIELD_SOURCE = ''BOOLEAN'') ' +
          'order by rFields.RDB$FIELD_POSITION ';
begin
  inherited Create(AOwner);
  FQuery := TSqlQuery.Create(self);
  FQuery.SQL.Text := QUERY;
  FQuery.Params.AddParameter.Name := 'tablename';
end;

procedure TInternalSQLDataSet.InternalInitFieldDefs;
var
  def: TFieldDef;
begin
  inherited InternalInitFieldDefs;
  if not (self.CommandType in [ctTable, ctQuery]) then
    Exit;

  FQuery.Active := false;
  //Why is TCustomSQLDataSet.SetConnection both virtual and private?!?
  FQuery.SQLConnection := self.SQLConnection;
  FQuery.ParamByName('tablename').AsString := UpperCase(GetTableNameFromQuery(self.CommandText));
  FQuery.Active := true;
  FQuery.First;
  FieldDefs.BeginUpdate;
  try
    while not FQuery.Eof do
    begin
      def := TFieldDef(TDefCollection(FieldDefs).Find(Trim(FQuery.FieldByName('NAME').AsString)));
      if assigned(def) then
         def.DataType := ftBoolean;
      FQuery.Next;
    end;
  finally
    FieldDefs.EndUpdate;
  end;
end;

end.

First comes the outer dataset.  It should operate exactly the same way as the original dataset, except with a modified inner dataset.  So I override AllocDataSet and replace it with a copy of the original code, except that it creates the new internal dataset and uses RTTI to access the private fields.  The internal dataset is where the interesting stuff takes place.

You can find out about Firebird domains by querying the internal metadata tables for the database.  That’s basically the only option available this late in the game, so I gave the new internal dataset an internal dataset of its own, a TSqlQuery which is designed for… well, you can probably guess.  The constructor creates the TSqlQuery and loads it with a parameterized query that will find any columns in a table that belong to a smallint-based domain named BOOLEAN.

Then, when InternalInitFieldDefs runs, it calls the inherited method, then sees if there’s anything it needs to fix.  Currently only single-table queries of type ctTable or ctQuery are supported.  If you run a ctQuery with a join on it, it’ll only get the first table.  Working around this would require building an actual SQL parser into the dataset, and I don’t really have time for that this evening.

It calls the RTL function DBCommon.GetTableNameFromQuery, which uses a very simple SQL parser to extract the table name. (If your query type is ctTable, it’ll just return the same table name you passed in.)  It sets the param on the internal TSqlquery and runs it, and then goes over the results. Any fields that get returned get checked against the field defs and their type is switched to ftBoolean.  Then this function returns, and InternalOpen goes on its merry way, setting up boolean fields for you.  Hooray!

But this really should be handled within the driver.  I hope that that can be arranged for the next version. (Of either one.)  Then we won’t need hacks like this.

I’m able to build and run some simple scripts, but I don’t have all the language features implemented yet, and I’m waiting on a bugfix for an issue in Collections that’s causing trouble for me.  (A big thanks to Alex for putting up with all my bug reports and feature requests, BTW!)

RTTI generation probably sounds intimidating, but the framework for it all condenses down to about 1200 lines of code spread across 4 units.  There are a couple more minor things I still need to add to it, but it’s not all that complicated when you get down to it.  Here’s a quick overview of the four units and what they do:

vmtStructure: A very simple unit that redefines the VMT of a class and several of the RTTI structures from TypInfo.pas in ways that are easier to work with.  A lot of it is based on work by Hallvard Vassbotn.

vmtBuilder: This unit has four routines for creating parts of the VMT and RTTI tables.  The RTTI structures can be very fiddly; they pack data in ways that is not actually supported by the Object Pascal language, such as inline variable-length arrays.  Reading them (or writing them) requires a lot of pointer-based math, so I wrote up these routines to encapsulate the scary work.  Just pass in the data that the structures are supposed to contain, and they’ll arrange it for you.

The vmtBuilder unit also introduces an interface called IBuffer, which provides a simple way to pack arbitrary data together.  It’s mostly just for internal use by the other RTTI-creation routines.

newClass: Here’s where the real fun begins.  The TNewClass class contains a bunch of methods for defining your new class itself.  You create the TNewClass, call the Add*Field methods (AddIntegerField, AddObjectField, etc.) in order, call AddMethod for each method, and then call the ClassPtr method, which takes all the information you’ve fed it and turns it into a new VMT and accompanying RTTI tables.  You can optionally pass a TPrivateHeap instance to ClassPtr, which will place the new data into the private heap instead of the standard application memory.  (I put that in because it’s very useful for dealing with an arcane but important detail of the RTTI system.)

TNewClass currently does not support creating properties, because they’re just syntactic sugar and I was focused on real functionality.  This will change once my script compiler gets to the point that it requires them.  It also doesn’t support creating classes that implement interfaces.  That would add a lot of complexity and I don’t think it’s worth the effort.  But the whole thing’s open source, so if someone needs it and feels like creating a patch for it, they’d be welcome to.

rttiPackage: Here’s the part where I stop drawing on Hallvard’s work and instead owe a big debt of gratitude to Barry Kelly for explaining some of the trickier details.  If you only want to create a new class, you could stop with the three units above.  But if you want to integrate it into the RTTI system so you can get at its members, (which is basically the only way to use it since they aren’t available at compile time,) you need a way to feed all this new RTTI you’ve created into RTTI.pas.  That’s what the rttiPackage unit does.

It contains two new interfaces, INewUnit and IRttiPackage, and classes that implement them.  INewUnit can contain any number of TNewClass instances, and it organizes them so the RTTI system thinks they were all declared in the same unit.  Then IRttiPackage is the construct that pulls everything together.  It collects INewUnit instances and creates and registers a new TLibModule record, which makes RTTI.pas think you’ve loaded a new runtime package, so it loads the RTTI tables for the (fake) new package, containing the data for your new classes.

IRttiPackage.Install, the method that sets up all the data, can take an optional reference to a procedure that allows you to alter the code address of your new classes’ methods at install time.  This was to support the script engine.  When the script compiler builds the metadata for the methods, the actual code address doesn’t exist yet because the RTTI hasn’t been loaded, so it sets the code address as the offset within the script.  (This will also make it a lot easier to save the script and load it later.)  Then when the RTTI table gets loaded as the script engine is starting up, it can use TRttiMethod.CreateImplementation to build a bridge between native Delphi land and the script engine, and set the implementation stub’s address to the official code address for the new method.  It’s kind of an ugly trick, but it works, and it’s the only way I could find to make it work.

I mentioned in my last post that the compiler will require Delphi XE because it uses this trick. I’ll have to revise that further.  There’s a glitch in the setup code for Method Implementations in XE RTM that can cause bogus FPU exceptions if you use it enough. This was fixed in XE Update 1, so that will be the minimum requirements for the script compiler when it’s ready.  The RTTI generation code itself, though, will work under D2010, if anyone has other uses for it.

More to come soon…

In the podcast, Allen mentioned certain features that were “holding the language back.”  Now, if you were to ask me what needs to be taken out of Delphi, I’d say with.  And I wouldn’t actually take it out; I’d change the syntax so it works like it does in VB, and I’d disallow nested with blocks.  Between those two changes, that should eliminate the ambiguities that make with troublesome and turn it into a very useful feature.

But Allen’s thinking on a bit of a lower level.  He mentioned absolute.  When I asked why absolute was problematic in the comments, he responded:

“absolute” can cause significant type-safety holes by allowing one to “overlay” a variable of one type on top of a variable of another type. In the future this will cause significant confusion for things like further enhanced meta-data and run-time stack information leading up to a richer/safer run-time experience. This would certainly interfere with the potential future inclusion of garbage collector, for instance.

This raises some serious philosophical questions about the future of Delphi.  From what I’ve seen, most of us tend to keep our diets rather low on kool-aid and don’t actually want garbage collection in Delphi.

But above that, mentioned ever-so-casually, is a term I don’t think I’ve ever seen before: run-time stack information. In fact, a Google search for “run-time stack information” (in quotes; without quotes we get nothing useful) turns up this podcast as the #1 result.  So what is run-time stack information?  It’s hard to say exactly, since I’m not Allen Bauer or a member of the Delphi team, but I can make a few educated guesses.

I’ll begin by assuming that the obvious similarity to “run-time type information” doesn’t end with the name.  So then Allen’s likely thinking about information about stack data that’s available at run-time, that would otherwise only be available in the source code and then get thrown away by the compiler.

So what goes on the stack?  Four things:  Parameters, local variables, implicit local variables, and stack frames.

Pretty much all the relevant data on parameters is already available through extended RTTI, at least for methods that have RTTI available.  So RTSI on parameters could probably be implemented simply by referencing the RTTI for the current function.  (Of course that would limit RTSI coverage to methods that actually have RTTI available.)  Not much of interest here.

At first glance, information on local variables would seem to be even less useful.  At its core, RTTI is about discovering things at runtime that can’t be statically verified at compile-time.  A routine already has all the information it needs about its own locals at compile-time.  Things get more interesting after looking at stack frames, though.

Implicit local variables can also go on the stack.  They’re for intermediate values. For example, if I write “x := x + y * z;” the compiler has to calculate y * z and store the result somewhere before it adds it to x.  Depending on various factors, that could go in a register or it could go on the stack.  Either way, I don’t think there’s much usefulness in creating data tables about implicit variables.

Stack frames are what makes the concept really interesting, though.  Here’s where we get information that’s not available at compile time.  Imagine if there was some way for a routine to determine what called it, and if that process could be continued for an arbitrary number of iterations.  We’d have an official API for examining the call stack from within Delphi.  There are various call stack examiners already available, especially as part of exception-logging packages, but they tend to depend on hacks and implementation details.  An official call-stack tracing API would be very useful, especially as Delphi starts to branch out into other platforms and architectures.

(Having said that, I’ve noticed that MadExcept frequently gets the call stack right in places where Delphi’s debugger gets lost.  Maybe Embarcadero should talk to Mathias Rauen about the possibility of integrating some of his code into the debugger?)

Call stack RTSI is also what make local variable and parameter RTSI interesting.  A function knows everything it needs to know about its own locals, but what if it could examine the locals and parameters of functions further up the call stack?  Admittedly, there’s probably not much practical use for this for most cases, but it would make it theoretically possible to build a debugger into your program.

Of course, a debugger runs on debug data, which is traditionally not included in a program, and for a good reason.  To make that information available at runtime, it needs to be included in the binary, where anyone with the right tools could get at it.  This is the sort of stuff that makes .NET assemblies trivially decompilable, unless you use an obfuscator that eliminates the benefits of having that level of detailed metadata around in the first place.  Anyone want to see a Delphi Reflector?

So, what is “run-time stack information”?  How is the team planning to implement it?  Is it merely a concept, or do they have any work in progress on implementing it so far?  It’s hard to say.  These are just my best guesses about what it might be and what might be done with it.  But it sure sounds interesting…

On a more technical level it suffices to say that they use custom and extremely compact and fast data structures, tricks and hacks to be able to represent millions and millions of objects within the constraints of a 32-bit Windows system. Throw in the use of Physical Address Extensions, storing per-class information in “virtual” class vars to reduce object instance size, creation of classes and their VMTs dynamically at runtime (!!), pointer packing, multithreading, the list just goes on and on.

I figured if Hallvard’s friend can create classes and their VMTs dynamically at runtime, then anyone could, with the right level of technical knowledge about how a VMT works.  My CodeRage presentation in 2009 was about theoretical research in that area, demonstrating that it could be done.  But then I just sorta sat on it.  It was a cool trick, but I didn’t have anything useful I could do with it, and  I had more important things to work on, like my game engine.

But a few months ago I started ran into some serious issues with making the scripting for my game engine work.  I’ve been using PascalScript, but its support for units is badly broken (and there’s no good way to fix it without basically rearchitecting the whole system) and its GOTO support is just broken enough to cause lots of trouble for me.  (I need to support GOTOs properly.  Don’t ask.)

So I looked at DWS, which has been getting a lot of hype lately with all the recent development.  But it also has bad unit support, no GOTOs at all, and no way to save a compiled script.  I raised each of these issues with Eric and he gave reasons why he has no intention to fix each of them.

There’s one interesting thing DWS can do, though: you can define new classes and use them in the script engine.  Being able to do this in my game engine would allow for a lot of customization opportunities.  But a bit of testing demonstrated that, like the holographic Professor Moriarty on Star Trek: The Next Generation, these newly-defined objects are purely virtual constructs that can’t step out into the real world.  If you try to bring a reference to one of these script objects into Delphi-land, you get a nil instead.  This greatly reduces the opportunity for customization.

So I figured I may as well try my hand at my own version, drawing on my class-creation research.  For the last few months, with a lot of hard work (and a fair bit of technical assistance from Barry Kelly on the arcane details of the RTTI system) I’ve been putting together a new Object Pascal-based script compiler.  It’s not completely finished yet, but the features that will set it apart are up and running:

• Unit-based compilation.  All your code doesn’t have to be in one unit.  It also doesn’t expect that you’re building a single script to be executed directly.  You can do that, but you can also run a valid compile cycle made up entirely of one or more units, with no program unit and no main routine, and use it as a library, calling individual routines from the script engine.  Units can be compiled and saved individually, similar to Delphi’s DCU system, and linked with other units to form a script program.
• Dynamic, RTTI-based creation of new classes.  You can define a new class in the script engine and the compiler will generate a new VMT and all the extended RTTI necessary to use this class from native Delphi code just like any other.  For example, you can create a class that descends from a native type, override a virtual method, pass an instance out of the script engine to a variable reference, call the virtual method, and the script engine will be invoked to run it.  (This particular trick require the TMethodImplementation class to set up, so Delphi XE or later will be required.)

The plan is to use the RTTI capabilities to achieve the most seamless integration possible between scripts and native code.  In my next few posts I’ll be going over some details as to how I made this all work.

I can’t comment on this article as it requires the commenter to be
logged in, and registration is disabled.

There are no plans for a more compact RTTI format. I’d love for there to
be, but the backward compatibility concerns are simply too large.
However, as more code relies on higher-level RTTI constructs, the scope
for freedom there increases incrementally. That’s all I was trying to
say.

– Barry

*sigh*  I thought I’d fixed the comments thing.  Anyone’s supposed to be able to post here with just a name and email address.  That’s why registration is disabled: no one should need to register!  Looks like it got reset somehow during the last WordPress update.

As for the RTTI, I guess I’m not surprised by that.  Backwards compatibility is really important to the Delphi team.  That’s why monstrosities like nested WITH statements are still with us.

I just hope Barry and the rest of the team finds time to fill in holes for XE2.  Specifically, it would be very nice to be able to use RTTI to access array properties.  Being able to get at standalone functions (not methods of objects) would be nice too.  That might require the addition of some sort of TRttiUnit class, which I definitely wouldn’t mind having either.

But that’s just an implementation detail.  What I really want are array properties and standalone methods, and a public interface to the TMethodImplementation class.  There’s a lot of potential in that class (introduced in the RTTI unit in Delphi XE) that’s unfortunately not available for public use.  When I asked Allen Bauer about it, he said the team wasn’t sure they wanted to have to support it in its current form, but it might be made more available in a future release.  I certainly hope so.

What do you think?  Are there any other holes in RTTI that need filling in, and what would you do with them if you had them available?

I mentioned in a comment that I wouldn’t like to see much change in the RTTI data structures because it would make dynamic class creation code difficult to maintain, and I’ve been doing some serious work in that area recently. Barry responded with the rationale:

It’s less a case of completely changing, than having more data appended on the end. It’s a real PITA to jam in extra data without breaking code. As it is, the RTTI format contributes a fairly substantial size penalty largely because its encoding can’t be optimized much. It would be nice to consider something like e.g. .NET metadata tables, which use a conciser representation. Instead, Delphi RTTI is left with every xref taking up a minimum of 4 bytes, and frequently with redundant strings for names that could be pooled. But that can’t happen within the current framework.

He mentions two things here: changing the way references work to make the RTTI take up less space, and making it less difficult to update the RTTI structures with new information.  I could get behind something like that, especially since the update would almost by necessity involve cleaning up the type definitions in the process.  It’s a real pain to work with some of the existing RTTI structures, because you get stuff like this:

{ vmtFieldTable entry in VMT }
  PVmtFieldTable = ^TVmtFieldTable;
  TVmtFieldTable = packed record
    Count: Word; // Published fields
    ClassTab: PVmtFieldClassTab;
   {Entry: array[1..Count] of TVmtFieldEntry;
    ExCount: Word;
    ExEntry: array[1..ExCount] of TVmtFieldExEntry;}
  end;

Why are those last three fields commented out?  Because they’re declared as static arrays, but the upper bound isn’t known at compile time and changes from one instance to another.  Basically, they contain dynamic arrays inserted inline into the record, which is something that Delphi’s type system doesn’t support.  And since the type system doesn’t support it and the compiler can’t locate the members of the records, extracting data from a TVmtFieldTable (or several other structures that do similar things) is a real pain.

RTTI that’s easier to work with and takes up less space in the EXE?  If they can pull it off, I’d love to see it!

It uses a lot of generic interfaced collections to pull this off without making a huge mess of the memory management.  This is particularly interesting because during the keynote at Delphi Live, someone (I think it was Mike Rozlog, but I’m not certain–where are the videos I saw some people taking?) said that they’re working on a more modern VCL.  I asked what they meant by that, and one of the answers was more interfaces in more places.

Between that and a lot of other things they’ve done recently, it seems like they want to bring the VCL to parity with the .NET framework.  But we’ve got a lot of problems with the interfaces, especially when you bring RTTI into it.  For example, a bit of fun I discovered yesterday.  What would you expect this program to output?

program Project3;

{$APPTYPE CONSOLE}

uses
  SysUtils, RTTI,
  dehl.Collections.base;

type
   IntList = IList<integer>;

   TMyObject = class
   private
      FName: string;
      FNumbers: IntList;
   end;

procedure Test;
const
   FIELD_STRING = 'Field: %s, Type: %s';
var
   ctx: TRttiContext;
   field: TRttiField;
begin
   ctx := TRttiContext.Create;
   for field in ctx.GetType(TMyObject).GetFields do
      writeln(format(FIELD_STRING, [field.Name, field.FieldType.Name]));
end;

begin
  try
    { TODO -oUser -cConsole Main : Insert code here }
    test;
    readln;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

You’d expect to see:

Field: FName, Type: string
Field: FNumbers, Type: IntList

Or maybe, IList<integer> as the type for FNumbers.  But do you know what you do get?  On both D2010 and Delphi XE, you get an access violation, since the compiler doesn’t generate RTTI for generic interfaces, so field.FieldType returns nil.

I had really hoped that that would be fixed in XE.  It’s disappointing to see that it hasn’t been.  The compiler needs to be able to generate RTTI for every valid type and member in the language.  (For even more fun, try using RTTI to discover information on the members of an interface, even simple ones like IInterface.  Go ahead, see what you come up with.)  We need full RTTI support for all generics, for indexed properties, and for odd types such as sets larger than 32 bits or non-contiguous enumerations.  Without this, it just feels like a crippled feature that’s just waiting to trip over something and throw an access violation in code that ought to work.