Note: This is an alpha version.  There are probably plenty of bugs in it.  There are still features I haven’t gotten around to implementing.  The sight of the code would probably send Barry Kelly fleeing from the room, screaming in horror.  So use at your own risk, and I take no responsibility if this causes your system to crash, your data to come out all wrong, or demons to fly out your nose.  Specifically, the program keyword is not implemented yet, (only units,) var parameters don’t work yet, and a handful of features aren’t implemented in the script executor.

Having said that, here’s a few cool features I’ve come up with:

• Class creation:  Defining a new class inside the script code is not supported yet, but the basic framework is there, and script units implement new classes with their routines as methods, accessible to the script executor through the RTTI system.  I’ll add class creation pretty soon.
• Automatic registration: Ever try to register a class with the script engine in PascalScript?  It’s a painful process involving generating a registration unit that basically redeclares your entire class, and then has to be kept in sync or things break.  (And if you get the dependency order wrong, it fails silently,  leading to weird errors further down the line.)  I figured that was way too complicated, so for RTTIScript, all you have to do is register a class reference, and it uses the RTTI system to figure out the internal members and dependencies for you.
• Execution environment: You’ll often have a bunch of little standalone routines and global variables that you want to import into a script to have available at all times, to provide an interface between the script and your program.  So I introduced the concept of the “environment:” You create a class and register it with the compiler, and it treats all the class’s public methods and properties as globals for the script.  (This also means that you can use property getters and setters to have the script’s interaction with its “global variables” invoke side effects.)
• Script saving: I know Eric Grange thinks that a fast compiler is good enough and a bytecode version of a script is not necessary, but I don’t agree.  By his own tests, even a very fast script compiler can take 2 seconds to compile a 100,000 line script.  I’ve actually got an example of one that big in a project that the author wants to convert over once the TURBU project is ready.  Adding a two-second lag into the loading process and making the player wait while a level loads, which should feel instantaneous, is unacceptable.  I don’t have script saving implemented yet, but it’s definitely on the to-do list.

Anyway, the alpha version is available for any guinea pigs curious developers out there who’d like to take a look at it.  It requires Delphi XE, it’s pretty fast, and when I get the rest of the features set up it’s going to be awesome!  Feel free to check it out, and to submit a bug report if you find some errors. (Which I’m sure plenty of people will.)

I expanded on his idea and sent a suggested implementation to Barry Kelly.  Unfortunately, it didn’t make it into XE, but it’s in XE2 now.  It’s not exactly the code I suggested–a few things have changed in the way TValue works internally since D2010, and there’s at least one bugfix that I can see–but the basic idea is there.  When storing values into a TValue and retrieving them again, there’s special code to set and retrieve the most common types directly, without having to go into the generic RTTI-based (much slower) conversion routines.

I reran the timing test under XE2, and the results were much better this time:

Variants: 2140
TValues: 2895

Still about 30% slower than Variants, but at least it’s not 2000% slower anymore.  Thanks, Barry, and thanks to Robert too for the original suggestion!

Both Firebird DBX drivers blew up on me when I tried to do this, in different ways.  The Embarcadero driver may or may not have had a problem with the booleans, but I never really got the chance to test that out because the object I was trying to write contained a blob field, and I got an exception from Firebird: “Incorrect values within sqlda structure.”  A bit of Googling revealed that this is a known issue with writing blobs to Firebird that’s been in the Embarcadero driver for several years now. Apparently it’s still around.  Not much I can do about that without the source code.

So I tried Chau Chee Yang’s driver.  It didn’t have any trouble with the blobs, but it did choke on writing a Boolean field.  I traced into the call into the database driver and looked at the assembly, and as near as I can tell, his implementation of the DBXWritableRow_SetBoolean function is:

begin 
 assert(false); 
 result := 0; 
end;

Not hard to guess why.  Firebird doesn’t support booleans, so no need to waste time writing a method for it.  However, this is something I can fix without access to the driver source, with a little bit more RTTI surgery.

The standard version of the BOOLEAN domain is implemented as a special-case version of a smallint.  So what we need is a way to write the value back out as a smallint instead of a boolean.  When the driver DLL gets loaded by the DBExpress engine, it builds a method table object containing a bunch of function pointers to various functions in the DLL.  One of them is the aforementioned DBXWritableRow_SetBoolean.  There’s also a DBXWritableRow_SetInt16 that deals with writing out smallints.  So a redirector function is easy enough to write:

var
  LSetSmallint: TDBXWritableRow_SetInt16;

function Passthrough_SetBoolean(Handle: TDBXWritableRowHandle;
  Ordinal: TInt32; Value: LongBool): TDBXErrorCode; stdcall;
begin
  if value then
     result := LSetSmallint(handle, ordinal, 1)
  else result := LSetSmallint(handle, ordinal, 0);
end;

(Yes, I need that if statement in there instead of simply calling Ord(value).  As a LongBool, any nonzero value is considered True, and sometimes you’ll get things other than 1, for whatever reason.)  LSetSmallint represents the current driver’s DBXWritableRow_SetInt16 function.  Now you’ve just got to insert it into the method table.  It’s buried under several inaccessible layers of encapsulation, of course, but the RTTI objects can dig it out.

procedure FixConnection(connection: TSqlConnection);
var
   ctx: TRttiContext;
   cls: TRttitype;
   fld: TRttiField;
   driver: TDBXDriver;
   table: TDBXMethodTable;
begin
   cls := ctx.GetType(connection.DBXConnection.ClassType);
   driver := cls.GetField('FDriverDelegate').GetValue(connection.DBXConnection).AsType<TDBXDriver>;

   cls := ctx.GetType(driver.classtype);
   driver := cls.GetField('FDriver').GetValue(driver).AsType<TDBXDynalinkDriver>;

   fld := ctx.GetType(TDBXDynalinkDriver).GetField('FMethodTable');
   table := fld.GetValue(driver).AsType<TDBXMethodTable>;
   table.FDBXWritableRow_SetBoolean := Passthrough_SetBoolean;
   LSetSmallint := table.FDBXWritableRow_SetInt16;
end;

This has to be run after you open the connection but before you try to write any booleans out.

BTW I emailed Chau Chee Yang about all this.  He wrote back that he plans to support booleans in his driver “as soon as possible.”  So let’s hope that he finds the time soon, and that Embarcadero is able to free up some of their limited development resources to fix up their driver issues.  Tricks like this may be fun to dig around and build workarounds for, but in the end RTTI surgery is kind of an ugly hack that violates encapsulation, and it would be nice to not have to use it.

The first part is easy.  The second, not so much, if you want to use DBExpress.  This really should be handled internally as a special case inside the DBX driver.  Unfortunately neither Embarcadero nor Chau Chee Yang, maker of the alternative dbExpress Firebird driver, has released the source to their drivers, neither driver handles the BOOLEAN domain, and neither driver has any sort of callback/event handler that you can set up to intercept and modify the schema of a query result.  But I’m not gonna let a little thing like that stop me!

To know how to fix this, we first need to know what’s going wrong.  Here’s the basic overview:

To access database data through DBExpress, you generally use TSimpleDataset.  This is a TCustomClientDataset descendant that contains an internal dataset for DB access called TInternalSQLDataSet.  The outer layer has all the functionality that makes TClientDataset wonderful to work with.  You run a query, and it has the internal dataset access the database and pull back data, which gets stuffed into the client dataset for you to manipulate.

When the internal dataset talks to the database, it runs the query and gets back a record set, which includes data and a schema.  It iterates over the schema and prepares field defs for each column, then creates fields based on the field defs.  If you could catch and modify the field defs before it builds fields from them, you’d be good, but the whole process happens inside the InternalOpen method, with no conveniently-placed event handlers for you to interpose any code.

However, the InternalInitFieldDefs method, which sets up the field defs, is virtual, so it can be overridden in a descendant class.  Only problem is, we’re talking about the internal dataset here.  It’s kind of difficult to replace the internal dataset of your TSimpleDataset, because it’s a private field.  But the AllocDataSet method, which sets up the internal dataset, is also virtual.  With a little bit of RTTI surgery, we can set a value to a private field.

Things could get really messy at this point.  I could create a new dataset that descends from TSimpleDataset, but then I’d have to put it in a package, register it on the component palette, etc… unless I use an interposer class, a component with the same name that’s hidden from the form designer but gets instantiated instead of the original class at runtime.  So that’s our starting point.

Here’s my unit, weighing in at just under 100 lines of code plus the MPL header.  I’ll explain how it works at the bottom.  This does require Delphi 2010 or greater.

{*****************************************************************************
* The contents of this file are used with permission, subject to
* the Mozilla Public License Version 1.1 (the "License"); you may
* not use this file except in compliance with the License. You may
* obtain a copy of the License at
* http://www.mozilla.org/MPL/MPL-1.1.html
*
* Software distributed under the License is distributed on an
* "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
* implied. See the License for the specific language governing
* rights and limitations under the License.
*
*****************************************************************************
*
* This file was created by Mason Wheeler.  He can be reached for support at
* tech.turbu-rpg.com.
*****************************************************************************}
unit FirebirdDataset;

interface
uses
   SimpleDS;

type
  TSimpleDataset = class(SimpleDS.TSimpleDataset)
  protected
    procedure AllocDataSet; override;
  end;

implementation
uses
  Classes, SysUtils, RTTI, DB, Provider, SqlExpr,
  DBCommon;

type
  TInternalSQLDataSet = class(SimpleDS.TInternalSQLDataSet)
  private
    FQuery: TSqlQuery;
  protected
    procedure InternalInitFieldDefs; override;
  public
    constructor Create(AOwner: TComponent); override;
  end;

{ TSimpleDataset }

procedure TSimpleDataset.AllocDataSet;
var
  ctx: TRttiContext;
  cls: TRttiType;
  fld: TRttiField;
  FDataset: TInternalSQLDataSet;
  FProvider: TDataSetProvider;
begin
  cls := ctx.GetType(self.ClassType);

  fld := cls.GetField('FDataset');
  FDataSet := TInternalSQLDataSet.Create(Self);
  FDataSet.Name := 'InternalDataSet';
  FDataSet.SQLConnection := Connection;
  FDataSet.SetSubComponent(True);
  fld.SetValue(self, FDataset);

  fld := cls.GetField('FProvider');
  FProvider := fld.GetValue(self).AsType<TDataSetProvider>;
  FProvider.DataSet := FDataSet;
end;

{ TInternalSQLDataSet }

constructor TInternalSQLDataSet.Create(AOwner: TComponent);
const
  QUERY = 'select rFields.RDB$FIELD_NAME NAME ' +
          'from  RDB$RELATION_FIELDS rFields ' +
          'join RDB$FIELDS fields ON (rFields.RDB$FIELD_SOURCE = fields.RDB$FIELD_NAME) ' +
          'where  (rFields.RDB$RELATION_NAME=:tablename) and (fields.RDB$FIELD_TYPE = 7) ' +
          '  and (rFields.RDB$FIELD_SOURCE = ''BOOLEAN'') ' +
          'order by rFields.RDB$FIELD_POSITION ';
begin
  inherited Create(AOwner);
  FQuery := TSqlQuery.Create(self);
  FQuery.SQL.Text := QUERY;
  FQuery.Params.AddParameter.Name := 'tablename';
end;

procedure TInternalSQLDataSet.InternalInitFieldDefs;
var
  def: TFieldDef;
begin
  inherited InternalInitFieldDefs;
  if not (self.CommandType in [ctTable, ctQuery]) then
    Exit;

  FQuery.Active := false;
  //Why is TCustomSQLDataSet.SetConnection both virtual and private?!?
  FQuery.SQLConnection := self.SQLConnection;
  FQuery.ParamByName('tablename').AsString := UpperCase(GetTableNameFromQuery(self.CommandText));
  FQuery.Active := true;
  FQuery.First;
  FieldDefs.BeginUpdate;
  try
    while not FQuery.Eof do
    begin
      def := TFieldDef(TDefCollection(FieldDefs).Find(Trim(FQuery.FieldByName('NAME').AsString)));
      if assigned(def) then
         def.DataType := ftBoolean;
      FQuery.Next;
    end;
  finally
    FieldDefs.EndUpdate;
  end;
end;

end.

First comes the outer dataset.  It should operate exactly the same way as the original dataset, except with a modified inner dataset.  So I override AllocDataSet and replace it with a copy of the original code, except that it creates the new internal dataset and uses RTTI to access the private fields.  The internal dataset is where the interesting stuff takes place.

You can find out about Firebird domains by querying the internal metadata tables for the database.  That’s basically the only option available this late in the game, so I gave the new internal dataset an internal dataset of its own, a TSqlQuery which is designed for… well, you can probably guess.  The constructor creates the TSqlQuery and loads it with a parameterized query that will find any columns in a table that belong to a smallint-based domain named BOOLEAN.

Then, when InternalInitFieldDefs runs, it calls the inherited method, then sees if there’s anything it needs to fix.  Currently only single-table queries of type ctTable or ctQuery are supported.  If you run a ctQuery with a join on it, it’ll only get the first table.  Working around this would require building an actual SQL parser into the dataset, and I don’t really have time for that this evening.

It calls the RTL function DBCommon.GetTableNameFromQuery, which uses a very simple SQL parser to extract the table name. (If your query type is ctTable, it’ll just return the same table name you passed in.)  It sets the param on the internal TSqlquery and runs it, and then goes over the results. Any fields that get returned get checked against the field defs and their type is switched to ftBoolean.  Then this function returns, and InternalOpen goes on its merry way, setting up boolean fields for you.  Hooray!

But this really should be handled within the driver.  I hope that that can be arranged for the next version. (Of either one.)  Then we won’t need hacks like this.

I’m able to build and run some simple scripts, but I don’t have all the language features implemented yet, and I’m waiting on a bugfix for an issue in Collections that’s causing trouble for me.  (A big thanks to Alex for putting up with all my bug reports and feature requests, BTW!)

RTTI generation probably sounds intimidating, but the framework for it all condenses down to about 1200 lines of code spread across 4 units.  There are a couple more minor things I still need to add to it, but it’s not all that complicated when you get down to it.  Here’s a quick overview of the four units and what they do:

vmtStructure: A very simple unit that redefines the VMT of a class and several of the RTTI structures from TypInfo.pas in ways that are easier to work with.  A lot of it is based on work by Hallvard Vassbotn.

vmtBuilder: This unit has four routines for creating parts of the VMT and RTTI tables.  The RTTI structures can be very fiddly; they pack data in ways that is not actually supported by the Object Pascal language, such as inline variable-length arrays.  Reading them (or writing them) requires a lot of pointer-based math, so I wrote up these routines to encapsulate the scary work.  Just pass in the data that the structures are supposed to contain, and they’ll arrange it for you.

The vmtBuilder unit also introduces an interface called IBuffer, which provides a simple way to pack arbitrary data together.  It’s mostly just for internal use by the other RTTI-creation routines.

newClass: Here’s where the real fun begins.  The TNewClass class contains a bunch of methods for defining your new class itself.  You create the TNewClass, call the Add*Field methods (AddIntegerField, AddObjectField, etc.) in order, call AddMethod for each method, and then call the ClassPtr method, which takes all the information you’ve fed it and turns it into a new VMT and accompanying RTTI tables.  You can optionally pass a TPrivateHeap instance to ClassPtr, which will place the new data into the private heap instead of the standard application memory.  (I put that in because it’s very useful for dealing with an arcane but important detail of the RTTI system.)

TNewClass currently does not support creating properties, because they’re just syntactic sugar and I was focused on real functionality.  This will change once my script compiler gets to the point that it requires them.  It also doesn’t support creating classes that implement interfaces.  That would add a lot of complexity and I don’t think it’s worth the effort.  But the whole thing’s open source, so if someone needs it and feels like creating a patch for it, they’d be welcome to.

rttiPackage: Here’s the part where I stop drawing on Hallvard’s work and instead owe a big debt of gratitude to Barry Kelly for explaining some of the trickier details.  If you only want to create a new class, you could stop with the three units above.  But if you want to integrate it into the RTTI system so you can get at its members, (which is basically the only way to use it since they aren’t available at compile time,) you need a way to feed all this new RTTI you’ve created into RTTI.pas.  That’s what the rttiPackage unit does.

It contains two new interfaces, INewUnit and IRttiPackage, and classes that implement them.  INewUnit can contain any number of TNewClass instances, and it organizes them so the RTTI system thinks they were all declared in the same unit.  Then IRttiPackage is the construct that pulls everything together.  It collects INewUnit instances and creates and registers a new TLibModule record, which makes RTTI.pas think you’ve loaded a new runtime package, so it loads the RTTI tables for the (fake) new package, containing the data for your new classes.

IRttiPackage.Install, the method that sets up all the data, can take an optional reference to a procedure that allows you to alter the code address of your new classes’ methods at install time.  This was to support the script engine.  When the script compiler builds the metadata for the methods, the actual code address doesn’t exist yet because the RTTI hasn’t been loaded, so it sets the code address as the offset within the script.  (This will also make it a lot easier to save the script and load it later.)  Then when the RTTI table gets loaded as the script engine is starting up, it can use TRttiMethod.CreateImplementation to build a bridge between native Delphi land and the script engine, and set the implementation stub’s address to the official code address for the new method.  It’s kind of an ugly trick, but it works, and it’s the only way I could find to make it work.

I mentioned in my last post that the compiler will require Delphi XE because it uses this trick. I’ll have to revise that further.  There’s a glitch in the setup code for Method Implementations in XE RTM that can cause bogus FPU exceptions if you use it enough. This was fixed in XE Update 1, so that will be the minimum requirements for the script compiler when it’s ready.  The RTTI generation code itself, though, will work under D2010, if anyone has other uses for it.

More to come soon…

On a more technical level it suffices to say that they use custom and extremely compact and fast data structures, tricks and hacks to be able to represent millions and millions of objects within the constraints of a 32-bit Windows system. Throw in the use of Physical Address Extensions, storing per-class information in “virtual” class vars to reduce object instance size, creation of classes and their VMTs dynamically at runtime (!!), pointer packing, multithreading, the list just goes on and on.

I figured if Hallvard’s friend can create classes and their VMTs dynamically at runtime, then anyone could, with the right level of technical knowledge about how a VMT works.  My CodeRage presentation in 2009 was about theoretical research in that area, demonstrating that it could be done.  But then I just sorta sat on it.  It was a cool trick, but I didn’t have anything useful I could do with it, and  I had more important things to work on, like my game engine.

But a few months ago I started ran into some serious issues with making the scripting for my game engine work.  I’ve been using PascalScript, but its support for units is badly broken (and there’s no good way to fix it without basically rearchitecting the whole system) and its GOTO support is just broken enough to cause lots of trouble for me.  (I need to support GOTOs properly.  Don’t ask.)

So I looked at DWS, which has been getting a lot of hype lately with all the recent development.  But it also has bad unit support, no GOTOs at all, and no way to save a compiled script.  I raised each of these issues with Eric and he gave reasons why he has no intention to fix each of them.

There’s one interesting thing DWS can do, though: you can define new classes and use them in the script engine.  Being able to do this in my game engine would allow for a lot of customization opportunities.  But a bit of testing demonstrated that, like the holographic Professor Moriarty on Star Trek: The Next Generation, these newly-defined objects are purely virtual constructs that can’t step out into the real world.  If you try to bring a reference to one of these script objects into Delphi-land, you get a nil instead.  This greatly reduces the opportunity for customization.

So I figured I may as well try my hand at my own version, drawing on my class-creation research.  For the last few months, with a lot of hard work (and a fair bit of technical assistance from Barry Kelly on the arcane details of the RTTI system) I’ve been putting together a new Object Pascal-based script compiler.  It’s not completely finished yet, but the features that will set it apart are up and running:

• Unit-based compilation.  All your code doesn’t have to be in one unit.  It also doesn’t expect that you’re building a single script to be executed directly.  You can do that, but you can also run a valid compile cycle made up entirely of one or more units, with no program unit and no main routine, and use it as a library, calling individual routines from the script engine.  Units can be compiled and saved individually, similar to Delphi’s DCU system, and linked with other units to form a script program.
• Dynamic, RTTI-based creation of new classes.  You can define a new class in the script engine and the compiler will generate a new VMT and all the extended RTTI necessary to use this class from native Delphi code just like any other.  For example, you can create a class that descends from a native type, override a virtual method, pass an instance out of the script engine to a variable reference, call the virtual method, and the script engine will be invoked to run it.  (This particular trick require the TMethodImplementation class to set up, so Delphi XE or later will be required.)

The plan is to use the RTTI capabilities to achieve the most seamless integration possible between scripts and native code.  In my next few posts I’ll be going over some details as to how I made this all work.

I mentioned in a comment that I wouldn’t like to see much change in the RTTI data structures because it would make dynamic class creation code difficult to maintain, and I’ve been doing some serious work in that area recently. Barry responded with the rationale:

It’s less a case of completely changing, than having more data appended on the end. It’s a real PITA to jam in extra data without breaking code. As it is, the RTTI format contributes a fairly substantial size penalty largely because its encoding can’t be optimized much. It would be nice to consider something like e.g. .NET metadata tables, which use a conciser representation. Instead, Delphi RTTI is left with every xref taking up a minimum of 4 bytes, and frequently with redundant strings for names that could be pooled. But that can’t happen within the current framework.

He mentions two things here: changing the way references work to make the RTTI take up less space, and making it less difficult to update the RTTI structures with new information.  I could get behind something like that, especially since the update would almost by necessity involve cleaning up the type definitions in the process.  It’s a real pain to work with some of the existing RTTI structures, because you get stuff like this:

{ vmtFieldTable entry in VMT }
  PVmtFieldTable = ^TVmtFieldTable;
  TVmtFieldTable = packed record
    Count: Word; // Published fields
    ClassTab: PVmtFieldClassTab;
   {Entry: array[1..Count] of TVmtFieldEntry;
    ExCount: Word;
    ExEntry: array[1..ExCount] of TVmtFieldExEntry;}
  end;

Why are those last three fields commented out?  Because they’re declared as static arrays, but the upper bound isn’t known at compile time and changes from one instance to another.  Basically, they contain dynamic arrays inserted inline into the record, which is something that Delphi’s type system doesn’t support.  And since the type system doesn’t support it and the compiler can’t locate the members of the records, extracting data from a TVmtFieldTable (or several other structures that do similar things) is a real pain.

RTTI that’s easier to work with and takes up less space in the EXE?  If they can pull it off, I’d love to see it!

The first thing I knew about the release was when I received my SA email yesterday afternoon, as opposed to having to wait a few days or even weeks to get my upgrade as has happened in previous years.  Big improvement there, and I hope Embarcadero keeps it up for subsequent releases. Good.

No OSX support, which we all thought was going to be in this release up until a few weeks ago. Bad. But it’s not there because the people at Embarcadero preferred to focus on quality and not release something that wasn’t ready. Good. No one wants to see (or pay for) another D8.

So I open it up and start browsing through some code.  My settings from D2010 didn’t get migrated forward. Annoying. There really should be some way to import them as part of the installation process.

ErrorInsight is the same old same old, with false positives all over the place because it can’t find basic standard library units. Annoying. But at least it doesn’t choke on methods of objects from a generic list anymore. Good.

Speaking of Generics, huge improvement here.  I was able to go through my codebase and remove all the annoying workarounds for generics-related glitches, and it compiles and runs properly now! Good.

When it’s time to debug, I immediately noticed a few improvements.  We finally, finally, after all these years, have debug DCPs available for the standard libraries.  (For those not familiar with them, DCPs are to Delphi packages as DCUs are to units.)  That means that people like me who build stuff with packages are able to trace into the RTL when debugging.  Good.

Also, something that always drove me up the wall about the debugger has been fixed.  When you put the cursor in an expression and hit CTRL-F7, it pops up the Evaluate/Modify window and evaluates whatever you had the cursor on.  But if you moved the cursor and hit CTRL-F7 again without closing the Evaluate/Modify window, it would focus the Evaluate/Modify window, but it wouldn’t evaluate the new expression you had selected.  You’d have to close the window and hit CTRL-F7 again to get a new evaluation.  (Or copy/paste or type it in, of course.)  That always bugged me, but it’s been fixed in XE.  Good.

And while we’re on the subject of Evaluate/Modify, you can now select more than whatever the limit used to be (80 characters?) and have your entire selection copied when you hit CTRL-F7. Good.

But while poking around in there, I found a new way to crash the IDE.  Try to evaluate a call that will open a form, and it locks up Evaluate/Modify.  Then hit CTRL-F2 to kill the debugger, and it brings the whole program down. Bad. (Not sure if this was in earlier versions, though. I’ve never run across it before.)

Oh, there’s a new regular expression library in the RTL.  Some people will probably find this a good thing.  I personally see it as Annoying. Delphi code is supposed to be easy to read.  That’s an explicit design goal going all the way back to Wirth.  But PCREs are anything but, and having it right there in the standard library will just encourage people to use them in code that I might end up having to debug some day.

In addition to Generics bugs getting cleaned up, the Generics.Defaults and Generics.Collections units got a bit of polish.  There were some minor tweaks to speed up various low-level comparison functions in Generics.Defaults.  Over in Generics.Collections, all the collections have gained a ToArray method, TStack<T> and TQueue<T> gained a Capacity, and there’s a new TThreadedQueue<T> class that looks a lot like the Blocking Collection that Primoz Gabrijelcic set up for OmniThreadLibrary, only without all the Omni stuff.  These new features look Good.

The Math unit has been given overloads for Single, Double and Extended precision versions of just about all its calculation routines, which is Good because there’s no more implicit conversions to introduce rounding errors.  However, I had this bite me when compiling some code at work.  Someone was passing an array of integers to Math.Sum, which worked before, but now the compiler gave an ambiguous overload error for that. Annoying, but not too hard to fix, since there’s a SumInt function right after Sum.  Not sure why the original author of this code didn’t use it in the first place.  Oh well…

In RTTI.pas, there’s a new class called TMethodImplementation.  You create it and give it data that describes the signature of a method, then pass it an anonymous method that takes an array of TValues and returns a TValue, and it creates a little thunk of machine code that builds a function with the correct signature that invokes your anonymous method.  Very cool, lots of potential… except that you can’t use it.  The constructor’s private, and there’s a public constructor that raises an exception, so you can’t get at it via TObject.Create.  It’s apparently for internal use only.  Annoying. I asked Allen Bauer about this, and he said that it’s because they weren’t sure they had the final form of the class worked out yet, and they didn’t want to make it publicly available yet in a form that they’d end up having to preserve and support forever afterwards.

There’s also a new class called TVirtualMethodInterceptor that is publicly available, and uses TMethodImplementation internally to allow you to replace virtual methods on a class.  I’m a bit curious as to what this is useful for, but apparently the team thought it was good for something, because a lot of work went into setting it up. Good?

Oh, and apparently certain aspects of the compiler have slowed down.  Most things will compile about the same speed or even a little faster, but for really large projects (millions of lines) with complex interdependencies between units, you’ll notice some slowdown, and it’ll apparently get worse the larger your project is.  I timed it at work, on a project of about 3.5 million lines of code.  It builds in about 2 minutes on D2010, closer to 3 minutes on XE.  Bad, but still a heck of a lot better than the C family could do.  And apparently it has something to do with making Generics work right, so I can tolerate that.  Hopefully they’ll find some way to regain some of that lost speed in updates, though.

Oh, and there are little {$IFDEF POSIX} and {$IFDEF MACOS} tags scattered throughout the RTL and VCL, even more than there were in Delphi 2010.  This is Good if you’re looking forward to OSX or Linux cross-platform development in the next release(s).  I haven’t seen any 64-bit ifdefs, but that could be because it doesn’t require much change at the Pascal level, the way a completely different OS platform does. (UPDATE: According to PhiS, there’s a lot of {$IFDEF CPUX64} tags in System.pas.  I was poking around the libraries with BeyondCompare last night but I didn’t bother to look in there because I knew that, with all the cross-platform stuff and all the ASM they’ve got in System.pas, the diff view would be a big, confusing mess no matter what.)

Well, it’s getting late and I haven’t had time to dig through too much of the new stuff yet, (and with a new 1000-page novel and a new video game, it might be a while,) but overall this is looking like a pretty good release.  Not a huge leap forward like the last two were, but a lot of polish and incremental improvements.  I think this is going to be a pretty good release overall.

After a bit of digging, I found out that when CloneCursor builds the field structure for the cloned dataset, it copies the FieldDefs from the original.  And FieldDefs only define data fields.  The field I was trying to display was a Calculated field, so I ended up without it.  Well, OK, that’s not such a big problem.  Just add a new calculated field, right?

begin
   calc := TWideStringField.Create(nil);
   calc.FieldName := 'DisplayName';
   calc.Size := 255;
   calc.FieldKind := fkCalculated;
   calc.Dataset := DataSet;

That ought to work, but it raises an exception.  “Cannot perform this operation on an open dataset.”   OK, I can understand that, almost.  It makes sense for data fields, because the underlying data store has to be set up in a certain way.  But for lookup and calculated fields that aren’t bound to the data store, it doesn’t make any sense.  Unfortunately, TField.SetDataset doesn’t care what kind of field you’re adding.  It checks to see if the dataset is inactive, and if not, boom!  It blows up in your face.

So I can’t add a calculated field to the dataset after CloneCursor has run, because CloneCursor calls Open and after that it’s too late.  So maybe I can add it before CloneCursor has run?  That almost works.  You end up with that calculated field in your cloned dataset… and nothing else.  Why?  Because about 2/3 of the way through TClientDataset.InternalOpen, it says “if DefaultFields then CreateFields;”  And DefaultFields is set by the method that calls InternalOpen, which starts like this:

procedure TDataSet.DoInternalOpen;
begin
  FDefaultFields := FieldCount = 0;
  InternalOpen;

Oops!  Again, not making any distinction between data fields and auxiliary fields.  FDefaultFields is a private field, and DefaultFields is a read-only property, so that makes it very difficult to change.

I could have created an entirely new TClientDataset on my DFM that duplicates the entire field structure of the dataset I’m trying to clone, including the calculated field, and that would have solved the immediate problem.  But then I’d have to make another copy of another dataset every time I find myself in a situation like this.  I’d really prefer to make it recognize that there are some predefined fields that it can integrate into the field structure.

Fortunately, InternalOpen is virtual.  All I need to do is create a subclass of TClientDataset that overrides InternalOpen and resets FDefaultFields to the correct value before calling inherited.  Except… how do you do that?  It’s a private field of TDataset, accessible only through a read-only property.

Prior to Delphi 2010, I wouldn’t have been able to do this.  But now, there’s a way to fix this sort of problem with extended RTTI. The solution looks like this. (This solution works specifically for TClientDataset, but you can apply it to any TDataset descendant easily enough.)

unit extensible_cds;

interface
uses
   DBClient;

type
   TExtensibleClientDataset = class(TClientDataset)
   protected
      procedure InternalOpen; override;
   end;

implementation
uses
   db, RTTI;

{ TExtensibleClientDataset }

procedure TExtensibleClientDataset.InternalOpen;
var
   context: TRttiContext;
   field: TField;
   value: boolean;
begin
   value := true;
   for field in self.Fields do
      value := value and (field.FieldKind <> fkData);
   context.GetType(self.ClassInfo).GetField('FDefaultFields').SetValue(self, TValue.From(value));
   inherited InternalOpen;
end;

end.

If this sounds overly complicated, that’s probably because it is. TField.SetDataset should only care about the dataset being open if you’re trying to add a data field. That would have solved everything. I submitted a report to QC about this a few months ago.  Hopefully it’ll be one of the things they fix for Delphi 2011.

“Thanks again Mason! You (and the other folks here at SO) pretty much close the documentation gap. Thank you so much. + 1″

Now, StackOverflow is a wonderful resource.  I’ve been using it and loving it ever since Jim McKeeth introduced me to it about a year and a half ago.  But I’ve never really thought of it that way.  It makes me wonder.

StackOverflow has been a huge success ever since it was launched.  It’s what Experts-Exchange should have been.  But I sort of got the impression that it’s supposed to be for asking somewhat difficult technical questions, whatever “somewhat difficult” happens to mean to you at your current level of competence.  If Delphi users are instead using it to compensate for bad documentation, maybe there’s a more appropriate way to go about doing that?

I know that Embarcadero has a documentation wiki set up.  Ideally, that’s where documentation issues ought to be resolved.  Perhaps there would be some way to create a request/answer system for improving the quality of the documentation on the wiki?  There would need to be some way to create documentation requests, a way to respond to them and tie your responses to doc wiki pages, and some way to actually bring users in and get them to participate.  Does anyone have any thoughts as to how this could be accomplished?