The first part is easy.  The second, not so much, if you want to use DBExpress.  This really should be handled internally as a special case inside the DBX driver.  Unfortunately neither Embarcadero nor Chau Chee Yang, maker of the alternative dbExpress Firebird driver, has released the source to their drivers, neither driver handles the BOOLEAN domain, and neither driver has any sort of callback/event handler that you can set up to intercept and modify the schema of a query result.  But I’m not gonna let a little thing like that stop me!

To know how to fix this, we first need to know what’s going wrong.  Here’s the basic overview:

To access database data through DBExpress, you generally use TSimpleDataset.  This is a TCustomClientDataset descendant that contains an internal dataset for DB access called TInternalSQLDataSet.  The outer layer has all the functionality that makes TClientDataset wonderful to work with.  You run a query, and it has the internal dataset access the database and pull back data, which gets stuffed into the client dataset for you to manipulate.

When the internal dataset talks to the database, it runs the query and gets back a record set, which includes data and a schema.  It iterates over the schema and prepares field defs for each column, then creates fields based on the field defs.  If you could catch and modify the field defs before it builds fields from them, you’d be good, but the whole process happens inside the InternalOpen method, with no conveniently-placed event handlers for you to interpose any code.

However, the InternalInitFieldDefs method, which sets up the field defs, is virtual, so it can be overridden in a descendant class.  Only problem is, we’re talking about the internal dataset here.  It’s kind of difficult to replace the internal dataset of your TSimpleDataset, because it’s a private field.  But the AllocDataSet method, which sets up the internal dataset, is also virtual.  With a little bit of RTTI surgery, we can set a value to a private field.

Things could get really messy at this point.  I could create a new dataset that descends from TSimpleDataset, but then I’d have to put it in a package, register it on the component palette, etc… unless I use an interposer class, a component with the same name that’s hidden from the form designer but gets instantiated instead of the original class at runtime.  So that’s our starting point.

Here’s my unit, weighing in at just under 100 lines of code plus the MPL header.  I’ll explain how it works at the bottom.  This does require Delphi 2010 or greater.

{*****************************************************************************
* The contents of this file are used with permission, subject to
* the Mozilla Public License Version 1.1 (the "License"); you may
* not use this file except in compliance with the License. You may
* obtain a copy of the License at
* http://www.mozilla.org/MPL/MPL-1.1.html
*
* Software distributed under the License is distributed on an
* "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
* implied. See the License for the specific language governing
* rights and limitations under the License.
*
*****************************************************************************
*
* This file was created by Mason Wheeler.  He can be reached for support at
* tech.turbu-rpg.com.
*****************************************************************************}
unit FirebirdDataset;

interface
uses
   SimpleDS;

type
  TSimpleDataset = class(SimpleDS.TSimpleDataset)
  protected
    procedure AllocDataSet; override;
  end;

implementation
uses
  Classes, SysUtils, RTTI, DB, Provider, SqlExpr,
  DBCommon;

type
  TInternalSQLDataSet = class(SimpleDS.TInternalSQLDataSet)
  private
    FQuery: TSqlQuery;
  protected
    procedure InternalInitFieldDefs; override;
  public
    constructor Create(AOwner: TComponent); override;
  end;

{ TSimpleDataset }

procedure TSimpleDataset.AllocDataSet;
var
  ctx: TRttiContext;
  cls: TRttiType;
  fld: TRttiField;
  FDataset: TInternalSQLDataSet;
  FProvider: TDataSetProvider;
begin
  cls := ctx.GetType(self.ClassType);

  fld := cls.GetField('FDataset');
  FDataSet := TInternalSQLDataSet.Create(Self);
  FDataSet.Name := 'InternalDataSet';
  FDataSet.SQLConnection := Connection;
  FDataSet.SetSubComponent(True);
  fld.SetValue(self, FDataset);

  fld := cls.GetField('FProvider');
  FProvider := fld.GetValue(self).AsType<TDataSetProvider>;
  FProvider.DataSet := FDataSet;
end;

{ TInternalSQLDataSet }

constructor TInternalSQLDataSet.Create(AOwner: TComponent);
const
  QUERY = 'select rFields.RDB$FIELD_NAME NAME ' +
          'from  RDB$RELATION_FIELDS rFields ' +
          'join RDB$FIELDS fields ON (rFields.RDB$FIELD_SOURCE = fields.RDB$FIELD_NAME) ' +
          'where  (rFields.RDB$RELATION_NAME=:tablename) and (fields.RDB$FIELD_TYPE = 7) ' +
          '  and (rFields.RDB$FIELD_SOURCE = ''BOOLEAN'') ' +
          'order by rFields.RDB$FIELD_POSITION ';
begin
  inherited Create(AOwner);
  FQuery := TSqlQuery.Create(self);
  FQuery.SQL.Text := QUERY;
  FQuery.Params.AddParameter.Name := 'tablename';
end;

procedure TInternalSQLDataSet.InternalInitFieldDefs;
var
  def: TFieldDef;
begin
  inherited InternalInitFieldDefs;
  if not (self.CommandType in [ctTable, ctQuery]) then
    Exit;

  FQuery.Active := false;
  //Why is TCustomSQLDataSet.SetConnection both virtual and private?!?
  FQuery.SQLConnection := self.SQLConnection;
  FQuery.ParamByName('tablename').AsString := UpperCase(GetTableNameFromQuery(self.CommandText));
  FQuery.Active := true;
  FQuery.First;
  FieldDefs.BeginUpdate;
  try
    while not FQuery.Eof do
    begin
      def := TFieldDef(TDefCollection(FieldDefs).Find(Trim(FQuery.FieldByName('NAME').AsString)));
      if assigned(def) then
         def.DataType := ftBoolean;
      FQuery.Next;
    end;
  finally
    FieldDefs.EndUpdate;
  end;
end;

end.

First comes the outer dataset.  It should operate exactly the same way as the original dataset, except with a modified inner dataset.  So I override AllocDataSet and replace it with a copy of the original code, except that it creates the new internal dataset and uses RTTI to access the private fields.  The internal dataset is where the interesting stuff takes place.

You can find out about Firebird domains by querying the internal metadata tables for the database.  That’s basically the only option available this late in the game, so I gave the new internal dataset an internal dataset of its own, a TSqlQuery which is designed for… well, you can probably guess.  The constructor creates the TSqlQuery and loads it with a parameterized query that will find any columns in a table that belong to a smallint-based domain named BOOLEAN.

Then, when InternalInitFieldDefs runs, it calls the inherited method, then sees if there’s anything it needs to fix.  Currently only single-table queries of type ctTable or ctQuery are supported.  If you run a ctQuery with a join on it, it’ll only get the first table.  Working around this would require building an actual SQL parser into the dataset, and I don’t really have time for that this evening.

It calls the RTL function DBCommon.GetTableNameFromQuery, which uses a very simple SQL parser to extract the table name. (If your query type is ctTable, it’ll just return the same table name you passed in.)  It sets the param on the internal TSqlquery and runs it, and then goes over the results. Any fields that get returned get checked against the field defs and their type is switched to ftBoolean.  Then this function returns, and InternalOpen goes on its merry way, setting up boolean fields for you.  Hooray!

But this really should be handled within the driver.  I hope that that can be arranged for the next version. (Of either one.)  Then we won’t need hacks like this.

Oh, and apparently certain aspects of the compiler have slowed down.  Most things will compile about the same speed or even a little faster, but for really large projects (millions of lines) with complex interdependencies between units, you’ll notice some slowdown, and it’ll apparently get worse the larger your project is.  I timed it at work, on a project of about 3.5 million lines of code.  It builds in about 2 minutes on D2010, closer to 3 minutes on XE.  Bad, but still a heck of a lot better than the C family could do.  And apparently it has something to do with making Generics work right, so I can tolerate that.  Hopefully they’ll find some way to regain some of that lost speed in updates, though.

Well, I finally got around to installing XE Update 1 at work today, and I timed the build.  2:13.

Unfortunately, one of my coworkers stumbled across a nasty new edge case that didn’t exist in XE RTM.  If you open a very large project, (the one at work is millions of lines of code, with thousands of units used in the DPR,) pull up the DPR, and do something that produces an error dialog, such as trying to Find an invalid text snippet, before compiling, the IDE locks up on you, burning one core at 100% until you kill it.

I opened a second instance of Delphi and attached the debugger and got some interesting results.  I don’t have Andreas Hausladen’s IDE-spelunking brilliance, but apparently the main thread is waiting on something to finish, and if I had to guess, I’d say that it’s running a background compile to build *Insight tables.

I submitted a QC report on this, but something went wrong somewhere and it’s not publicly viewable.  I put a comment on the report to that effect; hopefully someone at Emb. will change it.  So if you can see QC# 90020, feel free to vote for it.

So Congrats to Barry and the compiler team for getting the compile time back down on par with D2010, and a big raspberry to whoever introduced yet another bit of behavior that blocks the main thread for a perceptible length of time.  That’s something that should never happen. It’s gotten better recently, but it still happens far too often in the IDE.  Hopefully the next update will fix this issue.

procedure AnonLoop;
var
  i: integer;
  proc: TProc;
  ProcList: TList<TProc>;
begin
  ProcList := TList<TProc>.Create;
  for i := 1 to 5 do
    ProcList.Add(
      procedure()
      begin
        write(i, ' ');
      end);

  for proc in ProcList do
    proc;
  procList.Free;
end;

Intuitively, you’d think it would write “1 2 3 4 5″, but you’d be wrong.  It actually prints out “6 6 6 6 6″.  Huh?  6s?  You never fed it a 6!  And why are they all the same?  Well, it has to do with the way anonymous methods work.  Take a look at my analysis from a while back and see if it becomes any clearer.

Basically, you aren’t creating a new anonymous method each time you go through the loop.  The functor object is created one time only, on the begin line of the enclosing method, not once each time you hit the declaration in the code.

This is a bit disorienting, but technically necessary; if there could be more than one instance of the functor object created each time you run the enclosing method, it would be impossible to support multiple anonymous methods enclosing the same local variables, among other gotchas.

So what to do about it?  Seems to me that we have two problems.  First, you can’t create multiple anonymous methods by putting a declaration in a loop.  Second, the compiler hides this issue from you by allowing the anonymous method to capture the index variable of a for loop, which almost certainly results in silently generating bad code.

To create multiple anonymous methods in a loop, you need to call another function that returns an anonymous method.  This can make things a bit difficult, especially since it negates (or at least lessens) the ability of the anonymous method to capture local variables, which is what makes it so useful.

The compiler already can’t capture Result and var/out parameters, because they don’t belong to the current procedure.  Seems to me that the next version of the Delphi compiler should similarly check for this, and emit an error if an anonymous method tries to capture the index variable of a for loop, because that almost certainly doesn’t belong to the anonymous method.

BTW I know Barry Kelly reads at least some of my posts, because he occasionally responds with comments here when I write about technical issues. Barry, if you see this, would you mind weighing in on the problem? Thanks.

procedure TMyCustomChecklistPopupControl.ClosePopup;
var
  i: integer;
begin
  inherited ClosePopup;
  FInternalItemStringList.Clear;
  for i := 0 to Self.CheckedCount - 1 do
    FInternalItemStringList.Add(Self.CheckedItems[i].Name);
end;

At first glance, it looks just fine. It’s semantically correct–it will do what you want it to. If you happen to have seen a certain issue before, something might jump out at you, but if not, you probably think this is OK. And most of the time, it is.

This is a simplified version of something I ran into at work today, in one of our custom controls. I ran into it in the debugger, but not because it was raising exceptions or corrupting data. No, the problem was that when I hit the Check All button, selecting all 200 or so items, and then closed the popup, it took left the UI unresponsive for a good 15 seconds or so.

Turns out the problem isn’t in what this code was written to do, but in what else it does. You see, there’s an OnUpdate event handler attached to the internal TSwissArmyKnife TStringList which goes over the data in the list, calculates a few things, and updates some UI elements. And yeah, you want that to happen when you make a change. But you want it to happen once per change, from the user’s perspective. This was happening once per change from the TStringList’s perspective, or in other words, 200+ times in total for a single user action. And it took forever to finish.

You can be a really good programmer and still not know all the ins and outs of the framework you’re working with. I’m always discovering new little details about how things work. Turns out I’ve seen this one before, so when I hit Pause a few seconds in and dropped to the debugger, and saw the following right in the middle of the call stack, I knew what was going on right away.

TStringList.Changed
TStringList.InsertItem
TStringList.AddObject
TStringList.Add

What whoever coded this control apparently didn’t know, probably because they’d just never run across it before, was that Borland anticipated this very problem–or more liklely, because so many VCL classes use TStrings descendantes internally, they ran into it themselves at one point–and put a little switch into TStrings to turn off the OnChanged event handler temporarily.

Once I surrounded this code with a BeginUpdate and EndUpdate pair, the delay on closing up the box went from an angonizing 15 seconds to a tiny fraction of a second that I wouldn’t have noticed at all if I wasn’t watching for it.

Hopefully most of the people reading this are familiar with BeginUpdate and EndUpdate. But if anyone who hasn’t seen it runs across this, now you have a new trick. Please make sure to use it, to spare your end-users some pain. Even if you don’t think it’s likely to be necessary, please use it anyway. When this special checklist control was originally written, years ago, it was intended to hold a dozen or so items at most, not hundreds, and it probably performed fine at that scale. But growing client demand means the app’s working with more data than it used to, and eventually you hit something like this unless you’re careful in your design.

Delphi

It’s the standard, imperative, object-oriented paradigm that most coders are familiar with, but it gets all the little details right that the C family always gets wrong. Plus the more recent versions have started to add support for mixing in functional programming concepts, without running into the ugly abstraction inversions that typically come with functional languages.

I’ve gotten some interesting replies to that.  One of them asked, “to justify the C-bashing (so they don’t clobber you!), and to satisfy my curiosity, what are the little details Delphi/Pascal gets right?”

Well, I’d answer that in a comment, but there isn’t enough space in 500 characters to do it justice, so I figured I’d write up my answer here.  Let’s start with the elephant in the room.

SECURITY!!!

There is no good reason why anyone should ever under any circumstances write a networked program with security requirements–such as an operating system or a Web browser–in C, C++ or Objective-C.  We’ve known that for 22 years now, ever since Robert Tappan Morris released the Internet Worm that used buffer overflow exploits in Unix to crash about 10% of the Internet, causing tens of millions of dollars in damage.  (It was a very small Internet at the time.  Today, the cost would be measured in hundreds of billions, and it probably could be, if not more, if you add up all the damages lost due to buffer overflow and other C-specific vulnerabilities exploited over the last couple decades.)

This should have been a major wake-up call.  You cannot write programs that require reliable security in a language that was designed with no thought to it! At least not with any degree of consistency.  Just look at how many patches are still being issued today for Windows, Linux, OSX, iOS, and various Internet services and Web browsers, all due to buffer overflows.  And it’s getting worse.  We’ve got a lot of people talking about computerizing the power grid, which makes a lot of sense in theory, but it’s likely to open up a few hundred million new vulnerabilities to terrorists both domestic and foreign. (Who all’s seen Live Free Or Die Hard?  Anyone want to live in that world, except without Bruce Willis and that “I’m a Mac” kid to conveniently step in and save the day?)

It’s the same old issue over and over and over again.  It keeps showing up in C programs because it simply can’t be solved in C without breaking backwards compatibility.  In any sane world, the C language would have been dead by 1989–the Morris Worm having shown it to be utterly unsuitable for its intended purpose: building operating systems–and all our computers would be safer for it.  And it’s not like there are no alternatives.  By the time the worm hit, Apple had been building the most advanced operating system of its day for several years already, in Pascal.  Continuing to write Internet-facing OSes, browsers and apps in C (or C++ or Objective-C) ought to be treated as an act of criminal negligence.

In Delphi, on the other hand, we have a real string type, the best-thought-out string type I’ve seen in any language.  It’s reference-counted and grows and resizes automatically as needed, which frees the coder from dealing with string-size and string-memory hassles.  It’s bounds-checked, and it does not live on the stack, so there’s no way to use a Delphi string buffer overflow for a stack-smashing exploit.  Likewise, for non-string data, Delphi has a real array type which is also bounds-checked.  In fact, we’ve got two real array types, both bounds-checked, and the one whose size is not fixed (the more dangerous kind) also does not live on the stack.  We’ve also got a string-format routine that doesn’t use varargs, and string-output code that doesn’t assume its input is a format string in the first place, which means that Delphi programs are immune to format string exploits.

In the interest of fairness, of course, I should point out that as with any language that supports pointers, it’s still possible to write unsafe code in Delphi.  But you have to really go out of your way to do it; it’s not the default state of the language, the way it is in C!  (And removing pointers from a language causes more problems than it fixes.  This is why Java and C# both have explicit “unsafe” features: because they’re necessary to actually accomplish important tasks.)

Syntax and semantics

OK, enough ranting about security.  Let’s move on to other things.  A lot of C’s security problems have been fixed by managed languages like C# and Java.  But what they can’t fix is the syntax, at least not without abandoning a great deal of their C roots, which is a very important marketing device.  For example, C has no boolean type.  Some of its descendants do, but they haven’t managed to escape the ramifications of this blunder: because C has no boolean type, anything can be treated as a boolean.

I heard a really horrible pun a while ago:  The cake may be a lie, but Pi is always True.  (Because it’s a nonzero number, and in C, anything can be treated as a boolean.)  When everything is a boolean, including an assignment operation, it’s not safe to write “if x = 5″, no matter how intuitive that looks.  And when everything is a boolean, including a number, if you try to and two expressions, the compiler doesn’t know if you mean a logical or a bitwise and, so you need two versions of all the boolean operators.  And if you get them wrong, it might work, or you might end up with some very hard-to-debug issues.

Java, JavaScript and C# still have double versions of all the operators.  And I know the “if x = 5″ bug still exists in JavaScript.  (I’ve heard the compiler doesn’t accept it in C#.  Not sure about Java.)

In Delphi, a boolean is a boolean, and a number (or a string or an object) is not.  This means that we have one and, one or, one xor and one not, and the compiler knows what to do with them by looking at the operands.  And if you try to do something nonsensical like anding a boolean and a number, the compiler throws an error instead of silently accepting it and generating nonsensical code.

And while we’re on the subject of operators, can anyone tell me what * or & do in C?  “Well, it depends on whether–” Oh, I see.  Fundamental syntactic elements whose meanings are context sensitive.  How lovely.  In Delphi, “a * b” means multiplication and nothing else.  And for addressing and dereferencing, we’ve got the @ and ^ symbols, which actually make sense mnemonically.

Then we’ve got object-oriented programming.  C++’s object model is a big mess.  There’s no base object class, which means that there’s no way to pass an object of arbitrary type between one routine and another.  This also means that there’s no standardized way to take an object and get RTTI information about it.  And objects are value types, declared by default on the stack (or inline in the larger object that contains them), and passed around by value by default.  This wreaks havoc on inheritance and polymorphism.

For example, what’s the output of this program?  And if you change the signature of Foo to pass the object by reference, does it alter the output of the program?

#include <iostream>

class Parent
{
public:
        int a;
        int b;
        int c;

        Parent(int ia, int ib, int ic) {
                a = ia; b = ib; c = ic;
        };

        virtual void doSomething(void) {
                std::cout << "Parent doSomething" << std::endl;
        }
};

class Child : public Parent {
public:
        int d;
        int e;

        Child(int id, int ie) : Parent(1,2,3) {
                d = id; e = ie;
        };
        virtual void doSomething(void) {
                std::cout << "Child doSomething : D = " << d << std::endl;
        }
};

void foo(Parent a) {
        a.doSomething();
}

int main(void)
{
        Child c(4, 5);
        foo(c);
        return 0;
}

If you have to stop and reason about it for any length of time, that’s a warning sign.  I asked our local C++ expert at work what this would do, and he thought about it for a few minutes, came to a logical-sounding conclusion about what he thought it should have to do.  Then he wrote up the code above to test it, just to be sure.  (This is a guy who’s been writing C++ professionally since I was in high school, and he’s really good at it.  But even with all that experience, he’s not experienced enough to be confident what it would do without testing it.)

In Delphi, there aren’t a bunch of arcane passing and copying rules to keep track of when you’re working with polymorphism.  Objects are always reference types, so when you pass an object to a function, it passes that object, and when you call a virtual method on an object, it calls that object’s class’s version.  Always.

External code

There are at least technical reasons that can explain a lot of the above issues.  But here’s something really bizarre that I’ve never heard a good explanation for.  I had to debug a C DLL that one of my Delphi programs calls into a while ago to fix some problems in it.  I opened it up in Visual Studio and tried to get it to build.  Everything was syntactically correct, and it compiled just fine… and then failed at the link phase, because it couldn’t find the .lib file for a second DLL that this DLL requires.

.lib file?  What in the name of Turing is a .lib file?!?  Turns out it’s a file that describes… something… about the other DLL so that the linker can hook… something… up properly.  I really have no clue what it is or why it’s necessary.  I’ve never had to deal with them before.  In Delphi, if you need to link to a DLL, you put a function header in the code, declare that it’s an external reference, and provide the name of the DLL it’s found in, and that’s it.

The C code had all the same information: there was a .h file containing the function headers and… ohhhh, wait. Now I see what’s going on!  That’s actually the exact same .h file that’s in the DLL I’m linking against. So it doesn’t specify that these functions are external references, or where they’re found.  That information, even though it’s an important part of your source code, needs to be provided in a .lib file, a binary blob generated by the compiler of the external DLL, that’s not human-editable and not version-control friendly.  (And if your external DLL wasn’t written in a C family language, you’re in for even more fun trying to generate a .lib file.)

What it all boils down to is that, for some bizarre reason, Delphi does a more hassle-free job of linking to C DLLs than C does.  That doesn’t even make any sense, but it’s true.  Delphi can talk to external C code better than C can!

I could go on, (I haven’t even mentioned templates yet!) but this post is getting long enough already.  But I think the facts speak for themselves.  By paying attention to little details like the things I’ve mentioned here and thinking through the ramifications, the Delphi language designers have managed to build a language that is easier to work with and easier to write correct code in.  I hope this clarifies what I meant when I posted that.

It uses a lot of generic interfaced collections to pull this off without making a huge mess of the memory management.  This is particularly interesting because during the keynote at Delphi Live, someone (I think it was Mike Rozlog, but I’m not certain–where are the videos I saw some people taking?) said that they’re working on a more modern VCL.  I asked what they meant by that, and one of the answers was more interfaces in more places.

Between that and a lot of other things they’ve done recently, it seems like they want to bring the VCL to parity with the .NET framework.  But we’ve got a lot of problems with the interfaces, especially when you bring RTTI into it.  For example, a bit of fun I discovered yesterday.  What would you expect this program to output?

program Project3;

{$APPTYPE CONSOLE}

uses
  SysUtils, RTTI,
  dehl.Collections.base;

type
   IntList = IList<integer>;

   TMyObject = class
   private
      FName: string;
      FNumbers: IntList;
   end;

procedure Test;
const
   FIELD_STRING = 'Field: %s, Type: %s';
var
   ctx: TRttiContext;
   field: TRttiField;
begin
   ctx := TRttiContext.Create;
   for field in ctx.GetType(TMyObject).GetFields do
      writeln(format(FIELD_STRING, [field.Name, field.FieldType.Name]));
end;

begin
  try
    { TODO -oUser -cConsole Main : Insert code here }
    test;
    readln;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

You’d expect to see:

Field: FName, Type: string
Field: FNumbers, Type: IntList

Or maybe, IList<integer> as the type for FNumbers.  But do you know what you do get?  On both D2010 and Delphi XE, you get an access violation, since the compiler doesn’t generate RTTI for generic interfaces, so field.FieldType returns nil.

I had really hoped that that would be fixed in XE.  It’s disappointing to see that it hasn’t been.  The compiler needs to be able to generate RTTI for every valid type and member in the language.  (For even more fun, try using RTTI to discover information on the members of an interface, even simple ones like IInterface.  Go ahead, see what you come up with.)  We need full RTTI support for all generics, for indexed properties, and for odd types such as sets larger than 32 bits or non-contiguous enumerations.  Without this, it just feels like a crippled feature that’s just waiting to trip over something and throw an access violation in code that ought to work.

The upside of all this, though, is a very simple scripting system that doesn’t look much like a programming language, with a simple interface that almost anyone can pick up.  I wanted to keep that simplicity as much as possible, while adding the full flexibility and power of a real scripting language.  So I dreamed up EventBuilder, a set of objects which represent a high-level scripting interface and can also express themselves as PascalScript code.

I needed some way to create EventBuilder objects that could form a hierarchical tree that can represent blocks of code.  They needed to be easily serializable to some human-readable format so people can copy and paste blocks of EventBuilder script in order to share scripts, ask for help with debugging, etc.  And it needed to be ready quickly, since I want to be able to present as much of this as possible at Delphi Live! in August.

So is there any pre-existing system that supports hierarchical trees of objects and easy serialization to a simple text-based format?  The answer should be obvious to any experienced Delphi user:  descend from TComponent and use its built-in serialization to “DFM format.”  I tried that and, once I’d figured out how to handle a few quirks related to object ownership, it worked great!  All the infrastructure was there for me, tested and tried and proven over the last 15 years, and I could focus on the actual Event Builder logic.  It’s taken me about a month to get the system to a workable state, and now it’s more or less all ready.

Then I tried running a very, very large RPG Maker project through my project importer, and it took a long time on converting the global script block.  That’s sort of to be expected, since there are almost 2000 event scripts in there, but even so it felt like it was taking far too long for the amount of work involved.  I looked at my code and couldn’t find any obvious issues, so I ran it through Sampling Profiler.

It’s a good thing I did, too.  It found a very clear bottleneck in a place I’d have never thought to look.  Apparently I was spending 77% of my time in TComponent.Notification.  And why would I have never thought to look there?  Because I’ve never heard of it!  But apparently every time I added a component, it would recursively call this on the entire subtree, turning what ought to have been a O(n) conversion into O(n^2).

With a bit of research, it turns out that TComponent.Notification is for dealing with linked components.  For example, when you link a TDataset to a TDatasource, it needs a notification mechanism so it can clean up references if you free one of them.  Since EventBuilder doesn’t use linked components, I didn’t really need this functionality.  Good thing TComponent.Notification is virtual!  I overrode it with a blank method, and suddenly the conversion time dropped from about 12 seconds to about 3 seconds, and everything’s running smoothly again.

Moral of the story?  Be careful that you understand what you’re inheriting from, otherwise you might end up with killer kangaroos or other unwanted features.

After a bit of digging, I found out that when CloneCursor builds the field structure for the cloned dataset, it copies the FieldDefs from the original.  And FieldDefs only define data fields.  The field I was trying to display was a Calculated field, so I ended up without it.  Well, OK, that’s not such a big problem.  Just add a new calculated field, right?

begin
   calc := TWideStringField.Create(nil);
   calc.FieldName := 'DisplayName';
   calc.Size := 255;
   calc.FieldKind := fkCalculated;
   calc.Dataset := DataSet;

That ought to work, but it raises an exception.  “Cannot perform this operation on an open dataset.”   OK, I can understand that, almost.  It makes sense for data fields, because the underlying data store has to be set up in a certain way.  But for lookup and calculated fields that aren’t bound to the data store, it doesn’t make any sense.  Unfortunately, TField.SetDataset doesn’t care what kind of field you’re adding.  It checks to see if the dataset is inactive, and if not, boom!  It blows up in your face.

So I can’t add a calculated field to the dataset after CloneCursor has run, because CloneCursor calls Open and after that it’s too late.  So maybe I can add it before CloneCursor has run?  That almost works.  You end up with that calculated field in your cloned dataset… and nothing else.  Why?  Because about 2/3 of the way through TClientDataset.InternalOpen, it says “if DefaultFields then CreateFields;”  And DefaultFields is set by the method that calls InternalOpen, which starts like this:

procedure TDataSet.DoInternalOpen;
begin
  FDefaultFields := FieldCount = 0;
  InternalOpen;

Oops!  Again, not making any distinction between data fields and auxiliary fields.  FDefaultFields is a private field, and DefaultFields is a read-only property, so that makes it very difficult to change.

I could have created an entirely new TClientDataset on my DFM that duplicates the entire field structure of the dataset I’m trying to clone, including the calculated field, and that would have solved the immediate problem.  But then I’d have to make another copy of another dataset every time I find myself in a situation like this.  I’d really prefer to make it recognize that there are some predefined fields that it can integrate into the field structure.

Fortunately, InternalOpen is virtual.  All I need to do is create a subclass of TClientDataset that overrides InternalOpen and resets FDefaultFields to the correct value before calling inherited.  Except… how do you do that?  It’s a private field of TDataset, accessible only through a read-only property.

Prior to Delphi 2010, I wouldn’t have been able to do this.  But now, there’s a way to fix this sort of problem with extended RTTI. The solution looks like this. (This solution works specifically for TClientDataset, but you can apply it to any TDataset descendant easily enough.)

unit extensible_cds;

interface
uses
   DBClient;

type
   TExtensibleClientDataset = class(TClientDataset)
   protected
      procedure InternalOpen; override;
   end;

implementation
uses
   db, RTTI;

{ TExtensibleClientDataset }

procedure TExtensibleClientDataset.InternalOpen;
var
   context: TRttiContext;
   field: TField;
   value: boolean;
begin
   value := true;
   for field in self.Fields do
      value := value and (field.FieldKind <> fkData);
   context.GetType(self.ClassInfo).GetField('FDefaultFields').SetValue(self, TValue.From(value));
   inherited InternalOpen;
end;

end.

If this sounds overly complicated, that’s probably because it is. TField.SetDataset should only care about the dataset being open if you’re trying to add a data field. That would have solved everything. I submitted a report to QC about this a few months ago.  Hopefully it’ll be one of the things they fix for Delphi 2011.

The tradeoff, though, seems to have been compiler stability.  Trying to do anything with Generics in D2009 before Update 3 came out was a nightmare, and even after, (and even in D2010,) there were still plenty of dark corners where you can end up with an Internal Compiler Error or linker error on something that, syntactically speaking at least, is perfectly cromulent Object Pascal.

I think I ran into the darkest corner of all this morning, or at least the darkest one that’s been found yet.  I found something that not only won’t compile, but will either destabilize or completely crash the IDE if you try to compile it.

I was trying to build a specialized tree class.  Trees are pretty familiar to anyone who’s taken a Data Structures class:  Each TreeNode contains a Value and pointers to two or more other TreeNodes.  There’s an excellent implementation of the standard binary tree concept in DeHL, for example.  This works really well if you’re trying to build a standard binary tree, but when you want a hierarchical tree, (like the way the VCL’s TTreeView organizes information,) the standard binary tree concept doesn’t work so well.

I came across a situation while building the TURBU editor where I had a list of objects with an ID property and a Parent property, where both are integers, and I needed to come up with a list of all objects whose Parent value was equal to the ID value of a given object.  In other words, all immediate children of a certain node.

Obtaining all descendants of a binary tree node is trivial if you understand recursion.  But that’s not what I needed; I needed all immediate children, but not their children.  So a different kind of tree is needed, one composed of TLists of nodes instead of binary subtrees of nodes.  So that’s what I wrote.  Each node contains a value, a pointer to the list it’s in, and a pointer to the list containing its children, and each list is a list of nodes, plus a pointer to its parent node.  It ended up looking like this:

unit hierarchy_tree;

interface
uses
   SysUtils, generics.collections;

type
   THierarchyTreeList<T: class> = class;

   THierarchyTreeNode<T: class> = class
   private type

      TEnumerator = class(TEnumerator<T>)
      private
         FTopNode: THierarchyTreeNode<T>;
         FIndex: integer;
         FNext: TEnumerator;
         FTopRead: boolean;
      protected
         function DoGetCurrent: T; override;
         function DoMoveNext: Boolean; override;
      public
         constructor Create(value: THierarchyTreeNode<T>);
         function GetCurrent: T;
         function MoveNext: boolean;
         property Current: T read GetCurrent;
      end;

      TNodeList = THierarchyTreeList<THierarchyTreeNode<T>>;

   private
      FData: T;
      FRight: TNodeList;
      FList: TNodeList;
      function GetParent: THierarchyTreeNode<T>;
      function GetLevel: integer;
   public
      constructor Create(value: T);
      destructor Destroy; override;
      procedure Add(node: THierarchyTreeNode<T>); overload;
      procedure Add(value: T); overload;
      function GetEnumerator: TEnumerator;

      property List: TNodeList read FList;
      property Right: TNodeList read FRight;
      property Parent: THierarchyTreeNode<T> read GetParent;
      property Level: integer read GetLevel;
      property Data: T read FData write FData;
   end;

   THierarchyTreeList<T: class> = class(TObjectList<THierarchyTreeNode<T>>)
   private
      FParent: THierarchyTreeNode<T>;
      FLevel: integer;
      procedure SetParent(Value: THierarchyTreeNode<T>);
   public
      constructor Create(parent: THierarchyTreeNode<T>);
      function Add(value: THierarchyTreeNode<T>): integer;
      property Parent: THierarchyTreeNode<T> read FParent write SetParent;
      property Level: integer read FLevel;
   end;

implementation

{ No implementation provided.  Compiler will break before reaching this
point anyway. }

end.

You can try to compile this if you want, but make sure to save any work before you try.  Turns out this will break the compiler.  The compiler will try to process this, get stuck for about 30 seconds, all the while sucking up memory like there’s no tomorrow, until it hits 1.5 GB according to Task Manager, and then break.  It may crash, or it may simply become unstable.  If you’re really lucky, you’ll get an Out Of Memory error so at least you have some idea what’s going on.

After sharing this problem with a few other coders, I’ve got some idea what’s going on.  The node uses the list in its definition, and the uses the node as its generic parameter.  Apparently the compiler is trying to resolve one of these generics immediately, instead of waiting for instantiaion when a value of T can be provided, and getting stuck in an infinite loop, even though the definition is not infinitely recursive.

I’ve submitted a bug report about this to QC . Thanks to Moritz Beutel and Stephen Kamradt for the insights as to what was going on!