TURBU Tech

On TVTropes, one of my favorite websites, there’s a phrase that’s only invoked when an author writes about something real, but uses it in a way that’s impossible and completely contrary to its nature: “X does not work that way.”  For example, if a sci-fi show features space fighters dogfighting and making turns that would only be possible with atmospheric friction against the hull, or gradually drifting to a stop if the engines stop thrusting, that might get listed under “Space does not work that way.”

What does this have to do with Delphi programming?  Bear with me.  I was going to write more about anonymous methods this evening, but then I ran across Wings of Wind’s latest blog post, about some cool things you can do with class helpers, especially in the newer versions of Delphi.  It was a very good article, with some cool ideas about how to associate data with classes.  And then I got to the end, where he presents a wish list and the first thing that went through my mind was “Class helpers do not work that way!”  It asks for a couple of things that are impossible without breaking the Delphi language:

Allow to override the virtual / dynamic members of the helped class. (hmm… hmmmm…. mmmm….)

Remove the “[DCC Error] Unit4.pas(75): E2169 Field definition not allowed after methods or properties” limitation – of course this will help also the Class Helpers, no? (if we believe the compiler, then this is the limitation which keeps us away for having fields in class helpers)

To understand why this can’t be done, you need to understand how classes, units, and class helpers work, and how the compiler treats them.  A class is contained in a unit, which the Delphi compiler compiles and turns into a DCU, one DCU for each unit.  The compiler does this a bunch of times and produces a collection of DCUs.  This collection is fed to the linker, which outputs an application, DLL or package, and everyone’s happy.  So far, so good.  But then you introduce class helpers, and there’s a problem.

Class helpers are used for extending classes that you can’t change the source to.  This could be because the original is in the VCL or another library and you don’t want to go patching it and then having to put up with all the “compiled with a different version of” hassles that that creates.  Or you could be using a plugin system and you want to separate functionality between libraries.  I used this technique to build an importer into the TURBU editor that can read other file formats.  But whatever reason you have, one thing you’re certainly not doing with your class helper is putting it in the same unit as the class it’s helping was declared in.  That makes no sense; you could just write the new methods into the original class.

So you write your class helper for some class in some other unit, and you add the other unit in your uses clause, and eventually the compiler comes along and starts to compile the unit with your class helper in it.  It looks at the uses clause and checks for the DCU for that unit.  If it has it, great.  If not, it goes and compiles the other unit first, then comes back to this one when it’s done.  Either way, the DCU of the original unit already exists in final form before it ever starts compiling the code of the class helper.

In addition to compiled code, the DCU also stores a certain amount of symbolic information to enable object-oriented programming to work across units.  It has to have metadata available about the public interface of the unit and its classes, so that code in other units can use (or inherit from) the classes, even without the original source available.  But the private data about the classes are encapsulated in the DCU and not exposed.  This is why class helpers can access public and protected members of their base class, but not private ones.  But when you try to override virtual methods and add fields, you’ve got a problem: where do you put them?

In order to dispatch a virtual method to the right class’s implementation of it, the compiler builds a virtual method table for the class with a bunch of pointers to the right locations.  By the time the linker finds the DCU for the unit with your class helper in it, it’s already linked up the original class and written its VMT.  If we override a virtual method, it would then have to go back to the original class and change the VMT pointer, and then go through and find every descendant of that class that it’s created since then and change their VMT pointers too, unless the descendant has also overridden that method, of course.  This would cause the compiler team some real headaches to try and work out.  Plus, if you do that, you’ve violated the scoping rules of the Object Pascal language:  now every call to that virtual method gets the overridden one, not just the calls written from a unit that uses the unit that your class helper was in.  (And it could be an even bigger mess for dynamic methods, because of the way their tables are implemented.)

Adding fields to the underlying class, on the other hand, wouldn’t be such a mess for the linker, but it would be for the compiler.  I’m not sure why it gives the error message it does, but that’s not the only problem that attempting this would pose.  Since the original unit is already compiled, in order to add to it, the compiler would have to crack open the DCU and stuff new data into it, and do so without breaking anything.  Plus, since this modifies the instance size of the class, you now have to hunt down all descendant classes that have been generated so far, in whatever DCUs they may be in, that introduce new fields and bump their offsets forward X number of bytes.  (Unless the linker handles that.  Not certain on that point.)  You’d be almost certain to have this problem, in fact, since the only real reason to want to add fields in a class helper, as opposed to just creating a new descendant class where these problems wouldn’t exist, is to slip these fields into existing descendant classes.  Oh, and since some of these DCUs we’re modifying might be system libraries that have to be reusable in other projects, you can’t rewrite them like that.  So instead you need to either store the modified versions in memory, or make copies and put them in some temp directory, and then make sure the linker looks there instead of grabbing the original.

All in all, this would be a nightmare for Barry Kelly and his team of miracle workers to try to implement at all, let alone get right .  And if it did get done, what would we end up with?  Eventually people would use it to create some sort of nightmare like .NET’s partial classes, a horrible misfeature that I’d really prefer not to ever see in Delphi.