The first thing I knew about the release was when I received my SA email yesterday afternoon, as opposed to having to wait a few days or even weeks to get my upgrade as has happened in previous years.  Big improvement there, and I hope Embarcadero keeps it up for subsequent releases. Good.

No OSX support, which we all thought was going to be in this release up until a few weeks ago. Bad. But it’s not there because the people at Embarcadero preferred to focus on quality and not release something that wasn’t ready. Good. No one wants to see (or pay for) another D8.

So I open it up and start browsing through some code.  My settings from D2010 didn’t get migrated forward. Annoying. There really should be some way to import them as part of the installation process.

ErrorInsight is the same old same old, with false positives all over the place because it can’t find basic standard library units. Annoying. But at least it doesn’t choke on methods of objects from a generic list anymore. Good.

Speaking of Generics, huge improvement here.  I was able to go through my codebase and remove all the annoying workarounds for generics-related glitches, and it compiles and runs properly now! Good.

When it’s time to debug, I immediately noticed a few improvements.  We finally, finally, after all these years, have debug DCPs available for the standard libraries.  (For those not familiar with them, DCPs are to Delphi packages as DCUs are to units.)  That means that people like me who build stuff with packages are able to trace into the RTL when debugging.  Good.

Also, something that always drove me up the wall about the debugger has been fixed.  When you put the cursor in an expression and hit CTRL-F7, it pops up the Evaluate/Modify window and evaluates whatever you had the cursor on.  But if you moved the cursor and hit CTRL-F7 again without closing the Evaluate/Modify window, it would focus the Evaluate/Modify window, but it wouldn’t evaluate the new expression you had selected.  You’d have to close the window and hit CTRL-F7 again to get a new evaluation.  (Or copy/paste or type it in, of course.)  That always bugged me, but it’s been fixed in XE.  Good.

And while we’re on the subject of Evaluate/Modify, you can now select more than whatever the limit used to be (80 characters?) and have your entire selection copied when you hit CTRL-F7. Good.

But while poking around in there, I found a new way to crash the IDE.  Try to evaluate a call that will open a form, and it locks up Evaluate/Modify.  Then hit CTRL-F2 to kill the debugger, and it brings the whole program down. Bad. (Not sure if this was in earlier versions, though. I’ve never run across it before.)

Oh, there’s a new regular expression library in the RTL.  Some people will probably find this a good thing.  I personally see it as Annoying. Delphi code is supposed to be easy to read.  That’s an explicit design goal going all the way back to Wirth.  But PCREs are anything but, and having it right there in the standard library will just encourage people to use them in code that I might end up having to debug some day.

In addition to Generics bugs getting cleaned up, the Generics.Defaults and Generics.Collections units got a bit of polish.  There were some minor tweaks to speed up various low-level comparison functions in Generics.Defaults.  Over in Generics.Collections, all the collections have gained a ToArray method, TStack<T> and TQueue<T> gained a Capacity, and there’s a new TThreadedQueue<T> class that looks a lot like the Blocking Collection that Primoz Gabrijelcic set up for OmniThreadLibrary, only without all the Omni stuff.  These new features look Good.

The Math unit has been given overloads for Single, Double and Extended precision versions of just about all its calculation routines, which is Good because there’s no more implicit conversions to introduce rounding errors.  However, I had this bite me when compiling some code at work.  Someone was passing an array of integers to Math.Sum, which worked before, but now the compiler gave an ambiguous overload error for that. Annoying, but not too hard to fix, since there’s a SumInt function right after Sum.  Not sure why the original author of this code didn’t use it in the first place.  Oh well…

In RTTI.pas, there’s a new class called TMethodImplementation.  You create it and give it data that describes the signature of a method, then pass it an anonymous method that takes an array of TValues and returns a TValue, and it creates a little thunk of machine code that builds a function with the correct signature that invokes your anonymous method.  Very cool, lots of potential… except that you can’t use it.  The constructor’s private, and there’s a public constructor that raises an exception, so you can’t get at it via TObject.Create.  It’s apparently for internal use only.  Annoying. I asked Allen Bauer about this, and he said that it’s because they weren’t sure they had the final form of the class worked out yet, and they didn’t want to make it publicly available yet in a form that they’d end up having to preserve and support forever afterwards.

There’s also a new class called TVirtualMethodInterceptor that is publicly available, and uses TMethodImplementation internally to allow you to replace virtual methods on a class.  I’m a bit curious as to what this is useful for, but apparently the team thought it was good for something, because a lot of work went into setting it up. Good?

Oh, and apparently certain aspects of the compiler have slowed down.  Most things will compile about the same speed or even a little faster, but for really large projects (millions of lines) with complex interdependencies between units, you’ll notice some slowdown, and it’ll apparently get worse the larger your project is.  I timed it at work, on a project of about 3.5 million lines of code.  It builds in about 2 minutes on D2010, closer to 3 minutes on XE.  Bad, but still a heck of a lot better than the C family could do.  And apparently it has something to do with making Generics work right, so I can tolerate that.  Hopefully they’ll find some way to regain some of that lost speed in updates, though.

Oh, and there are little {$IFDEF POSIX} and {$IFDEF MACOS} tags scattered throughout the RTL and VCL, even more than there were in Delphi 2010.  This is Good if you’re looking forward to OSX or Linux cross-platform development in the next release(s).  I haven’t seen any 64-bit ifdefs, but that could be because it doesn’t require much change at the Pascal level, the way a completely different OS platform does. (UPDATE: According to PhiS, there’s a lot of {$IFDEF CPUX64} tags in System.pas.  I was poking around the libraries with BeyondCompare last night but I didn’t bother to look in there because I knew that, with all the cross-platform stuff and all the ASM they’ve got in System.pas, the diff view would be a big, confusing mess no matter what.)

Well, it’s getting late and I haven’t had time to dig through too much of the new stuff yet, (and with a new 1000-page novel and a new video game, it might be a while,) but overall this is looking like a pretty good release.  Not a huge leap forward like the last two were, but a lot of polish and incremental improvements.  I think this is going to be a pretty good release overall.

They’d have something almost ready, and then person A says “No, we can’t do this, we shouldn’t have X on there, it needs to be Y instead,” so they take X off and replace it with Y.  Then person B looks at it and says, “hey, what happened to X? That was critical!”  So Mike has to explain that it was removed because of person A, and get those two in a room together where they can discuss it and work out a compromise, and then they update the roadmap so it says XY.  And then of course, person C comes by and says, “hey, what’s with all this X stuff in there? I thought we took it out, because we won’t be able to get it implemented on this schedule!” And it’s back to square one.

Repeat a few dozen times, over a period of a few months, until finally they reach the Ultimate Compromise that everyone can bring themselves to agree upon, and then they finally publish the new version.  That’s the easy part.  Once everyone’s signed off on it, it takes about five minutes to get it posted to embarcadero.com.  He said that he’d have liked to publish the updated roadmap before the XE sneak previews started, but that just wouldn’t have worked on the timetable they had.  It was a real eye-opener, talking with him about the process.  It makes the roadmaps being the way they are a bit more understandable.

Mike also mentioned that a lot of people, including himself, don’t like the “PowerPoint slide” format that the last few roadmaps have been done in, and the next one will probably be done a different way.  Of course, working out what the new format will look like will be another long round of negotiations and compromises…

Again, apparently this is all completely normal for trying to publish a roadmap for a commercial product.  What do you guys think?  Has anyone out there been in a similar situation?  Are roadmaps always that messy, or has Mike just had a run of bad luck?

David I showed a fortune from a fortune cookie he got at a Chinese retaurant just last week.  “You will do well in the field of computer technology.”  With the release of the new Delphi just days away, let’s hope it turns out to be true.

Michael Swindell, one of the top executives at Embarcadero, gave some very interesting numbers.  Out of an estimated 16 million total programmers worldwide, 1.7 million are using Delphi.  More than 10%!  So you’re not alone, even when all the C# and Java people try to make you feel like you are.  I found this particularly interesting because last year, when I was on the phone with Emb. tech support for an installation hiccup with D2010, I was chatting with the rep and I asked him if he knew how many people were actually using Delphi.  He basically said that that was a trade secret of Embarcadero’s and he could lose his job if he told me that.  Looks like perspectives have changed somewhat, and for the better IMO.

Michael also mentioned that even with the economic downturn, Delphi’s grown pretty significantly last year, including the biggest increase in upgrades from D7 ever.  He attributed this specifically to their strong commitment to quality, which was the main reason behind the much-maligned OSX compiler schedule slip.  I don’t remember if it was him or Allen Bauer, who spoke a little later, but one of them said specifically that they have a working OSX compiler and OSX-ready VCL, that they could have shipped, but it wasn’t good enough to meet their quality standards so they held it back.  They’re really trying to avoid another Kylix or Delphi 8 style disaster, and they seem pretty sincere about it.

Probably the most exciting thing that Michael Swindell said was that they’re starting to aggressively market to schools again, “from universities down to middle schools,” trying to get Delphi into classrooms and into students’ hands, at the sort of prices that schools and students can afford, including some way (not many details on this) for students to use Delphi for free for educational purposes.  It’s good to hear some public recognition that new blood is very important to the heath of the Delphi community and that they’re working on targeting the low end of the market again.  Borland sent Delphi into a decline by being far too enterprise-oriented, and that attitude seems to have gotten inherited, and it’s good to see that they’re starting to really see past that now.

Tony de la Lama, the VP of R&D, mentioned that they’re putting a lot of emphasis within the company on actually doing new development in Delphi.  Before, a lot of it was built on other products and with other languages, but they’re moving towards using Delphi internally a lot more.

One complaint that I’ve heard raised several times since the launch of Delphi 2010 was about how DataSnap servers were crippled, or at least less useful than they could be, because there’s no built-in support for encryption.  That’s being remedied in XE, with a couple different encryption modes out of the box and a way to add other filters if necessary.

Also, I got an answer to a question that a lot of people have been asking about the XE previews.  I don’t know if they’ve answered this one before, but if they have I haven’t seen it yet.  A big part of the new functionality in XE will be bundled third-party software.  I asked at the end which products would be with which editions, and the answer was basically that all the new products they’ve been demoing would be in the Professional edition, except for FinalBuilder, which would be in Enterprise and above.

There was more, but those are the points that really stuck out to me.  Delphi XE may not be everything that everyone wanted or expected, but it looks like a pretty solid release and the team definitely has their heads on straight.  I think they’re really taking things in the right direction, much more so than in past years, and I’m (cautiously) optimistic about the future.  They presented a lot of really good plans.  We’ll see how they work out.

Code Review is a very useful principle for keeping bugs out of the codebase.  At work we’ve got a rule which is (usually) followed pretty strictly: Any non-trivial change that might affect functionality needs to be reviewed by another developer before checkin.  Usually this is just a “rubber stamp” sort of thing.  You pull up a diff view against the source control version in Beyond Compare and show the other guy what you changed, he says “OK, looks good, go ahead and check it in,” and you go ahead and check it in.  But every once in a while they’ll catch something.

It can be just about anything, from debug code (commented out or left active) still in the code you wrote, to formatting problems that make it harder to read, to “I really think you could do this better if you were to change X and Y”, or even “that looks like it won’t work if the user does X.”  Occasionally the reviewer will even catch a syntax error that would have broken the build if you’d checked it in, either because you didn’t test it by compiling first or because you forgot to save your latest changes in the IDE.

It happens to everyone.  I’ve found all of these issues in my coworkers’ code, and had them find all of them in mine, at one point or another.  And when these sorts of things do get caught before going into the official codebase, it improves the quality of your program.  This is basically a small-scale version of Linus’s Law.

But a formal code review is not the only way to apply the principle.  You can catch a lot just by reviewing your own code, especially in a diff viewer like Beyond Compare.  Seeing what’s changed highlighted in red gives you a slightly different perspective than just looking at the ordinary code view, and sometimes it brings a mistake to your attention.  I usually check the diffs myself at work before asking a coworker for a review, and I always check diffs before checking any code in to the TURBU project repository.

Independent review can also be useful in actually finding the problem you’re trying to solve.  Maybe once or twice a month, I get a “hey Mason, can you come look at this code with me?  I need another pair of eyeballs” request from someone at work.  Sometimes I can help track it down, sometimes I can’t, and sometimes I just provide a useful suggestion.

I remember one time someone asked me to help track down a data corruption issue.  When I started looking at the code, it was a really ugly implementation of an array-based stack that someone else had written long ago, using pointers and GetMem/ReallocMem calls all over the place.  Looked like something straight out of my Data Structures class in college.  We’ve got a joke at work that a bad programmer can write C in any language, and that’s exactly what this looked like: someone writing C in Delphi.

At least there was a well-defined interface encapsulating this whole mess, so I said “I don’t think I can trace this very well, but it’s probably an out-of-bounds issue.  Any good reason why you couldn’t rewrite this with Delphi’s dynamic arrays and let Bounds Checking find the error for you?”  He did, and sure enough the problem became obvious real quickly.  It was trying to do a Pop on an empty stack at one point, and the debugger pointed out exactly where once the compiler was able to apply error checking to the problem.

Reviewing your code, and having someone else review it if possible, is a good way to catch bugs that might end up slipping into the codebase unnoticed otherwise, and it can also be helpful a lot of times to track down tricky errors if someone else is watching the debugger with you.  If you don’t do it in your development work, try it and see if it makes a difference.  It’s sorta like a seatbelt.  Most of the time it won’t do much for you, but when it does it’s very helpful.

Then they mentioned AQTime, which looked interesting but IMO brings up more questions than it answers.  For example, “Since this is the standard edition, not all of the profilers are enabled.” (9:13)  But I don’t see any of the profilers on that list grayed out.  And again, will this be in all editions, or just the high-level ones?

Also, will it come with an extension to the Open Tools API to allow other profilers to be integrated?  AQTime has some great features, but its core feature, performance profiling, uses instrumenting that slows your app down horribly, which makes it unsuitable for a lot of uses.  I prefer Sampling Profiler, which gives a good picture of what you’re spending your time doing without instrumenting your program.  It doesn’t provide the perfect counting accuracy that instrumenting does, but most of the time you don’t actually need that anyway in order to track down performance problems.  If there was a way to integrate Sampling Profiler into Delphi, it would make a good compliment to AQTime’s other profilers.

As for CodeSite, I don’t have too much to say about that, because I don’t know enough about how it works to discuss it very well.  Logging’s definitely useful, but usually in different ways than what they were demonstrating on the video.  For example, if CodeSite can only output to that popup window, it’s worthless IMO.  But if it can write to a file, or better yet if it allows you to create and register custom outputs of some kind, that would be a big help.

The one thing I’m a bit worried about is price.  Delphi’s got three general problems, which need to be resolved independently.

1. Missing features.  No 64-bit, no cross-platform, no data binding, no LINQ, etc.
2. Bugs and quality issues.  Glitchy generics, *Insight, the helpfiles, etc.
3. Price.  No explanation needed

Delphi 2011′s main point was supposed to be addressing a missing feature.  That ended up not happening, as the team members have been explaining, because they don’t have enough time to get it working at a good quality level.  That’s definitely a praiseworthy attitude, but I’m a bit worried at seeing all these expensive third-party tools being suddenly announced as part of the product.  They don’t actually do anything to solve the three main problems facing Delphi, and I’m a bit worried that they’ll end up taking the price even higher.

They’ve mentioned their commitment to quality several times, and specifically that they’ve done a lot of work to fix the generics issues.  That’s great, and if they’ve allocated some resources to getting a Error Insight and Code Completion working properly that’ll be even better.  But even so, not having the main missing feature we were expecting would be delivered is going to do a lot to reduce the perceived quality of the release, even if the objective quality has gone up quite a bit since D2010.  Quality improvements are generally expected to be part of an update, not a new release.  (The “I paid good money for this with the expectation that the promised features would actually work” argument.)

I really hope I’m interpreting this wrong, but what it looks like to me is that in order to avoid the new version being perceived as “just a bugfix release” that isn’t worth the price, Embarcadero decided to bundle a bunch of expensive tools to make it appear to be more worth the cost.  If that’s what they’re doing, there are two ways to do it, and both will hurt them.  Either the price stays about the same as it was for D2010, but they’re paying a share of it to the owners of AQTime, FinalBuilder, etc, so they lose a lot of revenue, or they pass that cost along to us, the price goes up, (without delivering the anticipated new feature,) and a lot of people end up not wanting to buy it at that price, so they lose a lot of revenue.

What I would prefer, and what I would do if it was my product, is to negotiate some sort of deal with the creators of these other products where people who buy Delphi XE can pick them up at a discount, and then sell the actual Delphi license to us at a reduced price.  That would help improve sales, or at the very least help stem the loss of sales resulting from not having cross-platform available, and no one ends up paying for add-on products they don’t need.  (And not everyone will need them.  Where I work we’ve got a bunch of developers and one build machine.  If we all got a copy of FinalBuilder… what would we do with it?)

I really hope I’m wrong about this, but this is what it looks like Embarcadero is doing.  I just wish they wouldn’t, since I don’t think it will actually be good for Delphi users, or for Delphi itself.

The interesting thing here is that people from Embarcadero are responding immediately.  And it’s interesting what they’re saying, and what can be read between the lines.  We won’t know anything for certain until an official announcement is made, but it’s sure starting to sound like the cross-platform features got delayed for whatever reason and are still in active development but won’t be in Delphi XE.
Embarcadero’s Allen Bauer and John Kaster have both announced that they’re working on a new, updated roadmap that will be posted within a few days.  I haven’t seen it yet, but I already think it won’t really reassure or impress anybody.I can’t speak for everyone, or really for anyone but myself, but I tend to
be pretty good at reading people and particularly at reading groups of people, and from what I’ve seen, this new roadmap will strike most people in the Delphi community as rather underwhelming no matter what’s in it, not because of what it contains, but because of what it lacks.

The roadmap that would get people excited–the one that Embarcadero is never going to release, of course, because of the everpresent, nebulous, but all-important “business reasons” that always get in the way of doing stuff like this right–would replace the weasel words in slide 3 of the current roadmap.

If they were to lose the following:

This information describes Embarcadero’s general product
direction at this time, and should not be relied on in making a
licensing decision.  The future development, release and timing
of features and functionality remains at our sole discretion and
may be changed at any time without notice.

…and replace it with something like this:

This information describes Embarcadero’s commitment to the
Delphi community, the features that we *will* produce for the
next releases, and a concrete schedule that we are 100%
committed to meeting or surpassing.

…can you imagine the reaction?!?  It would finally give us something we’ve been sorely missing for the better part of a decade: a company behind Delphi that we know “gets it.”

Remember what Windows development used to be like before Delphi?  Delphi was the great innovator in programming, back in the day.  It was light-years ahead of VB, or any flavor of C++ for that matter.  Then Delphi 8 came along, and the message it gave everyone is, “.NET is the future and we can’t get it right.”  From that point on, Borland was dropping the ball on every occasion possible.  You’d think Murphy himself was running the company, and overseeing the development tools division personally.

Quality plummeted.  Delphi 8 was a joke, and 2005 and 2006 were scarcely better.  D2007 was tolerable, but that’s the best I can say about it.  We’re still waiting on 64-bit, which Dephi The Great Innovator, had it existed at the time, would have had ready and working no later than 2004.  And Borland managed to pretty much eradicate the community’s trust.  Then they sold off the dev tools division to Embarcadero, and we thought, “finally!  Delphi in the hands of a company that really ‘gets it’ again.”  But so far that’s only been half true.

The quality improvements have been immense.  For the first time since D7, Delphi 2009 gave us a solid IDE that was not downright painful to use.  (As long as you didn’t try to use Code Completion, or hit F1.)  Generics, anonymous methods, Unicode and extended RTTI have been major steps forward, but Delphi The Great Innovator is gone.

Borland fumbled and gave Microsoft 4 free years to work on Visual Studio and the .NET languages, and C# has gone from an utter joke, (“let’s take people from the Delphi team and have them rewrite Delphi in Java syntax, and leave out half the stuff that makes Delphi worth working with because we can’t implement them without stepping in Borland’s patents, but that’s all right because it’s Backed By Microsoft™ so everyone will use it anyway,”) to an industry-leading language.  Borland let that happen, Embarcadero inherited their mess, and we’re playing catch-up now.  And until we have working compilers for Windows, OSX and Linux, in 32-bit and 64-bit flavors, a helpfile that’s actually complete and doesn’t suck, *Insight that actually works and doesn’t suck, and some form of LINQ and some form of concurrency support at the language level, (that actually work and don’t suck,) we’re going to be stuck playing catch-up.

While the team under Embarcadero has made excellent progress in improving the quality of Delphi, they’ve done little to nothing to rebuild trust in the Delphi community.  The recent dual PR nightmares from firing Nick Hodges and the huge mess that the forum mods made of the whole Simon Kissel thing, it creates the sense that nothing has changed, that there’s still a bunch of PHBs undermining the team’s efforts.  And that brings me back to the roadmaps.  Apparently Borland had to talk in this sort of weasel words and say how they couldn’t really say anything until the product was ready to ship.  Had something to do with being a publicly traded company.  Well, that’s one anchor that Embarcadero doesn’t have around their necks.

Embarcadero is not a publicly traded corporation.  They aren’t beholden to a bunch of morons on Wall Street who only care about stock price and don’t know a thing about producing good development tools.  They aren’t bound by a lot of the laws and financial regulations that govern the behavior of publicly traded companies.  So why does it feel so much like they’re still doing business by Borland’s rules?

If Embarcadero wants to regain the trust of the Delphi community, it has to give us something to base that trust on.  A firm schedule with a  firm commitment behind it would do absolute wonders, and IMO the commitment would be even more important than the actual content of the roadmap.  Even if it said “x64 will be available in 2016,” if there was a firm date and we didn’t have to wonder if it would be here in 2012 or 2035 or ever, that would take a huge load off a lot of people’s shoulders.  (Including, I’d imagine, the shoulders of a lot of the engineers on the Delphi team!)

But of course, Embarcadero isn’t going to do that, are they?  They’re going to release another roadmap that says, right in the document itself, that we can’t actually trust what it says.  And then they’re going to wonder why the developers don’t seem to trust what they say…

“Why can’t I pass a TList<TMyDerivedObject> to a function that’s expecting a TList<TMyBaseObject>?  You can pass a TMyDerivedObject to a parameter expecting TMyBaseObject, so why doesn’t it work for lists?”

It really looks like it ought to be that simple, but unfortunately it’s not.  Trying to do this gets into a tricky issue in type theory known as covariance and contravariance, which is a formal way of describing the relationship between different types when one inherits from another.

It’s important to remember that object types don’t really exist at the binary level.  They’re an abstraction that makes it easier for us to work with them, but when you get down to it, all that’s there is a sequence of bytes.  As a strongly-typed language, it’s up to the Delphi compiler to make sure that, when you have a TMyDerivedObject, it doesn’t get replaced with a TMyIncompatibleObject that will cause things to crash or corrupt your data when you try to use it.  This is why you’re not allowed to pass descendant classes to a var parameter, for example:  because in the function using the var parameter, you’re allowed to replace the object with something else, and if the compiler can’t guarantee that the replacement will be of a compatible type, it prevents you from passing it in order to preserve type safety.

This is basically the same problem you get when dealing with generic lists.  Let’s say you have a TList<TMyDerivedObject> and you pass it to a function that expects a TList<TMyBaseObject>.  As long as all it does is reads the items in the list, you’re just fine.  But if this function calls .Add on the list and adds a TMyIncompatibleObject, (which also descends from TMyBaseObject, and so is perfectly legal in this context,) then you’ve violated type safety.  You have a list that’s supposed to only contain one type of object, and now there’s something incompatible inside.  And since this is done by calling a method on the list object and not by directly assigning one variable to another, the compiler can’t check to make sure you’re not doing something dangerous like this, so it forbids the entire concept in order to preserve type safety.

There’s a solution to this problem.  You can extend the language syntax a little so that the compiler does have a way to check this.  You have to mark the methods on the list so that the compiler knows that it’s safe for them to receive an object that descends from, or is an ancestor of, the specified generic type.  For example, you could mark TList<T>.Add with a hint that it’s OK to add descendants of the T type, but not ancestors.  Then the compiler uses this to determine more relaxed type safety rules that allow you to pass around different generic types without the danger of mixing incompatible types together.

The latest version of Prism supports this, but the Delphi team hasn’t implemented it yet.  I haven’t heard anything to indicate that they’re working on it for Delphi 2011 either.  Hopefully it they’ll find some way to implement it for Delphi 2012.

The upside of all this, though, is a very simple scripting system that doesn’t look much like a programming language, with a simple interface that almost anyone can pick up.  I wanted to keep that simplicity as much as possible, while adding the full flexibility and power of a real scripting language.  So I dreamed up EventBuilder, a set of objects which represent a high-level scripting interface and can also express themselves as PascalScript code.

I needed some way to create EventBuilder objects that could form a hierarchical tree that can represent blocks of code.  They needed to be easily serializable to some human-readable format so people can copy and paste blocks of EventBuilder script in order to share scripts, ask for help with debugging, etc.  And it needed to be ready quickly, since I want to be able to present as much of this as possible at Delphi Live! in August.

So is there any pre-existing system that supports hierarchical trees of objects and easy serialization to a simple text-based format?  The answer should be obvious to any experienced Delphi user:  descend from TComponent and use its built-in serialization to “DFM format.”  I tried that and, once I’d figured out how to handle a few quirks related to object ownership, it worked great!  All the infrastructure was there for me, tested and tried and proven over the last 15 years, and I could focus on the actual Event Builder logic.  It’s taken me about a month to get the system to a workable state, and now it’s more or less all ready.

Then I tried running a very, very large RPG Maker project through my project importer, and it took a long time on converting the global script block.  That’s sort of to be expected, since there are almost 2000 event scripts in there, but even so it felt like it was taking far too long for the amount of work involved.  I looked at my code and couldn’t find any obvious issues, so I ran it through Sampling Profiler.

It’s a good thing I did, too.  It found a very clear bottleneck in a place I’d have never thought to look.  Apparently I was spending 77% of my time in TComponent.Notification.  And why would I have never thought to look there?  Because I’ve never heard of it!  But apparently every time I added a component, it would recursively call this on the entire subtree, turning what ought to have been a O(n) conversion into O(n^2).

With a bit of research, it turns out that TComponent.Notification is for dealing with linked components.  For example, when you link a TDataset to a TDatasource, it needs a notification mechanism so it can clean up references if you free one of them.  Since EventBuilder doesn’t use linked components, I didn’t really need this functionality.  Good thing TComponent.Notification is virtual!  I overrode it with a blank method, and suddenly the conversion time dropped from about 12 seconds to about 3 seconds, and everything’s running smoothly again.

Moral of the story?  Be careful that you understand what you’re inheriting from, otherwise you might end up with killer kangaroos or other unwanted features.

Unfortunately, I don’t always get what I’d like.  I can’t really talk about stuff like that because I haven’t reached that point yet, due to various bugs and other implementation hurdles.

The installation and setup phase went quite well, actually.  The package was a 19 MB installer including full source code, DPKs and other requisite files for building the AnyDAC packages under anything from Delphi 5/C++ Builder 5 to RAD Studio 2010 (or FPC 2.2.x), a nice big helpfile and a fairly comprehensive Samples folder.  The download was nice and fast, and the only real annoyance was the installer getting a few paths wrong.  The helpfile is quite well-written in good, understandible English, which is good to see; that’s not always the case when working with libraries written by people from non-English-speaking countries.

I loaded up the TURBU project, backed up everything to source control, and went into dm_database.pas, where I keep my client datasets.  Time for the trial by fire:  Open the form designer, switch to text view, Select All, SyncEdit, replace all “TClientDataset” with “TADMemTable”, switch back to form view.  IMO, if something is presented as a replacement for something else, you ought to be able to swap it out and have it “just work.”  In practice, it’s rarely that simple.

TADMemDataset has the same support for filtering as TClientDatset, and it has a CloneCursor function, which is good because I use it pretty extensively.  What it doesn’t have, though, is the same public interface as TClientDataset.  For example, TClientDataset has a property called LogChanges, where if you turn it on it will save all edits to an intermediary datastore instead of the main one until you call MergeChangeLog or CancelUpdates.  Among other things, this makes implementing an editor dialog with Apply and Cancel buttons very simple.  TADMemDataset supports the same functionality, but the property that controls it is called CachedUpdates, not LogChanges, and to merge the change log you call CommitUpdates.

I got a bit of a rude surprise when I tried to convert one of my ClientDatasets to a TADMemTable: it didn’t like one of the fields.  The field was a TSingleField, and the internal storage table doesn’t know what a Single is, so it converts it to a Double, which causes errors when trying to open the dataset.  When I reported this to Dimitry, he replied that “ftSingle is not supported for a while.”

Also, using custom filtering with the OnFilterRecord event handler seems to break CloneCursor.  It looks like an order-of-operations issue inside TADDataset.InternalOpen: X needs to happen before Y, which needs to happen before Z, which is the way it’s currently set up, except that Z and Y both need to happen before X if you’re cloning a dataset with an OnFilterRecord event handler.  Apparently this is because X is doing too many different things, and needs to be reworked a bit.

I’ve reported all these issues to Dimitry and he says he’ll get a fixed version to me as soon as possible.  What I’ve seen so far of the dataset’s API looks interesting, but it’s kind of difficult to test for the moment, until the rough edges are polished.   I’ll post another update once I’m able to.

One other strange thing: In order to use the dataset, it requires access to some interface that has to be initialized by dropping a certain component onto your project that seems to have no other purpose except to make this interface available.  (Sorta like the XP Manifest component that does nothing except link a resource into your EXE.)  I’m not quite sure why TADMemTable can’t manage its own dependencies, but that’s kind of disappointing.  Hopefully that will get fixed in an update.

You have a base class that represents any code-generation object, and a bunch of classes that descend from it.  In order to manage things properly, you’re likely to have TList<TCodegenObject> and TStack<TCodegenObject> collections (or, worse still, non-generic TObjectList and TObjectStack containers) all over the place.  But TCodegenObject, like TObject, is an abstract base class that you only instantiate descendants of.

This can make debugging messy, especially if you break on an exception and Evaluate/Modify can’t read the locals anymore. You try to look inside your collections to see what’s going on, but all you see are TCodegenObject instances, not the actual object instances you’re working with. Sometimes that information is crucial to figuring out what’s going on, and you have to resort to ugly debugger hackery to track it down.

After this had happened to me a few too many times, it occurred to me that there’s something slightly ridiculous in Delphi’s object model. The first 4 bytes of the instance contain a reference to the object’s VMT, which defines the object’s class type. It’s a hidden area of memory, managed through “magic” and ugly pointer casting. For example, the TObject.ClassType function, which returns a TClass variable, which is defined as a pointer to a class’s VMT, (which defines the class type,) is implemented as:

function TObject.ClassType: TClass;
begin
  Pointer(Result) := PPointer(Self)^;
end;

Now maybe, for some strange reason, this made sense back in 1995 when Delphi’s object model was first being set up. But today there are two problems with it. 1, it’s ugly, and 2, it leads to unnecessary frustration by hiding useful information, especially since the Delphi debugger has a built-in evaluator that converts TClass references to class names.

You can probably see what I’m getting at. I’d like to see, in the next Delphi version if possible, that “magic” field replaced with a real field, like so:

TObject = class
  private
    FClassType: TClass;
  public
    //stuff that's already there
    //minus the ClassType method
    property ClassType: TClass read FClassType;
  end;

This remains syntactically equivalent to the existing implementation and would not break any existing code, and it has the added benefit of making polymorphism-related code easier to debug, since the FClassType field would be available in the debugger. I’ve thought about this and I can’t see any downsides. Can anyone see a potential problem with it?

I’ve submitted this to QC. Please vote it up if you like the idea.

EDIT:
I’m adding this because a few of the comments seem to be misunderstanding what I’m talking about. They think I want to add a new field to TObject, and they (correctly) think that adding one would increase overhead, which would be a bad thing.

Thing is, I’m not talking about adding any new overhead to TObject. That field is already there. It’s been there since Delphi 1. The first 4 bytes of every TObject instance is a field that contains a pointer to the object’s VMT, or in other words, a TClass. For whatever reason, the language architects decided not to expose it as a visible field of type TClass. What I’m suggesting is simply taking that hidden field and exposing it, not adding anything new that wasn’t already there all along.