Dynamic class creation: moving beyond the theoretical
A few years back, I ran across this post by Hallvard Vassbotn. (It’s a shame he stopped blogging, because he always had some very interesting stuff about the technical details of how stuff in Delphi works.) At the bottom was a paragraph that really fascinated me:
On a more technical level it suffices to say that they use custom and extremely compact and fast data structures, tricks and hacks to be able to represent millions and millions of objects within the constraints of a 32-bit Windows system. Throw in the use of Physical Address Extensions, storing per-class information in “virtual” class vars to reduce object instance size, creation of classes and their VMTs dynamically at runtime (!!), pointer packing, multithreading, the list just goes on and on.
I figured if Hallvard’s friend can create classes and their VMTs dynamically at runtime, then anyone could, with the right level of technical knowledge about how a VMT works. My CodeRage presentation in 2009 was about theoretical research in that area, demonstrating that it could be done. But then I just sorta sat on it. It was a cool trick, but I didn’t have anything useful I could do with it, and I had more important things to work on, like my game engine.
But a few months ago I started ran into some serious issues with making the scripting for my game engine work. I’ve been using PascalScript, but its support for units is badly broken (and there’s no good way to fix it without basically rearchitecting the whole system) and its GOTO support is just broken enough to cause lots of trouble for me. (I need to support GOTOs properly. Don’t ask.) 🙁
So I looked at DWS, which has been getting a lot of hype lately with all the recent development. But it also has bad unit support, no GOTOs at all, and no way to save a compiled script. I raised each of these issues with Eric and he gave reasons why he has no intention to fix each of them.
There’s one interesting thing DWS can do, though: you can define new classes and use them in the script engine. Being able to do this in my game engine would allow for a lot of customization opportunities. But a bit of testing demonstrated that, like the holographic Professor Moriarty on Star Trek: The Next Generation, these newly-defined objects are purely virtual constructs that can’t step out into the real world. If you try to bring a reference to one of these script objects into Delphi-land, you get a nil instead. This greatly reduces the opportunity for customization.
So I figured I may as well try my hand at my own version, drawing on my class-creation research. For the last few months, with a lot of hard work (and a fair bit of technical assistance from Barry Kelly on the arcane details of the RTTI system) I’ve been putting together a new Object Pascal-based script compiler. It’s not completely finished yet, but the features that will set it apart are up and running:
- Unit-based compilation. All your code doesn’t have to be in one unit. It also doesn’t expect that you’re building a single script to be executed directly. You can do that, but you can also run a valid compile cycle made up entirely of one or more units, with no program unit and no main routine, and use it as a library, calling individual routines from the script engine. Units can be compiled and saved individually, similar to Delphi’s DCU system, and linked with other units to form a script program.
- Dynamic, RTTI-based creation of new classes. You can define a new class in the script engine and the compiler will generate a new VMT and all the extended RTTI necessary to use this class from native Delphi code just like any other. For example, you can create a class that descends from a native type, override a virtual method, pass an instance out of the script engine to a variable reference, call the virtual method, and the script engine will be invoked to run it. (This particular trick require the TMethodImplementation class to set up, so Delphi XE or later will be required.)
The plan is to use the RTTI capabilities to achieve the most seamless integration possible between scripts and native code. In my next few posts I’ll be going over some details as to how I made this all work.
Coolest stuff, Mason…
Definitely very interested in this. Can’t wait to see the results.
Let’s see, let’s seeeee!! 😀
Wow, this sounds really good.
Great idea,
did you check PAXCompiler before reinventing the wheel!
could be very useful and handy.
cheers,
MB
Interesting! Though if you want to achieve the second point, why use a script engine in the first place and not a regular compiler? The main reason for using a script is sandboxing, isolation and being able to suspend, kill or cleanup any script at any point with no impact on the host application.
With a VMT-interceptor-like approach and two-way RTTI-based exposure, you just can’t achieve that, so why not just compile everything directly with FreePascal f.i.? All the mechanisms you’re after would be supported out of the box.
Also you’re a bit incorrect with DWS: you can compile a script and use it as library (calling its functions from Delphi without ever running the script as a program), and you can see script-objects Delphi-side as IScriptObj (low-level) or as IInfo (an RTTI-like helper), and then invoke any of their methods or properties (I guess you could build a VMT for those if you wished). You can create script objects directly, free them, etc. Actually some of the DWS expressions do that (f.i. script-side exceptions are created this way), and in a broader usage case, we have customers that use DWS classes in their C# or Java code (via SOAP in our case).
That’s one reason for using a script. The reason I use scripts is for extensibility: to add new, customized functionality into the program that wasn’t available at compile-time. That’s more important to me than isolation.
For example, my game engine declares basic classes like “character” and “monster”. Let’s say that someone wants to add a new feature to their game, where characters can steal items from monsters. In order to do that, they need some way to describe what monsters have that can be stolen. So they can declare a new class that descends from TMonster with new fields and properties on it. So far, so good. You can do this in DWS too.
Problem is, the (compiled) game engine only knows about TMonster, not TNewMonster. So when it goes to create monsters for an encounter, they won’t have the right properties available. How to resolve this?
One thing I could do is give TMonster a virtual constructor and have the game engine hold a class reference to a “class of TMonster” somewhere. The user could then specify the standard monster type for their game and the game engine would create it instead of the ordinary TMonster and never know that it’s not a normal, compiled Delphi class. If they override the constructor, the script engine will get invoked to run their customized setup code, and then when they call inherited it’ll reach out into native-land again. Because TNewMonster has a real VMT, native Delphi treats it as a real object.
Then, when the native code that powers the battle engine requires a TMonster instance, it will be able to operate on the TNewMonster instances that inherit from TMonster the same way any normal inheritance would work. No need to special-case things and fiddle around with interfaces. Just plug a new class in on top of the old one and things continue working exactly as they always have.
This is the sort of power you can’t achieve with any other Delphi script engine I’m aware of, including DWS. (Although you’ll be free to incorporate my RTTI-generation code into DWS once I release it.)
Well, in DWS the exception mechanism is already based on something very similar if you want sample code: Exception classes can be subclassed script-side, and then the subclasses can be created Delphi-side (in addition to script-side).
The only constraint is that you should invoke the script-side constructor, which can be done Delphi-side via TProgramInfo & IInfo.
Also what I meant about isolation, is that if you don’t need it, you could distribute the engine as dcu/ppu, and compile the final exe as part of your engine. You could f.i. use FreePascal for that, distributing it with your game engine, and compile the scripts into a main exe or as dlls. IIRC Morfik has a somewhat similar approach f.i.
In game engines, scripts like UnrealScripts or Lua are used for their isolation properties, so that game designers can pause, alter, debug and develop game scripts interactively. If you have no or limited isolation, bugs in the scripts can crash the whole game engine, making the interactive development process impractical. Also isolation means you can allow end-users to play with the scripts for mods f.i., without them having to face game engines crashes every time their script misbehaves.
Fair enough, but on the other hand there’s only so much isolation you can achieve without making your script engine useless. In order to interact with the game, the script needs to be able to call into the native code in the engine. For example, in UnrealScript there’s a very useful function called “FastTrace” that takes a starting point and an ending point and returns a boolean signifying whether a line between those two points would hit anything. That’s written in native code, not in UnrealScript, and trying to implement it purely in script would be a nightmare.
Any game system (or any other system that relies on script extensions) has to either implement the whole thing in script or provide a fairly substantial API for scripts to access the underlying engine. Mine certainly does! And at any point in that API, there could be a potential for bad input into an API function to cause trouble in the game engine. But without that access, you couldn’t do anything useful, so you add code to your API layer to validate potentially troublesome parameters.
In fact, if you look at some of the really useful, highly successful game scripting systems like UnrealScript or NWScript, there’s a lot of potential to screw up the underlying game state. I’ve seen poorly-done NWN mods completely trash things more than a few times, for example. But they also provide the ability to create some really amazing mods. IMO it basically comes back to “with great power comes great responsibility.”
Of course the script has to call native code, and that’s usually what a script spends most of its time doing ;), but there is a difference IMHO between being able to call just any code directly, and more “traditional” script exposure.
In UnrealScript the FastTrace calls can’t crash Unreal, and more generally, the calls from the script-side are checked for correctness and/or wrapped in exception frames, and what isn’t safe to call directly is further wrapped. You can trash the game state, but usually (unless you hit bugs), not the game engine, the script console can still allow to diagnose what went wrong. Without isolation, any incorrect call can result in an access violation and/or crash back to desktop.
YMMV, but when using a script, you pay in performance and overhead. If you are ready to accept crashes, you might as well not pay for the performance loss in the first place, and use a regular compiler to actually compile scripts to binary code (on the fly). This also gets you all the features and openness of a full-blown compiler (including caching of compiled source in PPUs, DLLs, etc.).
Hallvard Vassbotn was the best! I dont know why he stopped bloggin either. But he knows about the intrinsics of RTTI and hack around it like few. Really nice to read some content about RTTI.
Very interesting. Will certainly check it out when you release it.
I’m still alive and kicking, alas not much in the blogsphere. Too little time, too much life, is the short explanation ;). Still using Delphi 100% at work.
Nice post and technique. About my post, credit is due to Patrick van Logchem:
http://www.vanlogchem.nl/index.php/?e=59
Thanks @Hallvard, I was trying to be humble and held back on a reply, but I appreciate the gesture!
(Nowadays I’m working on an Xbox1 emulator in my spare time, also written 100% in Delphi – how about that!)
[…] Way back at the start of this year, I wrote about how I’d been working on a new Object Pascal-based script engine built around Delphi’s extended RTTI system. And then it got real quiet, because more immediate concerns took priority for me. But I’ve gotten to the point in my game engine development where I really need to put in the scripting system, so I spent the last couple weeks finishing up the implementation and tuning the performance a little, and now I’ve got an alpha version ready. If anyone would like to try it out, they can find it on Google Code. […]
I think your idea is near to PaxCompiler (http://www.paxcompiler.com)
Works good. You can extend you applications without limitations.