Dear Idera, please Don’t Clutter Up our build process anymore
I was compiling a new branch at work today, which used a different version of Delphi than the one I had been working on, and along the way I ran into tons of spurious File Not Found errors from the compiler. The file was, in fact, right there where it should have been able to find it easily, but it kept erroring out. After speaking with another developer, I managed to track down what was really going wrong: for some reason, it was finding outdated DCUs from the old build and choking on it, giving an incorrect error. After deleting the rogue DCUs, everything built smoothly.
It made me think about the many DCU headaches I’ve had over the years. I haven’t heard an official explanation from the Delphi team for why Delphi uses DCUs, but according to Verity Stob,
As I understand it, this speed advantage [of Delphi’s compiler over Java’s] is in part achieved by the .DCU object file format, which is actually a raw-ish dump of the Delphi compiler’s output buffer. … it’s really easy to write out and load quickly [because] it’s already in the right format.
In other words, DCU essentially functions as a cache to help the compiler go faster. Unfortunately, as anyone with a few years of Delphi experience can tell you, this makes the compiler subject to one of the Two Hard Problems of Computer Science. And it kind of makes me wonder if DCUs are worth having around anymore.
The upside is that it provides caching for the compiler. This was very important in the late 80s and early 90s, due to highly limited hardware. Today, though, when “2+ GHz quad-core CPU with 4 GB of RAM and an SSD drive” isn’t even the description of a high-end system anymore, (particularly not for a software developer,) that’s a lot less of a benefit than it used to be.
There are several downsides, though:
- Caching means cache invalidation problems, as listed above.
- It makes the process of searching for units more complicated for the compiler, and harder for developers to reason about.
- Caching means cache invalidation problems, as listed above. (It’s a big enough problem to be worth mentioning twice, especially if you’re maintaining more than one project, or more than one version of the same project!)
- It gives rise to the temptation to do ridiculous things like checking DCUs into source control. (No, seriously. I know of at least one software development company using Delphi that actually does this!)
- It gives rise to the temptation, among some of the more parasitically-minded component and library vendors, to sell DCU-only libraries which then keeps the legitimate purchaser from debugging them or updating them to work with the next version of Delphi. This is something that simply should not happen, but some people are doing it just because they can.
It all makes me wonder, is it time to do away with the DCU file once and for all? The compiler’s fast enough to not need it, particularly if you’re on an SSD. (And if you’re not… seriously, go get an SSD. Now. I promise you, it’ll be the best thing you ever did for your dev environment.)
What do you think? Is DCU an outdated concept that’s doing more harm than good these days, or is it still useful for modern Delphi development? Let me know what you think in the comments.
I guess that without DCUs, my 2.7 million line project at work couldn’t be compiled any more – it would require a (full) build every time, right?
That is a lot of ‘down’ time while waiting for it to compile^H^H^H^H^H^H^H^H build.
Don’t get me wrong, you have pointed out some valid points in getting rid of them (DCU-only libraries) but for the bigger projects, it will have a negative effect.
Sounds to me like you’ve just configured things wrongly. When you create a new copy of your project on a branch, there should only be source. Ask yourself why the compiler can see any DCU files. It’s surely because you pointed the compiler at them.
I must agree with cache invalidation is hard, checking a DCU if you have source is bad (and you *should* have the source) and that you should get an SSD.
But for me, speed still an issue so I can’t agree to remove this speed if you do not improve the compiler speed. Rats, that is the first thing Verity ask in the linked article. If speed is not a problem to you, then just use “Build All” every time as Nick Ring said…
I agree with David but, anyway, I must admit that DCU files are one of the most confusing configurations on Delphi. People just keep getting wrong. No matter how much I read about them, I’m never sure if it is right…
The problem with DCU-only sales vendors is that components would increase price and sell less if they are obligatory to include source. And you know that Delphi is already a high priced tool for today standards of IDE and compiler. To be honest, I think that a vendor should now get in this discussion to tell their point of view…
tl’dr -> I agree, but I disagree… 🙂
DCUs are good. They avoid code that doesn’t need to be recompiled each time isn’t. They are alike precompiled headers in C/C++ – and whoever has ever compiled a large project in C/C++, knows how good they are (try to compile the whole Boost library, for example, if you don’t believe me…)
The problem relies in the fact that many Delphi developers – including many library vendors – have very outdated projects setups and directory layouts thereby when compiling the chance of hitting the wrong DCU are high. One common error is to have .dcu along .pas files, especially when using different versions of Delphi. They should go into their own folders, each for every version of Delphi (and a separate folder for debug version as well). Even when compiling applications, dcus should go into their separate folders for platform and build configurations – so changing them won’t lead to use the wrong dcu. Moreover, ensure paths are never absolute, but are relative or use environment variables properly set. A proper setup makes switching from a version to another a breeze.
It just requires some proper setup upfront – and Delphi supports it fully, it’s not longer Delphi 1 times – and it saves a lot of headaches afterwards.
DCU’s are great. If Idera/Embarcadero decides to remove them from Delphi I will surely be switching to some other development environment.
When I stared programming almost 15 years ago it was in C++ using Cpp development environment. On my old computer it took about 30 seconds just to compile some small “Hello world!” typed applications. Much more when compiling something larger.
The worst thing was that I usually spent waiting all that time just for compiler to tell me that I forgot some semicolon or made some typo in the code. And that was a real killjoy.
Then luckily I stumbled upon Delphi programming related forum when I learned that Borland is offering Free license for Delphi 6 Personal Edition so I gave it a try. And I must admit that I was surprised of how quickly did Delphi manage to compile any project. Of course later on I learned that DCU’s are its secret for this.
Now I do agree that nowadays computers are much faster and therefore compilation times become much much shorter but still recompiling any big application could take quite some time.
Take a look at some other programming languages and you will see that many of them have quite long compile times.
For instance developers of a PC game Factorio had to split its code into multiple modules so that they are no longer forced to recompile the whole game for every small change they made. And once they do final build of the game it takes them over an hour despite the fact that they use precompiled headers for many thing.
Did you ever have to wait for over an hour to compile any Delphi project even when doing full rebuild? I seriously doubt it. And the main reason for this are all those DCU’s that ship with Delphi and contain precompiled source code for every component that ships with Delphi itself. And even when you use “Build All” you are only rebuilding your own code and not the precompiled code that ships with Delphi.
As for the component vendors that only sell precompiled DCU’s.
What would you rather buy?
Component that comes in precompiled DCU that can then be fully linked into your application giving you ability to have single executable file?
Or perhaps component that comes inside dynamic link library that you need to ship along with your application as it is quite common for many other programming languages?
The truth is that most component vendors fear that if they do ship source code to their buyers some of those buyers would go and reuse parts of that code to make their own components and sell them as if they have made them all by themselves.
First, you need to implement a cleaner development process, where you dont have random folders full of DCUs lying around.
Second, you need to have a clean batch file or some other way of cleaning your working copies.
Thirdly, consider having your important builds occur under a Jenkins instance on a clean room build machine.
W
Mason, I’ll go you one better. You’re completely correct about Delphi still using concepts that were useful in the 1980’s but don’t provide tangible benefit any more. It’s time to dispense with the outdated single-pass compiler design entirely. Delphi code could become much more optimized, we could dispense with forward declarations and other productivity-sapping stuff, etc.
What about compiler speed? That’s where we take a page from Swift. Swift is a statically-typed, compiled language. However, it also provides the “playground” (also called a REPL Run-Evaluate-Print-Loop or just an old fashioned command line) for interactive use to play with the language, explore data, etc. in real time. This is EXACTLY what we need. Imagine if Delphi had an interpreter or JITted system that could run Delphi code immediately! This could be used to test and debug code, and a formal compile would only need to be done on the finished product. That way you’d have *zero* build time and we wouldn’t care if the actual compiler wanted to make 72 passes through the code for optimization purposes.
The best of both worlds.