The trials and tribulations of advanced make
Post date: Aug 08, 2011 12:28:11 PM
This post comes about after spending a lot of time and not a few curses on trying to get a nice generic build system set up with make and avoid the deferred pain of recursive make files. Why am I using make? For the same reason I use vi, but that's another post.
I know, I know, ask pretty much anyone in the industry these days and they'll either tell you that using make directly for complex projects is old school and you should be on one of the various toolchains that wraps it such as autoconf/configure, or they'll tell you that they've never written a build file of any description because, well, the IDE takes care of it. The first stance has some merit I admit, proponents of the second will seriously contemplate suicide the first time they're asked to work on a code base that's not tied to an IDE.
Personally I think that experience of varied build systems should be a critical item in the skill set when hiring if you're a software shop with any degree of environmental heterogeneity. Even if you're not you should consider it as even IDE's shift over time and I've never come across an established environment where there are no legacy systems of some kind.
So, taking as a given that, for some reason, you are wanting to produce a complex build environment in make what would you like to support you?
Good documentation? I've found the online edition of O'Reilly's Managing Projects with GNU make utterly invaluable.
Template build system? The problem with these is that they generally impose a very particular organisational structure on your project which may not be viable and the tiniest tweaking leads to mayhem as you try to figure out how the various includes, defines, implicit rules and shell scripts interact. The O'Reilly's book contains various templates in the examples.
Debugging capabilities? Well, there is that -d flag, which becomes more useful if you use --debug and become more selective which what debug data you want. However I found nothing that would let me debug variable expansion, deferral and evaluation except for the ages old court of last resort, adding print statements. As with printf, $(warning myvar: $(myvar)) has been an amazingly useful construct and, as with printf, it's use demonstrates a lack of support tooling.
While it's not enough, what is there?
dry run (-n): useful, but only work for the eventual commands invoked and not how you get there
inspect make data base (-p): useful for control flow, but doesn't insert values into variables. If only this was closer to the set -x behaviour for bash
pretend a files new (-W): useful with -n for targeted testing
The tool I've wanted ever since attempting to port some very complex software with very, very complex build systems is a full stack debugger. Many of you at this point are thinking either what does he mean, or thinking that there are such things; if you're still think the latter is correct when I've elaborated then tell me, please, and put me out of my misery!
A full stack debugger - a tool capable of traversing control and data flow through the intricate stacks of varied shells, interpreters and native programs that get involved in a build system. Something that can deal with a call hierarchy of bash->perl->bash->make->sh->make->perl->[my custom binary 4GL language generator]->bash->make->awk->et al.
I want to be able to SEE where a given value came from, if it came from the shell then where was that shell created and where was the environment for that shell constructed. I want to be able to single step through these scripts and programs and look at the values of data at any given point.
Maybe you start to see why I grudgingly grant some merit to those who play in the utopian sandbox of an IDE instead of venturing into the post-apocalyptic wasteland called legacy.
This particular problem of full stack debugging actually has bearing on future as well as legacy development headaches. Think about a cloud environment, not any given cloud, the concept of a cloud; a heterogeneous collection of hardware with varying capabilities and potentially transient execution environments as we provision and dispose of our hosts on demand.
Now think about how you are going to debug exactly what is was that went wrong, somewhere down the line on an unknown machine with an OS that's since been erased, that caused your application to think that little Tommy really did have the privileges to access his fathers keyring and purchase that Russian bride suggest by, some rather confused, targeted advertising.
The upshot of this rant is that I've added another project to my wish list (not my todo list). Not the full stack debugger, for while I'd love to have it the time required is prohibitive, but an interactive Make debugger. Something that allows you to step, line-by-line, through your build, inspecting your variables and seeing precisely why libsecurity.so hasn't been rebuilt for four years despite all those patches you've applied to the source files.