Best practices for a (GNU) makefile in the XXI century

This is not really a tutorial on makefiles; there are lots of those around the web.

But most of those are very outdated, and/or follow dubious practices. You'll end up with a makefile which was OK for a make and a compiler from the 90's, maybe even around year 2000. Even the GNU make manual recommends unnecessarily complicated things, given the capabilities gcc has grown in the last decade.

So, if you know how makefiles generally work (it's enough to know what a rule, a recipe and a variable is), and want to improve yours or know of better options, the following may help.

Rationale

I have read (heard?) a couple of times the advice that one shouldn't write his own new makefile, but rather just get one already existing and somewhat close to what you want to do, and adapt it to its new role.

And that is what I have done up to now whenever I needed a small makefile. But maybe that's also the reason why even the makefile tips currently appearing on Stack Overflow are mostly rehashes of suggestions that were already outdated 10 or 15 years ago; things that have been obsoleted since GCC 3, and yet people are still using them to build new makefiles, for small projects, on current systems, for current GNU make, for current toolchains.

Holy cow. Reminded me of this…

So finally I have needed to make my own makefile for my own miniproject, and it's been rather hard to come up with something that could be called "best practices", or a style guide, or at least something that could be called a de facto standard. The make manual certainly doesn't help: lots of ways of doing the same things, each of them full of exceptions and subcases (and a strong deja vu towards Choose Your Own Adventure. At every paragraph they tell you to look something else – how are you supposed to read like that?). So, what is idiomatic and easily recognizable for "everyone"? What is easier? What is harder now but will pay back in the longish run? Customs? Etiquette? Pitfalls?

For example: I will be using GNU make, which is the standard even on Mac OS X, which I would have expected to use a BSD make. The thing is, there is a ton of built-in rules in GNU make. At first I guessed that maybe I should be trying to let the built-in rules do most of the work: since they are built-in, that must mean that they were the most commonly used, and furthermore the fact of getting built-in will only strengthen their "used by everyone" status. Right?

No. After some trying and looking around, it rather looks like the implicit rules are mostly for the cool factor of typing make myfile without having a makefile and still having a myfile executable pop out seemingly directly from myfile.c (a chain of (built-in) implicit rules will make the .o file, link the final binary and then delete the .o; or, depending on the case, a variation of one of the implicit rules will recognize that it can compile and link all at once, so really avoiding the creation of the .o file).

Now, exactly what rules triggered and why, and what exactly did they do? HA! You might be able to deduce what happened by parsing the printed commands, but the moment that a chain of built-in implicit rules kicks in, good luck.

Worse: built-in rules are make-version (and even make-compilation) dependant, and are kind of a dead-end the moment they stop working. Imagine that a rule was doing its job and now stops. Of course, no error gets printed – because probably make doesn't even consider it an error. So, what is happening? "I thought that my case still should be supported by the rule, but it just is not working. Why? Maybe I broke some assumption? Well, why was it working before? Oh, maybe before I was triggering a different rule than I thought? How can I fix it? Can I still use the old rule? Should I try to use a new one, or am I already past the domain of implicit rules and am supposed to use a makefile? A makefile doing what exactly?"

At the end of the day built-in rules are invisible, unsophisticated boilerplate. And since sooner than later you will have to reimplement them to your specifications, why not do it from the beginning? Maybe the built-in rules make sense for other kinds of projects; I just fail to see how they really help for any multi-file C project.

The requirements for my makefile

So, to begin with the list of goals for my makefile, I will …

1. ignore built-in rules

Their only good thing is that they print out their generated command line, so it's easy to detect that a built-in implicit rule triggered. And it will be easy to detect them because …

2. my makefile is silent by default

… unless some problem appears. When you use cp and rm and gcc itself, they do their jobs silently unless they find a problem or you ask for extra info; why should make be different? Even more so, given that we are interested in reacting quickly to any warning or error coming from the compilation process; if we train ourselves to ignore a mountain of haystack at every build, we will miss the needle whenever it appears.
So, a silent make run means a good run.

However, we do want to have the option to print out all the command lines, for debugging and because Eclipse CDT needs to parse them to make sense of the compilation process (for indexing mostly). Se we'll have ...

3. option for verbose compilation

Also, I'm conflicted about printing "CC file" and "LINK file" lines, as does the Linux kernel compilation process. I guess it's useless for small projects, but at some moment might be interesting; so I left in some minimalistic echo lines.

About the file organization: lots of makefiles out there seem to be written in terms of the object files, for example giving the prerequisites for the final binary as a list of object files, and letting some rule deduce what are the corresponding source files. To me that is counterintuitive: I have .c and .h files, I want an executable file, why should I think about the intermediate products? Luckily it's also easy to find makefiles doing it that way, so probably it's a matter of choice. So I will ...

4. define the product in terms of .c source files, not object files

Note: only .c files, not the .h files that might influence them. 

The thing is, getting right the way sources combine to create object files is the most important part of the makefile creation process. And it usually isn't trivial, and it can change in unexpected ways whenever included or includer files change, and breakage can go unnoticed while creating Undefined Behaviour (for example, when a function declaration in a modified .h file gets out of sync with what was compiled into an erroneously non-rebuilt object file).

And yet some makefiles try to keep track manually of the dependencies. That's crazy, because there have been increasingly simpler ways to automate the dependency tracking since well before the year 2000. First some shell scripting was needed, and auto-generated rules were actually added at the end of the main makefile – that seems to be the reason why lots of oldish makefiles do have a #DO NOT DELETE THIS LINE at the end to signal the insertion point. Later a combination of gcc flags, small sed scripts, and make being able to include files, simplified the proces; but still makefiles had a "depend" target which was run as an extra build pass. And finally since gcc 3 everything can be done with just 2 gcc flags, an include line in the makefile, and no extra passes! Clang and icc do implement those same flags: -MMD and -MP. So, I want to ...

5. use gcc's modern, simultaneous-to-compilation auto-dependency tracking

Again, remembering that "modern" here means "since gcc 3". It's good to keep that kind of thing in mind, because there seem to be two aspects to the whole makefile thing: one is that the make manual, and some big projects, might care for make versions that couldn't do "include", and for C compilers from the 80's, from before ISO C and before Linux. Those are beasts which I have only heard of, and those people might well have a point. 

But the other aspect is newish projects that maybe started life as an Eclipse project with automatic makefile generation and now need to get their own real makefile. And those probably don't need to follow the olden ways.

Lastly, continuing with file organization: 

6. sources stay undisturbed in their own directory, while all the building happens somewhere else

This can actually be somewhat harder/messier, and it can depend on the project. For a small project like mine it's easy to list the .c files (or even autodetect them with a find command line!), and use make functions to change file extensions and paths, and so generate object file paths. 

But what happens with a big project with lots of subdirectories? Do you want to mirror the source tree into an object tree? Or is it enough to dump all the object files into a single directory? (the only problem is possible filename collisions, since the linker will flatten the file structure anyway). There are different ways to do it, and I'm leaving this as an exercise for the advanced reader. Solar's makefile tutorial shows a directory-oriented way, which is probably the easiest solution – but puts objects together with sources; should not be difficult to change it to use separate trees. The classical, full-fledged, Autoconf-grade, smells-like-overkill solution is given in http://make.mad-scientist.net/papers/multi-architecture-builds/#vpath. 

Some secondary details:

7. using := is for immediate assignment, instead of the standard =, which in fact is defining a macro – which gets re-evaluated everytime it is used, which can cause surprises once the makefile grows. := is a GNU-ism, but so are lots of other things; the POSIX-standard make seems to be awful anyway, but improving little by little by incorporating what GNU make does. BSD make also got the :=.
8. using $(CC) and $(RM) and such makes the makefile more portable / easily configurable.
9. creating the directory for object files is a bit of a logistic problem which turns out to be easy to solve with order-only prerequisites, and is in fact an example in the GNU make manual.
10. the "-include" line for dependency files should come after any rule has been defined: keep in mind that the default target for the makefile is the first rule which appears, and the "include" will be injecting a lot of those.
11. speaking of which, there are explicit ways to define the default target, but I preferred using an "all : mytarget" rule, since "all" is a very typical target in any makefile you might find around; it's customary to try a "make all" with any unknown makefile. For a more involved makefile I might use as a default a "help" target which would only show the available targets and their purposes.
12. listing filenames (without a path) in a _SRC variable and then building the full paths in a SRC variable seems to be common; same thing if you wanted to use object files in vars _OBJ and OBJ.
13. THE RULES MUST BE INDENTED WITH TABS, which are not supported in this blog. Here they have been rendered into 8-spaces. Change them in your code editor.

Some references linking to even more references:

• POSIX-make : underwhelming, but improving by getting GNU-isms and standardizing lots of what I have described: http://www.dwheeler.com/essays/make.html
• The make manual mentions automatic dependencies, but strangely does it in a half-cooked way. For example, it mentions sed scripts instead of using GCC's -MP, and makes everything more complicated than needed. But for a clearer example of being outdated, it mentions including, but doesn't use the "-" prefix to avoid the inevitable first-run errors – even though it does mention it in the include page. It's even more strange given that one base resource on autodependencies is a blog post by make's maintainer – that post is itself outdated, but still not as outdated as the manual itself.
• Modern auto-dependency tracking: http://bruno.defraine.net/techtips/makefile-auto-dependencies-with-gcc
• Solar's makefile tutorial 

EDIT (August 2021):

6 years after writing this I find myself having to remember how to write a simplistic makefile. And it's interesting how pointless it feels.

Back then I also made a question in Stack Overflow asking for review/improvements. The answers were rather uninteresting, so I never took the time to implement them. But now that I am looking again into this I realize that the Makefile has a couple of silly problems: SRCS is never used, so it should be deleted; createdir should be phony. 

But none of the reviewers caught that. And instead, one of them recommended an useless change based on their misunderstanding of Make rules.  

These reviewers self-selected to give advice. What does this say about us software people and our tools?

There's something wrong in keeping using something with this many dark corners and sharp edges. Years ago I printed out the Autotools manual (shudder!) to try to learn how to use it; now that kind of thing is so below the bar that I can't care. Bash went under the bar some time ago too; I see now that Make also followed it. Interesting.

 

I'll hope it's a sign of my, or my tastes', evolution, from realizing that there's more promising ways to spend mental cycles than learning the arcana needed to navigate stacks of hacks like those tools.

The makefile

The (numbers in parenthesis) refer to the numbers in my list of requirements.

WARNINGS := -Wall -Wextra -pedantic -Wshadow -Wpointer-arith -Wcast-align \
            -Wwrite-strings -Wmissing-prototypes -Wmissing-declarations \
            -Wredundant-decls -Wnested-externs -Winline -Wno-long-long \
            -Wuninitialized -Wconversion -Wstrict-prototypes

CFLAGS ?= -std=gnu99 -g $(WARNINGS)

OBJDIR := obj
SRCDIR := src

# (2 & 3) : silent by default
ifeq ($(VERBOSE),1)
    SILENCER := 
else
    SILENCER := @
endif

#pass this environment variable to the C source
ifeq ($(DEBUG_BUILD),1)
    CFLAGS +=-DDEBUG_BUILD
endif

# (12): typical way to list files and build full paths
# (4): list the sources, not the object files (nor includes)
_SRCS := uthreads.c main.c 
SRCS := $(patsubst %,$(SRCDIR)/%,$(_SRCS))
OBJS := $(patsubst %,$(OBJDIR)/%,$(_SRCS:c=o))

# (5): generate phony deps during compilation
CFLAGS += -MMD -MP 
DEPS := $(patsubst %,$(OBJDIR)/%,$(_SRCS:c=d))
# (10): can't include the deps before the first, default target has appeared!

# (11): "all" is the classical default target
all: main

# (13): indent with TABs!!
createdir:
        $(SILENCER)mkdir -p $(OBJDIR)

main: $(OBJS) 
        @echo " LINK $^"
        $(SILENCER)$(CC) $(CFLAGS) -o $@ $^ 

# (6): put the object (and dependency!) files away from the sources
# (9): create the dir before building into it
$(OBJDIR)/%.o: $(SRCDIR)/%.c | createdir
        @echo " CC $<"
        $(SILENCER)$(CC) $(CFLAGS) -c -o $@ $< 

clean:
        $(SILENCER)$(RM) -f *~ core main
        $(SILENCER)$(RM) -r $(OBJDIR)

.PHONY: clean all

# (10): dependencies won't turn into a default target here
-include $(DEPS)