Using Makefiles Effectively

Other online guides only showed bits and pieces of a decent makefile but ones that were quick to the point were often missing a feature I’d have liked to seen. I figured I’d post and explain my one-and-done solution for efficient compilation.

Theory & Motivation

I’ve been working on a modestly sized project for a little while now, and up until now I’ve been using the most ineffective possible makefile. And it looked something like this:

CC=g++
CFLAGS=-O2 -lSomeLib -lOtherLib -I../include

arix-project: ../src/*.cpp 
	$(CC) ../src/*.cpp ../src/*/*.cpp $(CFLAGS) -o arix-project

This is terrible. Horrendous. Don’t do this. You’d be better off scratching out a shell script in your project root that does the same thing with simpler code.

The reason this is bad isn’t because it doesn’t compile anything, no. This compiles perfectly, believe it or not. The real problem lies in that this usage misses out on any of the benefits that a Makefile can provide. For myself and probably most programmers, the main benefit is to optimize compile time. Another helpful feature is to use multiple targets, even if the only other target you create is to clean your build folder.

But if creating a useful Makefile is so great, why didn’t I for so long? In short, it’s complicated. Various online guides only showed bits and pieces, and ones that were quick to the point were often missing a feature I’d have liked to seen. So, when creating this, I knew I wanted the following features:

No Listing of Every Object/Code File

This seems like it should be one of the most used features of a Makefile, but so rarely did I find a guide that did this well. This feature is especially crucial for new projects that are still being heavily developed. You’re likely already making big changes to your codebase so why should you also have to remember to update every change in your Makefile too? (The answer is that you shouldn’t)

Compile Each Object File Individually

This one wasn’t terribly difficult to figure out how to do. We just need a target for each object file so that we don’t recompile the entire project every time we make a change somewhere. But since I have no intention of manually making a target for every object file, we need to do this programmatically.

Correct Object Dependencies

This seems like it should go without saying. Every object should only be recompiled when something that it’s dependent on changes. Yet it seems like this is so often disregarded by such a simple mistake, even after following all of the other steps to set up good dependencies. I’ll explain below.

Project Directory Structure

Before I get started on the Makefile itself, I need to talk about directory structure. There’s no real standard for how C/C++ projects must be set up, and this makes harder to present something and say “This is how it should be done.” In any case, here’s a directory structure, I recommend that you structure it this way.

someproject
|
|-- build
|   |-- Makefile
|
|-- include
|   |-- class1.hpp
|   |-- class2.hpp
|
|-- src
|   |-- class1.cpp
|   |-- class2.cpp
|   |-- someproject.cpp

It’s okay to create a build folder within your directory structure like this. More importantly, it’s also okay to share that build folder with others, as long as you only include the Makefile in it.

It’s fairly clear what the include and src directories do, they contain your header and source files respectively. But why am I suggesting you structure your project exactly like this? Well, it makes it significantly easier to work with in a Makefile. It’s safe to assume that all files in src are C++ (or whatever language) source files, so we can treat every file in that directory like a single object. When we go to create object and dependency files later, the structure there will mirror the structure in our src folder.

I also recommend keeping your header files in a separate, parallel directory structure like this. It keeps both your source and header files uncluttered. By keeping them in a parallel structure, it’s also easier to remember where certain header files are, especially with larger projects.

The Makefile Itself

Now that we have the directory structure that I’m using cleared up, let’s get on to the Makefile itself. I’m going to put the entire file here and we’ll analyze it section by section.

# C++ Compiler
CC=g++

# Important directories, relative to this Makefile
SRCDIR=../src
INCDIR=../include
OBJDIR=obj
DEPDIR=dep

# Main Target object/name
TARGET=someproject

# Compile flags
CLIBS=-lSomeLib -lOtherLib
CFLAGS=$(CLIBS) -I$(INCDIR) -O2

SRCS=$(wildcard $(SRCDIR)/*.cpp) $(wildcard $(SRCDIR)/*/*.cpp)
OBJS=$(subst $(SRCDIR),$(OBJDIR),$(SRCS:.cpp=.o))
DEPS=$(subst $(SRCDIR),$(DEPDIR),$(SRCS:.cpp=.d))

# Main output executable
$(TARGET): $(OBJS)
	@echo Building final target $(TARGET)...
	@$(CC) -o $@ $^ $(CFLAGS)
	@echo Done.

# Dependencies
$(DEPS): $(subst $(DEPDIR),$(SRCDIR),$(@:.d=.cpp))
	@echo Generating dependency file '$@'...
	@mkdir -p $(@D)
	@cpp $(CFLAGS) $(subst $(DEPDIR),$(SRCDIR),$(@:.d=.cpp)) -MM -MT $(subst $(DEPDIR),$(OBJDIR),$(@:.d=.o)) > $@
	@echo '	@echo Building $$@...' >> $@
	@echo '	@mkdir -p $$(@D)' >> $@
	@echo '	@$$(CC) -c -o $$@ $$(subst $$(OBJDIR),$$(SRCDIR),$$(@:.o=.cpp)) $$(CFLAGS)' >> $@

-include $(DEPS)

.PHONY: clean
clean:
	@echo Removing $(OBJDIR), $(DEPDIR), and $(TARGET)...
	@rm -fr $(OBJDIR)
	@rm -fr $(DEPDIR)
	@rm -f $(TARGET)

Compile Variables

For the first section, we’re just declaring some common variables. I’m including these on their own, because if you want to copy this Makefile exactly and don’t really care how it works, these are the only things you’ll really need to change.

# C++ Compiler
CC=g++

# Important directories, relative to this Makefile
SRCDIR=../src
INCDIR=../include
OBJDIR=obj
DEPDIR=dep

# Main Target object/name
TARGET=someproject

# Compile flags
CLIBS=-lSomeLib -lOtherLib
CFLAGS=$(CLIBS) -I$(INCDIR) -O2

First, we set CC to our compiler. C++ should use g++, C should use gcc, etc.

Next, the directories of our project. ../src and ../include are exactly as above. These hold the source and header files respectively. If your directories are elsewhere, change these.

The next two, I haven’t talked about yet. They’re both a part of the build process, and they’ll be created automatically. I’ll explain both later.

The target is simply the name of our main target, our final executable. This is the file we want to run once we finish compiling.

Lastly, compiler flags. Add to CLIBS what library linker settings you need. This is also where pkg-config --cflags --libs somelib would go, if you use a library that needs that. Append to CFLAGS your extra compile options, like -Wall to enable all warnings, or -O2 as above to enable compiler optimization.

So far, this is pretty uninteresting. Every Makefile, even the bad ones, manages to use variables somehow. So how do we improve upon those?

Creating Lists for Objects and Dependencies

This next section addresses my first requirement. I’m too lazy to list every source file, and I know you are too. So, use wildcards, but carefully.

SRCS=$(wildcard $(SRCDIR)/*.cpp) $(wildcard $(SRCDIR)/*/*.cpp)
OBJS=$(subst $(SRCDIR),$(OBJDIR),$(SRCS:.cpp=.o))
DEPS=$(subst $(SRCDIR),$(DEPDIR),$(SRCS:.cpp=.d))

What does this do, exactly? Well, SRCS is going to hold every C++ file in our project with its path relative to the Makefile. How does it work? Make has wildcard matching functionality by using the wildcard directive, so it expands all the entries that match $(SRCDIR)/*.cpp into a list. If you have subdirectories, you can either do as I’ve done and include a second pattern to match all subdirectories, or check out some of the answers for Recursive wilcards in GNU Make? on StackOverflow. I couldn’t be bothered to come up with a less readable solution just to replace an easy to understand statement.

The lines above may seem complicated, but they’re not at all. They both use the substitution function to replace all occurences of the source directory, with the object/dependency directory. And, they replace .cpp with .o or .d. But what does this mean now?

OBJS now holds all of the object files we plan to compile, again, with their path relative to the Makefile. That means that we can create a rule for them all- but not yet! DEPS also holds files in a very similar fashion, but what does the .d extension do? These files will hold all of the dependencies of each .cpp file, and essentially give us an exact template for when an object file needs rebuilt. I’ll explain more below.

Target Executable

Next up, the first and main target. There isn’t anything terribly unique about it.

# Main output executable
$(TARGET): $(OBJS)
	@echo Building final target $(TARGET)...
	@$(CC) -o $@ $^ $(CFLAGS)
	@echo Done.

Just a quick note, whenever you see @ before a command in a Makefile, all that means is to not print out the literal command to the output before running it. Anyway, this target is dependent on our OBJS list from earlier, that is, it depends on every individual .o file to compile. Nothing more. All it runs is the compile command, where -o $@ says to output to the name of the target, or the text on the left side of the colon. $^ is just all of the prerequisites of the rule, or the OBJS on the right side of the colon. Then we just toss CFLAGS from earlier on the end, for our compiler options.

Dependency Targets

Alright, the main target is pretty straightforward too. So, next comes the weird bit. Let’s start with the issue first. When we compile an object file, we as programmers know that it has certain dependencies. It should be based on a single C/C++ file, and it might include a couple header files from our project. We know that we only want to recompile that object file when either the C/C++ source file changes, or one of the headers that it includes does. But how do we tell Make that?

Make has no idea what our object file depends on. We could blindly toss some prerequisites at it. It’s not hard to tell it that it’s dependent on a C/C++ source file with the same name. Here’s the part I alluded to above, where programmers just throw every header file in as a prerquisite for the object file. They set up everything well, get a rule to match all object files and match them with their source files, but ruin it by saying each object file depends on every header file in a project.

What that would mean is that whenever we, say, change a typo in a header file included only by one source, we must recompile the entire project from scratch. I’m just going to say it, that’s dumb. There’s no need for that, especially because there are tools available to avoid that entirely.

What I’m talking about is dependency files. These are the .d files I mentioned earlier. They hold the precise prequisite rules for each and every object file. No more, no less. That means we will only need to recompile object files when we absolutely need to.

But, I hear you ask, doesn’t that mean we’re just going to have to create a bunch more files for each object in our project? Well, yes, but we’re definitely not going to do that ourselves. Let’s look at this next target.

$(DEPS): $(subst $(DEPDIR),$(SRCDIR),$(@:.d=.cpp))
	@echo Generating dependency file '$@'...
	@mkdir -p $(@D)
	@cpp $(CFLAGS) $(subst $(DEPDIR),$(SRCDIR),$(@:.d=.cpp)) -MM -MT $(subst $(DEPDIR),$(OBJDIR),$(@:.d=.o)) > $@
	@echo '	@echo Building $$@...' >> $@
	@echo '	@mkdir -p $$(@D)' >> $@
	@echo '	@$$(CC) -c -o $$@ $$(subst $$(OBJDIR),$$(SRCDIR),$$(@:.o=.cpp)) $$(CFLAGS)' >> $@

This is a bit of a longer rule, so let’s take a look at exactly what it’s saying here.

The first line is simple. We’re saying that every dependency object’s prerequisite is its matching C/C++ source file. I.E. we only need to recreate the .d file when the .cpp file changes.

After a general output message, we make sure that the directory of the dependency file exists using mkdir. That way we won’t get any errors when we try to create it.

Next, the real magic happens. We use the standard C/C++ preprocessor to generate a Makefile dependency rule for us. We pass it the compiler arguments and the C/C++ source file first. -MM tells the preprocessor to parse the source file, determine the #include files, excluding any system header files, and generate a target rule for us. -MTjust signifies that we’d like to use a different target name for the rule, and that target is the object file. Lastly, we output all of that into the dependency file with > $@.

The next few lines are just extra rules that we output into the newly created dependency file. We output some text, make sure that a directory exists for the object file, and compile the object file with our usual compiler arguments. Just note that the $‘s are escaped by using $$. And, note that those are tabs before each line, not spaces. Make is very picky about tabs.

Include the Dependencies

So, we’ve created all of our dependency files, what’s left? We have to include them, of course.

-include $(DEPS)

There’s not much to say about this. Every dependency file is included in the main Makefile as if it were another target. Each object’s target from above now becomes a part of this one Makefile.

Obligatory Clean Target

Lastly, it wouldn’t be right to not include the obligatory clean target found in any Makefile guide you find online.

.PHONY: clean
clean:
	@echo Removing $(OBJDIR), $(DEPDIR), and $(TARGET)...
	@rm -fr $(OBJDIR)
	@rm -fr $(DEPDIR)
	@rm -f $(TARGET)

This declares that clean isn’t a file on disk, but a fake target. We output a message saying what we’re removing, then proceed to remove object files, dependency files, and the target executable. That’s it.

So that was a bit of a long one. But hey, now you have an ultra-versatile Makefile that’s super easy to reuse! If I did something wrong, send me a message and I’ll fix it. Thanks!