AMBuild

AMBuild is a tool for building software projects and creating release packages. It is targeted at C++ projects, though it can be used for anything. It has been tailored to solve three major problems that most build tools do not address:

• Accuracy. You should *never* need to clean a build. Clean rebuilds are unnecessary and a waste of your time. AMBuild always computes minimal rebuilds accurately - any failure to do so is considered a bug.
• Speed. Most build systems need to traverse the entire dependency graph for changes. AMBuild only needs to look at the set of changed files on the filesystem.
• Flexibility. Build scripts are written in Python, so they're very easy to write and provide full programmatic control.

Keep in mind, AMBuild is neither widely used nor has it been used on a wide variety of systems. AlliedModders has used it successfully for small to medium-sized C++ projects on Linux, Mac, and Windows. Our largest project, SourceMod, has 700 C++ files and over 200,000 lines of code. We're happy to receive feedback (see the bottom of this page) if you use it in your own projects.

Motivation

AlliedModders C++ projects require a lot of customization. The set of flags passed to the compiler, their order, how things get linked, is all delicate and complicated. In addition, each Source game requires different linkage, so our C++ files usually get compiled multiple times, over and over again, into separate binaries. Lastly, our projects are large, and minimizing build time through correct dependency computation and parallelization is important. Very few build systems can handle any of these scenarios well, much less all of them, so we sought to make a new build system.

The initial version of AMBuild only solved flexibility problems. By controlling the build pipeline through Python, we were able to generate 15+ different binaries from the same source files without any code duplication. Over time the AMBuild script syntax has become very simple; you no longer need a complex project to justify AMBuild.

The modern version of AMBuild (also known as AMBuild 2), is modeled after Tup. Tup is a huge advance forward in build systems, and it is likely that one day AMBuild will simply generate Tup scripts. If you are looking at AMBuild as a build platform for your projects, you may want to see whether Tup meets your needs instead.

Installation

AMBuild can be downloaded via Mercurial [1] or via the releases page. To install it,

$ cd ambuild
$ sudo python setup.py install

If you're on Windows, omit the "sudo".

Tutorial

See the AMBuild Tutorial for more information.

API

See the AMBuild API article for more information.

Technical Overview

AMBuild is separated into a frontend and a backend. The frontend is responsible for parsing build scripts (this is known as the configure step). The backend is responsible for actually performing builds. The frontend and backend are separate, and it is possible to use a different backend other than AMBuild. For example, the configure step can produce Visual Studio project files.

Configuring

Build scripts are written in Python, and they are parsed whenever you configure the build. If a build script changes, it will automatically re-trigger the configure process on the next build. When a configure takes place, any files left by an old build are removed if they are modified or removed in the new dependency graph.

The details of the generated dependency graph are stored in an SQLite database, which is located in a hidden .ambuild2 folder inside the build path.

Building

The build process involves a few important steps that are executed every time you use the ambuild command:

1. Damage Computation. Builds a list of all files that have changed since the last build. This is based on filesystem timestamps, and either a forward or backward time change is enough to be considered "damaged" (or dirty). For example:

   $ ambuild --show-changed
   /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp
   

2. Partial DAG Construction. A partial dependency graph is built based on the list of damaged files. This is the entire set of nodes in the graph which need to be recomputed. The output of this step can be seen with --show-damage. For example:

   $ ambuild --show-damage
    - package/addons/metamod/bin/server_i486.so
     - cp "../loader/server_i486/server_i486.so" "package/addons/metamod/bin/server_i486.so"
      - loader/server_i486/server_i486.so
       - c++ loader.o gamedll.o serverplugin.o utility.o -m32 -static-libgcc -shared -o server_i486.so
        - loader/server_i486/loader.o
         - [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
          - /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp
   

3. Task Construction. The partial DAG is simplified into a tree of commands to run. The output of this step can be seen with --show-commands. For example:

   $ ambuild --show-commands
    - cp "../loader/server_i486/server_i486.so" "package/addons/metamod/bin/server_i486.so"
     - c++ loader.o gamedll.o serverplugin.o utility.o -shared -o server_i486.so
      - [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
   

4. Updating. Each task in the task tree is executed and the results are processed to either reject the build, or update the state of the dependency graph. At this phase, tasks are executed in parallel based on the number of CPU cores available. The task graph is also shared to maximize throughput. You can see the sequential build steps with --show-steps:

   $ ambuild --show-steps
   task 0: [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
     -> loader/server/loader.o
   task 1: c++ loader.o gamedll.o serverplugin.o utility.o -m32 -static-libgcc -shared -o server.so
     -> loader/server/server.so
   task 2: cp "../loader/server/server.so" "package/addons/metamod/bin/server.so"
     -> package/addons/metamod/bin/server.so