Difference between revisions of "AMBuild"

From AlliedModders Wiki
Jump to: navigation, search
m
Line 1: Line 1:
AMBuild is a tool for building software projects and creating packages for release. It is targeted at C++ projects, though it can be used for anything. It has been tailored to solve three major problems that most build tools do not address:
+
AMBuild is a tool for building software projects and creating release packages. It is targeted at C++ projects, though it can be used for anything. It has been tailored to solve three major problems that most build tools do not address:
*'''Accuracy'''. AMBuild guarantees that you never need to "clean" a build. Clean rebuilds are unnecessary and a waste of developer time. Builds are always computed accurately, and any failure to do so is considered a bug in AMBuild.
+
*'''Accuracy'''. You should *never* need to clean a build. Clean rebuilds are unnecessary and a waste of your time. AMBuild always computes minimal rebuilds accurately, and any failure to do so is considered a bug.
*'''Speed'''. Most build systems discover changes by recursively searching downward through all build steps and their dependencies. This is very slow. AMBuild instead computes the set of ''files'' that have changed, and propagates those changes ''upward'', to each build step that cares. An empty-build takes no time, whereas a large project with other build systems can have 30-second empty builds.
+
*'''Speed'''. Most build systems need to traverse the entire dependency graph for changes. AMBuild only needs to look at the set of changed files on the filesystem.
*'''Flexibility'''. Distributing C++ software is hard. AMBuild provides full tuning of the C++ compiler and offers abstraction and introspection of system properties, to avoid boilerplate often seen with Make or Autoconf. Build scripts are written in Python, offering a great deal of programmatic control.
+
*'''Flexibility'''. Build scripts are written in Python, so they're very easy to write and provide full programmatic control.
  
 
Keep in mind, AMBuild is neither widely used nor has it been used on a wide variety of systems. AlliedModders has used it successfully for small to medium-sized C++ projects on Linux, Mac, and Windows. Our largest project, SourceMod, has 700 C++ files and over 200,000 lines of code. We're happy to receive feedback (see the bottom of this page) if you use it in your own projects.
 
Keep in mind, AMBuild is neither widely used nor has it been used on a wide variety of systems. AlliedModders has used it successfully for small to medium-sized C++ projects on Linux, Mac, and Windows. Our largest project, SourceMod, has 700 C++ files and over 200,000 lines of code. We're happy to receive feedback (see the bottom of this page) if you use it in your own projects.
 +
 +
=Motivation=
 +
 +
AlliedModders C++ projects require a lot of customization in the flags passed to C++ compilers. We even have to be careful about the order in which those flags appear. In addition, we require different linkage against every Source game, so many C++ files get compiled multiple times into new binaries. Lastly, our projects are large, and minimizing build time through correct dependency computation and parallelization is important. Very few build systems can handle these scenarios well, so we sought to make a new build system.
 +
 +
The initial version of AMBuild only solved flexibility problems. By controlling the build pipeline through Python, we were able to generate 15+ different binaries from the same source files without any code duplication.
 +
 +
The modern version of AMBuild (also known as AMBuild 2), is modeled after [http://gittup.org/tup/ Tup]. Tup is a huge advance forward in build systems, and it is likely that one day AMBuild will simply generate Tup scripts. If you are looking at AMBuild as a build platform for your projects, you may want to see whether Tup meets your needs.
  
 
=Technical Overview=
 
=Technical Overview=

Revision as of 04:00, 15 October 2013

AMBuild is a tool for building software projects and creating release packages. It is targeted at C++ projects, though it can be used for anything. It has been tailored to solve three major problems that most build tools do not address:

  • Accuracy. You should *never* need to clean a build. Clean rebuilds are unnecessary and a waste of your time. AMBuild always computes minimal rebuilds accurately, and any failure to do so is considered a bug.
  • Speed. Most build systems need to traverse the entire dependency graph for changes. AMBuild only needs to look at the set of changed files on the filesystem.
  • Flexibility. Build scripts are written in Python, so they're very easy to write and provide full programmatic control.

Keep in mind, AMBuild is neither widely used nor has it been used on a wide variety of systems. AlliedModders has used it successfully for small to medium-sized C++ projects on Linux, Mac, and Windows. Our largest project, SourceMod, has 700 C++ files and over 200,000 lines of code. We're happy to receive feedback (see the bottom of this page) if you use it in your own projects.

Motivation

AlliedModders C++ projects require a lot of customization in the flags passed to C++ compilers. We even have to be careful about the order in which those flags appear. In addition, we require different linkage against every Source game, so many C++ files get compiled multiple times into new binaries. Lastly, our projects are large, and minimizing build time through correct dependency computation and parallelization is important. Very few build systems can handle these scenarios well, so we sought to make a new build system.

The initial version of AMBuild only solved flexibility problems. By controlling the build pipeline through Python, we were able to generate 15+ different binaries from the same source files without any code duplication.

The modern version of AMBuild (also known as AMBuild 2), is modeled after Tup. Tup is a huge advance forward in build systems, and it is likely that one day AMBuild will simply generate Tup scripts. If you are looking at AMBuild as a build platform for your projects, you may want to see whether Tup meets your needs.

Technical Overview

AMBuild is separated into a frontend and a backend. The frontend is responsible for parsing build scripts (this is known as the configure step). The backend is responsible for actually performing builds. The frontend and backend are separate, and it is possible to use a different backend other than AMBuild. For example, the configure step can produce Visual Studio project files.

Configuring

Build scripts are written in Python, and they are parsed whenever you configure the build. If a build script changes, it will automatically re-trigger the configure process on the next build. When a configure takes place, any files left by an old build are removed if they are modified or removed in the new dependency graph.

The details of the generated dependency graph are stored in an SQLite database, which is located in a hidden .ambuild2 folder inside the build path.

Building

The configure process also generates a build.py which should be run to perform a build. The build process involves a few important steps:

  1. Damage Computation. Builds a list of all files that have changed since the last build. This is based on filesystem timestamps, and either a forward or backward time change is enough to be considered "damaged" (or dirty). For example:
    $ python build.py --show-changed
    /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp
    
  2. Partial DAG Construction. A partial dependency graph is built based on the list of damaged files. This is the entire set of nodes in the graph which need to be recomputed. The output of this step can be seen with --show-damage. For example:
    $ python build.py --show-damage
     - package/addons/metamod/bin/server_i486.so
      - cp "../loader/server_i486/server_i486.so" "package/addons/metamod/bin/server_i486.so"
       - loader/server_i486/server_i486.so
        - c++ loader.o gamedll.o serverplugin.o utility.o -m32 -static-libgcc -shared -o server_i486.so
         - loader/server_i486/loader.o
          - [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
           - /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp
    
  3. Task Construction. The partial DAG is simplified into a tree of commands to run. The output of this step can be seen with --show-commands. For example:
    $ python build.py --show-commands
     - cp "../loader/server_i486/server_i486.so" "package/addons/metamod/bin/server_i486.so"
      - c++ loader.o gamedll.o serverplugin.o utility.o -shared -o server_i486.so
       - [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
    
  4. Updating. Each task in the task tree is executed and the results are processed to either reject the build, or update the state of the dependency graph. At this phase, tasks are executed in parallel based on the number of CPU cores available. The task graph is also shared to maximize throughput. You can see the sequential build steps with --show-steps:
    task 0: [gcc] -> c++ -Wall -Werror -H -c /home/dvander/alliedmodders/mmsource-central/loader/loader.cpp -o loader.o
      -> loader/server/loader.o
    task 1: c++ loader.o gamedll.o serverplugin.o utility.o -m32 -static-libgcc -shared -o server.so
      -> loader/server/server.so
    task 2: cp "../loader/server/server.so" "package/addons/metamod/bin/server.so"
      -> package/addons/metamod/bin/server.so