Boost.Build Python Port Design
This document outlines internal design of the Boost.Build V2 port to Python.
Constraints
The primary goal of the port is to replicate exactly the same functionality, and the same design and the same code, and get all the tests still work. Therefore, for modules in the 'build' and 'tools' directory only minimal design changes are allowed.
- No globals -- for the 'build' modules all data that is global in existing code will be kept inside various Python classes. There will be a top-level Manager object keeping all important data together. For 'tools' modules, using globals at this point does not seem reasonable, so we'll provide functions that return the 'global' instances of classes.
- Example: See the boost.build.build.project.ProjectRegistry class
- Build actions in Python -- in case where jam code, for a given command line, has both a command line proper and procedural code to setup something, said procedural code will be moved in Python -- while the command line will be declared using Jam syntax.
- Example: See how boost.build.tools.common.py defines the 'mkdir' function -- only the actual command is defined at Jam level
- Using of Python facilities. Python datatypes and builtin modules will be used.
- Using of the property_set class. Unless required for compatibility with Jamfiles, we'll be using property_set in all remaining cases where raw property list is used now.
- Path normalization. Jam code uses 'path.make' and 'path.native' to convert from native to normalized pathnames. Since Python has robust path manipulation functions, this logic will be dropped -- we'll use os.path everywhere.
Python style
Please try to adhere to the Python style guidelines at: http://www.python.org/dev/peps/pep-0008/.
The current code base breaks against a lot of these rules. If you encounter any of these problems while working on the code, please fix them as you go.
The most common problems are extraneous whitespace.
NO
__declared_subfeature [str (t, name)] = True m = __re__before_first_dash.match (toolset)
YES
__declared_subfeature[str(t, name)] = True m = __re__before_first_dash.match(toolset)
This is bad style. No whitespace should be used before brackets or parentheses. Sometimes whitespace is also used inside braces:
NO
{ 'foo': 'bar' }
YES
{'foo': 'bar'}
These whitespace should also be removed. Whitespace should also not be present in argument defaults:
NO
def __init__(self, name, project, manager = None):
YES
def __init__(self, name, project, manager=None):
Another common issue is using has_key to determine if a key is present
in a dictionary. Instead operator in should be used:
NO
if foo.has_key('bar')
YES
if 'bar' in foo:
Don't use [] as an argument default without making absolutely sure the argument isn't mutated inside the function. Instead, use something like:
def foo(x=None):
if x is None:
x = []
Don't use len() to test for empty sequences:
NO
if len(x): if not len(x):
YES
if x: if not x:
Python has booleans. Don't use 0 or 1 instead of False and True.
NO
def add_constant(self, name, value, path=0):
YES
def add_constant(self, name, value, path=False):
Design: Physical structure
The entire code base is divided in two parts -- Boost.Build proper and bjam. Boost.Build is written in Python. bjam is a build engine, written in C, and provides two major features:
- Parsing of project description files, called Jamfiles
- Build engine functionality -- detecting out-of-date targets and running commands to make them out of date.
The entry point is bjam, which is linked to Python interpreter. On startup, bjam imports Python main function and calls it, from which point Python is in control. We can alternatively create an extension library from bjam, and call that from Python, but this is not important design aspect.
Design: Logical structure
There's one top-level object -- manager -- of type boost.build.Manager. It holds all global definitions -- set of tools, features, project tree.
Project loading level
[Note that 'project' is basically a directory where either Jamroot or Jamfile is present].
Project loading is implemented by the boost.build.build.project module. The ProjectRegistry class maintains all known projects. The Manager.projects() method obtains the project registry for a manager.
The ProjectRegistry.load method load a project at specific directory. Each project is associated with a Jam module where
the Jamfile code is execute, with a project target, and a attributes object ProjectAttributes -- which keeps things like project requirements.
The attributes object is carried over from previous design, and will be eventually merged with the project target.
Loading of project follows these steps:
- Find Jamfile or Jamroot
- Prepare default attributes object, where all attribute are inherited from parent. Create new project target.
- Import all functions that must be useable in Jamfile context to Jam module for the new project
- Actually load the Jamfile.
If a Jamfile contains call to the 'project' rule, all specified attributes are passed to the ProjectAttributes object, which
adjust attributes. For example, for requirements, the properties specified in the 'project' rule are merged with the properties inherited from parent.
The list of functions that must be callable from Jamfile is maintained by the ProjectRules class. It has several method that
are automatically imported in Jamfile, as well as the add_rule method that can be called from anywhere. The add_rule
method takes a name that must be available in Jamfile, and a callable that will be called. This is the mechanism used to make
various main target rules available in Jamfile.
The callables imported into Jamfile will be called with exactly parameters passed in Jamfile -- there are no extra parameters implicitly added. While a Jamfile is being loaded, the ProjectRegistry.current method will return project target corresponding to the
currently loaded Jamfile. Main target rules should use it to decide in which project targets should be defined.
Metatargets level
Porting is in progress yet. Check back later.
Virtual targets level
Porting is in progress yet. Check back later.
Error reporting mechanism
See PythonPort/Design/ErrorReporting
Bjam engine level
On the bjam engine level, we have bjam targets and bjam actions. Bjam target is just a string. It can have arbitrary set of (variable-name,value) pairs associated with it. An action has a name, and a body. Body actually specifies the command that should be executed, and body can make use of the variables defined on targets. See bjam docs for more details.
In Python port, the bjam engine level is represented by the boost.build.engine.Engine class. The basic operation is to specify that certain targets are produced from other other targets using specified action. This is done by the 'set_update_action' method.
The targets are strings, they don't need any declarations. The actions however do need declarations. A declaration provides an name, and tells how the action is actually run. The action names are global to the engine instance, and used by other parts of Boost.Build. There are two methods to declare an action:
- The 'register_action' provides a command string to be executed. It's also possible to provide a Python function that will be executed before the command, and can set variables that are subsequently substituted in the command string.
- The 'register_bjam_action' specifies that a given action is completely defined in Jam code. This is only useful when user has defined some actions himself in a Jamfile. No part of Boost.Build will ever define action this way, it will
Design: Bjam-Python interface
Bjam side
The EXTRA_PYTHONPATH variable, in the global module, should be a list with directories where Python modules are searched, in addition to PYTHONPATH.
From Bjam side, a single extra rule is available:
PYTHON_IMPORT_RULE <python-module> : <callable> : <bjam-module> : <name> ;
This imports a python callable object from the specified module into bjam.
Python side
The following functions are avaiable from python, in the bjam module.
call <name>, <param0>, <param1>, ....
Looksup <name> in bjam module 'python_interface'. If found, calls that rule, passsing <param0>, <paramN>. Each <paramI> should be
a list of strings.
import_rule <bjam module>, <bjam name>, <callable>
Adds new rule <bjam name> to <bjam module>. Calling that rule will call back to Python <callable> -- which should accept lists of strings as parameters.
define_action action_name, action_body, bind_list, flags.
Defines new action with specified name, body, list if bound variables and flags. The action is created in the global module.
variable name <name>
Obtains the value of a variable 'name' in Jam's global module.
backtrace
Returns a backtrace from the point where Python was called from Bjam, till either bjam top-level, or the place where it was called from Python. The result is a list, each element of the tuple of format (file, file, jam module, function).
