wiki:AlternativeSelection
Last modified 14 years ago Last modified on 12/23/07 19:18:32

Proposal for "Do the Right Thing" Alternative Selection, by Dave Abrahams

This page describes a proposal for a change to the way target alternatives are selected.

Motivation

It seems there are a significant number of bjam invocations for which there is a sensible interpretation, but for which Boost.Build currently generates an error instead and refuses to build.

Example 1

Given the following Jamfile:

lib a : a.cpp : <debug-symbols>off ;                   # alternative 1
lib a : a-dbg.cpp : <debug-symbols>on <profiling>on ;  # alternative 2

In response to

bjam debug-symbols=on

the result is

error: No best alternative for ./a
    next alternative: required properties: <debug-symbols>off
        not matched
    next alternative: required properties: <debug-symbols>on <profiling>on
        not matched

My claim is that it would be more appropriate to build the 2nd alternative because it is a better match for the explicit build request.

Boost.Build generates an error because the default value of the <profiling> feature is "off". Boost.Build combines that default value with the explicit build request to get <debug-symbols>on <profiling>off, which does not match the 2nd alternative.

Example 2

I know at this point you're probably saying "the error makes sense; you asked for something for which there's no alternative with matching requirements." However, "no matching requirements" is currently not an error when there's only one alternative declared:

# b and c both have only one alternative. 
lib b : b.cpp : <debug-symbols>off ;
lib c : c.cpp : <debug-symbols>on <profiling>on ;

in this case, bjam debug-symbols=on happily builds both b (with debug-symbols off) and c (with profiling on) despite the fact that the requirements for b directly contradict the explicit build request and that the requirements for c contradicts the default for profiling, which is implicitly off.

The behavior in example 2 may seem counterintuitive, but it is widely regarded as desirable to succeed in building something, even if it doesn't exactly match what the user asked for, than to cause an error. I'm not challenging that premise in this proposal. In fact, there's an easy way to understand this behavior: requirements describe how a target must be built, not exactly how it must be requested. One point of this proposal is to make the case where there are multiple alternatives more consistent with that principle.

Definition of "Best"

Indeed, the error message didn't say "there's no matching alternative;" it said there's "no best alternative." Whether or not we accept this proposal turns on how we define "best." I propose that in example 1, the fact that alternative 2 matches an explicitly specified build property should make it "better" than alternative 1.

Example 3

This example is a simplified, but equivalent, version of the example in ticket #16:

lib test : : <name>test <toolset>gcc <threading>multi ;
lib test : : <name>test.lib <toolset>msvc <threading>multi ;

a simple "bjam toolset=gcc" currently yields:

error: No best alternative for ./test
    next alternative: required properties: <threading>multi <toolset>gcc
        not matched
    next alternative: required properties: <threading>multi <toolset>msvc
        not matched

Here Boost.Build complains because the <threading>single is the default. If you read ticket #16, you can see that the intended result was that Boost.Build choose the first alternative. My proposed change would accomplish that with no change in the Jamfile. We would lose the capability of specifying these alternatives in such a way that the build request bjam toolset=gcc causes an error.

Example 4

This is a real-life example that resulted in a long thread on the boost-testing mailing list, and was finally resolved only through an intensive back-and-forth with Martin over IRC. I should add that it was this difficult to debug despite the fact that I had already written most of this proposal; I just failed to recognize this as a manifestation of the same problem.

Martin's test-config.jam file is shown at the bottom of http://thread.gmane.org/gmane.comp.lib.boost.testing/4202/focus=4271. The crucial fragment is:

using python : 2.3 : /usr : : ; 
   ...
using gcc : 3.4.5_linux : /usr/local/gcc-3.4.5/bin/g++wrap --limit-memory=600 --limit-cpu=1800 : <root>/usr/local/gcc-3.4.5 <debug-symbols>off ;
using python : 2.4 : /usr/local/python/2.4/gcc-3.4.4 : : : <toolset>gcc <toolset-gcc:version>3.4.5_linux ;

Here, Martin configures python 2.3 as the default, and then configures a specific build of python 2.4 to be used with gcc-3.4.5_linux. Then he runs his tests, in the simplest case, with:

bjam gcc-3.4.5_linux

and the system uses the wrong python (2.3) to build and run these tests. To understand the reasons, you must know that

using python : version : ... : : : condition ;

ends up declaring a python library with the following requirements:

condition <python>version

As a result of the first using python... invocation, the default value of <python> is "2.3". Since "python=..." was not given explicitly in the build request, the current semantics add the default value to the request and thus fail to match the intended python installation. In fact, to test with multiple toolsets and their intended versions of Python, Martin has to invoke bjam as follows:

bjam gcc-3.3.6_linux gcc-3.4.5_linux/python=2.4 gcc-4.0.2_linux/python=2.4 etc...

This is not only inconvenient, but counterintuitive and uses a syntax that is not widely known. Under my proposal, the intended version of Python would've been selected.

Why Not Separate Alternative Selection Criteria from Requirements?

One might try to address these cases by forcing the user to specify their criteria for selecting alternatives separately from the alternatives' build requirements. That would be a bad idea, for two reasons.

First, the user usually wants a target to be built with the properties he explicitly requested, and often isn't aware of the default values of features that he didn't request. Currently, the explicitly-requested features are treated exactly like the defaulted ones, but the information about which ones were explicitly-requested is valuable; there's no reason to throw it out just because occasionally one additionally wants properties that aren't in the build request. Certainly, when the build request turns out to match an alternative's requirements exactly, Boost.Build currently considers that to be unambiguous, and that has not caused any real difficulty. This proposal expands the cases that Boost.Build considers unambiguous to better leverage the correspondence between explicit requests and desired requirements.

Secondly, with alternative selection criteria completely separated from requirements, to express the same thing as one could express with two alternatives in this proposal:

lib foo : foo.cpp ;

lib foo : bar.cpp : <feature1>value1 <feature2>value2 ... <featureN>valueN ;

would take N! (yes, that's N factorial) alternatives.

Note: I concede that there are a few interesting cases one can't express today because Boost.Build doesn't allow one to separately specify alternative selection criteria, like an alternative that must be built with intel C++ when the requested toolset is msvc. However, these cases are at best very unusual; if they do need to be accomodated, a separate feature is warranted, and there's no reason not to make the usual cases work concisely.

Details of Proposed Semantics

The semantics I'm proposing depend on the notion of "explicitly-requested" build properties.

Definition: a top-level target's explicitly-requested build properties are those that are specified on the command line, e.g.

bjam threading=multi.

A dependency target's explicitly-requested build properties are those that are propagated from its dependent target(s). As a special case, <toolset> is always considered to be explicitly requested, even when not explicitly specified on the command line.

When choosing among target alternatives, the one with the greatest number of requirements matching its explicitly-requested properties is selected. An ambiguity error is only reported when two alternatives match the same number of requirements.

When searching for dependency targets, the dependent's requirements are combined with its explicitly-requested build properties to form a new set of explicitly-requested properties, to be applied to its dependencies. During combination, where a non-free feature in the dependent's requirements conflicts with its explicitly-requested properties, the requirement is selected over the the explicitly-requested value of that feature.

Not A Pure Extension

This is almost a pure extension (i.e. one that does not change the semantics of any currently-working cases) but not quite. One case that it would change is as follows (Example 3):

lib a : a.cpp ;                                           # 1st alternative
lib a : a-dbg.cpp : <threading>multi <debug-symbols>on ;  # 2nd alternative

Today,

bjam debug-symbols=on

will select the first alternative. Under my proposal, it would select the 2nd. That leaves these questions:

  1. Is that an important difference? That is for you to answer.
  2. How would we get the current semantics? Read on...

Optional Extension to this Proposal

Currently, sa target alternative (when it's not the only alternative) will only be selected when all its requirements are matched in the build request. It's not yet clear that the rule "match all requirements" is actually useful in real-world scenarios (I admit that not matching requirements is a bit weird, but remember the rule of thumb: requirements describe how a target must be built, not exactly how it must be requested). If we decide it is important to support alternatives whose requirements must all be present in the build request in order to be selected, we would need a new notation. I suggest:

lib a : a.cpp ;                                           # 1st alternative
lib a : a-dbg.cpp : <threading>multi/<debug-symbols>on ;  # 2nd alternative

In this case, the 2nd alternative would only be selected if both <threading>multi and <debug-symbols>on were in the explicitly-requested properties. It would probably make sense to weight matches to these alternatives more heavily. A reasonable way of scoring that accounts for these property paths might be:

Sum( for all matching property paths p, 10s(p) )

where

s(p) :== the number of slashes in p

This syntax would also be more consistent with the way compound build requests are specified on the command line:

bjam toolset=gcc-3.4.5/debug-symbols=on toolset=intel/optimization=off

Summary

This proposal would make several problematic cases into non-problems, would make the handling of multiple alternatives more uniform with the handling of "single alternatives," could make the syntax/semantics of specifying conditions for alternatives more consistent with that of build requests, and would make some currently-"ambiguous" bjam invocations into well-defined commands. Admittedly, it would change the semantics of a common class of target declarations when combined with what appears to be an uncommon class of bjam invocations (so uncommon, I claim, that the changes are probably not important to anyone). So far, we don't have a use case for the current semantics that isn't equally well-covered by this proposal. However, should we find such a use case, the "optional extension" detailed here would allow us to recover the current semantics (with a different target declaration syntax, of course).