Microsoft Team Build Details

Core MSBuild concepts and Team Build target details

Core MS Build Concepts and Ant/Nant Equivalents

Microsoft's Team Build product runs on a foundation provided by MSBuild. Therefore, to understand how to configure and extend Microsoft Team Build, you need a good grasp of the basic MSBuild concepts and mechanics. For those familiar with Ant or Nant, a superficial glance at MSBuild shows it to be very similar to Ant or Nant. Only a slightly deeper look, however, reveals significant differences despite the tools' identical raison d'etre.

Properties

Properties are name/value pairs. Both Ant/Nant and MSBuild use them profusely. They are similar to character string variables in traditional programming languages. In Ant and Nant you specify a property using the following syntax:

For example:

To declare a property in MSBuild, use the following unusual syntax:

<PropertyGroup>
    ...
    <PropertyName>PropertyValue</PropertyName>
    ...
</PropertyGroup>

For example:

<PropertyGroup>
    <BuildNumber>140</BuildNumber>
    <BuildType>debug</BuildType>
    <CopyrightOwner>Stephen R. Palmer</CopyrightOwner>
</PropertyGroup>

Unlike Ant and Nant, MSBuild uses the XML tag name to define the name of the property and the contents of the tag as the value of the property. This is unusual because it makes it harder to specify anything useful about property definitions in an XML schema. To make it clear to the MSBuild engine that an unrecognized tag name is a property definition, all properties must be defined within a PropertyGroup tag. In Ant and Nant, the equivalent of a PropertyGroup tag is not needed because the tag name indicates that a property is being defined while attributes supply the actual property name and value. Having used XML schema for a number of years, I do not know how to write one to constrain the contents of an arbitary-named tag enclosed by a specifically-named tag. I have never needed to know this before, and, if it is indeed possible, it would seem to me to add unnecessary complexity. It is not a design decision that I would have made. I much prefer the way Ant and Nant do it.

The other thing to note is that while And and Nant start tag and attribute names with a lower case letter, MSBuild tags and attribute names start with an upper case letter. I have to admit that this is a consistency of sorts. After all, while Java attibutes and operations start with lowercase letters, C# properties and operations start with upper case letters. Consistent inconsistency maybe? Anyway, having worked with Ant a number of times in the past, this difference in MSBuild has caught me out more times than I like to admit. It can be a very irritating source of problems.

Another little difference between Ant/Nant and MSBuild that can be very annoying appears when trying to use the value of a property. To refer to the value of a property in Ant, we put the name of the property inside braces (curly brackets) and prefix it with a dollar sign. To refer to a property in MSBuild, we do the same thing except we use normal curved brackets instead. Here it is Ant and Nant that have departed from the traditional convention because the venerable Make tool refers to property values in the same way MSBuild does, using the dollar prefix and curved brackets convention. For example, in Ant or Nant both

<echo>The build number is: ${buildNumber}</echo>

and

writes out a message displaying the build number while

does the same thing in MSBuild.

MSBuild and Ant/Nant property definitions do have one other thing in common, however; they both have more to them than the simple examples above show. For example, instead of defining the name and value of a single property, the Ant/Nant property element can load a number of property definitions from a specified file or URL. MSBuild property definitions do not provide this option but they do enable a condition to be specified, both in the property tag itself and in the property group tag. Specifying a condition means that the property is only defined if the condition evaluates to true. Conditions enable the writer to add conditional logic to the build script. For example:

<PropertyGroup>
<BuildLabel Condition="$(BuildType)!=''RC">Debug.140</BuildLabel>
<BuildLabel Condition="$(BuildType)=='RC'">Release.140</BuildLabel>
</PropertyGroup>

is essentially an if-then-else statement setting the build label property to different values depending on the value of the build type property. Ant and Nant also have this feature but the attribute used for the condition is called if instead of condition. For example:

AS you can see, within an expression, Ant uses the property::getValue() function to retrieve the value of a property. MSBuild retains the use of the $() notation. I find this MSBuild way of expressing conditions much more elegant than Ant/Nant.

The need in MSBuild to place property definitions inside a PropertyGroup tag is painful except when there is a need to apply a condition to a set of properties. Being able to specify the condition at the property group level is the tag's only saving grace. However, both Ant and MSBuild feel vulgar when compared to the far clearer and more concise variable definitions and conditional constructs found in most modern programming and scripting languages.

The complete set of attributes for the ANT property task can be found in property section of the Ant manual. The manual also contains full details on the Ant condition task. MSDN has more detail about MSBuild properties.

MSBuild Items

Unlike Ant and Nant, MSBuild differentiates between properties used to specify inputs and those for task configuration options or control flags. In MSBuild, for inputs such as sub-projects, source files, etc, you define Items within an ItemGroup. For task configuration options and control flags you define Properties within a PropertyGroup as described in the previous section. Ant and Nant make do with a single property concept for all three usages.

Items and ItemGroups are one of the most confusing aspects of MSBuild. Theu have no direct equivlent in Ant or Nant, or to my knowledge any in mainstream programming or scripting language.

The first thing to be aware of is that, just like the MSBuild PropertyGroup tag and its enclosed property tags, the ItemGroup tag has no meaning other than to indicate that the enclosed tags define items within collections. The ItemGroup tag itself does not define a collection. The names of the tags enclosed within it are the collection names. For example, consider the following:

This does not define two collections of items, it defines three collections called Compile, Resource and Schema. It does not matter which ItemGroup element that the item element is defined in, all items with the same tag name define the contents of that collection. The Include attribute can take single file name, a list of semicolon-separated file names, or a set of file names specified using wildcards. The Exclude attribute can also be useful when working with wildcards.

Using Items

Collections of items are usually passed to tasks as parameters using the syntax @(ItemCollectionName) to refer to the collection as a whole. For example, the list of files to compile in the previous examples, can be passed to the compile task as @(Compile). Under the covers, the collection is transformed into the format required by the task parameter, delimited character string, string array, or array of ITaskItem objects. Only the latter includes any associated metadata values. The default delimiter character is a semi-colon. this can be ovrriden by spcifying an alternative delimiter. For example, to use a comma as a delimiter in the compile item collection use @(Compile, ',').

Defining Meta-Data and Batching

The most confusing thing about using items has to be the ability to define meta-data for items, use that meta-data to split the items in a collection into batches and have targets and tasks execute multiple times, once for each batch of items in the collection.

Declare metadata for items by specifying a child element of the item, setting the name of the child element to the name of the metadata. An item can as many metadata elements as needed. For example, the following has metadata that indicates that this item represents part of the Security subsystem.

<ItemGroup>
    <Compile Include="CoreSec*.cs">
        <Subsystem>Security</Subsystem>
    </Compile>
</ItemGroup>

One thing the meta-data values can be used for is called batching. Instead of using the @(ItemCollectionName) notation to pass the whole of a collection to a task as a parameter, the %(ItemCollection.ItemMetaDataName) notation to cause the tasks to be executed repeatedly, once for each different value of the specified metadata name.
For example, the following colelction is split into two batches, SmokeTest and InDepthTest identified by the TestDepth metadata name.

<ItemGroup>
    <Test Include="$(OutDir)Party.dll">
        <TestDepth>SmokeTest</TestDepth>
    </Test>
    <Test Include="$(OutDir)Transactions2.dll">
        <TestDepth>InDepthTest</TestDepth>
    </Test>
</ItemGroup>
...
<ItemGroup>
    <Test Include="$(OutDir)Transactions1.dll">
        <TestDepth>SmokeTest</TestDepth>
    </Test>
    <Test Include="$(OutDir)Products.dll">
        <TestDepth>InDepthTest</TestDepth>
    </Test>
</ItemGroup>

Passing %(TestDepth) to a testing task would cause the task to be executed twice, once for the items with the SmokeTest metadata value and once for the items with the InDepthTest metadata value.

The above technique is called task batching. There is also target batching that is used to ensure that a target can compare input and output timestamps appropriately. If this is not done when task batching is being used, the target may not skip execution when it is already up to date. Target batching uses the %(ItemCollection.ItemMetaDataName) notation to specify the Inputs and Outputs attributes of the target element.

I did not find these batching techniques easy to get mu head around. Properties, tasks and targets make perfect sense to me as a developer but I confess to have struggled for a considerable time to fully understand items, item groups, target and task batching and their application in controlling iterations. I still find their use an uncomfortable set of thought processes but then I have never really become fully comfortable with the either Make or Ant's 'depends on' style of programming either. I find structured and object-oriented coding thought processes far more natural.

Maybe these other styles of programmig come more naturally to those that tend to make their careers as build-engineers (some I have worked with are simply brilliant at their jobs). I did say at the start that the 'Build Manager' hat does not fit me quite as well as it obviously fits others; my 'Development Lead' hat fits better even if it is sagging a little at the corners after more than (but not that much more I hasten to add) twenty years of wearing it.

Team Build Details

With a good understanding of how properties and expecially items work within MSBuild, getting to grips in detail with the targets in the TeamFoundation.build.targtes file becomes a little easier.

Check Settings for End to End Iteration Target

This first target in the build is simple. It checks that half-a-dozen specific properties that store information about the build definition, and the TFS server have values. If any are undefined, an error is reported and the build stops. The target uses Error tasks to accomplish this; error tasks evaluate a condition and if true, log an error message and cause the build to stop. The target also uses the Message task to log the values of the properties checked but only if the logging level is high enough to include Low importance messages. In Ant/Nant you achieve simialr using the fail and echo.

Initialize Build Properties Target

The second target executed sets a long list of properties by executing the GetBuildProperties task. Given the TFS server and an unique URI for this build, the task sets a swathe of initial values for properties including various success and status flags, start time, the build directory, number and label, etc. Interestingly, the properties set also include the build definition properties just checked in the previous target.

When the results of MSBuild tasks need to be used later in the build, Output elements enclosed within the task tags map a TaskParameter attribute to a PropertyName attribute. For example, the values of the two output paramters of MyTask below are copied into properties named Output1 and Output2.

The Ant/Nant world does not have an exact equivalent because it does not need it. There is no built-in database to maintain in the Ant/Nant world. We can load a set of properties from a file in Ant/Nant simply using the following:

This technique is frequently used to tailor a build for a particualr environment and really serves the same sort of job that the TFSBuild.proj file does. If we want to maintain a database of builds executed, we have to build that sort of support ourselves or look for a 3rd party product. Fortunately, I have never had a requirement to do so and, so far, the database tightly integrated with TeamBuild has provided nothing that I need or want; it has been more of a nuisance than a benefit.

Initialize End to End Iteration

The InitializeEndToEndIteration sets the values of a few properties that control whether the build does an incremental get and build or a complete 'from scratch' build. For a frequent or 'continuous integration' build, it is common to check out and compile only the files that changed since the last build. This can make the build significantly shorter and, therefore, increase the frequency with which the build can be run. Incremental builds require a 'from scratch' build to be run at least once. However, 'from scratch' builds generally run at lower frequency, nightly or even weekly, to compliment the incremental build. The 'from scratch' builds often do more than the incremental build including tasks such as executing longer-running tests, and generating in-depth documentation and reports.

The Team Build framework uses to properties to communicate to the build engine the desired type of build, incremental or not, and if incremental, what kind of incremental. The properties in question are IncrementalGet and IncrementalBuild.

If IncrementalGet is set to true, the build's workspace is not deleted, only the compilation output is deleted during the clean target execution, and only files that have changed are retrieved from TFS Source Control.

If IncrementalBuild is set to true, the build's workspace is not deleted, the clean targets are skipped altogther, and only files that have changed are retrieved from TFS Source Control.

I have never set either of these properties to true because I have had enough fun and games getting and keeping the 'from scratch' builds working correctly as the project has progressed.

The target also updates the TFS build database with the build number and build drop location, and defaults the build label (LabelName property) to the same as the build number (BuildNumber property).

In my previous experiences with Ant, we have combined it with a 'continuous integration' tool such as CruiseControl or Hudson that controls the incremental versus 'from scratch' behavior of the build.

Initialize Workspace Target

Team Foundation Server Source Control requires the definition of workspaces. A TFS workspace is a named mapping between a set of folders in the TFS Source Control repository and working directories in a filesystem. When a user, in our case the build engine, requests files from TFS Source Control, the workspace mapping determines the directories on the user's computer that the files are copied to.

Each build definition defines a workspace. The initialize workspace target deletes the build definition's workspace by invoking the DeleteWorkspaceTask task. For a 'from scratch' build this also deletes the contents of the mapped working directories but this can be prevented by setting the SkipClean or CleanCompilationOutputOnly properties to true. For incremetnal builds the deletion of the workspaces is skipped completely.

The target completes by creating or recreating the workspace by invoking the CreateWorkspaceTask task. The name and owner of the workspace are stored in the WorkspaceName and WorkspaceOwner properties respectively as a result.

Clean All Target

Incremental builds executes the CleanCompilationOutputOnly target but as stated earlier, I have not yet had a chance to try it. For 'from scratch' builds, the build engine executes the CleanAll target instead. This simply uses the RemoveDir task to delete, if they exist, the directories named in the properties SolutionRoot, BinariesRoot, TestResultsRoot.

Great stuff! Except when you cannot remember what directories these actually point to. A quick search of the targets file reveals that SolutionRoot is the parent directory of the directory pointed to by the MSBuildProjectDirectory property. A quick search of MSDN reveals that the MSBuildProjectDirectory property points to 'the absolute path of the directory where the project file is located'. Only the TFSBuild.proj file is in its own Source Control folder that is not mapped inside my build defintion's workspace. It turns out that the MSBuildProjectDirectory property points to the working directory specified by the Build Agent (the computer or build server in other words) that is running the build.

The BinariesRoot directory turns out to, by default, be a directory called Binaries located in the same directory as the SolutionRoot directory. Likewise, the TestResultsRoot directory turns out to, by default, be a directory called TestResults located in the same directory as the SolutionRoot directory.

Initialize Build Targets

Having deleted all the directories in the CleanAll target, the InitializeBuild target simply recreates the SolutionRoot directory, the directory into which the TFSBuild.proj file will be copied from TFS Source Control.

Get Target

This target and the Label target are actually grouped into a target called PreBuild that does nothing except execute the Get and Label targets.

The Get target exceutes the Get task that retrieves folders and files from the TFS Source Control folders specified in the build definition's workspace and copies them into the workspace's working directories.

The GetVersion property passed as a parameter to the Get task determines which versions of the folders and files are retrieved from Source Control. For a regular or frequent build, this is usually the latest or a particular label that indicates that the folder or file is ready to be part of the build. Setting the GetVersion property to LPromoted (the L indicates that what follows is a label and not a date/time, changeset version or workspace version) in the TFSBuild.proj file means that instead of building the latest versions of everything, the build extracts the versions of the files and folders with the Promoted label. This enables developers to control when folders and files checked into Source Control participate in this particular build. This is helpful when a small team of developers need to coordinate a set of check-ins before the build incorporates any of them. When a project has a number of small feature teams each coordinating sets of check-ins as they do within a process like Feature-Driven development (FDD), this sort of promote-to-build step almost becomes a must.

Also passed as a parameter, the GetFilespec property can be set to filter the types of file retrieved by the Get task but this can usually be left at its default value of the empty string. This results in all the files in all the folders defined in the workspace are retrieved.

Although the Get task in the targets file is set up to copy the lists of files retrieved, deleted, and replaced and any warnings to build properties, this is disabled by default because the GetPopulateOutput property is defaulted to false in the targets file.

Label Target

The Label taget executes the TFS Label task that by default labels all (actually project scope as defined by the default value for the LabelScope property) the folders and files in TFS Source Control that have equivalent directories and files in the workspace's working directories (defined by the default value for the Labelversion property) with the build number (because the default value of the LabelName property is the build number).

The target aslo records the label name in the build database by executing a SetBuildProperties task.

Compile Target

By the time we reach the compile target we have all the relevant source files in the working directories of the build defintion's workspace, and these files are also labeled in TFS Source Control as taking part in this particular execution of the build process. The preparation is complete and now the actual work of building begins.

Here life gets complicated again. Ignoring any compile targets provided specifically to be overridden to extend or modify the compilation behavior, we still have 4 interdependent compile targets:

Compile that depends on CallCompile that executes MSBuild.exe again with the same TFSBuild.proj file but this time invoking the CoreCompile target instead of the End2EndIteration target.

At this point it is important to understand how MSBuild evaluates properties. When MSBuild is run, it reads through all the property definitions in the supplied project file including properties defined in files imported by the project file. Later definitions of the same property override earlier definitions. Therefore, because almost the first thing the TFSBuild.proj file usually does is import the Team.Foundation.Build.targets file, subsequent property definitions in the TFSBuild.proj override those within the imported targets file. This happens before any targets are executed.

In addition properties defined on the MSBuild command line override both of these, but when executing MSBuild.exe again from within a target property values must be explicitly passed through to the execution on the MSBuild command line.

Therefore, by executing the MSBuild task, the CallCompile target causes the properties to be evaluated again for the scope of the task in the light of the property values specified to be command line parameters.

So MSBuild is run again this time starting at the CoreCompile target. This in turn invokes MSBuild again, once for each configuration being built. Again the TFSBuild.proj file is specified but this time the target to start with is CompileConfiguration.

The CompileConfiguration target again invokes MSBuild this time for each solution within the project (as defined by the SolutionToBuild item collection), still with TFSBuild.proj as the project file but this time CompileSolution as the starting target.

Of course CompileSolution invokes MSBuild but this time it executes the project files in the solution.

On eventually returning from this call to MSBuild any errors cause the build to be marked as failed and eventually halted.

Test Target

Like the compile target, the test target is executed iteratively, once for each build configuration.

On my project we supply a list of test containers to this target. By default it seems to order the test containers alphabetically before running all the tests in all the containers. The TestToolsTask task run in this case launches the VSHost process and executes all the tests within it. For us this eventually caused a problem. Either some part of the common set up for our tests was leaking memory or the VSHost process holds data from each test in memory until the end of the run. Either way, some of the tests at the end of the run started failing with out of memory errors. Moreover, there is no visual indication of progress within Visual Studio during a test run. This meant that for over an hour we had no idea of how many of the tests had been executed or if any of them had failed.

A quick search of the internet revealed a simple fix for the out of memory situation. Described by Bill Wang, it involves copying the CoreTestConfiguration target and changing @(LocalTestContainer) to %(LocalTestContainer.Identity). This causes the target to execute each test container within a separate test run. Of course, overriding CoreXXX targets is not generally recommended. Nevertheless, it seems a reasonable exception to the rules in this case; other alternatives require significantly more changes. In addition, when the running of each test container completes, Visual Studio displays the results for that container, and this gives us some indication of the progress of the testing. It also demonstrates again the power of task batching item collections, and how that power may be represented by a very small syntactic difference within a crowd of task parameters.

Batching up the test runs had another unintended 'benefit'. A small batch of tests that ran perfectly happily before batching the tests started failing, complaining that they could not load certain assemblies. Investigation revealed that the rouge tests loaded classes from these assemblies by reflection. Despite the test projects having references to the required assemblies in Visual Studio, the assemblies were not being copied to the test location. Visual Studio copies the assemblies and, therefore, the tests pass when run interactively, but the build does not copy the assemblies and the tests fail as a result in the build. This problem was being masked when the tests all ran as one big test run because other tests were either causing the assemblies to be copied or were loading the required classes into a statically held cache from which the rogue tests retrieved them instead of trying to load them from disk.

The fix for this inconsistency with assembly copying between Visual Studio and Team Build appears to require the assemblies to be added to the build's test configuration. Unfortunately, since the directory where the build plops its output is defined by a property and varies depending on the configuration being built. The entries in the test configuration file need to be absolute file names. This means that as part of the compiling of each Visual Studio project we copy the build output to a known location as it is built so it can be referred to easily in this way.

Drop Build Target

By default this target copies all the artifacts created during the build to the output directory specified in the build definition.

Conclusion

While MSBuild seems to provide all the nuts and bolts required to create build scripts despite the unusual definition and use of items. Unfortunately, the integration with Visual Studio and especially Team Foundation Server makes life much harder than it should be. The Team Build targets file is simply too complex and a number of the tasks simply too opaque to bend the process very far from the norm expected by the framework's developers.

Previous experience with builds of this level of sophistication suggests that a build based on Ant and CruiseControl would be in place within a couple of weeks. It has taken at least a couple of months to establish the build using Team Build 2008 and it is still far from ideal.
Significantly enhancing an Ant/CruiseControl build is simpler than significantly enhancing a Team Build 2008 build.
Team Build 2008 provides a workable integration with Visual Studio 2008. Nant/CruiseControl does not. This is even more the case with Team Foundation Server integration. There seems little incentive for this situation to change.
XML is not a great vehicle for creating build scripts. It never has been and never will be.
There is much more to be said and possible solutions and workarounds to try out but Team Build is almost completely rewritten in the 2010 version to use Microsoft's work flow engine, another different XML-based technology.