In my new project, we're using Team Foundation Server (TFS) as our source code repository and also as our continuous integration (CI) engine. The CI aspect is handled neatly by Automaton which leaves me to setup the build process (or the Team Build Type, as it is referred to in TFS parlance). Automated unit tests form the backbone of any decent CI process. It wasn't entirely straightforward to set these up, so I thought I'd share my experience with the community.
It's worth noting that we are using MS Test for unit tests, i.e. the unit tests built into Visual Studio Team System (VSTS), and not Nunit. I expected that this would simplify the build process since I wouldn't be integrating Nunit results back into the build report. This has been the case, but there are a few gotchas.
As stated on MSDN "How to setup a build server":
In order to run tests during build, Team Edition for Testers must be installed on the build computer. In order to run unit testing, code coverage or code analysis, Team Edition for Developers must be installed on the build computer.
I find it quite surprising that I need a Visual Studio installation on the server, but the docs are pretty clear on this point. Are there any licence implications of this? The Visual Studio 2005 Team System licensing white paper states:
As part of the build process, Team Foundation Server may run quality tests and/or analysis on the precompiled or compiled code. These tests rely on functionality found within Team System client products, typically within the Team Edition for Software Developers or Team Edition for Software Testers products. These products may be installed on the build machine by licensed users of those products, as long as they are not directly used by any individuals who are not licensed for those products.
In short, licencing is not a problem.
Now unit tests really are just a type of test, so I'm not entirely clear whether I need VS TE for Testers or VS TE for Developers. I'm planning to do some automated system testing and so I've installed the relevant bits from both editions on our build server. I'd guess that you really only need VS TE for Developers.
So you would think that now our build server ought to be fully setup and we now ought to be able to execute unit tests. Well you can, but only if they are part of a test list. VSTS groups tests into lists and you need to specify (in the TfsBuild.proj file) the name of a test list containing unit tests to execute. This is clear in the TfsBuild.proj file, where you need:
<TestList>List of Unit Tests</TestList>
I don't think this is good enough. It's enough of a problem for me to get developers to write unit tests. I don't want to have to then get hold of a VS TE for Testers installation (which is the edition that allows you to edit test lists) and then add these unit tests into the appropriate list to get executed. And it seems that I'm not the only one. Enough people have complained about this that Buck Hodges (the dev lead for Team Build) has posted some modifications to TFS that allow you to execute all unit tests in a DLL. These take the form of a DLL containing an updated MS Build task and a revised Microsoft.TeamFoundations.Build.targets file (plus some docs). Both of these files need to be installed on your build server. Having done that, you can get rid of the MetaDataFile element in TfsBuild.proj and replice it with:
<TestContainerInOutput Include="%2a%2a\%2a.UnitTests.dll" />
This (rather odd) syntax translates into **\*.UnitTests.dll. In other words, you can just specify which DLLs contain your unit tests and the Team Build will now execute them all.