The devil of regression testing

 

"No sympathy for the devil; keep that in mind. Buy the ticket, take the ride...and if it occasionally gets a little heavier than what you had in mind, well...maybe chalk it off to forced conscious expansion"

 Hunter S. Thompson (Fear and Loathing in Las Vegas)

On our mailing list where we discuss all sorts of important development issues and organise pub trips, a mate of mine came up with an interesting problem to do with testing - It went a bit like this ..

“I've just been talking to someone who I has attempted agile without any QA or automated tests at all. Unsurprisingly they've seen an increase in regression issues and are now looking to retrofit done automated testing.”

As the resident tester it got me thinking. What is the cost of regression testing in an agile project ? - well the cost in my opinion is actually 0, zip, nadda, snake eyes !!

Right about now everyone who has faced the cost of regression testing is thinking I am smoking crack? or maybe hill billy drunk on the 4th of july? - I’m not and here is why.

Lets say you have a project you estimated at 30 points - whatever - you figure as you are going along you can do roughly 5 points a week including getting it manually QA’d in a test environment and showed with great fan fare to your product owner  - Happy days!! - you will be out of there cashing the check in 6 weeks... 

or will you.. 

What was your ACTUAL velocity??

Easy I hear you all say - 5 points!! 

The problem is, it wasn’t really - The 5 points a week was stuffed with tech debit, Now I’m not talking the stupid tech debit i’ve cited before where you just sweep all the hard stuff into the tech debit backlog to temporarily inflate your velocity (and we all know how thats going to end) I’m talking about serious tech debit that wont bite you for a month or so. The question is actually what did you DO for that 5 points? 

Write code that solved the problem?

Or write code that solved the problem with unit tests, integration tests, platform tests, automated UI tests that prove every build and deployment you haven’t broken anything??

In most cases its the former  - And here is where logically (though not financially I’m sorry to say) it doesn’t cost. What is actually happening is your velocity gets corrected. You weren’t doing 5 points, you were actually doing maybe 3 but incurring the tech debit of poorly tested code allowed the team to inflate their velocity to an unrealistic 5. The over all correction theoretically will be 4 weeks in this case - but anyone who has tried to go back and retrofit tests to poorly tested code will know its harder than writing them at the time. so in this case it will take longer. 

I guess if you can guarantee some other poor stooge will have to deal with it later you *could* give them crappily tested code to work with but once the smell has pervaded the building you wont be working back there any time soon and your rep wont be the greatest.

 

July 5 2011

Finding all projects containing a reference to X assembly using POSH

 

Finding all the projects which reference a specific assembly can be a little tough in visual studio if you have hundreds of projects. NDepend can do this for you, but that requires you’re willing to pay for a licence (you should) if you need it commercially.

I wrote a little bit of POSH to do this for me…

function Get-ContainsReferenceTo($projectFile, $referenceName)
{
    [xml]$s = get-content $projectFile

    $references = $s.Project.ItemGroup | Where-Object { $_.Reference -ne $null }
    
    foreach($reference in $references.ChildNodes)
    { 
        if($reference.Include -ne $null -and $reference.Include.StartsWith($referenceName))
        {
            return $true
        }
    }

    return $false
}

 

After loading the project file into an XmlDocument, we can iterate through each reference and do a string compare against the reference name.

Running this will return “True” or “False” as to whether the file contains the reference or not.

We can then do this for all project files in a directory by doing:

function Get-ProjectsContainingReferenceTo($directory, $referenceName)
{
    Get-ChildItem $directory -include *.csproj,*.vbproj -Recurse | foreach ($_) { 
        if(Get-ContainsReferenceTo $_.fullname $referenceName -eq $true){ $_.fullname }
    }
}

 

Now if any project files match the reference, the full path of the project file will be output to the command window. We can use this to perform batch operations to change references, remove them or anything else we would like.

I hope you find this useful and you can find it bundled with SlightlyPosher.

The Template Method pattern gone wrong (or how to stop your overridden methods getting called)

The definition of the “Template Method pattern” may or may not be familiar to you, but the following bit of code sure will:

public abstract class BroadcasterBase 
{ 
    public void SendMessage(Message msg) 
    { 
        // Perform some validation and pre-processing on msg 
        base.SendMessageInternal(msg); 
    } 

    protected virtual void SendMessageInternal(msg)
    {
        // Some default behaviour
    }
}

public class NetworkBroadcaster : BroadcasterBase
{
    protected override void SendMessageInternal(Message msg) 
    {
        // Send the message, now that it has been validated and all pre-processing is complete
    }
}

For the pedants amongst us (which includes me), the Template Method pattern can be defined as:

A template method defines the program skeleton of an algorithm. One or more of the algorithm steps can be overridden by subclasses to allow differing behaviours while ensuring that the overarching algorithm is still followed.

Now this works well until it starts becoming over-engineered and we start having more than one base class. 

public abstract class BroadcasterBase 
{ 
    protected virtual void SendMessageInternal(msg)
    {
        // Some default behaviour
    }
}

public abstract class ValidatingBroadcaster : BroadcasterBase
{
    public void SendMessage(Message msg) 
    { 
        // Perform some validation and pre-processing on msg 
        base.SendMessageInternal(msg); 
    } 
}

public class NetworkBroadcaster : ValidatingBroadcaster
{
    protected override void SendMessageInternal(Message msg) 
    {
        // Send the message, now that it has been validated and all pre-processing is complete
    }
}

And at this point it’s actually broken!  The code inside NetworkBroadcaster will no longer get executed.  This is actually down to one very small bit of code … the use of the “base” keyword in the ValidatingBroadcaster.  This actually makes the call non-virtual and binds it to the specific class, in this case BroadcasterBase.SendMessageInternal.  Eric Lippert puts it nicely:

a base call is a non-virtual call to the nearest method on any base class, based entirely on information known at compile time.

I ran into this problem with the Microsoft Prism framework when extending the eventing mechanism.  Specifically I wanted to derive from CompositePresentationEvent and override the event publication mechanisms.  This particular problem prevented me from doing that.

There are probably two important lessons from this:

  1. Don’t go blithely putting “base.” in front of methods calls, e.g. because something like StyleCop thinks you should
  2. Just because you have a base class with some overridable methods, don’t assume they will get called
June 27 2011

NBuilder is fantastic for generating object graphs

 

A library which I have not seen used often enough is NBuilder. On my current project we needed to create a WCF service for consumption by a 3rd party. Since the structure of the web service was agreed beforehand, we wanted to publish an early version for them to connect to.

Instead of manually hand cranking the test data that the API was to return, we were able to use NBuilder to do 90% of the work, with overrides to handle more complex cases.

You can also use it in your unit tests wrapped by a builder pattern to create common object graphs for use by your tests.

We created some common interfaces for our builders to get a default setup of an object, which we could then use the provided builder methods to override.

public interface IBuilder<T>
{
    ISingleObjectBuilder<T> GetDefault();
}

public interface IManyBuilder<T>
{
    IListBuilder<T> GetDefault();
}

Next we implemented default builders to get the common configured setup. NBuilder has generators for many common types, and iterates through the object graph populating dummy data.

public class CustomerBasketBuilder : IBuilder<Basket>
{
    public ISingleObjectBuilder<Basket> GetDefault()
    {
        var basketItems = new CustomerBasketItemsBuilder().GetDefault().Build();

        var basket = Builder<Basket>
            .CreateNew()
            .With(basket1 => basket1.BasketItems = basketItems)
            .With(basket1 => basket1.ItemCount = basketItems.Count)
            .With(basket1 => basket1.CustomerId = new PrimitiveBuilder().GetCustomerId())
            .With(basket1 => basket1.TotalBasketValue = basketItems.Sum(item => item.BasePrice));

        return basket;
    }
}

 

Whenever we use the With method here, we are setting up an override over the default generators. I manually call the BasketItemsBuilder to get the default setup for a list of items, and then ask NBuilder to Build the object graph using a combination of my overrides with the default generators.

public class CustomerBasketItemsBuilder : IManyBuilder<BasketItem>
{
    public IListBuilder<BasketItem> GetDefault()
    {
        var generator = new UniqueRandomGenerator();
        
        var basketItems = Builder<BasketItem>
            .CreateListOfSize(10)
                .All()
                    .With(item => item.ItemId = generator.Next(100000, 999999))
                    .With(item => item.Colour = Pick<string>.RandomItemFrom(new[] { "Red", "Green", "Blue" }))
                    .With(item => item.ItemType = BasketItemType.Product)
                    .With(item => item.ValidOperations = new[]
                                                             {
                                                                BasketItemOperations.CanSaveForLater,
                                                                BasketItemOperations.CanDelete,
                                                                BasketItemOperations.CanEdit
                                                             });
                
        return basketItems;
    }
}

 

UniqueRandomGenerator is one of the OOTB generators, which is normally called implicitly, however we needed a defined range in this case. The Pick methods are also useful for one liner random item choosing. I strongly recommend going through the NBuilder codebase to see what’s available. There’s some real gems in there which unfortunately are not documented.

Here’s a refactored sample from one of our tests:

Before After
image image

 

It turns out refactoring unit tests is really satisfying, and reduces stress, I highly recommend it.

Newer Posts Older Posts