The devil of regression testing


"No sympathy for the devil; keep that in mind. Buy the ticket, take the ride...and if it occasionally gets a little heavier than what you had in mind, well...maybe chalk it off to forced conscious expansion"

 Hunter S. Thompson (Fear and Loathing in Las Vegas)

On our mailing list where we discuss all sorts of important development issues and organise pub trips, a mate of mine came up with an interesting problem to do with testing - It went a bit like this ..

“I've just been talking to someone who I has attempted agile without any QA or automated tests at all. Unsurprisingly they've seen an increase in regression issues and are now looking to retrofit done automated testing.”

As the resident tester it got me thinking. What is the cost of regression testing in an agile project ? - well the cost in my opinion is actually 0, zip, nadda, snake eyes !!

Right about now everyone who has faced the cost of regression testing is thinking I am smoking crack? or maybe hill billy drunk on the 4th of july? - I’m not and here is why.

Lets say you have a project you estimated at 30 points - whatever - you figure as you are going along you can do roughly 5 points a week including getting it manually QA’d in a test environment and showed with great fan fare to your product owner  - Happy days!! - you will be out of there cashing the check in 6 weeks... 

or will you.. 

What was your ACTUAL velocity??

Easy I hear you all say - 5 points!! 

The problem is, it wasn’t really - The 5 points a week was stuffed with tech debit, Now I’m not talking the stupid tech debit i’ve cited before where you just sweep all the hard stuff into the tech debit backlog to temporarily inflate your velocity (and we all know how thats going to end) I’m talking about serious tech debit that wont bite you for a month or so. The question is actually what did you DO for that 5 points? 

Write code that solved the problem?

Or write code that solved the problem with unit tests, integration tests, platform tests, automated UI tests that prove every build and deployment you haven’t broken anything??

In most cases its the former  - And here is where logically (though not financially I’m sorry to say) it doesn’t cost. What is actually happening is your velocity gets corrected. You weren’t doing 5 points, you were actually doing maybe 3 but incurring the tech debit of poorly tested code allowed the team to inflate their velocity to an unrealistic 5. The over all correction theoretically will be 4 weeks in this case - but anyone who has tried to go back and retrofit tests to poorly tested code will know its harder than writing them at the time. so in this case it will take longer. 

I guess if you can guarantee some other poor stooge will have to deal with it later you *could* give them crappily tested code to work with but once the smell has pervaded the building you wont be working back there any time soon and your rep wont be the greatest.


July 5 2011

Be The Headlights Not The Handbrake


Some people will tell you that slow is good – but I’m here to tell you that fast is better. I’ve always believed this, in spite of the trouble it’s caused me. Being shot out of a cannon will always be better than being squeezed out of a tube. That is why God made fast motorcycles, Bubba…"

— Hunter S. Thompson (Kingdom of Fear: Loathsome Secrets of a Star-crossed Child in the Final Days of the American Century)

I haven't been at ThoughtWorks long but its been a happy couple of weeks - There is good conversations on a wide number of lists about all sorts of crazy stuff.

One that really grabbed my interest was a whole fiery chain about test automation and much opinion. Again someone raised the point of view that testers are like tigers (In the future survival stakes rather than the scary stakes) and will be extinct soon. This is something I have been yearning after for years - Imagine a world where the development practices employed by development teams are so quality driven we didn't need to test at all after they were done :-) 

I actually think many of the projects I have worked on were like this - It didn't make me redundant - In fact I have never been so busy! I spent a lot more of my time on defining acceptance criteria, Defining automation Scenarios, writing automation fixtures, Working with the devs on their unit tests, Generally finding ways to prevent the issues from occurring in the first place. In some ways the vision of the room full of testers busily finding fault with a release is redundant. But I think that has been going the way of the dodo fast with the uptake of agile.

However it seems that people are not sure what to do with us (QA) these days, Its as if the quality of a tester is ranked by the number of bugs they find. I would argue that if you are doing your job right the answer should be none!

On a big bang project with epic specs and reduced testing time I remember getting into trouble when I found bugs. "You keep raising issues like this we'll never go live!" my standard defence was always "I didn't put the bug there - I just found it" These days I don't believe that is sufficient. I think any bug I find is a failing of a process I am an integral part of. It is a failure of mine to get the right acceptance criteria, Or a failing of the communication channels between myself and the devs to ensure we are building the right thing, Or a failing in the automation scenarios we designed that allows some of the common edge cases to be passed incorrectly. These are just some examples but where you compare this to the old Us and Them (test and dev) mentality that used to exist I much prefer driving the quality of the process and application from the front where I can make an impact rather than from the end where all I can really do is find fault.

So does all this mean I never look at an application in a test environment anymore? - Not a bit of it - It does however mean I try to never spend hours going through a costly tedious boring and above all expensive regression test suite. I spend my time exploring the application and assessing the strange edges where it may fail - Its much more fun and a lot faster to identify critical issues. and I think I started this post with a quote that says fast is better.

Don't squeeze your application out of a tube 


November 24 2010

Will we ever have the kahunies to publish or be damned!!


"I shared a vagrant optimism that some of us were making real progress, that we had taken an honest road, and that the best of us would inevitably make it over the top. At the same time, I shared a dark suspicion that the life we were leading was a lost cause, that we were all actors, kidding ourselves along on a senseless odyssey. It was the tension between these two poles - a restless idealism on one hand and a sense of impending doom on the other - that kept me going."

Hunter S Thompson - The rum diary

Publish or be damned is one of those scientific mantras you see sometimes, I believe it stems from needing to get your research out first before some other lab trumps you to it thus consigning you to an interesting foot note on the bibliography of a vegan honors students thesis.

Not a pleasant ending to a life's work.

In software development and testing especially it tends to be more publish and be damned. There is always a get out clause, a reason to not go live with something. In Agile there is this theory of producing potentially shippable code at the end of each sprint. What does that even mean?? I wouldn't be happy if I went to get my car from the mechanic and he told me I have potentially working brakes for example.

Now there are loads of places that will help define what "potentially shippable" is supposed to mean but I suspect that for most people it quickly translates into a BIG POTENTIALLY - and a very small shippable.

Is it shippable as soon as we fix that bug? or is it shippable as soon as we finish these five stories? Or more likely its shippable after an 8 week stabilization phase :-) so it was never potentially shippable at all.

That got me thinking, How would you change your work practices if your customer said to the team "You know what - I like this agile lark - Here are they keys to the live production environment and after every sprint review I want it in live within the hour!"

Can you imagine! A customer that put every sprints work directly into live!

I would probably react with total terror at first but then I started thinking about how the team and the process would change. I am willing to bet for a start that our velocity would half over night as various team members started thinking - "well I cant put that off, This is going live next week!". I would probably go on a one man uber crusade to automate every facet of the acceptance criteria before any of the code was written in case it came to me late and went live before I ever saw it!!

And then I thought "Why the hell am I not doing that now??"

I had a sudden image of a testers perfect world - Its a bad sprint, the developers are running late. Very little has made it through to be tested let alone marked as done and we have 5 stories in progress. Its 3 hours before sprint review and everything gets checked in all at once! - Normally that would be a nightmare scenario!!

[Scrum Master] "So what did we finish to demo?"

[Tester] "Did all the builds pass?"

[Scrum Master]  "Yes"

[Tester] "Demo the lot! Its done!"


Older Posts