The devil of regression testing

by Malcolm 5. July 2011 05:46

 

"No sympathy for the devil; keep that in mind. Buy the ticket, take the ride...and if it occasionally gets a little heavier than what you had in mind, well...maybe chalk it off to forced conscious expansion"

 Hunter S. Thompson (Fear and Loathing in Las Vegas)

On our mailing list where we discuss all sorts of important development issues and organise pub trips, a mate of mine came up with an interesting problem to do with testing - It went a bit like this ..

“I've just been talking to someone who I has attempted agile without any QA or automated tests at all. Unsurprisingly they've seen an increase in regression issues and are now looking to retrofit done automated testing.”

As the resident tester it got me thinking. What is the cost of regression testing in an agile project ? - well the cost in my opinion is actually 0, zip, nadda, snake eyes !!

Right about now everyone who has faced the cost of regression testing is thinking I am smoking crack? or maybe hill billy drunk on the 4th of july? - I’m not and here is why.

Lets say you have a project you estimated at 30 points - whatever - you figure as you are going along you can do roughly 5 points a week including getting it manually QA’d in a test environment and showed with great fan fare to your product owner  - Happy days!! - you will be out of there cashing the check in 6 weeks... 

or will you.. 

What was your ACTUAL velocity??

Easy I hear you all say - 5 points!! 

The problem is, it wasn’t really - The 5 points a week was stuffed with tech debit, Now I’m not talking the stupid tech debit i’ve cited before where you just sweep all the hard stuff into the tech debit backlog to temporarily inflate your velocity (and we all know how thats going to end) I’m talking about serious tech debit that wont bite you for a month or so. The question is actually what did you DO for that 5 points? 

Write code that solved the problem?

Or write code that solved the problem with unit tests, integration tests, platform tests, automated UI tests that prove every build and deployment you haven’t broken anything??

In most cases its the former  - And here is where logically (though not financially I’m sorry to say) it doesn’t cost. What is actually happening is your velocity gets corrected. You weren’t doing 5 points, you were actually doing maybe 3 but incurring the tech debit of poorly tested code allowed the team to inflate their velocity to an unrealistic 5. The over all correction theoretically will be 4 weeks in this case - but anyone who has tried to go back and retrofit tests to poorly tested code will know its harder than writing them at the time. so in this case it will take longer. 

I guess if you can guarantee some other poor stooge will have to deal with it later you *could* give them crappily tested code to work with but once the smell has pervaded the building you wont be working back there any time soon and your rep wont be the greatest.

 

Will we ever have the kahunies to publish or be damned!!

by Malcolm 12. March 2010 23:31

 

"I shared a vagrant optimism that some of us were making real progress, that we had taken an honest road, and that the best of us would inevitably make it over the top. At the same time, I shared a dark suspicion that the life we were leading was a lost cause, that we were all actors, kidding ourselves along on a senseless odyssey. It was the tension between these two poles - a restless idealism on one hand and a sense of impending doom on the other - that kept me going."

Hunter S Thompson - The rum diary

Publish or be damned is one of those scientific mantras you see sometimes, I believe it stems from needing to get your research out first before some other lab trumps you to it thus consigning you to an interesting foot note on the bibliography of a vegan honors students thesis.

Not a pleasant ending to a life's work.

In software development and testing especially it tends to be more publish and be damned. There is always a get out clause, a reason to not go live with something. In Agile there is this theory of producing potentially shippable code at the end of each sprint. What does that even mean?? I wouldn't be happy if I went to get my car from the mechanic and he told me I have potentially working brakes for example.

Now there are loads of places that will help define what "potentially shippable" is supposed to mean but I suspect that for most people it quickly translates into a BIG POTENTIALLY - and a very small shippable.

Is it shippable as soon as we fix that bug? or is it shippable as soon as we finish these five stories? Or more likely its shippable after an 8 week stabilization phase :-) so it was never potentially shippable at all.

That got me thinking, How would you change your work practices if your customer said to the team "You know what - I like this agile lark - Here are they keys to the live production environment and after every sprint review I want it in live within the hour!"

Can you imagine! A customer that put every sprints work directly into live!

I would probably react with total terror at first but then I started thinking about how the team and the process would change. I am willing to bet for a start that our velocity would half over night as various team members started thinking - "well I cant put that off, This is going live next week!". I would probably go on a one man uber crusade to automate every facet of the acceptance criteria before any of the code was written in case it came to me late and went live before I ever saw it!!

And then I thought "Why the hell am I not doing that now??"

I had a sudden image of a testers perfect world - Its a bad sprint, the developers are running late. Very little has made it through to be tested let alone marked as done and we have 5 stories in progress. Its 3 hours before sprint review and everything gets checked in all at once! - Normally that would be a nightmare scenario!!

[Scrum Master] "So what did we finish to demo?"

[Tester] "Did all the builds pass?"

[Scrum Master]  "Yes"

[Tester] "Demo the lot! Its done!"