The devil of regression testing

 

"No sympathy for the devil; keep that in mind. Buy the ticket, take the ride...and if it occasionally gets a little heavier than what you had in mind, well...maybe chalk it off to forced conscious expansion"

 Hunter S. Thompson (Fear and Loathing in Las Vegas)

On our mailing list where we discuss all sorts of important development issues and organise pub trips, a mate of mine came up with an interesting problem to do with testing - It went a bit like this ..

“I've just been talking to someone who I has attempted agile without any QA or automated tests at all. Unsurprisingly they've seen an increase in regression issues and are now looking to retrofit done automated testing.”

As the resident tester it got me thinking. What is the cost of regression testing in an agile project ? - well the cost in my opinion is actually 0, zip, nadda, snake eyes !!

Right about now everyone who has faced the cost of regression testing is thinking I am smoking crack? or maybe hill billy drunk on the 4th of july? - I’m not and here is why.

Lets say you have a project you estimated at 30 points - whatever - you figure as you are going along you can do roughly 5 points a week including getting it manually QA’d in a test environment and showed with great fan fare to your product owner  - Happy days!! - you will be out of there cashing the check in 6 weeks... 

or will you.. 

What was your ACTUAL velocity??

Easy I hear you all say - 5 points!! 

The problem is, it wasn’t really - The 5 points a week was stuffed with tech debit, Now I’m not talking the stupid tech debit i’ve cited before where you just sweep all the hard stuff into the tech debit backlog to temporarily inflate your velocity (and we all know how thats going to end) I’m talking about serious tech debit that wont bite you for a month or so. The question is actually what did you DO for that 5 points? 

Write code that solved the problem?

Or write code that solved the problem with unit tests, integration tests, platform tests, automated UI tests that prove every build and deployment you haven’t broken anything??

In most cases its the former  - And here is where logically (though not financially I’m sorry to say) it doesn’t cost. What is actually happening is your velocity gets corrected. You weren’t doing 5 points, you were actually doing maybe 3 but incurring the tech debit of poorly tested code allowed the team to inflate their velocity to an unrealistic 5. The over all correction theoretically will be 4 weeks in this case - but anyone who has tried to go back and retrofit tests to poorly tested code will know its harder than writing them at the time. so in this case it will take longer. 

I guess if you can guarantee some other poor stooge will have to deal with it later you *could* give them crappily tested code to work with but once the smell has pervaded the building you wont be working back there any time soon and your rep wont be the greatest.

 

by Malcolm
July 5 2011

Be The Headlights Not The Handbrake

 

Some people will tell you that slow is good – but I’m here to tell you that fast is better. I’ve always believed this, in spite of the trouble it’s caused me. Being shot out of a cannon will always be better than being squeezed out of a tube. That is why God made fast motorcycles, Bubba…"

— Hunter S. Thompson (Kingdom of Fear: Loathsome Secrets of a Star-crossed Child in the Final Days of the American Century)

I haven't been at ThoughtWorks long but its been a happy couple of weeks - There is good conversations on a wide number of lists about all sorts of crazy stuff.

One that really grabbed my interest was a whole fiery chain about test automation and much opinion. Again someone raised the point of view that testers are like tigers (In the future survival stakes rather than the scary stakes) and will be extinct soon. This is something I have been yearning after for years - Imagine a world where the development practices employed by development teams are so quality driven we didn't need to test at all after they were done :-) 

I actually think many of the projects I have worked on were like this - It didn't make me redundant - In fact I have never been so busy! I spent a lot more of my time on defining acceptance criteria, Defining automation Scenarios, writing automation fixtures, Working with the devs on their unit tests, Generally finding ways to prevent the issues from occurring in the first place. In some ways the vision of the room full of testers busily finding fault with a release is redundant. But I think that has been going the way of the dodo fast with the uptake of agile.

However it seems that people are not sure what to do with us (QA) these days, Its as if the quality of a tester is ranked by the number of bugs they find. I would argue that if you are doing your job right the answer should be none!

On a big bang project with epic specs and reduced testing time I remember getting into trouble when I found bugs. "You keep raising issues like this we'll never go live!" my standard defence was always "I didn't put the bug there - I just found it" These days I don't believe that is sufficient. I think any bug I find is a failing of a process I am an integral part of. It is a failure of mine to get the right acceptance criteria, Or a failing of the communication channels between myself and the devs to ensure we are building the right thing, Or a failing in the automation scenarios we designed that allows some of the common edge cases to be passed incorrectly. These are just some examples but where you compare this to the old Us and Them (test and dev) mentality that used to exist I much prefer driving the quality of the process and application from the front where I can make an impact rather than from the end where all I can really do is find fault.

So does all this mean I never look at an application in a test environment anymore? - Not a bit of it - It does however mean I try to never spend hours going through a costly tedious boring and above all expensive regression test suite. I spend my time exploring the application and assessing the strange edges where it may fail - Its much more fun and a lot faster to identify critical issues. and I think I started this post with a quote that says fast is better.

Don't squeeze your application out of a tube 

 

by Malcolm
November 25 2010

Sometimes bed is not an option

 

"What passed for society was a loud, giddy whirl of thieves and pretentious hustlers, a dull sideshow full of quacks and clowns and philistines with gimp mentalities."

Hunter S Thompson - The rum diaries

How many times do you hear that to do the right thing is “Not an option” because of your client engagement.

I sometimes suspect this is much more driven by fear of the Consultants engagement than by the actual need – Sometimes being unpopular is the only option. Let me illustrate a fairly common scenario for you.

You have a piece of work where the client hasn’t actually agreed to the tech and visual design yet, this piece of work may be quite important and needs to be done very soon and as a team you might really want to get it out of the way? – The correct response is to NOT do it until we have a consensus on what the functionality should do – Hold off it – drop it – The product owner may be unhappy in the short term but then it saves you generating waste willingly. The problem is you don’t actually know what they want so are you building it because;

You are arrogant enough to think you know your product owners business better than they do?

Or because you are in the “More haste less speed” camp of “How wrong can it be?” (See above)

Now this happens all the time but let’s take this situation out of its comfortable context and put it somewhere where it really shows its flaws.

You take your car to a mechanic (You are now the Product Owner and the mechanic is the team) and the mechanic states that there is some serious damage to your engine. He will likely be able to get away with re doing the head (£x) but thinks it is wise to replace the engine (£3x)

If you have a good quality late model car maybe the engine replacement makes sense

If it’s an older car you are considering replacing in a year or two most likely the head

If it is an old banger you will probably tell him to not bother and scrap it.

How do you think you are going to feel when you decide to tell him to scrap it, call him up, and he says “We got bored waiting so we put in a new engine. You owe us £1800”

I reckon you are going to be a little less than pleased.

by Malcolm
June 8 2010

Its Scrum (But)

 

"Call on God, but row away from the rocks"

Hunter S Thompson

So I ran a retrospective for a team a couple of weeks ago

Nothing exciting in that per se I thought but the most interesting thing happened at check in!

For those of you unfamiliar with this process it is a time at the very start where you go round the room and get everyone to say a few words. The thing they talk about is usually unimportant, what is important is that everyone speaks; it helps to engage everyone and empower them to talk further during the retrospective. So I usually ask something straight forward about the last sprint, how it made you feel etc.

Having not run a retrospective for this team before I figured I’d ask them what they hoped to gain during the retrospective and I have to confess I was surprised by some of the answers. One of the team members said “What do I hope to gain? Nothing!” which was a little disappointing but by far the most interesting was one of the tech guys. “I want to get pink cards!!” 

Now as a short explanation the pink cards are the items they keep their retrospective actions on, these go on the task board as reminders of the areas we want to improve upon. So a sensible answer you would think?

No I don’t think it is.

The telling point is he was focusing on the Artifact and not the purpose, it could just as easily be couched as “I want somewhere to park what’s going wrong so I can highlight that I have acknowledged it and no longer have to do anything about it”

Interestingly we came up with, I think, 4 pink cards? – at the next retrospective the self same team when answering “How did we perform against our last retrospective actions” are going to have to answer “We didn’t!”

 

by Malcolm
June 2 2010
Older Posts