The Plan

What is more important:

• Knowing and understanding the application you’re testing

 Or

• Being able to visualise how you are going to approach the testing

I suspect the answer is it depends.

From my experience, both are required for good testing. Knowing the application, its ins and outs and having the domain knowledge should help test design. Or, from the opposite point of view, not knowing how the application works will mean you won’t know anything about (any) coverage so you’re likely to miss out on bugs.

Being able to visualise your plan of attack means you know what you’re going to test, what areas of the application will be tested and which areas you can ignore (at least for the time being). This provides a structure that you can work in. Not having this can either make the testing daunting or allow the testing to go off on tangents.

This is why I like Session Based Test Management. SBTM provides enough structure and a plan to guide the testing but still allows flexibility for the tester learns more about the application and/or finds bugs, especially if the tester is new to the application and needs to get to grips with it.

I’ve found Rob Lambert and Darren MacMillan often write about visualising their testing or ideas (using mind maps, drawings, call flows). Here are their blogs to find out more:

Rob's http://thesocialtester.posterous.com/

Darren's http://www.bettertesting.co.uk/content/

thinking/drawing/dreaming

dream concepts

Bug Bash

Bug bash by Wikipedia (incase you're not aware of the term)

Pete Hawley: Rhubarb

The Cost of Automation

This cartoon was designed by Raimond Sinivee. Thanks Raimond!

Raimond had a few comments to say about the cartoon [added to post on 30-Aug-2011]

The idea behind is that, it is hard to justify automation as a testing method to managers. When it is said that we are going to automate stuff, then first answer is: “Yes!”. The answer may change when tester gives tool cost or additional working hours estimate. Manager might go: “Extra cost? I thought that we are going to save!” 

They want to see direct money benefit or labor cost savings. I think this comes from assembly line type of manufacturing. There has been lot of automation and cost savings through that. Testing is not assembly line type of work – it is engineering! So if we want to compare, then we should compare tools cost in engineering, not at assembly.

Automation could be taken as a helping method to cover new aspects of software that are hard or impossible to test manually. Usually these parts are not tested at all. Since nobody is testing newly covered aspects, then it is mostly extra expense to test them.

Tester job is to provide information about risks among other things. Tester tries to find ways to gather information about risks and how to level them. Sometimes a tester needs a tool to find out if the risk is covered. The result of executing a test could be that there is no problem. That is also valuable information if it cleared out a risk, but it is hard to actually calculate the amount of money that was saved. We could calculate the risk impact, but that also might be disputable and manipulated.

Robo-Cat

In testing we can be bombarded with the notion that test automation is best. I recently read an article saying all manual testing can be replaced with a new test automation tool – what craziness! I can’t wait for someone to say all developing and coding can be automated too! 

Test automation does have its place though. When a tester is repeating a step or task while testing, it’s worth investigating if that can be automated. Sometimes, writing quick and dirty automated scripts can save a lot of time. For example, when testing a web app, I often use a record and play back tool to log into the app, once logged in, I can carry on with manual testing. I can re-do these steps as many times as needed. Once the testing is finished, I throw away the automated script. At other times, for a large testing project, it’s good to invest in a test automation framework if you expect to run the same checks over and over again as regression tests.

Yet test automation always has flaws. One major flaw is the fact the automation can only check for what it is programmed to check. For example, if it is checking the login screen is displayed, it will check for this but won’t check whereabouts on the screen it is displayed, what colours, what font it is using or whether anything else is displayed on the screen.

Another flaw is that test automation can be brittle. A developer changing the layout of a screen may generate a false positive in the test automation. The tester will need to spend some time investigating why it failed and then update the script. All this adds to the maintenance costs.

From my experience, the biggest flaw is that test automation doesn’t find many bugs, and the bugs it finds don’t tend to be critical bugs compared to those found through manual testing. The main reason for this is that test automation is typically used for regression. The application has been manually tested in the past and most of the bugs have been identified, a test automation script has been developed to give confidence that the application has not regressed. One positive use for using automation is that regression testing is still required but can be and is boring to most testers – as it’s repeating the same tests executed previously without much hope of finding new bugs.

I recently read this article from Adam Goucher about test automation heuristics. It’s a good read :)

Watch out for next cartoon from Raimond Sinivee which has a similar theme.