Why Scripted Testing Sucks

Posted by Steve Green on 30 March 2012.

Even today, the vast majority of software testers are following the decades-old practice of writing test scripts, which is still being taught in tester training courses. It never worked, it never will, and it's part of the reason why there is so much bad software in the world. You could not invent a more inefficient or ineffective approach to testing.

The traditional approach to testing is based on the following sequence of events:

• All the stakeholders' requirements are gathered and documented. The resulting design documentation is deemed to be complete and accurate.
• An experienced test analyst identifies every test case that is required and writes a test script for it. In a waterfall development project, this is typically done while the software is being written.
• A traceability matrix is created that maps every requirement to a test.
• Junior testers begin to execute the test scripts when the software becomes available.
• The test manager gathers metrics to report progress to the stakeholders.
• Testing ends when all the test scripts pass or other exit criteria are met e.g. the CEO says "launch it on Monday or you're fired".
• The software is launched.
• It crashes and burns.
• A post mortem recommends that next time there should be more analysis, more scripts and more metrics i.e. more of everything that didn't work the first time (or any other time).

The following are just some of the reasons this approach doesn't work:

1. The documented requirements are never complete. In fact they are never even close to being complete. Developers are used to this, and they use their experience to fill in the gaps. And they sometimes just guess the bits that aren't specified.
2. 100% test coverage of the documented requirements will only equate to a small percentage of everything the system can do (and users can do to it). Far from guaranteeing full test coverage (whatever that means), scripted tests guarantee that you will NOT test everything you have built. Only exploratory testing enables you to find and test things that are not specified in the design documentation.
3. The documented use cases or user journeys only specify the most linear paths through the system, but people rarely behave like that. Actual user behaviour is far more complex.
4. Without knowing how a feature will be implemented, it is impossible to write all the tests that are necessary. However, this is precisely what happens because the code is often not yet written when the test scripts are being created.
5. The most experienced people on the project (the analysts and managers) never get their hands on the code.
6. The least experienced, least motivated people on the project spend the most time using the software but may never learn what it does. They may be incapable of spotting an obvious bug if it is not covered by the scripts they are following.
7. The test scripts are written at the time when the least is known about the system. This contrasts with exploratory testing, where tests are designed throughout the testing phase.
8. While the test scripts are being executed, there is usually no mechanism for learning and using the new information to create new tests. I have even spoken to testers who were forbidden from reporting bugs that did not derive from scripted tests. By contrast, rapid cycles of learning and test design are a key feature of exploratory testing.
9. Changes to the system design can result in the need for extensive rewriting of test scripts that may not have even been executed once. Worse-still, de-scoping of features means that some test scripts may never be used at all. This wastage does not occur with exploratory testing.
10. All metrics relating to software testing are bogus. They all derive from counting things that are not equivalent, such as test scripts or bugs. Second-order metrics such as 'defect escape ratios' are absurd - one meaningless number divided by another meaningless number. Managers may like numbers, but narrative is the only way to meaningfully convey the test coverage.

So what is the alternative?

It will come as no surprise that exploratory testing can address most of these issues.

• It is not constrained by what the system is supposed to do, so you can explore what it actually does. Scripted testing is just a subset of what can be achieved through exploratory testing.
• As you are testing, you learn things about the system that can be used to create increasingly-powerful tests. There often comes a time when you can predict that a bug will occur with such certainty that it is barely worth running the test to prove it (but you still do).
• You learn to embrace the infinite nature of testing instead of pretending it is finite.
• It is adaptable in response to changes in system design or if parts of the system are broken or incomplete.

And not only does it find bugs that scripted testing cannot find, it does all this in a fraction of the time. Why would anyone do testing any other way?