Head to Head - Scripted vs Exploratory Testing

Posted by Andrew Baird on 18 July 2017.

When exploratory testing goes head-to-head with scripted testing which do you think works best? Test Partners show there’s a clear winner from one of our recent website testing projects.

We’re often asked to quote for testing websites using scripts that the client provides. This is convenient for clients as it enables them to easily compare prices for testing on a like-for-like basis, but it ignores supplier specialities. By contrast, Exploratory Testing is typically (and incorrectly) characterised as random, unplanned, unstructured, unrepeatable and maybe even more expensive.

So why would anyone do Exploratory Testing?

At Test Partners we have specialised in Exploratory Testing since 2001. All the criticisms mentioned above can certainly be true when it is done badly, but we have developed a unique testing methodology that addresses all those issues – it’s planned, it’s structured, it’s repeatable and we know what we have and have not done. And there’s no waste – we only produce the minimum necessary documentation.

One of the key benefits of Exploratory Testing is that while we are testing the software, we learn about it, and we use that learning, together with previous knowledge, to design and conduct new tests. After a while we can think of tests that someone writing test scripts would never think of, because they do not go through that learning process. This approach is discussed in more detail in our other blogs.

It’s Showtime!

It’s not often that you get to test the same application in different ways to see which is best, but an opportunity recently arose on a website UAT (User Acceptance Testing) project for a digital marketing agency whose end client was a consumer electronics giant. It was a redesign of a global website, with a staged rollout starting in UK – so very high profile… The test requirements had been written months before, using the TestRail tool for test case management.

By the time the system was ready for UAT, the documentation was out of date – the test cases had not been kept up to date with development. They also specified older mobile devices that now had little or no market share.

The client was reluctant to feed our information back up the chain, but daily scrum calls helped to clarify the issues and to refocus the testing on the development, rather than the old test cases. We presented usage statistics on current mobile handsets. The client still wanted scripted testing on old devices, so we offered exploratory testing in addition. They agreed and we split the team and ran scripted and exploratory testing in parallel. 

It’s a Slam Dunk!

The results were astonishing. We know exploratory testing is more efficient and effective, but this direct comparison showed by how much.

Over a measured 2-day test period, we raised 2x more bugs through exploratory testing than through scripted testing. And that doesn’t even take into account all the time previously spent writing the test scripts!

Essentially, exploratory testing enabled us to find more issues, quicker than scripted testing. Our test coverage documents, which were produced at the same time, also provided more information and were more relevant than the scripts.

Why was there such a difference?

There are many reasons, such as:

1. The TestRail reporting system relied on metrics bound up with the original test cases, but these were out of date. For instance, test cases referred to functionality and content sections which had been removed or altered, and there was no mechanism for updating the test cases.
2. The scripted testers had no reason or incentive to understand the system. The exploratory testers learnt the system faster than the scripted testers and used that learning more effectively. They understood the purpose of the system quicker, and applied their knowledge of code, webpage structure, browser interactions and similar websites to pursue issues in the higher risk areas.
3. Test scripts are only as thorough as the functional specifications they are based on, but even the most comprehensive design documentation only captures a small proportion of the actual requirements. Also, the widespread use of JavaScript libraries and refactored components means that production code contains a lot of functionality that even the developers don’t know is in there.

Test scripts are typically written early in development, usually before there is any sort of application to look at, i.e. the time when you know the least about the system. By contrast, exploratory test design is mostly done after the application is built and you can see how the features have been implemented. This is a particularly good example to highlight the benefit of exploratory testing compared with scripted testing, but it is no more than we expected based on our experience of several thousand testing projects. 

Verdict

It wasn’t even close - Exploratory Testing is the hands-down winner.

Exploratory testing can be much more effective and efficient than scripted testing, but remember also that while exploratory testers can do scripted testing, scripted testers can’t necessarily do exploratory testing. Exploratory testing is hard to do well – and impossible to do well without training and experience. Contact us at Test Partners if you would like to bring the power of Exploratory Testing to your project.