What can go wrong with bad software testing?

Posted by Andrew Baird, on 7th June 2021, in Software Quality

A poorly constructed app that crashes your iPhone is a small inconvenience. But as software is used in almost every object with an electric current these days, it’s increasingly easy to come across software failures that are more serious. 

While good software development methodologies introduce testing in the early stages, poorly designed testing or testing that is not thorough enough can give the product team false security.

One of the obvious consequences of poor testing is financial. In January 2018, Andrew Green spent 6 hours playing a blackjack game on the BetFred mobile app, eventually winning £1.7m at odds of 7,777/1. When he tried to withdraw the money from his account, BetFred refused, because they claimed the game contained a ‘defect’. After three years of the stress and worry of bringing legal action the case got to court in 2021 and Andrew Green won the full amount plus interest.

The details of the ‘defect’ in the game, provided by Playtech to BetFred, have not been revealed as it is “too confidential to be shared”.

Betfred has apologised to Green for the delay in paying out and said it will not appeal against the judgment.

In March 2020, Playtech announced the extension of its long term partnership with Betfred for a further 4 years and the game, Frankie Dettori’s Magic Seven Blackjack is still available at BetFred and other online gaming sites.
In the end: “Man wins payout from bookies” and: “software company wins 4-year deal” are not unusual conclusions. 

However, if you were a postmaster working for the Post Office between 2000 and 2013, you could have ended up in prison because of software failures. During those 13 years, 736 postmasters were prosecuted and found guilty of theft, fraud and/or false accounting charges. Some postmasters pleaded guilty to false accounting because they were told the theft charges would be dropped. However, the Post Office then pursued those with criminal convictions, under the Proceeds of Crime Act, in order to seize their assets – bankrupting many.
The cases came about after the Post Office spent £1billion building and installing an accountancy IT system, Horizon, supplied by ICL/Fujitsu. 

Under a Freedom of Information request, the Horizon system is described: “encompasses the software, both bespoke and software packages, the computer hardware and communications equipment installed in branch and the central data centres. It includes the software used to control and monitor the systems. In addition…  [it includes] testing and training systems.”

Convictions against the postmasters were obtained on IT evidence alone. 

After class action civil litigation in 2019 against the Post Office by more than 550 postmasters, a high court judge declared the Horizon system contained “bugs, errors and defects”. The class action resulted in a £58m settlement from the Post Office to the claimants and the opening of a government-organised independent inquiry which is still investigating.

In 2020, the Criminal Cases Review Commission referred the cases of 42 convicted postmasters to the court of appeal. Tim Moloney QC, speaking for the postmasters, stated: “All had the shame and humiliation of arrest and prosecution. All experienced the enormous psychological toll associated with that process. Some saw their marriages break up, others suffered bankruptcy and some are dead, having gone to their graves with their previous convictions still extant… caused by … unfair trials stemming from the defective software and an abject failure on the part of the respondents to effectively assess, let alone effectively address, the defects in that software”. 

Software errors can destroy lives. In the Post Office case, postmasters had reported concerns with the Horizon system soon after it was implemented in 2000. Computer Weekly published an expose article in 2009. The BBC News reported in 2014 that an audit report by external investigators revealed the system omitted tracking revenue from several product lines. In 2015 Private Eye reported that an audit of Horizon, being conducted by an external company, had been cancelled one day before it was due to be published. 
Sometimes it’s not just the testing that is at fault, so much as the managers who decide what to do with the results.

On 8th April 2021, the Post Office announced the phased replacement of the Horizon system with a new cloud-based IT system. Nick Read, Post Office CEO announced: “it will after all be among the biggest, if not the biggest, IT roll-out in the country.”

Let’s hope lessons have been learned.

At Test Partners, we take testing seriously. If you want to take it seriously too, talk to us.

Article written with material sourced from theguardian.com, gamblinginsider.com, pokernewsdaily.com, sbcnews.co.uk, dailymail.co.uk, computerweekly.com, bbc.co.uk, private-eye.co.uk, postofficetrial.com