What is DevOps?
DevOps is a cross functional methodology that breaks down the barriers between development, operations and testing, helping to generate faster and higher quality code updates.
As its name suggests, DevOps overturns conventional notions of ‘development’ and ‘operations’ teams. Developers are distributed throughout business operations and this helps to address issues more rapidly. More open and regular communication leads to faster adjustments to existing software and quicker evolution of new projects.
How DevOps has Changed Testing
Prior to DevOps, it was common for development to take place in stages, with code released for review and testing only at the end of a development cycle.
DevOps works differently, focusing on prevention of error and speedy resolution through automated testing. The objective is to seek out quality issues as – or even before – they arise, rather than at the end of the process.
We haven’t seen it discussed before, but in essence, the proponents of DevOps have similar objectives to those who sought to streamline manufacturing in the last decades of the twentieth century. This involved relying on automation of processes and use of Total Quality Management (the prevention of poor quality components and processes) rather than Quality Control (testing at the end of the production line). The question is: how safely can DevOps integrate Quality Assurance practices and automated testing into the creation of software – and what is likely to get missed?
As every software project is different, there are only some ways in which Quality Assurance (software development using audited practices) can currently exist within a DevOps environment – utilising previously tested code would be an example.
Whilst Test Automation is not really Quality Assurance (as it relies on testing, albeit automated), it still marks a step forward in rapid bug detection against a prescribed set of criteria. Test Automation means that some faults can be identified automatically, shortly after a round of programming has taken place, enabling code to be updated almost immediately.
Automated testing is excellent for repetitive testing around functionality, stability and some (but not all) areas of compatibility testing.
Working as part of your DevOps team, our test consultants can review options and recommend the most appropriate and cost effective automated framework and regression testing solutions.
Whilst automated testing is valuable in providing fast and consistent results for certain tests, we’ve found that not everything can, or should, be automated.
We have always – and will continue to advocate – the essential role of exploratory testing in assuring the production of high quality software.
Whereas automation checks what is expected, exploratory testing investigates what is possible, asking questions such as “what if…?”, “how does that work?”, “what happens if I do this?” and “how might this not work…?”
In addition, some parts of testing are more experiential than functional. For example, it is hard to automate tests for unexpected transient behaviour.
Accessibility Testing and DevOps
It is also impossible to automate almost all Accessibility testing. Even the best automated accessibility testing can only verify 25% of WCAG checkpoints and even then, the results need to be assessed manually. If you want to build automated accessibility testing into a continuous integration test suite, you can only include tests that are 100% objective, and that is perhaps no more than 10% of the WCAG checkpoints.
Automation is a useful addition to DevOps, with the capability to identify bugs in carefully controlled, definable scenarios. You may find our consultants helpful in scoping the test automation part of your project.
However, we strongly recommend that exploratory testing is also factored into your DevOps test strategy. As a company, we have always said, “We test what your software can do, not what it should do.” Human beings do not always behave predictably and only exploratory testing can uncover these risks. Yet it doesn’t need to take place in a separate location or to slow down your project. If you’d like our help, our test lab can provide an efficient, flexible, scalable extension to your in-house test capability and we can also provide testers to work in house alongside your teams.
To discuss your DevOps testing requirements, please contact us.