by Don Mills
A correspondent asked, “Are any efficient test strategies (or) test case generation tools available for the generation of creative test cases that find UNIQUE bugs and accelerate time to market?” Here’s what I replied:
I suppose it depends on what you mean by unique bugs. I think perhaps you’re asking for a technique that delivers one bug per test case, one test case per bug (assuming the bugs are there to find). The fact is that any test case may stumble across bugs that it wasn’t designed to find, but it’s still possible to design test cases to look for specific bugs.
Glen Myers described such an approach in his 1979 classic, The Art of Software Testing (revised edition 2004, I think). He called it, “High-Yield Testing”, the idea being that well-designed testing can find a relatively high yield of bugs for a relatively low investmentof test cases. Studies in white-box testing by Horgan, London, and Lyu, and in black-box testing by Bender, showed that testers lacking these techniques typically required twice as many test cases to find the same quantity of bugs as testers using the techniques.
One part of high-yield testing is to use test techniques that address the risks present in the software. For example, use: boundary value testing to trap boundary bugs; equivalence partition testing to trap data class bugs; cause-effect graphs to trap combinatorial logic bugs; syntax testing to trap compositional rule bugs; statement testing to trap computational bugs; and so on.
Another part is the concept of basis set testing, which originated in the 1920s with electricians trying to find a minimal set of test cases that would isolate the location of a break in a complex circuit. Beizer calls it the “scientific test technique”, in which, from test case to test case, you change one factor at a time until all relevant factors have been tested both positively and negatively. In process-based testing, for example, you start with the shortest, simplest path from the start of the process to an exit point, then vary one decision at a time until all decision outcomes have been individually tested. This will normally produce a relatively small test set (maximum n+1 paths, where n is the number of decisions in the process) which covers all statements and decisions in the process in such a way that the failure of any test case isolates the probable location of the defect.
These techniques can be applied to both black-box testing (from specifications) and white-box testing (from code). Combining the two approaches test modelling techniques targeted at specific types of bug, and test design techniques that minimise debug time by isolating bug locations is probably as close as we can come to test cases which find UNIQUE bugs and help accelerate time-to-market.
Don Mills. 2 December 2008