Every once in a while, we encounter individuals on our teams who have a healthy dose of skepticism about these new Agile practices they are learning. For those who tend to be scientifically-minded, or need more evidence than just a good story, I have found that it is important to give them real data to look at.
As an example, it is interesting to combine the usual Cost per Defect curve for a software project with a histogram that maps the probability or frequency of finding defects to the corresponding project phase. The result is mathematical support for both Agile as well as the value of Automation and good Agile QA practices.
Figure A below shows the typical Cost curve for a waterfall project (source: The Economics of Testing, Rice Consulting) along with an overlay showing the probability of finding defects in various phases of the project.
As can be seen, most defects are found during the testing phase, as one might expect. Using some industry standard numbers, the cost to fix per defect is about $490.
Note, however, what would happen if you are able to find defects earlier in the process.
Figure B shows the same cost curve, but the histogram representing the probability of defects found is pushed earlier in the process. The kinds of practices that might result in such a change include collaborative QA and developer testing, pairing, and automation (which helps prevent the defects from being found in the expensive tail of the curve). This doesn’t mean spending more on QA, just utilizing tighter feedback loops that help prevent defects from getting to the later phase of the project. So, even with a waterfall process, or a non-agile iterative process, one can easily see how collaborative testing and automation can reduce the cost of defects considerably, in this case down to $220 per defect.
The Agile cost curve is actually a little different, as shown in Figure C.
There is still the hockey stick effect when the software goes into production, but the rest of the cost curve would be flat, since the cost of fixing is pretty much the same from iteration to iteration. The defect frequency histogram is drastically different and is flattened and spread out across the entire life cycle of the release.
In this model, Agile practices alone, such as sprint-based functional testing and having QA and developers working off of the same requirements, are responsible for about a factor of two improvement in overall cost of defect fixing per release. Automation and good collaborative practices are responsible for another factor of two, which gets the overall cost per defect down to about $130.
Agile Accelerate 