John Scarborough, vp of quality engineering at MindTree considers that quality, cost, risk and schedule are all critical factors in making decisions in software production, and looks at software production from the standpoint of testing only.
Without measurement, how will you know whether today's version of the software under test (SUT) is an improvement over yesterday's? This is the argument for quantification and qualification in software testing and we include qualification because some attributes of software quality resist quantification.
Saying that an interface is intuitive, for example is touchy-feely, but if your customer feels that your interface is not intuitive, they don't need a quantitative reason for rejecting your deliverable. So, for completeness, we should strive to track both qualitative and quantitative assessment of software quality.
Tracking attributes
What do you test? Everything? How many attributes can be tracked when testing? A dozen? A hundred? Two hundred? Is more better than less? We are limited by time, money and people. To get the most from our testing, we need to select our targets, our points of focus, very carefully.
At the very least our software should initialise and execute reliably; it should not abort unexpectedly or lose data and it should comply with these highest priority criteria in all supported environments. But what does the software do? Why do we want it, or why do we want others to want it? That is the business case, the set of business objectives that drive us to create our software.
Business case
The biggest reason for failures in software testing is lack of attention to the business case. If you don't know what matters, you won't know when you're done, or whether it's good enough. Once you know (not guess at) your business objectives, you can set your quality objectives.
If you need to support ten thousand concurrent users on your server, and you calculate on the basis of similar systems' logs that each user may in one instant execute 10 transactions per second, you have at least one quality objective.
Or maybe you are targeting a worldwide market, in which case you need to support DBCS as well as ASCII. You design your tests so that their results are delivered in a form that is relevant to confirmation of your set of selected quality objectives.
All of this is about measuring your return on investment (ROI) as Testing is an investment. Just as you don't throw your money away, you shouldn't throw testing away either. One principle of efficient test design is to ensure that, if you are testing multiple instances of an equivalence class, you have reason to suspect that they represent different implementations; if not, testing multiple instances is redundant - so you are throwing away your testing.
Again, why spend a month writing special assemblies in C# to enable the automation of a handful of timing-dependent test cases when manual testing of the same test cases can be completed in an hour? It doesn't save time or money. Maybe the test cases have to be executed at machine speed to replicate its use in the field - or in other words, you need to align your investment in testing with the quality objectives, which should support the business case.
How will you know whether a given method or practice gives you a higher return on your investment than another? You establish that through measurement. Measurement determines whether the quality of the SUT is improving or declining; it also enables comparison (and improvement) of means. This is essential as you won't see process improvement without both.
An organisation at CMMI Level 4 begins to enter this culture; at Level 5, this culture is driving the organisation. You measure change in product quality; and you measure your ability to effect deliberate changes in selected quality attributes.
Seeing the benefits
How long does it take to see benefits from improving ROI? Not long at all, really. Consider this scenario
It is Monday morning and you, the Product Manager, are conducting the Monday morning weekly status meeting. It's early in the SDLC, the development team is busy implementing features. Your QA Manager reports that, based on the Quality/Cost /Time/Risk tradeoff model derived from earlier releases' data, her team projects that the product release date will slip by about 6 weeks, based on an analysis of the data she projects on the whiteboard:
• Development schedule slippage in several components
• High defect density rate in 2 high priority areas
• Usability analysis shows low click-through efficiency in basic tasks
• Code profiling indicates need for re-factoring of 20+ APIs
Do you want to run a week more and see if the development team can catch up as they claim they can? Do you pull back on some features and focus the team on stabilising the product? Do you want to increase your spend (invest more) to bring the schedule back on track?
Whatever you decide, you can bet you are grateful to your QA organisation for providing data you need to help you make critical decisions on a continuous basis
Today, this scenario is still rare. A more likely outcome is that, without the attention to ROI, the product group will push the next release milestone further and further out as test engineers and developers stretch their work weeks to 70 or 80 hours, features are lopped off, and senior managers yell louder at line managers. In that darker version, everyone works in a reactive state of mind and no-one has time to improve processes. Decisions are made by gut feel only, rather than through the analysis of metrics that are aligned with delivery criteria. The result is delivering buggy software far past the intended delivery date.
This is what we mean by driving ROI culture in testing. ROI is about cost, but not cost alone. ROI should be viewed as a function of four attributes - cost, quality, risk, and time (schedule in the above example). A testing team that has thoroughly assimilated ROI culture can establish a balanced scorecard, simultaneously minimising costs, mitigating risks, reducing time to market, and managing quality.
So how do you drive the ROI culture in a quality organisation?
Before going deeper into this, a word of caution: achieving improvements in all four attributes in one project does not mean that you have assimilated ROI culture. The identifying marks of success are consistency and predictability.
A testing organisation emphasises finding defects, whereas a quality organisation emphasises defect prevention, which drives defect discovery to earlier stages in the SDLC. Quality organisations are more likely to provide a higher ROI, because defects are less expensive when detected earlier in the SDLC. Transforming a testing organisation into a quality organisation can be seen as a 4-step journey:
Align Quality Objectives with Business Objectives
A critical failure factor for quality organisations is inattention to business objectives and the cause is tunnel vision, a focus on technology only.
Quality objectives should support business objectives. The quality organisation needs to protect this charter against attempts by development, marketing and sales teams to enlist test engineers in the performance of work toward fulfillment of their own charters but at the cost of completing the quality organisation's plan.
The illustration below shows the organisational structure of a team that has been transformed from a developer-led testing team to a quality-focused quality organisation:
Key steps in the alignment include:
• The testing team reports to a senior executive other than the VP of Development.
• The organising principles for each project are the business objectives for that project. Quality criteria, test strategy, and test planning are derived from them.
• Information must flow between marketing & product management teams to prioritise test coverage & efforts.
• Set success criteria to evaluate product quality during release, post release
• Budget investments in high-end testing (moving validation and verification to as early a stage as possible in the SDLC)
Align SDLC and STLC schedules
The cost of not finding software defects early in the production cycle increases dramatically in successive stages. See the table below. (For example, a Requirements issue found in the Requirements phase will take one unit of time to fix; the same issue, found in the testing phase, can take 50 such units of time.)
Defining Key Quality Metrics
While earlier phases are important, this phase compares observed behaviour to test oracles to determine if the SUT is approaching release quality. At the end of the SDLC, the output of this phase - the key quality metrics - will be the most important in the entire testing project.
Here are some metrics that may be helpful in tracking the achievement of quality objectives in the SUT, and the ability of software testing and quality engineering procedures to deliver their promised benefits:
Resist the temptation of collecting measurements just because you can collect measurements. The metrics you collect and analyse should reflect business objectives, including the satisfaction of end-customer expectations.
Unless the quality of the test effort is mature and well-managed, decisions based on collected data will be suspect. There is no viable substitute for applying formal processes to the measurement and management of the quality of both the SUT and of software testing efforts.
Not one-size-fits-all
ROI culture is not a one-size-fits-all concept. Its applicability depends on your testing objectives, process maturity, staff qualifications, capacity for internal training, etc. This article has provided broad guidelines only.
Creating a mature quality organisation that is grounded in ROI culture may require the help of STQE experts who specialise in independent testing.
Do you need to promote and invest in ROI culture for your organisation? The following questions can help you make that decision:
• Can I justify the investments that we have made in testing in the last 12 months?
• Can I justify the investments needed over the next 12 months?
• Can my organization provide better solutions than it provides today?
• Is my organisation measuring product and process quality during the SDLC, or only as a part of project retrospective analysis?
• How much does my organisation need to invest in improved testing procedures and processes? Why?
• Do I know what metrics my organisation should collect for each of the systems or applications under test?
• Do I know how to select metrics that reflect realisation of quality objectives that support the SUT's business objectives?
• Do I know how to interpret test results expressed in goal-oriented metrics so that I know when the SUT is ready to release?
John Scarborough heads the software testing and quality engineering division of MindTree's Customer Solutions Group (CSG). Growing up in Silicon Valley, he fell in love with papermaking, book restoration, digital image processing and eigenvalues. He worked for 11 years as a test manager in Microsoft's OS testing division. Disha Technologies (later acquired by Aztecsoft) then recruited him as Principal Architect in 2001; Aztecsoft (later acquired by MindTree) appointed him VP Quality Engineering in 2004.