By Paul Fratellone, Programme Director Quality & Test Consulting Unit at MindTree
A couple of decades ago technology investments were classified as a cost of doing business, a cost- centre-centric view. In today’s world, the new paradigm is a 180-degree turn-around, with technology being a profit centre, adding business value through the delivery of software and software products to the market place.
It is indeed an integral part of a company’s competitive advantage and differentiator. Before we go any further, we need to baseline the reasons why we measure, analyse, compare information and ultimately improve. The reasons need to be anchored by the business goals/objectives, (qualified and/or quantified); if not then I question the value of collecting information in the first place.
What is return on investment?
The ROI calculation in its simplest form denotes the amount of return expressed as a percentage of investment. Going through an ROI calculation enables everyone to understand the total investment needed for this effort. Hardware, software, tools, licensing, and training are examples of what needs to be included to establish investment cost. Incremental costs over time need to be added to the total cost. For example, it is important to:
- establish reliable data points and statistics of staff productivity
- assess the value of a benefit by comparing the new to the old
- validate the fully loaded rate for labour expense
- talk to the finance department about the discount rate, since the actions listed above do not take into account the cost of using money over time
- remember amortization of fixed costs can also affect ROI if calculated over several quarters.
The internal rate of return (IRR) is a rate used in capital budgeting to measure and compare the profitability of investments. It is also called the discounted cash flow rate of return (ROR). Cost benefit analysis (C/BA) provides quantitative comparison of costs and benefits across two or more solutions and an accounting of when costs will be incurred and when benefits will accrue; “a time-based” cash flow analysis. C/BA often discusses parameters for financial value of soft benefits (mindshare, brand recognition, market entry). Seek guidance from your finance department: they really can help technology professionals in this area.
Justifying your investments in testing
Investments require the establishment of sufficient business rationale to determine the value that each is going to bring to the business to justify the required expenditures and resource allocations.
Call it a business case – business justification; we anchor our investments in testing to meet quality objectives in support of business goals. We measure success (meeting specifications) through testing/validation. Your ability to articulate the expected benefits and associated cost of quality through investments in testing will be decided by the merits of your business case/rationale. These goals or quality dimensions can be qualified (reliable, scalable, accuracy, performance, etc) and/or quantified (say by a 5% increase in sales revenue or market share) but regardless, you still need to identify the value and benefits the investments (testing) will bring and have contributed to the business.
In the case where the product owner needs to ensure that the product is highly reliable and stable, there needs to be an association between the quality dimensions and the test effort to deliver it. Knowing it is an impossible task to test every permutation within a code base, testers still must be able to quantify the effort of the test investment to assure reliability and stability.
First, a qualifier of highly reliability will agree a quantifiable number of test cases to be executed. Once you have this estimation you can see the effort and the associated duration it will take to deliver reliability. In the unlikely event that testing time is reduced, you now have quantifiable data showing that the effort needed to deliver reliability is in jeopardy due to the time constraint. The discussion about what to do next is now fairly simple. The tester might say, in order to give you 'highly reliable' I need 'n' days. Due to delays we have n-five days. There are no more available bodies (lack of budget) to complete the tests in the timeframe. Do you want to extend the date or accept the risk of less testing?
Of course, testing will focus on the most important/business-critical tests first and execute in a descending priority order, but the options are clear. The earlier you get your effort estimation documented and refined over the course of the test cycles, the better it will be for the project success and it will reduce the perception that testing is the bottleneck.
A qualitative goal, such as “the product has to be more intuitive to the customer”, is much harder to measure, but in the end, even this intuitive goal can be measured quantitatively; we can quantify/measure website traffic, usage, duration on site, increased sales/products sold, features purchased and increased advertising revenues.
The marketing department is likely to have done some studies on qualitative goals. They would be a good source for the data. Some general guidelines include:
- Understand the business rationale, quality goals, and success criteria – discuss this with your business owners and stakeholders but do not forget about marketing and finance to ensure a full understanding
- Know what you want from this initiative and how it will contribute to fulfilling your success criteria
- Quantify as many of your objectives/goals as possible
- Identify all the processes in support of the goals and business rational
- Set thresholds and targets
- Use different ROI models for different purposes for the same project
- Remember numbers can say whatever you want; don’t use ROI calculations to confirm a bad investment.
Costs of quality
Naturally, establishing costs is necessary for any ROI calculation. Quantitatively establishing the cause and effect (failures and costs) of processes is what QA management needs to articulate to business leaders. The Cost of Quality Concept (CoQ) developed by J M Juran (Quality Handbook) identifies costs as preventative and detective or appraisable. We invest in these costs to reduce the occurrence of failures. The cost of the failures is greater than the cost to invest in preventative and detective controls, hence your ROI.
What are the right metrics?
Metrics need to be transparent and provide understandable information to management in order to stay the course or to make course corrections. Peter Drucker said: “If it cannot be measured, it cannot be managed.”
When we think in terms of increasing productivity or efficiency without a baseline or benchmark, you do not know if your changes have actually improved performance.
Collecting data and reporting statistics has value, but it is negated if there are no actions taken to improve what we have been measuring. When we are faced with financial goals, quantitative measurements normally are applied and therefore much easier to know how we’re doing against plan. A qualitative goal, such as the product has to be more intuitive to the customer, is much harder to measure, but in the end even this intuitive goal can be measured quantitatively.
The more intuitive a website the more functionality will be used/adopted, therefore more products are sold which increases sales revenue, advertising revenues, market share etc – indeed it can all be measured.
Some examples of metric categories and measurements
Some general guidelines to follow:
- Understand the business rationale, quality goals, and success criteria
- Know what you want from this initiative and how it will contribute to fulfilling your success criteria
- Quantify as many of your objectives/goals as possible
- Identify all the processes in support of the goals and business rational
- Try to standardise the collection and analysis of data
- Harden the integrity and baseline the data for comparison/measurement
- Set thresholds and targets
- Monitor the data throughout the delivery and adjust/course correct as needed
- Use different ROI models for different purposes for the same project
- Numbers can say whatever you want; don’t use ROI calculations to confirm a bad investment.
Business and IT leaders faced with goals to reduce costs and increase profits need to figure out how to measure the delivery of software products. When you hear the battle cry of your business and technology leadership we need faster, better, cheaper delivery of our software products and we must be innovative while doing so, I am sure you get that tingling sensation throughout your body.
It truly is a very tough task, but it can be done, but not all at once. Ask yourself if you understand how this request aligns with the company’s goals and objectives.
Just as important is to understand and help your team understand the connection from quality to testing. If sales forecasts require that the product is available in the market place by a stated date in order to maximise the window of opportunity for revenue, then time to market is the most important criteria.
Those activities that will accelerate delivery time will be sought out by the team and finding the balance where increased speed does not create a quality issue is how the team will proceed and make trade-offs accordingly.
At some point the team will need to assess when speed to market causes lower quality that could impact this and possibly other objectives. Balance across objectives needs to be considered before the team proceeds with course corrections. Making sure the business owner is clear about the test investment in relation to the quality objectives will reduce the chance of surprises when the effort (time, duration, cost) is needed for the level of quality required.
Paul Fratellone is programme director in the test consulting in the testing business unit of MindTree. He has worked in information technology for 25 years, focused on testing, compliance and quality management.
By Paul Fratellone, Programme Director Quality & Test Consulting Unit at MindTree