An example of calculating ROI for
LoadRunner is included herein. This example could be quickly adapted for any performance
test simulation tool. One could easily translate the concepts and transform the calculations to fit tools of different classes. We begin...
This is about beans. Beans are known to be an agent of human flatulence. The term beans is also a financial term. Financial folk count beans. Financial folk like non-financial folk to count beans with them. They even want we the non-financial folk to justify our bean count. Counting beans in this context or the fear of having to count causes anxiety in many of us. Anxiety can cause acid indigestion. The remedy for flatulence is an anti-flatulence treatment. A remedy for indigestion is an antacid. If one wishes to avoid indigestion and not take an antacid, then one must count beans. If one is consuming beans and counting beans then one need only an anti-flatulence treatment.
Being asked for ROI is a fact of life in a fiscally responsible company. It is as sure as night and day. In an organization where beans are being counted to look for bean savings, all departments, groups, teams, etc. are asked to justify their existence and technologies. Asking about ROI is a both fair and reasonable question. You wish to keep your job – correct? If you have too many symptoms of flatulence at work, co-workers may wish for you to not keep your job. The stockholders wish to make money – correct? So one not only needs to get acclimated to this reality; one must know how to provide ROI. The people asking for ROI are justified in doing so. Someone needs to be responsible for spending – correct? So these people are doing their job and they need be educated with respect to a given department and its costs; more specifically for this article – tool and/or process/practice ROI. The bean counters are not stupid. They are skilled at their job. They are also very aware of and keen on waste. "Why did you buy this tool if it is collecting dust?" That is one reason they have questions. If they assumed answers to their questions rather than ask you, they might be making decisions for you that you might not appreciate. "We will not give you budget to purchase this tool or renew the maintenance on that tool." "You will have to dismiss Timmy Tester." "Your team will not be getting pay raises this year." "There is not funding for your bonus pool." "There is no money for you to take your team to a B Spears concert." (<- Thank God for that!) So, by their questions they are educating you with respect to your own fiscal responsibility. Do not get me wrong for I am not saying this is not painful. However, I can assure you that the pain is mild in comparison to the potential pain of not doing the necessary legwork.
Here is an experiment for you to demonstrate bean-sensitivity. Insist to your wife that you need to buy that US$10,000 video cam and a US$1.8M Bugatti. Prior to doing this you may want to consume a large amount of beans, or you will want to be dressed in armor as she may transform ordinary household objects into missiles and refine her brain-based targeting firmware. She will want you to prove that they are necessary to your household and/or family, or she may decide to not answer and begin a hunt for a lower-budget spouse.
For purposes of the examples herein and to frame this up a bit, let us establish a difference between QA and
Testing. Traditional QA consists of defect prevention-type processes and practices. Traditional "testing" or QC as some may dub it consists of defect-detection activities. Sometimes the two overlap. But that is not the point of concern in this blog entry. It is always a source of seemingly infinite debate for another topic – a topic which does not address tool-ROI concerns here and now. So let us stick to testing, as many tend to know it – the good old hands-on stuff or testing, with tools which replace some of the good old hands-on stuff. Let us focus on a specific area for example purposes. Let us look in on a performance testing team. Let us say this team has some fulltime performance test techies and they use LoadRunner. In this case we may be talking an annual budget that exceeds the budget of many towns in the USA and possibly some small countries. Here is how one might drive out the tool costs and ROI.
The short list of simplistic cost parameters and items to consider:
1. You cost your company money, salary/wages, benefits, floor space, equipment, etc. Your bean counters should be able to tell you your total hourly impact on the company.
2. The things you use to do your job have a cost. In my case they consist of LoadRunner (LR), the annual LR maintenance costs, the networks, network appliances, the servers and support used to put LR to work, and – the spaces that house this stuff. These items generally make up what is known as the Total Cost of Ownership (TCO). Some of the latter do not directly play into calculating ROI in the example detailed here.
Clearly the above is not enough to arrive at an accurate or even positive ROI. What else is needed? Read on. Typically, LR is used to show that a system meets its performance objectives or detect performance issues before software goes to production. Every time issues are detected and corrected, the company just avoided a potentially huge cost burden. We of course cannot project the costs associated with discovery of these issues by the end-user and the negative effects of word-of-mouth advertising with respect to potential loss of existing or new business, and – the gas we will reap! We can however estimate the savings associated with correction prior to deployment versus repair post deployment. To further simplify the example I will omit factoring in above list item 1 (one) since it involves some complexities not appropriate for this blog entry setting. Yet the ROI detailed here does work in consonance with the latter, ultimately. Now we can zoom in on ROI as related to the tools and other items in above 2. LR simulates users. Simulated users are far less costly than real users. I will call this the Virtual User Savings. These savings are critical to calculating ROI. The other major factor of ROI in this case is the cost savings of having found performance issues prior to production deployment. I will call this Cost of Repair. I now have flatulence and must leave. Please refer to the MS-Excel file example under this link (right-mouse, open in new window). This example is stripped of all proprietary information. It is set up for basic calculations. Simply plug in some values and observe. Can you use this? How would you change it? You are free to go and consume or count beans. Thank you!
