Why do we write a test plan | testingReflections.com
Skip navigation.

Why do we write a test plan

functional testing | general software testing | project management
This is the most tricky QA question I’ve ever run into during my almost 10 years of testing experience participating the most different projects. I’ve been considering blogging about test plans for months. I’ve seen a lot of templates, red articles, and even purchased a testing book that I refer to as test plan understanding handbook and discussed it . Still I believe that test plan is the most variable or even undefined artifact in whole IT industry. I have some ideas regarding possible contributing factors for that, but it is not my message. My message is how to optimize the artifact usage. And this is one of a few blogs that I pretend to write for wide context – for wide list situations I’ve been into. Sorry that this is so large – it is actually an article size blog, but I believe that the topic deserves that.


Summary
The issue is simple we should neither make test plan huge (so that is become product of fear), nor meaningless (so that it eventually become a product of ignorance).
Templates are good if continually used in your company as it makes common interface for information: reader know where to look for certain information even in a huge document that He wants to know at this moment, because different stakeholders want different info in different moments. However templates distract author’s focus from main goal of a plan - goal that differs from case to case unlike the template.
I have a better idea (by the way - I’ve applied that idea to my templates) – you simply (well this is quite a challenge to do actually) must understand how clear you need to make the why and the how you are going to do what you plan to do. Make it as clear as it need to be: no more, no less. Remember that calendar plan (aka microsoft project plan) is enough to show what and when you plan to do, so if you write anything more your concern is why and how. Not only how do you do what you planned to do, but also how do you know what you know, how did you plan what you plan. Even if your plan will look like you try to tell your PM what the project is about – you are actually telling him how do you perceive it and he could find issues in your perception.

Preface 0: SCRUM planning meetings
I’ve recently been on SCRUM training. That’s management methodology based on agile ideas. You know what’s exiting about the methodology: they has formal planning meetings that longs a whole day (8 hours) once per month, all team (7+/-2 persons) participating. And the product owner has done a pretty good preparation work for that. I’ve calculated that is almost equal to a half-time person working on project. They allocate half a person for only doing planning and do you know what they plan looks like? That’s bunch of “cards” with a few words and estimated hours of work on each of them. That’s about 1 Kb of uncompressed information, which required almost 2 person-weeks effort.
Conclusion: planning may be an extremely complex, inaccurate and still time consuming process, but the written test plan does not necessarily reflect that.
On the contrary plans that describe how existing process and methodology will be applied to current project/task may be as long as all the process and methodology documents existing in a company, but does not include any real planning effort in creating such a document.

Preface 1: planning cooking VS planning business
Webster has different descriptions of a plan , but basically it is talking about method of doing something. Planning is about thinking HOW to do something. My hobby is cooking on weekends. I do typically ask my family what are they preferences and how much they want to eat. I then consider if what I will cook will be good on the next day and other aspects to plan how much should I buy. The plan may be as simple as a real number – kilos of meet to buy, but there is a huge process of getting this number. The documented plan may optionally describe the process. For example Business Plan as far as I know describe not only what is the business but also why we think it will be a successful business (include market research and so one). I will need to share the why. I don’t need to share it when planning how much meet I will buy as it is my money I’m going to waste. If you want me to cook for you and will pay the bills for food – you will want to see the why in the plan, don’t you?
So I believe I’ve described the firs big issue in writing test plans. There is a question if and how much do we want to write about WHY in the test plan. It is more or least clear that we want to write WHAT. And it is clear what does it mean to write WHEN (schedule) although it is the hardest and typically the most incorrect part of a plan :) .

Preface 2: Planning a route
I live in Riga, almost a million people city with old (soviet time) traffic infrastructure and of course got some traffic issues. Consider how that my business partner call me as say – I want to see you at 14:00 in my office (address). I’ve never been there but I have e-map to plan my trip. So I need to consider what time I start. Of course I need to plan a trip in order to be able to somehow evaluate time required to make it. However I can’t plan the jams and the closed roads (there are a lot of maintenance), but I should at lest know if I have to cross the river (bridges are almost always in jam) and stuff like that. So I need to consider HOW I will get there, that’s a part of my plan, no matter if it is written or reside in my head. If we compare to cooking example above – given that it is my hobby and I don’t use any recipe – I never plan for HOW I will cook it. I plan it on the fly. I plan my path from office to home on the fly each day as I see the traffic and remember that I need to go shopping or something.
The HOW issue include two aspects – not only it a question if we want to write it down, but even if we want even to think about it, how detailed do we want to do it?


What’s so special about it nowadays?
Dragon is dead, long live the (new) dragon. History is repeating, you know. I hope you see what I mean when say that in year 2007 the documentation era seems to be going. The long, supposedly comprehensive documents, filled according to templates that accord to standards (actually have copy-pasted them), with a long list of supposed audience and assigned reviewers. The documents that are never red through however neither by audience nor reviewers. The era of agility is coming. The era of communication, this time supported by technologies: textual information exchange (chat, e-mail) is extended with voice (skype) and even limited visuals (webinars).
The era of communication the information that each person you communicate to needs to know (instead of communicating what all of them may ever need to know) at this moment. Empowering people, relying on their skills, commitment and teamwork. That’s not a silver-bullet and there are drawbacks, for sure. I’m not as old to do a complete analyze but could name a few however: no history kept (at least not reasonable searchable), even harder to evaluate each individual performance, which reduces the motivation on a long run (who wants to be just another person on a scrum team all his life even when new those newbies are populating it...).

What so specific about test plan ?
It was more that a year since I blogged why do we write test cases . This time I write about test plans. I now realize that there are a lot in common. Still It took me extra year to decide I know enough about test plans and it took about a moth to compile what I know about it.
There are at least one reasons I know why the test plan is differently used: test plan is an item in the test case management systems I know of. The item that have actual test cases to execute attached to it. Test plan is the only mechanism to organize test execution and that execution reporting. That’s why we create a separate “test plan” for each type of testing, each cycle, each feature or even each ester. On the other hand “test plan” is sometimes (well almost always) seen by management as a central document addressing quality. You know, there are 4 factors on a project: time, cost, scope and quality. Scope is covered by requirements document, time – by project plan (schedule), cost – contract, so they only need a document for quality. And it include both quality control (testing) and the quality assurance (the quality process).
Where does we write how the source control and defect tracking systems are used by developers in bug-fixing process? How do you think if you as a tester would answer “this is related to development, it should not be part of test plan”, what would your project manager think about you.
In our company we do have a document called QA plan. Which is supposed to cover all QA/QC issues. Then we have Master Test Plan with address testing issues and we have component test plans that are the result of actual planning process.


Proving the information within a test plan
Just as test cases (see above) test plan writing purpose I divide in two groups: First is when TP is delivered to customer. Credibility is goal: credibility of everything: process, skills, resources, tools, etc. You want to copy-paste information from one project to another describing what defect tracking system is used, how priorities are set, how build process, smoke tests, regression tests and other processes happens, how do you ensure the best coverage etc. Just as with TC I don’t want to talk about this type. I want to talk about internally useful test plans. Now as we talk about internally useful test plans there are again several reasons to write them:
1) To represent result of test planning and allow review by stakeholders, developers (whose code will be tested), etc. This is the classic reason well documented as “test plans to be communication channel” in one of my favorite testing text-books (given that I don’t like text-books in general) Rick D.Craig, Stephan P.Jaskiel "Systematic Software Testing"
2) To store details of execution the process of actual planning (I strongly recommend look at this process checklist by James Bach to understand that this is quite complex one) and allow to peer review it by other testers. Peer review process assumes that someone without deeper (and actually without any) knowledge of the product under test will do a review to suggest what the general/methodology ideas author has missed. This is OK for experienced testers reviewing novice’s work, but I don’t believe in a real “peer” review of plans, as peers do not know the entire context. Also a checklist linked above may quite well replace a peer review.
3) To store for further retrieval information on planned schedule, environment, etc. I don’t like this one – I prefer to store this in test reports.
So let me further investigate case 1. Let me first say that I always see testing as a service done for development team. Not a service for customers. That’s my context. So as I write test plans I show it to my customer – development team and management (sometimes also other stakeholders) – so that they could review what type of services I am going to provide them. I write only the information that I believe they have to know. I’m not going to describe them details of test techniques/strategies – they are not testers and are hardly going to understand that. I will say that I will do my best to test Scope according to Goals. Scope and Goals are what they have to review. Also my vision of product under tests. I tell them how I understand the software under tests: it’s architecture, business purpose, testability. They could review and help me to better understand it. They review the scope and could suggest the technical ideas that I’ve missed (like suggest to test that cache is synchronized correctly as they see that I have no idea that there are a cache involved).
Summary: not only I write what I’m going to test (scope), I also give a brief overview of how I’m going to cover that scope (what are quality goals). And of course – I write WHY I believe it is what I should do. That’s an essential part of my plan.


Test plan (following template) content
Let me analyze in details most of the commonly supposed part of test plan and show you context in which that makes no sense.
Test Strategy, models, approach, techniques. Never the fist chapters but almost always supposed as a part of test plan. If anyone does not kwno what it is I could give some hints: state-transition test model (see a nice refute to this by James Bach) boundary value test technique, approach: try to imagine myself an and user and perform scenarios accordingly... Do you really think your development team want to know your strategy? Do you think they even want to know you have one? Do you know if they have development strategy and what it is? I bet you don’t. There are two typical cases: 1) the strategy does not changes a lot between test plans and this chapter is populated with copy-paste or generic statements 2) the strategy changes all the time and your testers never actually follow what’s written in plan.
Test Schedule, resources, etc. test schedule planning up-front is the most comic idea in software development. Does anyone really believe it only depends on feature completeness? When a developer say code is complete... well 99% complete it only mean there is a chance that it will work, but a good chance that most of test cases will still be blocked. And how about time for defect reporting and retesting? How about regression test cycles number of which depends on initial quality?
Don’t you know why it is still a trend to ask for industry standard developers : testers ratio? And why it is the most attacked question by testers? How could you ignore this ratio – this is actually project’s investment into supporting test engineers – ourselves? This is how project decide if we have to be paid.

Metrics, reporting. It is sometimes assumed that testing progress and quality may be measured in % of tests passed or defects open. Well, if you do – proceed, but consider the Fundamental Issues in Software Metrics.

Now maybe you could analyze yourself if your stakeholders really want to know details of other items:
Tools,
Responsibilities (persons involved), approvers, etc.
Objectives, test goals, quality goals
Defect reporting, severity guidelines, etc.
Procedural: enter/exit criteria, change management, Suspension Criteria and Resumption Requirements, etc.
Scope, Feature to and not to be tested (high level only – a lot of places of misinterpretation)


Use Cases
Use case 1: regression test plan in test case management system
The test planning consists of only one step – selecting the right test cases out of list of all of them. The test cases attached to test plan perfectly describes scope. You may want to describe logic you have used to select them in a test plan, but in best case you have described in at a higher level – when planning whole project tests, so my regression test plans in test case management system are basically only test case holders without any line ox text in it’s content. The only information is build number, environment, etc. – items that are typically stored in a separate fields.

Use Case 2: Test plan for external customers or Bypassing company level requirements
There you are in a big company or your customer is the big company requesting that you fill in template, write down every detail of test process, including such details as defect severity evaluation and listing such buzzwords as test techniques and models to be used. I suggest you to remove this obstacle and deliver THE information within a test plan. There is a nice thing called appendix. You are mostly free to put into appendices whatever you want. So I suggest to do following: create one or several appendices that describes the WHY and the HOW are you going to do what is planned. Now make references to this from al the significant places within main body and ensure that you do not provide enough details within the body – just enough to intrigue the reader to open appendix and read it.

Use Case 3: Exploratory testing plan. I’ve discussed this already

Use Case 4: New component test plan
That is the most typical case for test design. There is a new set of functionality done or under development and you (will) have to test it. I use a template for this, but there are some significant hints:
- The first chapter is references. But rather than refer to whatever document that has anything to do with this plan (quality procedures, etc.) I only link to documents that I’ve actually used to do the planning. This answer WHY question. This will help reviewer to suggest me what sources of information I missed or which of used sources are not reliable (not up-to-date, etc).
- The next chapter is introduction where I introduce the component. I describe in my words how do I understand it’s business goal derived quality goals, etc. This actually describes my assumptions and will help reviewer to correct my assumptions if they are wrong.
- There are more chapters where I try to outline what tools/techniques I will use in preparing test data, executing tests, validating results. Developer may suggest better tools or deeper debugging options, how to test caching, synchronization, etc., how to use logs and debug info.
- I don’t write test scope details up-front. I fill that chapter in during test execution. Even if I have detailed SRS and I don’t do exploratory testing there are no sense to copy-paste items from SRS. I want to list what I’ve really tested during testing and them compare that to SRS.


Endnote
This is a really huge text for me, especially taking into account that English is my 3rd language. It would really require a complete rework but I’m afraid I’m exhausted writing this one. I reserve the rights to come back to this blog later and rework it significantly. Your feedback is more than welcome as it will help me to set goals for the rework.