testingReflections.com
Created and managed by Antony Marcano, testingReflections aggregates some of the best software development and testing blogs on the web.
more...
Good Defect Reporting: de-bugging without a debugger
Submitted by Ainars Galvans on Fri, 19/01/2007 - 12:20.
functional testing | general software testing
This is last item in series about testing concept that are simple in text-books, but so complicated in real life within a context.
It is a simple to report a defect. Just describe steps to repeat problems and submit the report to developer. Developer will be doing debugging using a debuger and fix the problem.
There are two issues, but both are related. What if a sequence of steps to repeat the problem is enormous or even defect is repeatable only for certain, single record in database. Do you want developer to connect to your database and edit it’s content? What if he fix defect by fixing the record?
Debugging: process of using a tool or something more?
If you google “define:debugging “ you will find a lot of definitions that describe it mostly as process of finding and removing either bugs or mistakes in software (mostly in code). The fact that there are tools called debugger in for main programming languages has limited debugging to use of the tool at least in developer’s jargon. However finding the bug quite often begins with failed test and more tests to localize the bug. I believe at least localization of failure is part of debugging. A manual tester could do this even better than a developer.
As a tester I always try to realize what was the actual problem of the issue: What is the shortest/simples scenario to repeat the problem? Which test data affects it and which does not? For example – a defect appears only because on step 5 while filling out form of 25 records you entered Yes instead of No in a single field but still you report defect that requires developer to go though the 10 steps filling a lot of fields with exact values on each step. I bet the developer will mistype time to time and will probably find defect not repeatable doe to that. More over – only when you have found the actual reason for a defect you could do defect triaging
Example
I’m testing component that have 3 parts: design-time when we define workflow (BPM-type application) and business logic, run-time where this logic is executed and configuration when I configure DB connections, security, 3rd party configuration and a lot of parameters such a maximal allowed number of simultaneous executions. Now suppose that I’ve got exception in run-time that reports me NullPointerException. The problem could be any of the following: a) wrong configuration + poor error handling in run-time code. Error handling should suggest what is wrongly configured; b) Business logic lack some validation. If it will be fixed run-time will never run into such a problem c) Plain run-time problem: developer never imagined about such a use case d) run-time limitation need to document that it can’t e used this way and optionally improve error message, e) 3rd party run-time error. Need to develop a workaround; f) ....
Now imagine developers who have implemented the design-time validation and run-time code are different. The developer who knew the 3rd party is reallocated to a different project, but the one who developed the business logic run-time parsing is quit work last moth. Who will be doing debugging for this defect?
My solution to example
First of all I assume that developers have run some unit tests and this functionality works in their environment, for their business logic, 3rd party configuration, etc. I need to realize what is so different in my case that makes the system to fail. I’m capable of doing that more frequently than I’m not. Especially if at least something works I could change the case that works to the case that doesn’t step-by-step and see on which step it fail. This way I could identify the single property the atomic element of my test case that was reason for failure. Atomic in black-box, not in white-box. It is still developer who will find the lines of code to be fixed, but I will at least tell him the item that fails instead system level scenario that involves all the components developed by hundreds of coders and reusing libraries developed by thousands.
It is a simple to report a defect. Just describe steps to repeat problems and submit the report to developer. Developer will be doing debugging using a debuger and fix the problem.
There are two issues, but both are related. What if a sequence of steps to repeat the problem is enormous or even defect is repeatable only for certain, single record in database. Do you want developer to connect to your database and edit it’s content? What if he fix defect by fixing the record?
Debugging: process of using a tool or something more?
If you google “define:debugging “ you will find a lot of definitions that describe it mostly as process of finding and removing either bugs or mistakes in software (mostly in code). The fact that there are tools called debugger in for main programming languages has limited debugging to use of the tool at least in developer’s jargon. However finding the bug quite often begins with failed test and more tests to localize the bug. I believe at least localization of failure is part of debugging. A manual tester could do this even better than a developer.
As a tester I always try to realize what was the actual problem of the issue: What is the shortest/simples scenario to repeat the problem? Which test data affects it and which does not? For example – a defect appears only because on step 5 while filling out form of 25 records you entered Yes instead of No in a single field but still you report defect that requires developer to go though the 10 steps filling a lot of fields with exact values on each step. I bet the developer will mistype time to time and will probably find defect not repeatable doe to that. More over – only when you have found the actual reason for a defect you could do defect triaging
Example
I’m testing component that have 3 parts: design-time when we define workflow (BPM-type application) and business logic, run-time where this logic is executed and configuration when I configure DB connections, security, 3rd party configuration and a lot of parameters such a maximal allowed number of simultaneous executions. Now suppose that I’ve got exception in run-time that reports me NullPointerException. The problem could be any of the following: a) wrong configuration + poor error handling in run-time code. Error handling should suggest what is wrongly configured; b) Business logic lack some validation. If it will be fixed run-time will never run into such a problem c) Plain run-time problem: developer never imagined about such a use case d) run-time limitation need to document that it can’t e used this way and optionally improve error message, e) 3rd party run-time error. Need to develop a workaround; f) ....
Now imagine developers who have implemented the design-time validation and run-time code are different. The developer who knew the 3rd party is reallocated to a different project, but the one who developed the business logic run-time parsing is quit work last moth. Who will be doing debugging for this defect?
My solution to example
First of all I assume that developers have run some unit tests and this functionality works in their environment, for their business logic, 3rd party configuration, etc. I need to realize what is so different in my case that makes the system to fail. I’m capable of doing that more frequently than I’m not. Especially if at least something works I could change the case that works to the case that doesn’t step-by-step and see on which step it fail. This way I could identify the single property the atomic element of my test case that was reason for failure. Atomic in black-box, not in white-box. It is still developer who will find the lines of code to be fixed, but I will at least tell him the item that fails instead system level scenario that involves all the components developed by hundreds of coders and reusing libraries developed by thousands.
