We
often use “error counters” to indicate the successful or failure of our test
targets. To make some concepts clearer, here’s some formal definitions on those
error-related concepts.
Some formal terminology in Testing (software testing, craftsman’s approach)
Error
“People make errors. A good synonym is “mistake”. This is the
origin of all system failures.”
Fault
“A fault is the result of an error. It is more precise to
say theat a fault is the representation of an error. E.g, when people make
errors, they could write wrong expressions, or they could design a wrong flow
char, etc…, these are all fault. A good synonym is “Bug”, I suggest we use
“bug” instead of Fault in future. “
Normally,
looking for a bug and making any correction on those bugs if necessary, is the
job of developers. However, a good tester should also provide some suggestions
on the possible bugs to developers when there are system failures found. For a
systematic testing, bugs should be collected and on which, statistics and
analysis should be made to improve testing effectiveness and developing
process.
Failure
“A failure occurs when a fault executes. E.g, invalid
outputs from an application. But this definition is not completed, “omission”
bugs can also result in “failure”.”
Incident
When a failure occurs, if may or may not be readily
apparent to the user/tester/developer. An incident is the symptom associated
with a failure that alerts the user to the occurrence of a failure.
Incident is actually the “error” we mentioned in our test
scrīpts. But to make life easier, we can use failure and incident interchangeable
as long as we are aware of the fact that they are only part of all possible
failures that we detect. In future, we should call our detected “error”
“failure”, so an “error counter” now will be named as failure counter.
Failure counter design
Failures
can propagate, normally, in our practice, the first error in certain test case
bears the significant information. We call those failures “key failures”,
and we should use a dedicated counter for them. This counter is called “key
failure counter(kfCnt)”. All failures including those propagated ones are
called “general failures”, and a separate counter should be designed for
them, this counter is named “general failure counter(gfCnt)”. So in our
scrīpts/testingApplications, we should keep two counters, key failure counter
and general failure counter. The basic rule is like this:
If
(kfCnt increase by n)
GfCnt increases at least by n.
A
switch can be used to enable/disable gfCnt displaying. Switching off the gfCnt
will make us understand the failure easier, also, this will help developer in
debugging.
Remarks
Besides all the failures mentioned above, there’s
another kind of failure, which is generated due to errors made during
testing-scrīpts design and implementation. E.g, when passing parameters to a
function call in a scrīpt, some invalid parameters will cause the function
failure. As a good tester, we should minimize, if not able to delaminate, those
errors during scrīpt implementation. For a better design, another failure
counter, scrīpt failure counter (sfCnt) should be embedded into our test
scrīpts. scrīpt failure may have a serious impact on final testing result, so
the best rule is to terminate the scrīpt whenever there’s a scrīpt failure
generated.
