AppScan results differ from one scan to the next identical scan

Here are the most common causes for varying results between scans:

Cause 1

By default, AppScan runs its automatic explore in 10 threads. Sometimes, due to timing, the threads may explore the site in a slightly different fashion from one scan to the next.

Cause 2:

Communication problems can lead to scans tests being skipped.

Cause 3:

Sometimes an application may return a different response from an identical request.

For example, AppScan may be testing a web application that is accessed through a proxy which for whatever reason periodically returns errors. This then causes inconsistent vulnerability results.

The easiest way to determine if tests are being skipped, is by opening AppScan's new real-time scan log (Scan > Scan Log) during the course of the scan. If tests were skipped, you will see entries similar to the following examples:

• Cannot connect to host: [IP/HOSTNAME]
  
  
• Test [TEST_NUMBER] [TEST_NAME] failed due to communication error: [PATH]
  
  
• Connection established with host: [IP/HOSTNAME]

Diagnosing the problem for Cause 3:

The way to isolate the issue is to run two identical scans with the traffic log enabled (seeTechnote 1287440for instructions), save the Traffic log after each scan, then determine the scan differences in the Issues view.

Solution for Cause 1:

If the visited URLs displayed under Application Data > Visited URLs differ in two comparable scans, you can set up AppScan to run in 1 thread (Scan > Scan Configuration > Connection > Number of Threads). This will ensure that AppScan will traverse the site in the same fashion as a previous scan, permitting that the site content hasn't changed in the interim.

Solution for Cause 2:

If you notice excessive entries similar to the above, refer totechnote 1283298-How to address communication problems during a scan in AppScan

Solutions for Cause 3

1. Once one issue shows up in one scan, but not in the next, copy the exact test payload in the scan that came up positive. (For example, if the payload is delivered on a parameter and the Test Request is a GET, copy the first line of the Test Request to your clipboard; otherwise, if its a POST request, copy the parameters which include the payload.) Then search the Traffic log for the scan which the vulnerability was missed in and when you find the request, find the response which is the next one for the listed thread id (like the thread id for "---- Begin Thread 2768[132c] ----" is 132c) to determine why it is different from that listed in the vulnerable scan.
   
   
2. If using AppScan 7.5 or higher, an alternate solution is to perform. a similar analysis by enabling the invulnerable Variants (under the Scan menu) and searching the results for responses which do not come up as vulnerable.