Optimizing job and scan configurations for large sites

A job or scan that you are running is taking an excessive amount of time and is finding a large number of links.

This might come a little surprising at first but from the security perspective, big sitesare very rare.

Let us elaborate a bit what this statement means. If you go over 500 links in your scan, you should ask yourself: did the developers of this application really write 500 individual scripts (JSP/ASP/PHP files)? Actually, most of the time you will encounter sites that contain tens of thousands or even a million pages.
There is no way that a development team no matter how large wrote a million pages.

Redundant path limit

From the security perspective, we are only interested in testing a limited set of instances for an individual script. (page).

The redundant path limit setting (Rational AppScan Standard- Scan Configuration > Explore Options,Rational AppScan Enterprise / Rational Policy Tester- Edit job properties > Explore Options) restricts the number of requests to a specific URL, which does not include the query parameters.

The path in this example URL
http:// www.site.com/folder1/folder2/index.jsp?query=123

is represented by the following section

folder1/folder2/index.jsp

The path usually specifies the name of the script. and its location on the servers file system.

In Rational AppScan Enterprise, due to its integration with Rational Policy Tester where the focus is content scanning, the query is included in the redundant path limit calculations.

Let us look at an example of a entertainment site that contains biography pages for different artists. The biography page is called biography.jsp, is based on the value of a query parameter calledartist, and will display a different text and picture for each artist.

As you can see, there is no difference in structure between

biography.jsp?artist=madonna&session=123

and

biography.jsp?artist=britney_spears&session=123

The only difference between the two is the text of the biography and the picture that are displayed in the page. So if there is a Cross-Site Scripting vulnerability on the session parameter, it will exist on bothartist=madonnaandartist=britney_spears, so it does not make sense to navigate to this page more than once.

The redundant path limit allows the artist parameter to change only 5 times by default, preventing Rational AppScan Standard / Rational AppScan Enterprise / Rational Policy Tester from testing this page for every single artist on the site. However in this specific case, the redundant path limit of 5 is too much.

Why then have the redundant path limit set to 5 instead of 1?

There are situations where parameters have effect on the structure of the page. For example the navigational parameters that are encountered in MegaScript. applications. To learn more about MegaScripts and advanced redundancy tuning in Rational AppScan Standard, reviewHandling MegaScript. sites with Rational AppScan Standard.

The restriction to 5 identical paths was set in an attempt to find a middle ground between parameters that affect the page content and parameters that affect the page structure. However, if the parameters only affect the page content and the site has 200 pages, Rational AppScan Standard / Rational AppScan Enterprise / Rational Policy Tester will discover 1000 pages and take 5 times longer to explore the site.

So in certain cases the redundant path limit should be decreased or set to 1. If a limited number of pages that change their structure based on parameters values exists, manual explore or multi-step operations should be used for those pages .

What if it is really hard to separate the path from parameters?

Web application developers often use a technology called URL Rewriting to hide parameters in the directory structure. Let us imagine that our entertainment site uses the following rewrite rule.

RewriteRule ^biography/(.+).jsp biography.jsp?artist=$1

This rule tells the web server to convert the URL that you see in the web browser, such as
http:// www.site.com/biography/madonna.jsp

to the following
http:// www.site.com/biography.jsp?artist=madonna.jsp

The main reason behind URL rewriting is to force Google and other search engines to index all the pages of the site. Another advantage of URL Rewriting is that questions marks and equal signs are removed from the URL making it easy to remember. The whole transformation is entirely hidden from the user.

The problem posed by URL Rewriting for Rational AppScan Standard / Rational AppScan Enterprise / Rational Policy Tester is that it renders the redundant path limit useless. The parameters are now part of the path and the product has no way of automatically knowing which is the script. and which is the parameter.

If there are ten thousands artists on our entertainment site you will now have ten thousand additional URLs in your scan when you should really have just one. Add to that another URL-rewrited parameter that handles the session and changes its value every time you login and you will now have a never ending scan. If this occurs, Rational AppScan Standard / Rational AppScan Enterprise / Rational Policy Tester will eventually run out of resources.

How to identify URL rewriting?

Rational AppScan Standard

If the scan goes past the 500 URL mark, perform. the following:

1. Pause the scan
   
2. Choose the Application Data view on the left
   
3. Highlight each folder and look at the number of visited URLs displayed in the "Show" drop-down located at top-center of the screen to find the folder with the most URLs. In our example the folderbiographywould show ten thousand pages.
   
4. Now that you located the problematic folder, you can check to see if all URLs in this folder follow a specific pattern. In our example you would notice that all the pages in the biography folder have celebrity names.

Rational AppScan Enterprise / Rational Policy Tester

If the job takes too long to execute and the number of pages scanned is very big in the status screen:

1. Save current results and stop. It is very important to selectSave current results and stopand notDiscard results and stopsince only the save option will also run the reports on the data gathered up to this point
   
2. Examine the pages report to identify URL rewriting patterns using a similar process as in Rational AppScan Standard

How to handle URL rewriting ?

Rational AppScan Standard - Custom Parameters

1. The first step is to identify the parameter values in the URL. This can be done by comparing the differences between the URLs that are part of the same folder
   
   Example:
   
   The difference between
   
   http:// www.site.com/biography/madonna.jspand
   
   http:// www.site.com/biography/britney_spears.jspis the page name. This difference could be comprised in the following regular expression:
   
   biography/(.+)\.jsp
   
   
2. Once identified the URL-rewrited parameters can be added to the list of Custom Parameters underScan Configuration > Parameters and Cookies > Custom Parameters
   
   If our example the Custom Parameter definition will be:
   
   Reference Name: artist
   Pattern: biography/(.+)\.jsp
   Location: Path
   
   
   
   Defining the parameter this way will actually allow Rational AppScan Standard to send application type tests to this entity. An example of a Cross-Site scripting attack for this site would look like this:
   http://www.site.com/biography/.jsp
3. After defining the parameter, you need to edit its redundancy settings. To do that, click on the Parameters and Cookies tab and then click on the plus sign.
   
   
4. In the "Type" drop down choose Custom Parameter and then choose the reference name you just defined.
   
   
5. Under the redundancy tuning settings at the bottom chooseWhen comparing explore requests ignore the parameter valueandDo not retest adjacent parameters when the parameters value changesas per in the attached screenshot.
   
   

Rational AppScan Enterprise / Rational Policy Tester - URL Substring Exclusions

1. In the Job Configuration,go toParameters and Cookies.
   
   
2. Scroll to the bottom until you find theNormalization Rulescategory
   
   
3. Add the pattern you have identified underIgnore the following URL substrings when applying normalization rulespreceded by the "regexp:" prefix.

For our example the pattern would be:regexp:biography/(.+)\.jsp. There is no need to add a delimiter for regular expressions.

Alternatively ,you can simply enter a substring and a corresponding delimiter. For our example the substring would bebiography/and the corresponding delimiter would be.jsp.



This will prevent the exploration and testing of the redundant URLs.