3 simple steps to evaluate a web vulnerability scanner

Step 1 - Choose a web app that will make testing easy

Pick an app where the issues are known

It's important to test a scanner on a modern environment with a known attack surface and vulnerabilities.

In an ideal world, you'd test a scanner against one of your own applications.

Unfortunately, many people don't feel comfortable scanning one of their company's production sites, or aren't confident that they know the complete attack surface or security issues present on their own sites. Further, the nature of modern web applications means that new security vulnerabilities may not yet be known to those testing it.

PortSwigger's Gin & Juice Shop provides you with a list of the vulnerabilities it contains.

Use a scan-me site to test quickly and objectively

Definition: A scan-me site is a deliberately vulnerable web application designed for evaluating web vulnerability scanners.

Scan-me sites provide an environment with known security issues that anyone can scan, thus making it easier to objectively test how well a web vulnerability scanner performs.

A good scan-me site will have a documented list of security issues and detail the URLs or exploitable "attack surface" on a web app.

Further, a good scan-me app will consist of the same technological components you'd expect to find in your own web applications. This may include certain hard-to-scan single page application frameworks such as React or Angular and authentication mechanisms.

OWASP produces a list of vulnerable web applications - including a "technology" field, to tell you which technologies a particular scan-me site uses. We recommend that you consider multiple scan-me sites with characteristics that are important to you.

Be aware that some scan-me sites are quite old and unmaintained, meaning they might no longer represent the modern web applications in your portfolio.

Step 3 - Determine how well the scanners performed

List each app's security issues in a spreadsheet

The difficulty of this task depends on the scan-me apps you've selected for testing.

Some (like PortSwigger's Gin & Juice Shop) provide you with a list of the vulnerabilities they contain. Other scan-me sites do not tell you what you should find.

If you plan to test your scanner against one of your company's own web applications, listing the security issues may even require running a penetration test to know what issues exist - a potentially expensive and time-consuming exercise.

Highlight the security issues that are most important to you

If you are simply looking to compare two scanner's capabilities, then you may be able to skip this step. If, however, you are looking for a scanner that is compliant with your company's security policy, or to see if a web vulnerability scanner will detect specific security issues, then this step will be important.

To carry out this step, you'll need a list of the security issues that are of particular importance to your organization.

This is likely to depend on the types of technologies you use in and around your web applications, and will require a degree of technical knowledge. Speak to your security / AppSec team if you are unsure.

Check off the security issues found by each scanner

In an ideal world, this would be a simple task. In reality, however, it requires a bit of judgment - as different vendors often call security issues by different names.

You can often get around this problem by looking at a security issue's CWE (Common Weakness Enumeration). PortSwigger provides CWEs for the security issues found by Burp Scanner, where available.

Compare results

The practical and best starting point for comparing web vulnerability scanner performance is to answer the question: "Did it find all of the security issues that are known to exist on my target application?".

These are your true positives.

You should also count the number of false positives, or the number of security issues reported by your chosen scanner that don't actually exist on your target site.

An excessive number of false positives cost the users of web vulnerability scanners time and money to investigate the issues.

Draw your conclusion - which web vulnerability scanner performs best?

If you completed the above steps using different web vulnerability scanners, we expect that you've found their performance varies widely.

To conclude your evaluation and decide which web vulnerability scanner performed best in your tests, you should answer the following questions:

• True positives: Did the web vulnerability scanner find all of the security issues that are known to exist on my target application?
• False positives: Did the product report too many false positives, or security issues that don't actually exist on my site?
• Critical issues and security policy: Did the product detect all of the critical security issues on my site and/or the issues my company wants to detect across its portfolio?
• Usability and accessibility: Was the scan easy to set-up and run? Were options provided for different out-of-the-box scan configurations?
• Speed: How quickly did the scans complete? Were options provided for quick and deep scans?
• Authentication and single page application crawling: Does the web vulnerability scanner handle technologies used in my web applications, including the ability to navigate through authentication and crawl JavaScript-heavy single page applications (SPAs)?
• Technology support: Does the product have other features that are important to your organization e.g. integrations, reporting, single sign-on, tech support etc?
• Cost: Just as the performance of web vulnerability scanners varies widely, so does the cost of products. You often get what you pay for ... but many commercial web vulnerability scanners are well known for their excessive prices.