Burp Suite Enterprise Edition is now available in our secure Cloud  –  Learn more

3 simple steps to evaluate a web vulnerability scanner

There are many web vulnerability scanners available on the market and their performance varies widely. Here we show you how you can quickly and objectively evaluate web vulnerability scanners, to help you find the best product for detecting security issues in your web applications.

It's easy - follow these 3 simple steps:

Step 1 - Choose a web app that will make testing easy

Pick an app where the issues are known

It's important to test a scanner on a modern environment with a known attack surface and vulnerabilities.

In an ideal world, you'd test a scanner against one of your own applications.

Unfortunately, many people don't feel comfortable scanning one of their company's production sites, or aren't confident that they know the complete attack surface or security issues present on their own sites. Further, the nature of modern web applications means that new security vulnerabilities may not yet be known to those testing it.

PortSwigger's Gin & Juice Shop provides you with a list of the vulnerabilities it contains.

Use a scan-me site to test quickly and objectively

Definition: A scan-me site is a deliberately vulnerable web application designed for evaluating web vulnerability scanners.

Scan-me sites provide an environment with known security issues that anyone can scan, thus making it easier to objectively test how well a web vulnerability scanner performs.

A good scan-me site will have a documented list of security issues and detail the URLs or exploitable "attack surface" on a web app.

Further, a good scan-me app will consist of the same technological components you'd expect to find in your own web applications. This may include certain hard-to-scan single page application frameworks such as React or Angular and authentication mechanisms.

OWASP produces a list of vulnerable web applications - including a "technology" field, to tell you which technologies a particular scan-me site uses. We recommend that you consider multiple scan-me sites with characteristics that are important to you.

Be aware that some scan-me sites are quite old and unmaintained, meaning they might no longer represent the modern web applications in your portfolio.

Gin & Juice Shop

PortSwigger's Gin & Juice Shop is a modern scan-me site that makes it easy to test web vulnerability scanners. It uses many of the technologies used to build today's web applications, including plenty of JavaScript and CSRF tokens. Gin & Juice Shop contains many modern web vulnerabilities, doesn't require deployment, and is just the right size to quickly assess a scanner's performance (not too many duplicate issues).

Step 2 - Select web vulnerability scanners and scan your apps

Select scanners that have the capabilities you need

If you're reading this page, chances are you already have a few web vulnerability scanners you want to test. There are a number of open-source and / or free products, and many commercial scanners offer free trials.

OWASP lists a range of web vulnerability scanning products, and you can always request a free trial of Burp Suite Professional or Burp Suite Enterprise Edition on PortSwigger's website.

Scan your apps

The good news is that most scanners make it pretty easy to initiate a scan. It's often as easy as entering a URL.

For example Burp Suite Enterprise Edition (PortSwigger's automated web vulnerability scanner) gives you the option to use one of four preset scan configurations to get started.

Step 3 - Determine how well the scanners performed

List each app's security issues in a spreadsheet

The difficulty of this task depends on the scan-me apps you've selected for testing.

Some (like PortSwigger's Gin & Juice Shop) provide you with a list of the vulnerabilities they contain. Other scan-me sites do not tell you what you should find.

If you plan to test your scanner against one of your company's own web applications, listing the security issues may even require running a penetration test to know what issues exist - a potentially expensive and time-consuming exercise.

Highlight the security issues that are most important to you

If you are simply looking to compare two scanner's capabilities, then you may be able to skip this step. If, however, you are looking for a scanner that is compliant with your company's security policy, or to see if a web vulnerability scanner will detect specific security issues, then this step will be important.

To carry out this step, you'll need a list of the security issues that are of particular importance to your organization.

This is likely to depend on the types of technologies you use in and around your web applications, and will require a degree of technical knowledge. Speak to your security / AppSec team if you are unsure.

Check off the security issues found by each scanner

In an ideal world, this would be a simple task. In reality, however, it requires a bit of judgment - as different vendors often call security issues by different names.

You can often get around this problem by looking at a security issue's CWE (Common Weakness Enumeration). PortSwigger provides CWEs for the security issues found by Burp Scanner, where available.

Compare results

The practical and best starting point for comparing web vulnerability scanner performance is to answer the question: "Did it find all of the security issues that are known to exist on my target application?".

These are your true positives.

You should also count the number of false positives, or the number of security issues reported by your chosen scanner that don't actually exist on your target site.

An excessive number of false positives cost the users of web vulnerability scanners time and money to investigate the issues.

Why might one scanner perform better than another?

Discovering a web application's attack surface can be difficult. While this task used to be relatively easy, how modern web applications are developed has made discovering a web application's attack surface much trickier than it used to be. Many web vulnerability scanners therefore fail to discover content when used to scan modern web applications.

The second reason that some scanners are better than others is that there's a huge variation in how different products audit the attack surface for vulnerabilities. Different web vulnerability scanners will use different "scan checks" and techniques for uncovering vulnerabilities that could pose a risk to your business, so it's worth bearing these points in mind.

Draw your conclusion - which web vulnerability scanner performs best?

If you completed the above steps using different web vulnerability scanners, we expect that you've found their performance varies widely.

To conclude your evaluation and decide which web vulnerability scanner performed best in your tests, you should answer the following questions:

Go do it yourself - helpful resources

Ready to evaluate a web vulnerability scanner? Here are some resources we can offer: