Support Center

Burp Community

See what our users are saying about Burp Suite:

How do I?

New Post View All

Feature Requests

New Post View All

Burp Extensions

New Post View All

Bug Reports

New Post View All
Documentation

Burp Suite Documentation

Take a look at our Documentation section for full details about every Burp Suite tool, function and configuration option.

Full Documentation Contents Burp Projects
Suite Functions Burp Tools
Options Using Burp Suite
Extensibility

Burp Extender

Burp Extender lets you extend the functionality of Burp Suite in numerous ways.

Extensions can be written in Java, Python or Ruby.

API documentation Writing your first Burp Suite extension
Sample extensions View community discussions about Extensibility
Support Center Issue Definitions Robots.txt file

Robots.txt file

Description

The file robots.txt is used to give instructions to web robots, such as search engine crawlers, about locations within the web site that robots are allowed, or not allowed, to crawl and index.

The presence of the robots.txt does not in itself present any kind of security vulnerability. However, it is often used to identify restricted or private areas of a site's contents. The information in the file may therefore help an attacker to map out the site's contents, especially if some of the locations identified are not linked from elsewhere in the site. If the application relies on robots.txt to protect access to these areas, and does not enforce proper access control over them, then this presents a serious vulnerability.

Remediation

The robots.txt file is not itself a security threat, and its correct use can represent good practice for non-security reasons. You should not assume that all web robots will honor the file's instructions. Rather, assume that attackers will pay close attention to any locations identified in the file. Do not rely on robots.txt to provide any kind of protection over unauthorized access.

Vulnerability classifications

Typical severity

Information

Type index

0x00600600