Getting Started With Burp Spider
Burp Spider is a tool for automatically crawling web applications. While
it is generally preferable to map
applications manually, you can use Burp Spider to partially automate
this process for very large applications, or when you are short of time.
Note: Using Burp Spider may result in unexpected effects
in some applications. Until you are fully familiar with its functionality
and settings, you should only use Burp Spider against non-production
To start getting to know Burp Spider, carry out the following steps:
- First, ensure that Burp is
installed and running,
and that you have configured
your browser to work with Burp.
- In Burp, go to the Proxy Intercept tab, and turn off Proxy
interception (if the button says "Intercept is off" then click it to
toggle the interception status).
- Browse around a few pages of the application.
- In Burp, go to the Target tab and look at the site map. This
contains all of the URLs you have visited in your browser, and also all
of the content that Burp has inferred from responses to your requests
(e.g. by parsing links from HTML responses). Items that have been
requested are shown in black, and other items are shown in gray.
- Note: When spidering, Burp uses the
Spider scope settings to
determine which URLs will be requested. If you are new to Burp, and have
modified any settings relating to target scope or spidering, go to the
Burp menu and restore default settings for the Target and Spider tools
- In the Target site map, find the application you want to spider
(this will typically be a specific branch of the site map, or sometimes
an entire host). Select the relevant node in the site map tree, and
choose "Spider this host / branch" from the context menu.
- Assuming the selected item is not within the currently defined
scope, Burp will prompt you to confirm you want to proceed. Click "Yes",
and Burp will modify the current target scope to include the selected
item, and all sub-items within the site map tree.
- Burp will then turn on the Spider, which will begin crawling. Go to
the Spider Control tab, and view the progress of the Spider (number of
requests made, bytes transferred, etc.). While the Spider it running, it
may prompt you for guidance in submitting some HTML forms. You can
cancel these dialogs or fill out the form fields if you prefer. (You can
later configure how the Spider submits forms in the
form submission options.)
- Go to the Target tool and browse the tree view of the site map. As
the Spider runs, more items that were previously not requested (shown in
gray) will be requested (shown in black), and further items that have
been discovered by the Spider may be added.
- If you want to monitor new items in the site map as they are added,
select the application branch or host in the tree view, and click twice
on the "Time requested" column header in the table view (the first click
sorts ascending; the second click sorts descending). This will sort all
the URLs in the application according to the time requested, with the
most recently request items at the top.
Use the links below for further help on starting to use Burp Spider:
Wednesday, April 22, 2015
This release contains a number of minor enhancements and bugfixes, including:
- The Proxy now uses SHA256 to generate its CA and per-host certificates
- There is a new button at Proxy / Options / Proxy Listeners to force Burp to regenerate its CA certificate.
- A bug in the "Paste from file" function which caused Burp to sometimes retain a lock on the selected file has been fixed.
- A bug in the Intruder "extract grep" function, which sometimes caused extracted HTML content to be rendered as HTML in the results table, has been fixed.
See all release notes ›