Getting Started With Burp Spider
Burp Spider is a tool for automatically crawling web applications. While it is generally preferable to map applications manually, you can use Burp Spider to partially automate this process for very large applications, or when you are short of time.
Note: Using Burp Spider may result in unexpected effects in some applications. Until you are fully familiar with its functionality and settings, you should only use Burp Spider against non-production systems.
To start getting to know Burp Spider, carry out the following steps:
- First, ensure that Burp is installed and running, and that you have configured your browser to work with Burp.
- In Burp, go to the Proxy Intercept tab, and turn off Proxy interception (if the button says "Intercept is off" then click it to toggle the interception status).
- Browse around a few pages of the application.
- In Burp, go to the Target tab and look at the site map. This contains all of the URLs you have visited in your browser, and also all of the content that Burp has inferred from responses to your requests (e.g. by parsing links from HTML responses). Items that have been requested are shown in black, and other items are shown in gray.
- Note: When spidering, Burp uses the Spider scope settings to determine which URLs will be requested. If you are new to Burp, and have modified any settings relating to target scope or spidering, go to the Burp menu and restore default settings for the Target and Spider tools before proceeding.
- In the Target site map, find the application you want to spider (this will typically be a specific branch of the site map, or sometimes an entire host). Select the relevant node in the site map tree, and choose "Spider this host / branch" from the context menu.
- Assuming the selected item is not within the currently defined scope, Burp will prompt you to confirm you want to proceed. Click "Yes", and Burp will modify the current target scope to include the selected item, and all sub-items within the site map tree.
- Burp will then turn on the Spider, which will begin crawling. Go to the Spider Control tab, and view the progress of the Spider (number of requests made, bytes transferred, etc.). While the Spider it running, it may prompt you for guidance in submitting some HTML forms. You can cancel these dialogs or fill out the form fields if you prefer. (You can later configure how the Spider submits forms in the form submission options.)
- Go to the Target tool and browse the tree view of the site map. As the Spider runs, more items that were previously not requested (shown in gray) will be requested (shown in black), and further items that have been discovered by the Spider may be added.
- If you want to monitor new items in the site map as they are added, select the application branch or host in the tree view, and click twice on the "Time requested" column header in the table view (the first click sorts ascending; the second click sorts descending). This will sort all the URLs in the application according to the time requested, with the most recently request items at the top.
Use the links below for further help on starting to use Burp Spider: