Meet urlhunter, the URLTeam companion and shortened URL search engine

Open source tool useful for threat intel experts and bug bounty hunters, says its architect

URL shorteners have a notorious reputation for being bad for both privacy and security.

Users often use them to share sensitive, unprotected files stored in the cloud via truncated links that give the user no indication where they’re navigating to.

Moreover, malicious actors can find these documents with a little brute force search through the shortened URL address space.

Urlhunter, a tool developed by security analyst Utku Sen, can search URL shortener archives. Security researchers can use it to find sensitive files and URLs that have been accidentally shared with the public.

The tool also highlights the dangers of using URL shorteners.

Brute forcing URL shorteners

Sen got the idea for urlhunter from a bug bounty hunter who told him about the URLTeam archive.

URLTeam regularly publishes a list of shortened URLs generated by services such as bit.ly, goo.gl, and ow.ly from their original long-form URLs.

URL shorteners compress long addresses into very short strings. This makes it easy to brute force the entire range of addresses of a shortener service and map each shortened URL to its original address, which is how URLTeam manages to update its list every day.

The URLTeam archive contains a lot of valuable information for bug bounty hunters and threat intel experts.

“For example, it's possible to find private information of companies via public Google Doc links,” Sen tells The Daily Swig. “Also, it reflects the danger of using shortener services for sensitive URLs.”

Happy hunting

Sen, who also develops other security tools and has delivered several presentations at DEF CON’s Demo Labs, quickly turned the idea into a working program.

Urlhunter is a Go program that can search the URLTeam archive for specific keywords. For instance, a researcher might use it to discover publicly shared Google Docs and Drive files, public Trello boards, and URLs with password reset tokens. Users can use urlhunter to search the latest copy of the archive or go through previous versions of the URL lists.

“Urlhunter is useful for cyber intelligence and bug bounty purposes. Bug bounty hunters can make money by finding private documents, admin panels, and other directories that are not easy to find,” Sen says.

Before running queries, urlhunter must download the URLTeam archive from the Archive.org servers where the files are stored. Since Archive.org throttles download speeds, the current version of urlhunter runs a little slowly when downloading new archives.

Sen told The Daily Swig that he plans to equip the tool with URLTeam’s alternative torrent download method to speed up performance.