Black Hat 2021: WARCannon simplifies web-wide vulnerability research

‘Even the most resource-constrained researcher can now add web-scale analysis to their arsenal’

An open source tool that makes grepping the internet for web vulnerabilities simpler, faster, and cheaper was unveiled at Black Hat USA today.

Security researchers and bug bounty hunters who unearth novel flaws in web applications, web frameworks, or open source components can use WARCannon to non-invasively test regex patterns across the entire internet for corresponding vulnerability indicators.

But this entails the parsing of a daunting volume of data – in the region of hundreds of terabytes.

DON’T MISS Black Hat 2021 keynote: Zero-days, ransoms, supply chains, oh my!

Parallel processing

Fed by Common Crawl spiders’ trove of JavaScript, CSS, and other website code, WARCannon tackles this challenge with parallel WARC processing across hundreds of CPU cores, while costs are reduced courtesy of spot fleets and same-region data transfer.

As a result, “WARCannon can process multiple regular expression patterns across 400TB in a few hours for around $100”, according to the tool’s GitHub repo.

Results are stowed in S3 storage and retrieved at speeds of up to 100Gbps per node.

Multiplier effect

WARCannon has a multiplier effect on productivity, says tool developer Brad Woodward, who is practice lead for cloud security architecture at Observian.

“Digging through a framework, integration, technology, or component to find a vulnerability is really labor intensive,” he told The Daily Swig ahead of his Black Hat Arsenal presentation.

“For researchers whose efforts extend beyond a single application or customer, WARCannon lets them multiply the fruits of these labors by finding similarly vulnerable implementations in the wild.”

Woodward suggests that bug bounty hunters could extract the greatest benefit.

“Consider a researcher who spends two days finding a vulnerability in one place,” he explains. “They can then plug their findings into WARCannon and turn one vulnerability into five, 500, or 50,000.”

Optimizing performance

Woodward says that “integrating custom searches and validations is relatively simple” and executed by many other tools.

WARCannon, he said, “is architected around scaling a single version of the code. This means that whatever works locally will work exactly the same with a one-off test or a full campaign.

“It’s built to make these ‘small scale’ tests easy, which helps researchers be more confident before they pull the trigger at scale.”

Woodward also “tightened down the thumbscrews to squeeze out as much performance-per-penny as possible. This took weeks of tweaking knobs and buttons in different permutations until only the most effective approaches remained.

“The optimizations here were so great that even the most resource-constrained researcher can now add web-scale analysis to their arsenal.”

‘Apache Project for hackers’

Woodward says he has no immediate plans for further developing the tool.

“That said,” he continues, “I really like the idea of aggregating vulnerability indicators from the community, routinely running campaigns on each new crawl as they become available, and sharing the results with the cybersecurity community. This poses a number of challenges that I don’t have good solutions to just yet (PunkSpider is facing many similar challenges), but I’d love to see something happen in that vein.”

After BlackHat, WARCannon will, like two other tools developed by Woodward – NPK and Hirogen – be supported by Porchetta Industries, which Woodward describes as “a sort of Apache Project for Hackers”.

He added: “Even if folks can’t contribute in other ways, at least dropping in on [the Porchetta Industries] Discord to share war stories is always appreciated.”