Once described as one of the most dangerous sites on the internet, Shodan is now a staple in the security professional’s toolkit
INTERVIEW In 2009, bioinformatics graduate John Matherly set up Shodan, a search engine that can discover all and any devices connected to the internet.
But Shodan’s prominence – and its controversy – comes from its ability to discover insecure IoT devices.
The free-to-use service can find IP cameras, TV sets, fridges, and coffee makers, as well as industrial infrastructure and control systems, plus conventional servers and routers.
This makes it is one of the few, if not the only, tools that can truly map the internet of things.
Shodan is now widely used by security professionals to analyze their network footprint, and identify connected devices with poor, or even no, security.
‘I discovered a love of large-scale data analysis’
Matherly was born and grew up in Switzerland, but moved to the US when he was 17.
At university he worked for the San Diego Supercomputer Center, and after graduation as a freelance software engineer.
“I took a bachelor’s degree in bioinformatics. In the process, I discovered a love of large-scale data analysis which I ended up applying to internet crawling,” he explains.
John Matherly
“The original idea for Shodan was to create a service that could provide trends on software that is used on the internet. For example, who is using Apache or Nginx or how fast do Cisco customers patch their devices.
“The goal was to offer a Netcraft on steroids: provide real-time market intelligence about everything that’s on the internet.”
Security was not part of the plan for Shodan: at least, not at first.
“The use case I designed Shodan for was market intelligence, not security, risk, fraud detection or the many other things our data are used for today,” Matherly said.
“I happened to be interested in an area of networking that turned out to be a lot more relevant in the future. And I had a background in data collection and analysis, which combined to create something new that came out at the right time.”
He certainly did not expect Shodan to grow to the size it today.
“I wish I could say that I had the foresight that refrigerators would get connected to the internet, but I didn’t,” he admits. “And I got lucky that people in the security community discovered Shodan.”
Shodan is the world’s first search engine for internet-connected devices
Growing in concert with the internet
The path to today’s Shodan service was, Matherly says, “slow and steady”. The current service, with data crawlers around the world, multiple availability regions, and its own IT in a local data center is a long way from the early days.
“The Shodan backend has actually been re-engineered multiple times over the past decade. In the early days, most of the crawling happened from a few US-based servers and was stored in a refurbished Dell Vostro in my closet,” he admits.
Shodan, though, has grown as the internet has grown, and changed dramatically in its scope and reach.
Today, it provides a range of monitoring services to businesses, and security teams use this to form a map-like view of their internet presence. Matherly has been described as an “internet cartographer”.
Read more of the latest IoT security news
“OT [operational technology] network security became a hot area for startups, threat intelligence became the thing to have and most organizations are either hybrid or entirely on the cloud,” he says.
“From my perspective, networks used to be a lot more homogenous 10 years ago. Nowadays networks can contain links to industrial control systems, or the smart coffee machine in the break room, alongside the usual desktops and servers. And all of them could be spread across multiple internet service providers but tied together over a VPN.”
And this adds up to a real challenge for security teams.
Security teams use Shodan to obtain a map-like view of the organization’s internet presence
On dual-purpose tools
Shodan’s automated crawlers do work that a human security monitoring team would struggle with. That struggle only becomes more difficult as the number of connected devices grows.
But Shodan can be used offensively as well as defensively. Does this bother Matherly?
“That duality is common for all security tools. By nature, everything in security can be used to either protect yourself or to find the weakness in others.
“Much like Google, we’ve taken many steps to limit access for offensive use cases, such as limiting the number of searches or downloads, while creating tools to make the defensive use cases more approachable, such as Shodan Monitor.”
RECOMMENDED IoT security: Singapore launches labeling scheme for WiFi routers and home hubs
And commentators asking for Shodan to be banned are less common than they were.
“We don’t hear that sort of criticism much anymore,” he says. “I believe in the early days of Shodan people hadn’t caught on to how connected the world became within the past decade.
“Twenty years ago, there were clear lines between a server, desktop, and your refrigerator. That isn’t as much the case anymore and people have started to realize that connectivity has exploded,” Matherly acknowledges. “So, we need a new breed of tools that can provide visibility.
“There’s also been misinformation spread about what Shodan does and I think we’re [now] better at explaining what we actually do.
“We don’t attempt to login to anything and we don’t try default passwords.”
Fixing a hole
Nonetheless, those poorly configured devices could be on sensitive networks. Has Shodan been asked to remove results, say for critical infrastructure, or a hospital?
“Yes, but we don’t remove search results,” explains Matherly. “In those cases, we work with the organization to fix the problem as soon as possible and then update the information in Shodan to reflect that the problem has been fixed.
“Bad guys are constantly scanning the internet using botnets so hiding a result on Shodan isn’t an effective strategy for protecting an insecure service.”
Instead, Matherly says security professionals should integrate Shodan into their monitoring.
“Make sure you’ve setup Shodan Monitor so you automatically get notifications if something on your network gets exposed to the internet,” he recommends.
“Secondly, take the time to look at the raw data that Shodan collects. The website only shows a small fraction of what’s available in our database and maybe some of the additional metadata can help you decide whether it’s something to investigate further.”
Information overload
But a challenge most – if not all – security teams currently face is what Matherly describes as “dashboard and alert fatigue”. This type of information overload, he believes, is one of the greatest challenges facing the information security profession.
“A decade of growth in security has left the industry with a lot of different products, all of which have their own dashboards and alerting systems,” he explained.
“I’ve spoken to people where their first question is what their daily routine with Shodan should look like. That basic premise is crazy to me.
“The software shouldn’t require daily check-ins on their dashboard or require you to look through a list of alerts and filter out the noise. I hope that security products will become more streamlined so security pros can spend more time doing what they’re good at and less time checking in to their various services to collect the data.”
YOU MIGHT ALSO LIKE Latest web hacking tools – Q1 2020
Shodan, though, is about more than vulnerability monitoring. The search engine, Matherly says, is about understanding the shape and scope of the internet.
“I have a background in science so fundamentally I believe that having data is useful for making better decisions,” he says.
“In this case, understanding the internet and how it’s changing can help guide policies at the national level – how secure is our electric grid? – all the way down to the individual – how secure is your refrigerator?
“There are many use cases we’ve demonstrated where Shodan can provide unique and valuable data that wouldn't be available otherwise.”
Is IoT regulation the way forward?
The type of information Shodan has uncovered over the last decade does though raise questions about who is responsible for securing connected devices, whether they are part of critical national infrastructure or in the home.
“I do believe that the government has a role to play in ensuring the devices are secure by default,” he explains.
“I don’t believe that the average user should be expected to make the right decisions and keep up with security best practices. The vendors should be responsible for ensuring their devices are secure to the best of their ability.
“In my view, most end-users aren’t willing to pay a premium for a product with better security. As a result, the market won’t put pressure on companies to offer more secure products.
“Consumers in other countries might be more security or privacy-sensitive in their choice of solutions but I don’t think that’s been a driving factor in the US.”
READ MORE Knowledge transfer: Bugcrowd’s Casey Ellis on IoT bug bounties and live hacking events
