Long favored by spooks and spies, OSINT is also a powerful weapon in the security pro’s armory
In July 2014, Malaysian Airlines Flight MH17, from Amsterdam to Kuala Lumpur, crashed some 50km from the Ukrainian-Russian border. All 298 passengers and crew on board the Boeing 777 lost their lives.
In the aftermath of the incident, separatists from Donetsk claimed to have shot down a Ukrainian transport aircraft, claims they later withdrew. The Ukrainian authorities said separatists had downed the airliner. The country’s president called the incident an act of terrorism.
Several independent and official investigations followed. One of the most comprehensive was conducted by Bellingcat, the investigative and citizen journalism group.
Bellingcat, along with other media organizations, used open source intelligence (OSINT) techniques and methodology to build a timeline of the incident, and to expose Russian claims and counterclaims as fabrications.
What is OSINT?
OSINT is intelligence “drawn from publicly available material”, according to the CIA. Most intelligence experts extend that definition to mean information intended for public consumption.
OSINT is information that can be accessed without specialist skills or tools, although it can include sources only available to subscribers, such as newspaper content behind a paywall, or subscription journals.
Read more of the latest open source security news
The CIA says that OSINT includes information gathered from the internet, mass media, specialist journals and research, photos, and geospatial information. Most of these sources were used in the Bellingcat MH17 investigation.
OSINT does not require its exponents to hack into systems or use private credentials to access data. Viewing someone’s public profile on social media is OSINT; using their login details to unearth private information is not. In intelligence agency terms, OSINT is also information drawn from non-classified sources.
When were OSINT techniques first used?
Open source intelligence predates the internet. Governments have long used newspapers, and later broadcasts, to track potential adversaries’ military, political, or economic plans and activities.
OSINT is low risk, cheap, and often highly effective, as corporate intelligence consultant Cameron Colquhoun has written in a Bellingcat article on the history of OSINT.
As Colquhoun suggests, OSINT fell out of fashion after World War Two, with intelligence agencies instead focusing on the more glamorous and dangerous world of HUMINT – human intelligence or spying – and SIGINT: signals and electronic intelligence.
But with the rise of the internet and social media, and online tools that can sift through vast amounts of information, OSINT is now more relevant than ever.
Not just for spies: OSINT and cybersecurity
Intelligence agencies use OSINT to track events, equipment such as weapons systems, and people. These are the ‘targets of interest’ (ToIs).
But hackers use OSINT to identify technical vulnerabilities as well as human targets for phishing and social engineering attacks. As a result, pen testing and security teams deploy similar techniques to find and close down weaknesses.
Read more of the latest social engineering news
“When explaining OSINT and how damaging it can be to clients, I do like the analogy of putting up a big poster in your front window with all your information on,” Liam Follin, penetration tester and web application security consultant at Pentest People, told The Daily Swig.
“We know where to look and we have the tools. But the nefarious side of hacking, the black hats, also know where to look.”
OSINT helps security teams unearth clues that individuals leave in the open that compromise security. Like using a vulnerability scanner to find flaws in systems, OSINT tools pick up on problem data, such as dates of birth, Social Security numbers, family members or even hobbies that could help attackers compromise an account.
OSINT techniques can be turned inward to gain information about your company’s exposure
Common OSINT techniques
There is no single playbook for OSINT: most pen testers have their own methods and preferred tools.
This often starts with manual reconnaissance, and reading up on the target subjects, including using non-technical sources such as an organization’s annual report, financial filings, and associated news coverage, as well as content on its websites and YouTube and similar services.
“Anything you can obtain via online as well as by traditional media research can be used for OSINT. And one of the simplest and yet most effective tools is using search engines,” Anita Bielicka, a cyber threat intelligence researcher at Orpheus, told The Daily Swig.
A hacker will also search for information on employees, cross checking against external social media and professional profiles.
“If I can identify the likely members of the development team and identify what they have done in public source code repositories, the technology meetups they have participated in, their blog posts, I can start to build a profile for how strong their skillset is and also an understanding of the types of mistakes they are likely to make,” Tim Mackey, principal security strategist at the Synopsys’ Cybersecurity Research Centre, told The Daily Swig.
“If I know what mistakes they’ve historically made, I can potentially build a targeted attack against that particular weakness in their coding… that all becomes part of the corpus of information we can use to define a potential attack.”
And although OSINT is often associated with social media and human tooIs, it can equally be used alongside security scanners to find problems with physical assets and IT systems, for example via a Shodan search.
RECOMMENDED Shodan founder John Matherly on dual-purpose hacking tools, and information overload
And researchers will look at GitHub, Stack Overflow, Reddit, vendor support forums, and even job listings for insights on a target’s technology.
“Since almost everyone uses LinkedIn, there is a massive amount of information there,” Andreas Georgiou, security consultant at Trustwave Spiderlabs, told The Daily Swig. “If you can find out how an email or username is constructed, you can look at past credential leaks, and have a good chance of finding a way into a network.”
The number of OSINT tools and services is constantly growing (image via osintframework.com)
OSINT in the open – examples of open source intelligence
Pentest People’s Follin recalls an OSINT engagement that found floor plans of a sensitive location online, and another where an online photo contained enough information to copy a keycard. Both could compromise the physical security of an organization.
This shows why OSINT is a valuable tool for raising security awareness, as well as a technical tool for identifying security risks.
“Organizations are potentially enabling cyber-attacks against themselves through the information they publish online,” James Dale, penetration testing and red team lead at PA Consulting, told The Daily Swig.
“OSINT is harvesting data from legitimate sources such as online search engines, websites, and professional social networks. But our cybersecurity experts have conducted client OSINT assessments and discovered information such as versions of software, names of devices used to print documents, and email addresses.
“Along with obvious sources, such as a company website and LinkedIn, this information can also be gathered through metadata stored within files created and published by an organization.”
Even fairly trivial information can have big security consequences, warns Dale. “A pet’s name, or the version of Office used to create a document, may seem insignificant, but it can be used to inform a potential cyber-attack,” he says.
Other increasingly important OSINT sources are open data feeds and geospatial information, from Google and other mapping tools.
And the use of OSINT can go even deeper, potentially right into the code of a company’s web applications.
If a security team can find those OSINT weaknesses first, they can move to close them down.
Social media has become a readymade OSINT channel for intel gatherers
Social media and OSINT
Although online media and search engines make OSINT quicker and easier, social media has been the most effective medium for gathering information on individuals with a view to defrauding them or stealing their identities.
“Social media has made [conducting] OSINT on people super easy,” Drew Porter, president and founder at security firm Red Mesa, told The Daily Swig.
"People willingly put their info out there. If a person is in scope, looking at their social media pages for 15 minutes will tell us more about the person than one hour of OSINT will most the time.”
Hackers, pen testers, and intelligence agencies view social media profiles on sites such as LinkedIn, Facebook, and Instagram as fair game for OSINT. Some firms use scraping tools, although these are against most sites’ terms, including LinkedIn for example.
This, of course, will not deter malicious actors. And although security teams have to be careful with social media, trawling these platforms can quickly reveal serious security flaws.
Read more of our latest deep dives into security topics and trends
“Social media analysis (or SOCMINT) is a subsection of OSINT, although its value can be hampered by privacy and platform restrictions,” Louise Taggart, manager for cyber threat detection and response, and Kirsten Ward, senior associate for threat intelligence, at PwC UK, told The Daily Swig.
“Sources can include social networking sites, professional networking sites, video sharing or vlog sites, or microblogging sites, for example.
“Social media platforms can betray a considerable amount of information on a person or organization, whether it is through a seemingly innocent picture, which could be the answer to a secret question, or a colleague wishing someone happy birthday and thereby inadvertently disclosing their date of birth.”
Security teams can use social media as an entry point for social engineering, or for physical site penetration. But the breadth of social media, including images and video, means it can all too easily provide malicious actors with information about security systems and IT, often without the business realizing. With no system compromise to detect, OSINT recon stays well below the radar.
“Social networks have long been used in pen testing and red team engagements,” Jordan Cheal, senior security consultant at Bridewell Consulting, told The Daily Swig.
“They can provide a host of information for performing passive reconnaissance or enumerating data that can be weaponized and used in phishing campaigns or for performing password sprays.”
Is OSINT legal or ethical?
In the US and the UK, OSINT is legal, but security teams need to stay within a clearly defined framework, which is agreed with their clients in advance of conducting OSINT.
Much will depend on where target information resides. OSINT that gathers information where there is a reasonable expectation of public access – a blog post or a LinkedIn profile, for instance – is generally considered legal. But where data are password-protected, obtained by deception, or anonymized and aggregated, the legality is less clear-cut.
RELATED ethicsFIRST: Maintaining ethical behavior across the cybersecurity industry
It often comes down to intent, says Red Mesa’s Porter. “When we are hired to do it for companies or high net worth individuals it is 100% ethical. When someone is doing it to stalk an ex, it’s not ethical.”
At PwC, Taggart and Ward point to the Berkeley Protocol, which sets out a framework for conducting open source investigations into war crimes and human rights violations, as a useful ethical standard for cybersecurity-related OSINT, too.
OSINT practitioners must ensure they are operating on the right side of the law
The right choice?
Should, then, security teams use OSINT?
If they are not already, they are overlooking a vital means of spotting, and removing, sensitive information from the public domain that could be abused by malicious actors to compromise an organization. And, unlike many other methods and tools, OSINT is largely free.
Set against these benefits are the legal and ethical considerations, and the fact that security teams must exercise a certain level of skill and caution to use OSINT effectively, ethically, and legally.
“It’s also important to bear in mind, that whatever a ‘good guy’ can find through OSINT, so can the ‘bad guy’,” warn PwC’s Taggart and Ward.
OSINT tools: An expanding list
Pentesters use a wide range of tools for OSINT, with consultants often using their own tools. Some of these, such as Pentest People’s Athena, are available on GitHub. Here are some other popular OSINT tools:
- Scrapesy: Scrapes both the clear web and dark web for exposed credentials
- O365 Squatting: Generates typosquatting permutations and cross-references them against Office 365 infrastructure to find potential phishing websites
- ZMap: Network scanner that discovers devices and services exposed to the internet
- Ghunt: Finds information associated with a Google ID
- Intel Owl: Pulls together threat analysis tool feeds into a single API
- ReNgine: Open source tool for aggregating recon feeds
- Shodan: IoT device search engine used to find unsecured equipment on LANs and other hardware-based weak spots
- Social Mapper: Developed by Trustwave Spiderlabs, Social Mapper uses facial recognition, as well as usernames, to track targets across platforms
- Spiderfoot: OSINT automation tool, available in open source and commercial versions
- Sublist3r: Python-based sub-domain enumerator
- theHarvester: Helps to “determine a company's external threat landscape on the internet” by gathering “emails, names, subdomains, IPs and URLs”
- Google dorking: Less a tool than a technique, Google Dorking involves using specialist search terms to find results not visible to natural language search
The SANS Institute has also published a detailed list of OSINT tools. Check out our Latest Hacking Tools page to keep track of the latest open source releases.
READ MORE Suspected Vietnamese cyber-spies targeting dissidents in Germany
