OSINT tools that will speed up online research

Carder

Professional
Messages
2,616
Reaction score
1,934
Points
113
Almost all occasionally, you may need to go into a new area, to conduct research or to quickly find a very specialized information. But the search engine doesn't always understand us. We try very hard to explain: we introduce large sentences to replace short phrases. We search through pages and links. And, probably, after a few hours, we find in some pdf document a request that should have been entered from the very beginning.

Sometimes, it's not always easy to Express your information needs in a search query. You may not have enough terminology, or you are not quite sure what you are looking for, and when you see the possible options, you will immediately understand what you need. Or it may be difficult to put what you're looking for into a concise search query at all.

We've put together OSINT tools that will make online research faster, more useful, and possibly more enjoyable. Most of the collected resources are in English, free of charge and without registration.

They will help you understand what to look for in General​


1. OneLook

AAaRrjjrfRM.jpg

This is a dictionary that selects words and phrases that somehow relate to your query. Here you can quickly find definitions of certain words — the result will come from different dictionaries.

2. Keyword Tool

The platform displays keywords for the entered query in any language and country. Some requests even show how popular they are, even though this service is paid for. You can search for keywords on Google, YouTube, Twitter, Instagram, Amazon, eBay, Play Store, Bing.

When searching on Google, you can, for example, select keywords that contain question words or prepositions. And on the left there are filters where you can search for keywords already in the resulting search results.

v5HzC937BH8.jpg


3. Answer the Public

Here you enter the word you are interested in and get variants of queries with it that people search for. There is a choice of languages, including Russian. This is what the section with queries containing question words looks like.

EhRcgXCT1xY.jpg

The only downside is that in the free version, you can only search for about 3-4 queries a day.

The handle will lead you to the desired source​

These are site reference pointers. In them, sites are divided into thematic folders. You need to choose a topic of interest and see what the platform's editors have carefully selected for you.

v6FnwdCoZXE.jpg

It will help you understand what information you need by offering your own options. Allows you to find the exact site by navigating you from folder to folder and shortening the search area.

4. Web Directory

4SAB1W7BDxI.jpg

The selection here is not very large, but it can be useful. Each link has a short description from the platform's editors, which reduces the time required to view each resource.

Sort the output by topic groups​


5. Open Knowledge Map

ZwdFv4fiLKw.jpg

The resource provides all materials with open access from BASE (a search engine for a huge number of scientific materials) and PubMed (a database of medical and biological publications). You can filter by content format: audio, books, maps, patents, reports, SOFTWARE, videos, research theses, and so on.

A feature of the resource is that it shows search results in the form of thematic circles. They touch and intersect. The closer the circles are to each other, the more related their themes will be. In the center are those that have the strongest subject similarity with all other circles on the map. On the right, you can search for keywords already within the resulting map. By hovering the mouse over a circle, you can see how many documents there are on the topic of this circle. If you click on the circle, it increases, and you can select a suitable document.

6Q_cEo0gCbM.jpg

Moreover, the platform hasSOFTWAREthat allows you to visualize your own set of documents in the same way.

The downside is that the platform produces only the 100 most relevant results. Although for some, on the contrary, this is a plus, as it reduces the search time.

6. Carrot2

This is a search engine that distributes search results into thematic folders and shows them as paychart and trimap. You can choose the type of clustering algorithm. Right here:

phiNBRXrQBo.jpg

By the way, if when searching, Carrot shows you this, then just change the country in the VPN and refresh the page.

RJGjpzo_lnQ.jpg

Search → select the folder of interest → there will be results on the right.

This is what the Treemap looks like if you search Carrot for information about metamodernism:

zscR3eHC-bg.jpg

The platform more convenient that looking from 15 search engines (including Google, Yandex, Bing).

7. BizNar

oVhiIdLRnkY.jpg

BizNar is presented as a resource for finding business information: market research and various industries. The platform searches on the Deep web (not the same as the Dark web) — results are returned that are not indexed by the usual search engines. Includes search for patents and social networks.BizNar also packages the output in thematic folders and in the form of a paychart (click "visual" on the left).

After entering the query, there is a graph at the bottom left showing in what year how many materials on the topic were published. You can select a year, and the output will display publications only for this period.

R5A5EjaZLag.jpg


8. World Wide Science

Search through international and national research databases. You can enter the request in any of the suggested languages (including Russian). If you want to receive email notifications about new content, you will need to register. However, this feature is only available for English sources.

The output can be filtered by various indicators, and it is also sorted by topic.

ATIlEnCIWDs.jpg

There is a map on the home page. When you click on the country of interest, a list of databases that the platform can search for is displayed.

P2MSc0RdUAI.jpg


Everything possible in one place​


9. Fagan Finder

NB_5MXLixI8.jpg

The platform contains a huge number of different search resources, divided into groups. Here you will find, for example, search engines only for encyclopedias, only for documents, for articles from magazines, or, in the end, for presentations. In General search, you can search not only by images and videos, but also by audio or by groups and forums. On the first page, scroll through below and you can get short guides to each source.

What is collected here may not be found through a regular search engine, because the source of information may not be online, the search may not consider it important enough, or it may not be published in the form in which the search engine has access to it.

It will find a mention of your query in all popular social networks and not only that​


10. Hash At It

The tool searches by hashtags, keywords, URLS, and profile mentions. It immediately returns the latest results, but by selecting a specific social network in the filter, you can quickly see older results as well.

jk5cO5A8EX0.jpg

If you register on the site, you will be able to subscribe to a specific hashtag or request to immediately receive new posts on the topic.

11. Social Search

DRehyTAA-Hc.jpg

Searches for mentions in both social networks and search engines. Interestingly, the list of social networks that he searches for includes VK. Usually, foreign search engines do not include it. Here, too, you can set up e-mail notifications about new publications based on your request.

Quick search for news to keep up to date with all the trends on the topic of interest​


12. Watson News Explorer

ynVZRzx7GQk.jpg

This is a Bank with an update. Here you can explore how individuals, businesses, and news relate to each other. The platform will tell you which topics, people, and organizations are more or less related to your request. The map on the right shows what country is - the source of information, you could filter the results by country.

You can configure the search so that it will only search by people, companies, or non-profit organizations. And advanced search will allow you to find the relationship between two queries.

The resource provides data for the last month, and a time line is displayed at the bottom, when which materials were published.

NxZWZ-KjjVo.jpg

This is a resource with all possible news, including in different languages. You can simply search by keywords for all messages, or you can choose a country and subject. Also, by the way, here you can search for radio and podcasts in different countries. If you click on "Classic" in the upper - right corner, you will see a more detailed search-everything is sorted by topic and sub-topic, and it is easier to search. There is a mobile app.

13. News Now

7YwrFE_mwhY.jpg

Here's all the latest news in one pile. The small checkboxes next to the title indicate where the message is coming from. You can filter news by country or topic. You can also see which news items are most viewed and read here.

14. All You Can Read

1M1eP7lN-TY.jpg

Choose a country, and the issue gives all possible online media that are available in this country. Plus, you can sort by specific topics. One of the largest news resources with magazines and Newspapers from all over the world. However, keyword search is not provided here.

They will deliver the necessary information directly to your e-mail address​

These tools will track the appearance of new content on the topic, as well as help you monitor changes on the sites you are interested in.

15. Google Alert

FIDyU1T6eA8.jpg

The tool automatically notifies you by e-mail about new materials on the topic. In order to use it, you need a Google account.

And so, we enter the request that we want to follow. Open "More options" and see how we can configure our search. "Auto-select" here means that information will come from all kinds of sources. But you can also choose one of them. For example, you are only interested in new books on the topic. "Only the best results" in the "Quantity" section means that information will only come from resources that have a good reputation.

16. Visual ping

yH0TZ_DYPys.jpg

Are you waiting for the results of the contest, discounts on air tickets, or are you monitoring changes in legislation? This resource will send you a notification when something changes on the link you specified. The program takes a screenshot and checks it against the previous one. This is how it notices changes.

The Chrome extension helps you create new notifications directly from the site you want to follow. You can also create group notifications in Slack (for more information on how to set them up, see the FAQ at the very bottom). It's a great tool for tracking changes from competitors.

The "actions before checking" function helps you automate a specific action before a new screenshot is checked. For example, you need to enter a password, close the pop-up window, or click on a button. Learn more about how to use it here and here (more advanced usage).

MbGOd4K6QFM.jpg

You can set up notifications so that they are sent to you only when the keyword you specified appears.

FYdN3SnmGnw.jpg

The free version has restrictions on the number of checks.

17. Watch That Page

r1YIsxSsy3s.jpg

This resource is simpler than the previous one. For example, it doesn't analyze changes in images, only text and links. However, for personal accounts, you can perform an almost unlimited number of checks here.

All updates to the specified sites are sent in one email. But you can also divide links by topic — the resource will send them in separate emails. For each site, you can choose: you just want to find out that something has changed there and check it yourself; or you can immediately receive these changes in an email. This section explainshow to add new tracking sites faster.

Extensions that will speed up your regular search​


18. Selection Search

Install the extension → select the text you are interested in → the extension offers options where you can search for information for this query.

fjnrNxu4_Fs.jpg


19. Infinite Scroll for Google

Here everything is very simple — you don't need to click on the next page of the search results again. You just scroll down, and the extension automatically opens new results for you.

20. Search Preview

2jrUzftvHDA.jpg

The extension allows you to immediately see what the site looks like, so that you don't need to visit it in vain. Supports Google, DuckDuckGo, Yahoo, and Bing.

What else might be useful​


21. Wolfram Alpha

Ob-YlMu08sw.jpg

This is a kind of search engine that does not search for resources where you can find answers, it immediately gives answers. It analyzes various types of files, makes complex calculations, gives accurate characteristics of the selected object, compares it, and so on.

To understand how to work with it, the resource offers application examples. Wolfram Alpha is perfect if you need to quickly find out some accurate data on work in the field of Finance, sociology, medicine, transport, chemistry, engineering, statistics, linguistics, and so on. It can be used for training and just for answering everyday questions.

22. Gigablast

ZG6N-VPZFpU.jpg

This is a simple and intuitive search engine, there are no ads, there is privacy and a directory of sites. As well as a large list of dorks for efficient search.

23. Symbol Hound

A search engine that does not ignore special characters such as&,%, and π, so it is useful when searching for information about the selected programming language.

24. Digle

mAh51_wLmK8.jpg

In the end, you can ask someone to find the information you need for money or earn money on the search yourself.
 

Intelligence and data collection - overview of the OSINT Toolkit​

OSINT involves collecting and analyzing publicly available information, mainly from online sources. In the context of cybersecurity/pentest, OSINT is most often used to collect public data about a company, and this applies not only to information about the email addresses of its employees. No less interesting will be the information about:
  • DNS names and IP addresses;
  • open ports;
  • running network services;
  • availability of remote access services;
  • unprotected applications and operating systems;
  • existing security mechanisms.
Fortunately, there are many tools available for running OSINT, and today we'll take a look at some of those that can help you gather information about an organization and build its digital footprint.

Console tools
Nmap
Nmap ("Network Mapper") is an open source utility for network research and security verification. It was designed for fast scanning of large networks, although it also works well for single targets. This is an ageless classic and the first tool that is used during the pentest. Its functionality is quite extensive, but in our case it will only need to identify open ports, the names of running services, and their versions.

Code:
# nmap -v -sV -p1-65535 pentestit.ru

tLkVRGGAaYE.jpg


You need to scan not only the TCP port range, but also UDP:

Code:
# nmap -sU pentestit.ru

88VsYly4H74.jpg


This information will help you determine possible entry points to the organization's network perimeter.

Dmitry
Another console tool for finding information about hosts of interest. The basic functionality allows you to:
  • collect possible subdomains;
  • collect email addresses;
  • get uptime information;
  • perform a TCP port scan (scans the first 150 ports).);
  • perform whois searches and much more.
It is recommended to run the tool using all available keys:

Code:
# dmitry -winsepfb pentestit.ru

Dsb25ktqBvY.jpg
Dsb25ktqBvY.jpg

  • -i - Perform a whois search by the host's IP address;
  • -w - Perform a whois search by host domain name;
  • -n - Get information from Netcraft.com by host;
  • -s - Search for possible subdomains;
  • -e 0 Search for possible email addresses;
  • -p - Perform a TCP port scan on the host;
  • -f - Perform a TCP port scan on the host and show filtered ports in the report;
  • -b - Read the banner received from the scanned port.
The functionality is quite extensive, but in terms of the capabilities and speed of detecting open ports, it is still inferior to the previous tool, although this does not prevent using them together. Still, such questions require the use of information from different sources for greater reliability.

TheHarvester
TheHarvester is a complete framework for collecting email addresses, subdomain names, virtual hosts, open ports/banners, and company employee names from various open sources. Allows you to perform both passive search in several search engines: google, yahoo, bing, shodan.io, googleplus, linkedinetc., and active, for example, searching subdomain names in a dictionary.

Code:
# theharvester -d pentestit.ru -b google -l 500

T-hE7UPOC9U.jpg

  • -d - domain;
  • -b - search engine;
  • -l - the number of search queries to process.

Recon-ng
Recon-ng is also a full-featured web intelligence framework written in Python. The package includes independent modules, interaction with your own database, convenient built-in functions, interactive help, etc. due to the similarity of the Recon-ng interface with Metasploit-framework, it will be much easier to get used to it if you have had experience using the latter. You can install all available modules using the command: marketplace install all when you first start the tool, because they may initially be missing. Some modules interact with the API of various services, such as Google, Github, Bing, etc. they will need to get an API key and specify it with the command keys add. You can view the list of keys using the command keys list where the table will be displayed.

Ukesz3wvN10.jpg


To view the basic commands, call help:
xVpek1U4e8U.jpg


TiDos
TiDos is a universal platform that covers everything from initial data collection to vulnerability analysis. When starting the program, you must enter the domain of interest and whether SSL is used:

5dKpNGUzyZY.jpg


In its Arsenal, TiDos has 5 sections that allow you to search for information about the target in open sources, as well as search for certain vulnerabilities. To select it, just specify the corresponding ID.

cEJFFXqLjHo.jpg


Since we are only interested in collecting information, we will consider only one section containing about 50 modules. Here you can perform passive data collection:
  • search for addresses and other contact information on the Internet;
  • domain information (Whois information);
  • DNS configuration information;
  • list of subdomains;
  • list of subnets, etc.

6zmZS-fI6mM.jpg


peRK4rJn_WM.jpg

so active:
  • collecting banners;
  • checking files robots.txt and sitemap.xml;
  • CMS definition;
  • defining alternative versions of the site by using different User-Agent parameters;
  • file search info.php and its possible variations, etc.

7fxvGJ0g2NE.jpg


5QTWhUJ1oVw.jpg


Online services
Using online services is also one of the possible ways to collect goal data. There is a generalized database of such services - this is osintframework. Here various tools are collected and sorted by sections. By the way, it also provides links not only to online services, but also to useful tools and frameworks related to a specific section, such as TheHarvester or Recon-ng — for OSINT.

Gg2SrXbTrS8.jpg


Search for email addresses
hunter
The service is quite easy to use. On the site, enter the name of the company, or its domain, and get a list of addresses that were found along with the resources where this data appeared. A plug-in is also available for installation in the Mozilla Firefox browser to automate the process of collecting accounts.

_MJRODuYyfM.jpg


mailshunt
Business analogs-specify the company's domain and get a list of found accounts. Different services may work according to different algorithms, so we recommend using multiple services together to get a more extensive list.

Whois
who.is
We enter the company's domain and get whois information about the domain, DNS records, and so on in the response.

ConrGZpreB0.jpg


2whois
In addition to receiving whois information, the site provides online services such as nslookup, dig, DNS analysis, and much more. Very useful service.

Fv6QxODpu9s.jpg


Hackertarget
The service is similar to the previous one, but the set of tools is significantly expanded, which allows you to additionally use, for example, vulnerability scanners OpenVAS, Nmap, Nikto, WhatWeb, etc.

oycyYbSRGns.jpg


Google Dorks
Google Dork Queries (GDQ) is a set of queries for identifying everything that is not properly hidden from search robots. Here is a small list of commands that are most commonly used:
  • site - search by specific site;
  • inurl - indicate that the search words should be part of the page/site address;
  • intitle - search operator in the title of the page itself;
  • ext orfiletype-search for files of a specific type by extension.
By combining the search commands in various ways, you can find almost everything, up to the administrator's login/password.

7OKDTQYahyw.jpg


Conclusion
Collecting information is an important step in conducting penetration testing. Periodic reconnaissance will help you detect potential flaws in the network perimeter before intruders take advantage of it.
 
OSINT-intelligence based on open sources of information. Links to resources.

https://www.hashatit.com
Search by hashtags, keys, and links in social media profiles.

https://wn.com/#/search
Search by news mentions. It is convenient if you need to collect information about a company or a public figure.

http://graph.tips/beta/
Functional Facebook search.

https://mattw.io/youtube-geofind/location
Search on YouTube with a specific geolocation, you can watch videos from the place of interest.

https://getcontact.com
A popular app that shows how a specific phone number is recorded by other people. There is a caveat it parses your phone book and adds it to the database. You can use it on the left handset.
 
Top