OSINT stands for Open Source Intelligence and is the practice of scraping the internet for publicly available information. This information, when examined individually, may seem relatively innocuous but when you gather enough individual pieces and can consider them together you can often build a eerily clear picture of someone or something.
As an example, if you have a Facebook account chances are your birthday is listed somewhere on it. By itself, this doesn’t seem like a big deal. We give our birthday out all the time to verify our age. Besides, all you may use Facebook for is exchanging dank memes with your best friends and the occasional family photo that your mom tagged you in.
Individually, none of this seems that important but when taken altogether it starts to sound a lot like some very common security questions that you have to answer to recover lost account passwords for emails, banks or your iCloud. Collecting this information could allow a malicious actor to take control over these accounts.
Buscador is a pre-built Linux virtual machine that can be downloaded from https://inteltechniques.com/buscador/ that comes pre-configured with dozens of tools to make gathering and correlating information on people and companies easy. While it sounds scary, and it is, it’s important to realize that there are plenty of legitimate uses for performing this kind of information gathering like finding and scrubbing your own personal data to secure your own identity, or for investigative journalists digging to bring light to the next breaking story.
I will be using Oracle’s VirtualBox as my hypervisor, so I downloaded the Buscador .OVA file that corresponds to VirtualBox:
Once the file is downloaded, open up VirtualBox, select the
File menu and click on
Import Appliance In the window that pops up, click on the little folder icon next to the input box and navigate to the .ova file you just downloaded:
Go ahead and click
Import to import the VM. We will adjust the system settings after the import rather than tweak them here. Once the import is complete, you should see the machine on the left panel of the VirtualBox dashboard. Select the new machine, then from the
Machine menu at the top open the
Settings On the General tab, you can rename the machine to whatever suits you:
Next, select the
Advanced tab in the
General options and set
Shared Clipboard to bidirectional:
Under the System option’s Motherboard tab, it is recommended you set the Base Memory to half your available total, although I am limiting mine to 8GB:
Next, navigate to the
Display options and set
Video Memory to 128 MB if available:
Finally, you’ll want to switch to the Shared Folders menu and choose a folder on your host system to share to this machine and select auto-mount:
With this final setting change, close the settings window and go ahead and start the VM. Upon boot, you will log in to the
osint user with the password
osint. Once at the desktop, go under the
Devices VirutalBox menu at the top and select
Insert Guest Additions CD. Allow the guest additions to install, reboot the machine and log back in:
If you have the same experience that I did with the guest additions installation, the installation will fail with an unknown error. You can download the VirtualBox Guest Additions .iso file directly from the website, being sure to grab the one for your version of VirtualBox. In my case I am running v6. When you download the file, select the option to automatically open it and allow it to auto-run upon completion. It should prompt you for a the password for your account, which is
osint, and then successfully install from there. After it finishes, reboot again:
If all went well, we should be able to enter full screen display, utilize our shared folder and copy files and text in and out of our machine. The last thing we need to do to setup is to update the core system, and the OSINT scripts specific to Buscador. To update the core system, click on
Activities in the top left, and search for
Download and install all updates available though Package Updater:
Afterwards, open a terminal by clicking on the black square icon on the dock to the left of your desktop. If prompted to update say yes:
At the terminal prompt, type
update_scripts. NOTE: As of this writing, the repository that this script pulls seems to be missing so this script is broken. As I have come across tools that don’t work, I have been able to manually update each to get them working and this will be shown when we talk about those tools.
With our setup out of the way, we want to completely shutdown the machine and create a snapshot so we can easily restore to a fresh install. This is important for a few reasons. Firstly, a lot of the tools we will use can generate a large amount of files and data. This can quickly fill up a virtual drive, so it’s a good idea to start every investigation with a fresh system. Secondly, and possibly most important, if you are using Buscador to perform actual legal investigations that will result in submitting findings as evidence in court, it is extremely important to maintain forensic integrity meaning you don’t want any residual data from unrelated investigations that could call the integrity of your evidence in to question.
Once shut down, select the hamburger menu to the right of your Buscador VM and switch to the snapshot menu. Select
Take at the top, give your snapshot an intuitive name and save it:
That concludes the initial setup of Buscador. It is a good idea to perform the updates each time you start the machine from the fresh snapshot and create a newer updated snapshot before starting an investigation. This way you will always be working with the latest packages and features.
Video Downloader is a utility that takes in a URL that contains a video, and will download it locally for further assessment. Like most of the functionality in Buscador, underneath the useful GUI provided by this custom virtual machine lies an elaborate python back-end. The official site for this python utility is https://ytdl-org.github.io/youtube-dl/download.html and contains useful documentation on the script itself. To launch Video Downloader, locate and click the red rectangular icon with a white play button in the dock:
Upon launching, you will be presented with a simple window asking for the URL of the page with the video or videos you would like to download. Paste your URL in and click OK. You will be greeted by an animated progress window and then the file explorer will automatically open to the default download directory (/home/osint/Video_Utilities/Youtube-DL/):
When I tested this with the URL in the picture, it appeared to progress normally but when the directory popped open it was empty. Unfortunately with
update_scripts being broken, it seems like the current version doesn’t work anymore. I was able to fix this by manually updating the youtube-dl application by opening the terminal and issuing:
sudo pip install --upgrade youtube-dl :
With the update complete, launch Video Downloader again and paste your URL in. Afterwards, you should see a new file in the file explorer that pops up:
As a final note for this section, this utility will download all videos found in the URL that you submit. While we pasted a URL in that corresponded to a single YouTube video, had we submitted a URL to,say, https://www.youtube.com/user/forevertrollin421/videos it would download every video that the highly trained unprofessional John Maxwell, aka The Harley Tech, has uploaded to his YouTube channel. This can be really useful to grab a lot of videos with one easy command, but be aware that it can eat up a lot of time and disc space.
As an aside: If you prefer 2 wheels over 4, check out his channel I just linked and find him on Instagram and Facebook because he puts out some great content that constantly makes me want to abandon computers altogether and switch careers.
This menu option available in the dock is an easy to use interface to a powerful video utility called FFmpeg. You can open it by clicking on the blue bubble with the white play icon on it:
This tool can play pretty much any video format, from the usual formats to corrupted files to security camera footage that has no associated video player. When you click on the icon in the dock, you will be asked to select a video file to open. Navigate to whatever video file you are interested in and click OK. Next, FFmpeg needs to know what you want to do with this file and it can do a LOT:
Play a video is relatively self explanatory, so all I will say is it will try to use every codec it knows to open and run the file. It’s probably safe to say that if FFmpeg can’t play it, it can’t be played.
The second option listed is
Convert a video to mp4. Since FFmpeg can play files that can’t always be played elsewhere, this allows you to convert it to a format that is commonly playable so you can distribute it to your stakeholders, or transfer it to one of your other systems where this tool isn’t available. Note, I am not a lawyer and I have no law enforcement experience but if I had to guess I would imagine that the converted file is probably not forensically sound since it’s technically altered. Use your best judgement if you are working in that domain.
The next option,
Extract video frames, will process the video and open a directory with each frame of the video exported as a still image in .bmp format. This is good for gathering important snapshots from a video to submit as evidence or include in reports. By default, the stills will output to /home/Video_Utilities/ffmpeg_out/<current_date_time>-frames/:
As you can see in the screenshots, it creates a lot of images taking up a lot of space. The video used in this example was 76 MB in size and the file output was 13.2 GB. This is why we take snapshots of this machine and start with a fresh instance for every investigation.
The next two options that FFmpeg presents us are very similar.
Shorten video (low activity) and
Shorten video (high activity) both do exactly what it sounds like. They will analyze the video and remove frames where there is nothing interesting happening. This can be good for long surveillance videos that need to be reviewed. The difference is in the tolerance used for determining what is interesting. the low activity version has a strict tolerance, and will keep sections that have pretty much any movement at all, where the high activity version is better suited for videos with a good deal of background movement like outdoor security cameras that might capture birds or pedestrians off in the distance.
Finally, the last option allows you to extract just the audio of the provided file out into an mp3 file. This can be situationally useful as you can imagine.
Metagoofil is a command-line utility that will take a domain that you provide and go out to collect all kinds of documents found related to the site. To launch it, click on the Domains Interact option in the dock and select Metagoofil from the resulting popup:
Enter your target domain in the next window and click OK. Select the maximum number of documents you would like to collect and click OK again:
Once it completes it’s run, it will store the results in /home/Domains/Metagoofil/ but will not open the directory automatically like many of the other tools on Buscador:
As you can see in the following screenshot, choosing my own domain as the example target was a bad idea because there was nothing to find. However you can still see the general format of what it will output, including an HTML report of it’s findings:
In general there will be 4 main files that are generated by this tool. Firstly, there will be a file called Full.txt that contains the full text of all metadata found in all documents found on the domain. Second will be an Authors.csv file, which will contain a list of names of all the people that wrote the documents that were found. Companies.csv will contain only information on companies that was found in the metadata, which helps identify associations between your target and other companies that may be relevant to the investigation. Last but not least, Modified.csv lists names of anyone who has modified the files found. This further assists in building a list of names of people associated with your target.
One important piece of information that you usually want to find when performing an OSINT investigation against an organization are employee email addresses. These can be used to for phishing campaigns, which is a common method of gaining initial access into a corporate network as long as it is in scope. One good tool to do this is Harvester. Harvester can be found under the Domain Interact menu on the dock where you previously found Metagoofil:
This time you will want to select Harvester as seen below:
It will prompt you for the domain of your target. Since I don’t have an email for my domain, I opted to use a different domain which I have blurred in all screenshots out of respect. This is all publicly available information, but I don’t see any need to put anyone specific in the spotlight here:
Harvester searches many different sources in order to find email addresses associated with your target domain that have been publicly announced on the internet. These sources include: Baidu, Google, Twitter, Bing, DuckDuckGo, VirusTotal and many more. It will generate two reports in /home/Domains/TheHarvester related to your target domain, one HTML report for easy viewing and one XML report for easy consumption by other tools:
Subdomain Enumeration Tools
There are a few tools included in Buscador that allow you to take a single domain name and quickly discover many subdomains associated with it. These include:
These can all be found under the same
Domain Interact dock menu. While these tools all perform similar tasks, they do vary in the sources that they pull from so it’s a good idea to run several. With that said, I personally run only two tools for this purpose which are Amass and Subfinder. While Subfinder is not part of Buscador, these two tools will have their own dedicated post in this blog. Given how similar all these tools are, I will leave the full write up for those posts, for now it is enough to know that they will output a list of all the subdomains available for a given domain.
Social media is a goldmine of oversharing. For whatever reason, people are comfortable blasting all kinds of personal information out on sites like Twitter and Instagram for fake internet points. Given such a trove of valuable information like birth dates, addresses, the name of their favorite security question and when they will be away from their house it’s no surprise that some decent tools exist to pull this information out of the proverbial ether(net). In Buscador, these tools can be found on the dock under the
Social Networks option:
Firstly, a custom script aptly named
Twitter script takes a Twitter handle and will pull all available photos, tweets and a list of friends:
The files will be stored in /home/Social/Twitter/Exporter/<handle>/:
Clearly, I need to step up my Twitter game since I only have one picture of my cat.. You get the point though.
There is one more Twitter tool that is publicly accessible outside of Buscador and that is Tinfoleak. Tinfoleak seems to be much more powerful than the Twitter script, but that comes with a layer of complexity that isn’t always necessary so both have their place. Under the same
Social Networks menu, this time select
Tinfoleak and click OK. Here you can select and configure many different options to customize what you want to capture. Lets collect everything I can to see what happens:
The output report will be save in /opt/tinfoleak/Output_Report/<handle>.html:
When you open the report, you will see that it gathered a LOT of information. You will see every tweet, your tweet-to-like ratio, who’s following you and who you are following along with a lot of other data that lends to a report that is too large for me to get a good screenshot of. Here is just the beginning of it:
One other social media scraping script available is instaLooter, a python based tool that works similarly to the Twitter script but scrapes Instagram account data instead of Twitter. I discovered that this script is broken in the same way that youtube-dl was broken and that we could again fix it by issuing the command:
sudo pip install --upgrade instaLooter:
As I don’t have an Instagram account myself, I will once again use John Maxwell as an example. Open instaLooter from the Social Networks menu option on the dock and enter an Instagram handle:
Once it completes, you should have a directory pop open with the information that it was able to scrape from the targetted account. Unfortunately, I didn’t make the appropriate sacrifices to the Demo Gods and it is not returning results for me right now to grab a screenshot of, so I will continue to troubleshoot the script and update this post if I get it working. In the mean time, if YOU get it working, be sure to leave a comment below to tell me what you did.
The last tool that I want to cover is Spiderfoot. Spiderfoot is a tool that takes in a domain name as a target and queries over 100 sources to gather information on everything from IP addresses, names, account names, emails and websites with the same account names registered. It creates a large report with graphs and tables to visualize this data and allow you to examine the details. There is a lot of correlation and guessing involved, and can possibly have a lot of false positives if your target doesn’t have a large web presence but it provides a good starting point for large scale investigations of a persons or organizations activities online.
It is availble as a bookmark at the top of the Firefox browser in Buscador:
When you first open it, you won’t have any scans listed so click on `New Scan` at the top to configure and start your first one. There are four modes to run a scan and each one can take a relatively long time to run. For my example, I ran a Footprint scan to gather information on the network perimete, associated identities and othe rinformation that is obtained through web crawling and search engines:
Once the scan completes, which took 2 hours for my example, you can see a list of categories of information that found along with the counts of datapoints for each group. By clicking on the type, you can drill down and see the specific datapoints. For my example, almost everything ended up being false positives since I have a very common name and not a large web presence, however I did discover a couple accounts on websites from 10 or 15 years ago that I had completely forgotten about.
In the tab graph, you can see the relationships between the datapoints, which your originally seeded target highlighted in red. This is a good way to judge how closely related the data points are. If someting is directly connect to your seed node in the graph, theres a lot higher confidence of the relationship to your target than if it’s linked to a different node, three or 5 nodes away.
OSINT is an enormous area of study which represents a great benefit and a great threat to everyone. The people that want to use this information to harm others will continue to craft their own tools and scripts to do so, so having a publically accessible and easy to use resource like Buscador is vitally import to allow us to discover what we have exposed and take necessary steps to protect ourselves.
While this was not an exhaustive list of the tools available in or out of Buscador, the tools covered here represent a great entry point for those of us justgetting started in our OSINT journey.
If you found this information and these tools interesting, head over to IntelTechniques and show support, and definitely pick up Michael Bazzells book Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information. It’s a great read and covers this information and a lot more in a much greater depth, and I am excited to continue reading through it.
If you found this content helpful, please consider donating your favorite cryptocurrency below.
- Bitcoin cash
- Ethereum classic
Donate Bitcoin to this address
Scan the QR code or copy the address below into your wallet to send some Bitcoin
Donate Ethereum to this address
Scan the QR code or copy the address below into your wallet to send some Ethereum
Donate Cardano to this address
Scan the QR code or copy the address below into your wallet to send some Cardano
Donate Xrp to this address
Scan the QR code or copy the address below into your wallet to send some Xrp
Donate Litecoin to this address
Scan the QR code or copy the address below into your wallet to send some Litecoin
Donate Stellar to this address
Scan the QR code or copy the address below into your wallet to send some Stellar
Donate Bitcoin cash to this address
Scan the QR code or copy the address below into your wallet to send some Bitcoin cash
Donate Dogecoin to this address
Scan the QR code or copy the address below into your wallet to send some Dogecoin
Donate Usdcoin to this address
Scan the QR code or copy the address below into your wallet to send some Usdcoin
Donate Aave to this address
Scan the QR code or copy the address below into your wallet to send some Aave
Donate Uniswap to this address
Scan the QR code or copy the address below into your wallet to send some Uniswap
Donate Wrappedbitcoin to this address
Scan the QR code or copy the address below into your wallet to send some Wrappedbitcoin
Donate Eos to this address
Scan the QR code or copy the address below into your wallet to send some Eos
Donate Cosmos to this address
Scan the QR code or copy the address below into your wallet to send some Cosmos
Donate Synthetix to this address
Scan the QR code or copy the address below into your wallet to send some Synthetix
Donate Dash to this address
Scan the QR code or copy the address below into your wallet to send some Dash
Donate Maker to this address
Scan the QR code or copy the address below into your wallet to send some Maker
Donate Dai to this address
Scan the QR code or copy the address below into your wallet to send some Dai
Donate Ethereum classic to this address
Scan the QR code or copy the address below into your wallet to send some Ethereum classic