OSINT stands for open-source intelligence
OSINT is the collection and analysis of data gathered from open sources (overt sources and publicly available information) to produce actionable intelligence. OSINT is primarily used in national security, law enforcement, and business intelligence functions and is of value to analysts who use non-sensitive intelligence in answering classified, unclassified, or proprietary intelligence requirements across the previous intelligence disciplines.
What began as curiosity soon collided with ideology.
But code has gravity, and mine began to pull.
~ Anonymous
When I first started the OSINTBuddy project 4, nearly 5 years ago now (wow, time flies) I wasnāt entirely sure what I was getting myself into. As it turns out, I was building a mirror, one that reflected back ethical issues and holes in my belief system. I used to be a big fan of individuals such as Snowden and Julian Assange and RMS type characters, I still am, but I no longer know how to fit those beliefs in with the fact that I am building a system that essentially amounts to being a surveillance/intelligence tool. I tell myself it isnāt mass surveillance but that doesnāt bring me much comfort anymore.
It started as a weekend crawler scraping CSE links I found on pastebins, bookmarking sites, and defunct forums. No real vision, no roadmap, just curiosity, caffeine, and a desire to build something. What I didnāt realize was that curiosity scales faster than conscience. While OSIB started simply, a Google CSE (custom search engine) crawler that would crawl hundreds of CSE links I collected from various places on the internet. We have since evolved into a fairly general data collection toolkit that will run any Python script you fancy to collect, transform, and amplify OSINT investigations similiar in style to Maltego (If any Maltego employees are reading this, Iām a huge fan of your work, thanks for inspiring me! :).
The first iteration of OSINTBuddy was uploaded to Github on August 25, 2021
The repo shows that date but I recall working on the project before that without uploading the code to Github, I was still a relatively new developer professionally speaking and I didnāt have many ambitions outside of building up a small portfolio of public work. But OSINTBuddy changed everything, this is the first project of dozens I started that I actually stuck with. It was the first project that hooked me deeply enough to refine and to imagine beyond the prototype.
The README was much less ambitious at this time as you can see:
And hereās a preview of what the old UI looks like, I still suck at design but at least I can claim Iāve improved since then:
At the time I was spending tons of time learning about open source and free software and the differences in their philosophies and licensing. I absolutely loved how free/open source software gave me the ability to glance at the code huge companies used and having the ability to see how renowned developers approached problems felt almost like real life ācheat-codesā. Free software was like a teacher that never tired, never judged. You could literally step through the logic of someone smarter than you with the code they wrote to gain deeper insights into real-world software problems and the developers mindspace. Iāll forever have eternal gratitude for the free/open source communities, these spaces have their warts, but theyāre beautiful spaces nonetheless.
Outside of learning lots from reading free/open source code it also gave me a sense of techno-optimism which I still hold to this day, albeit slightly less so considering current events. I ended up settling on a free software (AGPL) license for OSINTBuddy rather than a more permissive license like MIT. The idea of keeping free open source code āfreeā and in āthe commonsā really resonated with me. I wasnāt really thinking too deeply about the implications and consequences of what I was creating as the project was merely a crawler. A free open source crawler wouldnāt be able to do much harm, right?
Well, that brings us to today, OSINTBuddy isnāt only a single crawler anymore, itās going to be dozens of data collection scripts, visualization tools, and more. Making OSINT easier makes it more accessible to the wider public and sure it doesnāt invent new capabilities but it does collapes the ācost curveā, and every time you collapse a cost curve, you democratize both discovery and abuse. Thatās good. And alsoā¦ complicated.
Removing the barriers of needing to think and have experience in open source intelligence analysis all while rapidly speeding up the process of collecting such intelligence makes OSINT practitioners more efficient and lets new people make use of these practices for good, whatever that may be. However it also opens up the avenue of making it easier to utilize OSINTBuddy for malicious and nefarious purposes. Itās a tradeoff and an ethical dilemma Iām still grappling with to this day. I still donāt know where the ethical boundary lies between visibility and violation. The same transparency that once felt liberating now feels precarious. Open code used to mean accountability, now it can also mean weaponization. Maybe there isnāt a fixed ethical boundary to stand behind at all, maybe itās a moving horizon that recedes as we walk toward it, and our job as builders is to keep chasing it without pretending itās fixed. Iāll try my best to keep my eyes open, hands steady, and my code honest.
OSINTBuddy taught me how to build faster. Itās now taking me longer to learn when to stop. I still believe in the commons. But every time I run OSIB, I wonder what else Iāve set in motion.