Generating Targeted Google Dorks with GPT-4o System Prompts
LLMs accelerate OSINT reconnaissance by producing lists of advanced search operators tailored to specific targets. Instead of manually crafting dozens of dorks, practitioners feed a precise system prompt into GPT-4o and receive ready-to-test queries within seconds.
A reliable system prompt begins by assigning a role and output constraints: "Act as a senior OSINT analyst. Create 12 distinct Google dorks focused on exposed documents, login portals and employee data for target domain acme.com. Use operators including inurl, intitle, filetype, site, cache and ext. Return only a numbered list with one dork per line. Avoid duplication."
This prompt consistently yields high-quality output when using temperature=0 and the model gpt-4o. Typical results include patterns such as intitle:"index of" inurl:backup site:acme.com, filetype:xls "username" "password" site:acme.com and inurl:admin intitle:dashboard site:acme.com.
These dorks can be validated manually or scripted using the googlesearch Python package. The advantage lies in discovering non-obvious combinations that a single analyst might overlook after thirty minutes of manual brainstorming. However, LLMs occasionally suggest operators that violate Google TOS or return zero results, requiring a quick follow-up validation pass before scaling the search.
Parsing theHarvester Output with Structured JSON Responses
theHarvester remains a cornerstone OSINT tool for enumerating emails, subdomains and hosts. Version 4.6.0 supports over 30 data sources. A standard invocation for a domain looks like: theHarvester -d example.com -b google,bing,duckduckgo -l 300 -f output.xml.
The resulting XML or console output contains mixed noise and valuable intelligence. Passing this raw text to GPT-4o with a JSON mode request converts the unstructured data into clean, machine-readable records suitable for databases or further enrichment.
import json
from openai import OpenAI
client = OpenAI()
def enrich_harvester_results(raw_output: str) -> dict:
response = client.chat.completions.create(
model="gpt-4o",
temperature=0.0,
response_format={"type": "json_object"},
messages=[
{
"role": "system",
"content": "You are an OSINT data engineer. Parse the provided theHarvester output. Extract unique emails, subdomains, IPs and hosts. Categorize risk level as low/medium/high based on exposure type. Return only valid JSON with keys: emails, subdomains, hosts, risk_summary."
},
{
"role": "user",
"content": raw_output
}
]
)
return json.loads(response.choices[0].message.content)
# Example usage after running theHarvester
# with open("output.txt") as f:
# data = f.read()
# enriched = enrich_harvester_results(data)
The temperature=0.0 setting combined with JSON mode eliminates creative hallucinations and guarantees parseable output. This pipeline reduces the time needed to triage 300+ harvested items from roughly 45 minutes of manual spreadsheet work to under five minutes of compute time.
Named Entity Recognition Pipelines Using spaCy and LLM Post-Processing
Combining classical NLP libraries with LLMs produces robust entity extraction from OSINT sources such as news articles, leaked credential dumps or public records. The spaCy library with the en_core_web_lg model handles initial tokenization and entity detection while GPT-4o resolves ambiguous or context-dependent entities.
Installation and model download use two commands: pip install spacy==3.7.5 followed by python -m spacy download en_core_web_lg. The following pattern loads a document, extracts PERSON, ORG and GPE entities, then asks the LLM to disambiguate and link them to known OSINT data points.
import spacy
from openai import OpenAI
nlp = spacy.load("en_core_web_lg")
client = OpenAI()
def extract_and_enrich_entities(text: str):
doc = nlp(text)
initial_entities = [(ent.text, ent.label_) for ent in doc.ents if ent.label_ in ("PERSON", "ORG", "GPE")]
prompt = f"Resolve these extracted entities in OSINT context. Merge duplicates and add known aliases or related domains:\n{initial_entities}"
response = client.chat.completions.create(
model="gpt-4o",
temperature=0,
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
sample_text = "Johnathan Doe, CTO at Acme Dynamics, spoke at the 2024 conference in Berlin about their new cloud platform."
print(extract_and_enrich_entities(sample_text))
This hybrid approach catches both statistical entities and those requiring world knowledge or recent context that pure statistical models miss. The spaCy model processes roughly 800 tokens per second on a modern CPU while the LLM call adds contextual reasoning at additional latency and cost.
Integrating Shodan API Results into LLM Analysis Loops
Shodan provides internet-wide device and service intelligence. After obtaining an API key, the CLI command shodan search --fields ip_str,port,org "port:22 country:US org:Example" returns structured data that can be piped directly into an LLM for vulnerability hypothesis generation or asset mapping.
A production pattern wraps the Shodan Python library inside a loop that queries multiple ports or banners, aggregates the JSON responses, then requests the model to identify patterns such as outdated software versions or exposed management interfaces. Using the shodan package version 1.3.0, the code looks like this:
After retrieval, the same structured-output technique shown earlier converts raw banner text into risk scores and recommended follow-up OSINT steps. The combination surfaces relationships that static Shodan filters alone would not reveal, such as correlating an exposed RDP service with the same organization’s public GitHub repository containing hardcoded credentials.
Multimodal OSINT with GPT-4o Vision Capabilities
GPT-4o accepts image inputs, enabling direct analysis of screenshots, maps, ID documents and social media photos without separate OCR tools. A typical OSINT workflow uploads a publicly available image and uses the prompt: "Perform OSINT on this image. Transcribe all visible text exactly. Identify any logos, landmarks, vehicle license plates, clothing details or architectural styles that could indicate geographic location. List any metadata inferences."
The model reliably extracts text from even low-resolution screenshots and suggests candidate cities based on visible building styles or vegetation. When chained with reverse image search APIs, this creates a closed loop that enriches an investigation rapidly. Cost per image analysis averages three to five times higher than text-only calls, so most teams reserve vision calls for high-value images only.
Building OSINT Agents with LangChain Tool Calling
LangChain version 0.2+ allows creation of agents that dynamically select among OSINT tools. A custom agent can be equipped with tools for WHOIS lookup, Shodan queries, theHarvester execution via subprocess, and web scraping. The agent receives a high-level goal such as “Gather all public information on target CEO Jane Smith” and decides which tools to call in sequence.
Defining tools with the @tool decorator and binding them to an agent powered by gpt-4o enables autonomous iteration. The agent can call a WHOIS tool, feed the registrant data into a search tool, then ask the LLM summarizer to produce a concise profile. Error handling and retry logic are essential because external OSINT APIs frequently return rate-limit or transient failures.
Developers usually set max_iterations=15 to prevent runaway token consumption. The resulting agent dramatically reduces context switching for the analyst while still requiring human oversight for final validation of conclusions.
Mapping Extracted Entities with Neo4j and LLM-Generated Cypher
After entity extraction, storing relationships in a graph database reveals hidden connections. Neo4j Community Edition 5.18 paired with the official Python driver allows storage of nodes for Person, Organization, Domain and Location. The LLM can generate Cypher statements directly from enriched OSINT data.
A prompt such as “Convert this JSON of entities and relationships into valid Cypher CREATE statements using MERGE to avoid duplicates” produces executable queries. The resulting graph supports visualization in Neo4j Bloom and complex queries like “Find all persons connected to both acme.com and a known data-breach domain within two hops.”
This pattern turns flat CSV exports from earlier pipeline stages into interactive intelligence graphs that scale to thousands of nodes without performance degradation on modest hardware.
Evaluating Reliability, Cost and Privacy Trade-offs
LLM-assisted OSINT introduces three primary trade-offs. First, models can hallucinate relationships or misclassify risk even at temperature 0; every critical finding therefore requires at least one independent verification step using primary sources. Second, token costs accumulate quickly: processing 500 harvested emails plus associated context can consume 40,000 input tokens and 8,000 output tokens per run with gpt-4o. At current pricing this equals roughly $0.28 per analysis cycle, acceptable for targeted investigations but expensive when scaled across hundreds of domains weekly.
Third, sending raw OSINT data containing PII to third-party APIs creates privacy and compliance risks. Two common mitigations are self-hosted models via Ollama running Llama-3-70B or Mistral-Large, and local embeddings with sentence-transformers for similarity search without network calls. The local route trades accuracy and speed for complete data sovereignty.
Teams often adopt a hybrid model: lightweight tasks stay local with spaCy and Ollama while complex reasoning or multimodal analysis uses GPT-4o only on redacted or non-sensitive subsets.
End-to-End Python Pipeline for OSINT Data Enrichment
The following complete script ties together dork generation, theHarvester execution, entity extraction and graph preparation. It demonstrates production-grade error handling and logging that developers can adapt to their own environments.
import subprocess
import json
from pathlib import Path
from openai import OpenAI
import spacy
client = OpenAI()
nlp = spacy.load("en_core_web_lg")
def run_theharvester(domain: str, output_path: Path):
cmd = ["theHarvester", "-d", domain, "-b", "google,bing", "-l", "200", "-f", str(output_path)]
subprocess.run(cmd, check=True)
return output_path.read_text(encoding="utf-8")
def llm_structured_extract(raw_text: str):
response = client.chat.completions.create(
model="gpt-4o",
temperature=0,
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": "Parse OSINT data into structured JSON with keys: persons, organizations, emails, subdomains, relationships."},
{"role": "user", "content": raw_text[:8000]} # truncate to fit context
]
)
return json.loads(response.choices[0].message.content)
def main():
domain = "example.com"
raw = run_theharvester(domain, Path("harvester.xml"))
structured = llm_structured_extract(raw)
# Further entity enrichment
for person in structured.get("persons", []):
doc = nlp(person)
print(f"Extracted entities from {person}: {[ent.text for ent in doc.ents]}")
Path("enriched_osint.json").write_text(json.dumps(structured, indent=2))
print("OSINT enrichment pipeline completed.")
if __name__ == "__main__":
main()
This script executes in under two minutes on average hardware and produces a ready-to-import JSON artifact. Adding Neo4j insertion or automated reporting requires only a few additional functions. The pattern scales to small business security audits or startup due-diligence investigations while keeping the core logic readable and maintainable.
Choosing Between Cloud LLMs and Local Models for OSINT Tasks
Cloud models deliver superior reasoning on ambiguous OSINT data but introduce latency of 800-2500 ms per call and recurring expense. Local models such as those available through Ollama 0.1.30 with the command ollama run llama3:70b keep all data on-premise and incur zero marginal cost after hardware investment.
Local inference on an RTX 4090 workstation achieves approximately 35 tokens per second for 70B parameter models, sufficient for batch processing of harvested text but noticeably slower for interactive agent loops. Many teams therefore route simple classification and extraction to local models while reserving cloud LLMs for final synthesis and hypothesis generation. This split optimizes both cost and confidentiality without sacrificing overall pipeline effectiveness.