Projects

The problem

Threat-relevant posts are spread across services that are slow to browse, awkward to search and impossible to monitor consistently. The practical result is that the same pages get re-read over and over, and the genuinely new material is easy to miss in the noise.

What it does

The engine is a background collector for Tor hidden services. It crawls the onion sites you configure, filters what it finds against a keyword set, extracts indicators of compromise, asks a local language model whether a post is actually relevant and what it is about, stores the result, and pushes the matches to a chat channel. It runs on a schedule and keeps its own record of what it has already seen.

Verified capabilities

• Tor onion service crawling. Crawls configured onion services asynchronously through a local Tor SOCKS5 proxy, following links up to a configurable depth and page limit.
• Keyword pre-filtering. Target keywords are read from the KEYWORDS environment variable. With LLM_PREFILTER_KEYWORDS enabled, page text is screened for those terms before any inference runs, so irrelevant pages cost nothing.
• Local LLM classification. A local model classifies threat relevance and writes concise incident summaries. Nothing is sent to an external API, so the material being collected stays on the machine.
• Indicator of compromise extraction. Extracts cryptocurrency addresses (BTC legacy, SegWit and Bech32, ETH, XMR, LTC, BCH), contact handles (Telegram, Tox, Keybase, email), v3 onion URLs, PGP key blocks and SHA-256 hashes from raw page text.
• Deduplication and retention. Content hashes are stored in SQLite and enforced through deduplication indexes, so the same post is not alerted on twice. Records older than the retention window are purged.
• Telegram alert dispatch. Relevant findings are queued and delivered as formatted HTML alerts, with automatic backoff when the bot API rate limit is hit.
• Circuit rotation and scheduling. The Tor control port is signalled to renew circuits on an interval so crawling does not stall on a dead circuit. Background jobs handle health checks, scheduled scraping, seed synchronisation and log cleanup.

Technologies

Running it

The repository ships an installer that installs system dependencies, configures Tor to expose a SOCKS port and an authenticated control port bound to loopback, creates a virtual environment, downloads the model file, applies firewall rules and runs the test suite. It can be registered as a systemd service or run manually from the command line. A 4 GB machine is the floor for CPU inference with the 1.5B model, and CUDA acceleration is detected and used automatically when a compatible GPU is present.

Not built yet

The upstream README lists the following as planned work rather than existing features, and none of them are present in the current implementation:

• A lightweight web dashboard for uptime, health metrics and historical alerts.
• Additional alerting channels — Discord webhooks, Slack and direct email.
• Export pipelines for STIX 2.1 and TAXII.
• A custom rule engine for user-defined matching patterns and severity scores.

Scope and intended use

The project’s own disclaimer restricts it to authorised threat intelligence collection, defensive monitoring and academic research, and asks operators to satisfy the laws that apply to them. It is built for that purpose, and the same expectation applies here.