I open-sourced the architecture behind my onion-indexing search engine. Isolated Tor crawlers, JSONL pipeline, Elasticsearch. Looking for people to poke holes in it.

Followed up on my last post about the DNS leak problem with onion crawling. Here's the actual architecture I landed on for AcademiaVault:

Crawlers run in isolated processes, each routed exclusively through Tor with no fallback to system DNS. Surface web crawlers are a completely separate pipeline – different process pool, different network namespace, zero shared state. Every document gets normalized into JSONL before it touches the index: url, domain, source_type, title, content, snippet, category, language. That JSONL layer is the choke point where ethical crawl rules get enforced – robots.txt respect, rate limiting, no auth-walled scraping.

From there it lands in Elasticsearch, tuned for full-text search and linguistic analysis across four categories: computer-science, cybersecurity, networks, ethics.

The part I'm least confident about is scaling the ingestion pipeline without breaking compartmentalization. If anyone's built something similar – distributed crawlers with hard network isolation – I'd want to compare notes. What would you change about this setup?