Veila active
A search engine for the Tor network — crawls and indexes onion services so the hidden web is searchable like any other.
- role
- author & operator
- language
- Python
- transport
- Tor SOCKS5
- storage
- SQLite
- status
- active
Overview
Veila sets out to do something simple but surprisingly hard: make the Tor network searchable. It crawls and indexes onion services so you can explore that corner of the web the same way you’d explore any other — type what you’re looking for, get a list of places to start. Onion services are unindexed by traditional search engines, and that gap is exactly what Veila exists to fill.
How it works
seed .onion list ──▶ fetch via Tor ──▶ extract links ──▶ store ──▶ queue next
- Seeding — starts from a list of known v3 onion addresses.
- Crawling — fetches each page through Tor’s SOCKS5 proxy, rate-limited to stay a good citizen of the network.
- Extraction — discovers new onion links with regex matching for 56-char base32 v3 addresses plus DOM parsing of anchor and link tags.
- Storage — a SQLite database tracks each address, where it was discovered, first-seen / last-checked timestamps, and status (queued, unreachable, http_200).
- Queue — addresses are processed FIFO and marked as checked after each attempt.
On Tor’s reputation
Tor hosts marketplaces and other unsavory corners, and some of them are indexed here — pretending otherwise would be dishonest. But the same is true of the open web, which we don’t dismiss as nothing but fraud and abuse because it’s also how we learn, organize, and speak freely. Tor is a tool; its character comes from how it’s used. Veila indexes what’s there and gives people a way to navigate it.