LinkAtlas Crawler

active

The Undead Web Crawler

The LinkAtlas Crawler is an undead web-crawling machine that never rests. This horizontally scalable microservice powers the LinkAtlas search engine with tireless, zombie-like persistence. It features smart crawling with HTTP and Puppeteer fallback, distributed rate limiting, bloom filter deduplication, and circuit breaker reliability. Like a horde of zombies, it scales across multiple workers, always hungry for more pages to index while respecting robots.txt.