I was going through this web crawler system design problem, and the basic loop is almost deceptively simple: fetch page → parse HTML → extract links → repeat

The more interesting problems show up once you try to do this across billions of pages. For example, adding more crawler workers doesn’t automatically make the system better.

If 500 workers all discover URLs from the same domain, you can easily hammer that website unless the rate limit is coordinated globally per domain.

Then failures make things even more interesting:

• worker crashes after downloading the page
• website times out
• DNS starts becoming a bottleneck
• the same URL is discovered thousands of times
• different URLs return identical content
• one domain stays down and keeps getting retried

A design I like is to split fetching and parsing into separate durable stages: Frontier → Fetcher → raw HTML storage → Parser → discovered URLs → Frontier

That way, if parsing fails, you don’t need to hit the external website again. And with a durable queue, worker crashes become mostly a retry problem rather than a lost-work problem.

My takeaway: designing a large crawler is less about “how do I fetch pages fast?” and more about how do I avoid wasting work or becoming a terrible neighbor on the internet?

Full web crawler system design: [link]

Preparing for your next interview?

Chill Interview tracks recent interview experiences and recurring question patterns across top companies at here