# Web Crawler \
\
---\
\
0) Problem Restatement\
Design a distributed web crawler that can efficiently traverse the internet, download web pages, extract links, and index content for a search engine. The core challenges are avoiding duplicate crawling, respecting politeness policies, handling failures, and scaling to billions of pages.\
\
1) Requirements\
\
1.1 Functional\
- Crawl web pages starting from seed URLs.\
- Extract and follow links from downloaded pages.\
- Store page content and metadata.\
- Respect robots.txt and politeness policies.\
- Support incremental and recurring crawls (freshness).\
- Handle different content types (HTML, PDF, images, etc.).\
1.2 Non-Functional\
- Scalability: Crawl billions of pages across millions of domains.\
- Politeness: Avoid overloading servers (rate limiting per domain). Example: If you crawl *example.com/page1*, wait 1 second before crawling *example.com/page2*. But you can simultaneously crawl *other-site.com/page1* (different domain)\
- Robustness: Handle failures, malformed HTML, redirects, timeouts.\
- Efficiency: High throughput (thousands of pages/sec).\
- Freshness: Re-crawl important pages regularly.\
- Deduplication: Avoid crawling duplicate URLs and content.\
1.3 Scale Estimates\
- Total pages: 10 billion pages to crawl.\
- Crawl rate: 10,000 pages/second.\
- Avg page size: 100 KB.\
- Storage: 10B × 100 KB = 1 PB (raw HTML).\
- Recrawl interval: Monthly for most pages, hours/days for critical content.\
---\
\
2) High-Level Architecture\
\
2.1 Overview\
- Frontier (URL queue) → Crawler Workers → DNS Resolver → Downloader → Content Processor → Storage (HTML, metadata, index).\
- Key challenges: URL deduplication, politeness enforcement, distributed coordination, failure recovery.\
2.2 Architecture Diagram\
```mermaid\
---\
config:\
layout: elk\
---\
flowchart TB\
%% Seed URLs and Frontier\
Seeds[\