# Search Engine (Google) \
\
---\
\
0) Problem Restatement\
Design a web search engine like Google that can crawl billions of web pages, index content, and return relevant search results in milliseconds. Core challenges include distributed crawling at scale, building inverted indexes for fast lookups, ranking results by relevance, and handling billions of queries per day with low latency.\
\
1) Requirements\
\
1.1 Functional\
- Crawl and index web pages from the internet.\
- Accept user search queries and return relevant results.\
- Rank results by relevance (quality, freshness, user intent).\
- Support autocomplete/query suggestions.\
- Handle different query types (informational, navigational, transactional).\
- Display search results with title, snippet, and URL.\
- Support advanced search (filters, date range, site-specific).\
- Respect robots.txt and website politeness.\
1.2 Non-Functional\
- Scale: Index 100 billion+ web pages.\
- Latency: Return search results in < 200ms (P99).\
- Throughput: Handle 100K+ queries/second.\
- Freshness: Re-crawl updated content regularly.\
- Relevance: High-quality results ranked by multiple signals.\
- Availability: 99.99% uptime.\
- Storage: Petabytes of indexed data.\
1.3 Scale Estimates\
- Web pages: 100 billion pages to index.\
- Crawl rate: 10K pages/second.\
- Daily queries: 8 billion queries/day (~100K queries/sec avg).\
- Avg page size: 100 KB.\
- Index size: 100B pages × 100 KB = 10 PB (raw HTML), 1-2 PB (compressed index).\
- Storage: Original pages + inverted index + metadata = ~15 PB.\
---\
\
2) High-Level Architecture\
\
2.1 Overview\
- Crawling Pipeline: Crawlers → Content Parser → Indexer → Inverted Index.\
- Query Pipeline: User Query → Query Processor → Searcher → Ranker → Results.\
- Key components: Distributed crawlers, inverted index, PageRank, query processing, caching.\
2.2 Architecture Diagram\
```mermaid\
---\
config:\
layout: elk\
---\
flowchart TB\
%% Crawling Pipeline\
Web[\