News Aggregator (Google News)

0) Problem Restatement

Design a news aggregation service like Google News. It collects articles from thousands of publishers, groups articles about the same story, and shows each user a fresh feed. Users can browse by topic (Sports, Tech), open an article (which takes them to the publisher's site), and get a personalized "For you" feed.

This problem was asked many times at Rippling, in several versions:

• Focus on pulling articles on a schedule, storing them, and reading by topic.
• Make sure a feed never shows duplicates, even when each publisher gives each article a unique URL.
• Design the frontend feed (React, pagination, virtualization, offline).

1) Requirements

1.1 Functional

• Fetch new articles from publisher feeds (RSS/APIs) regularly.
• Store article metadata: URL, title, summary, image, publisher, topic, publish time.
• Group articles about the same story, and show one story with "5 more sources".
• Serve topic feeds and a personalized feed, newest or most relevant first, paginated.

1.2 Non-Functional

• Fresh: breaking news appears within a few minutes.
• Fast reads: feed loads in under ~200 ms.
• Read-heavy: many more reads than writes.
• No duplicates within a user's feed.

1.3 Scale Estimates

• 50,000 sources, ~2M new articles/day ≈ 25 articles/sec (small).
• 50M daily users × 10 feed loads = 500M reads/day ≈ 6K/sec, with spikes during big news.
• Storage: 2M × 2 KB = 4 GB/day. We only need recent articles hot (e.g., 30 days).

1.4 API Design

• GET /v1/feed?topic=tech&cursor=...&limit=20
• GET /v1/feed/for-you?cursor=... (personalized)
• GET /v1/stories/{story_id} (all sources for one story)
• Internal: POST /v1/sources (register a publisher feed)

2) High-Level Architecture

2.1 Overview

• Fetch Scheduler: decides when to poll each source.
• Fetchers: download feeds, parse new items and push them to a queue.
• Ingest pipeline: normalizes URLs, deduplicates, classifies the topic, and clusters articles into stories.
• Article store: articles and stories (SQL or a document DB).
• Feed builder: precomputes topic feeds (sorted lists of story IDs) in Redis.
• Feed API: reads the precomputed lists, applies personalization and filters out seen items, then hydrates details from cache.

2.2 Architecture Diagram

flowchart LR
    SCH["Fetch Scheduler"] -->|"due sources"| FQ[("Fetch queue")]
    FQ --> F["Fetchers"]
    F --> PUB["Publisher RSS / APIs"]
    F --> IQ[("Kafka - new articles")]
    IQ --> P["Ingest: normalize, dedupe, topic, cluster"]
    P --> DB[("Articles + Stories DB")]
    P --> FB["Feed builder"]
    FB --> R[("Redis - topic feeds")]
    U["Users"] --> API["Feed API"]
    API --> R
    API --> C[("Article cache")]
    C --> DB

3) Data Model

sources:   source_id, feed_url, avg_publish_interval, last_fetched_at, next_fetch_at, etag
articles:  article_id, canonical_url (unique), url_hash, title, summary, image_url,
           source_id, topic_id, published_at, story_id, content_fingerprint
stories:   story_id, headline_article_id, topic_id, first_seen_at, last_updated_at, article_count
Redis:     feed:topic:{topic_id} → sorted set of story_id by score (recency/importance)

4) Key Flows

4.1 Scheduling pulls

• Each source has next_fetch_at. The scheduler picks due sources (an index on next_fetch_at) and queues them.
• Adaptive frequency: a source that publishes every 5 minutes is polled every few minutes, while one that posts weekly is polled every few hours. If a fetch finds nothing new, wait longer next time. If it finds a lot, poll sooner.
• Use HTTP ETag/If-Modified-Since so unchanged feeds cost almost nothing.
• Be polite: limit concurrent requests per publisher domain.

4.2 Ingesting an article

1. Normalize the URL (remove tracking parameters like utm_source, lowercase the host) and hash it. If the hash exists, skip it (exact duplicate).
2. Classify the topic, using the source's category or a text classifier.
3. Cluster into a story: compare the article with recent stories in the same topic using text similarity (e.g., embeddings or MinHash on the title and summary). If it's close enough, attach it to that story. Otherwise, create a new story.
4. Update the story's score and add or bump it in feed:topic:{id}.

4.3 Reading a feed

Read story IDs from the Redis sorted set with cursor pagination, fetch story details from the cache, and return them. The "for you" feed merges the topic feeds the user follows and re-ranks them with the user's interests.

5) Deep Dive A — No duplicates

There are three layers of dedup:

• Same URL: canonical URL + hash, with a unique constraint in the DB.
• Same article on different URLs (syndicated copies like AP stories on many sites): a content fingerprint (e.g., a SimHash of the text) catches near-identical text.
• Same story from different publishers: story clustering, so the feed shows the story once, with other sources inside it.
• In the user's feed, also remove stories the user already saw (keep recent seen IDs per user, with a Bloom filter to save memory).

6) Deep Dive B — Freshness, spikes and the frontend

• Breaking news spike: feeds are precomputed and served from Redis plus CDN caching for anonymous topic pages (a 30-second TTL), so traffic spikes don't hit the DB.
• Precompute vs on-read: topic feeds are shared by millions of people, so precompute them. The personal feed is cheap to assemble on read from a few topic lists.
• Frontend (React): fetch the first page quickly and load more on scroll (infinite scroll with cursors). Render only visible rows (virtualization) so long feeds stay smooth. Cache the last feed for offline reading. Show hover cards for hashtags and mentions with lazy loading.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you rank?" Combine recency, the number of sources covering the story (importance), source quality and user interest. Old stories decay over time.
• "Paywalled or removed articles?" Store a status. A periodic check (or publisher signals) marks removed articles and hides them.
• "Local news?" Tag articles with locations and add a location filter to the feed.

9) Wrap-Up

Poll publisher feeds with an adaptive scheduler, send new items through Kafka to an ingest pipeline that normalizes URLs, fingerprints content and clusters articles into stories, and store them. Precompute topic feeds as Redis sorted sets, serve them with cursor pagination and caching, and build the personal feed by merging and re-ranking topic lists while filtering out already-seen stories.