Streaming Mention Analytics with Search and Alerts (Bloomberg)

0) Problem Restatement

Bloomberg asked: design a system that ingests high-volume text streams (news articles, social posts, press releases), detects mentions of entities (companies, tickers, people), counts mentions over time windows, lets users search recent mentions, and alerts subscribers when mention volume for an entity spikes (e.g., "AAPL mentions are 8x normal in the last 10 minutes").

1) Requirements

• Ingest thousands of documents per second from many sources.
• Extract entity mentions with good accuracy ("Apple" the company vs the fruit).
• Real-time counts per entity per minute, plus sentiment optionally.
• Search: "show recent mentions of TSLA with 'recall'".
• Alerts on spikes, with low false alarms. Latency from publish to alert under ~1 minute.

2) Architecture

flowchart LR
    SRC["News wires, social feeds"] --> ING["Ingestion + dedupe"]
    ING --> K[("Kafka - documents")]
    K --> NER["Entity extraction + disambiguation"]
    NER --> KM[("Kafka - mentions: entity, doc, time, sentiment")]
    KM --> AGG["Windowed counts per entity"]
    AGG --> TS[("Time-series store")]
    AGG --> AD["Spike detector vs baseline"]
    AD --> AL["Alert service - subscriptions"]
    AL --> U["Users / terminals"]
    KM --> IDX[("Search index - recent mentions")]
    U --> API["Search / chart API"]
    API --> IDX
    API --> TS

3) Key Stages

1. Ingest and dedupe: many outlets republish the same story. Remove exact duplicates by content hash and near-duplicates with SimHash/MinHash, so spikes aren't just syndication.
2. Entity extraction: named-entity recognition finds candidate names, then entity linking maps them to IDs (Apple Inc. → AAPL) using context (words like "iPhone" or "shares"), a dictionary of aliases and ticker symbols ($AAPL), and a model for ambiguous cases. Output: {entity_id, doc_id, ts, snippet, sentiment}.
3. Aggregation: a stream processor counts mentions per entity per minute (event time, with watermarks), and stores them in a time-series DB for charts.
4. Spike detection: compare the current window with a baseline for that entity and time of day (e.g., the rolling mean and standard deviation of the last 4 weeks, same hour). Alert when the z-score is > 4 and the absolute count is above a minimum (small entities jumping from 1 to 5 shouldn't alert).
5. Alerts: users subscribe to entities or watchlists with thresholds. The alert service deduplicates (one alert per entity per spike episode), and includes top headlines as context.
6. Search: index mentions with text, entity, source and time in a search engine, with time-based indexes (recent days hot).

4) Scale and Latency

• Partition mention streams by entity ID for aggregation. Hot entities (big tech on earnings day) may need pre-aggregation.
• NER models are the heaviest step, so scale them horizontally (GPU workers for large models, fast dictionary matching first).
• Target: document → alert in under 60 seconds.

5) Wrap-Up

Ingest and deduplicate text streams into Kafka, extract and link entity mentions with NER plus alias dictionaries and context, and publish mention events. Aggregate mentions per entity per minute in a stream processor (stored for charts), detect spikes against seasonal baselines with both relative and absolute thresholds, and deliver deduplicated alerts to subscribers, while a time-partitioned search index serves recent-mention search.