Real-Time Market Data Distribution and VWAP (Bloomberg)

0) Problem Restatement

Bloomberg asked two related questions:

1. Multiple exchanges (NASDAQ, NYSE, NSE, ...) send market data to loader servers. The loaders distribute several copies of each message over UDP to processor servers. Design this pipeline so that processors see every message exactly once and in order, quickly.
2. Build a real-time VWAP provider. VWAP (volume-weighted average price) = Σ(price × volume) ÷ Σ(volume) over a window. It's the average price actually traded, weighted by size. Compute it for thousands of symbols from high-volume trade ticks, and publish it to subscribers.

1) Requirements

• Ingest trades and quotes from many exchanges (millions of messages/sec at peaks).
• Deliver to processors with very low latency (microseconds to low milliseconds).
• No lost or duplicated messages, and the per-symbol order is preserved.
• VWAP per symbol for windows (e.g., since market open, and rolling 5 minutes), updated in real time.
• Subscribers get updates for the symbols they care about.

2) Architecture

flowchart LR
    EX1["Exchange A feed"] --> L1["Loader A"]
    EX2["Exchange B feed"] --> L2["Loader B"]
    L1 -->|"UDP multicast - copy 1"| P["Processors - partitioned by symbol"]
    L1 -->|"UDP multicast - copy 2"| P
    L2 -->|"UDP multicast - copies"| P
    P --> V["VWAP state per symbol"]
    V -->|"updates"| PUB["Publisher / fan-out"]
    PUB --> S["Subscribers - terminals, apps"]
    L1 --> RR[("Retransmit / replay store")]
    P -->|"gap request"| RR

3) Reliable Delivery over UDP

UDP multicast is used because it's fast and one packet reaches many receivers, but packets can be lost, duplicated or reordered.

• Sequence numbers: every message from a loader (per channel or partition) carries an increasing seq.
• A/B arbitration: each message is sent on two independent paths (copy 1 and copy 2). The processor takes whichever copy arrives first and drops the other (it's a duplicate seq). This hides most single-path losses with no delay.
• Gap detection: if the processor sees seq 101 then 103, it knows 102 is missing on both paths. It asks a retransmit service (which keeps recent messages) for 102, and buffers 103+ briefly to keep order. If recovery is too slow, mark the symbol "stale" rather than show a wrong price.
• Snapshots: a processor that restarts or joins late loads a snapshot of current state, then applies messages after the snapshot's seq.

4) Computing VWAP

For each symbol, keep running sums:

since-open VWAP:   pv_sum += price × qty;  v_sum += qty;  vwap = pv_sum / v_sum
rolling 5-min:     keep per-second buckets (pv, v) in a ring of 300; add new, drop expired;
                   vwap_5m = Σ bucket.pv / Σ bucket.v

• It's O(1) per trade and O(1) per update (keep running totals of the ring too).
• Use fixed-point integers for prices (e.g., price × 10^4) to avoid floating-point drift.
• Handle trade corrections and cancels from exchanges by subtracting the original trade's contribution.
• Partition by symbol: each processor owns a set of symbols, so all ticks for a symbol go to one processor in order, with no locks. Hot symbols (AAPL, TSLA) get dedicated cores.

5) Publishing to Subscribers

• Subscribers subscribe to symbols. The publisher sends VWAP updates on change, conflated (e.g., at most every 100 ms per symbol per subscriber). Slow subscribers get the latest value, not a backlog.
• Many subscribers → fan-out through a tier of distribution servers (multicast inside the data center, TCP/WebSocket to outside clients).

6) Latency and Failover

• Low latency: kernel-bypass networking, pinned CPU cores, no garbage-collection pauses in the hot path (C++/Rust or tuned Java), and in-memory state only.
• Failover: run hot-standby processors that consume the same multicast and compute the same state. If the primary dies, the standby takes over instantly with identical state (deterministic processing of the same ordered input).
• Normalization: each exchange has its own format and symbol codes. Loaders convert to one internal format and symbol ID.

7) Trade-offs & Alternatives

8) Wrap-Up

Loaders normalize exchange feeds and send each message with sequence numbers over two multicast paths. Processors partitioned by symbol take the first copy, drop duplicates, detect gaps and recover them from a retransmit store, and load snapshots when they (re)join. VWAP is kept as running sums (since open) and per-second ring buckets (rolling windows) in fixed-point math, published to subscribers as conflated updates, and hot-standby processors keep identical state for instant failover.