0) Problem Restatement
Walmart asked: design a service that answers one client request by calling several downstream microservices and combining the answers. For example, a product page needs product details, price, inventory, reviews, delivery estimate and recommendations, each from a different team's service. The aggregator should be fast (low tail latency), resilient (one slow or failing service shouldn't break the page), and not overload struggling services.
1) Requirements
- One API call returns the combined response.
- Latency target, e.g., p99 < 300 ms.
- If an optional part fails, still return the rest (degraded response).
- Protect downstream services from overload and retry storms.
2) Architecture
Architecture Diagram
flowchart LR
C["Client"] --> AGG["Aggregator - fan-out / fan-in"]
AGG -->|"required, 120ms"| P["Product service"]
AGG -->|"required, 120ms"| PR["Price service"]
AGG -->|"required, 150ms"| INV["Inventory service"]
AGG -->|"optional, 200ms"| REV["Reviews service"]
AGG -->|"optional, 200ms"| RECO["Recommendations"]
AGG --> CA[("Cache - per dependency")]3) Key Techniques
- Parallel fan-out: start all independent calls at once (async I/O). Total latency ≈ the slowest call, not the sum. Calls that depend on another's result (e.g., delivery estimate needs the product's warehouse) are chained only where necessary.
- Deadlines and timeouts: the aggregator has an overall budget (e.g., 250 ms). Each call gets a timeout within that budget, and the remaining deadline is passed to downstream services (a deadline header) so they stop work that can't be used.
- Required vs optional parts: if a required part fails (price), return an error or use a safe fallback (cached price). If an optional part fails (reviews), omit it and mark
reviews: unavailable. - Fallbacks: serve from cache (stale-while-revalidate), use default values, or hide the section.
- Circuit breaker per dependency: after many failures or timeouts, stop calling for a short time and use the fallback immediately. This protects both the aggregator's latency and the struggling service.
- Bulkheads: a separate connection pool or concurrency limit per dependency, so a slow reviews service can't use up all threads and block price calls.
- Retries, carefully: retry only idempotent calls, only once, only if there's time left in the deadline, with jitter, and with a retry budget (e.g., retries ≤ 10% of requests) to avoid retry storms that multiply load during outages.
- Hedged requests (for tail latency): if a call hasn't returned by its p95 time, send a second copy to another replica and use whichever answers first. Cap hedging to a small percentage.
- Caching: cache stable data (product details) for minutes, and volatile data (inventory) for seconds or not at all.
4) Example Timeline
At t=0, send all 5 calls. At 40 ms, product and price are back. Inventory arrives at 90 ms. Reviews hit a timeout at 200 ms → breaker counts a failure → the response omits reviews. Recommendations arrive at 150 ms. The response is sent at ~200 ms with reviews marked unavailable.
5) Observability
- Distributed tracing across the fan-out shows which dependency drives latency.
- Per-dependency metrics: latency, error rate, timeout rate, breaker state and fallback usage.
- Alert when fallback usage rises, which means the page is degrading quietly.
6) Wrap-Up
Fan out to downstream services in parallel within an overall deadline, giving each call its own timeout and passing the remaining deadline downstream. Separate required from optional parts, with fallbacks and cached data, and protect everything with per-dependency circuit breakers, bulkheads, careful budgeted retries and optional hedging for tail latency. Trace and monitor each dependency so degradation is visible.