0) Problem Restatement
When a process crashes, the OS can write a core dump: a snapshot of its memory at the moment of the crash, which engineers use to debug. Design a system (asked at Amazon) that collects core dumps from a large fleet of hosts, stores them, and helps engineers analyze them: which crashes are new, how often they happen, and on which versions.
Core dumps can be huge (GBs), a bad deploy can cause thousands of crashes at once, and dumps can contain sensitive data (memory may include secrets or customer data).
1) Requirements
- Capture dumps on crash, without filling the host disk.
- Upload reliably to central storage, with limits during crash storms.
- Extract metadata: service, version, host, time, signal, and the stack trace (after symbolication).
- Group crashes by signature (same bug → same group), and count occurrences.
- Search, download (for authorized engineers) and link to tickets.
- Alert on new crash signatures, and on spikes after deploys.
2) Architecture
Architecture Diagram
flowchart LR
P["Crashing process"] --> H["Host agent - core handler"]
H --> SP[("Local spool - size capped")]
SP -->|"compressed, resumable upload"| UP["Upload service - rate limited"]
UP --> OS[("Object storage - encrypted dumps")]
UP --> Q[("Processing queue")]
Q --> AN["Analyzer - symbolicate, extract stack"]
SYM[("Symbol store by build ID")] --> AN
AN --> DB[("Crash DB - metadata, signatures, groups")]
DB --> UI["Crash dashboard"]
DB --> ALR["Alerts - new signature, spike"]3) Key Steps
- Capture: configure the kernel to pipe core dumps to our agent (
core_patternwith a pipe), instead of writing wherever. The agent writes a compressed file (zstd) into a spool directory with a size cap. If the disk is almost full, it keeps a minidump (just stacks and registers) instead of the full dump. - Quick local metadata: service name, binary build ID, version, host, signal, timestamps.
- Upload: resumable multipart upload to object storage (encrypted). The upload service applies rate limits and sampling: during a crash storm, keep full dumps for the first N per signature per hour, and just counts and minidumps for the rest.
- Analyze: a worker fetches the dump, loads debug symbols for that build ID from the symbol store (uploaded by CI at build time), produces a symbolicated stack trace, and computes a signature (e.g., a hash of the top 5 meaningful frames, ignoring addresses and line numbers that change between builds).
- Group and store: add to the crash group for that signature, updating counts, first seen, last seen and affected versions.
4) Grouping and Alerts
- Signature quality matters: skip generic frames (abort, malloc internals) so different bugs don't merge, and normalize inlined or templated names so the same bug doesn't split into many groups.
- Alerts: a new signature in a production service → notify the owning team. A spike (crash rate per version jumps after a deploy) → alert and suggest rollback, and link it to the deploy system.
- Dashboard: top crashes by count, trend over time, and versions affected. Each group links to example dumps and the ticket.
5) Security and Retention
- Dumps may contain secrets or customer data: encrypt at rest, restrict download to the owning team with audit logs, and prefer analysis in a secure environment (engineers view stacks, not raw memory, unless approved).
- Retention: keep full dumps for 14–30 days (a few examples per group longer), and keep metadata and stacks for a long time. Delete by lifecycle policy.
6) Trade-offs & Alternatives
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Capture | Pipe to agent with capped spool | Protects host disk | Default core files: disks fill up |
| Crash storms | Sample per signature + minidumps | Bounded storage and bandwidth | Upload all: overload during incidents |
| Analysis | Central symbolication by build ID | Readable stacks | Raw addresses: useless for humans |
| Grouping | Normalized top-frame signature | Tracks bugs over time | Group by error message: too coarse |
7) Wrap-Up
Route core dumps to a host agent that compresses them into a size-capped spool (falling back to minidumps), uploads them resumably with rate limiting and per-signature sampling, and records metadata. Central analyzers symbolicate stacks using a build-ID symbol store, compute normalized signatures, and group crashes, which powers dashboards and alerts on new signatures and post-deploy spikes, with encryption, access control and lifecycle retention for sensitive memory contents.