Troubleshooting a Slow or Failing Production System

0) Problem Restatement

Two interview versions:

• Atlassian: "We scaled our service up, and now it's slower than before. How do you find out why?"
• Meta: "A web server running on a single machine is down or not responding. How do you troubleshoot it, and how do you prevent it next time?"

This isn't about drawing a new system. It tests whether you can debug methodically under pressure: stop the bleeding first, use data instead of guesses, and fix the root cause.

1) The Process (say this structure out loud)

1. Understand the impact: who is affected, since when, how bad (errors? latency? everything or one endpoint?).
2. Mitigate first: if something changed recently (deploy, config, traffic), roll back or scale up. Restore service before full diagnosis.
3. Look at the data: dashboards, logs, traces. Find where time is spent.
4. Form hypotheses and test them one at a time.
5. Fix the root cause, verify with metrics, and write a blameless postmortem with action items.

2) Two Simple Checklists

flowchart LR
    A["Alert / user report"] --> B["Assess impact - scope, since when"]
    B --> C["Mitigate - rollback, scale, shed load"]
    C --> D["Find the slow hop - traces, RED per service"]
    D --> E["Check resources - USE: CPU, memory, disk, pools"]
    E --> F["Hypothesis and test"]
    F --> G["Root cause fix + postmortem"]

3) Case A — "We scaled up and it got slower"

More servers but worse latency usually means the bottleneck is shared, something all servers use. Suspects, in order:

1. Database: more app servers → more DB connections and queries. Check DB CPU, slow query log, lock waits and connection count (hundreds of new connections can exhaust the DB). Fix: a connection pooler (e.g., PgBouncer), query and index fixes, read replicas, caching.
2. Connection pools / thread pools: each app instance has a pool. Too small and requests queue; too big in total and the DB is overloaded. Look at pool wait time.
3. Cache: new instances start with cold caches (local caches empty) → more DB load. Or a shared cache hits its network or CPU limit, or a hot key.
4. Lock contention: a shared lock (DB row, distributed lock, or a synchronized block) gets more contention as instances grow.
5. Downstream dependency: a service or third-party API with a rate limit. More callers → more throttling and retries → more latency (retry storms).
6. Load balancing: uneven distribution (sticky sessions, bad hashing) → some servers overloaded while others are idle.
7. Noisy neighbors / resources: new instances on smaller or shared machines, or garbage collection pauses from a new memory setting.

4) Case B — "A single-node web server is down"

Work from the outside in:

1. Is it reachable? Ping or DNS, security groups and firewall, load balancer health checks.
2. Is the process running? systemctl status, ps. Crashed? Check logs (journalctl, app logs) for panics or OOM kills (dmesg | grep -i oom).
3. Resources: top/htop (CPU), free -m (memory, swap), df -h (disk full is very common, often from logs), iostat (disk), open files (ulimit, too many connections), network (ss -s).
4. Is it hung? Too many threads blocked (take a thread dump), a deadlock, or a stuck dependency with no timeout.
5. What changed? A recent deploy, config change, certificate expiry, or OS update.

5) Lasting Fixes (after the incident)

• Observability: RED dashboards per service, USE for resources, tracing and alerts on SLOs.
• Resilience: timeouts, retries with backoff and jitter (and retry budgets), circuit breakers, bulkheads.
• Capacity: load tests before scaling events, and connection pool math (instances × pool size ≤ DB limit).
• Safe changes: canary deploys with automatic rollback.
• Postmortem: timeline, root cause, what went well and badly, and owned action items. Blameless, focused on systems, not people.

6) Common Follow-up Questions

• "Latency is high but CPU is low everywhere?" Then requests are waiting: on locks, I/O, pool slots or a slow dependency. Look at saturation (queues) and traces, not CPU.
• "Only p99 got worse?" Look at outliers: GC pauses, a slow shard, a hot key, retries or one bad host. Check latency per host.
• "Errors went up after a deploy but it's not obvious why?" Roll back first, then compare logs and traces between the old and new versions.

7) Wrap-Up

Start with impact and mitigation (roll back, scale, shed load), then use data. RED per service finds the slow hop and USE per resource finds the saturated component. After scaling, the usual culprits are shared ones: database connections and queries, pools, cold or hot caches, locks and rate-limited dependencies. On a single node, check reachability, the process, then CPU, memory, disk and dependencies. Finish with lasting fixes and a blameless postmortem.