Highly Available Software Load Balancer

0) Problem Restatement

Design a software load balancer (LB), the component that sits in front of a group of backend servers and spreads incoming traffic across them. It must send traffic only to healthy backends, handle many connections, and never be the single thing that takes the whole site down. Oracle asked to pick layer 4 or layer 7 and justify it. Amazon asked about high availability for HTTP/HTTPS traffic.

1) Requirements

1.1 Functional

• Distribute requests or connections across backend servers.
• Health checks: stop sending traffic to unhealthy backends.
• Add or remove backends without dropping traffic (draining).
• (L7) TLS termination, routing by path or host, sticky sessions.
• A configuration API.

1.2 Non-Functional

• High availability: an LB node failure must not cause an outage.
• Low added latency: well under a millisecond for L4, and a few ms for L7.
• Scale: millions of concurrent connections and hundreds of thousands of requests per second.

2) L4 vs L7

3) High-Level Architecture

3.1 Overview

• Clients resolve DNS to one or a few virtual IPs (VIPs).
• L4 layer: routers announce the same VIP from many machines (anycast plus ECMP, where the router splits traffic over several equal paths), so traffic spreads across L4 nodes. The L4 nodes use consistent hashing on the connection's addresses, so each connection sticks to one L7 proxy.
• L7 proxy fleet (e.g., Envoy): terminates TLS, routes and balances requests across backends.
• Control plane: stores configuration (routes, backend pools, certificates) and pushes it to all proxies. It collects health data.

3.2 Architecture Diagram

flowchart LR
    C["Clients"] --> DNS["DNS - VIP"]
    DNS --> R["Edge routers - anycast + ECMP"]
    R --> L4A["L4 node A"]
    R --> L4B["L4 node B"]
    L4A --> P1["L7 proxy 1"]
    L4A --> P2["L7 proxy 2"]
    L4B --> P1
    L4B --> P2
    P1 --> BE1["Backend pool - zone A"]
    P2 --> BE2["Backend pool - zone B"]
    CP["Control plane - config, certs, health"] --> P1
    CP --> P2

4) Balancing Algorithms

• Round robin: take turns. Simple, but ignores how busy each server is.
• Weighted round robin: bigger servers get more traffic.
• Least connections / least outstanding requests: send to the server with the fewest active requests. Good when request times vary.
• Power of two random choices: pick 2 servers at random and choose the less loaded one. Nearly as good as least-connections with far less coordination between LB nodes.
• Consistent hashing (by user or session ID): the same user goes to the same server. Useful for caches or sticky state, and only a few users move when servers change.

5) Key Mechanisms

5.1 Health checks

• Active: every few seconds, call /healthz on each backend. Mark it down after 3 failures and up after 2 successes. The different thresholds avoid flapping.
• Passive: watch real traffic. Too many 5xx errors or timeouts from a backend → eject it temporarily (outlier detection).
• Don't let a failing health endpoint take out the whole pool: if more than ~50% of backends fail at once, the problem is probably the health check itself, so keep serving ("fail open").

5.2 Connection draining

When removing a backend (deploy or scale-in), stop sending new requests to it, but let in-flight requests finish (e.g., up to 30 seconds) before it shuts down.

5.3 TLS termination

The L7 proxy decrypts HTTPS, which lets it route by path, then re-encrypts to backends if needed. Certificates are managed centrally and pushed by the control plane.

6) Deep Dive — No Single Point of Failure

• Multiple LB nodes, active-active: anycast/ECMP spreads traffic over many L4 nodes. If one dies, routers stop sending traffic to it within seconds.
• Consistent hashing at L4 (as in Google's Maglev): when an L4 node disappears, other nodes pick the same proxy for existing connections, so most connections survive.
• Smaller setups: a pair of LBs sharing a floating IP with VRRP/keepalived. The standby takes over the IP if the active one dies.
• Multi-zone: proxies and backends in several availability zones. Cross-zone balancing keeps capacity even if one zone fails.
• Config safety: push new routing config gradually to a few proxies first, then all, with automatic rollback, since a bad config is a common cause of LB outages.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you handle millions of connections?" L4 nodes keep a tiny amount of state per connection (or none, with consistent hashing). L7 proxies use event-driven I/O, and you add more nodes as needed.
• "DDoS?" Absorb at the edge with anycast across many sites, use SYN cookies at L4, and rate limit and filter at L7 (WAF).
• "Global load balancing?" Use GeoDNS or anycast to send users to the nearest healthy region, and fail over to another region when one is down.

9) Wrap-Up

Put a fast L4 layer (anycast + ECMP + consistent hashing) in front of an L7 proxy fleet that terminates TLS and balances requests with least-outstanding or power-of-two-choices. Keep backends healthy with active and passive health checks, drain connections on removal, and run every layer active-active across zones with config pushed gradually, so no single machine or bad config can take everything down.