Highly Available Spring Boot Service with Background Jobs

0) Problem Restatement

Walmart asked: design a highly available deployment for a Spring Boot application behind a load balancer, plus resilient background processing for long-running tasks (e.g., generating reports, bulk imports that take minutes). The service must survive instance failures and deployments without errors, and background jobs must not be lost or run twice.

1) Architecture

flowchart LR
    C["Clients"] --> LB["Load balancer - health checks"]
    LB --> A1["Spring Boot API - instance 1"]
    LB --> A2["Spring Boot API - instance 2"]
    LB --> A3["Spring Boot API - instance 3"]
    A1 --> DB[("Postgres primary + standby")]
    A1 --> R[("Redis - sessions / cache")]
    A1 -->|"enqueue job"| Q[("Queue - RabbitMQ / Kafka / SQS")]
    Q --> W1["Worker instance"]
    Q --> W2["Worker instance"]
    W1 --> DB
    SCH["@Scheduled jobs + ShedLock"] --> DB

2) High Availability for the API

• Stateless instances: no user session in local memory. Use tokens (JWT) or store sessions in Redis (Spring Session), so any instance can serve any request, and losing one instance doesn't log users out.
• At least 2–3 instances across availability zones behind a load balancer.
• Health checks with Spring Boot Actuator: a liveness probe (is the process alive?) and a readiness probe (can it serve? DB reachable, warmed up). The LB only sends traffic to ready instances.
• Graceful shutdown (server.shutdown=graceful): on deploy, stop accepting new requests, finish in-flight ones, then exit. Combined with rolling or blue-green deployments, users see no errors.
• Connection pools (HikariCP): size them so instances × pool size fits the database's limit.
• Database HA: a primary with a standby replica and automatic failover, plus backups.

3) Long-Running Work: Don't Do It in the Request

If a request takes minutes, it ties up a thread and times out at the LB, and if the instance restarts the work is lost. Instead:

1. The API validates the request, creates a job record (status = queued) in the DB, puts a message on a queue, and returns 202 Accepted with a job ID.
2. Worker instances (a separate Spring Boot app or profile) consume messages, run the task, update progress, and set succeeded or failed.
3. The client polls GET /jobs/{id} (or gets a webhook or notification).

• The queue gives at-least-once delivery: acknowledge only after finishing, so a crashed worker's message is redelivered.
• Make tasks idempotent (check the job status before starting, and write results with the job ID as the key), so redelivery doesn't duplicate work.
• Retries with backoff, and a dead-letter queue after N failures.
• Long jobs send heartbeats or save checkpoints, so a restart can resume.

4) Scheduled Jobs Across Many Instances

With @Scheduled on every instance, a nightly job would run 3 times. Fix it with a distributed lock: ShedLock (a lock row in the DB, or Redis) ensures only one instance runs each scheduled job at a time. Alternatively, use Quartz in clustered mode, or an external scheduler that enqueues jobs onto the queue.

5) Observability

• Metrics (Micrometer → Prometheus): request latency, error rate, queue depth, job duration and failures.
• Structured logs with a trace ID (Spring Cloud Sleuth / OpenTelemetry) across the API and workers.
• Alerts on queue backlog growth and on job failure rates.

6) Wrap-Up

Run multiple stateless Spring Boot instances across zones behind a load balancer, with Actuator liveness and readiness checks, externalized sessions, graceful shutdown with rolling or blue-green deploys, right-sized connection pools and an HA database. Move long-running work to a queue: the API returns 202 with a job ID, and idempotent workers process messages at-least-once with retries, a DLQ and checkpoints. Use ShedLock or clustered Quartz so scheduled jobs run once, and monitor everything.