Making Many HTTP Requests Fast (Within Rate Limits)

0) Problem Restatement

Anthropic asked: you need to make a large number of HTTP requests (say 100,000 calls to an external API) as fast as possible, but the API has rate limits (e.g., 500 requests/second) and sometimes returns errors or responds slowly. How do you design the client? What trade-offs exist between speed, limits and reliability?

1) Why the Naive Way Is Slow

One request at a time: if each takes 200 ms (mostly waiting on the network), 100,000 requests take 5.5 hours. The CPU is idle almost all the time. We're limited by latency, not work.

2) The Design

flowchart LR
    Q["Work queue - 100K URLs"] --> RL["Token bucket - 500 req/s"]
    RL --> SEM["Concurrency limit - e.g., 100 in flight"]
    SEM --> POOL["HTTP client - connection pool, keep-alive, HTTP/2"]
    POOL --> API["External API"]
    API -->|"429 / 5xx / timeout"| RETRY["Retry with backoff + jitter"]
    RETRY --> Q
    API -->|"200"| OUT["Results"]

• Async I/O (e.g., Python asyncio with aiohttp, or Go goroutines) handles hundreds of concurrent requests in one process cheaply. A thread pool also works for smaller scale.
• Concurrency cap (a semaphore): enough to hit the rate limit (~rate × latency), not unlimited (to avoid overloading the server or our memory).
• Client-side rate limiting: a token bucket at the allowed rate, so we don't blast requests and get 429s. Respect Retry-After headers.
• Connection reuse: a pool with keep-alive avoids a new TCP and TLS handshake per request (which can double latency). HTTP/2 multiplexes many requests on one connection.
• Timeouts: connect and read timeouts, so slow requests don't hold slots forever.
• Retries: only for retryable errors (timeouts, 429, 502/503), with exponential backoff + jitter, and a max attempt count. Record permanent failures separately.
• Batching: if the API has a batch endpoint (e.g., 100 items per call), use it. That's often the biggest win of all.

3) Code Sketch (Python asyncio)

import asyncio, random, time
import aiohttp

class TokenBucket:
    def __init__(self, rate, burst):
        self.rate, self.tokens, self.burst, self.t = rate, burst, burst, time.monotonic()
        self.lock = asyncio.Lock()
    async def take(self):
        async with self.lock:
            while True:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self.t) * self.rate); self.t = now
                if self.tokens >= 1:
                    self.tokens -= 1; return
                await asyncio.sleep((1 - self.tokens) / self.rate)

async def fetch_all(urls, rate=500, concurrency=100, max_tries=4):
    bucket, sem, results = TokenBucket(rate, burst=rate), asyncio.Semaphore(concurrency), {}
    timeout = aiohttp.ClientTimeout(total=10)
    async with aiohttp.ClientSession(timeout=timeout, connector=aiohttp.TCPConnector(limit=concurrency)) as s:
        async def one(url):
            for attempt in range(max_tries):
                await bucket.take()
                async with sem:
                    try:
                        async with s.get(url) as r:
                            if r.status == 200:
                                results[url] = await r.json(); return
                            if r.status not in (429, 500, 502, 503, 504):
                                results[url] = ("error", r.status); return
                            wait = float(r.headers.get("Retry-After", 0))
                    except (aiohttp.ClientError, asyncio.TimeoutError):
                        wait = 0
                await asyncio.sleep(max(wait, (2 ** attempt) * 0.1 + random.random() * 0.1))
            results[url] = ("error", "retries exhausted")
        await asyncio.gather(*(one(u) for u in urls))
    return results

4) Measure to Find the Bottleneck

• If throughput is stuck below the rate limit, check latency per request and concurrency (Little's Law).
• If you're at the rate limit, only batching or a higher quota helps.
• If CPU is maxed (JSON parsing), add processes.
• If bandwidth is maxed (large responses), compress (gzip) or request fewer fields.

5) Wrap-Up

Replace sequential calls with bounded concurrency (async I/O and a semaphore sized to about rate × latency), throttle with a client-side token bucket that honors Retry-After, reuse connections with keep-alive or HTTP/2, set timeouts, and retry only retryable errors with exponential backoff and jitter. Use batch endpoints when available, and measure latency, rate, CPU and bandwidth to find the real bottleneck.