APIs that stay up under pressure

Most API outages move through the same quiet sequence. A dependency slows down. Requests begin to gather. Clients retry. Workers block, and the connection pools fill. Meanwhile the database still has CPU to spare and the app still has memory, so the dashboards read as normal for a while. The thing the user actually feels is the waiting, and the waiting arrives before any alarm does.

Adding servers here only widens the loop. Each new server opens its own calls into the slow dependency, sends its own retries, and holds its own database connections.

What follows are the controls that interrupt the loop, and a calm way to reason about each one.

The failure is usually queueing

An API fails when code throws. It also fails, just as completely, when useful work sits behind work that will not finish in time.

Slow dependency calls hold onto request workers, and those workers hold memory, sockets, and database connections. New requests arrive and wait their turn. Clients time out and retry, and the retry lands while the first request is still running. Through that small chain, a modest rise in latency becomes an outage.

normal:
  500 requests/sec
  50ms service time
  25 concurrent in-flight requests

dependency slows:
  500 requests/sec
  2s service time
  1000 concurrent in-flight requests

The arrival rate is the same in both cases: 500 requests/sec. Only the service time changes, stretching from 50ms to 2 seconds, and in-flight work follows it from 25 to 1000.

This is why adding servers tracks the bottleneck rather than relieving it. Each new server simply opens more concurrent calls into the dependency that is already slow.

Rate limiting is admission control

Rate limiting decides which requests are allowed in. It answers a single question:

Should this request be allowed to enter the system right now?

A token bucket is a gentle way to express this. It gives each caller a burst allowance and a steady refill rate.

import time


class TokenBucket:
    def __init__(self, capacity, refill_per_second):
        self.capacity = capacity
        self.refill_per_second = refill_per_second
        self.tokens = capacity
        self.updated_at = time.monotonic()

    def consume(self, cost=1):
        now = time.monotonic()
        elapsed = max(0, now - self.updated_at)

        self.tokens = min(
            self.capacity,
            self.tokens + elapsed * self.refill_per_second,
        )
        self.updated_at = now

        if self.tokens < cost:
            return False

        self.tokens -= cost
        return True

This is a teaching version, kept small on purpose. In production the bucket needs to be shared across app instances, which usually means Redis with an atomic script, a gateway, or an edge proxy.

The configuration has a simple shape:

capacity = burst size
refill   = sustained rate
cost     = request weight

A cheap read might cost 1 token. A report export might cost 20. A login attempt can sit behind a separate security limit. A tenant-level write limit can protect the database from one large customer. Each weight is just an honest estimate of the work the request will ask for.

Pick the limit key carefully

The limit key decides who is isolated from whom. The common choices each carry their own tradeoff.

A common and forgiving approach is to layer the limits:

global limit
  tenant limit
    user/API key limit
      endpoint cost limit

Cost tracks weight. A GET /status spends a token or two; a POST /exports spends a larger share of the budget, in proportion to the work it sets in motion.

Fixed window, sliding window, token bucket, leaky bucket

Each strategy earns its place through how it behaves. Read the rows by what they do rather than by their names.

Fixed windows count with a single counter that resets on the boundary. That leaves an edge: a caller can send 100 requests at 12:00:59 and 100 more at 12:01:00, landing 200 requests inside two seconds.

Token buckets sit comfortably with the bursty traffic real systems actually see. They allow short bursts up to capacity, then settle back to the refill rate, which keeps sustained pressure in check.

Timeouts are part of the contract

Every network call needs a timeout. The public API, internal HTTP calls, database queries, Redis calls, search calls, queue publishes, object storage reads. Anything that waits on another system deserves a deadline.

Without one, the dependency decides how long your worker lives, and that is not a decision worth giving away.

def get_profile(user_id, request_deadline):
    remaining = request_deadline - time.monotonic()

    if remaining <= 0:
        raise TimeoutError("request deadline exceeded")

    return http.get(
        f"https://profiles.internal/users/{user_id}",
        timeout=min(0.150, remaining),
    )

A timeout budget makes the whole constraint visible at a glance:

public request deadline: 800ms
  auth:                  50ms
  database:             150ms
  profile service:      150ms
  recommendation call:  200ms
  rendering/marshal:     50ms
  slack:                200ms

By the time the outer request times out at 800ms, the user has already moved on. An inner call still waiting at 2 seconds is spending a worker on a result nobody is there to receive.

Retries are load multipliers

Retries earn their keep when the failure is temporary and the operation is safe to repeat. When the failure is permanent, or the operation carries side effects, each retry only adds load and leaves reliability exactly where it was.

They also compound across layers, quietly. One retry at three layers multiplies into something larger than it looks:

client retries 2x
  gateway retries 2x
    service retries 2x

one user request can become 8 downstream attempts

This is a retry storm.

A retry budget holds it in bounds:

import random
import time


def call_with_retries(fn, deadline, max_attempts=3, base_delay=0.025):
    last_error = None

    for attempt in range(max_attempts):
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("deadline exceeded") from last_error

        try:
            return fn(timeout=remaining)
        except TemporaryError as error:
            last_error = error

        if attempt == max_attempts - 1:
            break

        delay = min(base_delay * (2 ** attempt), 0.250)
        jitter = random.uniform(0, delay)
        time.sleep(min(jitter, max(0, deadline - time.monotonic())))

    raise last_error

Synchronized retries are a burst in their own right. Backoff spreads them out over time, and jitter makes that spread uneven, which gives a struggling dependency the room it needs to recover.

Circuit breakers fail fast

A circuit breaker simply stops calling a dependency that is already failing. It rests in one of three states:

closed:    calls flow normally
open:      calls fail fast
half-open: a few probe calls test recovery

In its simplest form:

import time


class CircuitBreaker:
    def __init__(self, failure_threshold, reset_after_seconds):
        self.failure_threshold = failure_threshold
        self.reset_after_seconds = reset_after_seconds
        self.failure_count = 0
        self.opened_at = None

    def call(self, fn):
        if self.opened_at is not None:
            elapsed = time.monotonic() - self.opened_at
            if elapsed < self.reset_after_seconds:
                raise ServiceUnavailable("circuit open")

            return self._half_open_call(fn)

        try:
            result = fn()
            self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.opened_at = time.monotonic()
            raise

    def _half_open_call(self, fn):
        try:
            result = fn()
            self.failure_count = 0
            self.opened_at = None
            return result
        except Exception:
            self.opened_at = time.monotonic()
            raise

This code is intentionally small. Real breakers add rolling windows, error rates, probe limits, and metrics. The principle underneath them does not change: once a dependency is unhealthy, there is little to gain from spending request threads on it.

Choosing the numbers

The patterns themselves are well known. The real work is choosing the specific numbers, and deciding what the product does when those numbers take effect:

• What status code is returned?
• What does the user see?
• Which endpoint is protected first?
• Which customer is allowed to burst?
• Which dependency is optional?
• Which write is safe to retry?

These are architecture decisions, and they are worth making calmly, before pressure forces them.

Bulkheads and concurrency limits

Rate limits govern arrival. Concurrency limits govern in-flight work. A reliable API tends to use both, each covering its own surface.

If an endpoint can spend 2 seconds waiting on a dependency, even a modest request rate can consume every worker. A concurrency limit caps how many calls run at once.

class ConcurrencyLimiter:
    def __init__(self, max_in_flight):
        self.semaphore = Semaphore(max_in_flight)

    def call(self, fn):
        if not self.semaphore.acquire(blocking=False):
            raise ServiceUnavailable("too many in-flight requests")

        try:
            return fn()
        finally:
            self.semaphore.release()

Give separate classes of work their own pools:

checkout pool:       small, protected, high priority
search pool:         medium, user-facing
export pool:         small, background
analytics pool:      best effort
admin report pool:   isolated

This is a bulkhead. Each pool holds its own capacity, so a flooded endpoint drains only its own pool and the rest of the product keeps its share. One part can struggle without pulling the whole down with it.

Graceful degradation

A dependency can fail while the API stays up. That is often a choice you can design for rather than an accident you suffer.

• If recommendations are unavailable, return the page without recommendations.
• If the personalization service is slow, return the default ranking.
• If a profile badge service fails, omit the badge.
• If a search cluster is unhealthy, return cached results with a stale marker.

The goal is steady: preserve the core product path when the optional pieces fall away.

def get_home_feed(user_id, deadline):
    feed = db.get_recent_feed_items(user_id, timeout=0.120)

    try:
        recommendations = recommender.get_items(
            user_id,
            timeout=min(0.080, deadline - time.monotonic()),
        )
    except (TimeoutError, ServiceUnavailable):
        recommendations = []

    return {
        "feed": feed,
        "recommendations": recommendations,
    }

The design question worth keeping nearby:

What can we remove and still serve something useful?

Load shedding

Load shedding is rejecting some work on purpose so the system stays available for the rest of the traffic. It is an act of care for the requests you can still serve well.

It pays off most when it rejects early:

1. Before expensive authentication fanout if possible.
2. Before opening database transactions.
3. Before calling slow dependencies.
4. Before accepting background jobs that cannot be processed.

A rejection that lands after most of the work is done has already spent the very capacity it was meant to save.

When the system is overloaded, a fast 429 or 503 with Retry-After hands the client a clear signal in milliseconds and frees the worker right away. A 30-second timeout holds that worker for the full 30 seconds and leaves the client to retry blindly into the dark.

HTTP/1.1 429 Too Many Requests
Retry-After: 3
Content-Type: application/json

{"error":"rate_limited","retry_after_seconds":3}

Status codes are control signals

Clients behave well when they are told clearly what happened. Distinct signals give them that.

A clear error tells the client exactly what to do next, so it waits, backs off, or stops. That kind of precision is itself a form of reliability.

The retry matrix

Before adding a retry, pause and decide whether it is safe:

Retry policy follows from operation semantics. A PUT is safe to retry when its implementation lands the same final state on every attempt. A POST is safe to retry when an idempotency key lets the server dedupe the command.

Metrics that help

A small handful of metrics deserves a place on the first dashboard for any API:

Watch in-flight work and saturation right alongside CPU. An API can go unresponsive at low CPU, simply because every worker is sitting and waiting on I/O.

Practical sequence

A workable order for putting these controls in place:

1. Define the core product path.
2. Put deadlines around every downstream call.
3. Limit in-flight work per route and dependency.
4. Add caller-aware rate limits.
5. Retry only idempotent operations, inside a deadline, with jitter.
6. Use circuit breakers for dependencies that fail slowly.
7. Decide which pieces can degrade.
8. Return clear status codes that teach clients what to do.
9. Watch in-flight work, tail latency, retries, and rejects.

It all starts with a few plain questions:

How many requests are allowed in?
How long can they run?
How many can wait?
What happens when dependency X is slow?
What should the client do next?

Once those answers exist, the API has a reliability story to stand on, and the architecture diagram stops being a hope and starts describing a system that already knows how it behaves under pressure.

Endpoint checklist

For any important API endpoint, it helps to fill in:

Then ask:

If this endpoint gets 10x traffic for five minutes, what protects the rest of the system?

A complete answer names a control beyond autoscaling. Autoscaling adds capacity; admission control governs what gets in. Both belong in the answer, and they work best together.

Summary

1. API failures often arrive as queueing failures, with the code still running fine.
2. Rate limits control which work enters the system.
3. Concurrency limits control how much work can be in flight.
4. Every downstream call needs a timeout and should respect the request deadline.
5. Retries multiply load; use budgets, backoff, jitter, and idempotency.
6. Circuit breakers fail fast when a dependency is already unhealthy.
7. Graceful degradation preserves the core product path.
8. Load shedding rejects work early to keep capacity for the traffic the system can still serve.
9. Status codes and Retry-After headers are part of the reliability contract.
10. Watch in-flight work, p99 latency, retry attempts, rate-limit rejects, and queue lag.

---

Pop quiz

Interactive quiz

API reliability controls

A relaxed, randomized review of rate limits, timeouts, retries, circuit breakers, and load shedding.

4of 11 questions

Question 1 of 425%

A dependency slows from 50ms to 2 seconds while request rate stays flat. What is the immediate API risk?

AThe API is automatically safe because request rate did not increaseBIn-flight requests grow, causing queueing, pool exhaustion, and timeoutsCThe database must become faster because CPU is lowDAll clients will stop retrying by default

Next question