Network calls are failure boundaries

In code, a network call looks just like a function call. The two share a syntax. They do not share a nature, and it helps to sit with the difference for a moment.

A local function does one of a few things. It returns, it raises, or it crashes the process. A network call lives in a wider world. The request might never leave the host. It might reach the server and time out on the way back. The server might finish the write quietly, after the caller has already given up waiting. The response might be perfectly correct and arrive too late to matter.

A network call, then, draws a boundary. On the near side, you know things. On the far side, your knowledge is partial. Everything that follows in this post is really one idea seen from different angles: how to work calmly with that partial knowledge instead of pretending it away.

Start with the boundary

There is one question worth asking before any network call, and it tends to settle the design on its own:

What does the caller know after this call returns, times out, or fails?

For a read, the answer comes easily. If a profile service times out, the caller can show no profile card, or it can serve slightly stale data. Either choice is fine, and the user is not harmed.

For a write, the same question carries more weight. If a payment gateway times out, the caller is left holding an open question: was the card charged or not? Retrying blindly can charge it twice. Giving up blindly can leave a paid order marked unpaid. Neither reflex is safe, because both answer a question the caller cannot actually see the answer to yet.

The instinct to collapse all of those states into one except branch is understandable. It is also where most of the trouble starts. Keeping them apart is the quiet work that pays off later.

Failure modes

A network call can fail in a handful of ordinary ways. None of them are exotic, and seeing them laid out tends to take the mystery out of them:

The second column is the one that matters.

Some of these failures mean "nothing happened." Some mean "something might have happened, and I cannot tell." Some mean "the other side handled it and said no." These are genuinely different situations, and each one deserves its own response. The table only looks like a list of errors. Really it is a list of distinct facts about the world.

A network call is a failure boundary

Here is a shape that appears in almost every codebase:

def create_order(user_id, cart_id):
    order = db.create_order(user_id=user_id, cart_id=cart_id)

    try:
        payments.charge(order.id)
    except Exception:
        db.cancel_order(order.id)
        raise

    return order

It catches the failure and cancels the order. It reads as careful code, and the intent behind it is good.

But a timeout from payments.charge carries one specific meaning, and it is not "the charge failed." It means the charge state is unknown. The payment service may well have charged the card and simply lost the response on the way back. In that case this code cancels the local order while the money has already moved.

So the catch block needs to draw a distinction it currently glosses over: was the result known, failed, or unknown? Once that question is in the open, the better version writes itself.

def create_order(user_id, cart_id, idempotency_key):
    order = db.create_order(
        user_id=user_id,
        cart_id=cart_id,
        idempotency_key=idempotency_key,
        status="payment_pending",
    )

    try:
        charge = payments.charge(
            request_id=idempotency_key,
            order_id=order.id,
            timeout=0.800,
        )
    except TimeoutError:
        db.mark_order_payment_pending(order.id)
        reconcile_payment_later(order.id)
        return {"order_id": order.id, "status": "payment_pending"}

    if charge.status == "succeeded":
        db.mark_order_paid(order.id, charge_id=charge.id)
        return {"order_id": order.id, "status": "paid"}

    db.mark_order_payment_failed(order.id, reason=charge.reason)
    return {"order_id": order.id, "status": "payment_failed"}

This version records what actually happened rather than what we hoped happened. A timeout no longer pretends to be a failure. It becomes a pending state, which is the truth, and the system can return to it later when the truth is knowable.

Timeouts are contracts

Every network call needs a timeout, and the timeout should fit inside the caller's own deadline. A timeout is less a safety net than a promise about how much patience this request is allowed to spend.

import time


def call_inventory(order_id, request_deadline):
    remaining = request_deadline - time.monotonic()
    if remaining <= 0:
        raise TimeoutError("request deadline exceeded")

    return inventory.reserve(
        order_id=order_id,
        timeout=min(0.200, remaining),
    )

Read that way, the timeout is a small sentence:

This dependency has this much time to help this request.

If the public request has an 800ms deadline, a 5-second internal call is doing work that no one is waiting for anymore. It holds sockets, workers, and memory long after the user-facing request has already been abandoned. The call keeps running for a reader who has left the room.

Timeouts also reward being specific. A connection timeout, a request write timeout, a response header timeout, and a response body timeout each carry their own meaning. Client libraries often fold all of these into a single timeout setting, and for simple paths that is perfectly fine. On important write paths, it is worth keeping enough detail to tell whether completion is unknown, because that distinction is the whole game.

Retry only when the operation allows it

Retries are a gift when the failure was temporary:

1. A connection could not be opened.
2. A dependency returned 503.
3. A read timed out before any state changed.
4. A write has an idempotency key and the server can dedupe it.

They turn risky the moment the operation changes state and has no dedupe boundary to lean on.

POST /payments without idempotency: unsafe to retry
POST /payments with idempotency: retry can be safe
PUT /profile/123 with version check: retry can be safe
POST /send-email without dedupe: may send duplicates

Notice that the client never gets to decide this on its own. The server's behavior is what decides whether a retry is safe.

Idempotency keys

An idempotency key gives a write a stable identity. With it, the caller can repeat the request as many times as the network forces it to, and the server can quietly hand back the original result each time. The retry stops being a gamble and becomes a question the server already knows the answer to.

The server should scope the key to the actor and store a hash of the request body alongside it. If the same key arrives with a different payload, that is a sign something is confused, and the right response is a conflict rather than a replay.

import hashlib
import json


def stable_hash(payload):
    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(encoded.encode("utf-8")).hexdigest()


def create_payment(account_id, idempotency_key, amount_cents):
    payload = {
        "account_id": account_id,
        "amount_cents": amount_cents,
    }
    request_hash = stable_hash(payload)

    with db.transaction() as tx:
        existing = tx.get_idempotency_record(
            scope=account_id,
            key=idempotency_key,
            for_update=True,
        )

        if existing is not None:
            if existing.request_hash != request_hash:
                raise Conflict("idempotency key reused with different payload")
            return existing.response

        payment = tx.insert_payment(
            account_id=account_id,
            amount_cents=amount_cents,
            status="pending",
        )
        tx.insert_idempotency_record(
            scope=account_id,
            key=idempotency_key,
            request_hash=request_hash,
            response={
                "payment_id": payment.id,
                "status": "pending",
            },
        )

    return {
        "payment_id": payment.id,
        "status": "pending",
    }

This is the database side of "safe to retry." The external charge may still need reconciliation of its own, but the local command now has one identity, one payment row, and one response that every repeated attempt resolves to. That is a calm place to retry from.

At-least-once is normal

It helps to make peace with this early: networked systems naturally give you at-least-once delivery. Exactly-once is not a property you receive; it is something you build on top, by adding dedupe at the handler. Once you expect duplicates, they stop being alarming.

The same logical command can arrive twice, and the path there is ordinary:

1. Client sends request.
2. Server commits the write.
3. Response is lost.
4. Client retries.
5. Server receives the same command again.

Once retries exist, this is simply the expected shape. It is not a bug to be stamped out; it is a condition to design around.

The response is to make handlers idempotent, so that a repeat is a no-op rather than a second effect:

def handle_user_created(event):
    inserted = db.insert_processed_event(
        event_id=event.id,
        handler="welcome_email",
    )

    if not inserted:
        return

    email.send_welcome_message(event.payload["user_id"])

This small dedupe table is all it takes to keep a duplicate event from sending a second welcome email. The same pattern shows up again and again, in queues, webhooks, stream consumers, and cron repair jobs. Learn it once and it keeps paying you back.

Partial failure

Partial failure is the situation where one step succeeds and a later step fails, leaving the work half-done.

reserve inventory: succeeded
charge card:       succeeded
create shipment:   failed
send receipt:      not attempted

The card has been charged. The inventory has been held. Returning a 500 to the user does not undo any of that. The system now holds facts, and facts do not disappear because the response was an error.

There are three common ways to meet this honestly:

Compensation deserves a closer look, because it is easy to mistake for a rollback. It is really a business operation.

Refunding a payment does not erase the charge; it records a new transaction that reverses an acknowledged one. Releasing inventory returns a held reservation to the available pool. Sending a correction email follows the first email with an update. In each case you are not pretending the earlier step never happened. You are adding a later step that makes the whole story come out right.

For that reason, the compensating action should be named in the domain, in the same language the business already uses.

def place_order(order_id):
    inventory_reservation = inventory.reserve(order_id)

    try:
        payment = payments.authorize(order_id)
    except Exception:
        inventory.release(inventory_reservation.id)
        raise

    try:
        fulfillment.create_shipment(order_id)
    except Exception:
        payments.void_authorization(payment.id)
        inventory.release(inventory_reservation.id)
        db.mark_order_failed(order_id, reason="shipment_failed")
        return {"order_id": order_id, "status": "failed"}

    db.mark_order_ready(order_id)
    return {"order_id": order_id, "status": "ready"}

This example is still simplified, and it is worth being honest about that. Real workflows need durable state between steps. If the process crashes after payment authorization and before shipment creation, the in-memory try/except is gone, and only what was written down survives. A worker should be able to pick the order back up from that saved state and carry on. Which leads naturally to the next idea.

Sagas are state machines

The word saga sounds grander than the idea behind it.

The useful version is just a state machine for a multi-step workflow:

order_created
  -> inventory_reserved
  -> payment_authorized
  -> shipment_created
  -> ready

failure paths:
  -> inventory_release_pending
  -> payment_void_pending
  -> failed

Each transition writes down what has happened. From there the recovery work becomes almost mechanical: workers retry pending transitions, compensate failed ones, and reconcile the unknown ones. The hard thinking has already been done by recording the state.

def advance_order(order):
    if order.state == "order_created":
        reservation = inventory.reserve(order.id)
        db.transition_order(
            order.id,
            state="inventory_reserved",
            reservation_id=reservation.id,
        )
        return

    if order.state == "inventory_reserved":
        try:
            payment = payments.authorize(order.id)
        except TimeoutError:
            db.transition_order(order.id, state="payment_unknown")
            return

        db.transition_order(
            order.id,
            state="payment_authorized",
            payment_id=payment.id,
        )
        return

    if order.state == "payment_unknown":
        payment = payments.lookup_by_order_id(order.id)
        if payment is None:
            db.transition_order(order.id, state="payment_retryable")
        elif payment.status == "authorized":
            db.transition_order(
                order.id,
                state="payment_authorized",
                payment_id=payment.id,
            )

Crash recovery becomes possible here only because the current state is durable. The machine can always be asked, "where were we?" and it can always answer.

A nice side effect: the endpoint no longer has to hold a request open while the entire workflow runs. It can record the starting state, hand the work to the machine, and return.

Correlation IDs

Once a request crosses services, it is easy to lose sight of it. A correlation ID gives that one request a single name that follows it across every boundary.

def handle_request(request):
    correlation_id = (
        request.headers.get("X-Request-ID")
        or generate_request_id()
    )

    response = profiles.get_user(
        request.user_id,
        headers={"X-Request-ID": correlation_id},
        timeout=0.150,
    )

    logger.info(
        "profile_lookup_completed",
        extra={
            "correlation_id": correlation_id,
            "user_id": request.user_id,
            "status": response.status_code,
            "duration_ms": response.duration_ms,
        },
    )

The ID should travel through HTTP headers, queue messages, background jobs, and logs, and it should show up in error reports and traces too. The goal is simple: no matter where you are looking, you can ask "what else belongs to this request?" and get a complete answer.

A few fields are worth carrying alongside every network call:

Without these fields, an incident turns into a search problem at the worst possible time. With them, the answers are already sitting there, waiting to be read.

Distributed tracing

A trace is structured timing for network calls. Where logs tell you what happened, a trace shows you when and in what order, laid out so the shape of the request is visible at a glance.

It answers questions like:

Where did the request spend time?
Which dependency failed?
Which retry attempt succeeded?
Which service returned after the user-facing deadline?

For small systems, logs with correlation IDs are often enough, and there is no need to reach for more. As systems grow, traces become the fastest way to see the path a request actually took. A good trace shows the caller span, the dependency spans, the attempts, the timeouts, and the error statuses, all in one view.

A trace is how a design becomes observable, and it is also how the design quietly tells on itself.

If the trace shows three retries across three layers, the retry policy is not centralized enough. If it shows optional dependencies sitting on the critical path, graceful degradation has not been wired up. If it shows a request still working after the deadline, cancellation is not being propagated. The trace does not scold; it just makes the gaps easy to see, which is the first step to closing them.

Service mesh, carefully

A service mesh or sidecar can take care of cross-cutting network behavior for you:

1. Timeouts.
2. Retries.
3. Circuit breakers.
4. Load balancing.
5. Mutual TLS.
6. Metrics and tracing.

This is genuinely useful, and offloading it is a relief. It also carries one quiet risk: the infrastructure may retry a request that the application never intended to be retried.

The mesh cannot know whether POST /payments is idempotent unless the application contract says so. It cannot know whether a stale fallback is acceptable. It cannot know whether a given timeout means "safe to retry" or "completion unknown." These are meanings, and meaning lives in the application, not in the wire.

So the division of labor is gentle but firm: let the infrastructure enforce the mechanics, and keep the semantics in the application.

mesh:
  connection timeout
  per-route deadline
  circuit breaker
  retry safe reads

application:
  idempotency key
  conflict detection
  pending states
  compensation
  reconciliation

Keeping that split deliberate is what keeps both sides understandable.

When to give up

Every call needs an answer to the question of when to stop trying. Giving up is not a failure of nerve; it is part of the contract, and deciding it ahead of time is a kindness to whoever is on call later.

Giving up can take many shapes, and the right one depends on what the call was:

There is one anti-pattern worth naming, because it is so common. The call waits until every deadline has expired and then returns a bare, ambiguous 500. The caller is given no signal it can act on, so it does the only thing it can: it retries blindly, and the cycle feeds itself. A clear answer, even a clear "no," is almost always kinder than an ambiguous one.

What I notice

Most network bugs, when you sit with them, turn out to be classification bugs.

The code catches Exception, logs "dependency failed," and returns an error. It feels complete. But the real state behind that single line might be any of these:

1. The request never reached the dependency.
2. The dependency rejected it.
3. The dependency completed it and the response was lost.
4. The dependency is still working.
5. The dependency completed an older version of the command.
6. The caller timed out but left work running.

Each of those states asks for a different recovery, and the single error branch erases the difference between them.

The hard part is rarely the code. It is giving each outcome a name, and then deciding, calmly and in advance, what the product should do next. Naming is most of the work.

Design the call contract

The reflex to reach for here is writing the contract down before you depend on the call. For each cross-service call, note:

Then ask the one question again, the same one from the very beginning:

If the response never arrives, what does the caller know?

That single question tends to reveal, gently and immediately, whether the call was designed or only implemented.

Metrics that help

When the contract is in place, a small set of metrics keeps the boundary honest. A first dashboard for network boundaries is worth building around these:

Tail latency is what the user actually feels. A dependency that answers comfortably at p50 can still blow the budget at p99, and it is the p99 that shapes someone's experience. The average hides the moments that matter.

Unknown completions are worth their own metric. They are where correctness work quietly accumulates, and folding them into a generic failure count is how that work stays invisible until it isn't.

What I think

If I had to compress all of this into a sequence, it would be this, taken one calm step at a time:

1. Remove unnecessary network calls from the critical path.
2. Put a deadline on every remaining call.
3. Classify failures into failed, rejected, timeout, and unknown.
4. Retry only operations with a clear idempotency or read-only contract.
5. Add idempotency keys to important commands.
6. Make pending states explicit for unknown write completion.
7. Use compensation for business-level undo.
8. Pass correlation IDs through every boundary.
9. Watch attempts, timeouts, unknowns, and pending reconciliation age.
10. Let infrastructure enforce mechanics, but keep semantics in application code.

The under-engineering move, and usually the right one, is to make fewer calls and make the ones that remain explicit.

A small number of well-defined boundaries stays legible. Each one has a named contract you can read, operate, and reason about directly, without holding the whole system in your head at once. Fewer, clearer boundaries are easier to live with than many clever ones.

Tutorial checklist

For any cross-service write, it is worth filling this out once, on paper or in a doc, before the code feels finished:

Then ask:

Can the same request arrive twice without creating two facts?

If the answer is not yet a clear yes, then retries are not safe yet, and that is simply a sign of where the next bit of work lives.

Summary

1. A network call is a failure boundary.
2. Timeout, failure, rejection, and unknown completion each name their own outcome.
3. Every call needs a deadline that fits inside the caller's deadline.
4. Retries are load multipliers and should require idempotency or read-only semantics.
5. Idempotency keys give one logical command a stable identity.
6. Partial failures need transactions, compensation, or reconciliation.
7. Sagas are durable state machines for cross-service workflows.
8. Correlation IDs and traces make boundaries observable.
9. Service mesh behavior enforces the mechanics; the application holds the semantics.
10. Design the call contract before relying on the call.

---

Pop quiz

Interactive quiz

Network call reliability

A randomized review of network failure modes, timeouts, retries, idempotency, partial failure, and observability.

4of 11 questions

Question 1 of 425%

Why is a network call different from a local function call?

AIt always runs faster than local codeBIt introduces uncertainty about remote receipt, completion, and response deliveryCIt cannot return errorsDIt automatically makes writes idempotent

Next question