The write path is your architecture

Most architecture diagrams leave out the part that matters most.

They show boxes. API, database, queue, worker, cache, search. Sometimes arrows between them. It looks complete, and it tells you almost nothing about how the system behaves.

There is a narrower diagram that does.

It answers one question: what happens after a user clicks submit?

That is the write path. It decides what the user waits for, what can be retried, what can be lost, what needs repair, and which read models are even possible later. When the write path is clear, the architecture is clear. When it is vague, everything downstream inherits the vagueness.

So it is worth sitting with this one question for a while.

Start with the acknowledgement boundary

There is one question to ask before any of the others:

What must be true before we tell the user the write succeeded?

That line is the acknowledgement boundary. Everything in the design tends to fall into place once you can answer it honestly.

For a blog comment, success might mean the canonical comment row exists. Notifications, search indexing, abuse scanning, and feed fanout can all happen later.

For a bank transfer, success might mean the ledger entries are durably committed, balanced, and visible to the account owner. The email receipt can happen later.

For a file upload, success might mean the object bytes and metadata are durable. Thumbnail generation can happen later.

Place the boundary far downstream and the user waits on work they never asked about. Place it too early and you say "success" for work that can still quietly disappear. The whole craft here is finding the line that is both true and small.

The naive write path

This is the shape most systems start with, and there is nothing shameful about it:

def create_post(user_id, content):
    post = db.insert_post(user_id, content)
    cache.invalidate(f"feed:{user_id}")
    search.index_post(post)
    notifications.send_to_followers(user_id, post)
    analytics.track("post.created", user_id=user_id, post_id=post.id)
    return post

It reads top to bottom and you understand it immediately.

It also adds latency. The user waits for the database, the cache, search, notifications, analytics, and every network hop hiding behind those calls.

And it is fragile under partial failure. Picture the sequence:

1. The post insert succeeds.
2. Search indexing times out.
3. The API returns 500.
4. The user retries.
5. The second request creates a duplicate post.

Now the code needs cleanup logic, dedupe logic, and a support ticket.

Synchronous writes still have their place. The thing worth noticing is subtler: this function blends the canonical write and the derived side effects into one undivided block, and the boundary between them is left implicit. The cost is not in any single line. It is in the missing line that would have separated them.

What I notice

The write path tends to grow quietly.

One team adds analytics. Another adds notifications. Another adds search. Another invalidates a cache. Another writes audit logs. The endpoint still looks like one tidy function, but the product contract underneath it has changed.

The user thinks they are creating a post. The system is doing six things.

So when p99 latency rises, the database usually gets blamed first, because it is the obvious stateful component sitting in the diagram. The real cause is usually elsewhere. Without anyone deciding it, the request path has become an integration path.

The fix is simply to name the critical path, out loud, on purpose.

Split canonical writes from derived work

A write path that scales tends to settle into this shape:

request
  -> validate input
  -> write canonical fact
  -> enqueue durable event
  -> return

worker
  -> consume event
  -> update read models
  -> send notifications
  -> index search
  -> update analytics

The user waits for the fact. Workers handle the effects. That single division is most of the work.

def create_post(user_id, content, idempotency_key):
    post_id = generate_id()

    with db.transaction() as tx:
        existing = tx.get_idempotency_result(
            user_id=user_id,
            key=idempotency_key,
        )
        if existing:
            return existing

        post = tx.insert_post(
            id=post_id,
            user_id=user_id,
            content=content,
            status="published",
        )
        tx.insert_outbox_event(
            event_id=generate_id(),
            event_type="post.created",
            aggregate_id=post_id,
            payload={
                "post_id": post_id,
                "user_id": user_id,
            },
        )
        tx.save_idempotency_result(
            user_id=user_id,
            key=idempotency_key,
            response={"id": post_id, "status": "published"},
        )

    return {"id": post_id, "status": "published"}

Three details carry the weight here.

First, the canonical row and the event are written in the same database transaction.

Second, the idempotency result is stored in that same transaction.

Third, nothing calls search, notifications, or analytics before returning to the user.

The request path stays small, and derived work is treated as real work that runs on its own schedule. Nothing is dropped. It is just held in the right place.

The outbox pattern

Queues open a small gap between the database commit and the queue publish. Small, but it is exactly where things go wrong.

Watch what the gap does:

def create_post(user_id, content):
    post = db.insert_post(user_id, content)
    queue.publish("post.created", {"post_id": post.id})
    return post

If the database write succeeds and the queue publish fails, the post exists but no worker ever hears about it. The fact is real and the effects never come.

The outbox pattern closes that gap by writing the event into the same database transaction as the canonical fact. Either both land or neither does.

CREATE TABLE posts (
  id bigint PRIMARY KEY,
  user_id bigint NOT NULL,
  content text NOT NULL,
  status text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox_events (
  id bigint PRIMARY KEY,
  event_type text NOT NULL,
  aggregate_id bigint NOT NULL,
  payload jsonb NOT NULL,
  published_at timestamptz,
  created_at timestamptz NOT NULL DEFAULT now()
);

Then a relay publishes the unpublished events:

def publish_outbox_batch(limit=100):
    events = db.fetch_unpublished_outbox_events(limit=limit)

    for event in events:
        queue.publish(
            event.event_type,
            key=str(event.aggregate_id),
            payload=event.payload,
        )
        db.mark_outbox_event_published(event.id)

This relay can crash after publishing but before marking the event as published. When that happens, workers receive duplicates.

That is fine. It is the expected shape of the world, not a defect.

The system should be designed around at-least-once delivery, unless you have a very specific reason and the infrastructure to support something stricter.

Idempotency is the retry contract

Retries are everywhere, whether you plan for them or not.

Clients retry after timeouts. Load balancers retry. Workers retry. Operators replay events after a bug. You replay outbox rows after a deploy. A mobile app sends the same request twice because the connection died before the response arrived.

So if retries are unsafe, the system is unsafe. There is no way around it.

For external API writes, require an idempotency key:

CREATE TABLE idempotency_keys (
  user_id bigint NOT NULL,
  key text NOT NULL,
  request_hash text NOT NULL,
  response jsonb NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (user_id, key)
);

The key carries a single meaning: for this user, this logical command may run once.

The request hash matters because clients sometimes reuse keys for different payloads by accident. When the same key arrives with a different request body, return a conflict. That keeps the server honest about what each key actually represents.

For worker handlers, make each effect idempotent too:

def handle_post_created(event):
    if db.event_already_processed(event.id, handler="feed_projection"):
        return

    with db.transaction() as tx:
        tx.upsert_feed_item(
            user_id=event.payload["user_id"],
            post_id=event.payload["post_id"],
        )
        tx.mark_event_processed(
            event_id=event.id,
            handler="feed_projection",
        )

The handler name is the quiet hero here. Search indexing, feed projection, notifications, and analytics are different effects, and each one needs its own dedupe boundary. They process the same event, but they are not the same work.

A queue keeps the hard decisions yours

A queue helps when work can happen later. What it does not do is make the decisions for you.

You still choose:

A queue mostly shapes how a spike lands.

A direct path turns a spike into waiting. Users hold the connection while the work runs.

A queue turns a spike into lag. The work accumulates behind the boundary and drains over time.

A queue earns its place when that lag is visible and bounded. Otherwise it just moves the problem somewhere you can no longer see it.

Backpressure

Backpressure is the system saying, plainly: I cannot accept work at this rate and still keep my promises.

Ignoring it means returning success for work that is already falling behind.

The arithmetic makes it concrete:

1. The API accepts 20,000 writes per second.
2. Workers can process 5,000 events per second.
3. The queue grows by 15,000 events per second.
4. Notification lag reaches 45 minutes.
5. The API keeps returning 200 while the product feels broken to the person using it.

It helps a great deal to decide on a policy before reaching that point, while there is still room to think.

A few options:

1. Return 429 or 503 when queue lag crosses a threshold.
2. Accept the write but mark derived effects as delayed.
3. Drop low-value effects such as analytics while preserving canonical writes.
4. Degrade expensive fanout into smaller batches.
5. Route large tenants through separate partitions.

The right policy depends entirely on the product.

For payments, slow intake down before risking ledger correctness.

For social notifications, preserve the post and let the notification arrive late.

For analytics, sample or drop events before touching the user-facing write.

What streams, queues, and direct writes each do

These words get overloaded, and the overload causes most of the confusion.

Here is what each one is actually for:

Seen through examples:

Direct write:
  create ledger entry before returning payment success

Queue:
  resize uploaded image
  send email receipt
  run abuse scan

Stream/log:
  post.created feeds search, ranking, notifications, analytics, audit

Use a queue when you want to distribute work.

Use a stream when you want a durable history that multiple consumers can read and replay independently.

Use a direct write when the user-facing command simply is not true until that write completes.

Ordering comes from partition keys

Ordering is not global by default, and assuming otherwise is a common source of quiet bugs.

Most scalable logs order records within a partition. So the key you choose is the thing that decides where order is preserved.

For posts, post_id is a reasonable key if all events for one post need to stay in order:

post.created
post.updated
post.deleted

For account balances, the key is usually account_id, because operations for one account need a single, undisputed sequence.

For notifications, the key may be recipient_user_id, so one person's notification timeline stays ordered.

The wrong key creates subtle bugs that are hard to trace later.

If every event uses one global key, a single partition becomes hot.

If events use random keys, related updates can be processed out of order.

Pick the key based on the invariant you actually need to preserve, and nothing more.

Update-in-place and append-only

A current-state table holds the latest value:

UPDATE posts
SET content = $2,
    updated_at = now()
WHERE id = $1;

An append-only log stores facts instead:

INSERT INTO post_events (
  id,
  post_id,
  event_type,
  payload,
  created_at
) VALUES (
  $1,
  $2,
  'post.updated',
  $3,
  now()
);

Append-only is useful when history matters, when replay matters, or when several projections need to be rebuilt from the same source.

Each shape earns its place. There is no winner.

Current-state tables answer "what is true now" in a single read. Logs preserve every fact in sequence, ready to audit and replay. Many systems run both together, side by side:

post_events     -> source of historical facts
posts           -> current state projection
feed_items      -> read model
search_index    -> external projection

This is event sourcing in the small. You do not need to redesign the whole company around it. You can keep append-only facts in the few places where replay and audit justify the cost, and leave the rest alone.

A practical decision matrix

Before designing a write path, it helps to fill this out:

My default for product systems is DB + outbox.

It keeps a simple source of truth, gives workers a durable event stream, and avoids coupling the request path to every side effect. It is rarely the most clever choice, and that is part of why it holds up.

Queue first can be the right call when the command itself is naturally asynchronous:

import a CSV
train a model
transcode a video
send a campaign

In those cases the user should get a job id and a status endpoint:

def start_import(user_id, file_id):
    job_id = generate_id()
    queue.publish("import.requested", {
        "job_id": job_id,
        "user_id": user_id,
        "file_id": file_id,
    })
    return {"job_id": job_id, "status": "queued"}

And it is worth being honest in the product language. If the work is queued, call it queued. Reserve "done" for work that has actually finished.

Failure states are product states

Async writes create intermediate states. They exist whether or not you acknowledge them.

So name them.

CREATE TABLE import_jobs (
  id bigint PRIMARY KEY,
  user_id bigint NOT NULL,
  status text NOT NULL CHECK (
    status IN ('queued', 'processing', 'completed', 'failed')
  ),
  error_message text,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now()
);

The status model is part of correctness, not decoration around it.

If a user can start a job, refresh the page, and see nothing, the write path is incomplete. The request returned before the real work finished, so the product needs an honest way to represent the work that is still pending.

What I think

The mental model I keep returning to is small:

command -> fact -> effects -> projections

A command is what the user asks for.

A fact is what the system durably records.

Effects are the things caused by the fact.

Projections are read models built from facts.

Most tangled architectures simply blur these together. A request handler validates a command, writes a fact, mutates three projections, calls two external systems, and returns success based on whatever happened to finish last.

That is how a write path becomes untestable, quietly, one helpful addition at a time.

Separate the pieces. Then decide, deliberately, where the acknowledgement boundary belongs. Most of the difficulty dissolves once those two things are done.

Tutorial checklist

For any important write path, it is worth filling this out:

Then sit with the harder question:

What happens if every step after the acknowledgement boundary fails?

If the answer is "we do not know," the write path is not designed yet. And that is alright. It just means there is one more honest question to answer before you ship.

Summary

1. The write path is the real architecture for user-facing state changes.
2. Start by defining the acknowledgement boundary.
3. Keep the request path limited to the smallest durable product promise.
4. Move derived effects behind queues, workers, or streams.
5. Use an outbox when a database write and event publish must move together.
6. Assume retries and duplicates; make commands and handlers idempotent.
7. Treat queue lag and backpressure as product concerns that shape the user experience.
8. Pick partition keys based on the ordering invariant you need.
9. Async workflows need explicit pending, completed, and failed states.

---

Pop quiz

Interactive quiz

Write path design

A randomized review of acknowledgement boundaries, outbox events, idempotency, and backpressure.

4of 10 questions

Question 1 of 425%

What is the acknowledgement boundary?

AThe first line of code in a request handlerBThe set of conditions that must be true before returning success to the userCThe number of workers assigned to a queueDThe maximum size of a database transaction log

Next question