Consistency has a cost

Consistency bugs are quiet at the start.

A user changes their display name and sees the old one on another screen. A follower count dips, then climbs back. A search result still shows a deleted post for a few seconds. An inventory number says one item remains, but checkout fails.

It is tempting to treat each of these as its own small puzzle, and each one does need its own fix. But underneath them sits a single question, and it is narrower than it first appears:

What promise does this read need from the write that came before it?

Sometimes the answer is "the latest value, always." Sometimes it is "the user's own write should be visible to them." Sometimes it is "within a few seconds is fine." Once the promise is named, the architecture has something to follow. Most of this post is just learning to ask that question a little earlier.

Start with the read promise

Consistency is a promise made by a read path. Not a property of the database, not a setting to turn up. A promise one read makes about what it will show.

These are the common ones:

Strong consistency feels safer, so it is easy to reach for everywhere. Eventual consistency feels scalable, so it is easy to reach for everywhere. Both instincts are reasonable, and both go wrong when applied without a reason.

Each level only earns its meaning from the product promise behind it. The level is the answer; the promise is the question.

The display name bug

Here is a common one, traced step by step:

1. User updates display name from "Mina" to "Mina K."
2. API writes the leader database.
3. Profile page reads from a replica.
4. Replica is 800ms behind.
5. User sees the old name and retries the update.

The write worked. Nothing was lost. What failed was the read promise: the writer expected to see their own change, and the replica had not caught up yet.

There are several fixes, and they cost different things:

The fix follows from one question: which read went stale, for whom, and for how long? Answer that, and the right row in the table usually picks itself.

CAP, the useful part

The simplified CAP line is usually "consistency, availability, partition tolerance: pick two." It is memorable, and on its own it does not help much with a real decision.

The version that does help asks one thing:

When the network is unreliable, should this operation reject work or serve a possibly stale answer?

Partitions rarely arrive as a dramatic outage. They take quiet forms. A replica falls behind. A region loses a link. A queue consumer pauses. A deployment routes traffic unevenly. In each of these, the system has to choose, and it is better if you chose first.

For an account balance write, reject or wait.

For a follower count read, serve the stale count.

For a permission check, read the authoritative source or fail closed.

For a recommendation module, return an empty list or a cached result.

CAP earns its keep when it pushes this decision into the design, where you can make it calmly, rather than into an incident, where you cannot.

Strong consistency is coordination

Strong consistency means the system coordinates before it answers. That is the whole idea, and it is also the whole cost.

The costs are concrete and worth naming plainly:

1. Extra network round trips.
2. More time spent holding locks or transactions open.
3. Lower availability during replica or region problems.
4. More pressure on the leader or quorum.
5. More complicated failover behavior.

None of these are bugs. They are the price of agreement, and that price is exactly why strong consistency is worth reserving for the places that genuinely need it: the invariants.

Moving credits between accounts is one of those places.

def transfer_credits(from_account_id, to_account_id, amount):
    with db.transaction() as tx:
        debited = tx.execute(
            """
            UPDATE accounts
            SET balance = balance - %(amount)s
            WHERE id = %(from_account_id)s
              AND balance >= %(amount)s
            """,
            {
                "from_account_id": from_account_id,
                "amount": amount,
            },
        )

        if debited.row_count != 1:
            raise InsufficientFunds()

        tx.execute(
            """
            UPDATE accounts
            SET balance = balance + %(amount)s
            WHERE id = %(to_account_id)s
            """,
            {
                "to_account_id": to_account_id,
                "amount": amount,
            },
        )

        tx.insert_ledger_entry(
            from_account_id=from_account_id,
            to_account_id=to_account_id,
            amount=amount,
        )

The transaction is the consistency boundary. Everything inside it agrees before anything is acknowledged.

This path reads from the leader to decide whether money can move, because a stale read here does not produce a slightly old screen. It produces an incorrect balance, and that is the kind of mistake the coordination exists to prevent.

Eventual consistency is a lag budget

Eventual consistency is gentler, and it deserves a clear definition rather than a vague hope:

This derived value may lag the source of truth by up to N seconds.
If lag exceeds N, the product changes behavior.

"Eventual" is honest only when N is a number you have chosen and can see. For a public profile card, the source of truth might be users, while search, feed headers, and mention previews update from an event.

def update_profile(user_id, display_name):
    with db.transaction() as tx:
        version = tx.next_profile_version(user_id)
        tx.update_user_profile(
            user_id=user_id,
            display_name=display_name,
            version=version,
        )
        tx.insert_outbox_event(
            event_type="profile.updated",
            aggregate_id=user_id,
            payload={
                "user_id": user_id,
                "version": version,
            },
        )

    return {
        "user_id": user_id,
        "display_name": display_name,
        "version": version,
    }

Workers update derived views:

def handle_profile_updated(event):
    user_id = event.payload["user_id"]
    version = event.payload["version"]

    profile = db.get_user_profile(user_id)

    search.upsert_user(
        user_id=user_id,
        display_name=profile.display_name,
        source_version=version,
    )

    cache.delete(f"profile_card:v3:user:{user_id}")

Search may lag, and that is fine by design. The profile edit response returns on its own as soon as the write commits, and search catches up within its lag budget.

The part that matters is that the lag is measured, not assumed.

Read-your-writes scoped to the writer

A lot of product paths ask for less than they seem to. The writer needs to see their own latest write. Everyone else can see it a little later, and usually does not notice.

A simple pattern covers this: return a version from the write, then carry it into the next read.

def update_settings(user_id, settings):
    version = db.update_settings(
        user_id=user_id,
        settings=settings,
    )

    return {
        "settings": settings,
        "version": version,
    }


def get_settings(user_id, min_version=None):
    if min_version is None:
        return replicas.any().get_settings(user_id)

    replica = replicas.find_one_with_version_at_least(
        user_id=user_id,
        version=min_version,
        timeout_ms=80,
    )

    if replica is not None:
        return replica.get_settings(user_id)

    return db.leader().get_settings(user_id)

Normal reads stay cheap. Only the reads that require a known version pay the cost of the leader or the wait, and those are the minority.

This works because of a quiet fact about traffic: most reads arrive well after the write they might have chased, so they have nothing to wait for.

Quorums

Quorums take the read/write tradeoff and make it arithmetic, which is a relief, because arithmetic is easy to reason about.

With N replicas:

W = number of replicas that must confirm a write
R = number of replicas that must answer a read

If W + R > N, the read set and the write set must share at least one replica. In the simple case, that shared replica is what lets a read observe the latest write.

For N = 3, the shapes look like this:

The idea, stripped down to what it is:

def quorum_intersects(replica_count, write_confirmations, read_confirmations):
    return write_confirmations + read_confirmations > replica_count


assert quorum_intersects(3, 2, 2)
assert not quorum_intersects(3, 1, 1)

Real systems carry far more than this. Clocks, leader terms, hinted handoff, read repair, partitions, and stale replicas all complicate the picture, and none of them disappear. But the core stays true and worth holding onto: stronger reads and writes need more agreement, and more agreement costs latency and availability. The arithmetic just makes that trade visible.

Replication lag is a product metric

Once a feature leans on eventual consistency, replica lag stops being an infrastructure detail. It becomes part of how the product behaves, and so it deserves to be watched the way product behavior is watched.

A few metrics carry most of the signal:

If users can observe the lag, it should have an SLO, the same as any other promise:

profile projection p99 lag < 5 seconds
search indexing p95 lag < 30 seconds
analytics dashboard lag < 10 minutes

Without a lag target, "eventual" has no boundary, and a system with no boundary cannot tell you when it has drifted out of bounds.

Conflicts are design problems

Conflicts arise when two writers can update related state and no single authority decides between them. They feel like runtime accidents, but they are usually decided much earlier, in the shape of the data model.

So the easiest conflict strategy is the one that prevents the conflict from being possible:

1. One owner for each entity.
2. One partition key for ordered updates.
3. One writer for a field when possible.
4. Optimistic concurrency for user-edited documents.

Optimistic concurrency is often enough:

UPDATE documents
SET body = $1,
    version = version + 1,
    updated_at = now()
WHERE id = $2
  AND version = $3;

If zero rows update, the client was editing an old version. The product can then ask them to reload, merge, or choose between versions, calmly, before anything is overwritten.

Last-write-wins keeps the most recent value and drops the rest. That is exactly right for fields where the latest setting is the whole intent:

display theme
sort preference
temporary draft title

And it is exactly wrong for fields where each write carries intent that must survive. Dropping a write here loses something real, so the conflict needs domain-specific handling:

account balance
inventory reservation
document body
permission grant
seat assignment

For counters and sets with multiple writers, reach for a known conflict-free approach or a database feature that already implements the merge semantics. Hand-rolling CRDTs is rarely where I would start; it tends to be the right tool only when the product genuinely needs offline multi-writer edits, and most products do not.

What I notice

A lot of confusion clears up once you separate three questions that consistency discussions tend to blur together:

1. Can the system store the correct fact?
2. Can every reader see that fact immediately?
3. Can the product tolerate some readers seeing an older fact?

They are separate questions, and a healthy system often answers them differently. It can hold a strongly consistent source of truth and serve eventually consistent read models from it. That combination is not a compromise; it is the common, practical shape.

A user profile might be strongly written to users, then eventually projected out to search, feed headers, notifications, and caches. One fact, many read paths, each with its own promise.

The bugs tend to come from a single, forgivable lapse: forgetting which of those reads are eventual.

Make pending states explicit

Async systems live in the middle. Work is accepted but not yet done, and that in-between is a real state, not a gap to paper over. The kindest thing the product can do is name it.

Some examples:

The rule is quiet but firm: the backend calls a projection complete only once it is complete.

When the write is accepted but the derived work is still in flight, the API response should say so plainly.

{
  "status": "saved",
  "search_status": "indexing",
  "profile_version": 42
}

Precise state names quietly remove a whole class of trouble: fewer retry loops, fewer support tickets, fewer clients guessing wrong about work that simply was not finished yet.

Strong where the invariant lives

A split I keep coming back to, as a starting point rather than a rule:

The search index can stay eventual alongside the profile table. The edit returns on its own, search catches up after, and almost no one is harmed by the gap. Coupling them strongly belongs to the rare product that truly needs search to update inside the edit response.

Inventory is the opposite case, and worth keeping strong when the product sells scarce items and every reservation must hold. Coordinated reservations are simply what keeps the last item from being sold twice.

Repair is part of eventual consistency

Eventual consistency is not complete without a repair path. The repair path is what keeps the word "eventual" honest when an event is dropped, and events do get dropped. It is the quiet safety net underneath the promise.

The common tools are unglamorous, which is a good sign:

1. Outbox replay from a durable source.
2. Periodic reconciliation from source of truth to projection.
3. Dead-letter queues for poison events.
4. Version checks on projections.
5. Idempotent handlers that can run more than once.

Example reconciliation:

def repair_profile_projection(batch_size=500):
    stale_profiles = db.find_profiles_newer_than_search(batch_size)

    for profile in stale_profiles:
        search.upsert_user(
            user_id=profile.user_id,
            display_name=profile.display_name,
            source_version=profile.version,
        )

A good repair job is boring, and it should stay that way. It compares the source version to the projection version and updates whatever is missing or old. Nothing clever.

There is one line worth remembering here: if a derived system cannot be repaired from the source of truth, then it is itself a source of truth, whether or not you meant it to be, and its consistency deserves to be treated with that seriousness.

Multi-region makes the tradeoff visible

Inside a single region, the round trips that strong consistency needs stay short. The cost is there, but small enough that many paths can comfortably afford it.

Across regions, those same round trips stretch over real distance, and the cost stops hiding.

A write that must coordinate between Tokyo, Virginia, and Frankfurt waits on the speed of light, on network variance, on failure handling. If every profile edit needs global agreement before it returns, the user pays that toll on every single edit, and they feel it.

A few common patterns:

A tight consistency boundary is what buys this freedom. Keep the coordinated work local and small, and the rest of the system is free to spread across regions without dragging that latency along.

What I think

When I have to do this for real, the sequence is roughly this:

1. Write down the invariant.
2. Decide which reads must observe it immediately.
3. Put those reads on the coordinated path.
4. Give every derived read model a lag budget.
5. Return versions from writes.
6. Use read-your-writes only for users who need it.
7. Make conflict policy explicit.
8. Measure lag and stale reads.
9. Build repair from the source of truth.

If there is one underengineering move to keep, it is this: make only the invariant strong.

That single discipline keeps the expensive guarantee close to the data that actually needs it, and lets everything else breathe.

Tutorial checklist

For any consistency-sensitive feature, it helps to fill this out before writing code:

And then one closing question, which sorts most decisions on its own:

If a stale read happens here, is it annoying, misleading, or incorrect?

Annoying can often stay eventual.

Misleading needs product handling.

Incorrect needs coordination.

Three words, and most of the design falls into place behind them.

Summary

If you carry only a handful of things away from this, let it be these:

1. Consistency is a read promise that each read path makes.
2. Strong consistency is coordination, and coordination costs latency and availability.
3. Eventual consistency should have a lag budget and repair path.
4. Many systems use a strong source of truth with eventual read models.
5. Read-your-writes can be targeted with versions and leader fallback.
6. Quorums make read/write agreement explicit, and the operational complexity stays for you to handle.
7. Conflict resolution should follow product semantics; last-write-wins fits fields whose latest value is their whole intent.
8. Pending states should be named when derived work happens later.
9. Measure replica lag, projection lag, stale reads, conflicts, and repair activity.
10. Make the invariant strong, then let lower-risk projections lag intentionally.

---

Pop quiz

Interactive quiz

Consistency tradeoffs

A randomized review of strong consistency, eventual consistency, lag budgets, quorums, and conflict handling.

4of 10 questions

Question 1 of 425%

What is the first consistency question to ask for a read path?

AWhich database has the most replicas?BWhat promise this read needs from the write that came before itCHow to remove every cache from the systemDWhether all data can be stored in memory

Next question