Deploys need a release path

A deploy is a production change. That is the whole of it, and it is enough.

Every deploy carries the same few things: a version, a path into service, a set of users who can reach it, health signals to watch, and a condition that tells the rollout when to stop. Once you can see those parts, the change stops being a moment of suspense and becomes something you can hold.

There is one question that organizes all of it:

What has to be true before the next slice of traffic sees this version?

Asked early, that question keeps the deploy small enough to reason about. It gives the system a way to move forward, and it gives the person at the controls a clear place to stop. Nothing more is required of you than to answer it honestly, one slice at a time.

Start with the release path

Three words do a lot of quiet work here, and it helps to let them mean different things.

A deploy places a version in production.

A release exposes behavior to users.

A rollout changes that exposure over time.

They help because each one has its own control points, and naming them separately gives you somewhere precise to act:

Once those controls are separate, you gain room to breathe. A version can sit in production at zero traffic. A feature can be enabled for a single tenant. A rollout can rest at a small percentage for as long as you need. The change is present, and nothing has happened yet that you cannot undo.

The release object

Give the release one durable record, in one place, that everyone can return to.

from dataclasses import dataclass
from datetime import datetime


@dataclass
class Release:
    version: str
    artifact_digest: str
    environment: str
    rollout_percentage: int
    status: str
    started_at: datetime
    owner: str

That record becomes the single thing automation reads and updates, so the truth lives somewhere other than memory.

The useful release states are few, and they stay simple on purpose:

With this record, humans and automation share one view of the same world. There is no private version of the story.

Blue-green deployment

Blue-green deployment keeps two production-ready environments side by side.

One environment serves traffic. The other quietly receives the new version, the migrations that are safe for the current system, the configuration, the warmup, and the smoke checks. When the prepared environment is healthy, routing moves traffic to it. The switch happens only after the new side has earned it.

A few things hold this together:

1. Both environments can run the production workload.
2. Routing can move traffic with a clear command.
3. The previous environment stays available for a defined window.
4. Shared dependencies can handle both versions during the transition.
5. State changes remain compatible across the switch.

Blue-green works well when the serving layer can be duplicated cleanly. The shared parts, databases, queues, caches, and object storage, still ask for release discipline, because they are touched by both sides. When the shared state is already compatible, the traffic switch becomes the easy step it was meant to be.

Rolling deployment

A rolling deployment replaces capacity gradually, a little at a time.

One instance leaves service. The new version starts. Readiness checks pass. The instance returns to service. Then the same small step repeats, instance by instance, until the whole fleet is running the version you intended.

The shape is ordinary, and that is its strength:

drain old instance
start new version
wait for readiness
serve traffic
continue with the next instance

A rolling deployment lives with version overlap, and there is no avoiding it. During the rollout, old and new instances may handle the same user, read the same database rows, consume the same queues, and write the same cache keys. They coexist for a while.

So compatibility is the requirement. If the two versions can hold the same data without confusion, the overlap is nothing to fear.

Canary deployment

A canary sends a small slice of production traffic to the new version, and then watches.

The slice should be stable. A user, tenant, or request key should hash into the same bucket each time, so that one user sees one consistent behavior throughout the measurement window. Stability is a kindness to your future self reading the metrics.

import hashlib


def stable_bucket(key: str, buckets: int = 100) -> int:
    digest = hashlib.sha256(key.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % buckets


def receives_canary(user_id: str, rollout_percentage: int) -> bool:
    return stable_bucket(user_id) < rollout_percentage

That is enough for a percentage rollout. The same user lands in the same bucket while the percentage moves underneath them, and their experience stays steady.

Canary traffic asks for the same monitoring as normal traffic, only broken down by version so you can compare like with like:

The rollout advances when the health gate is satisfied, and not before. There is no rush in it.

Feature flags

A feature flag controls behavior at runtime, which lets you separate shipping code from turning it on.

A flag worth keeping has a name, an owner, a default value, a rollout rule, and a removal date. The removal date matters as much as the rest.

def use_new_ranking(request, flags):
    enabled = flags.enabled(
        "feed.new_ranking",
        actor_id=request.user.id,
        tenant_id=request.tenant.id,
        default=False,
    )

    if enabled:
        return rank_feed_v2(request)

    return rank_feed_v1(request)

The flag lookup should be fast, local where it can be, and observable. When the behavior change is meaningful, let the request log carry the flag state alongside it, so the record explains itself later.

logger.info(
    "feed.ranked",
    extra={
        "event": "feed.ranked",
        "user_id": request.user.id,
        "tenant_id": request.tenant.id,
        "ranking_version": "v2" if enabled else "v1",
        "flag": "feed.new_ranking",
        "flag_enabled": enabled,
        "duration_ms": duration_ms,
        "result": "success",
    },
)

Flags are production code, not notes in the margin. They deserve tests, ownership, cleanup, and a known default, the same care you give anything else that runs.

Rollout gates

A rollout gate is a small decision function, and its smallness is the point.

def rollout_gate(version, baseline, current):
    checks = {
        "error_rate": current.error_rate <= baseline.error_rate * 1.25,
        "p99_latency": current.p99_latency_ms <= baseline.p99_latency_ms * 1.20,
        "success_rate": current.success_rate >= baseline.success_rate * 0.995,
        "saturation": current.max_saturation <= 0.80,
        "unknown_states": current.unknown_state_rate <= baseline.unknown_state_rate,
    }

    return {
        "version": version,
        "passed": all(checks.values()),
        "checks": checks,
    }

The numbers themselves are policy, and you can tune them. What matters is that the decision is explicit and written down by the gate rather than carried in someone's head. A deploy rests easiest when the choice to advance or stop is something you can point to.

Good rollout gates tend to share these qualities:

1. They use metrics tagged by version.
2. They wait long enough to collect useful data.
3. They include user-visible success signals.
4. They include saturation signals.
5. They produce a decision record.
6. They page a human when the decision needs human judgment.

From there the gate can advance, pause, or roll back. Three calm options, each of them safe.

Automatic rollback

Rollback is a routing action first, before it is anything else.

The simplest automatic rollback just moves traffic away from the version that failed the gate, back to the version that was already serving well.

def evaluate_release(release):
    baseline = metrics.for_version(release.previous_version)
    current = metrics.for_version(release.version)
    decision = rollout_gate(release.version, baseline, current)

    audit_log.write(
        {
            "event": "release.gate_evaluated",
            "version": release.version,
            "rollout_percentage": release.rollout_percentage,
            "passed": decision["passed"],
            "checks": decision["checks"],
        }
    )

    if decision["passed"]:
        return advance_rollout(release)

    route_traffic(version=release.previous_version, percentage=100)
    mark_release(release.version, status="rolled_back")
    alert_release_owner(release, decision)
    return release

Rollback works cleanly when the release kept its state compatible all along. Code can move back the moment the data still makes sense to the previous version. The exit was prepared before it was ever needed.

That is why database migrations matter so much.

Database changes

Database changes have to work across the entire rollout window, while both versions are alive.

A common and dependable sequence is:

1. Expand the schema.
2. Deploy code that can read and write both shapes.
3. Backfill existing rows.
4. Move reads to the new shape.
5. Stop writing the old shape.
6. Contract the schema after the old version is gone.

As an example, take splitting users.name into first_name and last_name. Each step below leaves the system whole.

First, add nullable columns:

ALTER TABLE users ADD COLUMN first_name text;
ALTER TABLE users ADD COLUMN last_name text;

Then write both shapes:

def update_name(user_id, full_name):
    first_name, last_name = split_name(full_name)

    db.execute(
        """
        UPDATE users
        SET name = %(full_name)s,
            first_name = %(first_name)s,
            last_name = %(last_name)s
        WHERE id = %(user_id)s
        """,
        {
            "user_id": user_id,
            "full_name": full_name,
            "first_name": first_name,
            "last_name": last_name,
        },
    )

Then read with a fallback:

def display_name(user):
    if user.first_name is not None and user.last_name is not None:
        return f"{user.first_name} {user.last_name}"

    return user.name

Backfill in batches:

def backfill_names(batch_size=500):
    rows = db.query(
        """
        SELECT id, name
        FROM users
        WHERE first_name IS NULL
        ORDER BY id
        LIMIT %(batch_size)s
        """,
        {"batch_size": batch_size},
    )

    for row in rows:
        first_name, last_name = split_name(row.name)
        db.execute(
            """
            UPDATE users
            SET first_name = %(first_name)s,
                last_name = %(last_name)s
            WHERE id = %(user_id)s
              AND first_name IS NULL
            """,
            {
                "user_id": row.id,
                "first_name": first_name,
                "last_name": last_name,
            },
        )

Once the rollout and backfill are both complete, a later migration can remove the old column. The removal belongs to its own release, given its own time. There is no need to crowd everything into one risky moment.

Queues and workers

Deploys move through queues too, in their own time, not just through request traffic.

A producer may publish a new event field before every consumer has the new code. A worker may read a job created by the previous version. A retry may replay a payload from hours ago, long after the deploy you thought was finished. None of this is a problem if you expect it.

So queue payloads carry a few compatibility rules:

1. Add fields as optional first.
2. Keep old fields during the rollout.
3. Give events a schema version.
4. Make consumers tolerant of unknown fields.
5. Keep handlers idempotent.
6. Hold old handler code until queued work drains or expires.

Example:

def handle_user_updated(event):
    version = event.get("schema_version", 1)

    if version == 1:
        email = event["email"]
        display_name = event.get("name", "")
    else:
        email = event["email"]
        display_name = " ".join(
            part for part in [event.get("first_name"), event.get("last_name")] if part
        )

    search_index.update_user(
        user_id=event["user_id"],
        email=email,
        display_name=display_name,
        event_id=event["event_id"],
    )

The worker treats old payloads as ordinary input, not as exceptions. Nothing breaks, because nothing was surprised.

Configuration

Configuration is part of the release, every bit as much as code.

Configuration worth trusting has:

Let configuration changes show up in the release record and in the logs for affected requests. A silent config change can wear the mask of a code regression during an incident, and at three in the morning that disguise costs real time. Visibility is mercy.

Smoke tests and readiness

Smoke tests check the version in its environment.

Readiness checks decide whether an instance is ready to receive traffic.

A small smoke test suite walks the production path at low volume:

1. Start the process.
2. Load configuration.
3. Connect to required dependencies.
4. Serve a health endpoint.
5. Execute one read path.
6. Execute one write path in a safe test scope.
7. Emit a structured log event.
8. Export metrics with the version tag.

Readiness should cover the dependency state the request path truly needs, stay cheap to run, and keep unavailable instances out of rotation until they are genuinely ready. It is a small gate that prevents a lot of grief.

What I notice

Calm deploy systems have a name for each stage.

The version is named. The artifact is named. The rollout scope is named. The flag is named. The health gate is named. The rollback condition is named. The database state is named. Naming a thing is how you stop being afraid of it.

When all of that is named, the team can point to one release record and simply see what is happening, together.

The systems that feel anxious are the ones that scatter those facts across chat messages, dashboards, a terminal history on someone's laptop, and memory. Nothing is wrong with the people; the path is just hidden from them.

So the work, gentle and unglamorous, is to make the path visible.

Metrics that help

The deployment metrics that earn their keep are tagged by version, environment, region, and rollout_percentage, so you can always ask which slice of the world a number belongs to.

The release process leaves logs of its own, too:

release.prepared
release.deployed
release.canary_started
release.gate_evaluated
release.rollout_advanced
release.paused
release.rolled_back
release.completed

With those events, deployments become queryable in the same system as application behavior. The deploy stops being a separate, mysterious thing and joins the rest of what you can see.

What I think

The practical sequence, in order, is this:

1. Build one artifact and give it a digest.
2. Deploy the artifact to production capacity.
3. Keep release behavior behind a flag when the behavior is meaningful.
4. Start with a small stable cohort.
5. Tag logs, metrics, and traces with the version and flag state.
6. Evaluate the health gate after enough traffic arrives.
7. Advance the rollout in clear steps.
8. Pause when the signal is incomplete.
9. Route traffic back when the gate fails.
10. Design database and queue changes for overlapping versions.
11. Remove old flags, columns, and compatibility paths in later releases.

The under-engineering move, the one that quietly saves you, is to make the deploy path explicit and mechanical. Let the steps be boring. Boring is what lets you stay calm.

A small release system with clear gates is genuinely enough for most teams. You do not need more than this to be safe.

Tutorial checklist

For any production deploy, it helps to fill this out before you begin:

Then ask:

Can this release pause or move back while keeping data in a shape the old code can read?

A complete release path can answer yes to that question without hesitation. When it can, the deploy is no longer something to brace against. It is just a change you can make, and unmake, in the open.

Summary

1. A deploy places a version in production.
2. A release exposes behavior to users.
3. A rollout changes exposure over time.
4. Blue-green deployment keeps two production-ready environments.
5. Rolling deployment replaces capacity gradually.
6. Canary deployment sends a small stable slice of traffic to the new version.
7. Feature flags control runtime behavior and need ownership.
8. Rollout gates turn health metrics into advance, pause, or rollback decisions.
9. Automatic rollback works cleanly when state remains compatible.
10. Database migrations need an expand, overlap, backfill, and contract sequence.
11. Queue payloads need compatibility across producer and consumer versions.
12. A release record makes the deploy path visible and auditable.

---

Pop quiz

Interactive quiz

Deployment and release safety

A gentle, randomized review of release paths, canaries, feature flags, rollout gates, rollback, and compatible migrations.

4of 8 questions

Question 1 of 425%

What does a rollout name?

AThe laptop used to write the codeBThe amount of exposure a version or behavior receives over timeCA permanent replacement for logsDThe SQL type of a new column

Next question