Logs are data too

Logs usually begin as small notes a program leaves for itself.

One line when a request starts. One line when it fails. A few values dropped into a string because someone, someday, might want to see them.

For a single process on a single host, that is enough. But add more services, more hosts, queues, retries, and deploy versions, and those notes quietly become something larger: data produced by the running system, scattered across every machine that touched the request.

So there is one question worth holding onto:

Could we query this record during an incident without already knowing the answer?

When the answer is yes, the log has earned its place. When the answer is no, it rests in the file as decoration, present but silent in the moment you most need it to speak.

Start with the record

A log line is a record, and the message is just one field within it.

The most useful fields tend to be the most ordinary ones:

The fields are the interface. When the only stable part of a log is the sentence, querying it later turns into a kind of string archaeology, slow and uncertain.

Logs are records first

This line is quick to type and reads cleanly on a local terminal:

logger.info(f"User {user_id} created post {post_id} in {duration_ms}ms")

Its values live inside a sentence written for a human eye. But to count post creation latency by service version, a log system has to take that sentence apart again, undoing the work that made it readable.

This shape keeps the same values in named fields:

logger.info(
    "post_created",
    extra={
        "event": "post.created",
        "service": "posts-api",
        "version": app_version,
        "environment": environment,
        "correlation_id": request.headers.get("X-Request-ID"),
        "user_id": user_id,
        "post_id": post_id,
        "content_length": len(content),
        "duration_ms": duration_ms,
        "result": "success",
    },
)

The sentence can still exist for the human. The data simply lives, in parallel, in named fields for the machine. Both can be true at once.

Event names

An event name should be stable enough to query for months without changing under you.

Good event names tend to be:

1. Specific.
2. Lowercase.
3. Past tense for facts.
4. Consistent across services.
5. Not packed with variable data.

For example:

The left column names a stable kind of fact and stays the same across millions of events. The right column folds variable data into prose, so each row becomes a fresh string to match against.

A simple division keeps things clear: the event name answers "what kind of thing happened?" and the fields answer "to which object, under which request, with which result?"

Correlation IDs

A correlation ID gives a single logical operation one name to carry as it crosses services, queues, workers, and storage systems. Wherever the work travels, the name travels with it.

At the edge, generate one if the caller did not send one:

def get_correlation_id(request):
    incoming = request.headers.get("X-Request-ID")
    if incoming:
        return incoming[:128]

    return uuid.uuid4().hex

Then pass it through every boundary:

def create_post(request):
    correlation_id = get_correlation_id(request)

    post = db.insert_post(
        user_id=request.user.id,
        content=request.json["content"],
    )

    queue.publish(
        "post.created",
        {
            "post_id": post.id,
            "user_id": request.user.id,
            "correlation_id": correlation_id,
        },
    )

    logger.info(
        "post_created",
        extra={
            "event": "post.created",
            "post_id": post.id,
            "user_id": request.user.id,
            "correlation_id": correlation_id,
        },
    )

    return {"post_id": post.id}

That one ID becomes a shared thread running through the whole workflow, tying together every service it touches.

Logs, metrics, and traces

These three tools overlap, and yet each answers its own kind of question.

A single incident often draws on all three in turn:

1. Metric shows payment.authorization_failed increased.
2. Trace shows failures cluster behind one dependency call.
3. Logs show the dependency returned 429 after a rollout changed request volume.

The log gives detail. The metric gives scope. The trace gives shape. Together they let you see the same event from three honest angles.

Levels are an operating contract

Log levels work best when they mean something stable.

ERROR should mean a real failure, one that warrants action. Reserve it for those, and the alert stream stays meaningful and worth trusting. Log real failures at the right level, and the system keeps its visibility exactly when it needs it most.

Each level carries its own quiet promise. WARN records a handled surprise. INFO records a lifecycle fact. DEBUG records detail for a narrow investigation. When every team keeps those promises, a reader can trust the level before ever reading the message.

Seen this way, levels are simply routing rules for attention.

Aggregation is part of the system

Local logs are temporary by nature. Containers restart, hosts disappear, disks fill. A service that only logs to local files is observable only while that local file remains within reach.

At scale, logs need a pipeline:

application -> stdout/file -> collector -> transport -> index/store -> query/alert

Each step has failure modes:

Logging is a production dependency like any other. So its backpressure, drop behavior, retention, and cost limits all deserve to be understood rather than left to chance.

Sampling

Some events arrive too frequently to keep in full detail.

Page views, health checks, cache hits, and successful background jobs can easily dominate log volume. Keeping every record may end up costing more than the value it returns.

So sampling is best done deliberately:

1. Always keep errors.
2. Always keep security and audit events.
3. Sample high-volume success paths.
4. Sample by correlation ID or stable key so whole workflows stay together.
5. Record the sample rate on the event.

Stable sampling keeps related logs in each other's company:

import hashlib


def should_sample(key, rate):
    digest = hashlib.sha256(key.encode("utf-8")).hexdigest()
    bucket = int(digest[:8], 16) / 0xFFFFFFFF
    return bucket < rate


def log_post_view(post_id, user_id, correlation_id):
    if not should_sample(correlation_id, rate=0.01):
        return

    logger.info(
        "post_viewed",
        extra={
            "event": "post.viewed",
            "post_id": post_id,
            "user_id": user_id,
            "correlation_id": correlation_id,
            "sample_rate": 0.01,
        },
    )

Sampling by workflow keeps the story whole, so that when a record is selected, it is still usable end to end.

Retention

Retention is part of the design, not an afterthought.

Different records call for different lifetimes:

Hot storage answers queries in milliseconds and costs the most per gigabyte. Cold storage answers in seconds or minutes and costs cents on the dollar. A retention policy that matches the way the data is actually used keeps both the cost and the access in balance.

If support tickets tend to arrive two weeks after the user action, the searchable window has to reach back two weeks to meet them. If debug logs carry large payloads, a short window keeps that volume from quietly accumulating into a year of storage no one will ever query.

Logs as audit trail

Some logs are operational. Others are audit records. The two look alike on the page, but each deserves its own standards, applied with care.

An audit trail usually needs these properties:

1. Stable schema.
2. Append-only storage.
3. Actor and target identifiers.
4. Before and after values when appropriate.
5. Authorization context.
6. Tamper evidence or restricted write access.
7. Clear retention.
8. PII minimization.

For example:

def change_role(actor_id, user_id, new_role):
    old_role = db.get_user_role(user_id)

    db.update_user_role(user_id, new_role)

    audit_log.write(
        {
            "event": "user.role_changed",
            "actor_id": actor_id,
            "user_id": user_id,
            "old_role": old_role,
            "new_role": new_role,
            "result": "success",
        }
    )

This is a domain record, and it deserves to be treated as one.

Event streams and logs

Logs and event streams are both append-only records. What sets them apart is the contract each one carries.

The same event shape can serve both observability and downstream work. When another system depends on a fact, give that fact a delivery contract so it arrives, holds its schema, and can be replayed. A debug log, by contrast, makes its best effort and stays purely observational, and that is fine for what it is.

If another system must react to the fact, write an event with a delivery contract:

def publish_post_created(tx, post):
    tx.insert_outbox_event(
        event_type="post.created",
        aggregate_id=post.id,
        payload={
            "post_id": post.id,
            "user_id": post.user_id,
            "created_at": post.created_at.isoformat(),
        },
    )

The logger can note that the event was written, while the outbox holds the delivery boundary itself.

PII and secrets

Logs spread quickly. They move from application memory into collectors, indexes, dashboards, tickets, notebooks, and cold storage. Anything placed into a log is, in practice, likely to be copied many times over.

For that reason, logging benefits from both a list of what to leave out and a list of what to keep.

Do not log:

1. Passwords.
2. Session tokens.
3. API keys.
4. Full payment details.
5. Raw authorization headers.
6. Large request bodies by default.
7. Private messages unless the feature explicitly requires secure audit handling.

Do log identifiers that allow lookup in the proper system:

logger.info(
    "payment_authorization_failed",
    extra={
        "event": "payment.authorization_failed",
        "payment_id": payment.id,
        "order_id": order.id,
        "gateway": "stripe",
        "gateway_error_code": error.code,
        "correlation_id": correlation_id,
    },
)

The log points to sensitive data through identifiers, while the values themselves stay safely in the system that owns them.

Schema drift

Structured logs fail quietly when field names begin to drift apart.

One service logs user_id. Another logs uid. A third logs userId. One version writes duration_ms as a number, another as a string. Queries become partial, dashboards lie by omission, and incident review slows down just when speed matters.

A small schema contract steadies this:

def log_event(logger, event, **fields):
    required = {
        "event": event,
        "service": SERVICE_NAME,
        "version": VERSION,
        "environment": ENVIRONMENT,
    }

    logger.info(event, extra={**required, **fields})

For larger systems, this grows into a shared logging package or a schema registry. The aim is the same throughout: to make the important fields boring and consistent, so they can be trusted without a second thought.

What I notice

The systems that are easiest to debug tend to log a steady, modest number of facts.

They log the same few facts consistently. They use stable event names. They pass correlation IDs through every path. They mark expected validation at one level and real failure at another. They keep the fields small enough to index. And they know which records are sampled and which are always kept.

A system can also carry many expressive log lines whose values live inside prose. During an incident, someone then sits and reconstructs the record by hand, line by line.

A good log record arrives already structured. It answers the query directly and quietly narrows the work that remains.

Design the log event

For a new user-facing write, it helps to define the log event before the incident rather than during it.

Example for post creation:

Then define the queries that should work:

event:post.created result:success service:posts-api
event:post.created user_id:1234
event:post.created correlation_id:abc123
event:post.created result:timeout version:2026.05.23.1

If those queries cannot be answered, the event is not yet shaped, and that is simply a signal to keep refining it.

Metrics that help

The most useful logging metrics are about the pipeline and the records themselves:

The logging system itself needs watching too. Without it, the logging can fail at the very moment the application needs it most.

What I think

The practical sequence is steady and unhurried:

1. Define common fields.
2. Use stable event names.
3. Pass a correlation ID through every boundary.
4. Put queryable values in named fields.
5. Keep secrets and large payloads out by default.
6. Treat log levels as an operating contract.
7. Aggregate logs outside the host or container.
8. Sample high-volume success paths by workflow.
9. Never sample records that carry correctness, security, billing, or audit meaning unless another durable source exists.
10. Set retention by record type.
11. Watch the logging pipeline for drops and schema rejects.

The underengineering move is to log a small set of facts and give each one a stable shape, and then to stop there.

That small set of stable records tends to carry most of the value when an incident arrives.

Tutorial checklist

For any important event, it helps to fill this out:

Then ask:

Can this record be found, grouped, and trusted without parsing a sentence?

If not, the log is not yet structured enough, and that is just the next thing to tend to.

Summary

1. Logs are data produced by the running system.
2. A useful log line is a record with named, queryable fields.
3. Event names should be stable, specific, and queryable.
4. Correlation IDs connect local service stories into one workflow story.
5. Logs, metrics, and traces answer different questions.
6. Log levels route attention and should have consistent meaning.
7. Aggregation, indexing, retention, and drops are part of the logging design.
8. Sampling should preserve workflows and never hide important correctness records.
9. Audit logs need stable schema, actor and target IDs, retention, and controlled writes.
10. Logs can feed event thinking; a fact other systems depend on belongs in an event stream with a delivery contract.
11. The best logs reduce guessing during incidents.

---

Pop quiz

Interactive quiz

Logs as operational data

A randomized review of structured logging, correlation IDs, sampling, retention, and audit boundaries.

4of 11 questions

Question 1 of 425%

What makes a structured log more useful than a plain sentence?

AIt always uses more disk spaceBIt stores important values in named fieldsCIt prevents all production failuresDIt removes the need for metrics

Next question