logs you can actually query

For years my Go logging was a graveyard of log.Printf calls. Useful enough when tailing a terminal, useless the moment the service was in production and I needed to find one request out of millions. The line "processing user 4823 failed: timeout" is human-readable and machine-hostile. You cannot filter on it, you cannot aggregate it, and you certainly cannot graph it.

The fix is structured logging: log key-value pairs, not sentences. Instead of baking the user ID into a string, you attach it as a field. The log line becomes data, and data you can query.

The difference at the call site is small:

// before: a sentence
log.Printf("processing user %d failed: %v", userID, err)

// after: structured fields
logger.Error("processing failed",
    "user_id", userID,
    "error", err,
)

That second form, emitted as JSON, gives you something a log aggregator can actually index. Now "show me every error for user 4823 in the last hour" is a filter on a field, not a desperate grep across a wall of free text.

The ecosystem in Go has settled into a few mature options for this. zap and zerolog are the two I reach for most, both built around the idea of zero-allocation structured logging because in Go the temptation is always to reach for something fast, and logging on a hot path can hurt if you are not careful. Both let you attach fields, both emit JSON by default, and both are quick enough that you stop thinking about the cost.

The pattern that changed how I write services, more than the library choice, is the contextual logger. You create a base logger, then derive child loggers that carry fields down through a request. So at the start of handling a request you do something like:

reqLogger := logger.With(
    "request_id", reqID,
    "user_id", userID,
)

and every log line that reqLogger emits for the rest of that request automatically carries the request ID and user ID. You set the context once, and you stop manually threading the same fields into every single log call. When something goes wrong three layers deep, the log line already knows which request it belongs to, because the logger carried it down. Correlating a failure across a dozen log lines becomes trivial: filter on the request ID and read them in order.

A few things I learned the slightly harder way:

• Pick your field names and stick to them. If one part of the code logs user_id and another logs userId and a third logs uid, your lovely queryable logs are queryable in three incompatible ways. Agree the keys early. A small shared package of field-name constants is not over-engineering, it is the thing that keeps the logs actually usable a year later.
• Do not log secrets into structured fields. This sounds obvious, but structured logging makes it easier to accidentally dump a whole struct, password field and all, with one careless call. The convenience cuts both ways.
• JSON in production, human-readable locally. Reading raw JSON log lines in a terminal during development is miserable. Most of these libraries let you switch the encoder, so I run a console-friendly format locally and JSON in production. Same code, different presentation, and I am not parsing JSON with my eyes at my desk.

Is it more verbose at the call site? A little. You write "user_id", userID instead of letting the value disappear into a format string. That is the trade, and it is a good one. The half-minute you spend naming a field is repaid the first time you are staring at a production incident at an unsociable hour and can answer "which users did this affect" with a query instead of a grep and a prayer.

The real test came a few weeks after I made the switch, when a service started misbehaving in production and I had nothing to go on but the logs. With the old printf logs that would have been an evening of grepping and guessing. With structured logs it was a filter on an error field, a glance at the request IDs that recurred, and the culprit narrowed down inside ten minutes. That evening I did not have to spend is the whole argument. Structured logging is not glamorous and it does not demo well, but it is the difference between logs that comfort you and logs that actually help.