Logs are narrative, not data. A search for an error message returns ten thousand results. Filtering by service yields five thousand. Filtering by time range yields five hundred. An hour is spent scrolling through irrelevant entries. By the time the relevant log is found, the original question is forgotten.

This is the problem with treating logs as data. When everything is logged, facts are collected without context. “Request received.” “User logged in.” “Query executed.” Each entry is a fact, but facts without narrative are noise. Everything is collected, but nothing can be found. A log entry should tell what happened, why it matters, and what the context is. “Request received from user 12345, processing payment of $500, order ID 67890” tells a story. It tells who, what, and why. It gives context. It helps understand the sequence of events.

This is why log structure matters. Unstructured logs are free text: “User logged in successfully.” Structured logs are key-value pairs: {"event": "user_login", "user_id": 12345, "status": "success", "ip": "192.168.1.1"}. Structured logs are searchable, filterable, aggregatable. All login events for a specific user can be queried. Login events can be aggregated by IP address. Login events can be correlated with other events. Unstructured logs require regex parsing, which is fragile and slow. Structured logs are indexed automatically by Elasticsearch.

The ELK stack is designed for structured logs. Elasticsearch stores and indexes them. Logstash parses and transforms them. Kibana visualizes and queries them. But the stack does not make decisions about what to log. That is the responsibility of the application. If the application logs everything as unstructured facts, the stack stores everything, and what is needed still cannot be found.

This is why log levels matter. DEBUG logs are for development. INFO logs are for routine operations. WARN logs are for unexpected but recoverable situations. ERROR logs are for failures that need attention. FATAL logs are for unrecoverable failures. Most production systems should log at INFO level, with the option to increase to DEBUG for specific services when debugging. If logging is at DEBUG level in production, the system is drowning in noise.

The practical implication is that logging is a design decision, not an afterthought. Logging is not added at the end of development; it is designed at the beginning. Decisions are made about what events are meaningful, what context is necessary, what structure makes logs searchable. Logs are treated as part of the API, not as an implementation detail.

This is also why log retention matters. All logs cannot be kept forever. Storage is expensive, and most logs are never queried. A retention policy is needed: keep ERROR logs for 90 days, keep INFO logs for 30 days, keep DEBUG logs for 7 days. Or keep logs based on compliance requirements. The retention policy is a trade-off between storage cost and debugging capability.

The trade-off is that good logging requires upfront design. Decisions must be made about what to log, how to structure it, what level to use, how long to retain it. This is work that does not directly add feature value, so it is often deferred. But the cost of deferring logging design is higher debugging time, more incidents, and more frustration. The trade-off is between upfront design cost and ongoing debugging cost.

Logs are narrative, not data. A search engine for narrative only works if a story worth finding has been written. If only facts without context have been collected, only noise exists. And noise does not help understand the system; it just distracts from the real problem.