The monitoring dashboard shows CPU at 90%, memory at 85%, and error rate at 5%. Something is wrong, but what? The logs contain thousands of entries. The services all seem healthy. There are metrics, there are logs, but the question remains unanswered: why is the system slow?

This is the observability gap. Data without understanding is not observability. Something is wrong, but why it is wrong or where it is happening cannot be seen. Three questions need answers, and each question requires a different pillar.

The first question is: what is happening? Metrics answer this question. CPU usage is 90%. Request latency is 500ms. Error rate is 5%. Metrics are aggregated, time-series data. They tell the state of the system at a point in time. They are good for detecting trends and anomalies. They are bad for explaining why something happened. A metric can tell that latency increased, but it cannot tell which service caused the increase or which request was affected.

The second question is: why is it happening? Logs answer this question. A log entry says: “Database query took 450ms for user 12345.” Logs are discrete events with context. They tell what happened at a specific moment. They are good for debugging specific issues. They are bad for understanding system-wide behavior. Logs can tell why a specific request was slow, but they cannot tell if the slowness is affecting all requests or just a subset.

The third question is: where is it happening? Traces answer this question. A trace follows a request through the entire system: API gateway → service A → service B → database. Traces show the latency breakdown at each step. They tell which component is the bottleneck. They are good for understanding distributed systems where a request flows through multiple services. Traces can tell that service B is the bottleneck, but they cannot tell what service B is doing internally.

Each pillar answers a different question. All three are needed to understand system behavior. With only metrics, something is known to be wrong but not why. With only logs, individual issues can be debugged but the big picture cannot be seen. With only traces, requests can be followed but anomalies cannot be detected.

This is why the GLTM stack exists. Grafana is the visualization layer. Loki collects logs. Tempo collects traces. Mimir collects metrics. All four are integrated: a metric can be the starting point (latency is high), drilling down to the traces that show which requests are slow, then drilling down to the logs that explain why those requests are slow. The data is correlated. Jumping between different tools or manually correlating timestamps is not necessary. The observability is unified.

The practical implication is that monitoring is not a one-time setup. It is a continuous process of improving observability. Every incident reveals a gap in observability. A question could not be answered during the incident, so instrumentation is added to answer that question next time. The system becomes more observable over time, not because it was planned, but because failures were learned from.

This is also why alert fatigue is a design failure, not a user error. If too many alerts are being received, the problem is not lack of discipline in handling them. The problem is that the alerts are not actionable. An alert should tell: something is wrong, here is why, here is what to do. If an alert just says “CPU is high,” it is not actionable because high CPU might be normal (batch job running) or it might be a symptom of a deeper problem (memory leak causing garbage collection). The alert should tell what is actually wrong and what action to take.

The trade-off is that observability requires instrumentation. Metrics must be added to code, log levels must be configured, traces must be instrumented. This is work that does not directly add feature value, so it is often deferred. But the cost of deferring observability is higher incident response time, more debugging effort, and more outages. The trade-off is between upfront instrumentation cost and ongoing incident response cost.

Observability requires answering three questions: what is happening, why is it happening, and where is it happening? If all three cannot be answered, then observability does not exist. Only noise exists. And noise does not help understand the system; it just distracts from the real problem.