Kubernetes, the reconciliation problem
https://sangtd.net/kubernetes-the-reconciliation-problem/Kubernetes uses a reconciliation loop to maintain desired state. The desired state is described in a YAML file, and the control plane continuously works to make reality match that description. No scripts, no deployment pipelines, no manual steps. Just a document and a loop that closes the gap between current state and desired state.
The abstraction is clean. The system is told what is wanted, not how to get it, and the system figures out the rest. It feels like magic until something breaks.
A pod enters CrashLoopBackOff. Logs are checked, the issue is fixed, the deployment is updated, applied again. The pod restarts, runs for a minute, then crashes again. Events are checked, resource limits, liveness probe configuration. Nothing obvious. The deployment is described, compared to what was applied, and the spec does not match what was written. The system mutated the configuration.
This is the reconciliation problem. Kubernetes does not execute steps; it closes gaps. The control plane watches the current state, compares it to the desired state, and takes action to close the gap. When a pod crashes, the ReplicaSet controller notices the gap (2 replicas instead of 3) and creates a new pod. When a node dies, the controller reschedules the pods elsewhere. The system is always working, always reconciling, always closing gaps.
The problem is that the reconciliation process is invisible. In an imperative system, steps are written and watched as they execute. When something fails, the failed step is visible. An execution trace exists. Logging can be added, retries, rollback logic. The complexity is in the script, but the script is visible, testable, debuggable.
In Kubernetes, there are no steps. There is only a desired state and a reconciliation loop. When reconciliation fails, the reason is not visible. The gap between current and desired state is visible, but the mechanism that is trying to close the gap is not. Maybe the desired state is impossible (10 replicas requested but the cluster only has resources for 5). Maybe the reconciliation is blocked by a webhook that rejects the pod creation. Maybe the controller is working correctly but the status reporting is delayed, so the information is stale.
The debugging question changes. In an imperative system, the question is: “Why did step 3 fail?” In Kubernetes, the question is: “Why isn’t the current state converging to the desired state?” The second question is harder because the answer could be anywhere in the reconciliation loop. The execution cannot be traced because there is no execution to trace.
This is not a bug in Kubernetes. It is a fundamental property of declarative systems. Choosing declarative means trading write-time simplicity for debug-time complexity. Writing the YAML is easy. Understanding why the system is not converging is hard.
Kubernetes becomes valuable when the orchestration problem is so complex that the complexity of state management is acceptable in exchange. When hundreds of containers need to be managed across dozens of nodes, when automatic rescheduling is needed when nodes fail, when rolling updates with configurable disruption budgets are required — then the reconciliation loop is worth the debug-time cost.
The trade-off should be explicit. Kubernetes is not chosen because it is simpler. It is chosen because it solves a different problem than orchestration. It solves state management at scale. And state management at scale is hard, whether Kubernetes is used or a custom reconciliation loop is built.
Choosing Kubernetes is choosing to trade one class of problems for another. Orchestration complexity is traded for state management complexity. The question is not whether Kubernetes is better than scripts; the question is whether the orchestration problem is complex enough to justify the state management problem that comes with it.