Sang's Blog

Infrastructure as code, drift is inevitable

Drift is the gap between what the code says should exist and what actually exists in the infrastructure. It is the bane of Infrastructure as Code, and it is inevitable.

A security group rule is added at 2 AM to fix a production issue. It is a temporary fix, the promise goes. The code will be updated tomorrow. Tomorrow comes, and the fix is forgotten. A week later, terraform plan shows a diff. The infrastructure does not match the code.

The reason drift exists is not that people are careless. It is that infrastructure is mutable in ways that code is not. When a change is committed to the application, the old version is replaced. There is no divergence between repository and production. But infrastructure has state that persists across deployments. A database continues running with its data. A load balancer continues routing traffic. When infrastructure is modified, it is not replaced; it is mutated in place.

This creates a fundamental asymmetry. Code is immutable once committed. Infrastructure is mutable once deployed. When a security group is manually changed, state is being modified that the code does not know about. The code says one thing; reality says another. That gap is drift.

Terraform attempts to bridge this gap with a state file — a cache that maps the code to actual resources. But the state file can become stale. If a resource is deleted outside of Terraform, the state file still references it. If a resource is created manually, the state file does not know about it. The state file is a snapshot of reality at a point in time, and reality changes.

When terraform plan runs, the drift is visible. Three options exist: update the code to match the manual change, revert the manual change, or import the manual change into the state file. Each option has costs. Updating the code means the code reflects history, not intent. Reverting the change means undoing a fix someone made for a reason. Importing the change means reconciling state, which is error-prone.

The practical implication is that Infrastructure as Code does not eliminate drift; it makes drift visible. Before IaC, drift existed but was invisible. Servers diverged, configurations drifted, no one knew what was actually running. With IaC, terraform plan reveals the drift. But the decision of what to do about it remains.

This is why IaC requires ongoing discipline. terraform plan must run regularly to detect drift. Decisions must be made whether to update code or revert changes. The state file must be kept in sync with reality. The work does not end when the code is written; the work is managing the gap between code and state.

The trade-off is between visible drift and invisible drift. IaC provides visible drift, which is better than invisible drift, but it does not eliminate drift. The choice is to manage the gap explicitly rather than letting it grow implicitly.

For simple infrastructure — a few servers, a database, a load balancer — the overhead of managing drift might not be worth it. Manual management with shell scripts and runbooks might be easier. The drift will be invisible, but so will the complexity of managing it. For complex infrastructure — dozens of services, multiple regions, dynamic scaling — IaC is worth the cost because the alternative is worse.

Drift is inevitable. Infrastructure will always diverge from code because infrastructure has state and code does not. The question is not whether drift exists; the question is whether the drift should be visible.

← Prev Post Next Post →