Sang's Blog

CI/CD, the cost of change curve

The cost of fixing a problem increases exponentially as it moves through the pipeline. A syntax error caught by the compiler costs minutes to fix. A logic error caught by tests costs hours. A configuration error caught by staging costs days. A performance issue caught in production costs weeks.

CI/CD pipelines exist to catch problems at the cheapest possible point. Every stage is a feedback loop. Every feedback loop reduces the cost of change.

The pipeline that started as “automate the deploy” becomes a complex dependency graph of checks and gates. Each failure reveals a gap. Each gap becomes a new stage. The cost of fixing a problem increases exponentially as it moves through the pipeline. Each stage in the pipeline exists to catch problems earlier, when they are cheaper to fix.

The misconception is that CI/CD exists to make deploys faster. It does make deploys faster, but that is a side effect. The purpose is to reduce the cost of change by detecting problems at the cheapest possible point. Every stage is a feedback loop that answers a specific question: Does the code compile? Do the tests pass? Does the linting check pass? Are there security vulnerabilities? Does the deployment succeed?

Each stage is a gate. If the gate fails, the process stops. The next stage does not proceed. This is not because the goal is to block deploys; it is because the goal is to detect problems before they become expensive. The pipeline is structured as a dependency graph because each stage answers a question that is more expensive to answer than the previous one.

The principle is called shift-left: detect problems as early as possible in the process. A syntax error should be caught by the compiler, not by the test suite. A logic error should be caught by the test suite, not by the staging environment. A configuration error should be caught by the staging environment, not by production. Each layer of feedback catches a different class of problem, and the earlier it is caught, the cheaper it is to fix.

This is why CI/CD pipelines grow over time. Every production failure reveals a gap in the feedback mechanism. A new stage is added to catch that class of problem earlier. The pipeline becomes more complex, but the complexity is not arbitrary; it is a reflection of the failure modes that have been encountered.

The practical implication is that CI/CD is not just for teams that deploy frequently. It is for any team that wants to reduce the cost of change. If deploys happen once a month, CI/CD helps catch problems before the deploy, not during. If deploys happen ten times a day, CI/CD helps catch problems in minutes, not hours. The frequency of deployment does not determine whether CI/CD is needed; the cost of failure determines whether CI/CD is needed.

This is also why CI/CD is not a one-size-fits-all solution. The pipeline should reflect the specific failure modes of the system. A web application needs different checks than a data pipeline. A microservice architecture needs different checks than a monolith. The pipeline is not a product that is bought; it is a feedback mechanism that is designed based on what can go wrong.

The trade-off is that feedback mechanisms have costs. Every stage in the pipeline takes time. Every check requires maintenance. Every gate adds complexity. The goal is not to add as many stages as possible; the goal is to add the right stages that catch the most expensive problems at the cheapest point.

A pipeline that only automates deploys is missing the point. The pipeline exists to catch problems early, when they are cheap to fix. Every stage is a feedback loop. Every feedback loop reduces the cost of change. The pipeline is not about speed; it is about confidence.

← Prev Post Next Post →