Sang's Blog

The on-call rotation is a social contract, not a schedule

An on-call rotation is the only place where engineering decisions are tested against real-world consequences at three in the morning. Code review tests assumptions. Automated tests verify behavior. Staging environments simulate load. But on-call is the exam — the moment when a design choice made six months ago wakes someone up, demands an answer, and refuses to wait until morning. Everything else is preparation.

This is why the way an organization structures its on-call rotation reveals more about its engineering values than any mission statement ever will. Not what the organization claims to value. What it actually values, measured by who carries the pager, what they are permitted to do when it goes off, and what happens the morning after.

When on-call is treated as a cost, the instinct is to minimize it. The metric becomes page frequency. Fewer pages means less cost. The path of least resistance is to make the pager quieter — raise alert thresholds, widen hysteresis windows, route everything through a single escalation point who learns to filter. The system becomes quieter, but not because it became more reliable. It became quieter because the feedback mechanism was disconnected. The pages that still come through are the ones too loud to suppress: the full outages, the customer-facing degradations, the incidents that are already public. By the time the pager fires, the problem has passed through every layer that could have caught it earlier and found them all deliberately muted.

This is not negligence. It is an incentive structure working exactly as designed. When the metric is “reduce pages,” the rational response is to reduce pages — by any mechanism available, including making the system less observable. The organization gets what it measures, and it measured the wrong thing.

When on-call is treated as a feedback loop, the same page triggers a different response. The page is not the problem. The page is evidence that a problem exists somewhere upstream. Every alert is an unanswered question: what stage should have caught this before it reached a human at three in the morning? Was there a test that should have detected this condition? A canary deployment that should have caught the regression? A runbook that should have documented the remediation? A monitoring check that was too sensitive, or not sensitive enough, or looking at the wrong metric entirely?

Each answer becomes an improvement — not to the on-call rotation, but to the pipeline that feeds into it. The pager does not get quieter because alerts are suppressed. It gets quieter because the system becomes more reliable, one postmortem action item at a time. The metric is not page frequency. The metric is whether every page, once investigated, produces a change that prevents that exact page from ever happening again.

This distinction — cost versus feedback — is not a matter of tooling or process. It is a matter of what the organization believes on-call is for. If on-call is for keeping the system alive until business hours, then suppressing alerts is rational. If on-call is for discovering where the system is fragile and making it less fragile, then every page is valuable data. The same event — a pager firing — means completely different things depending on which belief the organization holds.

The social contract dimension runs deeper than process design. Who carries the pager is the most honest answer to the question “who owns this system.” If only junior engineers are on-call, the organization is making a statement: reliability is not a leadership concern. The people who make architecture decisions do not experience their consequences. The people who experience the consequences do not have the authority to fix the root cause. The pager becomes a punishment for being early in a career, not a mechanism for improving the system.

If the people who write the code are not the people who get paged for it, the organization has severed the feedback loop at its most critical junction. The developer who merged a change at 4 PM does not know it caused a cascading failure at 2 AM. The on-call engineer who debugged it for two hours writes a postmortem that the developer may or may not read. The learning never closes the loop. The same class of failure happens again because the person who could have prevented it never learned that it happened.

The inverse is equally revealing. When senior engineers carry the pager, the organization is saying: the people who design the system are accountable for how it behaves. When the person who approved the architecture is the person who gets woken up when it fails, design decisions are made differently. Not because anyone is more careful, but because the feedback is immediate and personal. The architect feels the failure. The next design accounts for it.

This is also where the distinction between on-call and incident response blurs. A healthy on-call rotation is not just about answering pages. It is about what happens the morning after: the blameless postmortem, the action items that actually get prioritized, the budget for reliability work that does not compete with feature velocity. An organization that runs postmortems but never allocates time to fix what the postmortem found is performing reliability theater. The postmortem is a ritual, not a mechanism. The on-call engineer knows this, because the same page will come again next month, and the month after that, and the postmortem from six months ago will still have its action items in the backlog with no sprint assigned.

The connection to the broader DevOps feedback architecture is direct. The CI/CD pipeline exists to catch problems at the cheapest possible point — a syntax error in the editor costs seconds, a logic error in tests costs minutes, a configuration error in staging costs hours, a performance regression in production costs weeks. On-call is the last stage of that pipeline. It is the most expensive feedback loop in the entire system. Every page is a failure of every preceding stage — the compiler, the linter, the test suite, the staging environment, the canary deployment — to catch something that was catchable earlier. The most effective way to improve on-call is not to optimize the rotation schedule or write better runbooks. It is to move the detection left. Make the pipeline catch what the pager currently catches. The pager should only fire for failures that no automated system could have predicted.

Maintenance connects here too. A system that is not maintained — dependencies not updated, CVEs not patched, technical debt not addressed — will eventually fail in production. The pager will fire. The on-call engineer will spend hours tracing a problem that could have been prevented by a Renovate PR merged three months ago. The cost of deferred maintenance does not disappear. It transfers. It moves from the developer who could have merged the update to the on-call engineer who now has to debug the outage. The organization did not avoid the cost. It shifted the cost onto the person least equipped to bear it.

Monitoring closes the loop. An alert that fires and says “CPU is high” is not actionable. An alert that fires and says “the payment service is returning 503 errors because the database connection pool is exhausted, here is the trace showing the bottleneck, here is the runbook for restarting the connection pool” is actionable. The difference between these two alerts is not the severity of the problem. It is the amount of investigation the on-call engineer must do at three in the morning while half-asleep. The monitoring system that produces the first alert is treating on-call as a cost — it offloads the diagnostic work onto the human. The monitoring system that produces the second alert is treating on-call as a feedback loop — it does the diagnostic work automatically so the human can focus on remediation.

The throughput of an on-call rotation is not measured in pages handled. It is measured in lessons learned and applied. A team that handles twenty pages a night but never fixes the root cause is not performing well; it is drowning. A team that handles one page a month, investigates it thoroughly, and ensures the same page never fires again is performing exceptionally. The metric is not how well the team handles incidents. The metric is how quickly the team eliminates the need for itself.

The on-call rotation is the most honest performance review an engineering organization will ever receive. It cannot be faked. It cannot be optimized with dashboards that show the right colors. At three in the morning, when a database is down and a customer is losing money, the rotas and the runbooks and the escalation policies either work or they do not. And whether they work was determined not by the on-call schedule, but by every decision made in the months leading up to that moment — the architecture choices, the testing strategy, the monitoring investment, the maintenance discipline, the postmortem follow-through. The pager is just the messenger. What the messenger says about the organization is more important than what the organization says about itself.

← Prev Post