https://sangtd.net/tags/devops/Recent content in Devops on Sang's BlogHugo -- gohugo.ioenSun, 19 Jul 2026 00:00:00 +0700https://sangtd.net/the-on-call-rotation-is-a-social-contract-not-a-schedule/Sun, 19 Jul 2026 00:00:00 +0700https://sangtd.net/the-on-call-rotation-is-a-social-contract-not-a-schedule/An on-call rotation is the only place where engineering decisions are tested against real-world consequences at three in the morning. Code review tests assumptions. Automated tests verify behavior. Staging environments simulate load. But on-call is the exam — the moment when a design choice made six months ago wakes someone up, demands an answer, and refuses to wait until morning. Everything else is preparation. This is why the way an organization structures its on-call rotation reveals more about its engineering values than any mission statement ever will.https://sangtd.net/database-ha-without-kubernetes/Sun, 17 May 2026 00:00:00 +0700https://sangtd.net/database-ha-without-kubernetes/A production database must survive failures. The server crashes, the network partition occurs, the datacenter loses power. When Kubernetes is in the stack, the answer is an operator — the StatefulSet restarts the pod, the PersistentVolumeClaim re-attaches the volume, the service routes traffic to the new instance. The database is treated as cattle, and the operator translates infrastructure failures into pod lifecycle events. But not every environment runs Kubernetes. On bare metal, on virtual machines in a private datacenter, on a three-node cluster behind a jump server, the tools are different and the abstractions are thinner.https://sangtd.net/site-to-site-vpn-connecting-infrastructure-across-boundaries/Sun, 10 May 2026 00:00:00 +0700https://sangtd.net/site-to-site-vpn-connecting-infrastructure-across-boundaries/Two networks need to communicate. One sits behind a NAT gateway in a cloud VPC, the other behind a firewall in a colocation facility. There is no direct fiber between them. The internet is the only path, and the traffic must be encrypted because it crosses untrusted infrastructure. A site-to-site VPN is the standard answer: an encrypted tunnel between two gateways that makes the two networks behave as one. The scenario repeats across every infrastructure team.https://sangtd.net/from-vpn-to-zero-trust-the-perimeter-is-no-longer-real/Sat, 09 May 2026 00:00:00 +0700https://sangtd.net/from-vpn-to-zero-trust-the-perimeter-is-no-longer-real/An engineer connects to the corporate VPN from a coffee shop. The VPN client authenticates, the tunnel comes up, and the laptop receives an IP address from the internal subnet. From that moment, the laptop can reach any server on the corporate network — the ERP system, the CI/CD pipeline, the production database, the internal wiki, the monitoring dashboard, the Git repository, the entire internal attack surface. The VPN tunnel is encrypted, so the link between the coffee shop and the corporate gateway is secure.https://sangtd.net/cloud-providers-the-lock-in-curve/Sun, 03 May 2026 00:00:00 +0700https://sangtd.net/cloud-providers-the-lock-in-curve/Managed services reduce operational burden. A managed database eliminates backup and replication management. Object storage eliminates disk capacity management. Serverless functions eliminate scaling management. The cloud provider handles the infrastructure, and the team focuses on business logic. The result is agility, scalability, and efficiency. Then the first large bill arrives. The breakdown shows charges for data transfer between services, API requests, storage IOPS, provisioned capacity that is not being used. Estimating next month’s bill is impossible because the pricing model is too complex.https://sangtd.net/dns-records-for-email-reputation-takes-time/Sun, 12 Apr 2026 00:00:00 +0700https://sangtd.net/dns-records-for-email-reputation-takes-time/A mail server is configured. Postfix is set up, TLS is enabled, user accounts are created. A test email is sent to a Gmail account. It arrives in spam. The logs show everything is fine. The email was sent, the connection was successful. SPF, DKIM, and DMARC records are added to DNS. A day passes. Another test email is sent. Still spam. The headers show “SPF: pass,” “DKIM: pass,” “DMARC: pass.https://sangtd.net/replication-and-sharding-consistency-at-scale/Sun, 05 Apr 2026 00:00:00 +0700https://sangtd.net/replication-and-sharding-consistency-at-scale/A database cannot handle the load. Queries are slow, writes are blocking, connections are timing out. Replication offers a solution: create read replicas, route read traffic to them, and suddenly there is 10x the read capacity. The implementation works—for a while. Then a user updates their profile, but when they refresh the page, the old data is still there. They refresh again, and now it is updated. The read replica is lagging behind the primary.https://sangtd.net/building-infrastructure-from-bare-metal/Sun, 15 Mar 2026 00:00:00 +0700https://sangtd.net/building-infrastructure-from-bare-metal/A rack of servers arrives. Four nodes, each with two CPUs, 128GB of RAM, four network ports, and four hard drives. The task is to turn them into a private cloud: virtual machines for databases, Kubernetes nodes, storage, monitoring, and development environments. There is no cloud provider to abstract the hardware, no API to provision resources, no managed service to handle the network. Everything must be built from scratch, one layer at a time, and each layer constrains the layers above it.https://sangtd.net/elk-logs-tell-stories-not-facts/Sun, 08 Mar 2026 00:00:00 +0700https://sangtd.net/elk-logs-tell-stories-not-facts/Logs are narrative, not data. A search for an error message returns ten thousand results. Filtering by service yields five thousand. Filtering by time range yields five hundred. An hour is spent scrolling through irrelevant entries. By the time the relevant log is found, the original question is forgotten. This is the problem with treating logs as data. When everything is logged, facts are collected without context. “Request received.” “User logged in.https://sangtd.net/message-queues-delivery-processing/Sun, 01 Mar 2026 00:00:00 +0700https://sangtd.net/message-queues-delivery-processing/Message queues guarantee delivery, but delivery is not the same as processing. The queue ensures the message arrives; the application must ensure it is processed correctly. Service A sends a message. The queue holds it. Service B consumes it, processes it, and acknowledges it. The system works—until Service B crashes after processing but before acknowledging. The queue redelivers the message. Service B processes it again. Now there is duplicate data in the database.https://sangtd.net/monitoring-three-questions-three-pillars/Sun, 22 Feb 2026 00:00:00 +0700https://sangtd.net/monitoring-three-questions-three-pillars/The monitoring dashboard shows CPU at 90%, memory at 85%, and error rate at 5%. Something is wrong, but what? The logs contain thousands of entries. The services all seem healthy. There are metrics, there are logs, but the question remains unanswered: why is the system slow? This is the observability gap. Data without understanding is not observability. Something is wrong, but why it is wrong or where it is happening cannot be seen.https://sangtd.net/maintain-risk-compounds-silently/Sat, 14 Feb 2026 00:00:00 +0700https://sangtd.net/maintain-risk-compounds-silently/Risk compounds silently. Every day maintenance is deferred, the risk increases. A security vulnerability discovered today might have been patched last month if dependencies had been updated. A bug that causes a production outage might have been fixed in a newer version of the library. The cost of inaction is not static; it grows over time. A security audit reveals the application is running a version of lodash with a known prototype pollution vulnerability.https://sangtd.net/ci/cd-the-cost-of-change-curve/Sun, 08 Feb 2026 00:00:00 +0700https://sangtd.net/ci/cd-the-cost-of-change-curve/The cost of fixing a problem increases exponentially as it moves through the pipeline. A syntax error caught by the compiler costs minutes to fix. A logic error caught by tests costs hours. A configuration error caught by staging costs days. A performance issue caught in production costs weeks. CI/CD pipelines exist to catch problems at the cheapest possible point. Every stage is a feedback loop. Every feedback loop reduces the cost of change.https://sangtd.net/gitops-pull-vs-push/Sun, 01 Feb 2026 00:00:00 +0700https://sangtd.net/gitops-pull-vs-push/Traditional CI/CD pipelines push changes to infrastructure. The pipeline watches Git, builds artifacts, and executes commands against the cluster. It holds credentials, it knows the cluster endpoint, it has write access. The pipeline is the orchestrator, and the deployment is a one-way street: Git to cluster. GitOps inverts this relationship. Instead of the pipeline pushing changes to the cluster, a controller inside the cluster pulls changes from Git. The controller watches the Git repository.https://sangtd.net/infrastructure-as-code-drift-is-inevitable/Sun, 25 Jan 2026 00:00:00 +0700https://sangtd.net/infrastructure-as-code-drift-is-inevitable/Drift is the gap between what the code says should exist and what actually exists in the infrastructure. It is the bane of Infrastructure as Code, and it is inevitable. A security group rule is added at 2 AM to fix a production issue. It is a temporary fix, the promise goes. The code will be updated tomorrow. Tomorrow comes, and the fix is forgotten. A week later, terraform plan shows a diff.https://sangtd.net/kubernetes-the-reconciliation-problem/Sat, 24 Jan 2026 00:00:00 +0700https://sangtd.net/kubernetes-the-reconciliation-problem/Kubernetes uses a reconciliation loop to maintain desired state. The desired state is described in a YAML file, and the control plane continuously works to make reality match that description. No scripts, no deployment pipelines, no manual steps. Just a document and a loop that closes the gap between current state and desired state. The abstraction is clean. The system is told what is wanted, not how to get it, and the system figures out the rest.