A production database must survive failures. The server crashes, the network partition occurs, the datacenter loses power. When Kubernetes is in the stack, the answer is an operator — the StatefulSet restarts the pod, the PersistentVolumeClaim re-attaches the volume, the service routes traffic to the new instance. The database is treated as cattle, and the operator translates infrastructure failures into pod lifecycle events.

But not every environment runs Kubernetes. On bare metal, on virtual machines in a private datacenter, on a three-node cluster behind a jump server, the tools are different and the abstractions are thinner. There is no operator to restart a crashed process, no service mesh to redirect traffic, no etcd to maintain consensus about who the current master is. Every component of the HA stack must be installed, configured, monitored, and understood independently. The insight is not that this is harder — it is that the failure modes become visible instead of being handled inside an operator, and understanding them is the difference between an HA system that works and one that produces false confidence.

The standard stack for database HA without Kubernetes has three layers. Replication moves data between nodes. A proxy routes traffic and manages connection pooling. A virtual IP provides a single endpoint that moves when the master changes. For MariaDB, the stack is GTID-based replication, MaxScale, and Keepalived. Each layer handles one concern, and none of them knows what the others are doing — which is precisely where the failure modes live.

MariaDB replication is asynchronous by default. The master commits a transaction, writes it to the binary log, and the slave reads the log and applies it in its own time. The master does not wait for the slave to acknowledge the transaction. This means the slave is always a little behind — milliseconds in a well-tuned network, seconds under load, minutes or hours if something goes wrong. GTID (Global Transaction ID) tracks exactly which transactions have been applied on each node. The format is domain-server_id-seq_no: a domain ID for multi-source replication, a server_id that identifies the node, and a sequence number. When a slave connects to a master, it tells the master which GTID it has already applied by reading its gtid_slave_pos. The master sends only the transactions the slave has not seen. This makes failover deterministic — the new master knows exactly which transactions it has and which it needs.

MaxScale sits between the application and the database. It is a TCP proxy that understands the MySQL protocol. It can route read-write traffic to the master and read-only traffic to replicas. It can monitor the health of each database node, detect when the master is unreachable, and fail over to a replica that has been promoted to master. It can also provide read-write splitting — transactions that write go to the master, SELECT statements go to replicas — without the application knowing anything about the topology. MaxScale does not manage virtual IPs. It changes its internal routing table when a failover event occurs, but the application still connects to the MaxScale address. If MaxScale itself fails, the application cannot reach the database at all. This is why MaxScale is deployed at least in pairs behind a load balancer that understands TCP health checks.

Keepalived provides the virtual IP. It runs VRRP (Virtual Router Redundancy Protocol) between two or more nodes, electing one master that owns the virtual IP. When the master fails, the backup node takes over the IP. The application connects to the virtual IP, which floats between MaxScale nodes. Keepalived does not know about database health — it knows only about network availability. If the database process crashes but the server is still running, Keepalived considers the node healthy and does not move the IP. This is the first and most common gap in the stack: Keepalived checks the network, not the database. The fix is a health check script that Keepalived calls periodically, which tests whether MaxScale is actually serving traffic by attempting a real connection.

Split-brain is the most dangerous failure mode in this stack. It occurs when both nodes believe they are the master. The classic scenario: the network between two datacenters partitions, the slave loses connectivity to the master, MaxScale on the slave side detects that the master is unreachable and promotes the slave to master. Both nodes accept writes. When the partition heals, there are two datasets that have diverged. Reconnecting them requires resolving the conflict — which transactions to keep, which to discard — and that resolution is manual. MaxScale has built-in protection: it uses a consensus mechanism to ensure only one master is promoted. But the protection depends on a quorum of monitor nodes, and in a two-node setup (the most common for cost reasons), there is no quorum to establish. The solution is a third node or a STONITH (Shoot The Other Node In The Head) mechanism — a fence device that can power off the other node, a script that shuts down the database on the old master before the new one is promoted, or a combination of keepalived.conf configuration that ensures the old master releases the virtual IP before the new one acquires it. None of these are built into the database. They must be engineered into the deployment.

Async replication lag is the second failure mode that does not have a clean solution. When the application writes to the master and a failover occurs before the transaction reaches the replica, the transaction is lost. The RPO (Recovery Point Objective) is not zero. Semi-synchronous replication reduces the window: the master waits for at least one slave to acknowledge the transaction before committing. But semi-sync in MariaDB has a timeout — if the slave does not acknowledge within the configured window (default 10 seconds), the master falls back to async. Under network load, this fallback happens silently. The application sees the transaction committed. The slave never received it. A failover moments later loses the data. Monitoring the semi-sync status, alerting on fallback events, and ensuring that the slave is always fast enough to keep up are operational responsibilities that cannot be automated away.

Backup is the layer that is easiest to neglect because it is invisible — until data is lost and no backup exists to restore from. MariaDB’s native backup tool is mariabackup, a fork of Percona XtraBackup. It performs physical backups: it copies the data files, the redo log, and the binary log position, producing a point-in-consistent snapshot. A full backup copies everything. An incremental backup copies only the pages that changed since the last backup. The incremental chain is based on LSN (Log Sequence Number) — mariabackup tracks the exact LSN at the end of each backup, and the next incremental starts from that LSN.

The backup strategy determines how much data can be recovered and how fast. A typical schedule: a full backup weekly on Sunday night, a second full backup mid-week on Thursday, and incremental backups on the remaining days. The weekly full provides a baseline. The mid-week full reduces the length of the incremental chain — applying six incrementals to a Sunday backup is slower and more error-prone than applying two incrementals to a Thursday backup. Incremental backups on Tuesday, Wednesday, Friday, and Saturday cover the gaps. Retention is three weeks for weekly backups, two copies for daily backups, and four days for incrementals. The total storage cost is roughly three full backups plus four incremental copies per week, which is manageable for most production databases under a few hundred gigabytes.

Point-in-Time Recovery uses the binary log. A full backup is restored, then the binary log from the last backup position is replayed up to the target timestamp. The workflow: identify the last full or incremental backup taken before the target time, prepare the backup with mariabackup –prepare, copy-back the data to the data directory, then use mariadb-binlog to replay transactions from the position recorded in xtrabackup_binlog_info to the target time. The command is mariadb-binlog --start-position=<position> --stop-datetime='2026-07-01 14:00:00' /var/log/mysql/mariadb-bin.* | mariadb. The precision depends on the binary log retention. If binary logs are rotated every day and kept for seven days, recovery can go back exactly one week. Beyond that, only the last full backup snapshot is available.

Restoring from backup in a replication topology adds a complication: the restored node must rejoin replication at the correct position. The backup records the master’s binary log position and GTID. After restore, the node connects to the current master with CHANGE MASTER TO MASTER_USE_GTID=current_pos. MariaDB’s GTID makes this deterministic — the restored node knows which transactions it has applied and requests only the missing ones from the master. But there is a subtle failure mode: if the master has purged binary logs containing transactions that the restored node has not yet applied, the slave cannot catch up. This is fixed by ensuring that binary log retention on the master exceeds the longest expected gap between backups — typically seven days, matching the backup schedule.

The common thread across all these failure modes is that they live in the gaps between components. Keepalived does not know about database health. MaxScale does not know about async lag. Mariabackup does not know about the replication topology. The backup schedule does not know about the log retention policy. Kubernetes operators solve this by consolidating all these concerns into a single controller that manages the database lifecycle holistically. Without Kubernetes, the responsibility for connecting these concerns falls on the operator of the system — not the software operator, but the human one.

The most important distinction between HA with and without Kubernetes is not technical; it is operational. In a Kubernetes environment, the replication controller detects a failure and initiates a recovery workflow automatically — create a new pod, attach the volume, update the service endpoints. The human operator is notified after the recovery is complete. The mean time to recovery is measured in seconds. In a non-Kubernetes environment, the recovery is initiated by a monitoring alert. The human operator SSHes into the node, checks the logs, promotes the replica, moves the virtual IP, updates the MaxScale configuration. The mean time to recovery is measured in minutes — if the operator is awake and the runbook is up to date. The system is not less reliable; it is less autonomous. And autonomy matters most at 3 AM on a Sunday when the only person who knows the topology is on vacation.

This does not mean Kubernetes is the answer for every database. A three-node MariaDB stack behind MaxScale and Keepalived is a proven, battle-tested architecture that runs some of the largest production databases in the world. The replication is stable. The proxy is fast. The virtual IP is simple. The failure modes, once understood, are manageable. But the architecture requires the engineering team to invest in three things: monitoring that detects the specific failure modes of each layer, runbooks that document the recovery procedure for each scenario, and regular drills that verify the runbooks work. Without these three investments, the HA stack provides false confidence — it works during normal operation but fails during the incident it was built to survive.