Site-to-site VPN: connecting infrastructure across boundaries
https://sangtd.net/site-to-site-vpn-connecting-infrastructure-across-boundaries/Two networks need to communicate. One sits behind a NAT gateway in a cloud VPC, the other behind a firewall in a colocation facility. There is no direct fiber between them. The internet is the only path, and the traffic must be encrypted because it crosses untrusted infrastructure. A site-to-site VPN is the standard answer: an encrypted tunnel between two gateways that makes the two networks behave as one.
The scenario repeats across every infrastructure team. A Kubernetes cluster in Singapore needs to reach a MariaDB replica in the Hanoi datacenter. A CI/CD runner in a cloud VM needs to push container images to a private registry behind a branch office firewall. A monitoring stack in a central location needs to scrape metrics from servers spread across three cloud providers. The common thread is that the participants are networks, not people. A router does not authenticate with a username and password. A firewall does not have an Okta session. The connection is between administrative domains, and the trust is established once, at configuration time, through cryptographic key exchange.
Site-to-site VPN is older than the cloud and will outlive the current ZTNA trend — not because it is better, but because it solves a fundamentally different problem. ZTNA brokers access between a human identity and an application. Site-to-site VPN connects two networks that do not share an identity provider and never will. The machine-to-machine nature of the connection demands deterministic latency, predictable throughput, and automatic failover. These requirements map naturally to a network-level tunnel and a dynamic routing protocol, not to per-request authorization against an identity provider.
IPsec has been the standard protocol for site-to-site VPN for over two decades. It operates in tunnel mode: the original IP packet is wrapped inside a new IP packet with an ESP (Encapsulating Security Payload) header, and both packets are sent between the two gateways. The outer header addresses the gateways; the inner header carries the original source and destination. A packet from 10.0.1.5 in Hanoi arrives at the gateway at 203.0.113.1, gets encrypted and encapsulated with a destination of 198.51.100.1 (the remote gateway), arrives at the remote side, is decapsulated and decrypted, and continues to 10.0.2.10 in Singapore. The applications on both sides see a direct connection; the tunnel is invisible above the IP layer.
The complexity of IPsec lives in the control plane, not the data plane. The data plane is straightforward: once the Security Associations (SAs) are established, packets are encrypted with AES-256-GCM and authenticated with an integrity check value. The control plane — IKE (Internet Key Exchange) — negotiates the SAs, manages keys, and handles the state machine. IKEv1 had two phases with eight exchange modes; IKEv2 reduced this to a single exchange with four messages: initiator request, responder response, initiator authentication, responder authentication. Most deployments today use IKEv2, but legacy equipment often requires IKEv1 with aggressive mode — the mode that leaks the preshared key in plaintext over the wire, which is why it was deprecated.
The real difficulty with IPsec is not the protocol itself but the interop matrix. An AWS VPN gateway speaking IKEv2 with AES-256-GCM and DH group 14 must negotiate with a pfSense firewall that prefers IKEv1 with AES-128-SHA1 and DH group 2. A MikroTik router proposes ESP with 3DES and null authentication because it defaults to maximum compatibility. A cloud VPN gateway rejects proposals weaker than AES-128. The negotiation fails, the tunnel does not come up, and the logs on both sides say “no acceptable proposal” — which is technically correct but does not tell the operator which parameter mismatched. The fix is always the same: pick a single proposal and configure it explicitly on both sides. Do not rely on automatic negotiation.
Routing is where site-to-site VPN designs differentiate themselves. A single tunnel between two networks is simple: both sides configure a static route pointing to the tunnel interface and a firewall rule allowing traffic from the remote subnet. Add a third site, and static routes multiply. Add a fourth site, and every new site requires route updates on every existing site. Dynamic routing solves this by having the VPN gateways exchange route information through the tunnel. BGP is the standard protocol for this: each gateway advertises the subnets it can reach, and route withdrawal on tunnel failure provides automatic failover.
BGP over IPsec is not complicated to configure, but the failure modes are subtle. If the tunnel goes down but the BGP session does not detect the failure immediately, the gateway continues to advertise routes that are no longer reachable and traffic blackholes. This is why BGP hold timers must be configured to match the expected recovery time of the tunnel — too short causes flaps, too long causes blackholes during failure. A common starting point is a hold time of 30 seconds with a keepalive interval of 10 seconds, which detects tunnel failure within 30 seconds and withdraws the routes before applications time out.
The topology decision constrains everything else. A hub-and-spoke topology has every spoke site connect to a central hub, and the hub routes traffic between spokes. This is the simplest to manage: N sites require N tunnels, and all routing converges at the hub. The trade-off is that all inter-spoke traffic traverses the hub, adding latency and making the hub a bottleneck and a single point of failure. A full mesh connects every site to every other site, eliminating the hub bottleneck at the cost of N(N-1)/2 tunnels — unmanageable beyond a handful of sites. A partial mesh is the pragmatic middle: critical paths are meshed, non-critical spokes connect through a hub, and BGP route reflectors distribute routes without requiring a full mesh of BGP sessions.
WireGuard has become the preferred alternative to IPsec for teams that control both ends of the tunnel. A WireGuard interface is configured with a private key, a list of peers with their public keys and endpoints, and a list of allowed IPs per peer. The allowed IPs define both what can be sent through the tunnel and what can be received — they act as a cryptographically enforced routing table and firewall in one. There is no IKE daemon, no certificate authority, no proposal negotiation. The interface exists when it is configured and traffic flows when both peers have each other’s keys.
WireGuard’s simplicity is not without cost. It uses ChaCha20-Poly1305 exclusively — there is no cipher negotiation, which means both sides must support it (most Linux kernels do since 5.6). It does not support x.509 certificates, so key distribution must be solved separately — a configuration management tool, a shell script, or a control plane like Headscale or Netmaker. It does not generate logs by default, which makes debugging silent failures difficult — the peer appears to be up but no traffic flows, and the only clue is a packet count on the interface that does not increment. These are not reasons to avoid WireGuard, but they are reasons to pair it with monitoring that actively probes tunnel health rather than relying on the kernel to report problems.
Cloud providers each have their own VPN gateway implementation with quirks that are discovered only during setup. AWS VPN Gateway creates two tunnels per connection for high availability — one tunnel uses the first public IP of the virtual private gateway, the other uses the second IP. The tunnels must come up independently, and the on-premises firewall must accept connections from both IPs. GCP Cloud VPN with HA configuration requires four tunnels between two cloud routers and two peer gateways — the tunnels are grouped into two pairs, and BGP sessions run over each pair to provide active-active connectivity. Azure VPN Gateway with active-active mode creates two gateway instances, each with its own public IP, and the on-premises peer must establish separate IPsec tunnels to each.
The diversity of cloud VPN implementations means that a single on-premises firewall must maintain multiple tunnel configurations — one set for AWS, one for GCP, one for Azure — each with different proposal requirements, different BGP ASN assignments, and different health check expectations. A misconfiguration in one tunnel should not affect the others, but in practice a buggy IKE daemon on the firewall can crash and take down all tunnels simultaneously. This is the strongest argument for deploying multiple firewall instances or VPN concentrators: one device fails, but only the tunnels it serves fail, not every tunnel in the organization.
Monitoring a site-to-site VPN is different from monitoring a remote-access VPN. Remote-access monitoring focuses on concurrent user count and bandwidth per user. Site-to-site monitoring focuses on tunnel state, route propagation, and latency. A tunnel can be up (the IKE SA is established) while the BGP session is down (the hold timer expired), and the operator sees the tunnel interface as green but traffic is blackholing. This is the most common silent failure in IPsec deployments, and the only way to catch it is to monitor both layers independently: verify the IKE SA, verify the BGP session, and verify that traffic is actually passing through by sending synthetic probes across the tunnel.
Practical experience shows that the most reliable site-to-site VPN deployments share three characteristics. First, they use explicit crypto proposals rather than automatic negotiation — this eliminates interop failures at the cost of documenting the proposal in the runbook. Second, they run BGP over the tunnel with short hold timers — this ensures that routing converges within seconds of a tunnel failure rather than within the default BGP timer of 180 seconds. Third, they monitor tunnel health with active probes — a ping or a TCP connection test across the tunnel every 30 seconds, with alerting on three consecutive failures. These three practices eliminate the failure modes that most commonly cause silent outages in IPsec deployments.
Site-to-site VPN is not a legacy technology destined for replacement. It is the infrastructure layer that carries traffic between administrative domains that do not and cannot share an identity provider. The encryption protocol — IPsec or WireGuard — matters less than the routing design. The gateway hardware matters less than the monitoring coverage. And the topology matters more than anything else: a well-designed hub-and-spoke with BGP route reflectors and active probes will outlast a full mesh with perfect encryption that no one can troubleshoot when it breaks.