01 Background
The client
A telecom and internet service provider operating in Central Africa. Alongside connectivity, they run and maintain infrastructure for their customers, including one system that serves a government client.
We had already replaced that server: an eleven-year-old, unpatched system, rebuilt and hardened with zero downtime. That solved the security problem. It did not solve the next one, which was that the new server was current, hardened, and still a single machine.
- Sector
- Telecom and ISP
- Region
- Central Africa
- End customer
- Government body
- Engagement
- 7 days
Client withheld under NDA · end customer not identified
02 Diagnosis
The problem
Everything the institution ran sat on one box. Not just the data, but the operating system, the service configuration, the user directory, the certificates and the hardening we had just put in place.
A backup restores files. It does not restore a working server. That matters more than usual here, because when a mail server is unreachable, other systems retry for hours and then return messages to their senders as undeliverable. A bounced message from a citizen or another department is often never sent again.
Building a second server was the easy half. The hard half was the address, because this infrastructure is bound to its IP far more tightly than most systems:
-
Sending reputation is tied to the IP
The PTR record proves an address may send for a domain, and SPF lists which addresses are allowed to. Send from one that is not listed and the message is rejected.
-
Certificates are tied to the hostname
They are issued for one specific name. A failover that changes the name produces an encryption warning on every client in the organisation.
-
MX records are cached everywhere
Servers across the internet hold them for as long as the record’s lifetime allows, so a change does not take effect at the moment you make it.
A shared “floating” IP requires both servers in the same network segment at the same provider, which defeats the point of a disaster recovery site. A private tunnel adds a component that takes the service down when it fails. A second MX record helps traffic from other servers and does nothing for staff opening webmail.
03 Execution
Our solution
We put the failover decision in DNS itself, so both servers keep their permanent IP addresses and the hostname decides which one to answer with.
-
Built the standby as a duplicate server, not a data backup
Same platform, same configuration, same hardening, same certificates. Everything after this step depends on the standby being able to serve, unattended, the moment it is asked to.
-
Kept the two continuously in step
The stored data, the user directory and the application databases each replicate continuously rather than on a schedule, because a nightly backup means a failure costs you everything since last night. Configuration and certificates are mirrored too, so the machines do not drift apart over months of small changes.
-
Chose the failover layer deliberately
A managed DNS provider puts a government system’s availability in a third party’s hands. A lightweight scripted check was fast enough and cheap, and was ruled out because it has no logging and no alerting. We recommended a self-hosted DNS balancer, not for speed, but because a DR system nobody can see the state of is one you find out about during the incident.
-
Health-checked the services that matter
The failover layer tests mail transfer, mailbox access and web rather than pinging the machine. A server that answers a ping while its services are dead is exactly the failure a naive check misses. Record lifetime is held at 60 seconds, which bounds the worst case any user experiences.
-
Kept every published record untouched
Both servers keep their own static IP and PTR permanently, both are listed in SPF from the start, and certificates, DKIM and DMARC are unchanged. During a failover, nothing on the internet is edited. One DNS answer changes.
-
Made failback automatic, and built in the drill
When the primary returns, replication brings it current before the failover layer puts it back in service. Monitoring alerts on every event, and a monthly failover test is part of how the system is run. Untested disaster recovery is a document, not a capability.
04 Duration
Timeline
05 Stack
Tech stack
- Active–passive architecture
- Self-hosted DNS load balancing
- Service-level health checks
- Continuous data replication
- Directory and database replication
- Mirrored configuration and certificates
- Failover alerting
06 Outcome
The outcome
| Criteria |
// After
BPDoxS
|
// Before Single server |
|---|---|---|
| Recovery from total loss | Automatic, under 60 seconds |
Rebuild from backup, measured in days |
| What recovery restores | The whole server: OS, config, certificates, data |
Files only, the server still needs rebuilding |
| Failure detection | Continuous service-level health checks |
Manual, usually on a user complaint |
| Published DNS records | Nothing edited, one DNS answer changes |
Any move means editing PTR, SPF and certificates |
| Failback | Automatic once replication is current |
Manual rebuild and reconfiguration |
| Tunnel dependency | None, static public IPs only |
The usual answer adds a VPN that can fail |
| Failover testing | Monthly drill in the operating procedure |
Nothing to test |
A second server, a hostname that decides for itself which one to answer with, and not one published record touched when it switches.