01 The Problem

A checkout system runs entirely in one region. Nightly backups of the order database run at 2am, get retained for thirty days, and the team has genuinely tested that the backup files themselves are valid, checked every morning for a year. One afternoon, that region has a real, multi-hour outage, the kind where the entire region is unreachable, not one instance, not one service. Checkout is completely down. There's no infrastructure anywhere else to fail over to.

On the incident call, two questions come up that the backup strategy was never actually designed to answer. How much order data will be lost once service is restored? The last backup ran at 2am, the outage hit at 11pm, so the honest answer is up to 21 hours of real customer orders, gone, depending on exactly when the failure happened relative to that backup. How long until checkout is back? Nobody has ever actually timed a full rebuild from a cold start in a different region, provisioning fresh infrastructure, restoring a multi-terabyte backup, replaying whatever logs exist since. It turns out to be nine hours, not the one hour everyone had always assumed.

"We have backups" was never a complete answer to either question. It answered whether the data could eventually come back. It said nothing about how much would be missing, or how long the wait would be.

02 What It Is

RPO, recovery point objective, is the maximum acceptable amount of data loss, expressed as a window of time: if disaster strikes right now, how far back does the most recent copy you could actually recover from go. A nightly backup gives you an RPO of up to one backup interval, roughly 24 hours in the worst case, because that's the largest possible gap between "the last successful backup" and "the moment everything broke."

RTO, recovery time objective, is the maximum acceptable time to actually restore service after a disaster. Backups alone give you an RTO of however long a full cold restore takes: provisioning infrastructure from scratch, pulling the backup, restoring it, replaying anything since, and redirecting traffic, none of which is instant, and none of which most teams have actually measured until they're living it.

Both numbers move in the same direction for the same reason: money and effort spent on standing infrastructure. A warm standby, a second region already running a scaled-down copy of the system, continuously kept current through replication instead of periodic backups, improves both. RPO shrinks toward however far behind that replication actually is, seconds or minutes instead of a full day. RTO shrinks toward however long it takes to redirect traffic and scale the standby up, minutes instead of hours, because the infrastructure was already running the whole time.

Nobody gets to pick RPO and RTO of zero for free. Every improvement on either number is a specific, real thing someone built and is paying to keep running, not a setting you turn on.

In Plain Terms

Someone backs up their phone's photos to the cloud once a night. If the phone gets stolen at 11pm, right before that night's backup runs, an entire day of photos is gone for good, the RPO was set the moment they chose "once a night" as the backup frequency, not the moment the phone was stolen. Buying a replacement phone the next day is easy. Actually getting every photo back onto it, waiting for years of backed-up data to redownload, takes real time too, hours, depending on how much there is, a completely separate question from how many photos were lost in the first place. "I back up my photos" never answered either number on its own.

03 How It Works

Rendering diagram…

04 Trade-offs

	Backups only	Continuous replication + warm standby
RPO, the data loss window	Up to one full backup interval, often 24 hours	Seconds to minutes, bounded by replication lag
RTO, time to restore service	Hours: provision infrastructure, restore the backup, replay any logs	Minutes: redirect traffic to infrastructure that's already running
Standing cost	Minimal, storage for backups only	Real, a second region's worth of infrastructure running and largely idle
What it's actually for	Recovering from data corruption or a small-scale loss, where speed doesn't matter as much	Surviving a regional outage where the service needs to be back fast

05 Where You'll Actually See This

Database replication is the exact mechanism a warm standby's improved RPO is built on, just replicating to a different region instead of a different rack. The same asynchronous trade-off applies directly: a standby is only ever as caught-up as its replication lag allows, and a region failing at the wrong instant loses whatever writes hadn't streamed across yet, the same gap that article covers for a single replica.

A failover to a warm standby is CAP theorem's choice, made on purpose, in advance, instead of forced by an unplanned partition. Accepting a standby that's a few seconds behind is choosing availability over waiting for perfect consistency with a region that might not come back for hours. The mechanism isn't different from CAP's usual framing. What's different is that this time the trade was decided deliberately, on a schedule the team controls, instead of a network forcing the decision at 3am.

Availability covers the same redundancy logic at a smaller scope, worth being precise about the difference rather than treating this as the same problem scaled up. Availability's redundancy assumes losing one instance, or one AZ, is survivable because something else in the same region can pick up the traffic automatically, in seconds, with a load balancer and a health check doing the work. A regional outage takes out everything in that region at once, there's no "something else nearby" left to route to automatically. Disaster recovery is a genuinely different problem: the failover isn't instant, it isn't automatic by default, and RPO and RTO don't show up anywhere in a conversation about one failed instance.

Neither case study discusses regional failure at all, worth naming rather than glossing over. Both assume a single-region deployment throughout. The moment either needed to survive an entire region going down, backups and a cold restore would be the honest starting point, and RPO and RTO are exactly the numbers that conversation would need answered before anyone could say how bad that would actually be.

A documented failover plan that's never actually been run is exactly the unproven state chaos engineering is about. Actually failing over to the standby region on a schedule, watching RTO turn into a measured number instead of an assumed one, is a chaos experiment aimed at the biggest blast radius this article covers, not a different kind of exercise from tripping a circuit breaker on purpose.

How to reason about it

When asked how you'd recover from a disaster, don't just say "we have backups." Name your actual RPO and RTO, specifically, in units of time, and say what it would cost to make either number smaller: continuous replication for RPO, a warm standby for RTO. An answer that stops at "we back up nightly" hasn't said how much data would be lost or how long the outage would last. It's described a safety net, not the two numbers that safety net is supposed to produce.

06 Quick Check