Design Desk

Storage & Data

Database Replication

Copying a database onto more than one machine fixes a read ceiling and a single point of failure at once, but only after you decide how far behind a copy is allowed to be.

Storage & DataAdvanced7 min readPrerequisites: Availability, CAP Theorem

After this lesson, you should be able to:

  • Explain the trade-off between synchronous replication (safety, added write latency) and asynchronous replication (speed, a staleness window).
  • Define replication lag and identify when a stale read from a replica is acceptable versus when it isn't.
  • Diagnose the read-your-own-writes problem as an expected consequence of asynchronous replication, not a bug in the replica.
  • Connect the sync-versus-async choice to CAP theorem: reading from an async replica trades consistency for availability on every read.

At a Glance

  • One database instance is a performance ceiling, every read competes for the same machine, and a single point of failure, at the same time.
  • Replication fixes both by copying data onto other machines, but every copy is either waiting for confirmation (synchronous) or catching up after the fact (asynchronous), never both.
  • Synchronous replication costs latency on every write. Asynchronous replication costs a window where a replica can answer with a stale, or on failover, lost, write.

01 The Problem

A news publisher's single Postgres primary serves editors publishing articles and millions of readers loading them, on the same machine. During breaking news, reads queue up behind each other because there's only one server to ask. Separately, that same one machine is a single point of failure: a disk failure or a bad patch takes the entire site down, reads and writes both, until someone restores from backup.

Adding a replica to take reads off the primary raises the actual question: does the replica have to be perfectly caught up before it's allowed to answer a reader, waiting on confirmation from the primary first, or can it answer immediately with whatever it currently has, even if that's a few hundred milliseconds behind the latest write? Wait for confirmation, and every write to the primary now pays for a network round trip to the replica before it's considered done, real latency on every single write, whether or not the replica was ever close to failing. Don't wait, and a reader might load a page showing an article as unpublished, or a comment count that's a few seconds stale, and if the primary fails in that window before the replica caught up, that write is gone.

Neither option is free. This is the trade-off between synchronous and asynchronous replication.

02 What It Is

Replication is copying a database's data onto one or more additional machines, replicas, so more than one server can answer for the same data. It solves two separate problems that happen to share a fix: read capacity beyond what one machine can serve, and durability beyond what one machine's disk can guarantee if it fails. How a replica catches up is the whole decision. Synchronous replication makes the primary wait for a replica to confirm it received a write before telling the client the write succeeded, so the replica is never behind, at the cost of that wait on every write. Asynchronous replication lets the primary reply the moment the write is durable on itself, then streams the change to replicas afterward, so writes stay fast, but a replica can lag by some measurable window, replication lag, during which it will hand out an answer that's already wrong.

There's no version of replication that avoids this. Copying data across a network takes time, no matter how fast the network is, so a replica is either behind by that time, or the write is delayed by that time waiting for it to catch up.

In Plain Terms

A manager keeps the master schedule on her desk and a copy pinned to the break-room wall so the whole team doesn't queue at her door to check their shift. If she updates the wall copy herself before telling anyone their new shift is confirmed, everyone waits a little longer, but the wall is always right. If she tells people their shift is confirmed and updates the wall whenever she gets a free minute, people find out faster, but for a while the wall might show yesterday's schedule, and if she gets pulled into a meeting before updating it, someone could show up for a shift that no longer exists.

03 How It Works

Rendering diagram…

The primary waits for Replica A to confirm before telling the client the write succeeded, so Replica A is never behind, at the cost of that round trip on every write. Replica B gets the same change, but only after the client already has its answer, so there's a window where Replica B is behind and would answer a read with the old value.

Try it: switch to Asynchronous, write a value, then Read from Replica before clicking Replica Catches Up, the replica still answers with the old value. Then write again, Fail Primary before the replica catches up, and Promote Replica, the write that never replicated is gone for good.

Replication mode

Primary

v1

Up

Replica

v1

Matches primary

No writes yet.

04 Trade-offs

 Synchronous replicationAsynchronous replication
Write latencyEvery write waits on a network round trip to the replica, on top of the write itselfWrite returns as soon as it's durable on the primary alone
Data safety on primary failureThe confirmed replica has everything; failover loses nothingAnything written but not yet streamed to a replica is gone if the primary fails at the wrong moment
Reading from a replicaAlways current, since a replica can't confirm a write it hasn't appliedCan be behind by the replication lag; a stale read is possible, not hypothetical
Best forData where losing a confirmed write is unacceptable: the ledger, not the dashboard reading itRead scaling where a few hundred milliseconds of staleness is a non-issue

Watch out

It's tempting to add a read replica and assume every read is now interchangeable with reading the primary. It isn't, not with asynchronous replication. A user who just wrote something and immediately reads it back can land on a replica that hasn't caught up yet and see their own write disappear for a moment. That's not a bug in the replica, it's the lag working exactly as designed. The fix isn't "make replication faster," it's deciding which reads are allowed to be stale and routing the ones that aren't back to the primary.

05 Where You'll Actually See This

The URL shortener case study takes reads off the primary a different way, caching in Redis instead of adding a read replica. Both solve the same problem, too many reads for one machine, and the choice between them usually comes down to whether the data changes often enough that a cache's TTL fits better than a replica's continuously-streaming copy.

This is also the concrete mechanism behind two things covered elsewhere. Availability names the general idea, redundant instances behind something that detects failure and routes around it, and a database replica is that mechanism applied specifically to the layer where the copies also have to agree, or explicitly not, on what the current data actually is. And CAP is this same choice playing out continuously rather than only during a network partition: reading from an asynchronous replica is choosing availability over consistency on every single read, not just when the network breaks. Reading only from a primary, or only from a synchronously confirmed replica, is choosing consistency, at the cost of the capacity a replica would otherwise have bought.

The read-your-own-writes problem in the Watch Out above has a mirror image on the write side, covered in idempotency keys: there, a retried write risks running twice instead of a read landing on stale data. Different symptom, same underlying cause, a network that doesn't guarantee anything happens exactly once or instantly.

CQRS has this exact staleness shape one layer up the stack. A replica lags the primary because the copy has to travel across the network before it's visible; a CQRS read model lags the write model for the same reason, an update has to propagate before a query sees it. The difference is a replica only copies, a read model can reshape the data on the way.

Disaster recovery is this same mechanism pushed to a different scale: replicating to a standby region instead of a standby instance, so a data loss window is measured in seconds of replication lag instead of however long since last night's backup. The asynchronous gap this article covers, whatever hasn't streamed across yet is gone if the primary fails at the wrong moment, is exactly the number that becomes a disaster recovery plan's RPO.

How to reason about it

When asked how you'd scale reads, don't just say "add a read replica." Say whether it's synchronous or asynchronous, and say which reads in your system are allowed to be stale and which ones have to go to the primary no matter what. If every read in your answer goes to "the replica" without that distinction, you haven't actually designed the read path, you've just drawn an extra box.

06 Quick Check

A primary database acknowledges a write to the client, then crashes one second later, before that write has streamed to any replica. The system fails over to a replica automatically. What happened to that write?