genv.

PlatformBackup and restore drills for a hosting platform

10 min readPublished Reviewed By the GENV team

In short

Restore from a real backup into an isolated environment on a schedule, and measure how long it took and what was missing. Backing up the database is the easy half; the hard half is everything that is not in it — secrets, DNS, provider credentials and the knowledge of what order to bring things back in.

Every hosting business has backups. Far fewer have ever restored one under time pressure, and the gap between those two states is where the outage becomes a closure. A drill is cheap and it is the only thing that converts a backup into a recovery capability.

What a drill actually is

  1. Take a backup that was produced by the normal automated process, not one made for the drill.
  2. Restore it into an environment isolated from production — a separate database, separate credentials, no outbound network to providers.
  3. Bring the application up against it.
  4. Check a list of facts you wrote down in advance: record counts, the most recent invoice, a known customer, the latest migration version.
  5. Write down how long it took and what was missing.
  6. Destroy the environment.

Step two matters more than it looks. A restore drill that can reach your real providers is a drill that can send real emails and make real API calls with a copy of production data.

The things that are not in the database

What a database restore does not bring back
ThingWhere it actually livesHow to cover it
Provider API credentialsA secrets store, or somebody's password managerDocumented rotation procedure; restoring a database gives you references, not values
DNS recordsYour DNS providerExport the zone, keep it with the runbook
TLS certificatesThe load balancer or issuing serviceAutomated issuance, tested from cold
Object storageIts own serviceIts own backup, with its own restore drill
The order to start things inSomebody's headThe runbook. This is the single highest-value document

The last row is the one that turns a two-hour recovery into a two-day one. Write down the order: database, migrations, application, workers, then providers. Write down which environment variables are required. Write it for somebody who was not there.

Migrations are part of the restore

A backup is taken at a schema version. If you restore it into an application that has moved on, the application will either refuse to start or — much worse — run against a schema it does not expect. Record the schema version with the backup, and rehearse the upgrade path from the restored version to the current one as part of the drill. That rehearsal is also how you find out whether your migrations are actually reversible.

How often, and what to measure

  • Monthly for the database restore. It is the most likely thing you will need and the cheapest to rehearse.
  • Quarterly for a full recovery including the runbook, ideally performed by somebody who did not write it.
  • After any schema change that is hard to reverse, because that is when the restore path is most likely to have broken.
  • Measure time to first byte of a working application, and the count of steps that were not in the runbook. The second number should trend to zero.

Point-in-time and the honest recovery point

Know your actual recovery point, not the advertised one. If backups run nightly, your recovery point is up to twenty-four hours, and that means a day of invoices and payments. If that is unacceptable, you need continuous archiving — and you need to have restored from it, because point-in-time recovery has more moving parts and more ways to be silently misconfigured.

Where GENV is

In the software: a schema change that alters shape ships a documented rollback, and that rollback is executed by an automated test against a scratch database — several are deliberately written to refuse while dependent records exist rather than deleting them to make a downgrade succeed.

What has been rehearsed is the database. For a recent schema change, a stored backup of the staging database was restored into an isolated PostgreSQL instance and the upgrade run against it: 111 user tables afterwards, and the 110 that existed beforehand confirmed intact. The backup taken after the migration was then restored in turn, so the post-upgrade backup is known to be restorable and not merely written. Both restores are recorded against that release.

What has not been rehearsed is everything else. Bringing back the identity server, the background workers and the secret store — the whole-stack recovery this article is mostly about — is a separate exercise, and GENV has not drilled it. So take the checklist above as what to set up for your own platform, not as a description of what GENV has proven about its own.

Questions

Is a managed database provider's backup enough?

It is a good backup and it is not a recovery plan. You still need to know how long a restore takes, what is missing from it, and in what order to bring the rest back.

Should backups be tested automatically?

The restore can be automated and should be. The runbook drill should be done by a person, because what it tests is whether the document is usable.

How long should backups be kept?

Long enough to cover the time it might take to notice a problem — corruption found three weeks later needs a backup older than three weeks. That is a business decision; make it explicitly rather than accepting a default.