VPS Snaps
← All articles

Kubernetes Restore Guide: Namespaces, etcd and Full Clusters

Kubernetes Restore Guide: Namespaces, etcd and Full Clusters

Engineer reviewing a Kubernetes cluster recovery

There are three kinds of Kubernetes restore, and they use different tools: bringing back one application or namespace into a healthy cluster, restoring the control plane from an etcd snapshot, and rebuilding a whole cluster somewhere new. Pick the one that matches what broke, restore into a clean target, and verify the data and the traffic path before you call it done. This guide walks through each case, the order of operations, and how to test recovery before you need it.

Vpssnaps
Automate Your Server Backups
VPS Snaps automates encrypted, S3-compatible backups across multiple cloud providers, with run logs that track successes and alert you to failures.

TL;DR

  • Match the restore to the failure: a deleted namespace needs an application restore, a corrupted control plane needs an etcd restore, and a lost cluster needs a rebuild plus a restore.
  • An etcd snapshot covers cluster state only. Persistent volume data, DNS, load balancers, IAM and secrets management need their own backups and notes.
  • Restore into a clean target first, then check the data and the user path, not just that pods are running.
  • Drill it: restore a namespace to a test cluster on a schedule, and rebuild a full cluster less often, then compare real recovery time with your target.
  • Automate backups into storage you own so a restore never depends on the system that just failed.

Table of Contents

Decide what you are restoring

Most panicked searches for a restore procedure start with the wrong assumption: that “restore the cluster” is one operation. It is not. The Kubernetes community’s data protection white paper lists application recovery options as recovery to an alternate namespace, cluster or region (Kubernetes data protection workflows). That range maps to three situations.

Situation Typical cause Restore method Typical effort
One app or namespace is gone or broken Bad deploy, accidental delete, bad migration Object-level restore (for example Velero) plus volume data Minutes to an hour
Control plane state is corrupt or lost etcd corruption, failed upgrade, lost control plane nodes etcd snapshot restore One to a few hours
The whole cluster is gone Region outage, account problem, ransomware New cluster, then restore state and data Hours, depending on prep

Start at the top of the table. If a namespace restore solves the problem, do not touch etcd. Restoring etcd rolls back the whole cluster to the snapshot moment, which also undoes legitimate changes made to every other workload since.

What a Kubernetes backup has to contain

A backup that cannot rebuild a working service is just a file. Check that yours covers five layers:

  1. Cluster state. The API objects: deployments, services, config maps, custom resources, RBAC. This lives in etcd, and you can also export it as manifests.
  2. Persistent data. The contents of persistent volumes, plus database dumps for anything transactional. A volume snapshot taken mid-write may not be consistent for a database, so dump the database as well.
  3. Secrets and certificates. Secrets are in etcd, but if you rely on an external secrets manager you need access to it from the new cluster.
  4. Configuration outside the cluster. A good disaster recovery plan accounts for external DNS records, load balancer configuration, IAM roles and cloud networking, according to one community handbook. Keep these as infrastructure code, or at least written down.
  5. The way you build the cluster. Version, add-ons, CNI, storage classes and ingress controller. A restored workload needs a storage class with the same name it was created with.

Disaster recovery is also not the same as high availability. High availability keeps the service up through a node or zone failure. Disaster recovery is what you do after that has failed, or after someone deleted the wrong thing. Replication copies mistakes instantly, so it is never a substitute for a backup.

Restore one application or namespace

This is the case you will hit most often. The cluster is healthy, but something inside it is wrong.

With Velero

Velero stores backups in an object storage bucket and restores through a Restore object. Per the Velero restore reference, the controller validates the request and accepts only backups in the Completed or PartiallyFailed phase as restore sources. In practice:

velero backup get
velero restore create --from-backup nightly-2026-10-24 \
  --include-namespaces shop \
  --namespace-mappings shop:shop-restore

The namespace mapping matters. Restoring into shop-restore lets you inspect the recovered data next to the broken original before you cut anything over. Afterward:

velero restore describe <restore-name> --details
velero restore logs <restore-name>

Read the warnings. Resources that already exist are skipped by default, which is usually what you want, but it means a half-broken original will not be overwritten silently. Our walkthrough on backing up a Kubernetes cluster with Velero covers the backup side, which decides how clean the restore will be.

Persistent volume data

Object restores recreate the claim, but the data has to come from somewhere: a CSI volume snapshot, a file-level copy, or a database dump. For databases, restore the dump into a fresh instance and compare row counts and a few known records before pointing the application at it. Do not assume that a Running pod means good data.

A short checklist for any app restore

  • Scale the broken workload to zero so it stops writing.
  • Restore into an alternate namespace first when you can.
  • Confirm secrets, config maps and persistent volume claims exist.
  • Check the ingress, service and DNS path, then check a real user action.
  • Cut over, then delete the old namespace only after a retention window.

Restore etcd on a self-managed control plane

Only self-managed control planes give you this option. On managed services such as EKS, GKE or AKS, you cannot reach etcd, so you rely on object-level backups and rebuild instead.

Taking the snapshot

The Kubernetes documentation describes two ways: run etcdctl snapshot save against a live member, or copy member/snap/db from a data directory that no etcd process is using. If etcd sits on a volume that supports snapshots, a volume snapshot also works. The same page advises encrypting snapshot files, because they contain every secret in the cluster.

Three methods for taking an etcd snapshot

Two timing notes from OKD’s documentation, which apply to other distributions as a habit: don’t take etcd backups before the first certificate rotation has completed, and schedule them in off-peak hours because snapshots carry a high I/O cost (OKD etcd backup guide). It also recommends keeping backups somewhere outside the environment itself. That is the single most important rule here: a snapshot stored on the node that died is not a backup.

Some installers automate this. In kOps-managed clusters, backups of both the main and events etcd clusters go to object storage alongside the cluster configuration, and by default they run every 15 minutes (kOps etcd backup). If you use kOps, you may already have what you need.

Restoring

The Kubernetes docs name etcdutl as the tool that works directly on etcd data files, including restoring snapshots, while etcdctl handles network operations. The general flow on a kubeadm-style cluster:

  1. Stop the API server and etcd on the control plane nodes (on kubeadm, move the static pod manifests out of /etc/kubernetes/manifests).
  2. Restore the snapshot into a new data directory: etcdutl snapshot restore snapshot.db --data-dir /var/lib/etcd-restored.
  3. Point the etcd manifest at the new data directory.
  4. Put the manifests back and wait for etcd and the API server to come up.
  5. Run kubectl get nodes and kubectl get pods -A, then look for drift: workloads created after the snapshot will be gone, workloads deleted after it will be back.

On a multi-member control plane, every member must be restored from the same snapshot and brought up with consistent cluster settings. Do not mix a restored member with an old one.

Version discipline matters. OKD tells you to restore using a backup taken from the same z-stream release. Even where your distribution is less strict, restore to the same Kubernetes minor version that took the snapshot, and upgrade afterward. We keep a longer command-by-command reference at Back up and restore etcd for Kubernetes, and a lightweight-distribution variant in our k3s backup and restore guide.

Rebuild a cluster after a total failure

When the cluster is gone, you are building a new one and loading state into it. The order that works:

  1. Provision the infrastructure. Networks, node pools, IAM, from infrastructure code if you have it.
  2. Install the cluster at a compatible version. Same minor Kubernetes version as the backup, same CNI and storage classes.
  3. Install the restore tooling. Point Velero (or your tool) at the existing backup bucket. This is why the bucket must live outside the failed environment.
  4. Restore cluster-scoped resources and namespaces. Custom resource definitions and storage classes first, then the applications.
  5. Restore data. Volumes and database dumps.
  6. Reconnect the outside world. DNS, load balancers, certificates, firewall rules, external secrets.

Step 6 is where most real recoveries stall. The CNCF’s September 2026 write-up on reproducible failure scenarios makes the point that recovery can fail at the joins between layers even when each layer works on its own, for instance when restored data has no traffic path (CNCF disaster recovery guidance). A pod can be healthy, its volume full of correct data, and still unreachable because the load balancer points at a dead address or an ingress class is missing. If your cluster sits behind complex network controls, review container security and network practices so restored policies don’t silently block traffic.

If you run in more than one region, an active-passive or active-active layout shortens recovery, at the price of cost, complexity and data consistency work. For most small teams a documented rebuild from backups in a second region is the realistic starting point.

Set recovery targets before you choose tooling

RPO is the maximum data loss you accept, measured in time. RTO is the maximum downtime. They drive every decision above: a 15-minute RPO means frequent etcd and volume backups, and a one-hour RTO means a rebuild you have practiced, not one you read about the night before. Write the numbers per application, not per cluster. A payments database and an internal wiki should not share the same target.

For a deeper treatment, see our RPO and RTO explained with worked examples and the tiering templates. A separate piece on setting recovery targets that hold is also worth a look when you need to argue the numbers with management.

Test the restore, or assume it fails

The CNCF guidance describes a recovery test as restoring a complete stateful application into a clean target and validating the data and the user path against expected contents. Do exactly that. A practical cadence:

  • Monthly: restore one namespace into a test cluster. Check data, not just pod status.
  • Quarterly: build a new cluster from scratch and restore everything into it. Time it.
  • After big changes: a new storage class, a Kubernetes upgrade or a new database engine each deserve a rerun.

This cadence follows what one community handbook suggests: restore a namespace to a test cluster monthly, do a full restore to a new cluster quarterly, and compare actual RTO with your target.

During a drill, record these:

Check What good looks like
Backup found Latest completed backup identified in under five minutes
Restore status No unexplained warnings in restore logs
Data integrity Row counts and known records match expected values
User path A real login or API call succeeds through the public address
Elapsed time Measured RTO is within target

Every miss becomes a fix: a missing step in the runbook, a credential that was only in someone’s head, a storage class that was not recorded.

Our perspective

We built VPS Snaps around one belief: a restore should never depend on the tool that made the backup. Too many backup products stage your data in their own storage or require a proprietary restore client, which means that on the worst day of the quarter you are asking a vendor for permission to get your own data back.

So we stream backups directly into buckets you own, S3-compatible and in your account. Provider snapshots stay in your provider account and can be restored without us. The part we automate is the part people skip: scheduling, retention, encrypted credential storage and run logs that show every success and flag every failure. A backup you have never seen fail is not the same as a backup that has never failed, and a log is the difference.

In Kubernetes specifically, we back up manifests and volumes, and we detect applications on a server so a database or Docker volume does not get missed. Our advice to any team: automate backups first, write the restore runbook second, and run the drill third. Most outages we hear about were not caused by missing tools, but by a restore path nobody had walked.

— Scott

Automating the backup side

Restores are only as good as the backups feeding them, and scripts that nobody monitors are the usual weak link. VPS Snaps schedules provider snapshots, database dumps, Docker volume backups, file backups and Kubernetes manifest and volume backups into storage you own, with retention handled for you. You can start on the Free plan, and paid plans on the pricing page run from Starter at $10 per month ($120 per year) through Standard at $30 per month, Pro at $57 per month and Agency at $117 per month. If a backup silently stops, you will want alerts for backup failures before you want a restore guide.

Common restore mistakes

  • Backups stored in the failed environment. Keep them in a separate account or region.
  • Unencrypted snapshots. An etcd snapshot holds every secret. Encrypt it, and store the key separately; our encryption guide covers GPG and age.
  • Ransomware reaching the backups. Restrict delete permissions and keep older copies. See how to protect backups from ransomware.
  • Too-short retention. If corruption went unnoticed for two weeks, a seven-day window leaves only bad copies. A retention policy should reflect how long problems stay hidden.
  • A restore run for the first time during an incident. Drill it.

FAQ

Can I restore a Kubernetes cluster from an etcd snapshot alone?

You can restore the cluster state: deployments, services, secrets and other API objects. You cannot recover persistent volume contents, external DNS, load balancers or IAM from etcd. Treat the snapshot as one of several pieces, and pair it with volume and database backups.

Do I need Velero if I already back up etcd?

Etcd backups are all-or-nothing for the cluster, so they are a poor fit for bringing back one namespace. An object-level tool lets you restore a single application, to an alternate namespace or even another cluster. Most teams run both.

Can I restore etcd on EKS, GKE or AKS?

No, the control plane is managed and etcd is not accessible to you. Back up your manifests and persistent data with an object-level tool, keep cluster creation in code, and recover by building a new cluster and restoring into it.

How often should I test a Kubernetes restore?

A monthly namespace restore into a test cluster and a quarterly full rebuild is a workable rhythm for most teams. Re-test after upgrades or storage changes. Time each run and compare it with your RTO.

What should I restore first in a new cluster?

Start with cluster-scoped resources such as custom resource definitions and storage classes, then namespaces and applications, then data. Finish with DNS, load balancers and certificates, and check a real user request before you declare recovery complete.

Sources