Backup and restore¶

Read the first section before you need it. A restore is the one routine operation that can admit a person you revoked, so its steps run in a fixed order and each safety step has a check.

What a restore does¶

A restore puts the store back to the moment of its snapshot. Two things follow, and the rest of this page exists to undo them:

Revocations made since the snapshot are gone, and lifts made since are undone. A revoked principal or certificate is admitted again until the revocation is put back.
Every single-use token redeemed since the snapshot is back, unspent. Redeemable again until it expires (a token lives at most 15 minutes).

port49 defends against both on its own, and still asks you for steps, because some restores it cannot see (below). Keep the order of the runbook: the token delete comes before anything reaches the store, and the revocations are re-issued before any broker starts.

What each shape must keep¶

All three shapes (systemd, container, Helm) keep the same things. Only the paths differ.

What	Why it is in the backup	systemd	container (compose)	Helm
The etcd data of every cluster: the directory store, and each region's session store	The revocation marks, second factors, pins and sessions. A snapshot holds the plaintext index (principal, device, username, certificate, `sid`): guard it as the store itself.	the hourly `port49-etcd-snapshot` timer, from an etcd host	each `etcd-N-data` volume	the `port49-etcd-snapshot` CronJob (`examples/backup-cronjob.yaml`)
`broker.state_dir` of every broker: `marks.json`, `witness.json`, and `issuing-ca.pem` and `issuing-ca.key` when the broker issues certificates	`marks.json` is the copy that puts back revocations a restore lost. `witness.json` is how a broker tells a restore. The CA is the private key your devices trust: whoever reads the backup can issue certificates.	`/var/lib/port49-broker`	the `broker-state` volume	one PVC per broker replica, `state/` inside it
port49d's accounting spool and the file beside it, `<accounting.spool_path>.restore.json`	`restore.json` is port49d's record of the restore witness, so a restart still knows the store was restored. Two port49d with spools in one directory each keep their own.	`/var/spool/port49d`	the `port49d-spool` volume	the port49d PVC (`/var/spool/port49d/data`)
On the memory store only: the directory holding `broker.revocations_file`	The memory store runs the broker inside port49d, which refuses `broker.state_dir`; the broker keeps its state beside the revocations file. That directory holds the revocations file (the marks), `enrollments.json` (the sealed enrollment store), the CRLs the broker verified, and, when the broker issues certificates, `issuing-ca.pem` and `issuing-ca.key` (with `-next` or `-previous` copies during a CA rotation): whoever reads the backup can issue certificates. On etcd the store keeps the marks and none of this applies.	the directory of `broker.revocations_file`	same	same
The enrollment store of a `directory_password` verifier	Second factors. On etcd it is sealed records in the directory store; on the memory store it is the sealed file `enrollments.json`, in the directory above.	the daily `port49-enrollments-export` timer, on a broker host	no job is shipped: run `port49-broker export-enrollments --out FILE` from your scheduler	the chart's `backup.enrollments` CronJob (off by default)

The configuration, the policy and the secrets are not on this list: they are in your configuration management and your secrets manager. Two cautions:

A backup is only restorable while every key that sealed its records is kept. Every record inside an etcd snapshot, a Velero backup or an enrollments export stays sealed under the encryption: key that was current when the record was written. An enrollments export keeps each record's own seal and seals only its envelope under the key current at export time.

Keep every old key file, offline, as long as any backup holding its records is kept. Velero also stores the encryption Secret in its bucket, so the bucket is as sensitive as the key. Key rotation has the rule and what a missing key costs.
Back up broker.state_dir with the broker's host, not from a different moment. A broker whose state_dir survived the restore is the one that puts back the revocations the store lost, so the runbook starts it first.

The enrollments export¶

port49-broker export-enrollments --out FILE writes the sealed enrollment store to a new file (mode 0600, never replacing one). The unit and the CronJob run it on a schedule, keep 14 days, and write a temporary file that is renamed, so a failed run changes nothing. On Helm a stale-check init container fails the Job when a .new file is left from a run that did not finish; remove it, and the next run goes.

A new deployment that uses directory password sign-in runs port49-broker init-enrollments once, before the first start (on Helm, examples/init-enrollments-job.yaml). It creates the store the export reads, and a second run is refused (the enrollment store is already there). Until it has run, sign-ins answer enrollment_store_lost.

To restore from an export, as an admin with a key press:

port49 directory import < enrollments-<date>.json

The import refuses when the store holds a second factor enrolled or reset after the export was taken, because it would undo it. --replace replaces the store anyway. An export sealed under a key the broker does not hold is refused, and the message tells you to set that key as encryption.previous.

Unmeasured

The Helm export has been run only against a freshly initialised store, which holds the creation mark and no enrolled person. An export that holds enrolled people has not been measured. The systemd timer has not been run on a real host: run it once by hand before you rely on it.

The restore floor¶

An etcd restore can bring back a token that was redeemed after the snapshot. port49 refuses such a token in three places, and the runbook adds a fourth, the delete, for the cases where the first two cannot tell.

port49d's restore floor. Every token records the store revision it was created at. port49d reads meta/witness once a second. When the witness or the store's revision is lower than its own record (<accounting.spool_path>.restore.json), the store was restored. The revision of that read becomes the floor: a token at or below it is refused FAIL with reason token_spent, once, and the user mints again. A token minted after the read is served. Each refusal adds one to port49_redemption_restored_refused_total.
The runner's purge. The broker that holds the keyspace compares meta/witness with its own broker.state_dir/witness.json. A store below the record, or no record at all, makes it purge every unspent token and every once-only record (sign-in challenges, proofs, nonces).
The mint gate. While a purge is owed, that broker mints nothing, so a token is not minted into a purge that would delete it.
The delete, step 3 of the runbook, before any port49d or broker reaches the restored store.

When port49 cannot tell¶

port49d tells a restore only from a record that is newer than the restored store. So it does not see one when:

the restore also restored the spool directory or broker.state_dir (Velero, a volume snapshot), so restore.json and witness.json came back with the store. They are newer than the store only when the store had writes in the backup window, which is usual on a busy store and not on an idle one;
port49d's spool directory is new or was lost. It has no restore.json and takes the restored witness as its first record. It logs port49d: store: no restore record at <path>: a restore made before this port49d's first read of the store cannot be told; after a restore, delete the unspent tokens before this port49d reaches the store (step 3 of Restoring etcd from a snapshot).

Either way a restored token is redeemed a second time with no line saying so. The delete is required in both cases. A broker with no witness.json (an empty or new broker.state_dir) purges once at its first takeover, which costs every user one new token and is expected on a cold start of a whole fleet on new volumes.

A record that cannot be read or decoded is moved aside (<path>.bad-<time>), and the first read is taken as a restore: every token in the store is refused token_spent once.

What you see¶

You see	It means	You do
port49d's stderr: `port49d: store: the session store was restored from a snapshot: meta/witness is …` and the tail `Revocations and marks made after the snapshot are lost: follow the restore runbook, Restoring etcd from a snapshot`	port49d found the restore and set its floor. The line names the floor revision. After a restore without `--bump-revision` it also names the band (below).	Follow the runbook from the revocations step if you have not. The line is a report, not a recovery.
`port49_store_restore_detected_timestamp_seconds` above 0, and the alert port49_store_restore_detected (ticket, for a day after)	Same event, as a metric.	The same.
A user's login fails and `port49_redemption_restored_refused_total` rises	Their token is from before the restore.	They mint a new token (`port49 token`). Nothing else.
A broker's stderr: `port49-broker: runner: <why>; every unspent token is purged and must be minted again (port49 token): N purged; M once-only records (sign-in challenges, proofs, nonces) deleted`	The runner purged. `<why>` is `the store's meta/witness is … below the … this replica saw: the store was restored from a snapshot`, or `no witness record in broker.state_dir; …`.	Nothing. Every user mints again.
A broker's stderr: `port49-broker: runner: N revocation(s) in broker.state_dir/marks.json were missing from the store (an etcd restore, …) and are put back: <marks>; lift any that were lifted on purpose with port49 revoke --lift`	The runner put back the revocations its copy held.	Lift again, with `port49 revoke --lift`, any of them that were lifted on purpose before the snapshot.
A broker's stderr: `broker.state_dir had no marks.json at this start; a revocation a store restore lost is put back only if this copy saw it`	This broker had no copy, so it put back nothing.	If no broker's `state_dir` survived, re-issue every revocation made since the snapshot (step 4 of the runbook).
`port49_broker_mint_gate_closed` at 1 and `POST /v1/token` answering `503 store_unavailable` with `Retry-After: 5`	A broker owes the purge and mints nothing until it is done.	Wait. If it stays at 1 for more than a minute (ticket port49_broker_mint_gate_closed), no runner is purging: check that one broker holds the keyspace, and read its stderr for a purge that failed. A purge cut short is made again by the next takeover.
`port49_redemption_restore_unjudged` at 1, and `port49d: store: reading meta/witness, to tell whether the store was restored: …; token redemptions are refused (store_unavailable) until a read answers; tried again every 1s`	port49d cannot read `meta/witness`, so it cannot tell whether the store was restored and refuses every token login. Page after 2 minutes: port49_redemption_restore_unjudged.	Fix the store, or the daemon role's read of `meta/witness`. A device whose authentication `ERROR` is final (IOS XE, and Junos under `tacplus` alone) refuses break-glass until the read answers or port49d is stopped; stopping port49d lets those devices fall to break-glass. Do not delete `restore.json` to clear it: without the record port49d cannot tell a restore.

The revision bump and the 5-minute band¶

Restore an etcd snapshot with --bump-revision. It raises the restored store's revision by a billion (eleven days of writes at a thousand a second), so etcd never reuses a revision for another write. port49d tells a restore from its witness whether or not --bump-revision was used, and with it the floor needs no band: nothing is refused but the old tokens.

Measured on vEOS with etcd 3.6.15 (L35-restore-arista_eos.md): a snapshot at revision 338 of a store at 347 restored to 1000000338, its next write was 1000000339, and a read at 347 was refused as compacted.

Without --bump-revision (a Velero restore cannot use it) the restored store climbs through the revisions the old timeline already used. port49d then also refuses, for 5 minutes after its read, every token minted at a revision up to the old high mark, token_spent. The log line names the time: a token minted until <time> (5 minutes after this read) at a revision up to <N> is refused until <time>; restore with --bump-revision (step 2 of Restoring etcd from a snapshot) to end that. Users see a failed login, and so does every retry until the window ends. A token minted inside the window keeps its mint time, so it stays refused until the band ends, 20 minutes after the read, and pasting the same token again keeps failing. Tell users to mint a new token after the window. Break-glass is unaffected.

Restoring etcd from a snapshot¶

This is the runbook for a restore of etcd on any shape. Do the steps in order. A step marked Check is a stop: do not go on until the check passes.

Stop what could reach the restored store.

On Helm, first suspend every GitOps controller (Argo CD auto-sync or self-heal, Flux) that manages port49 or port49-etcd, with the commands in the Velero runbook below (the paragraph before its step 1). Resume it after step 6. A StatefulSet cannot serve loopback only (step 2), so on Helm the brokers held at 0 are the only guard. A controller that puts the replicas back lets a broker reach the restored store before the delete and before the revocations are re-issued, and can admit a revoked principal.

Stop every port49-broker (systemd: systemctl stop port49-broker on every broker host; container: stop the port49-broker service; Helm: scale the broker StatefulSet to 0).

port49d may run on: it goes dark while the store is down. Also stop every port49d in any of these cases, because port49d may then not see the restore, and the delete below must land before it reaches the store:
- the restore also restored the spool directory or broker.state_dir;
- a port49d's spool is new;
- you cannot serve the restored store on loopback only (a StatefulSet cannot).
Restore with --bump-revision and --mark-compacted on each member, from the same snapshot: etcdutl snapshot restore <snapshot> --data-dir <new dir> --bump-revision 1000000000 --mark-compacted, with the member flags your cluster uses (--name, --initial-cluster, --initial-advertise-peer-urls).

Start the members serving clients on loopback only (for example --listen-client-urls https://127.0.0.1:2379 --advertise-client-urls https://127.0.0.1:2379, the TLS flags unchanged, the peer URLs as they were so quorum forms). On systemd and compose, no port49d or broker can reach the restored store yet; on Helm only the stopped StatefulSets keep them off it.

If the store was restored without --bump-revision anyway, also restart every port49d only after step 5's first runner has said its put-back line. Two things go wrong otherwise:
- a port49d restarted before then loads the restored marks, so it enforces none of the revocations the restore lost;
- a port49d started after the restore seals below what a port49d that ran across it knows, so its logins are denied store_unavailable until that process writes at the key.
Delete the unspent tokens and the judged records, on the loopback address, as etcd's root. Use your store prefix (stores.session.etcd.prefix, /port49/ by default):
```
etcdctl --endpoints https://127.0.0.1:2379 del --prefix /port49/v1/t/
etcdctl --endpoints https://127.0.0.1:2379 del /port49/v1/meta/judged
etcdctl --endpoints https://127.0.0.1:2379 del /port49/v1/meta/judged_bundle
```
Add the --cacert, --cert and --key of etcd's root identity; in a container, run etcdctl in the member's own network namespace. Check: all three returned. Only then restart the members on their usual client URLs.

On Helm the one-shot Job restore-unspent runs these three deletes from etcd's own image, as etcd's root; the Velero runbook below uses it.
Re-issue the revocations no broker's marks.json holds, while every broker is still stopped. These are:
- those made while every broker was down, since no broker saw them;
- if no broker's state_dir survived the restore, every revocation made since the snapshot.
Run port49-broker revoke --offline --principal <principal> on a broker host as the broker's user (sudo -u port49-broker; as root it refuses and names the line). On Helm, deploy/helm/port49/scripts/revoke-offline.sh does the same from the repository root. It leaves the brokers at the 0 it finds.
Check broker.policy_file on every broker host before any broker starts. Step 3 deleted meta/judged, so the reload guard at a broker's start has no judged bundle to check against. The first start says no bundle has been judged on this keyspace; this start is not judged against one and goes on, and a policy file the guard would have refused is not caught. Confirm each host holds the policy you mean to serve.

Start the brokers, first a broker whose broker.state_dir survived the restore. The first runner puts back only the revocations its own marks.json holds that the store lacks, and names them. A broker on an empty or new state_dir that runs first puts back none of the revocations made since the snapshot: its line says had no marks.json at this start. If no broker's state_dir survived, step 4 had to re-issue them all.

The first runner also applies every mark to the restored contexts and purges the unspent tokens. Check: a runner printed every unspent token is purged. If none did, delete the restored tokens again as root (step 3). Then start any port49d you stopped.
Lift again, with port49 revoke --lift, any revocation that was lifted since the snapshot and that the put-back line named.
Run port49 directory forget again for every pin forgotten since the snapshot: a restore brings the old pin back (Stores).
If a replay fails token_not_found with the log line etcd: superseded record "<prefix>v1/meta/superseded_all" does not decode, delete that one key as etcd's root (etcdctl del <prefix>v1/meta/superseded_all); the next purge writes it again. That is availability, never an admission.

A restored snapshot also brings back every context it held. One that a revocation by principal or certificate ended is denied again at once by the marks. One that a self or key revocation ended has no mark and is live again until its idle TTL: revoke it again.

Restoring with Velero¶

Prefer restoring etcd from an etcd snapshot (the CronJob's), as above, with --bump-revision, and use Velero for the other volumes. A Velero restore that includes port49-etcd is a store restore: it cannot use --bump-revision, so the 5-minute band applies, and the spool and broker.state_dir come back with the store, so the delete of step 3 is required, not belt and braces. The runbook below is the repository's own file, deploy/helm/port49/examples/velero/README.md, so it matches the chart you run. It restores etcd alone first, runs the restore-unspent Job (the delete of step 3), holds etcd down, restores port49 with its pods running so Velero can copy their volumes, then holds both StatefulSets at 0 and brings etcd back, so the revocations are re-issued before any broker or port49d reaches the store.

Unmeasured

The Argo CD and Flux suspend and resume commands in the runbook follow those tools' documented commands, and no GitOps controller has run the restore. Restoring port49 while etcd is held down in its own namespace has not been measured either: the measurement ran against an etcd that shared the release's namespace and was never held at 0. A file-system copy of a live etcd member under load, and a CSI volume snapshot in place of file-system backup, are also unmeasured. Skipping the suspend step can admit a revoked principal until the revocations are re-issued.

What is measured is on kind, with Velero 1.18.4 and its node agent (L89-kind.md): a backup of both StatefulSets, a deleted namespace and a restore left both ready with no fence refusal, the issuing CA unchanged, and a mint and a login working; a restore with the StatefulSets at 0 replicas hung; and the restore order below, with the pods running and then scaled to 0, completed with marks.json, the CA and the spool intact.

Everything from here to the end of the page is that README verbatim: what the backup holds, its requirements, how to pin the plugin, then the restore.

What the backup holds¶

The backup holds issuing-ca.key, the issuing CA's private key, in each broker's state/ directory, the broker state, and the accounting spool of each port49d. Whoever reads the bucket and the repository key can issue certificates your devices trust.

The bucket alone holds more. Velero encrypts only the volume data, under the repository key. It stores the namespaces' Secrets as it reads them, unencrypted: the TACACS+ keys, the store's encryption key, the etcd CA and client identities, and the TLS key. Read access to the bucket is enough to take them. The repository key protects the volumes and nothing else. Protect the bucket like the CA key.

Requirements¶

The repository key is yours, from a Secret, created BEFORE Velero is installed and before the first backup. Velero's default repository password is a well-known static value, and Velero writes that Secret itself when it starts and finds none. A repository made under the default key keeps it, so a backup made first is encrypted with a key everyone knows. Generate the key and create the Secret from a file, never from a value on the command line:
```
umask 077
head -c 32 /dev/urandom | base64 > repository-password
kubectl create namespace velero
kubectl -n velero create secret generic velero-repo-credentials --from-file=repository-password=./repository-password
velero install ...
```
Keep a copy outside the cluster: a restore into a new cluster needs it. If Velero already runs and has made no backup, delete the Secret Velero made, create yours, and restart the Velero deployment.
Apply schedule.yaml only after step 1.
The bucket must be encrypted at rest (server-side encryption in the object store).
Access to the bucket is limited to the backup's own credentials: no other principal, no public access, no credentials shared with another workload.

Pinning the plugin¶

velero install --plugins cannot take an image by digest: Velero names the plugin's init container after the reference, and a digest makes that name invalid. Give it the tag, then check the running pod's imageID (kubectl -n velero get pods -o jsonpath='{..initContainerStatuses[*].imageID}') against the digest you reviewed, or mirror the plugin by tag into a registry you control.

Restore¶

A Velero restore that includes port49-etcd is a store restore, and a store restore rolls the revocation marks back too. It loses every revocation and lift made since the backup, so a revoked principal or certificate is admitted again until it is re-issued, and it brings back every token redeemed since, redeemable again while it has not expired. The backup also restores port49d's <accounting.spool_path>.restore.json and the brokers' witness.json with the store (two port49d with spools in one directory each keep their own record). port49d can then tell the restore only if <accounting.spool_path>.restore.json is newer than the etcd copy, which is usual on a store with writes in the backup window and not so on an idle one, and a broker tells it on the same terms (its witness.json is then newer than the restored witness), and purges once more and says so at its first takeover: the token delete (step 3 of Restoring etcd from a snapshot) is required, and it must land before port49d or a broker reaches the restored store. A Velero restore cannot use --bump-revision, so when port49d does tell the restore it also arms the 5-minute band: see "What you see after step 9" below. So the restore is made in this order, with nothing reaching the store until the end.

Run step 2 in this directory (deploy/helm/port49/examples/velero/), which holds restore-unspent.yaml; step 8 from the repository root. <release> is your Helm release name, <backup> the Velero backup, <brokers> and <port49d> the broker.replicas and port49d.replicas you run, <prefix> the store prefix (/port49/ by default), and <restore> a name for the restore of port49, new each time you make one (port49-restore-1, then port49-restore-2).

Do not restore port49 with its StatefulSets at 0 replicas (a resource modifier setting /spec/replicas to 0, as the restore-scale-zero.yaml of earlier versions of this page did). Velero's file-system restore copies a volume through the pod that mounts it, and at 0 replicas there is none: measured on kind in L89 cell (c), run 42053716808, the restore stayed InProgress past 8 minutes with every pod volume restore at New and its pod absent. It does not finish. If you started one, see "Recovering from a restore at 0 replicas" below.

So the pods run for the restore, and etcd is held down while they do, so none of them reaches the store.

First, silence the alerts for the restore's window. This order keeps port49d away from the store for longer than 5 minutes, so port49_target_absent (page, 5m) fires, and port49_target_down (page, 2m) fires if step 4's pods crash-loop while etcd is held down. Silence both, and the etcd rules, in Alertmanager for the length you expect the restore to take, and remove the silence after step 11. Expect port49_store_restore_detected (ticket) afterwards; step 9 says why.

Then, if a GitOps controller manages the release, suspend it for both namespaces. These follow Argo CD's and Flux's documented commands and are not measured in port49's lab (L89 runs no GitOps controller). Argo CD with auto-sync or self-heal, or Flux, puts back what this order takes away: it re-creates the HorizontalPodAutoscaler that step 4 leaves out and reverts step 6's scale to 0, so brokers and port49d start before etcd is back or before step 8 re-issues the revocations. Skipping this can admit a revoked principal until step 8 runs.

For each Argo CD Application that manages port49 or port49-etcd, first note its sync policy: copy the .spec.syncPolicy.automated object of argocd app get <app> -o json as it is (its prune, selfHeal, allowEmpty and enabled), so step 11 restores exactly that. Suspend the parent too when the app is managed by another (app-of-apps, or generated by an ApplicationSet): the parent's next sync turns the child's sync back on. For a parent Application, suspend it the same way, first. For an ApplicationSet, first suspend the Application that syncs the ApplicationSet itself, if one does (or it puts the ApplicationSet back as Git has it), then keep the ApplicationSet from overwriting the child's sync policy: add .spec.syncPolicy to its spec.ignoreApplicationDifferences (- jqPathExpressions: [.spec.syncPolicy]), and remove that again after step 11 (first, check your Argo CD version supports ignoreApplicationDifferences). Then:

argocd app set <app> --sync-policy none

Under Flux, suspend every HelmRelease and every Kustomization that applies port49 or port49-etcd (the etcd example is plain manifests, reconciled by a Kustomization):

flux suspend helmrelease <name> -n <namespace>
flux suspend kustomization <name> -n <namespace>

Then:

kubectl delete namespace port49 port49-etcd

etcd alone. No port49d or broker exists yet, so nothing reaches the store.

velero restore create --from-backup <backup> --include-namespaces port49-etcd --wait
kubectl -n port49-etcd wait --for=condition=Ready pod -l app.kubernetes.io/name=etcd --timeout=5m

The token delete (step 3 of Restoring etcd from a snapshot), from a one-shot pod holding etcd's root client certificate (the Secret port49-etcd-root, as etcd-init-job.yaml uses it). Edit PREFIX in the file if your prefix is not /port49/.
```
kubectl apply -f restore-unspent.yaml
state=
for i in $(seq 120); do
  state=$(kubectl -n port49-etcd get job/restore-unspent -o jsonpath='{.status.conditions[?(@.status=="True")].type}')
  case "$state" in *Complete*|*Failed*) break ;; esac
  sleep 2
done
kubectl -n port49-etcd logs job/restore-unspent --all-containers || true
case "$state" in
  *Complete*) kubectl -n port49-etcd delete job/restore-unspent ;;
  *) echo "restore-unspent did not complete ($state): read its log above, and do not go on to step 3" >&2; exit 1 ;;
esac
```
This ends when the Job completes or fails, and prints its log either way. If it did not complete, do not go on: the log says why. Each delete is its own container (unspent, judged, judged-bundle), so kubectl -n port49-etcd logs job/restore-unspent -c unspent shows one, and kubectl -n port49-etcd describe job/restore-unspent has its pods. Delete the Job before applying it again. The deletes need a quorum of etcd, not every member: one member not yet answering after the restore is fine as long as the Job completes. context deadline exceeded from every container means no quorum: bring etcd's members back before trying again. The Job runs etcdctl from etcd's own image, so it needs no egress.
Hold etcd down: kubectl -n port49-etcd scale statefulset/etcd --replicas=0.
Restore port49, without the HorizontalPodAutoscaler (the chart's autoscaling.enabled): a pod it added while the pods run would have a claim with no backup, and step 5 would find a pod with no line. Step 11 brings it back. Its pods start, so the node agent can restore their volumes, and with etcd down they cannot reach the store. The lab cannot hold its etcd down apart from the release (one namespace), so L89 (d) runs this command with its etcd up.
```
velero restore create <restore> --from-backup <backup> --include-namespaces port49 \
  --exclude-resources horizontalpodautoscalers.autoscaling --wait
```
Check that the volumes came back, by the restore's name. This prints the restore's phase, then one line per pod volume restore:
```
kubectl -n velero get restore <restore> -o jsonpath='{.status.phase}{"\n"}'
kubectl -n velero get podvolumerestores -l velero.io/restore-name=<restore> -o custom-columns=PHASE:.status.phase,POD:.spec.pod.name,VOLUME:.spec.volume
```
The phase must be Completed, and there must be a Completed line with volume data for every broker pod (<release>-broker-0, -1, ...) and every port49d pod (<release>-port49d-0, ...). A pod with no line did not have its volume restored. A line still New or InProgress is still copying: run the check again. Never go on to step 7 while the restore or any of these lines is not Completed. A broker that starts on an empty state_dir puts back none of the revocations the store restore lost. If one is missing or Failed, stop: no broker or port49d has reached the store, so start over from the first command with a new <restore>.

Scale both StatefulSets to 0, and wait until no port49 pod is left.

kubectl -n port49 scale statefulset/<release>-broker statefulset/<release>-port49d --replicas=0
kubectl -n port49 wait --for=delete pod -l app.kubernetes.io/name=port49 --timeout=3m

Bring etcd back: kubectl -n port49-etcd scale statefulset/etcd --replicas=3, and wait for it as step 1 does.
Then, with no broker and no port49d running, re-issue the revocations made since the backup (step 4 of Restoring etcd from a snapshot). The script leaves the brokers at the 0 it finds. Run it from the repository root.
```
deploy/helm/port49/scripts/revoke-offline.sh port49 <release> -- --principal user:bob@example.com
```
The brokers first, a broker whose broker.state_dir survived the restore first (the check in step 5 showed every broker's volume restored, so any of them does); then port49d.
```
kubectl -n port49 scale statefulset/<release>-broker --replicas=<brokers>
kubectl -n port49 rollout status statefulset/<release>-broker --timeout=10m
kubectl -n port49 scale statefulset/<release>-port49d --replicas=<port49d>
kubectl -n port49 rollout status statefulset/<release>-port49d --timeout=10m
```
What you see after step 9. port49d usually logs the session store was restored from a snapshot (its <accounting.spool_path>.restore.json is newer than the etcd copy) and port49_store_restore_detected fires. This restore did not use --bump-revision, so for the 5 minutes after port49d's read every token a user mints is refused token_spent (FAIL): the new timeline's revisions sit below the old high_header. Users see a failed login, and so does every retry until the window ends. The log line names the time (a token minted until <time> (5 minutes after this read)). Tell users to mint a new token after it: a token minted inside the window keeps its mint time, so it stays refused until the band ends (20 minutes after the read), and a user who pastes the same token again keeps failing. A fresh mint after the window is served. No restored token is accepted at any point. Break-glass is unaffected. The band itself lasts 20 minutes, but only tokens minted in its first 5 are refused.
Lift again what was lifted since (step 6 of Restoring etcd from a snapshot).
Bring back what step 4 left out, and resume what you suspended first. With a GitOps controller, resuming it does both (unmeasured, as above). Under Argo CD, put back the automated object you noted, exactly, children first, then any parent and the Application that syncs an ApplicationSet:
```
argocd app patch <app> --type merge -p '{"spec":{"syncPolicy":{"automated":<noted JSON>}}}'
```
Under Flux, resume each one you suspended:
```
flux resume helmrelease <name> -n <namespace>
flux resume kustomization <name> -n <namespace>
```
Without one, upgrade the release to the chart version it runs, which re-creates the HorizontalPodAutoscaler and changes nothing else:
```
helm upgrade <release> <chart> --version <version> -n port49 --reuse-values
```

With the chart's autoscaling.enabled, the HPA is not restored (step 4) until step 11.

Recovering from a restore at 0 replicas¶

A restore made with the StatefulSets at 0 replicas waits for pods that never start, and Velero restores one at a time, so it holds up every restore after it. No broker or port49d ran, so nothing reached the store. Restart Velero, delete that restore (<restore> here is its name), then start over from the first command above with a new <restore>. Measured on kind in L89 cell (d), run 42058930263, on (c)'s hung restore: after these commands the restore was gone and no restore was New or InProgress.

kubectl -n velero rollout restart deployment/velero
kubectl -n velero rollout status deployment/velero --timeout=5m
velero restore delete <restore> --confirm
kubectl -n velero wait --for=delete restore/<restore> --timeout=5m

Prefer restoring etcd from the etcdctl snapshot CronJob (backup-cronjob.yaml) with --bump-revision, as Restoring etcd from a snapshot gives it, and use Velero for the other volumes. A file-system copy of a live etcd member is crash-consistent at best, and the three members are copied at different moments; whether a busy member's database survives that is unmeasured. L89 measured one restore on an idle cluster.

The data-directory init containers refuse a volume whose files are not owned by uid 65532 (see the chart README), so what a restore does to ownership, group and mode matters. L89 (a) measured it on kind: owner, group and mode of state/, issuing-ca.key and the spool directory were unchanged, and both StatefulSets became ready with no fence refusal.