Backup and restore¶
The custom resources in Git are the source of truth for OpenBao configuration. The operator rebuilds configuration from them. It cannot rebuild data, generated values or server-side IDs, so a complete recovery needs both the resources and an OpenBao snapshot.
What lives where¶
| state | lives in | recovered by |
|---|---|---|
| desired configuration (policies, roles, mounts, auth config, identities) | custom resources in Git | re-applying the resources |
| secret inputs (passwords, JWTs, client secrets) | Kubernetes Secrets referenced by <field>SecretRef |
your Secret source (ESO, sealed secrets, Velero) |
one-time outputs: AppRole secret IDs, OIDC client IDs and secrets, exported PKI private keys, RandomSecret copies |
Kubernetes Secrets written through writeSecretRef, owned by the resource |
Velero, or generating new ones |
| server-generated IDs and accessors (entities, groups, aliases, mounts) | OpenBao, copied into status.id and status fields |
an OpenBao snapshot; recreating gives new IDs |
data: KV values not set by a spec, RandomSecret values, transit, PKI and SSH key material, database root passwords after rotate-root, tokens and leases |
OpenBao only | an OpenBao snapshot only |
ownership and progress: status.acquired, status.path, status.namespace, status.writeOnlyHash |
resource status | not needed; see status is not restored |
members a GroupMembership added, children of an AccessGrant or AccessLayout |
resource status | lost with the status; see below |
the key of status.writeOnlyHash |
the hash-key Secret | Velero or a manual copy |
What to back up¶
- The custom resources: Git, or
kubectl get openbao -A -o yamlfor a point-in-time copy. - The Secrets the resources and connections reference.
- The output Secrets written through
writeSecretRef. - The hash-key Secret,
openbao-config-operator-hash-keyin the operator namespace. - OpenBao itself, with its own snapshot tooling (
bao operator raft snapshot savefor integrated storage).
The hash-key Secret¶
The Secret holds the HMAC key of status.writeOnlyHash. Losing it costs one write: each resource
with write-only inputs finds that its recorded hash no longer matches and writes once. It matters
only when the resources and their status survive, for example when the operator namespace is
deleted and recreated. When the status is gone as well, every such resource writes once anyway.
kubectl get secret openbao-config-operator-hash-key -n openbao-config-operator -o yaml > hash-key.yaml
Restore it before the manager starts. The manager creates a new key only when the Secret is absent.
Restore scenarios¶
OpenBao restored from a snapshot, resources intact¶
Restore the snapshot, unseal, and wait for the connections to become Ready. The resources keep
their status. On the next reconcile each resource reads its object and:
- writes it again if the object differs from the spec (changes made after the snapshot was taken);
- creates it again if it is missing and the resource had created it (
status.acquired: Created), even withcreationPolicy: Adopt. A resource that adopted the object (status.acquired: Adopted) and hascreationPolicy: AdoptreportsNotFoundinstead.
Objects created out of band after the snapshot are lost. One-time outputs generated after the snapshot, such as AppRole secret IDs, exist in Kubernetes Secrets but no longer in OpenBao: generate them again.
OpenBao rebuilt empty, resources intact¶
Re-initialise OpenBao and restore the connection identities first (auth method, role, policy), by
hand or with a bootstrap token. The operator then creates every object again. Server-generated
IDs change; resources that reference another resource (entityRef, mountRef) resolve the new
ID from the referent's status once it is Ready. Data is lost.
Kubernetes rebuilt, OpenBao intact¶
Install the operator, restore the hash-key Secret and the referenced Secrets, then apply the resources from Git. The new resources have no status, so the operator meets every object as an existing object it does not own yet:
creationPolicy |
result |
|---|---|
Create (default) |
Conflict for an existing object, except a configuration singleton |
CreateOrAdopt |
adopts the object, emits an Adopted event, sets status.acquired: Adopted |
Adopt |
adopts it; reports NotFound if it does not exist |
Set creationPolicy: CreateOrAdopt or Adopt for a restore. Each resource with write-only inputs
writes once, because its status has no hash yet.
Kinds with an ownership marker in OpenBao (entities, groups, namespaces) also compare the cluster
ID. Without --cluster-id, the cluster ID is the UID of the kube-system namespace, which changes
when the cluster is rebuilt, so these objects stop with Conflict ("carries the ownership marker
of another cluster") whatever the creationPolicy. Set the same --cluster-id through the chart's
extraArgs on the rebuilt cluster as on the old one; if the old cluster had none, set it to the old
kube-system UID.
Status is not restored¶
Velero and GitOps tools restore spec and metadata but not status by default. The operator does not need the status for most kinds: it re-acquires objects by adoption as above. Three kinds keep state in status that cannot be rebuilt:
GroupMembershiprecords the members it added (status.memberEntityIDs,status.memberGroupIDs) and removes only those. After a restore it adds the members in its spec again, but it no longer removes members it added earlier and that the spec no longer lists.AccessGrantandAccessLayoutrecord the objects they wrote instatus.children. After a restore they cannot remove children that their spec no longer produces. Remove those by hand.RandomSecretadopts the existing value when its status is empty and does not regenerate it, even ifspec.refreshNonceis set.
Velero¶
Back up the operator namespace (for the hash-key Secret) and every namespace that holds custom resources, with their Secrets:
velero backup create openbao-config \
--include-namespaces openbao-config-operator,team-a,team-b \
--include-cluster-resources=true
Points to check:
- Restore the CRDs, through the chart or
crds.yaml, before the resources. - Restore the hash-key Secret before the operator starts, or the operator creates a new key.
- Output Secrets written through
writeSecretRefcarry an owner reference to the resource's UID. A restored resource has a new UID. Check that the garbage collector has not removed the output Secrets. If an output Secret survives with the old owner, the operator refuses to overwrite it ("the Secret exists and is not controlled by this resource"); delete it so the operator writes it again. A missingAppRoleSecretIDSecret makes the operator generate a new secret ID. - Restored resources keep their finalizers. That is expected: the operator handles them.
Deletion policy and orphaned objects¶
deletionPolicy decides what deleting a resource does in OpenBao. Every kind defaults to Orphan:
the OpenBao object stays (ADR 0015). Delete is set
per resource. It removes the object, or resets a configuration singleton to its defaults; data holders
are removed only while the resource carries the annotation
openbaoconfig.io/allow-destructive-delete: "true". The kind catalogue lists these
destructive kinds; among them Entity, whose delete erases the MFA TOTP secrets enrolled for it. Kinds whose OpenBao object cannot be deleted reject Delete.
Without the annotation, a data holder with Delete in a live Kubernetes Namespace stays Terminating with
reason DeleteGuarded. In a terminating Namespace the operator leaves the object in OpenBao, records a Warning
event with reason Orphaned and removes the finalizer, so deleting a Namespace never destroys guarded data.
The operator adds the finalizer openbaoconfig.io/finalizer only when the effective policy is
Delete, and removes it when the policy changes to Orphan. On deletion with Delete it deletes
status.path in status.namespace. If the connection or a credential is missing, the finalizer
stays and Synced reports CleanupBlocked until the delete succeeds.
An orphaned object stays in OpenBao, unmanaged. Re-creating the resource adopts it again
(CreateOrAdopt or Adopt).
Uninstall and CRD deletion¶
The uninstall steps, including how to remove the operator and leave OpenBao untouched, are in the install guide.
With crds.keep=false, helm uninstall deletes the CRDs, and Kubernetes deletes every custom
resource with them. What happens in OpenBao then depends on timing: while a manager still runs, it
deletes every object whose policy is Delete, such as policies and roles. Once the manager is
gone, the finalizers block the deletion and the CRDs stay Terminating. Neither outcome is safe,
so keep crds.keep=true.