Status and events¶
The exact status contract of every kind: conditions, reasons, status fields, events and the
columns kubectl get prints. Facts on this page come from api/common/v1alpha1/types.go,
api/core/v1alpha1, internal/engine/reconciler.go and internal/controller/core. For what to do
about a failing resource step by step, use troubleshooting; for
metrics and alert rules, use observability.
Who writes which status¶
| kind | status written by | conditions |
|---|---|---|
every managed kind (sys, auth, secrets, identity, OpenBaoEndpoint, AccessGrant, AccessLayout) |
the generic engine (internal/engine) |
Ready, Synced |
AccessLevel |
its own reconciler (internal/controller/access/accesslevel.go); it writes nothing to OpenBao |
Ready, Synced |
OpenBaoServer |
the server controller (internal/controller/core/server_controller.go) |
Ready |
OpenBaoConnection, ClusterOpenBaoConnection |
the connection controller (internal/controller/core/connection_controller.go) |
Ready |
Every status is written with a merge patch, and only when it changed. The engine and AccessLevel
use the field manager openbao-config-operator; the server and connection controllers use the
manager's default field manager.
Conditions of managed kinds¶
| condition | True means |
False means |
|---|---|---|
Ready |
the OpenBao object matched the spec at the end of the last pass that reached OpenBao | the object is missing, differs, cannot be reached or must not be touched; the reason says which |
Synced |
the last pass completed | the last pass failed or is waiting; the reason and message say why |
Each condition carries observedGeneration, the metadata.generation it describes. Messages
name paths and the keys that differed, never values.
observedGeneration¶
- Every condition update also sets
status.observedGenerationto the current generation, whether the pass succeeded or failed.status.observedGenerationalone therefore shows that the operator has seen a spec, not that it applied it. - A spec is applied when
ReadyandSyncedare bothTrueand both conditions'observedGenerationequalmetadata.generation. The engine uses the same test (engine.Reconciled) to decide that a pass is a drift check of a settled resource. - A failed pass with an error leaves an existing
Readycondition as it was. A resource can therefore showReady=True(for an earlier generation) andSynced=Falsewhile OpenBao refuses a change. - Annotation changes do not bump the generation. Only
openbaoconfig.io/pausedandopenbaoconfig.io/allow-destructive-deletestart a pass when they change.
Reasons¶
The retry column is what the engine does after the pass. "Drift interval" means
spec.driftInterval, or 15 minutes for access-critical kinds and 1 hour for the rest, each with
±20% jitter. "Backoff" without a cap is controller-runtime's exponential error backoff.
| reason | Ready |
Synced |
meaning | retry | what to do |
|---|---|---|---|---|---|
Available |
True |
the object matches the spec | drift interval | nothing | |
ReconcileSuccess |
True |
the pass completed | drift interval | nothing | |
Observed |
True |
True when equal |
managementPolicy: Observe; the object exists and was read; Synced is Drifted when it differs |
drift interval | nothing |
Drifted |
False |
managementPolicy: Observe; the object differs from the spec in the keys named in the message |
drift interval | change the spec or OpenBao, or switch to Manage to let the operator converge it |
|
NotFound |
False |
False |
creationPolicy: Adopt, or managementPolicy: Observe, and the object does not exist |
drift interval | check the name, mount path and spec.namespace; use CreateOrAdopt to create it |
Conflict |
False |
False |
another resource or cluster owns the path, or creationPolicy: Create found an object it does not own; the claim in status is dropped |
drift interval | give the path one owner, or set Adopt or CreateOrAdopt to take an unowned object over |
ConnectionNotReady |
False |
False |
the connection does not exist, is being deleted or is not Ready for its current generation |
2 s, growing to 1 min; a connection change re-queues at once | check the connection's own reason |
ConnectionDenied |
False |
False |
the ClusterOpenBaoConnection does not select this namespace, or the OpenBaoServer no longer admits the OpenBaoConnection |
drift interval | allowedNamespaces and the namespace labels |
ReferenceNotReady |
False |
False |
a referenced Secret, key or resource is missing or not Ready, the OpenBao namespace does not exist yet, or OpenBao has no route for the path yet (the mount is missing) |
2 s, growing to 30 s; a change of a referenced Secret, of the Plugin behind a mount's pluginRef, of the AuthMount behind a Policy's accessorRefs or of a cited AccessLevel re-queues at once; other referents are picked up by the backoff |
create the referent named in the message, or wait for the OpenBaoNamespace or mount resource that creates it |
PermissionDenied |
False if no Ready yet |
False |
OpenBao answered 403: the connection's policy does not allow the request | backoff | extend the OpenBao policy of the connection identity (OpenBao policy) |
ReconcileError |
False if no Ready yet |
False |
OpenBao or Kubernetes returned another error | backoff | read the message; a 400 is usually an invalid value (common OpenBao errors) |
InvalidSpec |
False |
False |
the spec, or a Secret it references, passed the schema but can never work as written, such as a spec.namespace that starts with root or a payload that is not a JSON object |
none until the spec or a referenced Secret changes | fix the spec or the Secret |
RootOnlyKind |
False |
False |
a root-only kind (Audit, AuditRequestHeader, RateLimitQuota, QuotaConfig, Plugin, CORSConfig, UIHeader, Logger, RaftAutopilot) resolves to a child OpenBao namespace; refused before any OpenBao call |
none until the spec changes | use a connection based in the root namespace and no spec.namespace |
Unsupported |
False |
False |
the server cannot serve the kind, for example the plugin, the mount or raft storage is missing | drift interval | mount the plugin or engine, or remove the resource |
PluginNotRegistered |
False |
False |
a mount needs a plugin that is not in the server's catalog | drift interval | register the plugin (Plugin kind) |
NamespaceChanged |
False |
False |
the object was acquired in one OpenBao namespace and the connection now resolves to another; nothing is written | drift interval | restore the connection's namespace, or recreate the resource (NamespaceChanged) |
ServerChanged |
False |
False |
the object was acquired on another OpenBao cluster than the one the connection reaches now; nothing is written or deleted | drift interval; when detected right before a create, only Synced is set and the pass is retried with backoff |
restore the connection, or release the resource (architecture) |
OutOfSync |
False |
False |
a write succeeded but the read after it returned different values | backoff from 2 s, capped at the drift interval | the message names the keys; write them as OpenBao returns them, or report the kind |
WriteBudgetExceeded |
False if no Ready yet |
False |
the server's write budget (default 60 rewrites per minute) is spent; this rewrite waits | when the budget frees, plus a random delay of up to 1 min | nothing; investigate if it persists (observability) |
DeleteGuarded |
False, except during deletion |
False |
deleting, or recreating after a create-only change, a destructive kind needs the annotation openbaoconfig.io/allow-destructive-delete: "true"; a deletion in a terminating Kubernetes Namespace is orphaned instead (Orphaned event) |
drift interval, or no requeue during deletion; the annotation change re-queues at once | set the annotation if the data may go, or set deletionPolicy: Orphan to keep it (stuck finalizers) |
CleanupBlocked |
False |
deletionPolicy: Delete could not delete the OpenBao object; the finalizer stays |
backoff | the message; usually the connection or its credential is gone (stuck finalizers) | |
ReconcileInProgress |
False |
OpenBao runs an asynchronous operation, such as a namespace deletion | every 10 s | wait | |
SecretNotOwned |
False |
False |
the output Secret named in writeSecretRef exists and is not controlled by this resource; nothing is written to OpenBao |
drift interval | rename writeSecretRef or delete the foreign Secret |
OutputNotDelivered |
False |
False |
a value OpenBao returned once is not in the output Secret | 2 s, growing to 30 s | the message names the Secret and how to rotate |
Paused |
False |
openbaoconfig.io/paused: "true" on the resource, its Kubernetes Namespace or its OpenBaoServer; nothing is read, written or deleted, and a deletion keeps the finalizer |
on resume: at once for the resource's own annotation, within about 1 min for a Namespace or server annotation | remove the annotation | |
InUse |
False |
AccessLevel only: its deletion waits while an AccessGrant or AccessLayout cites it (status.citedBy) |
on change of a citing resource | change or delete the citing resources |
A blank cell means the pass does not change that condition. Orphaned is an event reason only
(see events).
Conditions of connections and servers¶
OpenBaoServer, OpenBaoConnection and ClusterOpenBaoConnection have only the Ready
condition. Both controllers probe every 1 to 1.2 minutes and on changes of the objects
they watch, and write the status only when it changed.
| kind | reason | meaning | what to check |
|---|---|---|---|
OpenBaoServer |
Available |
reachable, initialised and unsealed | nothing |
OpenBaoServer |
Unreachable |
sys/seal-status or sys/health failed: DNS, network, TLS or the address |
the address, the NetworkPolicy egress to OpenBao, the CA bundle (x509 in the message) |
OpenBaoServer |
Sealed |
the server is sealed or not initialised | unseal OpenBao; resources wait meanwhile |
OpenBaoServer |
ReferenceNotReady |
the CA bundle Secret, ConfigMap or key does not exist | the caBundle reference |
OpenBaoServer |
InvalidSpec |
allowedNamespaces is not a valid selector, an http:// address without allowInsecureHTTP, or another unusable value |
the message |
| connection | Available |
the server is Ready, admits the connection, and login and auth/token/lookup-self succeeded |
nothing |
| connection | ServerNotReady |
the OpenBaoServer is missing or not Ready for its generation; the message carries the server's reason |
kubectl get openbaoserver <name> and the server reasons above |
| connection | ConnectionDenied |
the server does not select the connection's namespace, an audience is not in allowedAudiences, or an audience is one the Kubernetes API server accepts |
the server's allowedNamespaces against the namespace labels, and allowedAudiences against the connection's audiences |
| connection | CredentialsNotFound |
a referenced Secret, key or token file does not exist | the names, keys and namespaces in spec.auth; for a file, the volume mount |
| connection | AuthenticationFailed |
the login or the token lookup failed | the message; the OpenBao role, the audience, the ServiceAccount, authNamespace (common OpenBao errors) |
| connection | ReferenceNotReady |
the connection's OpenBao namespace does not exist yet; retried from 2 s, growing to 30 s | bao namespace list in the parent namespace; the OpenBaoNamespace that creates it |
| connection | InvalidSpec |
the spec cannot be turned into a client configuration | the message |
Field reference for these kinds: connections.
Status fields¶
Common to every managed kind¶
| field | type | meaning |
|---|---|---|
conditions |
list of conditions, keyed by type |
Ready and Synced (see reasons) |
observedGeneration |
integer | the generation the operator last processed (see observedGeneration) |
path |
string | full OpenBao API path including the namespace, as last written; deletion uses it even after the spec changed. Recorded as a claim right before a create, then kept. Example: team-a/sys/policies/acl/app-read |
namespace |
string | absolute OpenBao namespace the object was written in; empty is the root namespace. A later mismatch gives NamespaceChanged |
serverID |
string | cluster id (from sys/health) of the OpenBao server the object was claimed or acquired on. A later mismatch gives ServerChanged |
acquired |
Created or Adopted |
how the operator came to own the object; empty until owned |
writeOnlyHash |
string | HMAC-SHA256 of the write-only inputs under a key derived from the operator's hash key and the resource's UID, group and kind; a change forces a write. It never reveals the values |
id |
string | identifier OpenBao generated (entity, group or alias ID, mount accessor), for kinds that have one; other resources reference it |
rotationNonce |
string | the resource's rotation nonce when one-time inputs (a root credential OpenBao rotates afterwards) were last sent |
Under managementPolicy: Observe the engine leaves acquired, path and namespace unchanged,
and serverID too while the resource holds a path.
Kind-specific fields¶
| kind | field | meaning |
|---|---|---|
AccessGrant |
roles |
Kubernetes auth role names, one per subject in spec order |
AccessGrant, AccessLayout |
children |
every OpenBao object the resource owns; with deletionPolicy: Delete exactly these are deleted |
AccessLevel |
citedBy |
<kind>/<name> of the resources that cite the level; the openbaoconfig.io/in-use finalizer holds a deletion while it is not empty |
AppRoleSecretID |
rolePath, expirationTime, retiredAccessors |
the role path, expiry of the current secret ID, accessors of replaced secret IDs still to destroy |
GroupMembership |
groupID, memberEntityIDs, memberGroupIDs |
the group and the members this resource added (it removes only those) |
KVSecret |
managedKeys, pathCreated |
key names written (never values); whether this resource created the secret, which decides whether a destructive delete removes the version history |
RandomSecret |
refreshNonce, pathCreated |
the spec.refreshNonce of the current value; as for KVSecret |
PKIIssuer |
keyID, csr |
the issuer's key ID; the CSR of an intermediate waiting to be signed |
SSHCA |
publicKey |
the CA's OpenSSH public key |
TransitKey |
latestVersion, rotatedNonce |
the key's latest version; the last applied spec.rotateNonce |
Connections and servers¶
| field | OpenBaoServer |
connections | meaning |
|---|---|---|---|
conditions |
yes | yes | Ready only |
observedGeneration |
yes | yes | the generation last probed or checked |
version |
yes | copied from the server | OpenBao version reported by sys/health |
sealed |
yes | copied from the server | true when sealed or not initialised; no connection to the server is Ready then |
standby |
yes | copied from the server | true when the probed node is a standby |
authenticated |
yes | the last login and auth/token/lookup-self succeeded |
|
lastProbeTime |
yes | yes | when a probe last changed the status |
Events¶
Events are written with the events.k8s.io/v1 API by the engine; the reporting controller is
<kind>-controller in lower case (for example policy-controller). Each event has a type, a reason
and an action. Notes name paths, owners and keys, never values. The connection, server and
AccessLevel controllers emit no events.
| type | reason | action | emitted when |
|---|---|---|---|
Normal |
Created |
Create |
the operator created the object: created <path> |
Normal |
Updated |
Update |
the operator wrote a difference: updated <path>: <keys>, or write-only inputs |
Normal |
Adopted |
Adopt |
an existing object was taken over under Adopt or CreateOrAdopt |
Normal |
Recreated |
Delete |
a destructive object was deleted to apply a create-only change (only with the destructive annotation) |
Normal |
Deleted |
Delete |
deletionPolicy: Delete removed the object |
Warning |
Conflict |
Acquire |
the resource entered Conflict (once per entry) |
Warning |
Conflict |
Delete |
deletion left the object in place: an older resource owns it, or its marker names another owner |
Warning |
ServerChanged |
Reconcile |
a pass stopped with ServerChanged |
Warning |
ServerChanged |
Delete |
deletion left the object on the server it was acquired on |
Warning |
NamespaceChanged |
Reconcile |
a pass stopped with NamespaceChanged |
Warning |
Drifted |
Observe |
an observed object differs from the spec, when the differing keys change |
Warning |
OutOfSync |
Update |
a write did not read back as written (once per entry) |
Warning |
DeleteGuarded |
Delete |
a destructive delete or recreate waits for the annotation (once per entry) |
Warning |
WriteBudgetExceeded |
Update |
a rewrite waits for the write budget (once per entry) |
Warning |
Orphaned |
Delete |
the Kubernetes Namespace is terminating and its connection is gone; the finalizer is released and the OpenBao object stays |
Warning |
Orphaned |
Delete |
the Kubernetes Namespace is terminating and a destructive resource with deletionPolicy: Delete lacks openbaoconfig.io/allow-destructive-delete; the finalizer is released and the OpenBao object stays |
Print columns¶
kubectl get shows these columns. Every kind is in the openbao category, so
kubectl get openbao -A lists all managed kinds.
| kinds | columns |
|---|---|
| every managed kind | Ready, Synced, Age, plus the extra columns below |
OpenBaoServer |
Ready, Address, Version, Sealed, Age |
OpenBaoConnection, ClusterOpenBaoConnection |
Ready, Server, Version, Age |
| kind | extra columns (-o wide only in brackets) |
|---|---|
OpenBaoEndpoint |
Path |
Audit, AuthMount, SecretsMount, Group |
Type |
Plugin |
Type, Version |
TransitKey |
Type, Version (status.latestVersion) |
SSHRole |
Key Type |
Logger |
Level |
RateLimitQuota |
Rate |
EntityAlias, GroupAlias |
Alias |
AppRoleSecretID |
Expires |
GroupMembership |
(Group) |
MFAMethod |
(Method ID) |
OIDCClient |
(Client ID) |
DatabaseConnection |
(Plugin) |
Conditions to alert on¶
The engine exports each resource's Ready condition as the gauge
openbao_resource_ready{kind,namespace,name,reason} and each connection's as
openbao_connection_ready{kind,namespace,name}; observability
has the rules.
| condition | why it matters | suggested for |
|---|---|---|
any connection Ready=False |
every resource of that connection stops converging | 10m |
OpenBaoServer Ready=False with Sealed or Unreachable |
every connection to that server stops | 5m |
Ready=False with Conflict, NamespaceChanged or ServerChanged |
needs a person: two owners, or a connection that moved | 15m |
Ready=False with PermissionDenied or ReconcileError |
a policy gap or a refused value; retried with backoff | 30m |
Ready=False with OutputNotDelivered or SecretNotOwned |
a one-time value did not reach its Secret | 15m |
Synced=False with WriteBudgetExceeded for a long time |
something keeps rewriting objects | 1h |
Synced=False with CleanupBlocked |
a finalizer blocks a deletion | 30m |
Paused, CleanupBlocked and WriteBudgetExceeded set only Synced, so they are not visible in
openbao_resource_ready (see observability).