Runbooks¶
The chart renders these alerts with prometheusRule.enabled=true. They use only metrics the
manager exposes: up, controller_runtime_reconcile_total, controller_runtime_reconcile_errors_total,
openbao_writes_total{kind} and openbao_requests_total{connection,method,code}. The connection
label is the connection UID; observability shows how to map it to a
connection.
The shipped rules do not alert on objects or connections that are not Ready. The manager exports
openbao_resource_ready and openbao_connection_ready for that; example rules are in
observability.
OpenBaoConfigOperatorDown¶
No manager replica has been scraped as up for 5 minutes. Nothing is reconciled, so drift stays and new resources stay pending. OpenBao itself is not affected.
- List the manager pods:
kubectl -n <namespace> get pods -l app.kubernetes.io/name=openbao-config-operator. - For a crash loop, read the previous container's log:
kubectl -n <namespace> logs <pod> --previous, with a crash-looping pod from step 1. A failure to load the hash-key Secret or a missing RBAC rule shows at startup. - If the pods run but are not scraped, check the ServiceMonitor labels and that the scraper is
bound to the
<fullname>-metrics-readerClusterRole; without the binding the scrape answers 403.
The alert resolves when Prometheus scrapes at least one replica as up == 1.
OpenBaoConfigOperatorReconcileErrors¶
More than prometheusRule.reconcileErrorRatio (10%) of the reconciles of one controller have
failed for 15 minutes.
kubectl get <kind> -Aand look forSynced=False;kubectl describeshows the reason and message. troubleshooting.md explains each reason.- Many kinds failing at once usually means OpenBao or a connection is down: check OpenBaoConfigOperatorOpenBaoServerErrors and OpenBaoConfigOperatorOpenBaoDenied.
The alert resolves when the failed share of the controller's reconciles drops below the threshold.
OpenBaoConfigOperatorSteadyStateWrites¶
The operator has written objects of one kind to OpenBao in every 15-minute window for 30 minutes. At rest the operator only reads, so this is a fight.
- Find the objects:
kubectl get events -A --field-selector reason=Updatednames the path and the keys that differed (never their values). - Another writer (Terraform, a script, a person, a second cluster) changes the same path: give the
path one owner. Set
managementPolicy: Observeon the CR, or pause it with the annotationopenbaoconfig.io/paused: "true", while you sort it out. - The same key differs on every read with no other writer: the server normalises the value differently from the spec. Write the value as the server returns it and report the kind.
The alert resolves when openbao_writes_total for the kind stops increasing for 15 minutes.
OpenBaoConfigOperatorOpenBaoServerErrors¶
More than prometheusRule.openbaoErrorRatio (5%) of the requests of one connection have returned
5xx or failed in transport for 10 minutes.
- Check OpenBao:
bao status(sealed, standby, leader), its logs and its load balancer. code="error"is a transport failure: DNS, TLS (wrong CA bundle), a NetworkPolicy (networkPolicy.openbao) or OpenBao being down.- The client retries 5xx with backoff, so a short blip does not fire this alert.
The alert resolves when the share of 5xx and error responses of the connection drops below the
threshold.
OpenBaoConfigOperatorOpenBaoDenied¶
OpenBao has answered 403 to one connection for 15 minutes.
- Find the denied paths in the OpenBao audit log (filter on the connection's token accessor or
role), or in the
Syncedmessages of the resources that use the connection. - Extend the OpenBao policy of the connection identity (openbao-policy.md), or move the resource to a connection that may manage that path.
- A 403 on login means the auth role no longer accepts the identity (bound ServiceAccount, audience or namespace).
The alert resolves when the connection gets no 403 for 5 minutes.