Load testing¶
This page describes the load test in test/load, the numbers it produced, the bottlenecks it found and how to
tune the operator for large object counts.
Method¶
hack/load/run.sh builds the manager binary once, then runs go test -tags load ./test/load/ for each object
count and kind set. Each run starts a fresh environment:
- envtest: a real kube-apiserver and etcd with every CRD from
config/crd/bases. - OpenBao: one
bao serveron raft storage with a single unseal key, so its data survives the restart in the outage scenario.bao server -devkeeps data in memory and would come back empty. - The manager binary from
./cmd/, run against envtest with--watch-namespaces=load,--leader-elect=false(one replica; lease renewals would count as Kubernetes writes in the steady window), both drift intervals at 1 minute,--max-concurrent-reconciles=4and the default--max-concurrent-reconciles-high-volume=8unless a run says otherwise. It authenticates with its own AppRole (design rule 9), not a root token.
The harness creates the objects with 16 parallel workers, follows their Ready and Synced conditions through
informers, scrapes the manager's /metrics every 2 s and reads the manager's RSS and CPU time and OpenBao's RSS
with ps. Goroutine
counts come from go_goroutines; the harness leaves pprof off.
Kind sets:
| Set | Kinds |
|---|---|
simple |
Policy and KVSecret, one Secret per KVSecret |
mixed |
Policy, KVSecret, PasswordPolicy, AuditRequestHeader, TokenRole, UserpassUser, AppRole, Entity, Group, TransitKey |
refs |
EntityAlias with entityRef to an Entity; aliases are applied 5 s before their entities |
members |
GroupMembership with entityRef and groupRef; memberships are applied first. Use a multiple of 3 for N |
tree |
an OpenBao namespace hierarchy, sized by a shape instead of N; see Namespace tree |
Scenarios, in order:
- ramp: create N objects; time from each create to
Ready=True,Synced=Truefor the current generation. - steady: three drift intervals with no change. OpenBao writes and Kubernetes writes must be zero.
- drift: change N/10
PolicyandKVSecretobjects in OpenBao and time their repair. Five otherKVSecretpaths get a key the operator does not own; their KV version counts the operator's writes to them exactly. - outage: stop OpenBao for 60 s, start and unseal it, and time until the connection and every object are healthy again.
- delete: delete every object (
deletionPolicy: Delete) and time until each is finalized; check OpenBao.
After each phase the harness lists auth/token/accessors, identity/entity/id and identity/entity-alias/id.
Only the managed Entity and EntityAlias objects may change them (design rule 2).
Run it¶
hack/load/run.sh -n "200 1000" -k "simple mixed"
hack/load/run.sh -n 1000 -k simple -b -1 # write budget off
hack/load/run.sh -n 201 -k members -s ramp,steady,delete
hack/load/run.sh -k tree -t "5x3x4 20x10x20" -s ramp,steady,drift,delete
Flags: -n object counts, -k kind sets, -t tree shapes, -c --max-concurrent-reconciles, -s scenarios
(an unknown name fails the run before anything starts), -d drift
interval, -b --write-budget-per-minute (-1 disables it), -o output directory (default
.tmp/night/loadtest). LOAD_MANAGER_BIN and LOAD_CRD_DIR reuse a prebuilt binary and its CRDs. Each run
writes results.json, timeseries.csv, objects.csv, events.txt, manager.log and bao.log.
.github/workflows/load.yml runs the tree set every Sunday and on demand (workflow_dispatch takes the shape,
scenarios, drift interval and --max-concurrent-reconciles). It defaults to 20x10x20 with
ramp,steady,drift,delete, uploads the run directory as the load-tree artifact and writes the main numbers to
the job summary. A scheduled failure opens an issue through report-failure.yml.
Limits of these numbers¶
- One laptop (darwin/arm64, 10 CPUs) runs envtest, OpenBao, the manager and the harness. Other work kept the
load average between 50 and 180 during these runs (recorded per run in
results.json), so latencies are pessimistic and vary between runs. Counts of requests, writes, reconciles and logins are exact. - No TLS, one OpenBao node, no network latency, no Kubernetes API priority and fairness limits.
- The 1 minute drift interval compresses time; production defaults are longer.
Results¶
Manager build of 2026-11-14 ~01:50. wb is --write-budget-per-minute; -1 disables the budget.
Ramp¶
| Run | All ready | Ready p50 / p95 / p99 | OpenBao requests | Reconciles | Peak RSS |
|---|---|---|---|---|---|
| simple, N=200, wb 60 (default) | 190.9 s | 64.2 / 185.9 / 190.0 s | 1262 (1062 GET, 200 PUT) | 761 | 124 MiB |
| simple, N=1000, wb off | 25.8 s | 12.8 / 23.9 / 24.7 s | 3500 (2500 GET, 1000 PUT) | 1000 | 153 MiB |
| mixed, N=200, wb off | 2.9 s | 0.9 / 2.5 / 2.7 s | 660 (440 GET, 220 PUT) | 200 | 125 MiB |
| mixed, N=1000, wb off | 7.7 s | 2.1 / 5.9 / 6.6 s | 3300 (2200 GET, 1100 PUT) | 1000 | 159 MiB |
| simple, N=5000, wb off | 91.6 s | 33.1 / 77.7 / 84.2 s | 19106 (14106 GET, 5000 PUT) | 6605 | 196 MiB |
The same harness on the build of ~01:25, which had no write budget, made N=200 ready in 3.2 s.
With the budget off, a create costs one reconcile (1.3 at N=5000), about 2.5 OpenBao reads and 1 write (read,
write, read back; KVSecret also reads the current version for check-and-set) and 3 Kubernetes PATCHes plus one
event. Throughput is bounded by the workers per kind: Policy (8 workers) and KVSecret (4 workers) together
reached 39 objects/s at N=1000 and 55 objects/s at N=5000 on the loaded host; the ten mixed kinds reached 130
objects/s.
Steady state¶
| Run | Reconciles/s (expected N/60 s) | OpenBao requests/s | Requests per 10 s bucket min / p50 / p95 / max | OpenBao writes | Kubernetes writes |
|---|---|---|---|---|---|
| simple, N=200 | 3.23 (3.33) | 3.22 | 0.5 / 2.9 / 6.3 / 6.3 | 0 | 0 |
| simple, N=1000 | 15.8 (16.7) | 15.8 | 0 / 13.8 / 34.3 / 34.3 | 0 | 0 |
| mixed, N=200 | 2.99 (3.33) | 3.27 | 0 / 1.2 / 9.9 / 9.9 | 0 | 0 |
| mixed, N=1000 | 15.2 (16.7) | 16.7 | 0 / 5.1 / 44.1 / 44.1 | 0 | 0 |
| simple, N=5000 | 81.3 (83.3) | 81.3 | 52.6 / 76.9 / 107.1 / 107.1 | 0 | 0 |
Every drift check costs one OpenBao read and no Kubernetes write. The rate stays below N/interval in the first interval after a ramp, because objects created together first come back between 0.8 and 1.2 intervals later. The ±20 % jitter spreads that cohort over 24 s; the 10 s buckets show the spread, which widens with every interval. Logins stayed at one per connection, and token accessors and entity aliases did not change in any run.
Drift storm¶
| Run | Tampered | Repair p50 / p95 / max | Writes | Writes after repair |
|---|---|---|---|---|
| simple, N=200 | 20 | 22.7 / 52.8 / 61.9 s | 20 | 0 |
| simple, N=1000 | 100 | 35.4 / 61.0 / 69.7 s | 100 | 0 |
| mixed, N=200 | 20 | 48.2 / 58.7 / 60.3 s | 20 | 0 |
| mixed, N=1000 | 100 | 42.1 / 63.1 / 72.7 s | 100 | 0 |
| simple, N=5000 | 500 | 39.9 / 66.2 / 75.7 s | 500 | 0 |
Repair time is the time to the object's next drift check, at most 1.2 drift intervals. Each tampered object is
written exactly once. A KVSecret path with a key the operator does not own was never rewritten.
OpenBao outage (60 s)¶
| Run | Objects unhealthy | Connection ready after restart | All objects ready after restart | Peak requests/s in recovery | Reconciles in recovery |
|---|---|---|---|---|---|
| simple, N=200 | 200 | 11.5 s | 41.2 s | 14 | 261 |
| simple, N=1000 | 1000 | 38.2 s | 68.3 s | 157 | 2734 |
| mixed, N=200 | 200 | 18.8 s | 48.9 s | 21 | 310 |
| mixed, N=1000 | 1000 | 20.2 s | 50.1 s | 66 | 1713 |
| simple, N=5000 | 5000 | 30.7 s | 61.0 s | 513 | 10149 |
No object was rewritten after the restart, and the operator did not log in again: its token was still valid.
Delete¶
| Run | All finalized | Finalize p50 / p95 |
|---|---|---|
| simple, N=200 | 3.6 s | 1.3 / 3.3 s |
| simple, N=1000 | 16.0 s | 8.0 / 14.4 s |
| mixed, N=200 | 1.3 s | 0.3 / 1.0 s |
| mixed, N=1000 | 7.6 s | 1.4 / 6.2 s |
| simple, N=5000 | 66.6 s | 24.4 / 59.8 s |
References¶
Referrers wait for their referents with a jittered backoff from 2 s to 30 s. On the build of ~01:25, 20
EntityAlias objects applied 5 s before their Entity objects were ready 0.4 to 6.6 s (p50 4.2 s) after the
entities were applied. On the ~01:50 build the refs run at N=200 could not measure this: 97 of 100 aliases stayed
in Conflict (bottleneck 2). The members set hits the same conflict.
On the build of ~04:00 (default write budget) both sets are clean: refs at N=200 made all 100 aliases ready
(OpenBao holds 101 entity aliases, the operator's own included), members at N=201 made all 67 memberships ready,
with no Conflict event in either run. Referrers were ready 4.8 s (p50) and 8.0 s (max) after their referents were
applied for refs, 5.8 s and 9.1 s for members.
Resources¶
The idle manager on that run used about 4,400 goroutines: 68 controllers (plugin kinds off), each with its workers, informers and queues. A default install now starts 74 controllers, one per built-in kind. The count stayed flat through every scenario and after deletion, so nothing leaks. RSS was about 55 MiB idle, 125 to 160 MiB with 1,000 objects and 232 MiB with 5,000. CPU peaked at 70 % of one core during the mixed N=1000 ramp; in steady state it was 7.5 % at N=1000 and 31 % at N=5000 with the 1 minute interval (81 drift checks/s).
Namespace tree¶
The tree set builds the OpenBao namespace hierarchy of a multi-tenant server. -t takes the shape
PARENTSxCHILDRENxKV[xROLESxPOLICIES] (roles and policies default to 3); the default 100x10x20 is 100 parent
namespaces with 10 children each: 1,100 namespaces, 32,000 objects and 23,100 mounts.
- Each parent namespace and the objects in it go through the root connection, with
spec.namespaceset to the parent (theroot-to-childmode ofexamples/live). - Each parent gets its own
OpenBaoConnectionwithnamespace: <parent>andauthNamespace: ""(thechild-connection-root-loginmode). It creates the child namespaces and the objects in them, so a run with 100 parents has 101 connections and 101 operator tokens. - Every namespace holds one
kubernetesAuthMountwith itsKubernetesAuthConfig(pinned to the envtest CA), KVSecretsMounts of typekv-v2, ROLESKubernetesAuthRoles and POLICIESPolicyobjects. - The harness applies everything at once, in the order of the Argo CD sync waves but without waiting between them: parent namespaces, connections, parent mounts and policies, parent auth configs and roles, child namespaces, child mounts and policies, child auth configs and roles. The ramp reports latency per level.
- Drift tampers with 50
Policyobjects, which stays under the default write budget of 60 rewrites a minute. - Delete goes level by level in reverse, each level after the one before is gone. Mounts, policies and roles use
deletionPolicy: Deletewith the destructive-delete annotation. Namespaces keep the defaultOrphan, andKubernetesAuthConfigcan only beOrphan. - After the ramp and after delete, the harness walks every namespace with the root token and counts the KV mounts, the kubernetes auth mounts and the policies. Token accessors must grow by one per connection and drop back after the connections are deleted.
Results¶
Apple M1 Max (darwin/arm64, 10 CPUs, 32 GB), load average 2 to 6, other envtest and OpenBao processes running
next to the harness. OpenBao 2.7.2 on raft, drift intervals 1 minute (30 s for 5x3x4), default write budget.
-c is --max-concurrent-reconciles; Policy and KubernetesAuthRole use the high-volume default of 8.
| Shape | Objects | -c |
All ready | Objects/s | Ready p50 / p95 / max | Manager peak RSS / goroutines / CPU | OpenBao peak RSS |
|---|---|---|---|---|---|---|---|
| 5x3x4 | 265 | 4 | 68.1 s | 3.9 | 3.5 / 67.0 / 68.0 s | 131 MiB / 3,982 / 11 % | 114 MiB |
| 20x10x20 | 6,400 | 4 | 393.2 s | 16.3 | 132.5 / 364.4 / 392.8 s | 206 MiB / 4,272 / 24 % | 457 MiB |
| 20x10x20 | 6,400 | 8 | 365.8 s | 17.5 | 123.4 / 338.7 / 365.8 s | 248 MiB / 4,538 / 24 % | 465 MiB |
Ready latency by level, 20x10x20 with -c 4 (mounts include the policies of the same namespace):
| Level | Objects | p50 | p95 | max |
|---|---|---|---|---|
| parent namespaces | 20 | 2.1 s | 3.6 s | 3.6 s |
| connections | 20 | 6.4 s | 71.8 s | 72.0 s |
| parent mounts | 480 | 37.8 s | 58.4 s | 375.8 s |
| parent auth | 80 | 8.1 s | 19.5 s | 20.7 s |
| child namespaces | 200 | 25.2 s | 76.0 s | 79.1 s |
| child mounts | 4,800 | 190.2 s | 369.4 s | 392.8 s |
| child auth | 800 | 37.9 s | 101.7 s | 118.0 s |
The other phases of 20x10x20 with -c 4:
- Steady: 104.7 reconciles/s (expected 106.3), 105.4 OpenBao requests/s, no OpenBao write and no Kubernetes write over three intervals. The manager used 29 % of a core and 228 MiB.
- Drift: 50 policies tampered, repaired after 24.4 s (p50), 58.3 s (p95), 64.8 s (max), each written once and none again in the two intervals after.
- Delete: all 6,400 objects gone after 330.7 s (370.1 s with
-c 8); finalize p50 64.5 s, p95 273.8 s. OpenBao then held the 220 orphaned namespaces with no KV mount, no kubernetes auth mount and no policy left in them; the auth configs went with their mounts. - Token accessors went from 2 to 22 after the ramp (one per connection) and back to 2 after delete. Entities and entity aliases did not change, and the goroutine count went back to its idle level of about 3,900.
OpenBao grew from 76 MiB idle to about 460 MiB for 220 namespaces and 4,620 mounts, about 1.7 MiB per namespace, and kept that memory after delete because the namespaces stay. The harness's OpenBao has no telemetry stanza.
The full 100x10x20 shape was not run on this laptop. Its ramp and delete are bound by OpenBao's mount writes (below) at about 12 to 14 mounts per second, so 23,100 mounts would take about 30 minutes to create and as long to delete, and OpenBao would need about 2 GiB by the same per-namespace growth.
Bottlenecks¶
The numbers above predate the fixes below.
- OpenBao serialises mount changes. Every enable, tune and disable takes the server-wide mount table lock
(
Core.mountsLock,internal/vault/mount.goin OpenBao), whatever the namespace, so mount creation is the ramp's ceiling and doubling the workers gained 7 %. Fixed in the operator's share: a mount create now sends its config with the enable, one locked write instead of an enable and a tune (a tune follows only for user lockout, which enable does not accept). - A tenant connection created before its namespace waited for the next probe (1 minute ±20 %), and every
child object behind it waited too. Fixed: namespace-not-found now reports
ReferenceNotReadyand retries after 2 to 30 s. - Objects created before their mount failed with errors. A write that gets 404 "no handler for route" (a role
before its auth mount, objects in a namespace whose
sys/is still mounting) logged an error with a stack trace on each retry, about 3,000 lines in the 20x10x20 ramp. Fixed: it is now aReferenceNotReadywait with one Info line per retry. Reads are unchanged: a 404 on read still means absent.
10x5x10 tree (1,150 objects, 660 mounts), -s ramp,delete, same laptop, before (main at a0a0b5b) and after:
| Before | After | |
|---|---|---|
| Ramp, all ready | 75.7 s (15.2 objects/s) | 40.2 s (28.6 objects/s) |
| Mounts per second | 8.7 | 19.1 |
| Connection ready p95 | 71.3 s | 3.6 s |
| OpenBao PUTs in the ramp | 3,403 | 1,857 |
| Failed reconciles | 1,121 | 0 |
| Manager error lines | 998 | 3 (startup cache warnings for kube-system namespaces) |
| Delete, all gone | 45.8 s | 46.1 s |
Before and after the engine fixes¶
The build of ~04:00 fixes the three bottlenecks below. Same harness and host, N=1000, default write budget (60),
60 s outage. Re-run with hack/load/run.sh to reproduce the numbers.
| Run | All ready (ramp) | Reconciles per object (ramp) | PATCHes per object (ramp) | Ready after restart | Peak OpenBao req/s in recovery | Reconciles in recovery |
|---|---|---|---|---|---|---|
| simple, before, wb 60 | at least 17 min (N/60); N=200 took 190.9 s | 3.8 at N=200 | 4.2 at N=200 | not run | not run | not run |
| simple, before, wb off | 25.8 s | 1.0 | 3.0 | 68.3 s | 162 | 2734 |
| simple, after, wb 60 | 12.1 s | 1.0 | 3.0 | 28.1 s | 100 | 1326 |
| mixed, before, wb off | 7.7 s | 1.0 | 3.0 | 50.1 s | 86 | 1713 |
| mixed, after, wb 60 | 9.6 s | 1.0 | 3.0 | 33.4 s | 114 | 1599 |
The three PATCHes per object are the claim recorded before the create, the finalizer and the final status. The peak is the highest 2 s sample between the restart and the last object ready. The recovery window held one status PATCH and one OpenBao read per object.
- Creates and spec changes no longer spend the write budget, so a first sync runs at full speed and no create
got
WriteBudgetExceeded. Only rewrites of objects the operator already holds spend it: drift repairs, changed defaults and new write-only inputs. The drift storm (100 tampered objects) now exceeds the budget: 36 (simple) and 29 (mixed) repairs waited, and the last repair landed after 122 s and 120 s instead of 70 s and 73 s. Each tampered object was still written exactly once. - A held-back rewrite keeps a fixed condition message and retries at a random point of the next window. A retry that is held back again patches nothing.
- Resources whose connection is not ready retry with a jittered backoff from 2 s to 60 s, and the connection watch
re-queues them when the connection turns ready. The first pass after that goes through a per-server ramp of 100
passes per second (burst 10), and its requests run without priority, so
OpenBaoServer.spec.rateLimitholds them to the drift share when it is set. Without the ramp, the watch released the whole backlog at once: 500 (simple) and 550 (mixed) req/s for about 2 s. With it, recovery ran at about 100 req/s for 10 s. The connection probe (1 minute ±20 %) still decides when the connection turns ready: 18 to 23 s after the restart in these runs. - The backoff makes more cheap retries while OpenBao is down: about 6 reconciles per object during the outage, against 1.1 to 1.8 before. They read only the cache: no OpenBao request and no status PATCH (about one PATCH per object for the whole outage, as before).
Steady state still wrote nothing to OpenBao or Kubernetes over three intervals, and delete took 12.9 s (simple) and 6.1 s (mixed).
Bottlenecks found¶
All three are fixed on the build of ~04:00 (see above).
- The write budget throttles creates.
--write-budget-per-minute(default 60) applies to every write, including creates and spec changes. A first sync of N objects therefore takes at least N/60 minutes: about 17 minutes for 1,000 objects and 83 minutes for 5,000. While waiting, each object re-reads OpenBao and patches its status every minute, so at N=200 the ramp cost 3.8 reconciles, 5.3 reads and 4.2 status patches per object instead of 1, 2.5 and 3. All waiting objects wake in the same 12 s band. - Shared collection paths conflict.
EntityAliasandGroupAliasrecord the collection path, so the ownership check lets only one of them ownidentity/entity-alias. At N=200, 97 of 100 aliases stayed inConflict.GroupMembershiphas the same problem withidentity/group. - Connection recovery waits a fixed 30 s. A resource whose connection is not ready requeues after exactly 30 s and does not watch the connection. During an outage the work queue backs up behind slow failing reconciles; when the connection turns not ready the backlog stalls within seconds, so the 30 s timers line up. After the restart, recovery peaked at 10 times the steady request rate at N=1000 (157 against 15.8 req/s) and 6 times at N=5000 (513 against 81 req/s), and every stalled object re-ran every 30 s until its connection was ready again: 10,149 reconciles for 5,000 objects. Objects were ready 50 to 68 s after OpenBao.
Steady state writes nothing (design rule 1), the operator logs in once per connection (design rule 2), and drift checks are
jittered. Errors use controller-runtime's per-item exponential backoff. On the ~01:50 build the connection-not-ready requeue was the one periodic wait without jitter. The
cache is scoped to --watch-namespaces, the client reads Secrets live and the Secret cache drops their data.
Tuning¶
--write-budget-per-minute: caps the rewrites per minute of objects the operator already holds (drift repairs, changed defaults, new write-only inputs). Creates and spec changes are not counted. With the default of 60, a drift repair or default change across N objects takes N/60 minutes.--max-concurrent-reconciles(default 4 per kind, 8 for high-volume kinds): creates scale with workers per kind until OpenBao or the Kubernetes API slows down. Raise it for a few kinds with thousands of objects. Every worker is a goroutine on each of the 74 built-in controllers, so do not raise it for everything.--drift-interval-criticaland--drift-interval-standard(the old--drift-intervalflag is removed): the steady OpenBao read rate is N divided by the interval, for example 5,000 objects at 10 minutes is 8.3 reads/s. Repair of out-of-band changes takes up to 1.2 intervals.OpenBaoServer.spec.rateLimit(unset means no limit): a token bucket shared by every connection to one server, with a smaller share for drift reads. Setqpsbelow what the OpenBao cluster can serve next to its other clients; at the default intervals the operator needs only a few requests per second.- OpenBao availability: after an outage, resources wait for the next connection probe (1 minute ±20 %). Then the connection watch re-queues them and each server takes them back at 100 per second, so expect up to about 72 s plus N/100 s before every object is ready again.
- Namespace trees: OpenBao creates and removes mounts one at a time, so 1,000 mounts take a little over a minute
on storage like this laptop's, whatever
--max-concurrent-reconcilessays. A tenant connection created before its OpenBao namespace exists reportsReferenceNotReadyand retries after 2 to 30 s.