test(authentik): fix slow contract stack startup in CI (Postgres JIT) #65

Merged

james.maes merged 3 commits from ci/authentik-ci-stability into develop 2026-11-07 00:08:28 +00:00

Conversation 0 Commits 3 Files Changed 5

+155 -40

james.maes commented 2026-11-06 18:00:15 +00:00

Owner

Copy link

Copy source

Makes the Authentik contract-test stack reliable on busy shared Gitea runners (blocked PR #62; failing runs 5754, 6277, 6363).

Root cause

Postgres JIT. Authentik validates every policy binding in a blueprint by loading PolicyBindingModel.select_subclasses(), a ~20-way LEFT JOIN. On a fresh database the planner estimates it at cost ~1e8, far above the JIT thresholds, so Postgres JIT-compiles it on every call: 7.6 s instead of 77 ms at 0.5 CPU (EXPLAIN ANALYZE: JIT 7.585 s). While the stack is starting, the Postgres container sits at its CPU cap and the worker is idle.

The fixtures' nested metaapplyblueprint chain (kof22-qqq-tokens -> bootstrap -> oauth2-apps) validates and applies oauth2-apps 8 times for every tokens apply. On a loaded runner that one apply ran past the worker's 10-minute task limit (run 6363: TimeLimitExceeded inside that SELECT). It was killed and retried, so the stack never became ready. Long transactions also widened the window for the concurrent-apply deadlock seen in 6277.

Changes (tests only)

• AuthentikTestStack: start Postgres with -c jit=off (keeping the Testcontainers default fsync=off).
• Waits on results the worker delivers (emails, notifications, events) share a documented 3-minute budget (ASYNC_TIMEOUT); blueprint re-applies get 10 minutes (APPLY_TIMEOUT). The waits still return as soon as the condition holds. The 60-second negative observation window is unchanged.
• From the earlier commits on this branch: drain blueprint work and then re-apply errored kof22 blueprints one at a time, in dependency order; print the task queue when a wait times out; BlueprintReapplyIT restores the seed state in finally.

No test is skipped, no assertion is weakened, and the ITs stay in the default build.

Evidence

All 108 ITs pass locally with every stack container capped at 0.5 CPU, which reproduces CI's 2m45s server start. Blueprint startup drops from >14 min (where it was still running when I stopped it) to ~1.5 min. BlueprintReapplyIT drops from 228 s (idle CI) to 34 s.

Follow-up for the owner: production Postgres (CNPG) for Authentik probably also has jit=on (the PG default), so the same blueprint applies are likely slow there too.

🤖 Generated with Claude Code

Makes the Authentik contract-test stack reliable on busy shared Gitea runners (blocked PR #62; failing runs 5754, 6277, 6363). ## Root cause Postgres JIT. Authentik validates every policy binding in a blueprint by loading `PolicyBindingModel.select_subclasses()`, a ~20-way LEFT JOIN. On a fresh database the planner estimates it at cost ~1e8, far above the JIT thresholds, so Postgres JIT-compiles it on every call: **7.6 s instead of 77 ms** at 0.5 CPU (`EXPLAIN ANALYZE`: JIT 7.585 s). While the stack is starting, the Postgres container sits at its CPU cap and the worker is idle. The fixtures' nested `metaapplyblueprint` chain (kof22-qqq-tokens -> bootstrap -> oauth2-apps) validates and applies oauth2-apps 8 times for every tokens apply. On a loaded runner that one apply ran past the worker's 10-minute task limit (run 6363: `TimeLimitExceeded` inside that SELECT). It was killed and retried, so the stack never became ready. Long transactions also widened the window for the concurrent-apply deadlock seen in 6277. ## Changes (tests only) - `AuthentikTestStack`: start Postgres with `-c jit=off` (keeping the Testcontainers default `fsync=off`). - Waits on results the worker delivers (emails, notifications, events) share a documented 3-minute budget (`ASYNC_TIMEOUT`); blueprint re-applies get 10 minutes (`APPLY_TIMEOUT`). The waits still return as soon as the condition holds. The 60-second negative observation window is unchanged. - From the earlier commits on this branch: drain blueprint work and then re-apply errored kof22 blueprints one at a time, in dependency order; print the task queue when a wait times out; `BlueprintReapplyIT` restores the seed state in `finally`. No test is skipped, no assertion is weakened, and the ITs stay in the default build. ## Evidence All 108 ITs pass locally with every stack container capped at 0.5 CPU, which reproduces CI's 2m45s server start. Blueprint startup drops from >14 min (where it was still running when I stopped it) to ~1.5 min. BlueprintReapplyIT drops from 228 s (idle CI) to 34 s. Follow-up for the owner: production Postgres (CNPG) for Authentik probably also has `jit=on` (the PG default), so the same blueprint applies are likely slow there too. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 2 commits 2026-11-06 18:00:16 +00:00

test(authentik): print the worker task queue on contract-test timeouts; restore seed state in BlueprintReapplyIT ... 2f6a83db27

Most contract-test waits depend on Authentik worker tasks (blueprint
applies, recovery/invitation emails, notification rules). When a wait
times out, print the global task status and the running, queued,
warning and error tasks with their logs, so CI shows whether the worker
was stuck, backlogged or failing.

BlueprintReapplyIT now restores the seed groups and the service-account
role's view_user permission in a finally block. A failed re-apply used
to leave those edits behind and cascade into unrelated failures
(QbitReadContractIT.groups saw "authentik.*, cms.publish";
TokenContractIT got 403 on /core/users/).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(test): drain Authentik's first-boot blueprint work before re-applying ours ... 7adc25bf7e

Root cause of the flaky contract stack (diagnostics from run 6277): on
first boot the worker (2 threads) applies ~40 built-in blueprints plus
the three kof22 fixtures concurrently. On a slow shared runner ours can
validate before the built-in objects they reference exist (managed
OAuth2 scope mappings, default flows) and end in "error", which
Authentik never retries for an unchanged file; concurrent applies also
deadlocked in Postgres. The earlier throttled re-apply made it worse:
it enqueued three applies every 30s regardless of pending work (69
queued apply_blueprint tasks, applies running for 6+ minutes), and the
backlog then starved the tests' own worker tasks (blueprint applies,
recovery/invitation emails, notification rules) and held locks that
timed out plain API writes.

Now startup waits until no blueprint discovery/apply task is queued or
in progress, then re-applies the first non-successful kof22 blueprint
in dependency order, one at a time, never enqueueing while an apply is
pending. The startup budget is 15 minutes (with a comment) because a
cold shared runner can need well over 8 minutes for the built-in storm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

james.maes added 1 commit 2026-11-06 23:36:52 +00:00

fix(test): turn off Postgres JIT in the contract stack ... d3f334cd42

Root cause of the slow/failed Authentik contract stack on busy CI runners:
Authentik validates every policy binding in a blueprint by loading
PolicyBindingModel.select_subclasses(), a ~20-way LEFT JOIN whose planner
estimate on a fresh database (cost ~1e8) is far above Postgres' JIT
thresholds. Postgres JIT-compiled it on every call: 7.6 s instead of 77 ms
at 0.5 CPU, with the Postgres container pegged and the worker idle.
The fixtures' nested metaapplyblueprint chain (tokens -> bootstrap ->
oauth2-apps) runs oauth2-apps 8 times per tokens apply, so one apply
outran the worker's 10-minute task limit, was killed and retried, and
the stack never became ready (runs 5754, 6277, 6363).

With jit=off the full contract suite passes locally with every
container capped at 0.5 CPU; blueprint startup drops from >14 minutes
to ~1.5 minutes and BlueprintReapplyIT from 228 s to 34 s.

Also give waits on worker-delivered results (emails, notifications,
events) a shared 3-minute budget and blueprint re-applies 10 minutes;
the waits still return as soon as the condition holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

james.maes changed title from test(authentik): fix flaky contract stack startup in CI to test(authentik): fix slow contract stack startup in CI (Postgres JIT) 2026-11-06 23:37:15 +00:00

james.maes merged commit 914b3ac58f into develop 2026-11-07 00:08:28 +00:00

james.maes deleted branch ci/authentik-ci-stability 2026-11-07 00:08:30 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

Architecture review: coupling, cohesion, UI API

bug

Something isn't working

dependencies

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

enhancement

New feature or request

github_actions

Pull requests that update GitHub Actions code

good first issue

Good for newcomers

help wanted

Extra attention is needed

invalid

This doesn't seem right

java

Pull requests that update java code

question

Further information is requested

release:rc

Merging this develop PR cuts the next release candidate

secuurity

wontfix

This will not be worked on

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

forge-recovery (Forge Recovery) james.maes (James Maes)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-authentik#65