perf(chain): normalized detail coverage table and sweep floors #14

2026-10-15T12:40:59Z

KofTwentyTwo commented

2026-10-15 12:40:59 +00:00

(Migrated from github.com)

Why

Production (kof22, 50k audits / 120k details) showed the chain writer as the top database CPU consumer:

lateDetails re-read the whole 24 h lookback every 30 s tick and probed the GIN array index once per detail: 18.4 s, 118k GIN probes, 1.78M buffer hits per tick, almost always returning nothing.
The coverage trigger's EXISTS (... && ...) ran as a cached generic plan that chose a sequential scan of worm_audit_chain for every inserted link (verified on prod: one seq scan per insert, 80 ms each and growing; 56k seq scans / 1.2B tuples read).
The unchained-header anti-join scanned every settled audit header each tick.

(The "two concurrent queries" seen in pg_stat_activity were one backend plus its parallel worker: the tick already takes pg_try_advisory_xact_lock before any query, so only one replica runs a tick.)

What

worm-audit:13: worm_audit_chain_detail (detail_id PK, chain_id), one append-only row per covered detail, written by the chain trigger in the same statement as the link, backfilled from the chain under the chain advisory lock. WORM row/truncate guards, SELECT/INSERT grants, no foreign keys. The purge deletes coverage before its links (worm_audit_reject() handles the new table).
The trigger now inserts the coverage rows and lets the primary key reject a detail covered twice (no EXISTS pre-check, no plan to go wrong).
lateDetails is a primary-key probe of the coverage table. Both sweeps read only ids above per-JVM sweep floors (highest ids seen at least settle horizon + 5 min before the last complete sweep); after restart and once an hour a tick ignores the floors, so a very late commit is still chained.
Verification counts coverage from the links' own detail_ids in one aggregate (no per-detail probes) and breaks if the coverage table disagrees with the chain. Link hash inputs, exports and the sealed-export verifier are unchanged.
Memory/H2: the writer records coverage rows itself. The table is a read-only QQQ table ("Detail Coverage") with help.
The GIN index stays for this release so an RC.3 writer running during a rolling deploy keeps an indexed probe.

Benchmark (synthetic prod-shaped data, PG 16)

Query	Before	After
late details, first tick after restart	181 ms (prod: 18.4 s)	37 ms
late details, steady state	same	0.27 ms
unchained headers	18 ms (prod: 45 ms)	0.11 ms
trigger per link	seq scan, prod 80 ms	PK insert

Tests

./mvnw verify green (103 tests, Checkstyle, JaCoCo), including PostgreSQL Testcontainers: trigger-written coverage, write-once guards, backfill after dropping the table, purge removes coverage, tamper detection; H2: writer-written coverage, floors advance only past settled ids, below-floor late detail chained by the hourly full sweep.

🤖 Generated with Claude Code

## Why Production (kof22, 50k audits / 120k details) showed the chain writer as the top database CPU consumer: - `lateDetails` re-read the whole 24 h lookback every 30 s tick and probed the GIN array index once per detail: **18.4 s**, 118k GIN probes, 1.78M buffer hits per tick, almost always returning nothing. - The coverage trigger's `EXISTS (... && ...)` ran as a cached generic plan that chose a **sequential scan of `worm_audit_chain` for every inserted link** (verified on prod: one seq scan per insert, 80 ms each and growing; 56k seq scans / 1.2B tuples read). - The unchained-header anti-join scanned every settled audit header each tick. (The "two concurrent queries" seen in `pg_stat_activity` were one backend plus its parallel worker: the tick already takes `pg_try_advisory_xact_lock` before any query, so only one replica runs a tick.) ## What - `worm-audit:13`: `worm_audit_chain_detail (detail_id PK, chain_id)`, one append-only row per covered detail, written by the chain trigger in the same statement as the link, backfilled from the chain under the chain advisory lock. WORM row/truncate guards, SELECT/INSERT grants, no foreign keys. The purge deletes coverage before its links (`worm_audit_reject()` handles the new table). - The trigger now inserts the coverage rows and lets the primary key reject a detail covered twice (no EXISTS pre-check, no plan to go wrong). - `lateDetails` is a primary-key probe of the coverage table. Both sweeps read only ids above per-JVM sweep floors (highest ids seen at least settle horizon + 5 min before the last complete sweep); after restart and once an hour a tick ignores the floors, so a very late commit is still chained. - Verification counts coverage from the links' own `detail_ids` in one aggregate (no per-detail probes) and breaks if the coverage table disagrees with the chain. Link hash inputs, exports and the sealed-export verifier are unchanged. - Memory/H2: the writer records coverage rows itself. The table is a read-only QQQ table ("Detail Coverage") with help. - The GIN index stays for this release so an RC.3 writer running during a rolling deploy keeps an indexed probe. ## Benchmark (synthetic prod-shaped data, PG 16) | Query | Before | After | |---|---|---| | late details, first tick after restart | 181 ms (prod: 18.4 s) | 37 ms | | late details, steady state | same | 0.27 ms | | unchained headers | 18 ms (prod: 45 ms) | 0.11 ms | | trigger per link | seq scan, prod 80 ms | PK insert | ## Tests `./mvnw verify` green (103 tests, Checkstyle, JaCoCo), including PostgreSQL Testcontainers: trigger-written coverage, write-once guards, backfill after dropping the table, purge removes coverage, tamper detection; H2: writer-written coverage, floors advance only past settled ids, below-floor late detail chained by the hourly full sweep. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-worm-audit#14