perf: keep the access fingerprint for the request and read rule rows once per change #41

2026-11-10T21:22:51Z

james.maes commented

2026-11-10 21:22:51 +00:00

What

Two things this qbit did on every field of every record are now done once.

Access fingerprint. Core calls FieldAccessResolverInterface.getAccessFingerprint() before every field decision (FieldAccessHelper.getDecision, QQQ 4.1.0-RC.29 line 149): once per field of every record of a query result, and once per field of every table when the metaData document is built. SecuredDataResolver.getAccessFingerprint recomputed it each time (a JSON key in state(), a permission lookup for every table of the instance, a second JSON document, SHA-256). It is now kept in the request context (QContext.setObject) and returned while everything it is derived from is unchanged: the session object, its permission set (identity and size), the channel, the policy and classification versions, and the earliest expiry of an active break-glass grant. Nothing outlives the request.
Retention rules. Core asks AuditValueProtectorInterface.classificationOf for every audited field and every filter criterion (ValueBehaviorApplier.validateClassifiedCriteriaValues), and each call ran RetentionRules.active, one query. The active rules are now read once per change (SecuredDataRuntime.activeRetentionRules).

state() no longer serializes JSON for its per-user key.

Invalidation, and what several nodes do

Mechanism: QQQ core's cluster cache registry (ClusterCaches.shared(name, cache, clearAll, clearKey) and the memo-invalidate event on ClusterBus). The qbit depends on nothing new; qbit-redis is not needed, because the rows are tiny, each node can hold its own copy, and core already carries the invalidation to every node over whatever transport the application configured.
A rule write (insert, update, delete of a retention rule, policy, classification or break-glass grant) moves the version on the writing node after commit and announces it. This also fixes a gap: policy and classification versions were node-local, so a second node kept old policies until it restarted.
Safety net for a lost event: no node uses rule rows older than MAX_RULE_AGE (30 seconds) without reading them again. So a changed rule takes effect at commit on the node that changed it, when the event arrives on the others, and after at most 30 seconds anywhere.
A transaction that edits retention rules reads its own uncommitted edits through that transaction, as before; other threads see the change at commit.
Fail closed: a read that fails throws (Unable to read secured-data state) and leaves nothing in place of the rules. The answer is never an empty rule list because the rules could not be read, and never the rows read before (rulesThatCannotBeReadAreNeverAnEmptyAnswer). If the cluster bus cannot send, the change still applies locally and is logged; the other nodes read again within 30 seconds.

Measured

GG Forge's composed instance (222 tables, secured data and collaboration wired as Forge wires them, in-memory backend, platform administrator), one JVM, released 0.1.0-RC.4 against this branch, same machine, back to back:

Request	0.1.0-RC.4	This branch
`GET /metaData`, warm median	866.6 ms	125.3 ms
allocated per metaData request	930 MB	74 MB
queries per metaData request	96	34
list of 300 rows (25 fields)	1,566 to 1,734 ms	24 to 36 ms
list of 27 rows	174 to 230 ms	13 to 17 ms
queries per list / per count	23 / 13	8 / 5

Before the change a stack sampler put 98 % of a 300-row list in getAccessFingerprint. On GG Forge's dev pod the same requests take 2.8 s (metaData) and 10 to 17 ms per returned row.

Tests and gates

New SecuredDataRuleCachesTest (5 tests): one read for 200 questions; insert, update and delete each seen by the next question; a cluster event clears all three kinds of rules and a local change is announced; unreadable rules throw; maximum age; the fingerprint is the same object within a request and changes when a relevant permission is added in place or the permission set is replaced.
The existing fingerprint tests (fingerprintTracksOnlyRelevantState, break-glass expiry, channel) pass unchanged.
./mvnw -B verify on ab283c7 (Maven 3.9.11 from the wrapper, QRun-IO registry): 101 tests, 0 failures; Checkstyle 0; coverage checks met; BUILD SUCCESS.

Not changed

Core still calls the fingerprint once per field decision; with this change each call is a few comparisons. Calling it once per result set is a possible core follow-up.
validateClassifiedCriteriaValues in core could ask for the classification only when an operand fails to convert; with cached rules it no longer costs a query.

🤖 Generated with Claude Code

## What Two things this qbit did on every field of every record are now done once. 1. **Access fingerprint.** Core calls `FieldAccessResolverInterface.getAccessFingerprint()` before every field decision (`FieldAccessHelper.getDecision`, QQQ 4.1.0-RC.29 line 149): once per field of every record of a query result, and once per field of every table when the metaData document is built. `SecuredDataResolver.getAccessFingerprint` recomputed it each time (a JSON key in `state()`, a permission lookup for every table of the instance, a second JSON document, SHA-256). It is now kept in the request context (`QContext.setObject`) and returned while everything it is derived from is unchanged: the session object, its permission set (identity and size), the channel, the policy and classification versions, and the earliest expiry of an active break-glass grant. Nothing outlives the request. 2. **Retention rules.** Core asks `AuditValueProtectorInterface.classificationOf` for every audited field and every filter criterion (`ValueBehaviorApplier.validateClassifiedCriteriaValues`), and each call ran `RetentionRules.active`, one query. The active rules are now read once per change (`SecuredDataRuntime.activeRetentionRules`). `state()` no longer serializes JSON for its per-user key. ## Invalidation, and what several nodes do - Mechanism: QQQ core's cluster cache registry (`ClusterCaches.shared(name, cache, clearAll, clearKey)` and the `memo-invalidate` event on `ClusterBus`). The qbit depends on nothing new; qbit-redis is not needed, because the rows are tiny, each node can hold its own copy, and core already carries the invalidation to every node over whatever transport the application configured. - A rule write (insert, update, delete of a retention rule, policy, classification or break-glass grant) moves the version on the writing node after commit and announces it. **This also fixes a gap:** policy and classification versions were node-local, so a second node kept old policies until it restarted. - Safety net for a lost event: no node uses rule rows older than `MAX_RULE_AGE` (30 seconds) without reading them again. So a changed rule takes effect at commit on the node that changed it, when the event arrives on the others, and after at most 30 seconds anywhere. - A transaction that edits retention rules reads its own uncommitted edits through that transaction, as before; other threads see the change at commit. - Fail closed: a read that fails throws (`Unable to read secured-data state`) and leaves nothing in place of the rules. The answer is never an empty rule list because the rules could not be read, and never the rows read before (`rulesThatCannotBeReadAreNeverAnEmptyAnswer`). If the cluster bus cannot send, the change still applies locally and is logged; the other nodes read again within 30 seconds. ## Measured GG Forge's composed instance (222 tables, secured data and collaboration wired as Forge wires them, in-memory backend, platform administrator), one JVM, released 0.1.0-RC.4 against this branch, same machine, back to back: | Request | 0.1.0-RC.4 | This branch | |---|---|---| | `GET /metaData`, warm median | 866.6 ms | 125.3 ms | | allocated per metaData request | 930 MB | 74 MB | | queries per metaData request | 96 | 34 | | list of 300 rows (25 fields) | 1,566 to 1,734 ms | 24 to 36 ms | | list of 27 rows | 174 to 230 ms | 13 to 17 ms | | queries per list / per count | 23 / 13 | 8 / 5 | Before the change a stack sampler put 98 % of a 300-row list in `getAccessFingerprint`. On GG Forge's dev pod the same requests take 2.8 s (metaData) and 10 to 17 ms per returned row. ## Tests and gates - New `SecuredDataRuleCachesTest` (5 tests): one read for 200 questions; insert, update and delete each seen by the next question; a cluster event clears all three kinds of rules and a local change is announced; unreadable rules throw; maximum age; the fingerprint is the same object within a request and changes when a relevant permission is added in place or the permission set is replaced. - The existing fingerprint tests (`fingerprintTracksOnlyRelevantState`, break-glass expiry, channel) pass unchanged. - `./mvnw -B verify` on `ab283c7` (Maven 3.9.11 from the wrapper, QRun-IO registry): 101 tests, 0 failures; Checkstyle 0; coverage checks met; BUILD SUCCESS. ## Not changed - Core still calls the fingerprint once per field decision; with this change each call is a few comparisons. Calling it once per result set is a possible core follow-up. - `validateClassifiedCriteriaValues` in core could ask for the classification only when an operand fails to convert; with cached rules it no longer costs a query. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 1 commit 2026-11-10 21:22:51 +00:00

perf: keep the access fingerprint for the request and read rule rows once per change

maven / build (push) Successful in 1m32s

Details

build / maven (push) Successful in 1m32s

Details

ab283c79b9

Core asks the field access resolver for its fingerprint before every field
decision (FieldAccessHelper.getDecision), that is once per field of every
record of a result and of every table in the metaData document. Each call
recomputed it: a JSON key, a permission lookup for every table, another JSON
document and a digest. Core also asks for the retention rules of a field on
every audited field and every filter criterion, and each question was a
query.

- getAccessFingerprint keeps its value in the request context (QContext) and
  returns it while the session, its permission set, the channel, the policy
  and classification versions are unchanged and no active break-glass grant
  has expired. Nothing outlives the request.
- RetentionRules.active answers from the active rules the runtime read once
  (SecuredDataRuntime.activeRetentionRules). A rule write clears them after
  commit; a thread that edits rules inside a transaction reads its own edits.
- Policy, classification and retention changes are announced on QQQ's cluster
  bus (ClusterCaches, memo-invalidate), so every node reads again. Before,
  the versions were node-local and another node kept old rules until restart.
- No node uses rule rows older than 30 seconds without reading them again
  (the safety net for a lost event).
- A read that fails throws and replaces nothing: never an empty rule list.

Measured in one JVM with GG Forge's composed instance (222 tables, secured
data and collaboration wired, in-memory backend, platform administrator),
released 0.1.0-RC.4 against this change, same machine, back to back:
metaData request 866.6 ms to 125.3 ms (930 MB to 74 MB allocated, 96 to 34
queries); a 300-row list 1,566 ms to 24 ms; a 27-row list 174 to 230 ms down
to 13 to 17 ms; queries per list 23 to 8.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes marked the pull request as ready for review 2026-11-10 21:37:12 +00:00

james.maes merged commit 4cd0fd1f54 into develop

2026-11-10 21:37:14 +00:00

james.maes referenced this issue from a commit

2026-11-10 21:37:48 +00:00

Merge develop into release/0.1 for 0.1.0-RC.6 (#41 request-scoped fingerprint, cached rule rows)

james.maes referenced this pull request

2026-11-10 21:37:49 +00:00

Merge develop into release/0.1 for 0.1.0-RC.6 (#41 performance fix) #42

james.maes referenced this issue from a commit

2026-11-10 21:38:16 +00:00

Merge pull request 'Merge develop into release/0.1 for 0.1.0-RC.6 (#41 performance fix)' (#42) from merge/develop-into-release-0.1-rc6 into release/0.1