Commit Graph
5 Commits
Author SHA1 Message Date
james.maesandClaude Fable 5.2 d99db4738f fix: a rule read that asks for its own rows no longer recurses
maven / build (push) Successful in 1m17s
build / maven (push) Successful in 1m17s
Reading the rule rows a node keeps (classifications, active retention rules,
policies) is a query on a secured-data table, and other code runs inside that
query: a virtual field value provider or a post-query customizer that another
QBit or the host put on the table. When that code runs a query with filter
criteria, core validates the operands of classified fields and asks this QBit
for the classification of the field, which needs the rows that are being
read. The version of the rows was recorded only after the read returned, so
the nested question started the read again, and that read asked again:
StackOverflowError, and the same on every later request, since nothing was
ever recorded. One classification row was enough (core runs no value provider
for an empty result).

- System reads of the QBit's own tables (SystemWrites.query, the audit key
  lookup) name the stored fields of the table. Core fills a virtual field only
  for a query that selects it, so no value provider runs inside a system read.
- RuleRows holds one kind of rule rows and is used for all three kinds, which
  had the same shape. A question asked from inside a read is answered from the
  rows held when the read began (none before the first read) and never starts
  another read. The owner's monitor makes that answer private to the reading
  thread: every other thread waits and gets the rows the read returns.
- A read that fails records nothing and clears the reading mark, so the next
  question reads again. The rows held before are not given to a request in
  place of rows that could not be read.
- RetentionRules.active: a question asked from inside a transaction's read of
  its own rule edits is answered from the rules every thread shares instead
  of reading the edits again.

Who can be given the nested answer: only code on the reading thread, inside
SystemWrites.call, where the session is the system session and the resolver
answers FULL without looking at a classification. No field access decision
for a person is made from it; the request that started the read, and every
request on another thread, gets the rows the read returns.

Tests (SecuredDataRuleReloadTest): 9 of the 11 end in StackOverflowError
without the fix. They cover a value provider and a post-query customizer on
the classification, retention rule and policy tables, the answer a nested
question gets in both directions of a change, the session it is asked in, a
second thread waiting for the read, and failed reads (an exception and an
Error) being read again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 01:56:42 -06:00
james.maesandClaude Fable 5.2 ab283c79b9 perf: keep the access fingerprint for the request and read rule rows once per change
maven / build (push) Successful in 1m32s
build / maven (push) Successful in 1m32s
Core asks the field access resolver for its fingerprint before every field
decision (FieldAccessHelper.getDecision), that is once per field of every
record of a result and of every table in the metaData document. Each call
recomputed it: a JSON key, a permission lookup for every table, another JSON
document and a digest. Core also asks for the retention rules of a field on
every audited field and every filter criterion, and each question was a
query.

- getAccessFingerprint keeps its value in the request context (QContext) and
  returns it while the session, its permission set, the channel, the policy
  and classification versions are unchanged and no active break-glass grant
  has expired. Nothing outlives the request.
- RetentionRules.active answers from the active rules the runtime read once
  (SecuredDataRuntime.activeRetentionRules). A rule write clears them after
  commit; a thread that edits rules inside a transaction reads its own edits.
- Policy, classification and retention changes are announced on QQQ's cluster
  bus (ClusterCaches, memo-invalidate), so every node reads again. Before,
  the versions were node-local and another node kept old rules until restart.
- No node uses rule rows older than 30 seconds without reading them again
  (the safety net for a lost event).
- A read that fails throws and replaces nothing: never an empty rule list.

Measured in one JVM with GG Forge's composed instance (222 tables, secured
data and collaboration wired, in-memory backend, platform administrator),
released 0.1.0-RC.4 against this change, same machine, back to back:
metaData request 866.6 ms to 125.3 ms (930 MB to 74 MB allocated, 96 to 34
queries); a 300-row list 1,566 ms to 24 ms; a 27-row list 174 to 230 ms down
to 13 to 17 ms; queries per list 23 to 8.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 15:22:22 -06:00
james.maesandClaude Opus 5.5 470b0bd214 feat: retention and wipe with crypto-shredding; move to QQQ RC.6 (phase f)
Moves the pin to QQQ 4.1.0-RC.6 and removes the stand-ins RC.6 provides
(classification, client channels, table features, the UI API extension
point). Adds retention rules, wipe markers, legal holds, wipe runs and the
per-record-per-field audit key table; the wipe process (null with marker,
overwrite, keyed hash, delete, archive then wipe) runs in one transaction
per batch, destroying the audit keys with the field updates; the audit
protector and key source (AES-GCM wrapped DEKs, AAD-bound); the backups
warning; canary and wipe-proof tests.

Fixes a key-source bug that zeroed each DEK after returning it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:35:45 -05:00
james.maesandClaude Opus 5.5 3457481841 feat: app home and anomaly alerts (phase e)
One Secured Data home: KPIs, recent reveals, top revealers and reasons,
break-glass events, policy and classification changes and anomaly flags,
as full-width tables. Anomaly rules (3x daily median, distinct records per
hour, exact-match volume, break-glass) raise secured_data.anomaly and flag
the log row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:20:16 -05:00
james.maesandClaude Opus 5.5 0963b45467 feat: classification, policy, tier resolver and access log (phase b)
Tables secured_data_classification, _policy, _reveal_grant, _break_glass
and _access_log; SecuredDataClassifications; the tier resolver with
break-glass, table features, policy rows, channel caps and code defaults;
the asynchronous access log; enumeration guardrails (rate limit, miss and
distinct-value alerts, minimum entropy). Builds on QQQ RC.5 with marked
stand-ins for the unreleased fieldaccess API and a reflective bridge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:03:57 -05:00