Compare commits

...
Author SHA1 Message Date
james.maesandClaude Fable 5.2 1e2d6352d4 fix: an edit over HTTP is an edit of its row, not a new row
build / maven (push) Successful in 1m26s
maven / build (push) Successful in 1m26s
The table customizer finds the stored row of an update by its id, and
compared the two ids as they came: the id of a request over HTTP is text
(it is the id in the path), the stored id is a number. They were never
equal, so an edit made through the API or the UI had no stored row and was
validated and stamped as an insert. The uniqueness check of a
classification compared ids the same way. Both now compare the ids as text.

This is older than the standard dates, but they build on it: the customizer
gives a new row its create date, and a client that knows record versions
edits a table with a modify date by sending only the changed values and the
version it read.

What an edit over HTTP did before:
- A classification, policy or retention rule got a new create_date.
- Releasing a legal hold overwrote when and by whom it was placed (and,
  since the hold has one, its create date).
- Editing the reason of a legal hold without sending its status failed with
  a server error (the validation read the status of a row it did not have).
- Editing a classification while naming its table and field was refused as
  a second classification of that field.
- Editing a retention rule without sending every field was refused.

Tests (SecuredDataHttpEditTest, through the real middleware on the shipped
schema, a day after the rows were written): the reason of a hold is edited
with the changed value and the version, and the stale version is then
refused; a hold is released and keeps when and by whom it was placed and
its create date; a classification is edited naming its table and field; a
policy and a retention rule are renamed and keep their create dates; a row
with neither date is read and edited. All five fail without the fix. They
run in UTC: where the default zone of the JVM is not UTC, core refuses
every update that sends a version, which is core's to fix and is kept out
of these tests.

Notes for an upgrade (shipped DDL block and README): set a lock_timeout
before the ALTER statements and run the block again when one is cancelled;
on a large access log run its backfill UPDATE in ranges of id, or leave it
out, since rows without the two dates are read and written like any
others. Changelog: the fix, that an update naming the create date still
sets it, and that a wipe moves the record version of its retention rule.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 10:40:48 -06:00
james.maesandClaude Fable 5.2 3ddcf7ec9d feat: a create date and a modify date on every table
build / maven (push) Successful in 2m9s
maven / build (push) Successful in 2m9s
Every table of the QBit has a create date and a modify date: DATE_TIME, not
editable, with core's CREATE_DATE and MODIFY_DATE behaviors, in the columns
create_date and modify_date.

Seven tables did not have them: reveal grant, break-glass session, access
log, wipe run, wipe marker, legal hold and audit key. Their fields are named
createDate and modifyDate. Classification, policy and retention rule already
had create_date and modify_date, set by the table customizer. They keep those
field names, because a rename would change what a host reads and what their
audits record, and the two fields are now not editable and have the
behaviors.

Writes. Every write of the QBit is a core insert or update action, so the
dates are in the statement that writes the row. The table customizer sets
them, as it did for the three tables: a new row gets both, equal, from the
QBit's clock, and a changed row a new modify date. The access-log writer
still runs one insert per row and no other statement, lock or transaction.
The access log and wipe runs are only ever inserted, so their dates stay
equal.

Core uses a DATE_TIME field with the MODIFY_DATE behavior as the record
version of its table. That now holds for all ten tables: on an update core
sets the modify date itself, after the customizer has run, and an update
that sends the version it read is refused when the row has changed since.
Updates that send no version, the QBit's own among them, behave as before.

Schema. docs/ddl/secured-data-postgres.sql has a new last block that adds
the two columns to the seven tables: nullable, defaulting to the current
time, with the rows that were already there given the row's own time. No
CREATE TABLE statement is edited. An existing database runs that block
alone, before this release is deployed. The block rewrites no table, can be
run while the release before this one is running (its inserts name neither
column and get both from the default) and can be run again. This release
does not run without the columns. README: "Upgrading an existing database".

Unchanged. The dates have no classification, policy or assistant exposure of
their own: they are CONFIDENTIAL with their table and get the tier of its
other fields. Nothing signed or verified covers them: a wrapped data key is
bound to its table, field and record id, and the fingerprints of a log row
are made from search values. A changed policy or classification writes the
same core audit, field by field, and the same POLICY_CHANGE or
CLASSIFICATION_CHANGE row.

Tests. SecuredDataStandardDatesTest: every table the producer declares has
the two dates (the tables are taken from the instance, so a table added
later without them fails); the dates resolve to the tier of the other fields
of their table for five kinds of session; every writer sets both and only a
change moves the modify date (access-log writer, policy, classification,
legal hold, reveal, break-glass start, end and expiry on the memory backend;
audit key, wipe run, wipe marker and retention rule in the shipped schema);
the access-log writer runs exactly one insert per row; a changed policy and
classification write the audit and log rows they wrote before; rows written
before the columns existed are unchanged by the upgrade block, their wrapped
keys unwrap to the same data keys and their audit token verifies the same
value and no other. SecuredDataDdlTest: the shipped DDL gives every declared
table the two columns, the upgrade block adds them where they were missing,
an insert that names neither succeeds, and the block can be run twice. Seven
of the eight new tests fail without the change; the eighth is the record of
what a changed policy wrote before and passes on both sides.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 09:01:39 -06:00
james.maes 7d0e54eeca Merge pull request 'fix: a reveal and the access-log writer no longer wait for each other' (#45) from fix/reveal-access-log-deadlock into develop
build / maven (push) Successful in 1m43s
maven / build (push) Successful in 1m43s
2026-11-14 14:25:54 +00:00
james.maesandClaude Fable 5.2 a366e8b96e fix: a reveal and the access-log writer no longer wait for each other
build / maven (push) Successful in 2m34s
maven / build (push) Successful in 2m34s
A reveal, and the start, end and expiry of a break-glass session, hold the
runtime's monitor from their first check to their last write. Asking for a
classification (SecuredDataRuntime.classificationRows, and
activeRetentionRules) took the same monitor, and every thread asks: core
asks for every field it decides on and for every filter criterion of every
query, including the queries made inside this QBit's own writes.

So a thread that asked while holding another lock of this QBit could not be
answered during a reveal, and if the reveal needed that lock next, both
waited for ever. The access-log writer is such a thread. It holds the
anomaly counters while it inserts a row; the insert counts earlier rows and
writes an audit, both queries with criteria. A reveal inserts its own
access-log row.

    reveal:  runtime's monitor, then the anomaly counters
    writer:  the anomaly counters, then the runtime's monitor

Seen in a host on PostgreSQL: a person read a record with a masked field and
revealed it at once; the writer's tick drained the read's log rows at the
same moment. The request and the writer deadlocked (thread dump: the request
in SystemWrites.insertOne under RevealService.reveal, the writer in
SecuredDataRuntime.classificationRows under AuditAction.fetchIdFromName
under its own insert), and every later request blocked in
classificationRows. The same order existed between a reveal and the
resolver's monitor, and the key source's.

- The rule rows have a lock of their own (ruleLock), held only while they
  are answered or read. A read is a query made as the system and takes none
  of the QBit's other locks, so nothing that holds ruleLock waits for one.
- The runtime's monitor is left to what it was for: one reveal or
  break-glass change at a time.

Tests (SecuredDataLockOrderTest, on H2 in PostgreSQL mode with the audit
tables): a thread is parked inside a reveal, and inside the start of a
break-glass session, holding the runtime's monitor; the access-log writer
must drain its queue and another thread must be told a classification and
the active retention rules meanwhile, and then the parked operation must
finish. Both tests fail without the fix (neither thread finishes).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 03:39:54 -06:00
james.maes 7f1b5988af Merge pull request 'fix: a rule read that asks for its own rows no longer recurses' (#43) from fix/reentrant-rule-reload into develop
build / maven (push) Successful in 1m26s
maven / build (push) Successful in 1m26s
2026-11-14 08:23:02 +00:00
james.maesandClaude Fable 5.2 d99db4738f fix: a rule read that asks for its own rows no longer recurses
build / maven (push) Successful in 1m17s
maven / build (push) Successful in 1m17s
Reading the rule rows a node keeps (classifications, active retention rules,
policies) is a query on a secured-data table, and other code runs inside that
query: a virtual field value provider or a post-query customizer that another
QBit or the host put on the table. When that code runs a query with filter
criteria, core validates the operands of classified fields and asks this QBit
for the classification of the field, which needs the rows that are being
read. The version of the rows was recorded only after the read returned, so
the nested question started the read again, and that read asked again:
StackOverflowError, and the same on every later request, since nothing was
ever recorded. One classification row was enough (core runs no value provider
for an empty result).

- System reads of the QBit's own tables (SystemWrites.query, the audit key
  lookup) name the stored fields of the table. Core fills a virtual field only
  for a query that selects it, so no value provider runs inside a system read.
- RuleRows holds one kind of rule rows and is used for all three kinds, which
  had the same shape. A question asked from inside a read is answered from the
  rows held when the read began (none before the first read) and never starts
  another read. The owner's monitor makes that answer private to the reading
  thread: every other thread waits and gets the rows the read returns.
- A read that fails records nothing and clears the reading mark, so the next
  question reads again. The rows held before are not given to a request in
  place of rows that could not be read.
- RetentionRules.active: a question asked from inside a transaction's read of
  its own rule edits is answered from the rules every thread shares instead
  of reading the edits again.

Who can be given the nested answer: only code on the reading thread, inside
SystemWrites.call, where the session is the system session and the resolver
answers FULL without looking at a classification. No field access decision
for a person is made from it; the request that started the read, and every
request on another thread, gets the rows the read returns.

Tests (SecuredDataRuleReloadTest): 9 of the 11 end in StackOverflowError
without the fix. They cover a value provider and a post-query customizer on
the classification, retention rule and policy tables, the answer a nested
question gets in both directions of a change, the session it is asked in, a
second thread waiting for the read, and failed reads (an exception and an
Error) being read again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 01:56:42 -06:00
james.maes 4cd0fd1f54 Merge pull request 'perf: keep the access fingerprint for the request and read rule rows once per change' (#41) from perf/request-scoped-resolver-and-cached-rules into develop
build / maven (push) Successful in 1m55s
maven / build (push) Successful in 1m56s
2026-11-10 21:37:13 +00:00
james.maesandClaude Fable 5.2 ab283c79b9 perf: keep the access fingerprint for the request and read rule rows once per change
build / maven (push) Successful in 1m32s
maven / build (push) Successful in 1m32s
Core asks the field access resolver for its fingerprint before every field
decision (FieldAccessHelper.getDecision), that is once per field of every
record of a result and of every table in the metaData document. Each call
recomputed it: a JSON key, a permission lookup for every table, another JSON
document and a digest. Core also asks for the retention rules of a field on
every audited field and every filter criterion, and each question was a
query.

- getAccessFingerprint keeps its value in the request context (QContext) and
  returns it while the session, its permission set, the channel, the policy
  and classification versions are unchanged and no active break-glass grant
  has expired. Nothing outlives the request.
- RetentionRules.active answers from the active rules the runtime read once
  (SecuredDataRuntime.activeRetentionRules). A rule write clears them after
  commit; a thread that edits rules inside a transaction reads its own edits.
- Policy, classification and retention changes are announced on QQQ's cluster
  bus (ClusterCaches, memo-invalidate), so every node reads again. Before,
  the versions were node-local and another node kept old rules until restart.
- No node uses rule rows older than 30 seconds without reading them again
  (the safety net for a lost event).
- A read that fails throws and replaces nothing: never an empty rule list.

Measured in one JVM with GG Forge's composed instance (222 tables, secured
data and collaboration wired, in-memory backend, platform administrator),
released 0.1.0-RC.4 against this change, same machine, back to back:
metaData request 866.6 ms to 125.3 ms (930 MB to 74 MB allocated, 96 to 34
queries); a 300-row list 1,566 ms to 24 ms; a 27-row list 174 to 230 ms down
to 13 to 17 ms; queries per list 23 to 8.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-10 15:22:22 -06:00
james.maes eb98ec5fd0 Merge pull request 'build(deps): align with Gitea registry (parent 2.2.1, QQQ 4.1.0-RC.26)' (#39) from deps/align-gitea into develop
build / maven (push) Successful in 7m39s
maven / build (push) Successful in 4m16s
2026-11-07 18:16:02 +00:00
james.maesandClaude Opus 5.5 f09fe866ad build(deps): align with Gitea registry (parent 2.2.1, QQQ 4.1.0-RC.26)
build / maven (push) Successful in 3m34s
maven / build (push) Successful in 3m35s
pom.xml: parent -> 2.2.1
pom.xml: qqq.snapshot.version: 4.1.0-RC.15 -> 4.1.0-RC.26
pom.xml: qqq-frontend-next: 1.0.0-RC.14 -> 1.0.0-RC.38
pom.xml: added qqq.version=4.1.0-RC.26 (parent BOM import)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:08:54 +00:00
james.maes c4ebc6833c Merge pull request 'ci: pin shared workflow to v1' (#38) from ci/pin-v1-develop into develop
QRun-IO/ci-policy: native / fork-deny (push) Skipped
QRun-IO/ci-policy: native / dotnet-0 (push) Skipped
QRun-IO/ci-policy: ci / Scoped workflow [workflow 1e25bb5a243dfdcd95d4b9a4f780200ff8c8778a] (push) Successful in 13s
QRun-IO/ci-policy: native / native-result (push) Successful in 15s
QRun-IO/ci-policy: ci / head-profile (push) Skipped
QRun-IO/ci-policy: ci / repository-policy (push) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (push) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (push) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (push) Skipped
QRun-IO/ci-policy: ci / node-8 (push) Skipped
QRun-IO/ci-policy: ci / node-10 (push) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (push) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (push) Skipped
QRun-IO/ci-policy: ci / source-config-13 (push) Skipped
QRun-IO/ci-policy: ci / source-config-14 (push) Skipped
QRun-IO/ci-policy: ci / original-voyage (push) Skipped
QRun-IO/ci-policy: ci / original-cli (push) Skipped
QRun-IO/ci-policy: ci / ci-result (push) Successful in 11s
build / maven (push) Successful in 5m40s
QRun-IO/ci-policy: ci / fork-guard (push) Skipped
QRun-IO/ci-policy: ci / application-template-0 (push) Skipped
QRun-IO/ci-policy: ci / gradle-1 (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (push) Skipped
QRun-IO/ci-policy: ci / node-9 (push) Skipped
QRun-IO/ci-policy: ci / android-0 (push) Skipped
QRun-IO/ci-policy: ci / original-business (push) Skipped
QRun-IO/ci-policy: ci / original-mcp (push) Skipped
QRun-IO/ci-policy: ci / not-enrolled (push) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (push) Skipped
maven / build (push) Successful in 5m40s
2026-11-06 16:10:31 +00:00
james.maesandClaude Opus 5.5 2431b4723a ci: pin shared workflow to v1
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
build / maven (push) Successful in 3m34s
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: ci / Scoped workflow [workflow 1e25bb5a243dfdcd95d4b9a4f780200ff8c8778a] (pull_request) Successful in 7s
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-business (pull_request) Skipped
QRun-IO/ci-policy: ci / original-voyage (pull_request) Skipped
QRun-IO/ci-policy: ci / original-cli (pull_request) Skipped
QRun-IO/ci-policy: ci / ci-result (pull_request) Successful in 7s
maven / build (push) Successful in 3m34s
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: native / native-result (pull_request) Successful in 7s
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:13:29 +00:00
james.maes 9fb168b96d Merge pull request 'ci: use shared ci-workflows Maven workflow' (#37) from ci/ci-workflows into develop
QRun-IO/ci-policy: native / fork-deny (push) Skipped
QRun-IO/ci-policy: native / dotnet-0 (push) Skipped
QRun-IO/ci-policy: native / native-result (push) Successful in 4s
QRun-IO/ci-policy: ci / head-profile (push) Skipped
QRun-IO/ci-policy: ci / repository-policy (push) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (push) Skipped
QRun-IO/ci-policy: ci / node-8 (push) Skipped
QRun-IO/ci-policy: ci / node-10 (push) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (push) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (push) Skipped
QRun-IO/ci-policy: ci / source-config-13 (push) Skipped
QRun-IO/ci-policy: ci / source-config-14 (push) Skipped
QRun-IO/ci-policy: ci / original-voyage (push) Skipped
QRun-IO/ci-policy: ci / original-cli (push) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (push) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (push) Skipped
QRun-IO/ci-policy: ci / ci-result (push) Successful in 5s
build / maven (push) Successful in 2m46s
QRun-IO/ci-policy: ci / fork-guard (push) Skipped
QRun-IO/ci-policy: ci / Scoped workflow [workflow d20a578c7d818bc7ca32a2ac2c675da4e4f27385] (push) Successful in 6s
QRun-IO/ci-policy: ci / application-template-0 (push) Skipped
QRun-IO/ci-policy: ci / gradle-1 (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (push) Skipped
QRun-IO/ci-policy: ci / node-9 (push) Skipped
QRun-IO/ci-policy: ci / android-0 (push) Skipped
QRun-IO/ci-policy: ci / original-business (push) Skipped
QRun-IO/ci-policy: ci / original-mcp (push) Skipped
QRun-IO/ci-policy: ci / not-enrolled (push) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (push) Skipped
maven / build (push) Successful in 2m46s
2026-11-05 23:17:59 +00:00
CI e0459a99ff ci: rerun shared workflow
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: ci / Scoped workflow [workflow 1b3648d9421543350067e95e54959dfb471b26d5] (pull_request) Failing after 2s
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-business (pull_request) Skipped
QRun-IO/ci-policy: ci / original-voyage (pull_request) Skipped
QRun-IO/ci-policy: ci / original-cli (pull_request) Skipped
QRun-IO/ci-policy: ci / ci-result (pull_request) Failing after 1s
build / maven (push) Successful in 2m19s
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: native / native-result (pull_request) Failing after 1s
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (pull_request) Skipped
maven / build (push) Successful in 2m19s
2026-11-05 16:26:50 -06:00
james.maesandClaude Opus 5.5 84e2311d84 ci: use shared ci-workflows Maven workflow
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
build / maven (push) Failing after 2s
QRun-IO/ci-policy: ci / Scoped workflow [workflow 1b3648d9421543350067e95e54959dfb471b26d5] (pull_request) Failing after 2s
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-business (pull_request) Skipped
QRun-IO/ci-policy: ci / original-voyage (pull_request) Skipped
QRun-IO/ci-policy: ci / original-cli (pull_request) Skipped
QRun-IO/ci-policy: ci / ci-result (pull_request) Failing after 2s
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
maven / build (push) Failing after 2s
QRun-IO/ci-policy: native / native-result (pull_request) Failing after 1s
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: ci / shared-pipeline (pull_request) Skipped
Replace the platform-ci release callers with one caller of
KofTwentyTwo/ci-workflows maven.yaml (qqq-orb port).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:08:11 +00:00
forge-recovery 5987cb53ac ci: enroll shared Gitea lifecycle
QRun-IO/ci-policy: ci / head-profile (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (push) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (push) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (push) Skipped
QRun-IO/ci-policy: ci / node-8 (push) Skipped
QRun-IO/ci-policy: ci / node-10 (push) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (push) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (push) Skipped
QRun-IO/ci-policy: ci / source-config-13 (push) Skipped
QRun-IO/ci-policy: ci / source-config-14 (push) Skipped
QRun-IO/ci-policy: native / fork-deny (push) Skipped
QRun-IO/ci-policy: native / dotnet-0 (push) Skipped
Shared Java Maven CI / secrets (push) Successful in 26s
QRun-IO/ci-policy: native / native-result (push) Successful in 7s
QRun-IO/ci-policy: ci / repository-policy (push) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (push) Skipped
Shared Java Maven CI / verify (push) Successful in 2m17s
QRun-IO/ci-policy: Shared Java Maven CI / verify (push) Successful in 1m55s
QRun-IO/ci-policy: ci / ci-result (push) Successful in 4s
qrun-release / postpublish (push) Successful in 2m4s
QRun-IO/ci-policy: ci / fork-guard (push) Skipped
QRun-IO/ci-policy: ci / application-template-0 (push) Skipped
QRun-IO/ci-policy: ci / gradle-1 (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (push) Skipped
QRun-IO/ci-policy: ci / node-9 (push) Skipped
QRun-IO/ci-policy: ci / android-0 (push) Skipped
QRun-IO/ci-policy: ci / original-business (push) Skipped
QRun-IO/ci-policy: ci / original-mcp (push) Skipped
QRun-IO/ci-policy: ci / not-enrolled (push) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / secrets (push) Successful in 26s
QRun-IO/ci-policy: ci / java-docker-3-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (push) Skipped
Shared Java Maven CI / gate (push) Successful in 15s
qrun-release / release (push) Successful in 53m0s
QRun-IO/ci-policy: Shared Java Maven CI / gate (push) Successful in 12s
QRun-IO/ci-policy: ci / java-maven-6 (push) Successful in 52m55s
Shared QRun source version / version (push) Successful in 2m4s
2026-11-03 18:10:35 +00:00
forge-recovery a087988f11 ci: enroll shared Gitea lifecycle
QRun-IO/ci-policy: ci / repository-policy (push) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (push) Skipped
QRun-IO/ci-policy: ci / ci-result (push) Successful in 5s
QRun-IO/ci-policy: Shared Java Maven CI / verify (push) Successful in 2m5s
QRun-IO/ci-policy: ci / head-profile (push) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (push) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (push) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (push) Skipped
QRun-IO/ci-policy: ci / node-8 (push) Skipped
QRun-IO/ci-policy: ci / node-10 (push) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (push) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (push) Skipped
QRun-IO/ci-policy: ci / source-config-13 (push) Skipped
QRun-IO/ci-policy: ci / source-config-14 (push) Skipped
QRun-IO/ci-policy: native / fork-deny (push) Skipped
QRun-IO/ci-policy: native / dotnet-0 (push) Skipped
QRun-IO/ci-policy: native / native-result (push) Successful in 12s
QRun-IO/ci-policy: ci / java-docker-3-head (push) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (push) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / secrets (push) Successful in 29s
QRun-IO/ci-policy: Shared Java Maven CI / gate (push) Successful in 14s
QRun-IO/ci-policy: ci / java-maven-6 (push) Successful in 1h13m45s
QRun-IO/ci-policy: ci / fork-guard (push) Skipped
QRun-IO/ci-policy: ci / application-template-0 (push) Skipped
QRun-IO/ci-policy: ci / gradle-1 (push) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (push) Skipped
QRun-IO/ci-policy: ci / node-9 (push) Skipped
QRun-IO/ci-policy: ci / android-0 (push) Skipped
QRun-IO/ci-policy: ci / original-mcp (push) Skipped
QRun-IO/ci-policy: ci / not-enrolled (push) Skipped
2026-11-02 22:35:22 +00:00
forge-recoveryandCodex 894c3023b4 ci: enroll shared Gitea release-candidate lifecycle
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / secrets (pull_request) Successful in 25s
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / verify (pull_request) Successful in 2m28s
QRun-IO/ci-policy: Shared Java Maven CI / gate (pull_request) Successful in 20s
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Successful in 12m47s
QRun-IO/ci-policy: ci / ci-result (pull_request) Successful in 5s
Release candidate / release-rc (pull_request) Skipped
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: native / native-result (pull_request) Successful in 8s
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
Co-Authored-By: Codex <noreply@openai.com>
2026-11-02 20:19:48 +00:00
15c2906fc1 ci: add the isolated postpublish job to the QRun release caller
Shared Java Maven CI / secrets (push) Failing after 16s
Shared Java Maven CI / verify (push) Canceled after 0s
qrun-release / postpublish (push) Canceled after 0s
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: ci / head-profile (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-business (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / secrets (pull_request) Successful in 20s
QRun-IO/ci-policy: ci / repository-policy (pull_request) Skipped
QRun-IO/ci-policy: ci / java-core-2-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-6-head (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-7-head (pull_request) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / verify (pull_request) Successful in 1m56s
QRun-IO/ci-policy: Shared Java Maven CI / gate (pull_request) Successful in 11s
QRun-IO/ci-policy: ci / java-maven-6 (pull_request) Successful in 23m20s
QRun-IO/ci-policy: ci / ci-result (pull_request) Successful in 4s
Release candidate / release-rc (pull_request) Skipped
Shared Java Maven CI / gate (push) Canceled after 0s
qrun-release / release (push) Canceled after 2h30m11s
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-4 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-9 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-10 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-11 (pull_request) Skipped
QRun-IO/ci-policy: ci / repository-policy-12 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-13 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-14 (pull_request) Skipped
QRun-IO/ci-policy: ci / android-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / original-mcp (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: native / native-result (pull_request) Successful in 7s
QRun-IO/ci-policy: ci / java-docker-3-head (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
2026-11-02 15:52:40 +00:00
45015b6912 ci: move the QRun release caller to the thin v2 contract
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-9 (pull_request) Skipped
QRun-IO/ci-policy: native / fork-deny (pull_request) Skipped
QRun-IO/ci-policy: ci / ci-result (pull_request) Failing after 5s
QRun-IO/ci-policy: Shared Java Maven CI / verify (pull_request) Successful in 3m1s
Shared Java Maven CI / secrets (push) Failing after 20s
QRun-IO/ci-policy: Shared Java Maven CI / secrets (pull_request) Failing after 19s
QRun-IO/ci-policy: Shared Java Maven CI / gate (pull_request) Canceled after 0s
Shared Java Maven CI / verify (push) Canceled after 0s
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-2 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-7 (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: native / dotnet-0 (pull_request) Skipped
QRun-IO/ci-policy: native / native-result (pull_request) Successful in 5s
Shared Java Maven CI / gate (push) Canceled after 0s
qrun-release / release (push) Canceled after 27m5s
QRun-IO/ci-policy: ci / java-maven-4 (pull_request) Canceled after 26m44s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
2026-11-02 06:20:41 +00:00
forge-recoveryandCodex 0f94818084 ci: add the identical QRun release caller
QRun-IO/ci-policy: ci / ci-result (pull_request) Failing after 8s
QRun-IO/ci-policy: ci / fork-guard (pull_request) Skipped
QRun-IO/ci-policy: ci / application-template-0 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-docker-2 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-5 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-6 (pull_request) Skipped
QRun-IO/ci-policy: ci / node-playwright-7 (pull_request) Skipped
QRun-IO/ci-policy: Shared Java Maven CI / verify (pull_request) Failing after 1s
QRun-IO/ci-policy: Shared Java Maven CI / secrets (pull_request) Failing after 1s
QRun-IO/ci-policy: Shared Java Maven CI / gate (pull_request) Failing after 0s
QRun-IO/ci-policy: ci / gradle-1 (pull_request) Skipped
QRun-IO/ci-policy: ci / java-frontend-browser-3 (pull_request) Skipped
QRun-IO/ci-policy: ci / source-config-8 (pull_request) Skipped
QRun-IO/ci-policy: ci / not-enrolled (pull_request) Skipped
QRun-IO/ci-policy: ci / java-maven-4 (pull_request) Failing after 1m40s
Co-Authored-By: Codex <noreply@openai.com>
2026-11-02 04:14:35 +00:00
james.maesandCodex 7b302ed2b1 ci: adopt managed orb 0.8 patch track (#30)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 15:29:10 -05:00
james.maesandCodex b917b51656 ci: adopt shared orb 0.7.18 (#29)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 14:47:03 -05:00
james.maesandCodex dc4734eef1 ci: adopt shared orb 0.7.17 (#28)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 14:16:47 -05:00
james.maesandCodex eb912ebf07 ci: use the shared branch-aware CodeQL worker (#26)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 10:56:24 -05:00
james.maesandCodex 9091f26dd7 ci: adopt central orb 0.7.12 and stable gates (#24)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 08:53:36 -05:00
james.maesandCodex 4942b7e994 ci: centralize rc release completion in qqq orb (#23)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 02:25:10 -05:00
james.maesandCodex 0875069488 ci: adopt shared orb 0.7.11 (#22)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 00:59:13 -05:00
james.maes bf81c587bc Merge pull request #21 from QRun-IO/feature/central-ci
ci: adopt shared QQQ orb standard
2026-10-20 19:19:22 -05:00
james.maesandCodex 70058c1ef6 ci: adopt verified central QQQ orb 0.7.3
Co-Authored-By: Codex <noreply@openai.com>
2026-10-20 19:08:13 -05:00
james.maes af62471cf6 Merge pull request #19 from QRun-IO/feature/access-log-lifecycle
fix: the access logger no longer keeps its instance alive
2026-10-16 11:41:27 -05:00
james.maesandClaude Opus 5.5 325150a938 fix: the access logger no longer keeps its instance alive
The worker thread's task and the JVM shutdown hook referenced the logger
(and through it the QInstance) strongly for the life of the JVM, so every
instance a host or test built and dropped stayed in memory (kof22's suite
ran out of heap in CI). Both now hold the logger weakly: a dropped
instance can be collected and its worker stops at the next tick.

- withRegisterShutdownHook(false) for hosts that close the runtime
  themselves (default true, unchanged behaviour).
- isWorkerRunning() for hosts and tests.
- README and CHANGELOG (also notes #18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:39:12 -05:00
james.maes 72ec4a08be Merge pull request #18 from QRun-IO/feature/virtual-fields-and-idle-flush
fix: resolve virtual fields; keep the access-log worker off unfinished instances
2026-10-16 11:35:24 -05:00
james.maesandClaude Opus 5.5 ef5c096e84 fix: resolve virtual fields and keep the access-log worker off unfinished instances
- SecuredDataClassifications.effective looked fields up with
  QTableMetaData.getField, which throws for a virtual (computed) field, so
  any user read of a table with one failed (kof22's permalink sitePath and
  siteUrl on every content table, and API validation). Virtual fields now
  take the table classification.
- The access-log worker opened a QContext on every tick, even with an empty
  queue. QContext validates an unvalidated instance, and the worker starts
  while the host is still building it, so the first tick validated (and
  enriched) a half-built instance on another thread; one failure cancelled
  every later tick for good. Idle ticks now return before opening a
  context, and a failed tick is logged and retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:42:05 -05:00
james.maes 75b5f0b64f Merge pull request #17 from QRun-IO/ci/release-rc-secrets-inherit
ci(release-rc): pass secrets so the shared workflow can sign the RC tag
2026-10-16 08:05:16 -05:00
james.maesandClaude Opus 5.5 2cb37a067d ci(release-rc): pass secrets to the shared workflow so it can sign the RC tag
QRun-IO/.github#2 adds a tag-and-release job that signs vX with the
QRUN_TAG_SIGNING_KEY secret after Maven Central confirms the artifact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:18:59 -05:00
james.maes 971ddbed2d Merge pull request #16 from QRun-IO/feature/ui-api-1.6
feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.15)
2026-10-15 16:17:33 -05:00
james.maesandClaude Opus 5.5 e2f5199e06 test(retention): canary leak scan without false positives
The canary was 16 random letters g-v and any 5-letter window counted as a
leak, so ordinary JSON words built from those letters (options, joins)
matched by chance and RetentionProofTest failed intermittently. The canary
now has 24 random letters and a leak is the whole marker or a 12-letter
window of its random part (16^12 values): chance matches are impossible,
truncated or re-encoded fragments of a real leak are still caught. A new
test pins both directions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:15:09 -05:00
james.maesandClaude Opus 5.5 214bd75298 chore: target QQQ 4.1.0-RC.15 (UI API 1.6.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:27:07 -05:00
james.maesandClaude Opus 5.5 6bb55f332a feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.14)
Regenerated against UI API 1.6.0: every operation has an operationId and
the standard error responses (BasicErrorResponseV1: error, code, message,
fieldErrors, traceId); the extension's own error schema is kept but
deprecated; hand-written error bodies go through QJavalinUtils so they carry
code/message/traceId. Extension MINOR bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:48:13 -05:00
james.maes 781bae3efe Merge pull request #15 from QRun-IO/feature/remove-standins
build: QQQ 4.1.0-RC.7; remove core stand-ins
2026-10-14 16:33:33 -05:00
james.maesandClaude Opus 5.5 677d611b15 build: start on the 0.1 version line, like the other new qbits
The UI API extension keeps its own semver (secured-data 1.0.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:27:45 -05:00
james.maesandClaude Opus 5.5 527ee4fc96 build: pin QQQ 4.1.0-RC.7
RC.7 ships field access (#1172) and audit value protection (#1182), so the
qbit builds on the released core with no stand-ins. Also repairs a
non-UTF-8 character in the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:25:17 -05:00
james.maes c0a2d67026 Merge remote-tracking branch 'origin/develop' into feature/remove-standins 2026-10-14 16:22:15 -05:00
james.maes f415f2cc8d Merge pull request #14 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:46:31 -05:00
james.maesandClaude Opus 5.5 0dec9463a6 chore: use core field access and audit-token APIs directly; remove stand-ins
QQQ #1172 and #1182 are released, so the standin package and the
reflective core bridges go; the resolver, key source and audit protector
register through core metadata, and the core-dependent tests run
unconditionally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:10:18 -05:00
james.maesandClaude Opus 5.5 e9b8a05f87 ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:07:46 -05:00
james.maes f2736311da Merge pull request #13 from QRun-IO/chore/standin-rc8
chore: retag stand-ins removeBy=RC.8
2026-10-14 07:57:01 -05:00
james.maesandClaude Opus 5.5 d8e3cb976d chore: stand-ins are removed with the first QQQ RC that ships #1172 and #1182 (RC.8)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:52:48 -05:00
james.maes 9b2d8fd182 Merge pull request #11 from QRun-IO/feature/retention-and-wipe
feat: retention and wipe with crypto-shredding; QQQ RC.6 (phase f)
2026-10-14 07:42:39 -05:00
james.maesandClaude Opus 5.5 470b0bd214 feat: retention and wipe with crypto-shredding; move to QQQ RC.6 (phase f)
Moves the pin to QQQ 4.1.0-RC.6 and removes the stand-ins RC.6 provides
(classification, client channels, table features, the UI API extension
point). Adds retention rules, wipe markers, legal holds, wipe runs and the
per-record-per-field audit key table; the wipe process (null with marker,
overwrite, keyed hash, delete, archive then wipe) runs in one transaction
per batch, destroying the audit keys with the field updates; the audit
protector and key source (AES-GCM wrapped DEKs, AAD-bound); the backups
warning; canary and wipe-proof tests.

Fixes a key-source bug that zeroed each DEK after returning it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:35:45 -05:00
james.maes 9801c6b4d1 Merge pull request #9 from QRun-IO/feature/app-home
feat: app home and anomaly alerts (phase e)
2026-10-14 07:16:04 -05:00
james.maes 0ce1723019 Merge pull request #8 from QRun-IO/feature/reveal-and-break-glass
feat: reveal, break-glass and the secured-data UI API extension (phase c)
2026-10-14 07:15:51 -05:00
78 changed files with 6628 additions and 1923 deletions

No files matched your search

+10 -61
View File
@@ -1,65 +1,14 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
qqq-orb: kingsrook/qqq-orb@0.8
workflows:
test_only:
setup:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
context: [ qqq-maven-registry-credentials ]
filters:
branches:
ignore: /(develop|release/.*|integration.*)/
tags:
only: []
publish_snapshot:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [ qqq-maven-registry-credentials ]
branch_type: snapshot
filters:
branches:
only:
- develop
publish_release_candidate:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [ qqq-maven-registry-credentials ]
branch_type: release_candidate
filters:
branches:
only:
- /release\/.*/
publish_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [ qqq-maven-registry-credentials ]
branch_type: release
filters:
branches:
ignore: /.*/
tags:
only: [ /^v\d+\.\d+\.\d+$/ ]
- qqq-orb/generate_pipeline:
orb_version: '0.8'
filters:
branches:
ignore: automation/qbit-rc-pending
tags:
only: /^v\d+\.\d+\.\d+$/
+7
View File
@@ -0,0 +1,7 @@
name: build
on:
push:
jobs:
maven:
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@v1
secrets: inherit
+23
View File
@@ -0,0 +1,23 @@
name: CodeQL
'on':
push:
branches:
- main
- develop
- release/**
- hotfix/**
pull_request: {}
schedule:
- cron: 41 16 * * 0
workflow_dispatch: {}
permissions:
contents: read
actions: read
security-events: write
packages: read
concurrency:
group: codeql-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
analyze:
uses: QRun-IO/qqq-orb/.github/workflows/qbit-codeql.yml@c288a8864fe9054f68410e1d78f5863aea4c4d22
+62 -13
View File
@@ -1,25 +1,74 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
'on':
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
description: Plan a cut without changing release state
type: boolean
default: false
finalize:
description: Retry finalization of an existing cut; do not cut another RC
type: boolean
default: false
recover:
description: Recover the same published RC after a deployment timeout; never
reupload
type: boolean
default: false
circle_build:
description: Original Circle publisher build number (recovery only)
type: string
default: ''
merge_sha:
description: Existing cut merge_sha (finalize retry only)
type: string
default: ''
bump_sha:
description: Existing cut bump_sha (finalize retry only)
type: string
default: ''
branch:
description: Existing cut branch (finalize retry only)
type: string
default: ''
version:
description: Existing cut version (finalize retry only)
type: string
default: ''
pull_request:
types: [closed]
branches: [develop]
types:
- closed
branches:
- develop
repository_dispatch:
types:
- qbit-rc-published
permissions:
contents: write
actions: read
checks: read
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
if: github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch'
|| (github.event_name == 'pull_request' && github.event.pull_request.merged
== true && github.event.pull_request.base.ref == 'develop' && contains(github.event.pull_request.labels.*.name,
'release:rc'))
uses: QRun-IO/qqq-orb/.github/workflows/qbit-release-rc.yml@cc30f599e63c8e41629c99d23395bac1adb8bc44
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || false
}}
finalize: ${{ github.event_name == 'repository_dispatch' || (github.event_name
== 'workflow_dispatch' && inputs.finalize) }}
recover: ${{ github.event_name == 'workflow_dispatch' && inputs.recover || false
}}
circle_build: ${{ github.event_name == 'workflow_dispatch' && inputs.circle_build
|| '' }}
merge_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.merge_sha
|| inputs.merge_sha || '' }}
bump_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.bump_sha
|| inputs.bump_sha || '' }}
branch: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.branch
|| inputs.branch || '' }}
version: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.version
|| inputs.version || '' }}
secrets: inherit
+16
View File
@@ -0,0 +1,16 @@
profile: qbit
orb_version: '0.8'
default_branch: develop
java_version: '21'
browsers: true
api_checks: false
docs_only: false
release_on_main: false
release_tag_policy: stable
test_branch_policy: non-release-no-integration
publish:
snapshot: true
release_candidate: true
release: true
asciidoc: false
release_completion: true
+53
View File
@@ -4,6 +4,59 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
## [Unreleased]
### Added: a create date and a modify date on every table
- Every table has a create date and a modify date: `DATE_TIME`, not editable, with QQQ's `CREATE_DATE` and `MODIFY_DATE` behaviors, in the columns `create_date` and `modify_date`. Seven tables did not have them: reveal grant, break-glass session, access log, wipe run, wipe marker, legal hold and audit key. Their fields are named `createDate` and `modifyDate`. Classification, policy and retention rule already had `create_date` and `modify_date`; they keep those field names, and the two fields are now not editable and have the behaviors.
- A new row gets both dates, equal, in the insert that writes it. A changed row keeps its create date and gets a later modify date (for an edit that arrives over HTTP this holds from this release on: see the fix below). An update that names the create date itself still sets it: not editable is what the metadata tells a client. The access log and wipe runs are only ever inserted, so their dates stay equal. The access-log writer still runs one insert per row, with no other statement, lock or transaction.
- **Upgrade: run the DDL first.** Before deploying this release on an existing database, run the last block of `docs/ddl/secured-data-postgres.sql` (from the line `-- Standard dates:` to the end). It adds the two nullable columns to the seven tables, defaults them to the current time and gives existing rows the row's own time. It rewrites no table and can be run while the release before this one is running, which goes on working with the columns present. This release does not run without them. See "Upgrading an existing database" in the README.
- The dates have no classification, policy or assistant exposure of their own, and nothing that is signed or verified covers them. Rows written before the columns existed are unchanged apart from the two columns: wrapped data keys unwrap as before and audit tokens verify as before.
- QQQ uses a field with the `MODIFY_DATE` behavior as the record version of its table. That now holds for every table of the QBit, classification, policy and retention rule included: on an update core sets the modify date itself, and an update that sends the version it read is refused (409) if the row has changed since. Updates that send no version, the QBit's own among them, behave as before. The QBit's own updates move the version all the same: a wipe, the scheduled one included, records its run on its retention rule, so an API client that reads a rule and edits it with that version after a wipe has run gets 409 and has to read the rule again.
- On a live database, set a `lock_timeout` before running the upgrade block and run it again when a statement is cancelled (an `ALTER` waits behind an open transaction on its table, and other readers and writers wait behind it). On a large access log, run its backfill `UPDATE` in ranges of `id`, or leave it out: rows without the two dates are read and written like any others. The README has both.
### Fixed: an edit over HTTP was treated as a new row
- The table customizer matched an update with its stored row by comparing ids of different types: the id of a request over HTTP is text, the stored id a number. An edit made through the API or the UI was therefore validated and stamped as an insert. Ids are now compared as text.
- What that did, for an edit over HTTP: a classification, policy or retention rule got a new `create_date` on every edit. Releasing a legal hold overwrote when and by whom it was placed. Editing a classification while naming its table and field was refused as a second classification of the same field. Editing a retention rule without sending every field was refused. Editing a legal hold without sending its status failed with a server error; a client that sends only the changed fields of a table with a modify date does exactly that.
### Fixed: a reveal and the access-log writer waiting for each other
- A reveal, or the start, end or expiry of a break-glass session, no longer deadlocks with a thread that asks for a classification while it holds another lock of this QBit. The access-log writer is such a thread: it holds the anomaly counters while it inserts a row, and that insert asks for classifications (it counts earlier rows and writes an audit, both queries with criteria). A reveal holds the runtime's monitor and then inserts its own access-log row. When both happened at once, neither finished, and neither did any later request, because every request asks for a classification. A person who opens a record with a masked field and reveals it straight away does both at once.
- The rule rows a node keeps (classifications, active retention rules) have a lock of their own, held only while they are answered or read. Asking for a classification no longer takes the monitor that reveal and break-glass hold for their whole operation.
### Fixed: rule reads that ask for their own rows
- Reading the rule rows a node keeps (classifications, active retention rules, policies) no longer recurses when code that runs inside that read asks for the same rows. Another QBit's virtual field value provider, or a post-query customizer, on a secured-data table that ran a query with filter criteria did exactly that: with one classification row present, every request that needed a classification ended in a `StackOverflowError`.
- The QBit's system reads of its own tables (`SystemWrites.query`, the audit key lookup) name the stored fields of the table, so core runs no virtual field value provider inside them.
- A question asked from inside a rule read is answered from the rows held when that read began (none before the first read) and starts no other read. Only the reading thread, inside the QBit's system read, can be given that answer; every request gets the rows the read returns. The same holds for a transaction's read of its own retention rule edits.
- A rule read that fails is tried again by the next question instead of failing or recursing from then on.
### Fixed: access-log lifecycle
- The access-log worker and JVM shutdown hook hold the logger weakly, so an instance the host drops (tests, reloads) can be collected; its worker stops at the next tick.
- `withRegisterShutdownHook(false)` for hosts that call `SecuredDataRuntime.close()` themselves; `SecuredDataAccessLogger.isWorkerRunning()`.
- Virtual (computed) fields resolve through the table classification instead of throwing (#18).
- Idle access-log ticks open no context, so the worker never validates an unfinished host instance, and a failed tick no longer stops the worker (#18).
### Changed: UI API 1.6.0
- UI API extension secured-data 1.1.0 (requires QQQ 4.1.0-RC.15, UI API 1.6.0): operationIds, standard error responses with BasicErrorResponseV1, extension tags, and retained SecuredDataErrorV1 schema deprecated since 1.1.0.
### Changed — direct core field access and audit protection
- Remove all #1172/#1182 stand-ins and reflective bridges; use core field-access, classified-value token, audit protection, wipe flag and shredded-event APIs directly.
- Register the resolver, key source and audit protector through core supplemental metadata.
- Run core field-access integration and retention proof tests unconditionally.
- Pin QQQ to `4.1.0-RC.7`, which ships field access (#1172) and audit value protection (#1182); all core stand-ins and reflective bridges are removed.
### Changed — RC.6 migration
- Pin QQQ to 4.1.0-RC.6; use released classification, client-channel and table-feature APIs directly.
- Fold UI API code and tests into the main build and remove the profile and reflective registration.
- Remove released-API stand-ins and the local contract assertion facade; retain #1172 field-access stand-ins with RC.7 removal markers.
### Added: phase (f) retention and wipe
- Retention rules, wipe markers, legal holds, wipe runs and a hidden, locked audit-key table, with the `secured_data.retention` permission and a "Retention rules" navigation entry.
- Actions NULL_WITH_MARKER, OVERWRITE_TEXT, KEYED_HASH, DELETE_RECORD and ARCHIVE_THEN_WIPE (host `ArchiveSinkInterface`), with validation and low-entropy refusal.
- Atomic wipe batches (fields, markers, key deletions, audits, access log and run row in one rollback-capable transaction), per-record and per-table legal holds, a Review/Result "Run wipe now" process and a daily scheduled run.
- Per-record-field AES-GCM wrapped DEKs (AAD-bound) for crypto-shredding of audit tokens, `CHANGED_ONLY` audit treatment, and the audit-value protector.
- App-home warning when known backup retention exceeds a rule's `age_days`; `withBackupRetentionDays` and `withArchiveSink` options; README retention section and the backups residual risk.
- Tests: all actions, holds, rollback, Review/Result, schedule, permissions, wrap/unwrap, and the section 15 canary and wipe-proof tests.
### Fixed
- The key source returned a DEK array it then zeroed; it now returns copies, with a regression test.
### Added
- Phase (e) configurable reveal-median, distinct-record and exact-match anomaly detection, with system access-log counters, bounded daily median caching, per-user/day alerts, and flags retained without an alert sink.
+104 -20
View File
@@ -1,16 +1,14 @@
# QBit Secured Data
Phases (b), (c), and (e) of [SECURED-DATA](https://github.com/QRun-IO/qqq/blob/develop/docs/SECURED-DATA.md): classification overrides, policy resolution, enumeration guardrails, and value-free access logging. QQQ core owns masking and query enforcement. Reveal, break-glass, the app home and anomaly alerts are implemented. Retention and audit-value protection remain later phases.
Phases (b), (c), (e) and (f) of [SECURED-DATA](https://github.com/QRun-IO/qqq/blob/develop/docs/SECURED-DATA.md): classification overrides, policy resolution, enumeration guardrails, and value-free access logging. QQQ core owns masking and query enforcement. Reveal, break-glass, the app home and anomaly alerts are implemented. Retention, wipe and audit-value protection are implemented (phase f); see Retention and wipe.
## Compatibility
Java 21, Maven wrapper 3.9.11, **QQQ 4.1.0-RC.5**. The released dependency pin is unchanged.
Java 21, Maven wrapper 3.9.11, and **QQQ 4.1.0-RC.15** or later (UI API 1.6.0, field access #1172 and audit value protection #1182).
RC.5 contains AI exposure metadata, the alerts producer SPI and the tables picker. It does **not** contain field-access enforcement, `DataClassification`, classification properties on AI metadata, client channels, or table-feature permissions. Jar contents and embedded sources were inspected locally (`jar tf`; Git's `unzip` could not run in the Windows sandbox).
Core classifications, AI metadata, client channels, table-feature permissions and UI API extension points are used directly. The UI API and its contract test are part of the main build; the contract test calls `UiApiContract` directly.
Missing APIs live under `com.kingsrook.qbits.secureddata.standin`, with `STAND-IN` removal markers. `CoreFieldAccessBridge` registers a dynamic proxy only when core's `FieldAccessResolverMetaData` exists. It converts decision record components and event enums by name. On RC.5 it logs one INFO that enforcement requires RC.6+; installing this QBit on RC.5 **does not enforce field tiers**.
`CoreMetadataBridge` promotes temporary classifications into core AI metadata when supported, registers the channel wall and table features reflectively, and uses core feature-permission checks when present. A missing API is supported; a present but incompatible API fails startup. The public classification API temporarily returns the stand-in `DataClassification` because the core enum is absent from RC.5. At RC.6, move these imports to core and delete the adapters/stand-ins.
Field access (#1172), classified-value tokens and audit-value protection (#1182) use core types directly. The producer registers the resolver, key source and audit protector through core supplemental metadata; QQQ core enforces field tiers.
## Install and register
@@ -18,7 +16,7 @@ Missing APIs live under `com.kingsrook.qbits.secureddata.standin`, with `STAND-I
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-secured-data</artifactId>
<version>1.0.0-SNAPSHOT</version>
<version>0.1.0-RC.1</version>
</dependency>
```
@@ -32,7 +30,45 @@ new SecuredDataQBitProducer()
Register the host backend and application tables before producing the QBit. Enable core permissions (`qInstance.withEnforcePermissionsInCoreActions(true)`) at application entry points. Only one secured-data runtime is supported per instance. A namespace labels the QBit identity; table names remain fixed.
The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata; an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode.
The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata (the fields `createDate` and `modifyDate` are the columns `create_date` and `modify_date`); an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode.
### Upgrading an existing database: create and modify dates
Every table has the columns `create_date` and `modify_date`, and the QBit reads and writes them on all ten tables. Seven tables were first created without them: `secured_data_reveal_grant`, `secured_data_break_glass`, `secured_data_access_log`, `secured_data_wipe_run`, `secured_data_wipe_marker`, `secured_data_legal_hold` and `secured_data_audit_key`.
Before deploying a release that has the two dates on every table, run the last block of [the DDL](docs/ddl/secured-data-postgres.sql) on the existing database: everything from the line `-- Standard dates:` to the end of the file. The order matters. The release before it runs unchanged once the columns are there. The release with the dates does not run without them: its reads of the seven tables fail, and with them every request that needs a field decision. The block does three things to each of the seven tables:
- It adds the two columns, nullable. That changes the catalog only; the table is not rewritten.
- It gives each column a default of the current time, so a release that does not know the columns goes on inserting while both releases run.
- It gives the rows that were already there the row's own time (below). The `UPDATE` touches only rows whose `create_date` is null and takes row locks only.
Every statement of the block can be run again and then changes nothing. Two things to plan for on a live database:
- **Locks.** Each `ALTER` takes a brief exclusive lock on its table. It waits behind any open transaction that has touched the table (one that has inserted a log row, for one), and every other reader and writer of the table then waits behind the `ALTER`. Set a lock timeout for the session first, and run the block again when a statement is cancelled:
```sql
SET lock_timeout = '5s';
```
- **A large access log.** The `UPDATE` of `secured_data_access_log` writes every existing row in one transaction. One measurement on PostgreSQL 16 with 2,000,000 rows: 24 s, and the table grew from 146 MB to 324 MB. On a large log run it in ranges of `id` instead, one range per transaction, up to the highest `id`:
```sql
UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL AND id BETWEEN 1 AND 100000;
```
Or leave it out. A row without the two dates is read and written like any other, and the columns are filled for every row inserted from then on.
| Table | `create_date` of an existing row | `modify_date` of an existing row |
|---|---|---|
| `secured_data_reveal_grant` | `granted_at` | `granted_at` |
| `secured_data_break_glass` | `started_at` | `ended_at`, else `started_at` |
| `secured_data_access_log` | `created_at` | `created_at` |
| `secured_data_wipe_run` | `finished_at`, else `started_at` | the same |
| `secured_data_wipe_marker` | `wiped_at` | `wiped_at` |
| `secured_data_legal_hold` | `placed_at`, else the time of the upgrade | `released_at`, else `placed_at`, else the time of the upgrade |
| `secured_data_audit_key` | `created_at` | `created_at` |
Nothing else changes in a row: wrapped data keys, search fingerprints and every other column are left as they are.
## Tables, screens and permissions
@@ -46,27 +82,33 @@ The host supplies database connection metadata. Physical table names and field b
Every table has audit rules, explicit record sections and CONFIDENTIAL code classification. The Secured Data app navigates to Classification, Policy, Access log and Break-glass session. Reveal grants are a read-only child list joined by `access_log_id`; no process or home widget appears in navigation. The classification target `(table_name, field_name)` is unique, including the null field used for a table-wide override.
Every table has a create date and a modify date: date-time fields that nobody edits, in the columns `create_date` and `modify_date`. The fields are named `createDate` and `modifyDate`. Classification, Policy and Retention rules had the two dates before the other tables did and keep the field names `create_date` and `modify_date`. A new row gets both, equal. A changed row keeps its create date and gets a later modify date. (Not editable is what the metadata tells a client, as everywhere in QQQ: an update that names the create date itself still sets it.) Every write of the QBit is a QQQ insert or update action, so the dates are in the one statement that writes the row, the access-log writer's insert included. The access log and wipe runs are only ever inserted, so their two dates stay equal.
The two dates have no classification of their own. They are CONFIDENTIAL with the rest of their table, are given the tier of its other fields, and are closed to the assistant with it. Nothing that is signed or verified covers them: a wrapped data key is bound to its table, field and record id, and the search fingerprints of a log row are made from the search values.
The modify date has QQQ's modify-date behavior, and QQQ uses such a field as the record version of its table. On an update core sets it to the current time (or one millisecond after the stored value, if that is later). When a caller sends the version it read, core refuses the update (409) if the row has changed since. The QBit's own updates send no version, but they move it: a wipe records its run on its retention rule, so a client that reads a rule and saves it with that version after a wipe has run is refused and has to read the rule again.
`secured_data.admin` gates configuration editing and activity reads through a standard QQQ custom permission checker. It gives no implicit access to classified fields on application tables. `secured_data.hasAccess` is the catalog base permission; the custom checker requires admin. Activity writes are refused for ordinary users even when they hold admin. Customizers also protect system-only writes, and update/delete capabilities are disabled on the access log, including system calls.
Both table features are registered for **all tables**: `secured_data_full` (Secured data: full) and `secured_data_reveal` (Secured data: reveal). This deliberate catalog choice supports new admin classification rows without a restart. Core's permission-base, READ prerequisite and NOT_PROTECTED semantics are preserved: NOT_PROTECTED application tables grant their features without an explicit permission. Use protected table permission rules when field policies must restrict users. Feature grants have no standard WRITE/admin implication.
The assistant is denied `secured_data_*` through core client-channel metadata. The local custom checker enforces the same wall on RC.5 when the trusted host uses `standin.ClientChannelSession`. On newer core use `QSession.withClientChannel("assistant/t1")`. The normalized channel `assistant` is stored in logs and used for policy caps. A host must set the channel from trusted request context.
The assistant is denied `secured_data_*` through core client-channel metadata. Use `QSession.withClientChannel("assistant/t1")` from trusted request context. The normalized channel `assistant` is stored in logs and used for policy caps.
ESB integration follows webhooks' host-wiring pattern: `SecuredDataTables.esbTableNames()` lists Classification, Policy, Access log and Break-glass. Connect those names to the host ESB table-event producer. No ESB or qbit-alerts dependency is added.
## Classifying fields
Code metadata establishes defaults. While pinned to RC.5:
Code metadata establishes defaults using the core APIs:
```java
import com.kingsrook.qbits.secureddata.standin.AiFieldMetaData;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
email.withSupplementalMetaData(
new AiFieldMetaData().withClassification(DataClassification.PII));
```
The matching stand-in `AiTableMetaData` sets a table default. These supplemental objects coexist with RC.5's core AI exposure metadata. The producer promotes them into core AI metadata when the newer API is available. After moving to RC.6, use `com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData` and core `DataClassification` with the same classification pattern.
Core `AiTableMetaData` sets a table default. Runtime classification rows take precedence over code metadata.
Admin rows in Classification can raise **or lower** the default. Effective order:
@@ -103,7 +145,7 @@ Policy/break-glass caches are bounded per instance and keyed by relevant user pe
The extra `search_value_fingerprints` TEXT column persists JSON arrays of HMAC-SHA256 fingerprints for distinct-value detection after restarts and across nodes. Use a shared secret key supplier on multi-node deployments; the default is a random per-JVM key, so historical fingerprints from another JVM/key cannot be compared reliably.
One daemon worker drains a bounded queue as a system session. Enqueueing does not wait for persistence. Full queues and failed writes warn without values and increment `getDroppedCount()`. The app can expose this counter later. Explicitly call the instance's `SecuredDataRuntime.close()` at host shutdown; a JVM shutdown hook also drains accepted rows. An abrupt process crash can lose queued rows.
One daemon worker drains a bounded queue as a system session. Enqueueing does not wait for persistence. Full queues and failed writes warn without values and increment `getDroppedCount()`. The app can expose this counter later. Explicitly call the instance's `SecuredDataRuntime.close()` at host shutdown; a JVM shutdown hook also drains accepted rows unless `withRegisterShutdownHook(false)` turns it off (hosts that close the runtime themselves). Neither the worker nor the hook holds the instance strongly: an instance the host drops (tests, reloads) can be collected, and its worker stops at the next tick. An abrupt process crash can lose queued rows.
Flagged rows raise the optional alerts producer signal `secured_data.enumeration`, deduplicated by `secured_data.enumeration:<user>:<table>.<field>:<UTC date>`. RC.5 already provides `QAlerts`; without an installed sink it logs the signal. Its catalog validator rejects underscores in type names, so this mandated type is sent through the producer without catalog registration until that core restriction is reconciled. No values or fingerprints are sent to the alert sink.
@@ -123,13 +165,13 @@ Configure `.withSchedulerName("yourScheduler")` to register the hidden `secured_
## UI API extension
The `secured-data` extension is version **1.0.0**:
The `secured-data` extension is version **1.1.0**:
- `POST /qqq/v1/qbit/secured-data/reveal`
- `GET /qqq/v1/qbit/secured-data/reveal/options?tableName=&fieldName=`
- `GET /qqq/v1/qbit/secured-data/status`
Build with `-Duiapi=true` and a compatible core to compile `src/main/java-uiapi` and `src/test/java-uiapi`. The producer registers the extension only when both its classes and core's extension metadata exist. The committed contract is [secured-data.yaml](src/main/resources/openapi/qqq-v1/secured-data.yaml); regenerate intentionally with `-Dqqq.uiApi.regenerate=true`. The supplied core exposes `UiApiContract.check`, so a local test-only `UiApiContractAssertions` adapter supplies the documented assertion facade. All UI API roots and adapters are `STAND-IN(core=QRun-IO/qqq#1128, removeBy=RC.6)` and fold into the normal roots at RC.6. The core OpenAPI model cannot mark object properties required; the request schema documents required properties and the handler validates them.
The producer registers the UI API extension directly. The committed contract is [secured-data.yaml](src/main/resources/openapi/qqq-v1/secured-data.yaml); regenerate intentionally with `-Dqqq.uiApi.regenerate=true`. Contract and HTTP tests run in the normal build.
The classification field picker uses CUSTOM possible values, reading `table_name` or `tableName` from the form's other values. It returns only the selected readable table's labeled fields, returns none without a selection, and rejects an unknown table. TODO: add `dependsOn("table_name")` when the core metadata API supports it.
@@ -146,11 +188,14 @@ The classification field picker uses CUSTOM possible values, reading `table_name
| `withFingerprintKeySupplier(Supplier<byte[]>)` | Random 32-byte JVM key; supplied key must be at least 16 bytes |
| `withAccessLogQueueCapacity(int)` | 4096 rows |
| `withAccessLogFlushIntervalMillis(long)` | 250 ms |
| `withRegisterShutdownHook(boolean)` | true; JVM hook that drains the access log at shutdown |
| `withExactMatchPerHour(int)` | 20; zero disables |
| `withMissAlertConsecutive(int)` | 5; zero disables |
| `withDistinctValueAlert(int)` | 15; zero disables |
| `withMinEntropyEnabled(boolean)` | true |
| `withSchedulerName(String)` | Host scheduler for the 60-second break-glass expiry job; null means host-managed invocation |
| `withBackupRetentionDays(Integer)` | Unknown; physical backup window in days, used for the retention warning (must not be negative) |
| `withArchiveSink(ArchiveSinkInterface)` | None; required before any `ARCHIVE_THEN_WIPE` rule can run |
| `withClock(Clock)` | UTC system clock; injectable for deterministic expiry tests |
Configure before producing the QBit; runtime infrastructure and HMAC keys are captured at startup. Classification/policy/break-glass lookups always run as system, with a thread-local recursion guard. No reveal values are cached.
@@ -159,10 +204,9 @@ Configure before producing the QBit; runtime infrastructure and HMAC keys are ca
```powershell
./mvnw.cmd -q -o clean verify
./mvnw.cmd -q -o clean verify -Dqqq.snapshot.version=4.1.0-SECUREDATA-SNAPSHOT -Duiapi=true
```
The direct suite uses memory and H2. `CoreFieldAccessBridgeIT` is included by Surefire and uses JUnit assumptions to skip unless the actual core resolver metadata is present. With the installed core snapshot it exercises Query/Get, policy tiers, masking, exact matches and outcomes, sorting denial, rate/entropy refusal and the assistant wall. Checkstyle and JaCoCo remain 70% instructions / 90% classes; stand-ins are not excluded.
The suite uses memory and H2. `CoreFieldAccessIT` runs unconditionally through Surefire and exercises Query/Get, policy tiers, masking, exact matches and outcomes, sorting denial, rate/entropy refusal and the assistant wall. `RetentionProofTest` also runs unconditionally. Checkstyle and JaCoCo gates remain 70% instructions / 90% classes.
## Anomaly monitoring (phase e)
@@ -191,7 +235,7 @@ The single Secured Data app owns one full-width HTML home widget, following qbit
Lists order newest first with id tie-breaking; rankings order by count descending then group alphabetically. Narrow screens use wrapping labelled card rows. Dates stay compact on one line, text is ellipsized with full tooltips, and colors inherit the host with `--qqq-grid-border` for borders. There are no inner scrollers. Empty lists explicitly say there is no activity.
Navigation contains Classification, Policy, Access log and Break-glass sessions. Reveal grants remain a child list; retention is not implemented. Successful policy/classification inserts, updates and deletes produce value-free change rows with actor, record id and action in the source transaction. No configuration values are copied into these rows.
Navigation contains Classification, Policy, Access log, Break-glass sessions and Retention rules. Reveal grants remain a child list. A warning line appears under the KPIs for each active retention rule whose known backup retention exceeds its `age_days`. Successful policy/classification inserts, updates and deletes produce value-free change rows with actor, record id and action in the source transaction. No configuration values are copied into these rows.
### Local screenshot harness
@@ -205,9 +249,49 @@ java -cp $demoClasspath com.kingsrook.qbits.secureddata.SecuredDataDemoServer
Open `http://localhost:8098/app/secured_data`. Stop the demo before `clean` on Windows. Capture light and dark at 390, 768, 1024, 1440, 1920 and 3200 px under `target/screenshots/`. This session started the server, but screenshot capture was blocked by Chromium `spawn EPERM` in the sandbox and two connected-browser timeouts. The coordinator still needs to capture and inspect the twelve images through Next.
## Retention and wipe
Spec: SECURED-DATA sections 13 to 15. Retention rules live in the **Retention rules** table (permission `secured_data.retention`). Each rule names a host table, the fields to wipe, an age field, `age_days`, a batch size, an action and an audit treatment.
Actions:
- `NULL_WITH_MARKER`: set the fields to null (fields must be nullable) and record a wipe marker.
- `OVERWRITE_TEXT`: replace with the rule's text (STRING/TEXT fields only; the text must fit the field).
- `KEYED_HASH`: replace with an instance-key HMAC token (`hmac:v1:...`) that the host can still verify. Refused for low-entropy fields and for fields shorter than 200 characters.
- `DELETE_RECORD`: delete the record, with markers for every field.
- `ARCHIVE_THEN_WIPE`: write the field values as AES-GCM encrypted JSON through the host `ArchiveSinkInterface` (an adapter to object storage, ideally a WORM bucket), then null them. The wipe is refused if no sink is configured.
Audit treatment: `SHREDDABLE` (default) gives each audited record-field its own data key (DEK), so deleting the key makes the audit tokens unverifiable (crypto-shredding). `CHANGED_ONLY` stores only `changed:v1` in audits and never creates a DEK.
Mechanics:
- Eligibility is evaluated in the database and returns record ids only; values are read only for `KEYED_HASH` and `ARCHIVE_THEN_WIPE`.
- Each batch is one transaction in a rollback-capable backend: field updates, wipe markers, audit-key deletions, audit details, the access-log `WIPE` rows and the run row commit or roll back together. Cross-backend or non-transactional deployments are refused before any write.
- Legal holds (per record, or per table with a blank record id) skip records; age and holds are re-checked inside the batch.
- **Run wipe now** is a standard Review/Result process with counts only; the hidden `secured_data_wipe_daily` process runs daily from the host scheduler (`withSchedulerName`). Neither places a host value in a message, log, run row or marker.
- DEKs are wrapped with AES-GCM under the KEK, with the table, field and record id as additional authenticated data. A wrong KEK or a wrong binding fails the unwrap. The wrapped-key table is hidden, locked, not exportable and audited at level NONE.
### Backups are a residual risk (section 14.3)
A wipe removes the value and its audit key from the live database only. Database backups taken before the wipe still contain the value and the wrapped DEK, and remain recoverable until they expire. Set `withBackupRetentionDays(...)`; when it exceeds a rule's `age_days` the rule and the app home warn that wiped values stay recoverable from backups. Unknown retention is reported as unknown, not as safe. Backups that must be purged earlier have to be handled outside this qbit.
### Keys and environment
| Variable | Purpose |
|---|---|
| `QQQ_AUDIT_HMAC_KEY` | Base64 32-byte instance key for `hmac:v1:` audit tokens and `KEYED_HASH`. Without it audits fall back to `changed:v1`. |
| `QQQ_AUDIT_HMAC_KEY_ID` | Optional id embedded in tokens, default `env1`; change it when rotating the key. |
| `QQQ_SECURED_DATA_KEK` | Base64 32-byte key-encryption key that wraps the per-record DEKs and archive objects. Required for `SHREDDABLE` and archive; a missing or invalid KEK fails closed. |
Each is read from a JVM property or the environment and is never logged. Losing the KEK makes every existing DEK unrecoverable, so protect and back it up separately from the database.
### Core hooks
Audit-value protection, wipe audit flags and shredded audit events use the core #1182 APIs directly. Core tokenises ordinary audits and writes value-free wipe details. The section 15 canary and wipe-proof tests (`RetentionProofTest`) are mandatory in the normal build.
## Remaining phases
RC.6 migration removes temporary types and reflection. Retention, wipe and audit-value protection remain phases (f)/(g).
Field-access and audit-token stand-ins and reflective bridges have been removed. The coordinator will replace the temporary snapshot pin with the first QQQ RC containing #1172 and #1182.
## License
+103 -1
View File
@@ -86,7 +86,8 @@ CREATE TABLE secured_data_access_log (
anomaly_flag BOOLEAN DEFAULT false,
break_glass_id INTEGER,
created_at TIMESTAMP WITH TIME ZONE NOT NULL,
search_value_fingerprints TEXT
search_value_fingerprints TEXT,
rule_id INTEGER
);
CREATE INDEX secured_data_policy_lookup ON secured_data_policy (is_active, permission_name, classification, table_name);
@@ -97,3 +98,104 @@ CREATE INDEX secured_data_access_lookup ON secured_data_access_log (user_id, tab
-- Phase (e): bounded user counters and home time windows.
CREATE INDEX secured_data_anomaly_lookup ON secured_data_access_log (user_id, event_type, created_at);
CREATE INDEX secured_data_home_lookup ON secured_data_access_log (event_type, created_at);
-- Phase (f): all policy targets and audit tables must share this transactional backend.
CREATE TABLE secured_data_retention_rule (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name VARCHAR(255) NOT NULL, table_name VARCHAR(255) NOT NULL, field_names TEXT NOT NULL,
age_field_name VARCHAR(255) NOT NULL, age_days INTEGER NOT NULL CHECK (age_days >= 1),
action VARCHAR(40) DEFAULT 'NULL_WITH_MARKER' NOT NULL CHECK (action IN ('NULL_WITH_MARKER','OVERWRITE_TEXT','KEYED_HASH','DELETE_RECORD','ARCHIVE_THEN_WIPE')),
overwrite_text TEXT DEFAULT '[wiped per policy]',
audit_treatment VARCHAR(30) DEFAULT 'SHREDDABLE' NOT NULL CHECK (audit_treatment IN ('SHREDDABLE','CHANGED_ONLY')),
archive_storage_name VARCHAR(255), batch_size INTEGER DEFAULT 500 NOT NULL CHECK (batch_size >= 1),
is_active BOOLEAN DEFAULT true, last_run_at TIMESTAMP WITH TIME ZONE, last_run_count INTEGER,
create_date TIMESTAMP WITH TIME ZONE, modify_date TIMESTAMP WITH TIME ZONE, modified_by VARCHAR(255)
);
CREATE TABLE secured_data_wipe_run (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id),
started_at TIMESTAMP WITH TIME ZONE NOT NULL, finished_at TIMESTAMP WITH TIME ZONE, status VARCHAR(30),
eligible_count INTEGER, held_count INTEGER, wiped_count INTEGER, error_count INTEGER,
record_ids VARCHAR(4000), message VARCHAR(500)
);
CREATE TABLE secured_data_wipe_marker (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255) NOT NULL,
field_name VARCHAR(255) NOT NULL, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id), action VARCHAR(40) NOT NULL,
wiped_at TIMESTAMP WITH TIME ZONE NOT NULL, run_id INTEGER NOT NULL REFERENCES secured_data_wipe_run(id),
UNIQUE (table_name, record_id, field_name)
);
CREATE TABLE secured_data_legal_hold (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255), reason TEXT NOT NULL,
placed_by VARCHAR(255), placed_at TIMESTAMP WITH TIME ZONE, released_by VARCHAR(255), released_at TIMESTAMP WITH TIME ZONE,
status VARCHAR(20) DEFAULT 'ACTIVE' NOT NULL CHECK (status IN ('ACTIVE','RELEASED'))
);
-- Mutable key material: never register this table in ESB, WORM or long-term logical backups.
CREATE TABLE secured_data_audit_key (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL,
record_id VARCHAR(255) NOT NULL, field_name VARCHAR(255) NOT NULL, wrapped_key TEXT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL, UNIQUE (table_name, record_id, field_name)
);
CREATE INDEX secured_data_hold_lookup ON secured_data_legal_hold (table_name, status, record_id);
CREATE INDEX secured_data_retention_lookup ON secured_data_retention_rule (table_name, is_active);
CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id);
-- Standard dates: create_date and modify_date on the seven tables created above without them.
-- A new database runs the whole file. An existing database runs this block alone, BEFORE the release
-- that writes the two columns is deployed. The columns are nullable and default to the current time, so
-- a release that does not know them goes on inserting while both run. Adding a nullable column and
-- setting a default change the catalog only (no table rewrite). Each UPDATE gives the rows that were
-- already there the row's own time, once: it touches only rows whose create_date is still null and
-- takes row locks only. Every statement can be run again and then changes nothing.
--
-- Locks. Each ALTER takes a brief exclusive lock on its table. It waits behind any open transaction
-- that has touched the table (one that has inserted a log row, for one), and every other reader and
-- writer of the table then waits behind the ALTER. On a live database set a lock timeout for the
-- session first (SET lock_timeout = '5s') and run the block again when a statement is cancelled.
--
-- A large access log. The UPDATE of secured_data_access_log writes every existing row in one
-- transaction. One measurement on PostgreSQL 16 with 2,000,000 rows: 24 s, and the table grew from
-- 146 MB to 324 MB. On a large log run it in ranges of id instead, one range per transaction (add
-- AND id BETWEEN 1 AND 100000 to its WHERE, then the next range, up to the highest id), or leave it
-- out: a row without the two dates is read and written like any other, and the columns are filled
-- for every row inserted from then on.
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_reveal_grant ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_reveal_grant ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_reveal_grant ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_reveal_grant SET create_date = granted_at, modify_date = granted_at WHERE create_date IS NULL;
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_break_glass ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_break_glass ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_break_glass ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_break_glass SET create_date = started_at, modify_date = COALESCE(ended_at, started_at) WHERE create_date IS NULL;
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_access_log ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_access_log ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_access_log ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_access_log SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_wipe_run ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_wipe_run ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_wipe_run ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_wipe_run SET create_date = COALESCE(finished_at, started_at), modify_date = COALESCE(finished_at, started_at) WHERE create_date IS NULL;
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_wipe_marker ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_wipe_marker ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_wipe_marker ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_wipe_marker SET create_date = wiped_at, modify_date = wiped_at WHERE create_date IS NULL;
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_legal_hold ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_legal_hold ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_legal_hold ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_legal_hold SET create_date = COALESCE(placed_at, CURRENT_TIMESTAMP), modify_date = COALESCE(released_at, placed_at, CURRENT_TIMESTAMP) WHERE create_date IS NULL;
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS create_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_audit_key ADD COLUMN IF NOT EXISTS modify_date TIMESTAMP WITH TIME ZONE;
ALTER TABLE secured_data_audit_key ALTER COLUMN create_date SET DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE secured_data_audit_key ALTER COLUMN modify_date SET DEFAULT CURRENT_TIMESTAMP;
UPDATE secured_data_audit_key SET create_date = created_at, modify_date = created_at WHERE create_date IS NULL;
+6 -33
View File
@@ -22,8 +22,10 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<qqq.snapshot.version>4.1.0-RC.5</qqq.snapshot.version>
<revision>1.0.0-SNAPSHOT</revision>
<qqq.version>4.1.0-RC.26</qqq.version> <!-- feeds qbit-build-parent's qqq-bom-pom import -->
<!-- QQQ 4.1.0-RC.15 ships UI API 1.6.0, field access and audit value protection -->
<qqq.snapshot.version>4.1.0-RC.26</qqq.snapshot.version>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<jacoco.version>0.8.15</jacoco.version>
@@ -33,7 +35,7 @@
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
<version>2.2.1</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -91,7 +93,7 @@
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-frontend-next</artifactId>
<version>1.0.0-RC.14</version>
<version>1.0.0-RC.38</version>
<scope>test</scope>
</dependency>
<dependency>
@@ -215,33 +217,4 @@
</dependency>
</dependencies>
</dependencyManagement>
<profiles>
<!-- STAND-IN(core=QRun-IO/qqq#1128, removeBy=RC.6): fold these roots into main/test at RC.6. -->
<profile>
<id>uiapi</id>
<activation><property><name>uiapi</name><value>true</value></property></activation>
<build><plugins><plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<sourceDirectories>
<sourceDirectory>${project.basedir}/src/main/java</sourceDirectory>
<sourceDirectory>${project.basedir}/src/main/java-uiapi</sourceDirectory>
</sourceDirectories>
</configuration>
</plugin><plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.6.1</version>
<executions>
<execution><id>uiapi-source</id><phase>generate-sources</phase><goals><goal>add-source</goal></goals>
<configuration><sources><source>src/main/java-uiapi</source></sources></configuration>
</execution>
<execution><id>uiapi-tests</id><phase>generate-test-sources</phase><goals><goal>add-test-source</goal></goals>
<configuration><sources><source>src/test/java-uiapi</source></sources></configuration>
</execution>
</executions>
</plugin></plugins></build>
</profile>
</profiles>
</project>
@@ -1,166 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Proxy;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import com.kingsrook.qbits.secureddata.standin.FieldAccessDecision;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEvent;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEventType;
import com.kingsrook.qbits.secureddata.standin.FieldAccessOutcome;
import com.kingsrook.qbits.secureddata.standin.FieldAccessResolverInterface;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** The only boundary between the temporary field-access API and unreleased core.
*******************************************************************************/
public final class CoreFieldAccessBridge
{
private static final String PACKAGE = "com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.";
private static final QLogger LOG = QLogger.getLogger(CoreFieldAccessBridge.class);
private static final AtomicBoolean WARNED = new AtomicBoolean();
/*******************************************************************************
** Utility class.
*******************************************************************************/
private CoreFieldAccessBridge()
{
}
/*******************************************************************************
** Register only when core can actually enforce field access.
*******************************************************************************/
public static boolean register(QInstance instance, FieldAccessResolverInterface resolver)
{
try
{
Class<?> metadata = Class.forName(PACKAGE + "FieldAccessResolverMetaData");
Class<?> contract = Class.forName(PACKAGE + "FieldAccessResolverInterface");
Object proxy = Proxy.newProxyInstance(contract.getClassLoader(), new Class<?>[] {contract}, (self, method, args) ->
{
return (switch(method.getName())
{
case "resolve" -> decision(resolver.resolve((QTableMetaData) args[0], (QFieldMetaData) args[1]));
case "getAccessFingerprint" -> resolver.getAccessFingerprint();
case "isRevealActive" -> resolver.isRevealActive((String) args[0], (String) args[1], (Serializable) args[2]);
case "fingerprintSearchValue" -> resolver.fingerprintSearchValue((Serializable) args[0]);
case "checkExactMatch" ->
{
resolver.checkExactMatch(event(args[0]));
yield null;
}
case "onAccess" ->
{
resolver.onAccess(event(args[0]));
yield null;
}
case "toString" -> "SecuredData core bridge";
case "hashCode" -> System.identityHashCode(self);
case "equals" -> self == args[0];
default -> throw new UnsupportedOperationException(method.getName());
});
});
Object registry = metadata.getMethod("ofOrWithNew", QInstance.class).invoke(null, instance);
metadata.getMethod("withResolverInstance", contract).invoke(registry, proxy);
return (true);
}
catch(ClassNotFoundException e)
{
if(WARNED.compareAndSet(false, true))
{
LOG.info("Secured-data field enforcement needs QQQ RC.6+; policy and access-log services are available.");
}
return (false);
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Core field-access API is incompatible", unwrap(e)));
}
}
/*******************************************************************************
** Construct the core record by its declared component types and enum names.
*******************************************************************************/
private static Object decision(FieldAccessDecision decision) throws ReflectiveOperationException
{
Class<?> type = Class.forName(PACKAGE + "FieldAccessDecision");
Class<?> tier = type.getRecordComponents()[0].getType();
Class<?> mask = type.getRecordComponents()[1].getType();
return (type.getConstructor(tier, mask, String.class, boolean.class).newInstance(
enumValue(tier, decision.tier().name()), decision.maskStyle() == null ? null : enumValue(mask, decision.maskStyle().name()),
decision.reason(), decision.exactMatchAllowed()));
}
/*******************************************************************************
** Convert enum types without sharing class identity across the bridge.
*******************************************************************************/
private static Object enumValue(Class<?> type, String name) throws ReflectiveOperationException
{
return (type.getMethod("valueOf", String.class).invoke(null, name));
}
/*******************************************************************************
** Events carry identifiers and keyed fingerprints, never search literals.
*******************************************************************************/
@SuppressWarnings("unchecked")
private static FieldAccessEvent event(Object source) throws ReflectiveOperationException
{
Class<?> type = source.getClass();
Object outcome = type.getMethod("getOutcome").invoke(source);
return (new FieldAccessEvent()
.withType(FieldAccessEventType.valueOf(type.getMethod("getType").invoke(source).toString()))
.withTableName((String) type.getMethod("getTableName").invoke(source))
.withFieldNames((List<String>) type.getMethod("getFieldNames").invoke(source))
.withPrimaryKeys((List<Serializable>) type.getMethod("getPrimaryKeys").invoke(source))
.withRecordCount((Integer) type.getMethod("getRecordCount").invoke(source))
.withActionName((String) type.getMethod("getActionName").invoke(source))
.withOutcome(outcome == null ? null : FieldAccessOutcome.valueOf(outcome.toString()))
.withSearchValueFingerprints((List<String>) type.getMethod("getSearchValueFingerprints").invoke(source)));
}
/*******************************************************************************
** Preserve the startup failure, including duplicate resolver registration.
*******************************************************************************/
private static Throwable unwrap(ReflectiveOperationException exception)
{
return (exception instanceof InvocationTargetException invocation ? invocation.getCause() : exception);
}
}
@@ -0,0 +1,103 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.List;
import java.util.function.Supplier;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** One kind of rule rows a node keeps in memory (classifications, policies or
** active retention rules): read once per version, never used unread past
** SecuredDataRuntime.MAX_RULE_AGE, and safe against a read that asks for the
** same rows again.
**
** Not synchronized: every call must be made holding the owner's lock (the
** runtime's rule lock for classifications and retention rules, the resolver's
** monitor for policies). That lock is what makes the re-entry answer below
** private to one thread: while a read is running, only the thread running it
** can get in here; every other thread waits for the lock and is given the rows
** that read returns.
**
** Re-entry: reading rule rows is itself a query, and anything that runs inside
** it (a table customizer, a record-security lookup, a value provider) may ask a
** question that needs the very rows being read. Such a nested question is
** answered with the rows this node held when the read began - the answer every
** request was given until now, and no rows before the first read - and never
** starts another read. The request that started the read gets the new rows.
**
** Failure: a read that throws leaves the previous rows and version in place
** and is not remembered, so the next question reads again. The previous rows
** are never handed to a request in place of rows that could not be read.
*******************************************************************************/
final class RuleRows
{
private List<QRecord> rows = List.of();
private long version;
private long loadedAt;
private boolean loaded;
private boolean reading;
/*******************************************************************************
** The rows of this version, reading them when the rows held are of another
** version, too old, or were never read.
**
** @param currentVersion the owner's invalidation clock, read by the caller
** before this call: a change during the read leaves a
** version that no longer matches, so the next question
** reads again
** @param read the query; whatever it throws is thrown to the caller
*******************************************************************************/
List<QRecord> get(long currentVersion, Supplier<List<QRecord>> read)
{
if(loaded && version == currentVersion && !SecuredDataRuntime.tooOld(loadedAt))
{
return (rows);
}
if(reading)
{
///////////////////////////////////////////////////////////////////////////
// the read that is running on this thread asked for its own rows: give //
// it the rows held when it began, and do not start another read //
///////////////////////////////////////////////////////////////////////////
return (rows);
}
reading = true;
try
{
long startedAt = System.nanoTime();
List<QRecord> fresh = List.copyOf(read.get());
rows = fresh;
version = currentVersion;
loadedAt = startedAt;
loaded = true;
}
finally
{
reading = false;
}
return (rows);
}
}
@@ -21,6 +21,7 @@
package com.kingsrook.qbits.secureddata;
import java.lang.ref.WeakReference;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayDeque;
@@ -38,11 +39,11 @@ import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEvent;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEventType;
import com.kingsrook.qbits.secureddata.standin.FieldAccessOutcome;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
@@ -60,7 +61,6 @@ import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Bounded asynchronous value-free audit, with persisted and local enumeration accounting.
*******************************************************************************/
@@ -91,7 +91,12 @@ public class SecuredDataAccessLogger implements AutoCloseable
/*******************************************************************************
** One daemon drains this instance's queue; the JVM hook flushes on shutdown.
** One daemon drains this instance's queue; the optional JVM hook flushes on
** shutdown. Neither holds the logger strongly: once the host drops its
** instance (a test, a reloaded instance), the logger, runtime and instance
** can be collected, and the worker stops at its next tick. Hosts that close
** the runtime themselves can turn the hook off
** (SecuredDataQBitConfig.withRegisterShutdownHook(false)).
*******************************************************************************/
public SecuredDataAccessLogger(QInstance instance, SecuredDataRuntime runtime)
{
@@ -104,10 +109,69 @@ public class SecuredDataAccessLogger implements AutoCloseable
thread.setDaemon(true);
return (thread);
});
WeakReference<SecuredDataAccessLogger> self = new WeakReference<>(this);
long interval = runtime.config().getAccessLogFlushIntervalMillis();
worker.scheduleWithFixedDelay(this::flush, interval, interval, TimeUnit.MILLISECONDS);
shutdownHook = new Thread(this::close, "secured-data-access-log-shutdown");
Runtime.getRuntime().addShutdownHook(shutdownHook);
worker.scheduleWithFixedDelay(new WeakTick(self, worker), interval, interval, TimeUnit.MILLISECONDS);
shutdownHook = runtime.config().getRegisterShutdownHook() ? new Thread(new WeakClose(self), "secured-data-access-log-shutdown") : null;
if(shutdownHook != null)
{
Runtime.getRuntime().addShutdownHook(shutdownHook);
}
}
/*******************************************************************************
** The worker's task, holding the logger weakly: when the logger has been
** collected the worker shuts itself down.
*******************************************************************************/
private record WeakTick(WeakReference<SecuredDataAccessLogger> logger, ScheduledExecutorService worker) implements Runnable
{
/*******************************************************************************
** Drain, or stop once the logger is gone.
*******************************************************************************/
@Override
public void run()
{
SecuredDataAccessLogger target = logger.get();
if(target == null)
{
worker.shutdown();
return;
}
target.scheduledFlush();
}
}
/*******************************************************************************
** The JVM shutdown hook, holding the logger weakly.
*******************************************************************************/
private record WeakClose(WeakReference<SecuredDataAccessLogger> logger) implements Runnable
{
/*******************************************************************************
** Drain accepted rows at JVM shutdown, if the logger still exists.
*******************************************************************************/
@Override
public void run()
{
SecuredDataAccessLogger target = logger.get();
if(target != null)
{
target.close();
}
}
}
/*******************************************************************************
** Whether the background worker is still running (false after close).
*******************************************************************************/
public boolean isWorkerRunning()
{
return (!worker.isShutdown());
}
@@ -179,7 +243,7 @@ public class SecuredDataAccessLogger implements AutoCloseable
.withValue("user_name", session == null || session.getUser() == null ? null : session.getUser().getFullName())
.withValue("table_name", event.getTableName()).withValue("field_name", String.join(",", fields))
.withValue("record_count", event.getRecordCount()).withValue("action_name", event.getActionName())
.withValue("client_channel", CoreMetadataBridge.channel(session)).withValue("outcome", outcome)
.withValue("client_channel", SecuredDataMetadata.channel(session)).withValue("outcome", outcome)
.withValue("anomaly_flag", false).withValue("created_at", runtime.config().getClock().instant());
if(event.getPrimaryKeys() != null)
{
@@ -333,13 +397,38 @@ public class SecuredDataAccessLogger implements AutoCloseable
/*******************************************************************************
** The worker's tick. An exception would cancel every later run of a
** scheduled task, so a failed drain is logged (value-free) and retried on
** the next tick; the rows stay queued.
*******************************************************************************/
void scheduledFlush()
{
try
{
flush();
}
catch(RuntimeException e)
{
LOG.warn("Unable to drain the secured-data access log; retrying on the next tick");
}
}
/*******************************************************************************
** Drain safely both from the worker and explicit test/host shutdown calls.
** An empty queue opens no context: QContext validates an unvalidated
** instance, and the worker starts while the host is still building it.
*******************************************************************************/
public void flush()
{
synchronized(flushLock)
{
if(queue.isEmpty())
{
return;
}
QContext.withTemporaryContext(new CapturedContext(instance, new QSystemUserSession(instance)), () ->
{
PendingLog pending;
@@ -464,7 +553,7 @@ public class SecuredDataAccessLogger implements AutoCloseable
}
worker.shutdown();
flush();
if(Thread.currentThread() != shutdownHook)
if(shutdownHook != null && Thread.currentThread() != shutdownHook)
{
try
{
@@ -0,0 +1,63 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.audits.AuditValueProtectorInterface;
import com.kingsrook.qqq.backend.core.actions.audits.DMLAuditAction.DMLType;
import com.kingsrook.qqq.backend.core.actions.audits.ProtectedAuditValues;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Classifications and policy determine whether audits retain a keyed derivative.
*******************************************************************************/
public final class SecuredDataAuditProtector implements AuditValueProtectorInterface
{
/*******************************************************************************
** Core also uses this source for runtime classification overrides.
*******************************************************************************/
@Override
public DataClassification classificationOf(QTableMetaData table, QFieldMetaData field)
{
DataClassification classification = SecuredDataClassifications.effective(table, field.getName());
if(com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.ALL.contains(table.getName()))
{
return (classification);
}
return (RetentionRules.active(table.getName(), field.getName()).isEmpty() ? classification : DataClassification.max(classification, DataClassification.PII));
}
/*******************************************************************************
** Null stays absent; CHANGED_ONLY never creates a DEK.
*******************************************************************************/
@Override
public ProtectedAuditValues protect(QTableMetaData table, QFieldMetaData field, Serializable pk, Serializable oldValue, Serializable newValue, DMLType dmlType)
{
boolean changed = RetentionRules.changedOnly(table.getName(), field.getName());
String message = (oldValue == null ? "Set " : newValue == null ? "Removed " : "Changed ") + field.getLabel();
return (new ProtectedAuditValues(oldValue == null ? null : changed ? "changed:v1" : ClassifiedValueTokens.forRecordField(table, field, pk, oldValue),
newValue == null ? null : changed ? "changed:v1" : ClassifiedValueTokens.forRecordField(table, field, pk, newValue), message));
}
}
@@ -23,8 +23,8 @@ package com.kingsrook.qbits.secureddata;
import java.util.Collection;
import java.util.Objects;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -54,9 +54,13 @@ public final class SecuredDataClassifications
{
return (DataClassification.valueOf(row.getValueString("classification")));
}
QFieldMetaData field = table.getField(fieldName);
DataClassification classification = field == null ? null : CoreMetadataBridge.classification(table, field);
return (classification == null ? CoreMetadataBridge.classification(table, null) : classification);
////////////////////////////////////////////////////////////////////
// getFields, not getField: a virtual field is not in the map and //
// getField throws, while it still takes the table classification //
////////////////////////////////////////////////////////////////////
QFieldMetaData field = table.getFields() == null || fieldName == null ? null : table.getFields().get(fieldName);
DataClassification classification = field == null ? null : SecuredDataMetadata.classification(table, field);
return (classification == null ? SecuredDataMetadata.classification(table, null) : classification);
}
@@ -67,7 +71,7 @@ public final class SecuredDataClassifications
public static DataClassification effective(QTableMetaData table)
{
QRecord row = row(table, null);
return (row == null ? CoreMetadataBridge.classification(table, null) : DataClassification.valueOf(row.getValueString("classification")));
return (row == null ? SecuredDataMetadata.classification(table, null) : DataClassification.valueOf(row.getValueString("classification")));
}
@@ -126,4 +130,12 @@ public final class SecuredDataClassifications
}
return (tableRow);
}
/*******************************************************************************
** Shared phase-b entropy guardrail, including runtime overrides.
*******************************************************************************/
public static boolean lowEntropy(QTableMetaData table, String field)
{
return (SecuredDataResolver.lowCardinality(table.getField(field), row(table, field)));
}
}
@@ -0,0 +1,291 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import javax.crypto.Cipher;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.Serializable;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.time.Instant;
import java.util.Arrays;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySource;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Per-instance audit DEKs. Cache hits still require the key row to exist, so
** another node's committed deletion takes effect immediately on the next lookup.
*******************************************************************************/
public final class SecuredDataKeySource implements ClassifiedValueKeySource, AutoCloseable
{
private final ThreadLocal<Boolean> instanceOnly = ThreadLocal.withInitial(() -> false);
private static final SecureRandom RANDOM = new SecureRandom();
private final Map<String, CachedKey> cache = new LinkedHashMap<>();
/*******************************************************************************
** Short-lived cache; never populated by uncommitted work.
*******************************************************************************/
private record CachedKey(byte[] key, Instant expires)
{
}
/*******************************************************************************
** Use core's interpreter directly: convenience environment getters log values.
*******************************************************************************/
public static String environment(String name)
{
return (new QMetaDataVariableInterpreter().interpret("${prop." + name + "}??${env." + name + "}"));
}
/*******************************************************************************
** Instance HMAC version, without retaining or logging configuration text.
*******************************************************************************/
@Override
public Optional<KeyAndId> instanceKey()
{
String encoded = environment("QQQ_AUDIT_HMAC_KEY");
if(encoded == null || encoded.isBlank())
{
return (Optional.empty());
}
try
{
String id = environment("QQQ_AUDIT_HMAC_KEY_ID");
return (Optional.of(new KeyAndId(Base64.getDecoder().decode(encoded), id == null || id.isBlank() ? "env1" : id)));
}
catch(IllegalArgumentException e)
{
throw (new IllegalStateException("Invalid audit HMAC key configuration"));
}
}
/*******************************************************************************
** Policies and entropy share the phase-b classification rules.
*******************************************************************************/
public boolean shreds(String table, String field)
{
var metadata = QContext.getQInstance().getTable(table);
DataClassification classification = SecuredDataClassifications.effective(metadata, field);
return (RetentionRules.active(table, field).stream().anyMatch(rule -> "SHREDDABLE".equals(rule.getValueString("audit_treatment")))
|| ((classification == DataClassification.PII || classification == DataClassification.PHI) && SecuredDataClassifications.lowEntropy(metadata, field)));
}
/*******************************************************************************
** Verification never creates. Missing KEK fails closed rather than falling back
** to an instance HMAC for a field that must be shredded.
*******************************************************************************/
@Override
public synchronized Optional<KeyAndId> keyForRecordField(String table, String field, Serializable pk, boolean create)
{
if(instanceOnly.get())
{
return (Optional.empty());
}
String binding = table + "\u001f" + field + "\u001f" + pk;
try
{
List<QRecord> rows = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)
.withFilter(recordField(table, field, pk)).withShouldOmitHiddenFields(false).withFieldNamesToInclude(SystemWrites.storedFields(SecuredDataTables.AUDIT_KEY))
.withTransaction(QContext.getQBackendTransaction())).getRecords());
if(rows.isEmpty())
{
evict(binding);
if(!create || RetentionRules.changedOnly(table, field) || !shreds(table, field))
{
return (Optional.empty());
}
byte[] dek = new byte[32];
RANDOM.nextBytes(dek);
try
{
String wrapped = Base64.getEncoder().encodeToString(encrypt(dek, binding));
QRecord row = SystemWrites.insertOne(SecuredDataTables.AUDIT_KEY, new QRecord()
.withValue("table_name", table).withValue("field_name", field).withValue("record_id", String.valueOf(pk))
.withValue("wrapped_key", wrapped).withValue("created_at", SecuredDataRuntime.current().config().getClock().instant()));
return (Optional.of(new KeyAndId(dek.clone(), "dek" + row.getValueString("id"))));
}
finally
{
Arrays.fill(dek, (byte) 0);
}
}
QRecord row = rows.getFirst();
Instant now = SecuredDataRuntime.current().config().getClock().instant();
CachedKey cached = cache.get(binding);
if(cached != null && cached.expires().isAfter(now) && QContext.getQBackendTransaction() == null)
{
return (Optional.of(new KeyAndId(cached.key().clone(), "dek" + row.getValueString("id"))));
}
evict(binding);
byte[] dek = decrypt(Base64.getDecoder().decode(row.getValueString("wrapped_key")), binding);
try
{
if(QContext.getQBackendTransaction() == null)
{
if(cache.size() >= 128)
{
close();
}
cache.put(binding, new CachedKey(dek.clone(), now.plusSeconds(5)));
}
return (Optional.of(new KeyAndId(dek.clone(), "dek" + row.getValueString("id"))));
}
finally
{
Arrays.fill(dek, (byte) 0);
}
}
catch(QException e)
{
throw (new IllegalStateException("Unable to access audit key storage"));
}
}
/*******************************************************************************
** Exact key identity, shared by deletion and lookups.
*******************************************************************************/
public static QQueryFilter recordField(String table, String field, Serializable pk)
{
return (new QQueryFilter().withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table))
.withCriteria(new QFilterCriteria("field_name", QCriteriaOperator.EQUALS, field))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, String.valueOf(pk))));
}
/*******************************************************************************
** Erase cached key bytes, including on rollback and runtime shutdown.
*******************************************************************************/
private void evict(String binding)
{
CachedKey old = cache.remove(binding);
if(old != null)
{
Arrays.fill(old.key(), (byte) 0);
}
}
/*******************************************************************************
** AES-GCM envelope: version byte, 12-byte random nonce, ciphertext and tag.
** The binding is authenticated additional data and prevents key-row swapping.
*******************************************************************************/
public static byte[] encrypt(byte[] plaintext, String binding)
{
byte[] nonce = new byte[12];
RANDOM.nextBytes(nonce);
byte[] ciphertext = crypt(Cipher.ENCRYPT_MODE, plaintext, nonce, binding);
return (ByteBuffer.allocate(13 + ciphertext.length).put((byte) 1).put(nonce).put(ciphertext).array());
}
/*******************************************************************************
** Authenticate before exposing any plaintext.
*******************************************************************************/
public static byte[] decrypt(byte[] envelope, String binding)
{
if(envelope.length < 29 || envelope[0] != 1)
{
throw (new IllegalArgumentException("Invalid encrypted envelope"));
}
return (crypt(Cipher.DECRYPT_MODE, Arrays.copyOfRange(envelope, 13, envelope.length), Arrays.copyOfRange(envelope, 1, 13), binding));
}
/*******************************************************************************
** Require exactly a 256-bit KEK; never include secret text in failures.
*******************************************************************************/
private static byte[] crypt(int mode, byte[] input, byte[] nonce, String binding)
{
byte[] kek = null;
try
{
String encoded = environment("QQQ_SECURED_DATA_KEK");
kek = encoded == null ? new byte[0] : Base64.getDecoder().decode(encoded);
if(kek.length != 32)
{
throw (new IllegalArgumentException("QQQ_SECURED_DATA_KEK must be base64 for 32 bytes"));
}
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
cipher.init(mode, new SecretKeySpec(kek, "AES"), new GCMParameterSpec(128, nonce));
cipher.updateAAD(binding.getBytes(StandardCharsets.UTF_8));
return (cipher.doFinal(input));
}
catch(GeneralSecurityException | IllegalArgumentException e)
{
throw (new IllegalStateException("Unable to protect audit data; check QQQ_SECURED_DATA_KEK and envelope integrity"));
}
finally
{
if(kek != null)
{
Arrays.fill(kek, (byte) 0);
}
}
}
/*******************************************************************************
** Clear every decrypted cached key.
*******************************************************************************/
@Override
public synchronized void close()
{
cache.values().forEach(key -> Arrays.fill(key.key(), (byte) 0));
cache.clear();
}
/*******************************************************************************
** KEYED_HASH deliberately uses only the instance key and record binding.
*******************************************************************************/
public String instanceToken(com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData table,
com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData field, Serializable pk, Serializable value)
{
if(instanceKey().isEmpty())
{
throw (new IllegalStateException("KEYED_HASH requires QQQ_AUDIT_HMAC_KEY"));
}
instanceOnly.set(true);
try
{
return (ClassifiedValueTokens.forRecordField(table, field, pk, value));
}
finally
{
instanceOnly.remove();
}
}
}
@@ -29,13 +29,12 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Configuration for policy resolution and bounded asynchronous access logging.
*******************************************************************************/
@@ -49,10 +48,74 @@ public class SecuredDataQBitConfig implements QBitConfig
private int anomalyExactMatchesPerHour = 50;
private String tablePathPrefix = "/app";
private String schedulerName;
private Integer backupRetentionDays;
private int wipeIntervalSeconds = 86400;
private com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface archiveSink;
/*******************************************************************************
** Known CNPG/WAL backup retention; null means unknown.
*******************************************************************************/
public Integer getBackupRetentionDays()
{
return (backupRetentionDays);
}
/*******************************************************************************
** Configure the physical backup recovery window.
*******************************************************************************/
public SecuredDataQBitConfig withBackupRetentionDays(Integer days)
{
if(days != null && days < 0)
{
throw (new IllegalArgumentException("Backup retention cannot be negative"));
}
backupRetentionDays = days;
return (this);
}
/*******************************************************************************
** Optional archive adapter.
*******************************************************************************/
public com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface getArchiveSink()
{
return (archiveSink);
}
/*******************************************************************************
** Supply a durable encrypted-object sink.
*******************************************************************************/
public SecuredDataQBitConfig withArchiveSink(com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface sink)
{
archiveSink = sink;
return (this);
}
/*******************************************************************************
** Daily by default.
*******************************************************************************/
public int getWipeIntervalSeconds()
{
return (wipeIntervalSeconds);
}
/*******************************************************************************
** Configure the host scheduler interval.
*******************************************************************************/
public SecuredDataQBitConfig withWipeIntervalSeconds(int seconds)
{
if(seconds < 1)
{
throw (new IllegalArgumentException("Wipe interval must be positive"));
}
wipeIntervalSeconds = seconds;
return (this);
}
private String backendName = "memory";
private Clock clock = Clock.systemUTC();
private int accessLogQueueCapacity = 4096;
private long accessLogFlushIntervalMillis = 250;
private boolean registerShutdownHook = true;
private int exactMatchPerHour = 20;
private int missAlertConsecutive = 5;
private int distinctValueAlert = 15;
@@ -154,6 +217,29 @@ public class SecuredDataQBitConfig implements QBitConfig
/*******************************************************************************
** Whether the access logger registers a JVM shutdown hook that drains
** accepted rows (default true). Hosts that call SecuredDataRuntime.close()
** from their own shutdown sequence turn it off.
*******************************************************************************/
public boolean getRegisterShutdownHook()
{
return (registerShutdownHook);
}
/*******************************************************************************
** Configure registerShutdownHook.
*******************************************************************************/
public SecuredDataQBitConfig withRegisterShutdownHook(boolean registerShutdownHook)
{
this.registerShutdownHook = registerShutdownHook;
return (this);
}
/*******************************************************************************
** Configure accessLogFlushIntervalMillis.
*******************************************************************************/
@@ -27,9 +27,12 @@ import java.util.Locale;
import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTableCustomizers;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.audits.AuditValueProtectionMetaData;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySourceMetaData;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverMetaData;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -39,7 +42,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Entry point for secured-data policy, classification and access logging.
*******************************************************************************/
@@ -47,7 +49,7 @@ public class SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredData
{
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-secured-data";
public static final String VERSION = "1.0.0";
public static final String VERSION = "0.1.0";
private SecuredDataQBitConfig qBitConfig;
private String namespace;
@@ -153,21 +155,27 @@ public class SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredData
output.add(com.kingsrook.qbits.secureddata.app.SecuredDataApp.app(tables));
output.add(com.kingsrook.qbits.secureddata.app.SecuredDataApp.widget());
com.kingsrook.qbits.secureddata.processes.BreakGlassProcesses.processes(qBitConfig).forEach(output::add);
com.kingsrook.qbits.secureddata.retention.WipeProcesses.processes(qBitConfig).forEach(output::add);
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.backups());
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.markersJoin());
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.markers());
output.add(SecuredDataTables.revealsJoin());
output.add(SecuredDataTables.revealsWidget());
output.add(new com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType().withName(com.kingsrook.qbits.secureddata.metadata.RetentionTables.KEY_LOCK));
output.addSelfToInstance(qInstance);
CoreMetadataBridge.register(qInstance);
com.kingsrook.qbits.secureddata.support.UiApiBridge.register(qInstance);
SecuredDataMetadata.register(qInstance);
com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension.register(qInstance);
if(qBitConfig.getNameHeuristics())
{
seedHeuristics(qInstance);
}
CoreMetadataBridge.promoteClassifications(qInstance);
SecuredDataRuntime runtime = new SecuredDataRuntime(qInstance, qBitConfig);
runtime.addSelfToInstance(qInstance);
try
{
CoreFieldAccessBridge.register(qInstance, runtime.resolver());
FieldAccessResolverMetaData.ofOrWithNew(qInstance).withResolverInstance(runtime.resolver());
ClassifiedValueKeySourceMetaData.ofOrWithNew(qInstance).withKeySourceInstance(runtime.keys());
AuditValueProtectionMetaData.ofOrWithNew(qInstance).withProtectorInstance(new SecuredDataAuditProtector());
}
catch(RuntimeException e)
{
@@ -192,18 +200,18 @@ public class SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredData
}
for(QFieldMetaData field : table.getFields().values())
{
if(CoreMetadataBridge.classification(table, field) != null || CoreMetadataBridge.classification(table, null) != null)
if(SecuredDataMetadata.classification(table, field) != null || SecuredDataMetadata.classification(table, null) != null)
{
continue;
}
String name = field.getName().toLowerCase(Locale.ROOT);
if(name.startsWith("diagnosis"))
{
CoreMetadataBridge.classifyField(field, DataClassification.PHI);
SecuredDataMetadata.classifyField(field, DataClassification.PHI);
}
else if(List.of("ssn", "dob", "date_of_birth", "email", "phone").contains(name))
{
CoreMetadataBridge.classifyField(field, DataClassification.PII);
SecuredDataMetadata.classifyField(field, DataClassification.PII);
}
}
}
@@ -36,14 +36,14 @@ import java.util.Objects;
import java.util.TreeSet;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessDecision;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEvent;
import com.kingsrook.qbits.secureddata.standin.FieldAccessResolverInterface;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.standin.FieldMaskStyle;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessDecision;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverInterface;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
@@ -57,7 +57,6 @@ import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Seven-step policy resolution with instance-versioned, bounded per-user state.
*******************************************************************************/
@@ -75,10 +74,14 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
{
}
private long cachedVersion;
/** Used only inside methods synchronized on this resolver. */
private final RuleRows policyRows = new RuleRows();
private List<QRecord> policies = List.of();
private final Map<String, UserState> users = new HashMap<>();
/** Request-scoped (QContext) holder of the access fingerprint. */
private static final String FINGERPRINT_KEY = SecuredDataResolver.class.getName() + ".accessFingerprint";
/*******************************************************************************
@@ -156,12 +159,12 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
{
tier = runtime.config().defaultTier(classification);
}
if(session != null && CoreMetadataBridge.hasFeature(table, SecuredDataPermissions.REVEAL))
if(session != null && SecuredDataMetadata.hasFeature(table, SecuredDataPermissions.REVEAL))
{
tier = tier.leastRestrictive(FieldAccessTier.REVEALABLE);
}
if(activeBreakGlass(table.getName(), classification) != null
|| (session != null && CoreMetadataBridge.hasFeature(table, SecuredDataPermissions.FULL)))
|| (session != null && SecuredDataMetadata.hasFeature(table, SecuredDataPermissions.FULL)))
{
tier = FieldAccessTier.FULL;
}
@@ -268,7 +271,7 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
*******************************************************************************/
private List<QRecord> matching(QTableMetaData table, DataClassification classification)
{
String channel = CoreMetadataBridge.channel(QContext.getQSession());
String channel = SecuredDataMetadata.channel(QContext.getQSession());
return (state().policies().stream().filter(row -> classification != null && classification.name().equals(row.getValueString("classification")))
.filter(row -> blank(row.getValueString("table_name")) || table.getName().equals(row.getValueString("table_name")))
.filter(row -> blank(row.getValueString("client_channel")) || Objects.equals(channel, row.getValueString("client_channel"))).toList());
@@ -289,16 +292,19 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
/*******************************************************************************
** Refresh policy rows once per version and matched state per relevant permissions.
** A read that fails throws and is tried again by the next call. A question
** asked from inside the read itself (see RuleRows) is answered from the
** policies held when the read began; whatever it matched for its user is
** dropped with every other user's state when the read returns.
*******************************************************************************/
private synchronized UserState state()
{
long version = runtime.policyVersion();
if(cachedVersion != version)
policies = policyRows.get(runtime.policyVersion(), () ->
{
policies = List.copyOf(SystemWrites.query(SecuredDataTables.POLICY, new QQueryFilter()));
List<QRecord> rows = SystemWrites.query(SecuredDataTables.POLICY, new QQueryFilter());
users.clear();
cachedVersion = version;
}
return (rows);
});
QSession session = QContext.getQSession();
TreeSet<String> held = new TreeSet<>();
for(QRecord policy : policies)
@@ -309,7 +315,7 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
held.add(permission);
}
}
String key = JsonUtils.toJson(List.of(userId(), held));
String key = userId() + '\u0000' + String.join("\u0000", held);
UserState state = users.get(key);
if(state == null)
{
@@ -357,11 +363,82 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
** Stable across unrelated permissions/records; includes channel and exact expiry.
*******************************************************************************/
@Override
public synchronized String getAccessFingerprint()
public String getAccessFingerprint()
{
state();
TreeSet<String> relevant = new TreeSet<>();
/////////////////////////////////////////////////////////////////////////////
// core asks for the fingerprint before every field decision (once per //
// field of every record of a result), so the value is kept for the //
// request - and only while everything it is derived from is unchanged //
/////////////////////////////////////////////////////////////////////////////
QSession session = QContext.getQSession();
String channel = Objects.requireNonNullElse(SecuredDataMetadata.channel(session), "");
Instant now = runtime.config().getClock().instant();
RequestFingerprint kept = QContext.getObject(FINGERPRINT_KEY, RequestFingerprint.class).orElse(null);
if(kept != null && kept.matches(this, session, channel, runtime.policyVersion(), runtime.classificationVersion(), now))
{
return (kept.value());
}
RequestFingerprint computed = computeAccessFingerprint(session, channel, now);
QContext.setObject(FINGERPRINT_KEY, computed);
return (computed.value());
}
/*******************************************************************************
** The fingerprint and what it was derived from. It stops matching when the
** session, its permissions, the channel, a policy or classification version
** changes, or when the first active break-glass grant expires.
*******************************************************************************/
private record RequestFingerprint(int resolver, QSession session, int permissionsIdentity, int permissionCount, String channel, long policyVersion,
long classificationVersion, Instant validUntil, String value) implements Serializable
{
/*******************************************************************************
** Whether the kept value is still the one a fresh computation would give.
*******************************************************************************/
boolean matches(SecuredDataResolver owner, QSession current, String currentChannel, long currentPolicyVersion, long currentClassificationVersion, Instant now)
{
return (resolver == System.identityHashCode(owner) && session == current && permissionsIdentity == identityOfPermissions(current) && permissionCount == countOfPermissions(current)
&& channel.equals(currentChannel) && policyVersion == currentPolicyVersion && classificationVersion == currentClassificationVersion && now.isBefore(validUntil));
}
}
/*******************************************************************************
** The permission set object of a session (it is replaced, not edited, when a
** session is given new permissions).
*******************************************************************************/
private static int identityOfPermissions(QSession session)
{
return (session == null || session.getPermissions() == null ? 0 : System.identityHashCode(session.getPermissions()));
}
/*******************************************************************************
** How many permissions a session holds (an edit in place changes it).
*******************************************************************************/
private static int countOfPermissions(QSession session)
{
return (session == null || session.getPermissions() == null ? -1 : session.getPermissions().size());
}
/*******************************************************************************
** Stable across unrelated permissions/records; includes channel and exact expiry.
*******************************************************************************/
private synchronized RequestFingerprint computeAccessFingerprint(QSession session, String channel, Instant now)
{
/////////////////////////////////////////////////////////////////////////////
// versions are read before the rows they describe, so a change during the //
// computation leaves a kept value that no longer matches //
/////////////////////////////////////////////////////////////////////////////
long policyVersion = runtime.policyVersion();
long classificationVersion = runtime.classificationVersion();
TreeSet<String> relevant = new TreeSet<>();
List<QRecord> breakGlass = state().breakGlass();
for(QRecord row : policies)
{
String permission = row.getValueString("permission_name");
@@ -372,7 +449,7 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
}
for(QTableMetaData table : QContext.getQInstance().getTables().values())
{
String base = CoreMetadataBridge.permissionBase(table);
String base = SecuredDataMetadata.permissionBase(table);
for(String suffix : List.of(SecuredDataPermissions.FULL, SecuredDataPermissions.REVEAL))
{
if(session != null && session.hasPermission(base + "." + suffix))
@@ -381,12 +458,15 @@ public class SecuredDataResolver implements FieldAccessResolverInterface
}
}
}
List<String> breaks = activeBreakGlass().stream().map(row -> row.getValueString("id")).sorted().toList();
String payload = JsonUtils.toJson(List.of(userId(), relevant, runtime.policyVersion(), runtime.classificationVersion(), breaks,
Objects.requireNonNullElse(CoreMetadataBridge.channel(session), "")));
List<QRecord> activeBreaks = breakGlass.stream().filter(row -> "ACTIVE".equals(row.getValueString("status")))
.filter(row -> row.getValueInstant("expires_at") != null && row.getValueInstant("expires_at").isAfter(now)).toList();
List<String> breaks = activeBreaks.stream().map(row -> row.getValueString("id")).sorted().toList();
Instant validUntil = activeBreaks.stream().map(row -> row.getValueInstant("expires_at")).min(Instant::compareTo).orElse(Instant.MAX);
String payload = JsonUtils.toJson(List.of(userId(), relevant, policyVersion, classificationVersion, breaks, channel));
try
{
return (Base64.getUrlEncoder().withoutPadding().encodeToString(MessageDigest.getInstance("SHA-256").digest(payload.getBytes(StandardCharsets.UTF_8))));
String value = Base64.getUrlEncoder().withoutPadding().encodeToString(MessageDigest.getInstance("SHA-256").digest(payload.getBytes(StandardCharsets.UTF_8)));
return (new RequestFingerprint(System.identityHashCode(this), session, identityOfPermissions(session), countOfPermissions(session), channel, policyVersion, classificationVersion, validUntil, value));
}
catch(java.security.NoSuchAlgorithmException e)
{
@@ -21,9 +21,21 @@
package com.kingsrook.qbits.secureddata;
import java.lang.ref.WeakReference;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.cluster.ClusterBus;
import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
import com.kingsrook.qqq.backend.core.cluster.ClusterEvent;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
@@ -34,15 +46,58 @@ import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaDa
public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCloseable
{
public static final String NAME = "securedDataRuntime";
private static final QLogger LOG = QLogger.getLogger(SecuredDataRuntime.class);
/////////////////////////////////////////////////////////////////////////////////
// the three sets of rule rows a node keeps in memory. A change clears them on //
// this node at once and, through QQQ's cluster bus (ClusterCaches), on every //
// other node. MAX_RULE_AGE is the safety net for a lost event: no node uses //
// rule rows older than that without reading them again. //
/////////////////////////////////////////////////////////////////////////////////
public static final String POLICY_CACHE = "securedData.policies";
public static final String CLASSIFICATION_CACHE = "securedData.classifications";
public static final String RETENTION_CACHE = "securedData.retentionRules";
public static final Duration MAX_RULE_AGE = Duration.ofSeconds(30);
static
{
ClusterCaches.shared(POLICY_CACHE, POLICY_CACHE, () -> clearOnThisNode(POLICY_CACHE), null);
ClusterCaches.shared(CLASSIFICATION_CACHE, CLASSIFICATION_CACHE, () -> clearOnThisNode(CLASSIFICATION_CACHE), null);
ClusterCaches.shared(RETENTION_CACHE, RETENTION_CACHE, () -> clearOnThisNode(RETENTION_CACHE), null);
}
private static final AtomicLong STARTUP_VERSION = new AtomicLong(System.currentTimeMillis());
private final AtomicLong classificationVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
private final AtomicLong policyVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
private final AtomicLong retentionVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
private final transient WeakReference<QInstance> owner;
private final transient SecuredDataQBitConfig config;
private final transient SecuredDataResolver resolver;
private final transient SecuredDataKeySource keys = new SecuredDataKeySource();
private final transient SecuredDataAccessLogger logger;
private final transient SecuredDataAnomalies anomalies;
private transient long cachedClassificationVersion;
private transient List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows = List.of();
/////////////////////////////////////////////////////////////////////////////////
// both are used only while holding ruleLock: one lock for the two, because //
// reading either may ask for the other. //
// //
// It is a lock of its own, never this runtime's monitor. A reveal and the //
// start, end and expiry of a break-glass session hold the runtime's monitor //
// for their whole operation, and every thread asks for a classification all //
// the time (core asks for every field it decides on and for every criterion //
// of every query), including threads that hold another lock of this QBit: the //
// access-log writer inside the anomaly counters, the resolver, the key //
// source. When the question needed the runtime's monitor, such a thread and //
// a reveal that needed its lock next waited for each other for ever. //
// //
// ruleLock is held only while rule rows are answered or read. A read is a //
// query made as the system: it takes none of the QBit's other locks, so no //
// thread holding ruleLock ever waits for one of them. //
/////////////////////////////////////////////////////////////////////////////////
private final transient Object ruleLock = new Object();
private final transient RuleRows classificationRows = new RuleRows();
private final transient RuleRows activeRetentionRules = new RuleRows();
@@ -52,6 +107,7 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
public SecuredDataRuntime(QInstance instance, SecuredDataQBitConfig config)
{
this.config = config;
this.owner = new WeakReference<>(instance);
this.resolver = new SecuredDataResolver(this);
this.anomalies = new SecuredDataAnomalies(config);
this.logger = new SecuredDataAccessLogger(instance, this);
@@ -136,6 +192,7 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
public void invalidateClassifications()
{
classificationVersion.incrementAndGet();
announce(CLASSIFICATION_CACHE);
}
@@ -146,23 +203,115 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
public void invalidatePolicies()
{
policyVersion.incrementAndGet();
announce(POLICY_CACHE);
}
/*******************************************************************************
** Read classification rows once per version, always as the system.
** A retention rule changed: forget the active rules here and on every node.
*******************************************************************************/
synchronized List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows()
public void invalidateRetentionRules()
{
long version = classificationVersion();
if(cachedClassificationVersion != version)
retentionVersion.incrementAndGet();
announce(RETENTION_CACHE);
}
/*******************************************************************************
** Version of the retention rules this node holds.
*******************************************************************************/
public long retentionVersion()
{
return (retentionVersion.get());
}
/*******************************************************************************
** Whether rows loaded at this System.nanoTime are too old to use unread.
*******************************************************************************/
static boolean tooOld(long loadedAtNanos)
{
return (System.nanoTime() - loadedAtNanos > MAX_RULE_AGE.toNanos());
}
/*******************************************************************************
** Every active retention rule, read once per version and at most MAX_RULE_AGE
** old. A read that fails throws and leaves nothing in place of the rules:
** callers never see "no rules" because the rules could not be read, and the
** next call reads again. A question asked from inside the read itself (see
** RuleRows) is answered with the rules held when the read began.
*******************************************************************************/
public List<QRecord> activeRetentionRules()
{
synchronized(ruleLock)
{
classificationRows = List.copyOf(com.kingsrook.qbits.secureddata.support.SystemWrites.query(
com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CLASSIFICATION, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter()));
cachedClassificationVersion = version;
return (activeRetentionRules.get(retentionVersion(), () -> SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true)))));
}
}
/*******************************************************************************
** Tell the other nodes (this node has already moved its version). A failure
** to send is logged: the other nodes then read again within MAX_RULE_AGE.
*******************************************************************************/
private void announce(String cache)
{
try
{
QInstance instance = owner.get();
if(instance != null)
{
ClusterBus.publish(instance, new ClusterEvent(ClusterEvent.TYPE_MEMO_INVALIDATE, cache, null));
}
}
catch(RuntimeException e)
{
LOG.warn("Could not announce a secured-data rule change; other nodes read again within " + MAX_RULE_AGE.toSeconds() + " seconds", e);
}
}
/*******************************************************************************
** A rule change announced on the cluster bus (also by this node): move the
** version of the runtime of the instance the event arrived on.
*******************************************************************************/
private static void clearOnThisNode(String cache)
{
SecuredDataRuntime runtime = current();
if(runtime == null)
{
return;
}
switch(cache)
{
case POLICY_CACHE -> runtime.policyVersion.incrementAndGet();
case CLASSIFICATION_CACHE -> runtime.classificationVersion.incrementAndGet();
default -> runtime.retentionVersion.incrementAndGet();
}
}
/*******************************************************************************
** Read classification rows once per version, always as the system. A read
** that fails throws and is tried again by the next call. A question asked from
** inside the read itself (see RuleRows) is answered with the rows held when
** the read began, and never starts another read.
*******************************************************************************/
List<QRecord> classificationRows()
{
synchronized(ruleLock)
{
return (classificationRows.get(classificationVersion(), () -> SystemWrites.query(SecuredDataTables.CLASSIFICATION, new QQueryFilter())));
}
return (classificationRows);
}
@@ -173,6 +322,7 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
@Override
public void close()
{
keys.close();
logger.close();
}
@@ -183,4 +333,12 @@ public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCl
{
return (anomalies);
}
/*******************************************************************************
** Instance-scoped audit keys.
*******************************************************************************/
public SecuredDataKeySource keys()
{
return (keys);
}
}
@@ -99,6 +99,14 @@ public class SecuredDataHomeData
List.of("user_name", "table_name", "record_id", "action_name", "created_at"), List.of("User", "Table", "Record", "Action", "When"));
data.recent("Anomaly flags", "No anomaly flags in the last 7 days", anomalies,
List.of("user_name", "event_type", "table_name", "field_name", "created_at"), List.of("User", "Event", "Table", "Field", "When"));
for(QRecord rule : new QueryAction().execute(new QueryInput(SecuredDataTables.RETENTION_RULE)).getRecords())
{
String warning = com.kingsrook.qbits.secureddata.retention.RetentionRules.backupWarning(rule);
if(warning != null)
{
data.backupWarnings.add(rule.getValueString("name") + ": " + warning);
}
}
return (data);
}
@@ -182,4 +190,14 @@ public class SecuredDataHomeData
{
return (sections);
}
private final List<String> backupWarnings = new ArrayList<>();
/*******************************************************************************
** One warning for each policy outlived by known physical backups.
*******************************************************************************/
public List<String> getBackupWarnings()
{
return (backupWarnings);
}
}
@@ -53,6 +53,10 @@ public final class SecuredDataHomeView
.append("\"><div style=\"font-size:1.6rem;font-weight:600\">").append(value)
.append("</div><div>").append(esc(label)).append("</div></div>"));
html.append("</div>");
for(String warning : data.getBackupWarnings())
{
html.append("<p role='alert'>").append(esc(warning)).append("</p>");
}
for(SecuredDataHomeData.Section section : data.getSections())
{
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0;font-size:1rem\">")
@@ -28,10 +28,10 @@ import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
@@ -235,7 +235,7 @@ public class BreakGlassService
{
if(QContext.getQSession() == null || QContext.getQSession().getUser() == null
|| "anonymous".equals(SecuredDataResolver.userId()) || SecuredDataResolver.userId().isBlank()
|| "assistant".equals(CoreMetadataBridge.channel(QContext.getQSession())))
|| "assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
throw (new QPermissionDeniedException("Permission denied."));
}
@@ -0,0 +1,148 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.util.List;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.AUDIT_KEY;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CREATE_DATE;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.LEGAL_HOLD;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.MODIFY_DATE;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.RETENTION_RULE;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_MARKER;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_RUN;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.base;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.createDate;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.field;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.modifyDate;
/*******************************************************************************
** Retention policies, value-free history, legal holds and mutable audit keys.
*******************************************************************************/
public final class RetentionTables
{
public static final String KEY_LOCK = "secured_data_internal_key";
/** A key value no session is ever given, so the lock denies every user (validator requires a non-empty override list). */
public static final String KEY_LOCK_NEVER_GRANTED = "secured_data_never_granted";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionTables()
{
}
/*******************************************************************************
** No backup or WORM registration is made for the key table.
*******************************************************************************/
public static List<QTableMetaData> tables(SecuredDataQBitConfig config)
{
QTableMetaData rule = base(config, RETENTION_RULE, "Retention rules")
.withField(field("name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("field_names", QFieldType.TEXT).withIsRequired(true))
.withField(field("age_field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_fields"))
.withField(field("age_days", QFieldType.INTEGER).withIsRequired(true))
.withField(field("action", QFieldType.STRING).withMaxLength(40).withDefaultValue("NULL_WITH_MARKER"))
.withField(field("overwrite_text", QFieldType.TEXT).withDefaultValue("[wiped per policy]"))
.withField(field("audit_treatment", QFieldType.STRING).withMaxLength(30).withDefaultValue("SHREDDABLE"))
.withField(field("archive_storage_name", QFieldType.STRING).withMaxLength(255))
.withField(field("batch_size", QFieldType.INTEGER).withDefaultValue(500))
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("last_run_at", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("last_run_count", QFieldType.INTEGER).withIsEditable(false))
.withField(createDate("create_date"))
.withField(modifyDate("modify_date"))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
rule.withSection(SectionFactory.defaultT1("id", "name", "table_name", "field_names"));
rule.withSection(SectionFactory.defaultT2("age_field_name", "age_days", "action", "overwrite_text", "audit_treatment", "archive_storage_name", "batch_size", "is_active"));
rule.withSection(SectionFactory.defaultT3("last_run_at", "last_run_count", "create_date", "modify_date", "modified_by"));
rule.withSection(SectionFactory.customT2("backups", new com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon("backup"))
.withLabel("Backups").withWidgetName(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.BACKUPS));
rule.withSection(SectionFactory.customT2("markers", new com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon("delete"))
.withLabel("Wipe markers").withWidgetName(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.MARKERS));
QTableMetaData marker = base(config, WIPE_MARKER, "Wipe marker")
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true))
.withField(field("action", QFieldType.STRING).withMaxLength(40).withIsRequired(true))
.withField(field("wiped_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("run_id", QFieldType.INTEGER).withIsRequired(true))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE))
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
marker.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "rule_id", "action", "wiped_at", "run_id"));
marker.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE));
QTableMetaData hold = base(config, LEGAL_HOLD, "Legal hold")
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255))
.withField(field("reason", QFieldType.TEXT).withIsRequired(true))
.withField(field("placed_by", QFieldType.STRING).withMaxLength(255))
.withField(field("placed_at", QFieldType.DATE_TIME))
.withField(field("released_by", QFieldType.STRING).withMaxLength(255))
.withField(field("released_at", QFieldType.DATE_TIME))
.withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE"))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE));
hold.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "reason", "status"));
hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at", CREATE_DATE, MODIFY_DATE));
QTableMetaData run = base(config, WIPE_RUN, "Wipe run")
.withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true))
.withField(field("started_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("finished_at", QFieldType.DATE_TIME))
.withField(field("status", QFieldType.STRING).withMaxLength(30))
.withField(field("eligible_count", QFieldType.INTEGER))
.withField(field("held_count", QFieldType.INTEGER))
.withField(field("wiped_count", QFieldType.INTEGER))
.withField(field("error_count", QFieldType.INTEGER))
.withField(field("record_ids", QFieldType.STRING).withMaxLength(4000))
.withField(field("message", QFieldType.STRING).withMaxLength(500))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE));
run.withSection(SectionFactory.defaultT1("id", "rule_id", "status", "eligible_count", "held_count", "wiped_count", "error_count"));
run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message", CREATE_DATE, MODIFY_DATE));
QTableMetaData key = base(config, AUDIT_KEY, "Audit key")
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
.withoutCapabilities(Capability.TABLE_EXPORT)
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("wrapped_key", QFieldType.TEXT).withIsRequired(true).withIsSensitive(true).withIsHidden(true)
.withFieldSecurityLock(new FieldSecurityLock().withSecurityKeyType(KEY_LOCK).withDefaultBehavior(FieldSecurityLock.Behavior.DENY).withOverrideValues(List.of(KEY_LOCK_NEVER_GRANTED))))
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE))
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
key.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "created_at"));
key.withSection(SectionFactory.defaultT3(CREATE_DATE, MODIFY_DATE));
return (List.of(rule, marker, hold, run, key));
}
}
@@ -23,15 +23,14 @@ package com.kingsrook.qbits.secureddata.metadata;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.standin.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
/*******************************************************************************
** String-valued possible-value sources shared by UI metadata and the DDL.
*******************************************************************************/
@@ -63,30 +63,71 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
{
for(QRecord record : records)
{
QRecord old = oldRecords.orElse(List.of()).stream().filter(row -> Objects.equals(row.getValue("id"), record.getValue("id"))).findFirst().orElse(null);
///////////////////////////////////////////////////////////////////////////
// ids are compared as text: a request over HTTP names its row by the id //
// in its path, which is text, while the stored id is a number //
///////////////////////////////////////////////////////////////////////////
QRecord old = oldRecords.orElse(List.of()).stream().filter(row -> Objects.equals(row.getValueString("id"), record.getValueString("id"))).findFirst().orElse(null);
QRecord effective = old == null ? new QRecord() : new QRecord(old);
record.getValues().forEach(effective::setValue);
validate(table, record, effective);
Instant now = SecuredDataRuntime.current().config().getClock().instant();
if(old == null)
dates(table, record, old == null, now);
if(SecuredDataTables.LEGAL_HOLD.equals(table))
{
record.setValue("create_date", now);
if(old == null)
{
record.setValue("placed_at", now);
record.setValue("placed_by", SecuredDataResolver.userId());
}
if("RELEASED".equals(effective.getValueString("status")))
{
record.setValue("released_at", now);
record.setValue("released_by", SecuredDataResolver.userId());
}
continue;
}
record.setValue("modify_date", now);
record.setValue("modified_by", SecuredDataResolver.userId());
}
}
else
{
Instant now = SecuredDataRuntime.current().config().getClock().instant();
for(QRecord record : records)
{
dates(table, record, input instanceof InsertInput, now);
}
}
return (records);
}
/*******************************************************************************
** The two dates every table has, set in the write that is already being made:
** a new row gets both, equal, from the clock every other time of this QBit
** comes from, and a changed row a new modify date. Core has the last word on
** a change: the modify date is the record version of its table, and core
** moves it forward after this has run. A row of a table that is only ever
** inserted (the access log, a wipe run) so keeps equal dates.
*******************************************************************************/
private static void dates(String table, QRecord record, boolean inserted, Instant now)
{
if(inserted)
{
record.setValue(SecuredDataTables.createDateField(table), now);
}
record.setValue(SecuredDataTables.modifyDateField(table), now);
}
/*******************************************************************************
** Access logs are append-only even for the system.
*******************************************************************************/
private static void assertWrite(String table, boolean mutation) throws QPermissionDeniedException
{
if((mutation && SecuredDataTables.ACCESS_LOG.equals(table))
if((mutation && List.of(SecuredDataTables.ACCESS_LOG, SecuredDataTables.WIPE_RUN).contains(table))
|| (SecuredDataTables.SYSTEM_WRITTEN.contains(table) && !SystemWrites.active() && !(QContext.getQSession() instanceof QSystemUserSession)))
{
throw (new QPermissionDeniedException("Secured-data activity is system-written; access logs are insert-only."));
@@ -106,7 +147,25 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
{
record.addError(new BadInputStatusMessage("Select an existing table."));
}
if(SecuredDataTables.CLASSIFICATION.equals(table))
if(SecuredDataTables.RETENTION_RULE.equals(table))
{
try
{
com.kingsrook.qbits.secureddata.retention.RetentionRules.validate(effective);
}
catch(IllegalArgumentException e)
{
record.addError(new BadInputStatusMessage(e.getMessage()));
}
}
else if(SecuredDataTables.LEGAL_HOLD.equals(table))
{
if(!List.of("ACTIVE", "RELEASED").contains(effective.getValueString("status")))
{
record.addError(new BadInputStatusMessage("Select ACTIVE or RELEASED."));
}
}
else if(SecuredDataTables.CLASSIFICATION.equals(table))
{
String field = effective.getValueString("field_name");
if(field != null && field.isBlank())
@@ -122,7 +181,7 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
{
for(QRecord row : SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, target))))
{
if(Objects.equals(field, row.getValueString("field_name")) && !Objects.equals(row.getValue("id"), effective.getValue("id")))
if(Objects.equals(field, row.getValueString("field_name")) && !Objects.equals(row.getValueString("id"), effective.getValueString("id")))
{
record.addError(new BadInputStatusMessage("A classification for this table and field already exists."));
}
@@ -206,7 +265,8 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
}
SecuredDataRuntime runtime = SecuredDataRuntime.current();
Runnable invalidate = SecuredDataTables.CLASSIFICATION.equals(table) ? runtime::invalidateClassifications
: SecuredDataTables.POLICY.equals(table) || SecuredDataTables.BREAK_GLASS.equals(table) ? runtime::invalidatePolicies : null;
: SecuredDataTables.POLICY.equals(table) || SecuredDataTables.BREAK_GLASS.equals(table) ? runtime::invalidatePolicies
: SecuredDataTables.RETENTION_RULE.equals(table) ? runtime::invalidateRetentionRules : null;
if(invalidate != null)
{
QBackendTransaction active = transaction == null ? QContext.getQBackendTransaction() : transaction;
@@ -216,6 +276,10 @@ public class SecuredDataTableCustomizers implements TableCustomizerInterface
}
else
{
if(SecuredDataTables.RETENTION_RULE.equals(table))
{
com.kingsrook.qbits.secureddata.retention.RetentionRules.editedInTransaction();
}
active.addAfterCommitCallback(invalidate);
}
}
@@ -25,13 +25,14 @@ import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
@@ -58,9 +59,16 @@ public final class SecuredDataTables
public static final String REVEAL_GRANT = "secured_data_reveal_grant";
public static final String BREAK_GLASS = "secured_data_break_glass";
public static final String ACCESS_LOG = "secured_data_access_log";
public static final List<String> ALL = List.of(CLASSIFICATION, POLICY, REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG);
public static final List<String> MAIN = List.of(CLASSIFICATION, POLICY, ACCESS_LOG, BREAK_GLASS);
public static final List<String> SYSTEM_WRITTEN = List.of(REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG);
public static final String RETENTION_RULE = "secured_data_retention_rule";
public static final String WIPE_MARKER = "secured_data_wipe_marker";
public static final String LEGAL_HOLD = "secured_data_legal_hold";
public static final String WIPE_RUN = "secured_data_wipe_run";
public static final String AUDIT_KEY = "secured_data_audit_key";
public static final List<String> ALL = List.of(CLASSIFICATION, POLICY, REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, RETENTION_RULE, WIPE_MARKER, LEGAL_HOLD, WIPE_RUN, AUDIT_KEY);
public static final List<String> MAIN = List.of(CLASSIFICATION, POLICY, ACCESS_LOG, BREAK_GLASS, RETENTION_RULE);
public static final List<String> SYSTEM_WRITTEN = List.of(REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, WIPE_MARKER, WIPE_RUN, AUDIT_KEY);
public static final String CREATE_DATE = "createDate";
public static final String MODIFY_DATE = "modifyDate";
public static final String REVEALS_JOIN = "secured_data_access_log_reveals";
public static final String REVEALS_WIDGET = "securedDataReveals";
@@ -90,7 +98,9 @@ public final class SecuredDataTables
*******************************************************************************/
public static List<QTableMetaData> tables(SecuredDataQBitConfig config)
{
return (List.of(classification(config), policy(config), revealGrant(config), breakGlass(config), accessLog(config)));
List<QTableMetaData> tables = new ArrayList<>(List.of(classification(config), policy(config), revealGrant(config), breakGlass(config), accessLog(config)));
tables.addAll(RetentionTables.tables(config));
return (tables);
}
@@ -98,13 +108,13 @@ public final class SecuredDataTables
/*******************************************************************************
** Common identity, audit and administration rules.
*******************************************************************************/
private static QTableMetaData base(SecuredDataQBitConfig config, String name, String label)
static QTableMetaData base(SecuredDataQBitConfig config, String name, String label)
{
QTableMetaData table = new QTableMetaData().withName(name).withLabel(label).withBackendName(config.getBackendName())
.withPrimaryKeyField("id").withField(new QFieldMetaData("id", QFieldType.INTEGER).withBackendName("id").withIsEditable(false))
.withPermissionRules(SecuredDataPermissions.rules()).withIcon(new QIcon("shield"))
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD));
CoreMetadataBridge.classifyTable(table);
SecuredDataMetadata.classifyTable(table);
if(ACCESS_LOG.equals(name))
{
table.withoutCapabilities(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
@@ -117,7 +127,7 @@ public final class SecuredDataTables
/*******************************************************************************
** Every database column has an explicit snake_case name.
*******************************************************************************/
private static QFieldMetaData field(String name, QFieldType type)
static QFieldMetaData field(String name, QFieldType type)
{
QFieldMetaData field = new QFieldMetaData(name, type).withBackendName(name).withLabel(name.replace('_', ' '));
// TODO(contingent): dependsOn("table_name") once the core RC has it
@@ -126,6 +136,51 @@ public final class SecuredDataTables
/*******************************************************************************
** When a row was written. Every table has this field and a modify date;
** neither is editable, and the columns are create_date and modify_date.
*******************************************************************************/
static QFieldMetaData createDate(String name)
{
return (field(name, QFieldType.DATE_TIME).withBackendName("create_date").withLabel("create date")
.withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE));
}
/*******************************************************************************
** When a row last changed: its create date until something changes it.
*******************************************************************************/
static QFieldMetaData modifyDate(String name)
{
return (field(name, QFieldType.DATE_TIME).withBackendName("modify_date").withLabel("modify date")
.withIsEditable(false).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE));
}
/*******************************************************************************
** The name of a table's create date field. Classification, policy and
** retention rule had the two dates before the other tables did and keep the
** field names they have always had.
*******************************************************************************/
static String createDateField(String table)
{
return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "create_date" : CREATE_DATE);
}
/*******************************************************************************
** The name of a table's modify date field (see createDateField).
*******************************************************************************/
static String modifyDateField(String table)
{
return (List.of(CLASSIFICATION, POLICY, RETENTION_RULE).contains(table) ? "modify_date" : MODIFY_DATE);
}
/*******************************************************************************
** Define Classification.
*******************************************************************************/
@@ -138,8 +193,8 @@ public final class SecuredDataTables
.withField(field("mask_style", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_mask_style"))
.withField(field("low_cardinality", QFieldType.BOOLEAN))
.withField(field("notes", QFieldType.TEXT))
.withField(field("create_date", QFieldType.DATE_TIME))
.withField(field("modify_date", QFieldType.DATE_TIME))
.withField(createDate("create_date"))
.withField(modifyDate("modify_date"))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
table.withSection(SectionFactory.defaultT1("id", "table_name", "field_name", "classification"));
table.withSection(SectionFactory.defaultT2("mask_style", "low_cardinality", "notes"));
@@ -172,8 +227,8 @@ public final class SecuredDataTables
.withField(field("distinct_value_alert", QFieldType.INTEGER).withDefaultValue(15))
.withField(field("min_entropy_enabled", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("create_date", QFieldType.DATE_TIME))
.withField(field("modify_date", QFieldType.DATE_TIME))
.withField(createDate("create_date"))
.withField(modifyDate("modify_date"))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
table.withSection(SectionFactory.defaultT1("id", "name", "permission_name", "classification"));
table.withSection(SectionFactory.defaultT2("table_name", "tier", "reveal_minutes", "daily_reveal_cap", "reason_options", "allow_free_text_reason", "require_free_text", "client_channel", "exact_match_per_hour", "miss_alert_consecutive", "distinct_value_alert", "min_entropy_enabled", "is_active"));
@@ -198,10 +253,12 @@ public final class SecuredDataTables
.withField(field("reason_text", QFieldType.TEXT))
.withField(field("granted_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true));
.withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE));
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "table_name"));
table.withSection(SectionFactory.defaultT2("record_id", "field_name", "reason", "reason_text", "access_log_id"));
table.withSection(SectionFactory.defaultT3("granted_at", "expires_at"));
table.withSection(SectionFactory.defaultT3("granted_at", "expires_at", CREATE_DATE, MODIFY_DATE));
return (table);
}
@@ -223,10 +280,12 @@ public final class SecuredDataTables
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("ended_at", QFieldType.DATE_TIME))
.withField(field("ended_by", QFieldType.STRING).withMaxLength(255))
.withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status"));
.withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status"))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE));
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "reason"));
table.withSection(SectionFactory.defaultT2("reason_text", "classification_ceiling", "table_name", "ended_by", "status"));
table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at"));
table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at", CREATE_DATE, MODIFY_DATE));
return (table);
}
@@ -253,10 +312,13 @@ public final class SecuredDataTables
.withField(field("anomaly_flag", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("break_glass_id", QFieldType.INTEGER))
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("search_value_fingerprints", QFieldType.TEXT));
.withField(field("search_value_fingerprints", QFieldType.TEXT))
.withField(field("rule_id", QFieldType.INTEGER))
.withField(createDate(CREATE_DATE))
.withField(modifyDate(MODIFY_DATE));
table.withSection(SectionFactory.defaultT1("id", "event_type", "user_id", "user_name"));
table.withSection(SectionFactory.defaultT2("table_name", "record_id", "field_name", "record_count", "action_name", "reason", "reason_text", "client_channel", "outcome", "anomaly_flag", "break_glass_id", "search_value_fingerprints"));
table.withSection(SectionFactory.defaultT3("created_at"));
table.withSection(SectionFactory.defaultT2("table_name", "record_id", "field_name", "record_count", "action_name", "reason", "reason_text", "client_channel", "outcome", "anomaly_flag", "break_glass_id", "rule_id", "search_value_fingerprints"));
table.withSection(SectionFactory.defaultT3("created_at", CREATE_DATE, MODIFY_DATE));
table.withSection(SectionFactory.customT2("reveals", new QIcon("visibility")).withLabel("Reveals").withWidgetName(REVEALS_WIDGET));
return (table);
}
@@ -22,7 +22,7 @@ package com.kingsrook.qbits.secureddata.permissions;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
@@ -45,7 +45,7 @@ public class SecuredDataPermissionChecker implements CustomPermissionChecker
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
boolean write = input instanceof InsertInput || input instanceof UpdateInput || input instanceof DeleteInput;
if(QContext.getQSession() == null || "assistant".equals(CoreMetadataBridge.channel(QContext.getQSession()))
if(QContext.getQSession() == null || "assistant".equals(SecuredDataMetadata.channel(QContext.getQSession()))
|| !QContext.getQSession().hasPermission(SecuredDataPermissions.ADMIN)
|| (write && SecuredDataTables.SYSTEM_WRITTEN.contains(metadata.getName())))
{
@@ -28,8 +28,8 @@ import java.util.Map;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
@@ -1,6 +1,6 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
@@ -18,26 +18,19 @@
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
package com.kingsrook.qbits.secureddata.retention;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** A resolver decision; the reason must never contain a field value.
** Host adapter to qbit-object-storage. Receives only AES-GCM ciphertext.
** The host should use a locked WORM bucket when available. Return after durable write.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessDecision; delete when the pin moves to RC.6.
public record FieldAccessDecision(FieldAccessTier tier, FieldMaskStyle maskStyle, String reason, boolean exactMatchAllowed)
@FunctionalInterface
public interface ArchiveSinkInterface
{
public static final FieldAccessDecision FULL = new FieldAccessDecision(FieldAccessTier.FULL, null, null);
/*******************************************************************************
** Compatibility constructor: exact-match searches are allowed by default.
** Store an encrypted JSON record at an opaque, unique object key.
*******************************************************************************/
public FieldAccessDecision(FieldAccessTier tier, FieldMaskStyle maskStyle, String reason)
{
this(tier, maskStyle, reason, true);
}
void put(String storageName, String objectKey, byte[] ciphertext) throws QException;
}
@@ -0,0 +1,221 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.util.Arrays;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.secureddata.SecuredDataClassifications;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Shared validation and effective audit treatment; no record values are queried.
*******************************************************************************/
public final class RetentionRules
{
public static final Set<String> ACTIONS = Set.of("NULL_WITH_MARKER", "OVERWRITE_TEXT", "KEYED_HASH", "DELETE_RECORD", "ARCHIVE_THEN_WIPE");
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionRules()
{
}
/*******************************************************************************
** Validated comma selection; preserve order and reject duplicates.
*******************************************************************************/
public static List<String> fields(QRecord rule)
{
String names = rule.getValueString("field_names");
return (names == null ? List.of() : Arrays.stream(names.split(",", -1)).map(String::trim).toList());
}
///////////////////////////////////////////////////////////////////////////////
// set on a thread that changed a rule inside a transaction: until that //
// transaction ends, the thread reads the rules through it (its own edits), //
// not the rules every other thread shares //
///////////////////////////////////////////////////////////////////////////////
private static final ThreadLocal<Boolean> EDITED_IN_TRANSACTION = ThreadLocal.withInitial(() -> false);
/** Set while this thread reads its own edits, so that a question from inside that read does not read them again. */
private static final ThreadLocal<Boolean> READING_OWN_EDITS = ThreadLocal.withInitial(() -> false);
/*******************************************************************************
** The active rules of a table that name this field. Core asks this for every
** field of every audited write and every filter criterion of every query, so
** the active rules are read once (SecuredDataRuntime.activeRetentionRules: per
** change, and at most MAX_RULE_AGE old) instead of once per call. A committed
** change is seen at once on the node that made it and on every node the
** cluster event reaches; a transaction that edits rules reads its own edits.
** A failed read throws: it is never "no rules". A question asked from inside
** a read of the rules is answered from the rules held before that read (see
** RuleRows) and never starts another.
*******************************************************************************/
public static List<QRecord> active(String table, String field)
{
SecuredDataRuntime runtime = SecuredDataRuntime.current();
List<QRecord> rules;
if(runtime != null && (!readsOwnEdits() || READING_OWN_EDITS.get()))
{
//////////////////////////////////////////////////////////////////////////
// also the answer to a question asked from inside this thread's read //
// of its own edits (code that runs inside that query): the rules //
// every other thread shares, and never another read of the edits //
//////////////////////////////////////////////////////////////////////////
rules = runtime.activeRetentionRules();
}
else
{
READING_OWN_EDITS.set(true);
try
{
rules = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table)));
}
finally
{
READING_OWN_EDITS.remove();
}
}
return (rules.stream().filter(rule -> table != null && table.equals(rule.getValueString("table_name"))).filter(rule -> fields(rule).contains(field)).toList());
}
/*******************************************************************************
** A rule was written on this thread inside a transaction that is still open.
*******************************************************************************/
public static void editedInTransaction()
{
EDITED_IN_TRANSACTION.set(true);
}
/*******************************************************************************
** Whether this thread must read the rules through its open transaction.
*******************************************************************************/
private static boolean readsOwnEdits()
{
if(!EDITED_IN_TRANSACTION.get())
{
return (false);
}
if(QContext.getQBackendTransaction() == null)
{
EDITED_IN_TRANSACTION.remove();
return (false);
}
return (true);
}
/*******************************************************************************
** A changed-only policy takes precedence over derivatives.
*******************************************************************************/
public static boolean changedOnly(String table, String field)
{
return (active(table, field).stream().anyMatch(rule -> "CHANGED_ONLY".equals(rule.getValueString("audit_treatment"))));
}
/*******************************************************************************
** Validate both edits and persisted policies before any execution.
*******************************************************************************/
public static void validate(QRecord rule)
{
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
if(table == null || SecuredDataTables.ALL.contains(table.getName()))
{
throw (new IllegalArgumentException("Select a host table for retention."));
}
List<String> fields = fields(rule);
if(fields.isEmpty() || fields.size() != Set.copyOf(fields).size() || fields.stream().anyMatch(f -> table.getField(f) == null || f.equals(table.getPrimaryKeyField())))
{
throw (new IllegalArgumentException("Select unique, non-key fields of the selected table."));
}
var age = table.getField(rule.getValueString("age_field_name"));
if(age == null || !Set.of(QFieldType.DATE, QFieldType.DATE_TIME).contains(age.getType()))
{
throw (new IllegalArgumentException("Age must use a date or date-time field of the selected table."));
}
if(rule.getValueInteger("age_days") == null || rule.getValueInteger("age_days") < 1 || rule.getValueInteger("batch_size") == null || rule.getValueInteger("batch_size") < 1)
{
throw (new IllegalArgumentException("Age days and batch size must be positive."));
}
String action = rule.getValueString("action");
if(!ACTIONS.contains(action) || !Set.of("SHREDDABLE", "CHANGED_ONLY").contains(rule.getValueString("audit_treatment")))
{
throw (new IllegalArgumentException("Select a supported action and audit treatment."));
}
for(String name : fields)
{
var field = table.getField(name);
if(Set.of("OVERWRITE_TEXT", "KEYED_HASH").contains(action) && !Set.of(QFieldType.STRING, QFieldType.TEXT).contains(field.getType()))
{
throw (new IllegalArgumentException("Overwrite and keyed hash require STRING or TEXT fields."));
}
if("KEYED_HASH".equals(action) && SecuredDataClassifications.lowEntropy(table, name))
{
throw (new IllegalArgumentException("Keyed hash is forbidden for low-entropy fields."));
}
if("KEYED_HASH".equals(action) && field.getMaxLength() != null && field.getMaxLength() < 200)
{
throw (new IllegalArgumentException("Keyed hash requires capacity for a 200-character token."));
}
if(Set.of("NULL_WITH_MARKER", "ARCHIVE_THEN_WIPE").contains(action) && field.getIsRequired())
{
throw (new IllegalArgumentException("Null wipe requires nullable fields."));
}
String overwrite = rule.getValueString("overwrite_text");
if("OVERWRITE_TEXT".equals(action) && (overwrite == null || field.getMaxLength() != null && overwrite.length() > field.getMaxLength()))
{
throw (new IllegalArgumentException("Overwrite text must fit every target field."));
}
}
if("ARCHIVE_THEN_WIPE".equals(action) && (rule.getValueString("archive_storage_name") == null || rule.getValueString("archive_storage_name").isBlank()))
{
throw (new IllegalArgumentException("Archive then wipe requires an archive storage name."));
}
}
/*******************************************************************************
** Known backup window; unknown is explicit rather than assumed safe.
*******************************************************************************/
public static String backupWarning(QRecord rule)
{
Integer days = SecuredDataRuntime.current().config().getBackupRetentionDays();
Integer age = rule.getValueInteger("age_days");
return (days != null && age != null && days > age ? "Backups keep data for " + days + " days, longer than this rule's " + age
+ " days. Wiped values remain recoverable from backups until they expire." : null);
}
}
@@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Value-free policy panels and host-opt-in record marker lists.
*******************************************************************************/
public final class RetentionWidgets
{
public static final String BACKUPS = "securedDataBackups";
public static final String MARKERS = "securedDataRuleMarkers";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionWidgets()
{
}
/*******************************************************************************
** Markers belong to retention rules and are never editable in child lists.
*******************************************************************************/
public static QJoinMetaData markersJoin()
{
return (new QJoinMetaData().withName("secured_data_rule_markers").withType(JoinType.ONE_TO_MANY)
.withLeftTable(SecuredDataTables.RETENTION_RULE).withRightTable(SecuredDataTables.WIPE_MARKER).withJoinOn(new JoinOn("id", "rule_id")));
}
/*******************************************************************************
** Standard rule child list.
*******************************************************************************/
public static QWidgetMetaData markers()
{
return (ChildRecordListRenderer.widgetMetaDataBuilder(markersJoin()).withName(MARKERS).withLabel("Wipe markers")
.withCanAddChildRecord(false).withMaxRows(50).getWidgetMetaData().withPermissionRules(SecuredDataPermissions.rules()));
}
/*******************************************************************************
** Known physical backup retention and its residual recovery risk.
*******************************************************************************/
public static QWidgetMetaData backups()
{
return (new QWidgetMetaData().withName(BACKUPS).withLabel("Backups").withType(WidgetType.HTML.getType()).withGridColumns(12)
.withCodeReference(new QCodeReference(Backups.class)).withPermissionRules(SecuredDataPermissions.rules()));
}
/*******************************************************************************
** Host adds this metadata and a section referencing its name on its own table.
** The immutable widget configuration selects the table, not request parameters.
*******************************************************************************/
public static QWidgetMetaData wipedFields(QTableMetaData table)
{
return (new QWidgetMetaData().withName("securedDataWipedFields_" + table.getName()).withLabel("Wiped fields").withType(WidgetType.HTML.getType())
.withGridColumns(12).withDefaultValue("hostTable", table.getName()).withCodeReference(new QCodeReference(WipedFields.class)));
}
/*******************************************************************************
** Render the rule through normal user permissions.
*******************************************************************************/
public static class Backups extends AbstractWidgetRenderer
{
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
List<QRecord> rules = new QueryAction().execute(new QueryInput(SecuredDataTables.RETENTION_RULE)
.withFilter(new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, input.getQueryParams().get("id"))))).getRecords();
StringBuilder html = new StringBuilder("<div>");
Integer days = SecuredDataRuntime.current().config().getBackupRetentionDays();
html.append("<p>Physical backup retention: ").append(days == null ? "unknown" : days + " days").append(".</p>");
for(QRecord rule : rules)
{
String warning = RetentionRules.backupWarning(rule);
if(warning != null)
{
html.append("<p role='alert'>").append(escape(warning)).append("</p>");
}
}
html.append("<p>CNPG physical backups and WAL cannot exclude audit keys. A backup and the KEK can recover wiped data until that backup expires.</p></div>");
return (new RenderWidgetOutput(new RawHTML(null, html.toString())));
}
}
/*******************************************************************************
** Authorize the host record with an ID-only query before looking up markers.
*******************************************************************************/
public static class WipedFields extends AbstractWidgetRenderer
{
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
String tableName = (String) input.getWidgetMetaData().getDefaultValues().get("hostTable");
var table = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getTable(tableName);
String id = input.getQueryParams().get("id");
List<QRecord> visible = new QueryAction().execute(new QueryInput(tableName).withFieldNamesToInclude(Set.of(table.getPrimaryKeyField()))
.withFilter(new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.EQUALS, id)))).getRecords();
StringBuilder html = new StringBuilder("<ul aria-label='Wiped fields'>");
if(!visible.isEmpty())
{
for(QRecord marker : SystemWrites.query(SecuredDataTables.WIPE_MARKER, new QQueryFilter()
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, tableName))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, id))))
{
List<QRecord> rules = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, marker.getValue("rule_id"))));
String age = rules.isEmpty() ? "unknown" : rules.getFirst().getValueString("age_days");
html.append("<li>").append(escape(marker.getValueString("field_name"))).append(": Wiped per policy (").append(escape(age)).append(" days)</li>");
}
}
return (new RenderWidgetOutput(new RawHTML(null, html.append("</ul>").toString())));
}
}
/*******************************************************************************
** Escape user-managed names and all text before HTML rendering.
*******************************************************************************/
private static String escape(String text)
{
return (text == null ? "" : text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace("\"", "&quot;").replace("'", "&#39;"));
}
}
@@ -0,0 +1,258 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Standard Review/Result action and a daily system scheduler entry point.
*******************************************************************************/
public final class WipeProcesses
{
public static final String WIPE = "secured_data_wipe";
public static final String DAILY = "secured_data_wipe_daily";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private WipeProcesses()
{
}
/*******************************************************************************
** Processes are table actions or hidden schedules, never navigation entries.
*******************************************************************************/
public static List<QProcessMetaData> processes(SecuredDataQBitConfig config)
{
Map<String, Serializable> values = new LinkedHashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, SecuredDataTables.RETENTION_RULE);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, SecuredDataTables.WIPE_RUN);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, "Eligible records will be wiped; active legal holds are preserved.");
QPermissionRules permissions = new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
.withPermissionBaseName("secured_data").withCustomPermissionChecker(new QCodeReference(Permission.class));
QProcessMetaData wipe = StreamedETLWithFrontendProcess.defineProcessMetaData(Extract.class, Review.class, Load.class, values)
.withName(WIPE).withLabel("Run wipe now").withTableName(SecuredDataTables.RETENTION_RULE).withMinInputRecords(1).withPermissionRules(permissions);
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily retention wipe").withIsHidden(true).withPermissionRules(permissions)
.withStep(new QBackendStepMetaData().withName("wipe").withCode(new QCodeReference(Daily.class)));
if(config.getSchedulerName() != null)
{
daily.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(config.getWipeIntervalSeconds()).withInitialDelaySeconds(30));
}
return (List.of(wipe, daily));
}
/*******************************************************************************
** Explicit retention permission is independent of classified-value access.
*******************************************************************************/
public static class Permission implements CustomPermissionChecker
{
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
WipeService.checkPermission();
}
}
/*******************************************************************************
** Select policies using the standard table-selection filter, then stream IDs.
*******************************************************************************/
public static class Extract extends ExtractViaQueryStep
{
@Override
public Integer doCount(RunBackendStepInput input)
{
return (null);
}
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
for(QRecord rule : SystemWrites.query(SecuredDataTables.RETENTION_RULE, getQueryFilter(input)))
{
new WipeService().extract(rule, packets ->
{
for(QRecord packet : packets)
{
getRecordPipe().addRecord(packet);
}
});
}
}
}
/*******************************************************************************
** Counts per policy and hold status; no record or field values in summaries.
*******************************************************************************/
public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final Map<String, ProcessSummaryLine> lines = new LinkedHashMap<>();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
for(QRecord packet : input.getRecords())
{
String id = packet.getValueString("rule_id");
boolean held = Boolean.TRUE.equals(packet.getValueBoolean("held"));
String label = held ? "held" : "to wipe";
lines.computeIfAbsent(id + label, key -> line(id, held, false)).incrementCountAndAddPrimaryKey(null);
output.getRecords().add(packet);
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
return (new ArrayList<>(lines.values()));
}
}
/*******************************************************************************
** Group only the current ETL page, splitting again by each policy's batch size.
*******************************************************************************/
public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final Map<String, ProcessSummaryLine> lines = new LinkedHashMap<>();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
Map<Serializable, List<Serializable>> rules = new LinkedHashMap<>();
for(QRecord packet : input.getRecords())
{
rules.computeIfAbsent(packet.getValue("rule_id"), ignored -> new ArrayList<>()).add(packet.getValue("id"));
}
for(var entry : rules.entrySet())
{
int size = WipeService.rule(entry.getKey()).getValueInteger("batch_size");
for(int offset = 0; offset < entry.getValue().size(); offset += size)
{
QRecord run = new WipeService().batch(entry.getKey(), entry.getValue().subList(offset, Math.min(offset + size, entry.getValue().size())));
output.getRecords().add(run);
for(boolean held : List.of(false, true))
{
String id = String.valueOf(entry.getKey());
ProcessSummaryLine line = lines.computeIfAbsent(id + held, ignored -> line(id, held, true));
for(int n = 0; n < run.getValueInteger(held ? "held_count" : "wiped_count"); n++)
{
line.incrementCountAndAddPrimaryKey(null);
}
}
}
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
return (new ArrayList<>(lines.values()));
}
}
/*******************************************************************************
** Scheduler identity must already be system; never elevate a user invocation.
*******************************************************************************/
public static class Daily implements BackendStep
{
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
if(!(QContext.getQSession() instanceof QSystemUserSession))
{
throw (new QPermissionDeniedException("Daily retention requires the system session."));
}
for(QRecord rule : SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))))
{
List<QException> errors = new ArrayList<>();
new WipeService().extract(rule, packets ->
{
try
{
output.getRecords().add(new WipeService().batch(rule.getValue("id"), packets.stream().map(row -> row.getValue("id")).toList()));
}
catch(QException e)
{
errors.add(e);
}
});
if(!errors.isEmpty())
{
throw (new QException("Daily retention had rolled-back batches."));
}
}
}
}
/*******************************************************************************
** Standard future/past summary text.
*******************************************************************************/
private static ProcessSummaryLine line(String id, boolean held, boolean result)
{
ProcessSummaryLine line = new ProcessSummaryLine(held ? Status.INFO : Status.OK, 0, null);
String future = "eligible records for rule " + id + (held ? " held by legal hold" : " to wipe");
String past = "records for rule " + id + (held ? " held by legal hold" : " wiped");
line.setSingularFutureMessage(future);
line.setPluralFutureMessage(future);
line.setSingularPastMessage(past);
line.setPluralPastMessage(past);
return (line);
}
}
@@ -0,0 +1,421 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
import com.kingsrook.qbits.secureddata.SecuredDataKeySource;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.audits.AuditShreddedEvent;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.tables.WipeAuditFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** ID-only extraction and atomic, system-owned batches. No host values are placed
** in process records, run messages, access logs or exceptions.
*******************************************************************************/
public final class WipeService
{
/*******************************************************************************
** Checked-exception-aware streaming boundary.
*******************************************************************************/
@FunctionalInterface
public interface PageConsumer
{
/*******************************************************************************
** Consume one bounded page.
*******************************************************************************/
void accept(List<QRecord> records) throws QException;
}
public static final String PERMISSION = "secured_data.retention";
/*******************************************************************************
** Enforce at every callable boundary, including process resumption.
*******************************************************************************/
public static void checkPermission() throws QPermissionDeniedException
{
if(!(QContext.getQSession() instanceof QSystemUserSession)
&& (QContext.getQSession() == null || !QContext.getQSession().hasPermission(PERMISSION) || "assistant".equals(QContext.getQSession().getClientChannelName())))
{
throw (new QPermissionDeniedException("Permission denied: secured_data.retention is required."));
}
}
/*******************************************************************************
** Reload the rule instead of trusting process-state configuration.
*******************************************************************************/
public static QRecord rule(Serializable id) throws QException
{
List<QRecord> rows = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id)));
if(rows.isEmpty())
{
throw (new QException("Retention rule no longer exists."));
}
QRecord rule = rows.getFirst();
RetentionRules.validate(rule);
return (rule);
}
/*******************************************************************************
** A bounded keyset scan. Per-field predicates return IDs only, even for PHI.
** Packets carry IDs, field names and a hold indicator, never stored values.
*******************************************************************************/
public void extract(QRecord rule, PageConsumer consumer) throws QException
{
checkPermission();
RetentionRules.validate(rule);
if(!Boolean.TRUE.equals(rule.getValueBoolean("is_active")))
{
return;
}
String tableName = rule.getValueString("table_name");
QTableMetaData table = QContext.getQInstance().getTable(tableName);
String pkName = table.getPrimaryKeyField();
Serializable after = null;
while(true)
{
QQueryFilter filter = eligible(rule, null).withOrderBy(new QFilterOrderBy(pkName)).withLimit(rule.getValueInteger("batch_size"));
if(after != null)
{
filter.withCriteria(new QFilterCriteria(pkName, QCriteriaOperator.GREATER_THAN, after));
}
List<QRecord> ids = query(tableName, filter, Set.of(pkName));
if(ids.isEmpty())
{
return;
}
List<QRecord> packets = new ArrayList<>();
for(QRecord id : ids)
{
Serializable pk = id.getValue(pkName);
packets.add(new QRecord().withValue("id", pk).withValue("rule_id", rule.getValue("id"))
.withValue("held", held(tableName, pk)));
}
consumer.accept(packets);
after = ids.getLast().getValue(pkName);
}
}
/*******************************************************************************
** Candidate fields are tested in SQL, not fetched.
*******************************************************************************/
private static QQueryFilter eligible(QRecord rule, String field)
{
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
Instant cutoff = SecuredDataRuntime.current().config().getClock().instant().minus(rule.getValueInteger("age_days"), ChronoUnit.DAYS);
Serializable age = table.getField(rule.getValueString("age_field_name")).getType() == QFieldType.DATE ? cutoff.atZone(ZoneOffset.UTC).toLocalDate() : cutoff;
QQueryFilter filter = new QQueryFilter().withCriteria(new QFilterCriteria(rule.getValueString("age_field_name"), QCriteriaOperator.LESS_THAN, age));
QQueryFilter nonnull = new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR);
for(String name : field == null ? RetentionRules.fields(rule) : List.of(field))
{
nonnull.withCriteria(new QFilterCriteria(name, QCriteriaOperator.IS_NOT_BLANK));
}
return (filter.withSubFilter(nonnull));
}
/*******************************************************************************
** Active record and whole-table holds pause all policies.
*******************************************************************************/
public static boolean held(String table, Serializable pk)
{
return (!SystemWrites.query(SecuredDataTables.LEGAL_HOLD, new QQueryFilter()
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table))
.withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"))
.withSubFilter(new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR)
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.IS_BLANK))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, String.valueOf(pk))))).isEmpty());
}
/*******************************************************************************
** Explicit projection and transaction for every host read.
*******************************************************************************/
private static List<QRecord> query(String table, QQueryFilter filter, Set<String> fields) throws QException
{
return (SystemWrites.call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withFieldNamesToInclude(fields)
.withShouldFetchHeavyFields(true).withTransaction(QContext.getQBackendTransaction())).getRecords()));
}
/*******************************************************************************
** Revalidate eligibility and holds inside the batch, then commit all effects.
** Cross-backend and nontransactional deployments are refused before any writes.
*******************************************************************************/
public QRecord batch(Serializable ruleId, List<Serializable> ids) throws QException
{
checkPermission();
QRecord rule = rule(ruleId);
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
if(ids.size() > rule.getValueInteger("batch_size"))
{
throw (new QException("Wipe batch exceeds the policy batch size."));
}
if(QContext.getQBackendTransaction() != null)
{
throw (new QException("Wipe batches must own their transaction."));
}
for(String name : List.of(SecuredDataTables.RETENTION_RULE, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD,
SecuredDataTables.WIPE_RUN, SecuredDataTables.AUDIT_KEY, SecuredDataTables.ACCESS_LOG, "audit", "auditDetail"))
{
QTableMetaData metadata = QContext.getQInstance().getTable(name);
if(metadata != null && !Objects.equals(metadata.getBackendName(), table.getBackendName()))
{
throw (new QException("Wipe, audit and secured-data tables must share one transactional backend."));
}
}
if("ARCHIVE_THEN_WIPE".equals(rule.getValueString("action")) && SecuredDataRuntime.current().config().getArchiveSink() == null)
{
throw (new QException("ARCHIVE_THEN_WIPE requires a host ArchiveSinkInterface backed by object storage."));
}
try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(table.getName())))
{
if(transaction == null || transaction.getClass() == QBackendTransaction.class)
{
throw (new QException("Wipe requires a backend with rollback-capable write transactions."));
}
QContext.setTransaction(transaction);
try
{
QRecord result = SystemWrites.call(() -> apply(rule, table, ids, transaction));
transaction.commit();
return (result);
}
catch(Exception e)
{
transaction.rollback();
throw (new QException("Wipe batch failed and was rolled back; no values are included in this message."));
}
finally
{
SecuredDataRuntime.current().keys().close();
QContext.clearTransaction();
}
}
}
/*******************************************************************************
** Batch body. Markers, key deletion, audit, access log and run row share tx.
*******************************************************************************/
private QRecord apply(QRecord initialRule, QTableMetaData table, List<Serializable> ids, QBackendTransaction transaction) throws QException
{
QRecord rule = rule(initialRule.getValue("id"));
Instant started = SecuredDataRuntime.current().config().getClock().instant();
List<Serializable> wiped = new ArrayList<>();
int held = 0;
Map<Serializable, List<String>> fieldsById = new LinkedHashMap<>();
if(Boolean.TRUE.equals(rule.getValueBoolean("is_active")))
{
for(String field : RetentionRules.fields(rule))
{
QQueryFilter filter = eligible(rule, field).withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, ids));
for(QRecord row : query(table.getName(), filter, Set.of(table.getPrimaryKeyField())))
{
fieldsById.computeIfAbsent(row.getValue(table.getPrimaryKeyField()), ignored -> new ArrayList<>()).add(field);
}
}
}
List<QRecord> markers = new ArrayList<>();
Map<String, List<Serializable>> logged = new LinkedHashMap<>();
for(var entry : fieldsById.entrySet())
{
Serializable pk = entry.getKey();
if(held(table.getName(), pk))
{
held++;
continue;
}
List<String> fields = entry.getValue();
String action = rule.getValueString("action");
if("DELETE_RECORD".equals(action))
{
fields = new ArrayList<>(table.getFields().keySet());
fields.remove(table.getPrimaryKeyField());
}
change(rule, table, pk, fields, transaction);
for(String field : fields)
{
List<QRecord> keys = query(SecuredDataTables.AUDIT_KEY, SecuredDataKeySource.recordField(table.getName(), field, pk), Set.of("id"));
for(QRecord key : keys)
{
var deleted = new DeleteAction().execute(new DeleteInput(SecuredDataTables.AUDIT_KEY).withPrimaryKeys(List.of(key.getValue("id"))).withTransaction(transaction));
if(deleted.getDeletedRecordCount() != 1)
{
throw (new QException("Audit key deletion failed."));
}
AuditShreddedEvent.shreddedDetail(table, pk, table.getField(field), rule.getValueString("id"), transaction);
}
markers.add(new QRecord().withValue("table_name", table.getName()).withValue("record_id", String.valueOf(pk)).withValue("field_name", field)
.withValue("rule_id", rule.getValue("id")).withValue("action", action).withValue("wiped_at", started));
logged.computeIfAbsent(field, ignored -> new ArrayList<>()).add(pk);
}
wiped.add(pk);
}
QRecord run = SystemWrites.insertOne(SecuredDataTables.WIPE_RUN, new QRecord().withValue("rule_id", rule.getValue("id"))
.withValue("started_at", started).withValue("finished_at", SecuredDataRuntime.current().config().getClock().instant())
.withValue("status", "COMPLETE").withValue("eligible_count", fieldsById.size()).withValue("held_count", held)
.withValue("wiped_count", wiped.size()).withValue("error_count", 0).withValue("record_ids", boundedIds(wiped))
.withValue("message", "Policy batch completed."), transaction);
for(QRecord marker : markers)
{
marker.setValue("run_id", run.getValue("id"));
List<QRecord> previous = SystemWrites.query(SecuredDataTables.WIPE_MARKER, SecuredDataKeySource.recordField(marker.getValueString("table_name"), marker.getValueString("field_name"), marker.getValue("record_id")));
if(previous.isEmpty())
{
SystemWrites.insertOne(SecuredDataTables.WIPE_MARKER, marker, transaction);
}
else
{
marker.setValue("id", previous.getFirst().getValue("id"));
update(SecuredDataTables.WIPE_MARKER, marker, transaction);
}
}
for(var entry : logged.entrySet())
{
for(int offset = 0; offset < entry.getValue().size(); offset += 50)
{
List<Serializable> chunk = entry.getValue().subList(offset, Math.min(offset + 50, entry.getValue().size()));
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, ActivityLog.row("WIPE", table.getName(), "ALLOWED")
.withValue("rule_id", rule.getValue("id")).withValue("field_name", entry.getKey()).withValue("record_id", boundedIds(chunk))
.withValue("record_count", chunk.size()).withValue("action_name", rule.getValue("action")), transaction);
}
}
update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", rule.getValue("id")).withValue("last_run_at", started).withValue("last_run_count", wiped.size()), transaction);
return (run);
}
/*******************************************************************************
** The two value transformations read only the fields they transform; all
** other operations work entirely by ID. Archive receives encrypted JSON only.
*******************************************************************************/
private static void change(QRecord rule, QTableMetaData table, Serializable pk, List<String> fields, QBackendTransaction transaction) throws QException
{
String action = rule.getValueString("action");
if("DELETE_RECORD".equals(action))
{
if(new DeleteAction().execute(new DeleteInput(table.getName()).withPrimaryKeys(List.of(pk)).withTransaction(transaction)).getDeletedRecordCount() != 1)
{
throw (new QException("Record deletion failed."));
}
return;
}
QRecord stored = null;
if(Set.of("KEYED_HASH", "ARCHIVE_THEN_WIPE").contains(action))
{
stored = query(table.getName(), new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.EQUALS, pk)), new LinkedHashSet<>(fields)).getFirst();
}
if("ARCHIVE_THEN_WIPE".equals(action))
{
String objectKey = "secured-data/" + UUID.randomUUID() + ".aesgcm";
Map<String, Serializable> values = new LinkedHashMap<>();
for(String field : fields)
{
values.put(field, stored.getValue(field));
}
byte[] json = JsonUtils.toJson(values).getBytes(StandardCharsets.UTF_8);
try
{
byte[] encrypted = SecuredDataKeySource.encrypt(json, objectKey);
SecuredDataRuntime.current().config().getArchiveSink().put(rule.getValueString("archive_storage_name"), objectKey, encrypted);
}
finally
{
Arrays.fill(json, (byte) 0);
values.clear();
}
}
QRecord patch = new QRecord().withValue(table.getPrimaryKeyField(), pk);
for(String field : fields)
{
Serializable value = switch(action)
{
case "OVERWRITE_TEXT" -> rule.getValue("overwrite_text");
case "KEYED_HASH" -> SecuredDataRuntime.current().keys().instanceToken(table, table.getField(field), pk, stored.getValue(field));
default -> null;
};
patch.setValue(field, value);
}
UpdateInput input = new UpdateInput(table.getName()).withRecord(patch).withTransaction(transaction);
input.setFlags(Set.of(new WipeAuditFlag(rule.getValueString("id"))));
check(new UpdateAction().execute(input).getRecords());
}
/*******************************************************************************
** Check record-level backend failures, which need not throw exceptions.
*******************************************************************************/
private static void check(List<QRecord> records) throws QException
{
if(records.isEmpty() || records.stream().anyMatch(row -> row.getErrors() != null && !row.getErrors().isEmpty()))
{
throw (new QException("Wipe write failed."));
}
}
/*******************************************************************************
** System metadata updates still validate in the transaction.
*******************************************************************************/
private static void update(String table, QRecord row, QBackendTransaction transaction) throws QException
{
check(new UpdateAction().execute(new UpdateInput(table).withRecord(row).withTransaction(transaction)).getRecords());
}
/*******************************************************************************
** Bounded identifiers, never values. Full record evidence is in audit rows.
*******************************************************************************/
private static String boundedIds(List<Serializable> ids)
{
String joined = ids.stream().limit(50).map(String::valueOf).collect(Collectors.joining(","));
return (joined.length() > 4000 ? joined.substring(0, 4000) : joined);
}
}
@@ -30,13 +30,13 @@ import java.util.Objects;
import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
@@ -77,7 +77,7 @@ public class RevealService
private QRecord policy(String tableName, String fieldName) throws QException
{
requireTarget(tableName, fieldName);
if("assistant".equals(CoreMetadataBridge.channel(QContext.getQSession())))
if("assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
throw (new QPermissionDeniedException("Reveal is unavailable on the assistant channel."));
}
@@ -120,7 +120,7 @@ public class RevealService
SecuredDataRuntime runtime = SecuredDataRuntime.current();
synchronized(runtime)
{
if("assistant".equals(CoreMetadataBridge.channel(QContext.getQSession())))
if("assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
denied(tableName, primaryKey, fieldName, "DENIED");
throw (new QPermissionDeniedException("Reveal is unavailable on the assistant channel."));
@@ -1,67 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QSupplementalFieldMetaData;
/*******************************************************************************
** Temporary classification metadata alongside the released AI exposure metadata.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1142, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData classification properties; delete when the pin moves to RC.6.
public class AiFieldMetaData extends QSupplementalFieldMetaData
{
public static final String TYPE = "securedDataAi";
private DataClassification classification;
/*******************************************************************************
** Metadata key, separate from RC.5's exposure-only AI metadata.
*******************************************************************************/
@Override
public String getType()
{
return (TYPE);
}
/*******************************************************************************
** Classification.
*******************************************************************************/
public DataClassification getClassification()
{
return (classification);
}
/*******************************************************************************
** Code classification until the released core supports it.
*******************************************************************************/
public AiFieldMetaData withClassification(DataClassification classification)
{
this.classification = classification;
return (this);
}
}
@@ -1,67 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QSupplementalTableMetaData;
/*******************************************************************************
** Temporary classification metadata alongside the released AI exposure metadata.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1142, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData classification properties; delete when the pin moves to RC.6.
public class AiTableMetaData extends QSupplementalTableMetaData
{
public static final String TYPE = "securedDataAi";
private DataClassification classification;
/*******************************************************************************
** Metadata key, separate from RC.5's exposure-only AI metadata.
*******************************************************************************/
@Override
public String getType()
{
return (TYPE);
}
/*******************************************************************************
** Classification.
*******************************************************************************/
public DataClassification getClassification()
{
return (classification);
}
/*******************************************************************************
** Code classification until the released core supports it.
*******************************************************************************/
public AiTableMetaData withClassification(DataClassification classification)
{
this.classification = classification;
return (this);
}
}
@@ -1,66 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import java.util.Locale;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** RC.5 host adapter for a trusted, server-supplied client channel.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1141, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.model.session.QSession client channel properties; delete when the pin moves to RC.6.
public class ClientChannelSession extends QSession
{
private String clientChannel;
/*******************************************************************************
** Trusted host channel value; never take this from an unvalidated request.
*******************************************************************************/
public ClientChannelSession withClientChannel(String clientChannel)
{
this.clientChannel = clientChannel;
return (this);
}
/*******************************************************************************
** Full channel including optional request detail.
*******************************************************************************/
public String getClientChannel()
{
return (clientChannel);
}
/*******************************************************************************
** Normalized channel for policy matching.
*******************************************************************************/
public String getClientChannelName()
{
return (clientChannel == null || clientChannel.isBlank() ? null : clientChannel.strip().split("/", 2)[0].toLowerCase(Locale.ROOT));
}
}
@@ -1,79 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import java.util.LinkedHashMap;
import java.util.Map;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
/*******************************************************************************
** Temporary registry for the subset needed by this QBit.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1141, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData registration subset; delete when the pin moves to RC.6.
public class ClientChannelsMetaData implements QSupplementalInstanceMetaData
{
private final Map<String, String> entries = new LinkedHashMap<>();
/*******************************************************************************
** Registry name, distinct from the future core registry.
*******************************************************************************/
@Override
public String getName()
{
return ("securedDataClientChannels");
}
/*******************************************************************************
** Instance registry.
*******************************************************************************/
public static ClientChannelsMetaData ofOrWithNew(QInstance instance)
{
return (QSupplementalInstanceMetaData.ofOrWithNew(instance, "securedDataClientChannels", ClientChannelsMetaData::new));
}
/*******************************************************************************
** Register the QBit's entry.
*******************************************************************************/
public ClientChannelsMetaData withDeniedPattern(String key, String value)
{
entries.put(key, value);
return (this);
}
/*******************************************************************************
** Read-only registration view.
*******************************************************************************/
public Map<String, String> getEntries()
{
return (Map.copyOf(entries));
}
}
@@ -1,72 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
/*******************************************************************************
** Data sensitivity, ordered from least to most sensitive.
** Null means not classified and is less sensitive than every declared value.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1142, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.ai.DataClassification; delete when the pin moves to RC.6.
public enum DataClassification
{
/*******************************************************************************
** May be shown to anyone.
*******************************************************************************/
PUBLIC,
/*******************************************************************************
** Business-confidential information.
*******************************************************************************/
CONFIDENTIAL,
/*******************************************************************************
** Personally identifiable information.
*******************************************************************************/
PII,
/*******************************************************************************
** Protected health information under HIPAA.
*******************************************************************************/
PHI;
/*******************************************************************************
** Whether this classification is at least as sensitive as other.
** Every classification is at least as sensitive as null.
*******************************************************************************/
public boolean isAtLeast(DataClassification other)
{
return (other == null || ordinal() >= other.ordinal());
}
/*******************************************************************************
** The more sensitive classification; null loses to any declared value.
** Returns null when both values are null.
*******************************************************************************/
public static DataClassification max(DataClassification a, DataClassification b)
{
return (a == null ? b : a.isAtLeast(b) ? a : b);
}
}
@@ -1,272 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
/*******************************************************************************
** Value-free audit notification. Primary key samples contain at most 50 keys.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent; delete when the pin moves to RC.6.
public class FieldAccessEvent
{
private FieldAccessEventType type;
private String tableName;
private List<String> fieldNames;
private List<Serializable> primaryKeys;
private Integer recordCount;
private String actionName;
private FieldAccessOutcome outcome;
private List<String> searchValueFingerprints;
/*******************************************************************************
** Getter for type
*******************************************************************************/
public FieldAccessEventType getType()
{
return (this.type);
}
/*******************************************************************************
** Setter for type
*******************************************************************************/
public void setType(FieldAccessEventType type)
{
this.type = type;
}
/*******************************************************************************
** Fluent setter for type
*******************************************************************************/
public FieldAccessEvent withType(FieldAccessEventType type)
{
setType(type);
return (this);
}
/*******************************************************************************
** Getter for tableName
*******************************************************************************/
public String getTableName()
{
return (this.tableName);
}
/*******************************************************************************
** Setter for tableName
*******************************************************************************/
public void setTableName(String tableName)
{
this.tableName = tableName;
}
/*******************************************************************************
** Fluent setter for tableName
*******************************************************************************/
public FieldAccessEvent withTableName(String tableName)
{
setTableName(tableName);
return (this);
}
/*******************************************************************************
** Getter for fieldNames
*******************************************************************************/
public List<String> getFieldNames()
{
return (this.fieldNames);
}
/*******************************************************************************
** Setter for fieldNames
*******************************************************************************/
public void setFieldNames(List<String> fieldNames)
{
this.fieldNames = fieldNames == null ? null : new ArrayList<>(fieldNames);
}
/*******************************************************************************
** Fluent setter for fieldNames
*******************************************************************************/
public FieldAccessEvent withFieldNames(List<String> fieldNames)
{
setFieldNames(fieldNames);
return (this);
}
/*******************************************************************************
** Getter for primaryKeys
*******************************************************************************/
public List<Serializable> getPrimaryKeys()
{
return (this.primaryKeys == null ? null : Collections.unmodifiableList(this.primaryKeys));
}
/*******************************************************************************
** Setter for primaryKeys
*******************************************************************************/
public void setPrimaryKeys(List<Serializable> primaryKeys)
{
this.primaryKeys = primaryKeys == null ? null : new ArrayList<>(primaryKeys.subList(0, Math.min(50, primaryKeys.size())));
}
/*******************************************************************************
** Fluent setter for primaryKeys
*******************************************************************************/
public FieldAccessEvent withPrimaryKeys(List<Serializable> primaryKeys)
{
setPrimaryKeys(primaryKeys);
return (this);
}
/*******************************************************************************
** Getter for recordCount
*******************************************************************************/
public Integer getRecordCount()
{
return (this.recordCount);
}
/*******************************************************************************
** Setter for recordCount
*******************************************************************************/
public void setRecordCount(Integer recordCount)
{
this.recordCount = recordCount;
}
/*******************************************************************************
** Fluent setter for recordCount
*******************************************************************************/
public FieldAccessEvent withRecordCount(Integer recordCount)
{
setRecordCount(recordCount);
return (this);
}
/*******************************************************************************
** Getter for actionName
*******************************************************************************/
public String getActionName()
{
return (this.actionName);
}
/*******************************************************************************
** Setter for actionName
*******************************************************************************/
public void setActionName(String actionName)
{
this.actionName = actionName;
}
/*******************************************************************************
** Fluent setter for actionName
*******************************************************************************/
public FieldAccessEvent withActionName(String actionName)
{
setActionName(actionName);
return (this);
}
/*******************************************************************************
** Getter for outcome
*******************************************************************************/
public FieldAccessOutcome getOutcome()
{
return (outcome);
}
/*******************************************************************************
** Fluent setter for outcome
*******************************************************************************/
public FieldAccessEvent withOutcome(FieldAccessOutcome outcome)
{
this.outcome = outcome;
return (this);
}
/*******************************************************************************
** Getter for searchValueFingerprints
*******************************************************************************/
public List<String> getSearchValueFingerprints()
{
return (searchValueFingerprints);
}
/*******************************************************************************
** Fluent setter for searchValueFingerprints
*******************************************************************************/
public FieldAccessEvent withSearchValueFingerprints(List<String> searchValueFingerprints)
{
this.searchValueFingerprints = searchValueFingerprints == null ? null : List.copyOf(searchValueFingerprints);
return (this);
}
}
@@ -1,33 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
/*******************************************************************************
** Kinds of field access reported to the resolver.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType; delete when the pin moves to RC.6.
public enum FieldAccessEventType
{
VIEW, EXACT_MATCH_FILTER, DENIED
}
@@ -1,33 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
/*******************************************************************************
** Outcome of an exact-match search.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome; delete when the pin moves to RC.6.
public enum FieldAccessOutcome
{
MATCH, NO_MATCH, REFUSED
}
@@ -1,90 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import java.io.Serializable;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Session-aware policy, reveal grants and value-free access notifications.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverInterface; delete when the pin moves to RC.6.
public interface FieldAccessResolverInterface
{
/*******************************************************************************
** Resolve using the current QContext session.
*******************************************************************************/
FieldAccessDecision resolve(QTableMetaData table, QFieldMetaData field);
/*******************************************************************************
** Whether this record and field have an active reveal grant.
*******************************************************************************/
default boolean isRevealActive(String tableName, String fieldName, Serializable primaryKey)
{
return (false);
}
/*******************************************************************************
** Stable identity of effective access, excluding reveal grants.
*******************************************************************************/
default String getAccessFingerprint()
{
return ("");
}
/*******************************************************************************
** Receive an audit notification; failures never prevent enforcement.
*******************************************************************************/
default void onAccess(FieldAccessEvent event)
{
}
/*******************************************************************************
** Refuse an exact-match search before execution; the exception message is user-facing.
*******************************************************************************/
default void checkExactMatch(FieldAccessEvent event) throws QPermissionDeniedException
{
}
/*******************************************************************************
** One-way fingerprint of a search literal. Raw values never enter access events.
*******************************************************************************/
default String fingerprintSearchValue(Serializable value)
{
return (null);
}
}
@@ -1,126 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
/*******************************************************************************
** Effective field access, ordered from least to most restrictive.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier; delete when the pin moves to RC.6.
public enum FieldAccessTier
{
FULL, REVEALABLE, MASKED, HIDDEN;
/*******************************************************************************
** canReadValue
*******************************************************************************/
public boolean canReadValue()
{
return (this == FULL);
}
/*******************************************************************************
** canSort
*******************************************************************************/
public boolean canSort()
{
return (this == FULL);
}
/*******************************************************************************
** canGroupOrAggregate
*******************************************************************************/
public boolean canGroupOrAggregate()
{
return (this == FULL);
}
/*******************************************************************************
** canQuickSearch
*******************************************************************************/
public boolean canQuickSearch()
{
return (this == FULL);
}
/*******************************************************************************
** canWrite
*******************************************************************************/
public boolean canWrite()
{
return (this == FULL);
}
/*******************************************************************************
** Whether the operator is available for this tier.
*******************************************************************************/
public boolean canFilter(QCriteriaOperator operator)
{
return (this == FULL || (this == REVEALABLE && (operator == QCriteriaOperator.EQUALS || operator == QCriteriaOperator.IN)));
}
/*******************************************************************************
** Whether the field may be described to the caller.
*******************************************************************************/
public boolean isInMetaData()
{
return (this != HIDDEN);
}
/*******************************************************************************
** Combine tiers; null means FULL.
*******************************************************************************/
public FieldAccessTier mostRestrictive(FieldAccessTier other)
{
FieldAccessTier candidate = other == null ? FULL : other;
return (ordinal() > candidate.ordinal() ? this : candidate);
}
/*******************************************************************************
** Combine tiers; null means FULL.
*******************************************************************************/
public FieldAccessTier leastRestrictive(FieldAccessTier other)
{
FieldAccessTier candidate = other == null ? FULL : other;
return (ordinal() < candidate.ordinal() ? this : candidate);
}
}
@@ -1,33 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2026. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
/*******************************************************************************
** Presentation rules for a restricted value.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1172, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle; delete when the pin moves to RC.6.
public enum FieldMaskStyle
{
AUTO, EMAIL, LAST_4, FIRST_LAST, FULL
}
@@ -1,79 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.standin;
import java.util.LinkedHashMap;
import java.util.Map;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
/*******************************************************************************
** Temporary registry for the subset needed by this QBit.
*******************************************************************************/
// STAND-IN(core=QRun-IO/qqq#1142, removeBy=RC.6): copy of com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData registration subset; delete when the pin moves to RC.6.
public class TableFeaturePermissionsMetaData implements QSupplementalInstanceMetaData
{
private final Map<String, String> entries = new LinkedHashMap<>();
/*******************************************************************************
** Registry name, distinct from the future core registry.
*******************************************************************************/
@Override
public String getName()
{
return ("securedDataTableFeatures");
}
/*******************************************************************************
** Instance registry.
*******************************************************************************/
public static TableFeaturePermissionsMetaData ofOrWithNew(QInstance instance)
{
return (QSupplementalInstanceMetaData.ofOrWithNew(instance, "securedDataTableFeatures", TableFeaturePermissionsMetaData::new));
}
/*******************************************************************************
** Register the QBit's entry.
*******************************************************************************/
public TableFeaturePermissionsMetaData withPermission(String key, String value)
{
entries.put(key, value);
return (this);
}
/*******************************************************************************
** Read-only registration view.
*******************************************************************************/
public Map<String, String> getEntries()
{
return (Map.copyOf(entries));
}
}
@@ -45,7 +45,7 @@ public final class ActivityLog
return (new QRecord().withValue("event_type", event).withValue("table_name", table)
.withValue("user_id", SecuredDataResolver.userId())
.withValue("user_name", QContext.getQSession().getUser() == null ? "system" : QContext.getQSession().getUser().getFullName())
.withValue("client_channel", CoreMetadataBridge.channel(QContext.getQSession()))
.withValue("client_channel", SecuredDataMetadata.channel(QContext.getQSession()))
.withValue("created_at", SecuredDataRuntime.current().config().getClock().instant())
.withValue("outcome", outcome).withValue("anomaly_flag", false));
}
@@ -1,286 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import java.lang.reflect.InvocationTargetException;
import com.kingsrook.qbits.secureddata.standin.ClientChannelsMetaData;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Adapters for #1141/#1142 APIs missing from the pinned released core.
*******************************************************************************/
public final class CoreMetadataBridge
{
private static final String PERMISSIONS = "com.kingsrook.qqq.backend.core.model.metadata.permissions.";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private CoreMetadataBridge()
{
}
/*******************************************************************************
** Normalized client channel, with RC.5 session-adapter support.
*******************************************************************************/
public static String channel(QSession session)
{
return (session == null ? null : (String) optional(session, "getClientChannelName"));
}
/*******************************************************************************
** Read a code classification from released/future core or temporary metadata.
*******************************************************************************/
public static DataClassification classification(QTableMetaData table, QFieldMetaData field)
{
Object ai = field == null ? AiTableMetaData.of(table) : AiFieldMetaData.of(field);
Object result = ai == null ? null : optional(ai, "getClassification");
if(result != null)
{
return (DataClassification.valueOf(result.toString()));
}
Object fallback = field == null ? table.getSupplementalMetaData("securedDataAi") : field.getSupplementalMetaData("securedDataAi");
return (fallback == null ? null : (DataClassification) optional(fallback, "getClassification"));
}
/*******************************************************************************
** Protect the QBit's own AI routing metadata on either core version.
*******************************************************************************/
public static void classifyTable(QTableMetaData table)
{
AiTableMetaData ai = AiTableMetaData.ofOrWithNew(table);
ai.withExposure(AiExposure.NONE);
setClassification(ai, DataClassification.CONFIDENTIAL);
table.withSupplementalMetaData(new com.kingsrook.qbits.secureddata.standin.AiTableMetaData().withClassification(DataClassification.CONFIDENTIAL));
}
/*******************************************************************************
** Seed opt-in name heuristics without overriding explicit code metadata.
*******************************************************************************/
public static void classifyField(QFieldMetaData field, DataClassification classification)
{
AiFieldMetaData ai = AiFieldMetaData.of(field);
if(ai == null)
{
ai = new AiFieldMetaData();
field.withSupplementalMetaData(ai);
}
setClassification(ai, classification);
field.withSupplementalMetaData(new com.kingsrook.qbits.secureddata.standin.AiFieldMetaData().withClassification(classification));
}
/*******************************************************************************
** Copy the classification into core when that property exists.
*******************************************************************************/
private static void setClassification(Object ai, DataClassification classification)
{
try
{
Class<?> type = Class.forName("com.kingsrook.qqq.backend.core.actions.ai.DataClassification");
Object value = type.getMethod("valueOf", String.class).invoke(null, classification.name());
ai.getClass().getMethod("withClassification", type).invoke(ai, value);
}
catch(ClassNotFoundException | NoSuchMethodException expected)
{
// RC.5 has exposure-only metadata; the stand-in carries classification.
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Unable to set core classification", e));
}
}
/*******************************************************************************
** Register the assistant wall and both features. Catalog applies to all tables.
*******************************************************************************/
public static void register(QInstance instance)
{
ClientChannelsMetaData.ofOrWithNew(instance).withDeniedPattern("assistant", "secured_data_*");
registerChannel(instance);
for(String suffix : new String[] {"secured_data_full", "secured_data_reveal"})
{
String label = "Secured data: " + suffix.substring("secured_data_".length());
TableFeaturePermissionsMetaData.ofOrWithNew(instance).withPermission(suffix, label);
try
{
Class<?> registry = Class.forName(PERMISSIONS + "TableFeaturePermissionsMetaData");
Class<?> entry = Class.forName(PERMISSIONS + "TableFeaturePermission");
Object feature = entry.getConstructor().newInstance();
entry.getMethod("withSuffix", String.class).invoke(feature, suffix);
entry.getMethod("withLabel", String.class).invoke(feature, label);
Object metadata = registry.getMethod("ofOrWithNew", QInstance.class).invoke(null, instance);
registry.getMethod("withPermission", entry).invoke(metadata, feature);
}
catch(ClassNotFoundException expected)
{
// RC.5 uses the stand-in catalog until core exposes the registry.
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Unable to register table features", e));
}
}
}
/*******************************************************************************
** Core channel enforcement when the #1141 API is installed.
*******************************************************************************/
private static void registerChannel(QInstance instance)
{
try
{
Class<?> type = Class.forName(PERMISSIONS + "ClientChannelsMetaData");
Object metadata = type.getMethod("ofOrWithNew", QInstance.class).invoke(null, instance);
type.getMethod("withDeniedPattern", String.class, String.class).invoke(metadata, "assistant", "secured_data_*");
}
catch(ClassNotFoundException expected)
{
// The RC.5 custom permission checker also enforces the local wall.
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Unable to register client channel wall", e));
}
}
/*******************************************************************************
** Use core's feature semantics, including table READ and permission base names.
*******************************************************************************/
public static boolean hasFeature(QTableMetaData table, String suffix)
{
try
{
return ((Boolean) PermissionsHelper.class.getMethod("hasTableFeaturePermission", QTableMetaData.class, String.class).invoke(null, table, suffix));
}
catch(NoSuchMethodException expected)
{
if(table.getIsHidden() || !PermissionsHelper.hasTablePermission(new GetInput(table.getName()), table.getName(), TablePermissionSubType.READ))
{
return (false);
}
QPermissionRules rules = PermissionsHelper.getEffectivePermissionRules(table, QContext.getQInstance());
String base = permissionBase(table);
return (rules.getLevel() == PermissionLevel.NOT_PROTECTED || QContext.getQSession().hasPermission(base + "." + suffix));
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Core feature permission check failed", e));
}
}
/*******************************************************************************
** Missing properties are expected only on RC.5; invocation errors fail closed.
*******************************************************************************/
private static Object optional(Object target, String name)
{
try
{
return (target.getClass().getMethod(name).invoke(target));
}
catch(NoSuchMethodException expected)
{
return (null);
}
catch(IllegalAccessException | InvocationTargetException e)
{
throw (new IllegalStateException("Core metadata adapter failed", e));
}
}
/*******************************************************************************
** RC.5 keeps core's equivalent helper package-private.
*******************************************************************************/
public static String permissionBase(QTableMetaData table)
{
QPermissionRules rules = PermissionsHelper.getEffectivePermissionRules(table, QContext.getQInstance());
String base = rules.getPermissionBaseName();
return (base == null || base.isBlank() ? table.getName() : base);
}
/*******************************************************************************
** Promote temporary host code classifications so future core AI routing sees them.
*******************************************************************************/
public static void promoteClassifications(QInstance instance)
{
for(QTableMetaData table : instance.getTables().values())
{
DataClassification tableClassification = classification(table, null);
if(tableClassification != null)
{
setClassification(AiTableMetaData.ofOrWithNew(table), tableClassification);
}
for(QFieldMetaData field : table.getFields().values())
{
DataClassification fieldClassification = classification(table, field);
if(fieldClassification != null)
{
AiFieldMetaData ai = AiFieldMetaData.of(field);
if(ai == null)
{
ai = new AiFieldMetaData();
field.withSupplementalMetaData(ai);
}
setClassification(ai, fieldClassification);
}
}
}
}
}
@@ -0,0 +1,124 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermission;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Secured-data defaults and registration using the released core metadata API.
*******************************************************************************/
public final class SecuredDataMetadata
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataMetadata()
{
}
/*******************************************************************************
** Normalized trusted client channel.
*******************************************************************************/
public static String channel(QSession session)
{
return (session == null ? null : session.getClientChannelName());
}
/*******************************************************************************
** Read code classification without applying field/table precedence.
*******************************************************************************/
public static DataClassification classification(QTableMetaData table, QFieldMetaData field)
{
if(field == null)
{
AiTableMetaData ai = AiTableMetaData.of(table);
return (ai == null ? null : ai.getClassification());
}
AiFieldMetaData ai = AiFieldMetaData.of(field);
return (ai == null ? null : ai.getClassification());
}
/*******************************************************************************
** Protect the QBit's own AI routing metadata.
*******************************************************************************/
public static void classifyTable(QTableMetaData table)
{
AiTableMetaData.ofOrWithNew(table).withExposure(AiExposure.NONE).withClassification(DataClassification.CONFIDENTIAL);
}
/*******************************************************************************
** Seed opt-in name heuristics without replacing other AI field metadata.
*******************************************************************************/
public static void classifyField(QFieldMetaData field, DataClassification classification)
{
AiFieldMetaData ai = AiFieldMetaData.of(field);
if(ai == null)
{
ai = new AiFieldMetaData();
field.withSupplementalMetaData(ai);
}
ai.withClassification(classification);
}
/*******************************************************************************
** Register the assistant wall and both table features.
*******************************************************************************/
public static void register(QInstance instance)
{
ClientChannelsMetaData.ofOrWithNew(instance).withDeniedPattern("assistant", "secured_data_*");
for(String suffix : new String[] {"secured_data_full", "secured_data_reveal"})
{
TableFeaturePermissionsMetaData.ofOrWithNew(instance).withPermission(new TableFeaturePermission()
.withSuffix(suffix).withLabel("Secured data: " + suffix.substring("secured_data_".length())));
}
}
/*******************************************************************************
** Use core feature semantics, including table READ and permission base names.
*******************************************************************************/
public static boolean hasFeature(QTableMetaData table, String suffix)
{
return (PermissionsHelper.hasTableFeaturePermission(table, suffix));
}
/*******************************************************************************
** Effective permission base name.
*******************************************************************************/
public static String permissionBase(QTableMetaData table)
{
QPermissionRules rules = PermissionsHelper.getEffectivePermissionRules(table, QContext.getQInstance());
String base = rules.getPermissionBaseName();
return (base == null || base.isBlank() ? table.getName() : base);
}
}
@@ -21,7 +21,9 @@
package com.kingsrook.qbits.secureddata.support;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
@@ -32,6 +34,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
@@ -100,13 +103,15 @@ public final class SystemWrites
/*******************************************************************************
** Query without user locks or field access recursion.
** Query without user locks or field access recursion, for the stored fields of
** the table only (see storedFields).
*******************************************************************************/
public static List<QRecord> query(String table, QQueryFilter filter)
{
try
{
return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter)).getRecords()));
return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withFieldNamesToInclude(storedFields(table))
.withTransaction(QContext.getQBackendTransaction())).getRecords()));
}
catch(QException e)
{
@@ -116,6 +121,22 @@ public final class SystemWrites
/*******************************************************************************
** The names of the stored fields of a table, as the projection of a system
** read. A query that names its fields is not given virtual fields, so core
** runs none of the value providers another QBit or the host may have added to
** a secured-data table: this QBit reads its own columns and nothing else, and
** a provider cannot be made to run inside a rule read. Null (every field, and
** the error core reports) for a table the instance does not have.
*******************************************************************************/
public static Set<String> storedFields(String tableName)
{
QTableMetaData table = QContext.getQInstance().getTable(tableName);
return (table == null ? null : new HashSet<>(table.getFields().keySet()));
}
/*******************************************************************************
** Insert and reject backend record errors.
*******************************************************************************/
@@ -1,56 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
/*******************************************************************************
** Optional UI API registration, with no linkage to unreleased middleware on RC.5.
*******************************************************************************/
public final class UiApiBridge
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private UiApiBridge()
{
}
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1128, removeBy=RC.6): register directly at RC.6.
*******************************************************************************/
public static void register(QInstance instance)
{
try
{
Class.forName("com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtensionsMetaData");
Class.forName("com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension").getMethod("register", QInstance.class).invoke(null, instance);
}
catch(ClassNotFoundException expected)
{
// RC.5 or a build without the uiapi source profile.
}
catch(ReflectiveOperationException e)
{
throw (new IllegalStateException("Unable to register secured-data UI API extension", e));
}
}
}
@@ -25,7 +25,7 @@ import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExten
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtensionsMetaData;
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1128, removeBy=RC.6): move to main source root.
** Secured-data extension for the released UI API.
*******************************************************************************/
public final class SecuredDataExtension
{
@@ -37,17 +37,17 @@ public final class SecuredDataExtension
}
/*******************************************************************************
** The secured-data UI API, version 1.0.0.
** The secured-data UI API, version 1.1.0.
*******************************************************************************/
public static MiddlewareSpecExtension defineExtension()
{
return (new MiddlewareSpecExtension().withName("secured-data").withSemanticVersion("1.0.0")
return (new MiddlewareSpecExtension().withName("secured-data").withSemanticVersion("1.1.0")
.withDescription("Reveal grants and break-glass status")
.withSpec(SecuredDataRevealSpecV1.class).withSpec(SecuredDataRevealOptionsSpecV1.class).withSpec(SecuredDataStatusSpecV1.class));
}
/*******************************************************************************
** Called reflectively by the producer.
** Register directly with the middleware extension catalog.
*******************************************************************************/
public static void register(QInstance instance)
{
@@ -41,7 +41,7 @@ public class SecuredDataRevealOptionsSpecV1 extends AbstractJsonEndpointSpec
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/reveal/options").withHttpMethod(HttpMethod.GET)
.withTag(() -> "Secured data").withShortSummary("RevealOptions"));
.withTag(() -> "secured-data").withShortSummary("RevealOptions"));
}
/*******************************************************************************
@@ -28,6 +28,7 @@ import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
import com.kingsrook.qqq.middleware.javalin.specs.v1.responses.BasicErrorResponseV1;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import com.kingsrook.qqq.openapi.model.Parameter;
import com.kingsrook.qqq.openapi.model.Schema;
@@ -44,7 +45,7 @@ public class SecuredDataRevealSpecV1 extends AbstractJsonEndpointSpec
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/reveal").withHttpMethod(HttpMethod.POST)
.withTag(() -> "Secured data").withShortSummary("Reveal"));
.withTag(() -> "secured-data").withShortSummary("Reveal"));
}
/*******************************************************************************
@@ -81,10 +82,10 @@ public class SecuredDataRevealSpecV1 extends AbstractJsonEndpointSpec
@Override
public List<com.kingsrook.qqq.middleware.javalin.specs.BasicResponse> defineAdditionalBasicResponses()
{
return (List.of(new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.BAD_REQUEST, "Invalid reveal request", "SecuredDataErrorV1"),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.FORBIDDEN, "Reveal denied", "SecuredDataErrorV1"),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.NOT_FOUND, "Record not found or locked", "SecuredDataErrorV1"),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.TOO_MANY_REQUESTS, "Daily reveal cap reached", "SecuredDataErrorV1")));
return (List.of(new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.BAD_REQUEST, "Invalid reveal request", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.FORBIDDEN, "Reveal denied", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.NOT_FOUND, "Record not found or locked", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.TOO_MANY_REQUESTS, "Daily reveal cap reached", BasicErrorResponseV1.class.getSimpleName())));
}
/*******************************************************************************
@@ -58,7 +58,8 @@ public final class SecuredDataSchemas
schemas.put("SecuredDataRevealAllowanceV1", object("tableName", type(Type.STRING), "fieldName", type(Type.STRING), "remainingDailyAllowance", type(Type.INTEGER).withNullable(true)));
schemas.put("SecuredDataStatusResponseV1", object("breakGlass", new Schema().withRefToSchema("SecuredDataBreakGlassV1").withNullable(true),
"revealsUsedToday", type(Type.INTEGER), "revealAllowances", type(Type.ARRAY).withItems(new Schema().withRefToSchema("SecuredDataRevealAllowanceV1"))));
schemas.put("SecuredDataErrorV1", object("error", type(Type.STRING)));
schemas.put("SecuredDataErrorV1", object("error", type(Type.STRING))
.withDescription("Deprecated since 1.1.0: errors use BasicErrorResponseV1."));
return (schemas);
}
@@ -41,7 +41,7 @@ public class SecuredDataStatusSpecV1 extends AbstractJsonEndpointSpec
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/status").withHttpMethod(HttpMethod.GET)
.withTag(() -> "Secured data").withShortSummary("Status"));
.withTag(() -> "secured-data").withShortSummary("Status"));
}
/*******************************************************************************
@@ -1,5 +1,44 @@
components:
schemas:
BasicErrorResponseV1:
description: "The error body of every UI API operation (core and extensions)."
properties:
error:
description: "The error message (the same as message; kept for clients of\
\ UI API 1.0 to 1.5)."
type: "string"
code:
description: "A stable, machine-readable code. Well-known values: BAD_REQUEST\
\ (400), UNAUTHENTICATED (401),\nPERMISSION_DENIED (403), NOT_FOUND (404),\
\ CONFLICT (409), RATE_LIMITED (429), INTERNAL_ERROR (500),\nERROR (other).\
\ Extensions may add codes; treat an unknown code by the HTTP status.\
\ Since 1.6.0."
type: "string"
message:
description: "A message to show the user. Since 1.6.0."
type: "string"
fieldErrors:
description: "Errors about particular request fields, when known. Since\
\ 1.6.0."
items:
properties:
field:
description: "The field (or parameter) name."
type: "string"
message:
description: "The error about that field."
type: "string"
code:
description: "An optional machine-readable code, e.g. REQUIRED, TOO_LONG,\
\ PATTERN."
type: "string"
type: "object"
type: "array"
traceId:
description: "Identifies this request in the server logs (also the X-QQQ-Trace-Id\
\ response header); quote it when reporting a problem. Since 1.6.0."
type: "string"
type: "object"
SecuredDataBreakGlassV1:
properties:
id:
@@ -16,6 +55,7 @@ components:
type: "string"
type: "object"
SecuredDataErrorV1:
description: "Deprecated since 1.1.0: errors use BasicErrorResponseV1."
properties:
error:
type: "string"
@@ -96,19 +136,53 @@ components:
type: "object"
securitySchemes:
sessionUuidCookie:
description: "The session cookie set by POST /qqq/v1/manageSession (web clients)."
in: "cookie"
name: "sessionUUID"
type: "apiKey"
bearerAuth:
bearerFormat: "JWT or session UUID"
description: "`Authorization: Bearer <token>`, where the token is either\n-\
\ an OAuth2 access token (a signed JWT) issued by the app's identity provider\
\ to the app's client,\n or to its native client (`values.nativeClientId`\
\ in GET /qqq/v1/metaData/authentication), for\n example from the device-authorization\
\ grant or auth-code + PKCE. It is verified against the\n provider's keys\
\ (issuer, audience, expiry, signature) and gives the same user and permissions\n\
\ as signing in on the web. When it expires (401), refresh it at the provider\
\ and retry; or\n- a session UUID returned by POST /qqq/v1/manageSession,\
\ which the server keeps refreshed."
scheme: "bearer"
type: "http"
oauth2:
description: "The app's OAuth2 / OpenID Connect provider. The real endpoints\
\ come from the provider's discovery\ndocument: `{values.baseUrl}.well-known/openid-configuration`,\
\ with values from\nGET /qqq/v1/metaData/authentication. Native clients use\
\ `values.nativeClientId` (a public client)\nwith auth-code + PKCE (loopback\
\ or custom-scheme redirect URI registered at the provider) or the\ndevice-authorization\
\ grant (`device_authorization_endpoint`, RFC 8628), then send the access\
\ token\nas `Authorization: Bearer`."
flows:
authorizationCode:
authorizationUrl: "/authorize"
scopes:
email: "Email address"
offline_access: "A refresh token, to renew the access token without signing\
\ in again"
openid: "OpenID Connect sign-in"
profile: "Name, username and groups (permissions come from the groups)"
tokenUrl: "/token"
type: "oauth2"
info:
contact:
email: "contact@kingsrook.com"
description: "Reveal grants and break-glass status"
title: "QQQ Middleware API extension: secured-data"
version: "1.0.0"
version: "1.1.0"
openapi: "3.0.3"
paths:
/qqq/v1/qbit/secured-data/reveal:
post:
operationId: "postQbitSecuredDataReveal"
requestBody:
content:
application/json:
@@ -120,19 +194,19 @@ paths:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataErrorV1"
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Invalid reveal request"
403:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataErrorV1"
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Reveal denied"
404:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataErrorV1"
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Record not found or locked"
200:
content:
@@ -144,16 +218,37 @@ paths:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataErrorV1"
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Daily reveal cap reached"
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "Reveal"
tags:
- "Secured data"
- "secured-data"
/qqq/v1/qbit/secured-data/reveal/options:
get:
operationId: "getQbitSecuredDataRevealOptions"
parameters:
- in: "query"
name: "tableName"
@@ -172,14 +267,61 @@ paths:
schema:
$ref: "#/components/schemas/SecuredDataRevealOptionsResponseV1"
description: "Successful Response"
400:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Bad request: a parameter or the body is invalid (code BAD_REQUEST);\
\ fieldErrors may name the fields."
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
403:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "The user (or the credential's scope) may not do this (code\
\ PERMISSION_DENIED)."
404:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not found, or not visible to this user (code NOT_FOUND)."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
429:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Too many requests (code RATE_LIMITED); retry later."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "RevealOptions"
tags:
- "Secured data"
- "secured-data"
/qqq/v1/qbit/secured-data/status:
get:
operationId: "getQbitSecuredDataStatus"
responses:
200:
content:
@@ -187,9 +329,61 @@ paths:
schema:
$ref: "#/components/schemas/SecuredDataStatusResponseV1"
description: "Successful Response"
400:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Bad request: a parameter or the body is invalid (code BAD_REQUEST);\
\ fieldErrors may name the fields."
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
403:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "The user (or the credential's scope) may not do this (code\
\ PERMISSION_DENIED)."
404:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not found, or not visible to this user (code NOT_FOUND)."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
429:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Too many requests (code RATE_LIMITED); retry later."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "Status"
tags:
- "Secured data"
- "secured-data"
servers:
- description: "The QQQ server that serves this document (paths include /qqq/v1)."
url: "/"
tags:
- description: "Operations of the secured-data extension."
name: "secured-data"
@@ -1,34 +0,0 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.file.Path;
import com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension;
import org.junit.jupiter.api.Test;
class SecuredDataUiApiContractTest
{
@Test
void committedContractMatches() throws Exception
{
UiApiContractAssertions.assertExtensionMatchesCommitted(SecuredDataExtension.defineExtension(), Path.of("src/main/resources/openapi/qqq-v1/secured-data.yaml"));
}
}
@@ -35,8 +35,6 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
@@ -55,34 +53,13 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Core field access bridge it.
** Core field access integration tests.
*******************************************************************************/
class CoreFieldAccessBridgeIT
class CoreFieldAccessIT
{
static final String EMAIL = "CANARY-EMAIL-918273@example.invalid";
static final String SSN = "CANARY-PHI-192837465";
/*******************************************************************************
** Require core.
*******************************************************************************/
@BeforeEach
void requireCore()
{
boolean available;
try
{
Class.forName("com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverMetaData");
available = true;
}
catch(ClassNotFoundException expected)
{
available = false;
}
Assumptions.assumeTrue(available, "QQQ RC.6 field-access core is not on this classpath");
}
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@@ -0,0 +1,119 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.core.Appender;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.LoggerContext;
import org.apache.logging.log4j.core.appender.AbstractAppender;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.LoggerConfig;
import org.apache.logging.log4j.core.config.Property;
/*******************************************************************************
** Captures every log event, at every level, from every logger while open.
** Used by the canary test to prove no protected value reaches a log line.
*******************************************************************************/
final class LogCapture implements AutoCloseable
{
private final List<String> lines = new ArrayList<>();
private final Map<String, Level> originalLevels = new HashMap<>();
private final LoggerContext context;
private final Appender appender;
/*******************************************************************************
** Attach to the root logger and force all configured loggers to ALL.
*******************************************************************************/
LogCapture()
{
context = (LoggerContext) LogManager.getContext(false);
Configuration configuration = context.getConfiguration();
appender = new AbstractAppender("secured-data-log-capture", null, null, true, Property.EMPTY_ARRAY)
{
@Override
public void append(LogEvent event)
{
StringBuilder line = new StringBuilder(event.getLoggerName()).append(' ').append(event.getMessage().getFormattedMessage());
for(Throwable t = event.getThrown(); t != null; t = t.getCause())
{
line.append(' ').append(t.getClass().getName()).append(' ').append(t.getMessage());
}
synchronized(lines)
{
lines.add(line.toString());
}
}
};
appender.start();
configuration.addAppender(appender);
List<LoggerConfig> all = new ArrayList<>(configuration.getLoggers().values());
all.add(configuration.getRootLogger());
for(LoggerConfig config : all)
{
originalLevels.put(config.getName(), config.getLevel());
config.setLevel(Level.ALL);
}
configuration.getRootLogger().addAppender(appender, Level.ALL, null);
context.updateLoggers();
}
/*******************************************************************************
** Every captured line so far.
*******************************************************************************/
List<String> lines()
{
synchronized(lines)
{
return (new ArrayList<>(lines));
}
}
/*******************************************************************************
** Detach and restore levels.
*******************************************************************************/
@Override
public void close()
{
Configuration configuration = context.getConfiguration();
configuration.getRootLogger().removeAppender(appender.getName());
for(LoggerConfig config : configuration.getLoggers().values())
{
config.setLevel(originalLevels.get(config.getName()));
}
configuration.getRootLogger().setLevel(originalLevels.get(configuration.getRootLogger().getName()));
context.updateLoggers();
appender.stop();
}
}
@@ -0,0 +1,388 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.ByteArrayOutputStream;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySource.KeyAndId;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.reporting.ExportAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.rows;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.rule;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** SECURED-DATA section 15 proofs: a planted canary never leaves the host table,
** and shredded audit tokens cannot be brute-forced once the data key is gone.
*******************************************************************************/
class RetentionProofTest
{
private static final SecureRandom RANDOM = new SecureRandom();
private static final String OTHER_KEY = "/v79vLu6ubi3trW0s7KxsK+uraypqKempaSjoqGgn56dnJuamZiXlpU=";
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
RetentionTestSupport.close();
}
private static final String CANARY_PREFIX = "CANARY-PHI-";
private static final int CANARY_RANDOM = 24;
private static final int MIN_LEAK_FRAGMENT = 12;
/*******************************************************************************
** A fixed prefix and 24 random letters g-v (so it never coincides with hex
** digests). Leaks are matched on the whole marker or on 12-letter windows of
** its random part: 16^12 possibilities, so ordinary text, JSON words or
** base64 tokens cannot match by chance, while a truncated or re-encoded
** fragment of a real leak still does.
*******************************************************************************/
private static String canary()
{
StringBuilder marker = new StringBuilder(CANARY_PREFIX);
for(int i = 0; i < CANARY_RANDOM; i++)
{
marker.append((char) ('g' + RANDOM.nextInt(16)));
}
return (marker.toString());
}
/*******************************************************************************
** The whole marker, or any MIN_LEAK_FRAGMENT-letter window of its random part.
*******************************************************************************/
private static List<String> leaks(String haystack, String... markers)
{
List<String> found = new ArrayList<>();
for(String marker : markers)
{
if(haystack.contains(marker))
{
found.add("whole marker");
continue;
}
String random = marker.substring(CANARY_PREFIX.length());
for(int start = 0; start + MIN_LEAK_FRAGMENT <= random.length(); start++)
{
if(haystack.contains(random.substring(start, start + MIN_LEAK_FRAGMENT)))
{
found.add("fragment at " + start);
}
}
}
return (found);
}
/*******************************************************************************
** The leak scan: whole markers and 12-letter fragments are found; the shared
** prefix and ordinary words made of the same letters are not.
*******************************************************************************/
@Test
void leakScanHasNoFalsePositives()
{
String marker = canary();
String random = marker.substring(CANARY_PREFIX.length());
assertThat(leaks("x" + marker + "x", marker)).containsExactly("whole marker");
assertThat(leaks("{\"v\":\"" + random.substring(5, 5 + MIN_LEAK_FRAGMENT) + "\"}", marker)).isNotEmpty();
assertThat(leaks(CANARY_PREFIX + " options joins strings rows points", marker)).isEmpty();
assertThat(leaks(random.substring(0, MIN_LEAK_FRAGMENT - 1), marker)).isEmpty();
}
/*******************************************************************************
** Canary through CRUD, a failing validation, a wipe run, a masked export and
** the access log: it may appear only in the host table itself.
*******************************************************************************/
@Test
void canaryNeverLeavesTheHostTable() throws Exception
{
String first = canary();
String second = canary();
String rejected = canary();
try(LogCapture logs = new LogCapture())
{
QInstance instance = RetentionTestSupport.setup();
QLogger.getLogger(RetentionProofTest.class).info("log-capture-sentinel");
assertThat(logs.lines()).anyMatch(line -> line.contains("log-capture-sentinel"));
SecuredDataTestSupport.policy("viewer", "PHI", "MASKED");
SecuredDataTestSupport.policy("viewer", "PII", "MASKED");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = rule("NULL_WITH_MARKER", "secret,notes");
// CRUD with audits.
List<String> errors = new ArrayList<>();
QRecord inserted = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("secret", first).withValue("notes", first)
.withValue("systolic", 120).withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst();
assertThat(inserted.getErrors()).isNullOrEmpty();
Serializable id = inserted.getValue("id");
QRecord updated = new UpdateAction().execute(new UpdateInput("reading").withRecord(new QRecord().withValue("id", id).withValue("secret", second)
.withValue("notes", second))).getRecords().getFirst();
assertThat(updated.getErrors()).isNullOrEmpty();
// A failing validation: the secret value is over the field's maximum length.
QRecord invalid = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("secret", rejected + "x".repeat(300))
.withValue("notes", rejected).withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst();
assertThat(invalid.getErrors()).as("the invalid insert must be reported as an error").isNotEmpty();
invalid.getErrors().forEach(error -> errors.add(String.valueOf(error.getMessage())));
// A masked user reads and exports; the access log records the views.
QContext.setQSession(SecuredDataTestSupport.user("viewer", "viewer", "reading.read"));
assertThat(new QueryAction().execute(new QueryInput("reading")).getRecords()).hasSize(1);
ByteArrayOutputStream export = new ByteArrayOutputStream();
ExportInput exportInput = new ExportInput();
exportInput.setTableName("reading");
new ExportAction().execute(exportInput.withReportDestination(new ReportDestination().withReportFormat(ReportFormat.CSV).withReportOutputStream(export)));
String exported = export.toString(StandardCharsets.UTF_8);
assertThat(exported).isNotBlank();
SecuredDataRuntime.current().logger().flush();
// Positive control: the scan does detect the planted value where it legitimately lives.
assertThat(leaks(JsonUtils.toJson(rows("reading")), second)).isNotEmpty();
// The wipe run, as the retention administrator.
QContext.setQSession(SecuredDataTestSupport.user("admin", "secured_data.admin", "secured_data.retention", "reading.read", "reading.write"));
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(id));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(rows("reading").getFirst().getValue("secret")).isNull();
assertThat(leaks(JsonUtils.toJson(rows("reading")), first, second)).as("the wiped host table").isEmpty();
List<String> places = new ArrayList<>();
Map<String, String> haystacks = new LinkedHashMap<>();
haystacks.put("export", exported);
haystacks.put("errors", String.join(" ", errors));
haystacks.put("logs", String.join("\n", logs.lines()));
for(String table : List.of("audit", "auditDetail", SecuredDataTables.ACCESS_LOG, SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER,
SecuredDataTables.AUDIT_KEY, SecuredDataTables.RETENTION_RULE, SecuredDataTables.LEGAL_HOLD, SecuredDataTables.POLICY, SecuredDataTables.CLASSIFICATION))
{
haystacks.put(table, JsonUtils.toJson(rows(table)));
assertThat(instance.getTable(table)).isNotNull();
}
haystacks.forEach((name, text) ->
{
if(!leaks(text, first, second, rejected).isEmpty())
{
places.add(name);
}
});
assertThat(places).as("places where the canary or a long fragment of it appeared").isEmpty();
assertThat(haystacks.get("audit")).isNotBlank();
assertThat(rows("auditDetail")).isNotEmpty();
assertThat(rows(SecuredDataTables.ACCESS_LOG)).isNotEmpty();
}
}
/*******************************************************************************
** HMAC-SHA256 over the agreed record binding, exactly as the token format does.
*******************************************************************************/
private static String digest(byte[] key, QTableMetaData table, String field, Serializable pk, int value) throws Exception
{
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(key, "HmacSHA256"));
byte[] out = mac.doFinal((String.join("\u001F", table.getName(), field, String.valueOf(pk)) + "\u001F" + value).getBytes(StandardCharsets.UTF_8));
return (Base64.getUrlEncoder().withoutPadding().encodeToString(out));
}
/*******************************************************************************
** After the wipe nothing can recover a systolic reading from its audit tokens,
** not even with every key still reachable, while before the wipe the same
** procedure does find the value (so the search genuinely detects a match).
*******************************************************************************/
@Test
void shreddedAuditTokensCannotBeBruteForcedAfterTheWipe() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
System.setProperty("QQQ_AUDIT_HMAC_KEY", OTHER_KEY);
QTableMetaData table = instance.getTable("reading");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
List<QRecord> toInsert = new ArrayList<>();
for(int systolic = 60; systolic <= 250; systolic++)
{
toInsert.add(new QRecord().withValue("systolic", systolic).withValue("secret", "s").withValue("reading_time", RetentionTestSupport.OLD));
}
List<QRecord> inserted = new InsertAction().execute(new InsertInput("reading").withRecords(toInsert)).getRecords();
assertThat(inserted).hasSize(191).allSatisfy(r -> assertThat(r.getErrors()).isNullOrEmpty());
Map<Serializable, Integer> truth = new LinkedHashMap<>();
inserted.forEach(r -> truth.put(r.getValue("id"), r.getValueInteger("systolic")));
// A control record the wipe will not touch: its data key stays reachable.
QRecord control = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 111)
.withValue("reading_time", SecuredDataTestSupport.NOW))).getRecords().getFirst();
Map<Serializable, List<String>> tokens = auditTokens(truth.keySet());
assertThat(tokens).hasSize(191).allSatisfy((id, list) -> assertThat(list).isNotEmpty().allMatch(t -> t.startsWith("shred:v1:")));
// Before the wipe, the DEK verifies the true value and no other value.
var keys = SecuredDataRuntime.current().keys();
for(var entry : truth.entrySet())
{
Optional<KeyAndId> dek = keys.keyForRecordField("reading", "systolic", entry.getKey(), false);
assertThat(dek).isPresent();
for(String token : tokens.get(entry.getKey()))
{
assertThat(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), entry.getValue(), token)).isTrue();
assertThat(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), entry.getValue() + 1, token)).isFalse();
assertThat(token).endsWith(digest(dek.get().key(), table, "systolic", entry.getKey(), entry.getValue()));
}
}
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(192);
// Wipe the old readings in policy-sized batches.
List<Serializable> ids = new ArrayList<>(truth.keySet());
for(int offset = 0; offset < ids.size(); offset += 100)
{
new WipeService().batch(rule.getValue("id"), ids.subList(offset, Math.min(offset + 100, ids.size())));
}
assertThat(rows("reading").stream().filter(r -> truth.containsKey(r.getValue("id")))).allSatisfy(r -> assertThat(r.getValue("systolic")).isNull());
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(1);
assertThat(rows(SecuredDataTables.AUDIT_KEY).getFirst().getValueString("record_id")).isEqualTo(String.valueOf(control.getValue("id")));
// Every key the test can still reach.
List<byte[]> reachable = new ArrayList<>();
reachable.add(Base64.getDecoder().decode(OTHER_KEY));
reachable.add(Base64.getDecoder().decode(RetentionTestSupport.KEY));
reachable.add(keys.keyForRecordField("reading", "systolic", control.getValue("id"), false).orElseThrow().key());
reachable.add(new byte[32]);
assertThat(reachable.get(2)).isNotEqualTo(new byte[32]);
Map<Serializable, List<String>> after = auditTokens(truth.keySet());
int matches = 0;
for(var entry : truth.entrySet())
{
assertThat(after.get(entry.getKey())).containsAll(tokens.get(entry.getKey()));
for(String token : after.get(entry.getKey()))
{
String tail = token.substring(token.lastIndexOf(':') + 1);
for(int guess = 60; guess <= 250; guess++)
{
if(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), guess, token))
{
matches++;
}
for(byte[] key : reachable)
{
if(digest(key, table, "systolic", entry.getKey(), guess).equals(tail))
{
matches++;
}
}
}
}
}
assertThat(matches).as("brute-force matches after the wipe").isZero();
assertThat(rows("auditDetail")).anySatisfy(row -> assertThat(row.getValueString("newValue")).startsWith("shredded:v1:"));
}
/*******************************************************************************
** Every systolic audit-detail value for the given records, by record id.
*******************************************************************************/
private static Map<Serializable, List<String>> auditTokens(Iterable<Serializable> ids)
{
Map<Serializable, String> recordByAudit = new LinkedHashMap<>();
for(QRecord audit : SystemWrites.query("audit", new QQueryFilter()))
{
recordByAudit.put(audit.getValue("id"), audit.getValueString("recordId"));
}
Map<Serializable, List<String>> result = new LinkedHashMap<>();
for(Serializable id : ids)
{
result.put(id, new ArrayList<>());
}
for(QRecord detail : SystemWrites.query("auditDetail", new QQueryFilter()))
{
if(!"systolic".equals(detail.getValueString("fieldName")))
{
continue;
}
String record = recordByAudit.get(detail.getValue("auditId"));
for(Serializable id : ids)
{
if(String.valueOf(id).equals(record))
{
for(String value : new String[] {detail.getValueString("oldValue"), detail.getValueString("newValue")})
{
if(value != null && !value.isBlank())
{
result.get(id).add(value);
}
}
}
}
}
return (result);
}
}
@@ -0,0 +1,147 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.time.Instant;
import java.util.List;
import java.util.UUID;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
final class RetentionTestSupport
{
static String BACKEND = "retention_rdbms";
static final String KEY = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=";
static final Instant OLD = SecuredDataTestSupport.NOW.minusSeconds(60L * 86400);
static QInstance setup() throws Exception
{
SecuredDataTestSupport.close();
System.setProperty("QQQ_SECURED_DATA_KEK", KEY);
System.setProperty("QQQ_AUDIT_HMAC_KEY", KEY);
SecuredDataTestSupport.CLOCK.now = SecuredDataTestSupport.NOW;
String db = "retention_" + UUID.randomUUID().toString().replace("-", "");
BACKEND = db; // pools are cached by backend name, so each instance needs its own
String url = "jdbc:h2:mem:" + db + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1";
RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName(BACKEND).withVendor("h2").withHostName("mem").withDatabaseName(db).withUsername("sa");
backend.setJdbcUrl(url);
QInstance instance = SecuredDataTestSupport.base();
instance.addBackend(backend);
QTableMetaData table = new QTableMetaData().withName("reading").withLabel("Reading").withBackendName(BACKEND).withPrimaryKeyField("id")
.withBackendDetails(new RDBMSTableBackendDetails().withTableName("reading"))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS))
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("reading_time", QFieldType.DATE_TIME))
.withField(new QFieldMetaData("reading_date", QFieldType.DATE))
.withField(new QFieldMetaData("systolic", QFieldType.INTEGER).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)))
.withField(new QFieldMetaData("secret", QFieldType.STRING).withMaxLength(255).withBehavior(com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior.ERROR).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)))
.withField(new QFieldMetaData("notes", QFieldType.TEXT).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)));
table.getFields().values().forEach(f -> f.setBackendName(f.getName()));
instance.addTable(table);
new AuditsMetaDataProvider().defineAll(instance, BACKEND, t -> t.withBackendDetails(new RDBMSTableBackendDetails().withTableName(t.getName())));
try(Connection connection = DriverManager.getConnection(url, "sa", ""))
{
for(QTableMetaData t : instance.getTables().values())
{
StringBuilder sql = new StringBuilder("CREATE TABLE ").append(t.getName()).append(" (");
for(QFieldMetaData field : t.getFields().values())
{
field.setBackendName(field.getName());
sql.append(field.getName()).append(" ");
if(field.getName().equals(t.getPrimaryKeyField()))
{
sql.append("INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY");
}
else
{
sql.append(switch(field.getType())
{
case INTEGER -> "INTEGER";
case LONG -> "BIGINT";
case DATE_TIME -> "TIMESTAMP WITH TIME ZONE";
case DATE -> "DATE";
case BOOLEAN -> "BOOLEAN";
case DECIMAL -> "DECIMAL";
default -> "TEXT";
});
}
sql.append(",");
}
sql.setLength(sql.length() - 1);
connection.createStatement().execute(sql.append(")").toString());
}
for(String sql : Files.readString(Path.of("docs/ddl/secured-data-postgres.sql")).split(";"))
{
if(!sql.isBlank()) connection.createStatement().execute(sql);
}
}
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withBackendName(BACKEND).withClock(SecuredDataTestSupport.CLOCK)
.withBackupRetentionDays(35).withAccessLogFlushIntervalMillis(60000)
.withTableMetaDataCustomizer((i,t) -> t.withBackendDetails(new RDBMSTableBackendDetails().withTableName(t.getName())))).produce(instance);
QContext.init(instance, SecuredDataTestSupport.user("admin", "secured_data.admin", "secured_data.retention", "reading.read", "reading.write"));
SecuredDataTestSupport.RUNTIMES.add(SecuredDataRuntime.current());
new QInstanceValidator().validate(instance);
return instance;
}
static QRecord rule(String action, String fields) throws Exception
{
return SystemWrites.insertOne(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30-day policy").withValue("table_name", "reading")
.withValue("field_names", fields).withValue("age_field_name", "reading_time").withValue("age_days", 30).withValue("action", action)
.withValue("archive_storage_name", "archive"));
}
static QRecord reading(Object value) throws Exception
{
return SystemWrites.insertOne("reading", new QRecord().withValue("systolic", (java.io.Serializable)value).withValue("secret", "test-secret")
.withValue("notes", "test-notes").withValue("reading_time", OLD).withValue("reading_date", OLD.atZone(java.time.ZoneOffset.UTC).toLocalDate()));
}
static List<QRecord> rows(String table)
{
return SystemWrites.query(table, new QQueryFilter());
}
static void close()
{
SecuredDataTestSupport.close();
System.clearProperty("QQQ_SECURED_DATA_KEK");
System.clearProperty("QQQ_AUDIT_HMAC_KEY");
System.clearProperty("QQQ_AUDIT_HMAC_KEY_ID");
}
}
@@ -0,0 +1,265 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.retention.RetentionWidgets;
import com.kingsrook.qbits.secureddata.retention.WipeProcesses;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.audits.DMLAuditAction.DMLType;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class RetentionWipeTest
{
@AfterEach void cleanup() { close(); }
@ParameterizedTest
@ValueSource(strings={"NULL_WITH_MARKER", "OVERWRITE_TEXT", "KEYED_HASH", "DELETE_RECORD", "ARCHIVE_THEN_WIPE"})
void eachActionIsTransactionalAndValueFree(String action) throws Exception
{
var instance = setup();
List<String> archiveJson = new ArrayList<>();
SecuredDataRuntime.current().config().withArchiveSink((store, key, bytes) -> {
assertThat(store).isEqualTo("archive");
assertThat(new String(bytes, StandardCharsets.UTF_8)).doesNotContain("test-secret");
archiveJson.add(new String(SecuredDataKeySource.decrypt(bytes, key), StandardCharsets.UTF_8));
});
QRecord rule = rule(action, "secret");
QRecord record = reading(120);
var keys = SecuredDataRuntime.current().keys();
keys.keyForRecordField("reading", "secret", record.getValue("id"), true);
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isNotEmpty();
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isEmpty();
assertThat(keys.keyForRecordField("reading", "secret", record.getValue("id"), false)).isEmpty();
if(action.equals("DELETE_RECORD")) assertThat(rows("reading")).isEmpty();
else {
QRecord remaining = rows("reading").getFirst();
if(action.equals("OVERWRITE_TEXT")) assertThat(remaining.getValue("secret")).isEqualTo("[wiped per policy]");
else if(action.equals("KEYED_HASH")) {
assertThat(remaining.getValueString("secret")).startsWith("hmac:v1:");
assertThat(ClassifiedValueTokens.verifyRecordField(instance.getTable("reading"), instance.getTable("reading").getField("secret"), record.getValue("id"), "test-secret", remaining.getValueString("secret"))).isTrue();
}
else assertThat(remaining.getValue("secret")).isNull();
}
if(action.equals("ARCHIVE_THEN_WIPE")) assertThat(archiveJson).containsExactly("{\"secret\":\"test-secret\"}");
else assertThat(archiveJson).isEmpty();
assertThat(rows(SecuredDataTables.WIPE_MARKER)).isNotEmpty();
assertThat(rows(SecuredDataTables.ACCESS_LOG).stream().filter(r -> "WIPE".equals(r.getValueString("event_type"))).toList()).isNotEmpty()
.allSatisfy(r -> { assertThat(r.getValueString("record_id")).isEqualTo(record.getValueString("id")); assertThat(r.getValueInteger("record_count")).isEqualTo(1); });
assertThat(rows("auditDetail")).anySatisfy(r -> assertThat(r.getValueString("newValue")).isEqualTo("shredded:v1:" + rule.getValueString("id")));
}
@Test void recordAndTableLegalHoldsAndAgeAreRechecked() throws Exception
{
setup();
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
QRecord one = reading(120), two = reading(130);
QRecord hold = SystemWrites.insertOne(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", one.getValueString("id")).withValue("reason", "Preservation"));
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"), two.getValue("id")));
assertThat(run.getValueInteger("held_count")).isEqualTo(1);
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
SecuredDataTestSupport.update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("record_id", null));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"))).getValueInteger("held_count")).isEqualTo(1);
SecuredDataTestSupport.update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED"));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"))).getValueInteger("wiped_count")).isEqualTo(1);
QRecord young = reading(140);
SecuredDataTestSupport.update("reading", new QRecord().withValue("id", young.getValue("id")).withValue("reading_time", SecuredDataTestSupport.NOW));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(young.getValue("id"))).getValueInteger("eligible_count")).isZero();
}
@Test void failureAfterFirstRecordRollsBackUpdatesKeysMarkersAuditsAndRun() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret");
QRecord first = reading(120), second = reading(130);
for(QRecord row : List.of(first, second)) SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", row.getValue("id"), true);
int audits = rows("auditDetail").size();
FailSecond.id = second.getValue("id");
instance.getTable("reading").withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(FailSecond.class));
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(first.getValue("id"), second.getValue("id")))).hasMessageContaining("rolled back");
assertThat(rows("reading")).allSatisfy(r -> assertThat(r.getValue("secret")).isEqualTo("test-secret"));
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(2);
assertThat(rows(SecuredDataTables.WIPE_MARKER)).isEmpty();
assertThat(rows(SecuredDataTables.WIPE_RUN)).isEmpty();
assertThat(rows("auditDetail")).hasSize(audits);
}
public static class FailSecond implements TableCustomizerInterface
{
static java.io.Serializable id;
@Override public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> old) throws QException
{
if(records.stream().anyMatch(r -> id.equals(r.getValue("id")))) throw new QException("Injected failure");
return records;
}
}
@Test void reviewResultAndDailySchedule() throws Exception
{
setup();
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
reading(120);
QRecord held = reading(130);
SystemWrites.insertOne(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", held.getValueString("id")).withValue("reason", "Preserve"));
var lines = ProcessRunner.run(WipeProcesses.WIPE, rule.getValue("id"));
assertThat(lines.review()).anyMatch(s -> s.contains("to wipe")).anyMatch(s -> s.contains("held"));
assertThat(lines.result()).anyMatch(s -> s.contains("wiped")).anyMatch(s -> s.contains("held"));
reading(150);
assertThatThrownBy(() -> new WipeProcesses.Daily().run(new RunBackendStepInput(), new RunBackendStepOutput())).hasMessageContaining("system session");
QContext.setQSession(new QSystemUserSession(QContext.getQInstance()));
var output = new RunBackendStepOutput();
new WipeProcesses.Daily().run(new RunBackendStepInput(), output);
assertThat(output.getRecords()).anySatisfy(r -> assertThat(r.getValueInteger("wiped_count")).isEqualTo(1));
var scheduled = WipeProcesses.processes(new SecuredDataQBitConfig().withSchedulerName("test")).getLast();
assertThat(scheduled.getSchedule().getRepeatSeconds()).isEqualTo(86400);
}
@Test void permissionsKeyLockAndBackupsPanels() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret"), record = reading(120);
SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", record.getValue("id"), true);
var keyTable = instance.getTable(SecuredDataTables.AUDIT_KEY);
assertThat(keyTable.getField("wrapped_key").getFieldSecurityLock().getOverrideValues()).containsExactly(com.kingsrook.qbits.secureddata.metadata.RetentionTables.KEY_LOCK_NEVER_GRANTED);
assertThat(keyTable.getDisabledCapabilities()).contains(com.kingsrook.qqq.backend.core.model.metadata.tables.Capability.TABLE_EXPORT);
assertThat(SecuredDataTables.MAIN).doesNotContain(SecuredDataTables.AUDIT_KEY);
assertThat(new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)).getRecords()).allSatisfy(r -> assertThat(r.getValue("wrapped_key")).isNull());
assertThat(RetentionRules.backupWarning(rule)).contains("35 days", "30 days", "recoverable");
assertThat(com.kingsrook.qbits.secureddata.app.SecuredDataHomeView.render(com.kingsrook.qbits.secureddata.app.SecuredDataHomeData.load())).contains("Backups keep data for 35 days");
RenderWidgetInput input = new RenderWidgetInput().withWidgetMetaData(RetentionWidgets.backups());
input.getQueryParams().put("id", rule.getValueString("id"));
assertThat(new RetentionWidgets.Backups().render(input)).isNotNull();
new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")));
var widget = RetentionWidgets.wipedFields(instance.getTable("reading"));
input = new RenderWidgetInput().withWidgetMetaData(widget); input.getQueryParams().put("id", record.getValueString("id"));
assertThat(new RetentionWidgets.WipedFields().render(input)).isNotNull();
QContext.setQSession(SecuredDataTestSupport.user("ordinary"));
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")))).hasMessageContaining("secured_data.retention");
}
@Test void changedOnlyAndWrapUnwrap() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret");
SecuredDataTestSupport.update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", rule.getValue("id")).withValue("audit_treatment", "CHANGED_ONLY"));
var table = instance.getTable("reading");
var protectedValues = new SecuredDataAuditProtector().protect(table, table.getField("secret"), 1, "old", "new", DMLType.UPDATE);
assertThat(protectedValues.oldToken()).isEqualTo("changed:v1"); assertThat(protectedValues.newToken()).isEqualTo("changed:v1");
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isEmpty();
byte[] plain = Base64.getDecoder().decode(KEY);
byte[] encrypted = SecuredDataKeySource.encrypt(plain, "binding");
assertThat(SecuredDataKeySource.decrypt(encrypted, "binding")).isEqualTo(plain);
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(encrypted, "wrong-binding")).isInstanceOf(IllegalStateException.class);
encrypted[15] ^= 1;
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(encrypted, "binding")).isInstanceOf(IllegalStateException.class);
System.setProperty("QQQ_SECURED_DATA_KEK", "bad");
assertThatThrownBy(() -> SecuredDataKeySource.encrypt(plain, "binding")).hasMessageContaining("QQQ_SECURED_DATA_KEK");
}
@Test void validatesPolicyAndRefusesMissingArchive() throws Exception
{
setup();
assertThatThrownBy(() -> rule("KEYED_HASH", "systolic")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> rule("NULL_WITH_MARKER", "missing")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> rule("NULL_WITH_MARKER", "secret,secret")).hasMessageContaining("insert failed");
var rule = rule("ARCHIVE_THEN_WIPE", "secret"); var record = reading(120);
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")))).hasMessageContaining("ArchiveSinkInterface");
assertThat(rows("reading").getFirst().getValue("secret")).isEqualTo("test-secret");
assertThatThrownBy(() -> new SecuredDataQBitConfig().withBackupRetentionDays(-1)).isInstanceOf(IllegalArgumentException.class);
}
@Test void wrongKekCannotUnwrapAndKeyReturnedIsNeverZeroed() throws Exception
{
setup();
rule("NULL_WITH_MARKER", "secret");
QRecord record = reading(120);
var keys = SecuredDataRuntime.current().keys();
Object pk = record.getValue("id");
String binding = "readingsecret" + pk;
// Regression: the source once returned the array it then zeroed in finally.
byte[] created = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, true).orElseThrow().key();
assertThat(created).hasSize(32).isNotEqualTo(new byte[32]);
QRecord stored = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)
.withFilter(SecuredDataKeySource.recordField("reading", "secret", (java.io.Serializable) pk)).withShouldOmitHiddenFields(false)).getRecords()).getFirst();
byte[] wrapped = Base64.getDecoder().decode(stored.getValueString("wrapped_key"));
assertThat(created).isEqualTo(SecuredDataKeySource.decrypt(wrapped, binding));
// Lookups (uncached then cached) agree with the unwrapped DEK, and callers zeroing their copy cannot poison the cache.
byte[] looked = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, false).orElseThrow().key();
assertThat(looked).isNotEqualTo(new byte[32]).isEqualTo(created);
java.util.Arrays.fill(looked, (byte) 0);
byte[] cached = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, false).orElseThrow().key();
assertThat(cached).isEqualTo(created).isNotEqualTo(new byte[32]);
// A different KEK, or a different binding, cannot unwrap.
System.setProperty("QQQ_SECURED_DATA_KEK", "/v79vLu6ubi3trW0s7KxsK+uraypqKempaSjoqGgn56dnJuamZiXlpU=");
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(wrapped, binding)).isInstanceOf(IllegalStateException.class);
System.setProperty("QQQ_SECURED_DATA_KEK", KEY);
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(wrapped, binding + "x")).isInstanceOf(IllegalStateException.class);
assertThat(SecuredDataKeySource.decrypt(wrapped, binding)).isEqualTo(created);
}
@Test void keyedHashIsRefusedOnLowEntropyFields() throws Exception
{
setup();
assertThat(rule("KEYED_HASH", "secret").getValue("id")).isNotNull();
SystemWrites.insertOne(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "reading").withValue("field_name", "secret")
.withValue("classification", "PHI").withValue("low_cardinality", true));
assertThatThrownBy(() -> rule("KEYED_HASH", "secret")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> RetentionRules.validate(new QRecord().withValue("table_name", "reading").withValue("field_names", "secret")
.withValue("age_field_name", "reading_time").withValue("age_days", 30).withValue("batch_size", 10).withValue("action", "KEYED_HASH")
.withValue("audit_treatment", "SHREDDABLE"))).hasMessageContaining("low-entropy");
assertThat(rule("OVERWRITE_TEXT", "secret").getValue("id")).isNotNull();
}
}
@@ -31,9 +31,9 @@ import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.processes.BreakGlassProcesses;
import com.kingsrook.qbits.secureddata.reveal.RevealCappedException;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
@@ -22,6 +22,7 @@ package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.lang.ref.WeakReference;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
@@ -32,10 +33,10 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEvent;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEventType;
import com.kingsrook.qbits.secureddata.standin.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
@@ -46,10 +47,10 @@ import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
@@ -360,4 +361,92 @@ class SecuredDataAccessLoggerTest
{
}
}
/*******************************************************************************
** The worker starts while the host is still building its instance. An idle
** tick opens no context, so it never validates (or enriches) an unfinished
** instance on the worker thread, and a failing tick is logged instead of
** cancelling every later tick.
*******************************************************************************/
@Test
void idleTicksNeverValidateAnUnfinishedInstance() throws Exception
{
setup();
SecuredDataRuntime runtime = SecuredDataRuntime.current();
QInstance unfinished = SecuredDataTestSupport.base();
unfinished.addTable(new QTableMetaData().withName("unfinished").withBackendName("not_registered_yet"));
SecuredDataAccessLogger idle = new SecuredDataAccessLogger(unfinished, runtime);
try
{
idle.flush();
idle.scheduledFlush();
assertThat(unfinished.getHasBeenValidated()).isFalse();
}
finally
{
idle.close();
}
}
/*******************************************************************************
** Neither the worker thread nor the JVM shutdown hook keeps a discarded
** instance alive: a host or test that drops its instance without closing
** the runtime lets it be collected.
*******************************************************************************/
@Test
void aDiscardedInstanceCanBeCollected() throws Exception
{
close();
WeakReference<QInstance> discarded = discardedInstance();
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
List<byte[]> pressure = new ArrayList<>();
while(discarded.get() != null && System.nanoTime() < deadline)
{
pressure.add(new byte[1 << 20]);
if(pressure.size() > 64)
{
pressure.clear();
}
System.gc();
Thread.sleep(20);
}
assertThat(discarded.get()).as("instance still reachable from the access logger").isNull();
}
/*******************************************************************************
** Build, use and drop an instance whose worker ticks every 10 ms.
*******************************************************************************/
private static WeakReference<QInstance> discardedInstance() throws Exception
{
QInstance instance = SecuredDataTestSupport.base();
instance.addBackend(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
instance.addTable(person("memory"));
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withAccessLogFlushIntervalMillis(10)).produce(instance);
Thread.sleep(50);
QContext.clear();
return (new WeakReference<>(instance));
}
/*******************************************************************************
** Hosts that close the runtime themselves can skip the JVM hook; close
** still drains and stops the worker.
*******************************************************************************/
@Test
void shutdownHookIsOptional() throws Exception
{
setup(config -> config.withRegisterShutdownHook(false));
var logger = SecuredDataRuntime.current().logger();
assertThat(logger.isWorkerRunning()).isTrue();
logger.onAccess(event(FieldAccessEventType.VIEW, null, null));
logger.close();
assertThat(logger.isWorkerRunning()).isFalse();
assertThat(logs()).hasSize(1);
}
}
@@ -25,16 +25,16 @@ import java.util.List;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.standin.FieldAccessEventType;
import com.kingsrook.qbits.secureddata.standin.FieldAccessOutcome;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
@@ -33,7 +33,6 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaDat
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
@@ -97,8 +96,8 @@ class SecuredDataDdlTest
classification(null, "PII");
policy("cs_admin", "PII", "REVEALABLE");
assertThat(SecuredDataRuntime.current().resolver().resolve(instance.getTable("person"), instance.getTable("person").getField("email")).tier().name()).isEqualTo("REVEALABLE");
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qbits.secureddata.standin.FieldAccessEventType.EXACT_MATCH_FILTER,
com.kingsrook.qbits.secureddata.standin.FieldAccessOutcome.MATCH, "hmac"));
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.EXACT_MATCH_FILTER,
com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome.MATCH, "hmac"));
SecuredDataRuntime.current().logger().flush();
assertThat(logs()).extracting(row -> row.getValueString("event_type")).containsExactly("CLASSIFICATION_CHANGE", "POLICY_CHANGE", "EXACT_MATCH_FILTER");
// Equal timestamps must be counted by SQL as well as the memory backend.
@@ -121,4 +120,73 @@ class SecuredDataDdlTest
.isInstanceOf(java.sql.SQLException.class);
}
}
/*******************************************************************************
** Every table the producer declares has the columns create_date and
** modify_date in a database made from the DDL. The tables that were first
** created without them get them from the last block of the file, which is what
** an existing database runs: nullable columns that default to the current
** time, so an insert that names neither still succeeds, and a block that can
** be run again and then changes nothing.
*******************************************************************************/
@Test
void postgresDdlGivesEveryDeclaredTableTheTwoDateColumns() throws Exception
{
String url = "jdbc:h2:mem:secured_data_dates;MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1";
QInstance instance = SecuredDataTestSupport.setup();
try(Connection connection = DriverManager.getConnection(url, "sa", ""))
{
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(false));
java.util.List<String> added = new java.util.ArrayList<>();
for(var table : SecuredDataStandardDatesTest.declared(instance))
{
if(!hasColumn(connection, table.getName(), "create_date"))
{
assertThat(hasColumn(connection, table.getName(), "modify_date")).as(table.getName()).isFalse();
added.add(table.getName());
}
}
assertThat(added).hasSize(7);
connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'before the columns')");
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true));
for(var table : SecuredDataStandardDatesTest.declared(instance))
{
for(String column : java.util.List.of("create_date", "modify_date"))
{
try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table.getName(), column))
{
assertThat(columns.next()).as(table.getName() + "." + column).isTrue();
assertThat(columns.getString("TYPE_NAME")).as(table.getName() + "." + column).isEqualTo("TIMESTAMP WITH TIME ZONE");
assertThat(columns.getString("IS_NULLABLE")).as(table.getName() + "." + column).isEqualTo("YES");
if(added.contains(table.getName()))
{
assertThat(columns.getString("COLUMN_DEF")).as(table.getName() + "." + column).isEqualTo("CURRENT_TIMESTAMP");
}
}
}
}
connection.createStatement().execute("INSERT INTO secured_data_legal_hold (table_name, reason) VALUES ('person', 'by a release that does not know the columns')");
var holds = SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false);
assertThat(holds).hasSize(2).allSatisfy(hold -> assertThat(hold.get("create_date")).isNotNull().isEqualTo(hold.get("modify_date")));
SecuredDataStandardDatesTest.run(connection, SecuredDataStandardDatesTest.ddl(true));
assertThat(SecuredDataStandardDatesTest.rows(connection, SecuredDataTables.LEGAL_HOLD, false)).isEqualTo(holds);
}
}
/*******************************************************************************
** Whether a table has a column.
*******************************************************************************/
private static boolean hasColumn(Connection connection, String table, String column) throws java.sql.SQLException
{
try(java.sql.ResultSet columns = connection.getMetaData().getColumns(null, null, table, column))
{
return (columns.next());
}
}
}
@@ -57,7 +57,7 @@ class SecuredDataHomeTest
var app = instance.getApp("secured_data");
assertThat(app.getWidgets()).containsExactly(SecuredDataApp.HOME_WIDGET);
assertThat(app.getChildren()).extracting(c -> c.getName()).containsExactly(SecuredDataTables.CLASSIFICATION, SecuredDataTables.POLICY,
SecuredDataTables.ACCESS_LOG, SecuredDataTables.BREAK_GLASS);
SecuredDataTables.ACCESS_LOG, SecuredDataTables.BREAK_GLASS, SecuredDataTables.RETENTION_RULE);
assertThat(instance.getTable(SecuredDataTables.ACCESS_LOG).getSections()).anyMatch(s -> SecuredDataTables.REVEALS_WIDGET.equals(s.getWidgetName()));
}
@@ -88,8 +88,8 @@ class SecuredDataHomeTest
delete(SecuredDataTables.POLICY, policy);
classification("email", "PII");
var logger = SecuredDataRuntime.current().logger();
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qbits.secureddata.standin.FieldAccessEventType.VIEW, null, null));
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qbits.secureddata.standin.FieldAccessEventType.VIEW, null, null));
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.VIEW, null, null));
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.VIEW, null, null));
var data = SecuredDataHomeData.load();
assertThat(data.getKpis()).containsEntry("Reveals today", 15L).containsEntry("Active reveal grants", 1L)
.containsEntry("Active break-glass sessions", 1L).containsEntry("Anomaly flags (7 days)", 1L)
@@ -0,0 +1,352 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TimeZone;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
import com.kingsrook.qqq.middleware.javalin.QJavalinImplementation;
import org.h2.util.DateTimeUtils;
import org.json.JSONObject;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** A person edits a row of the QBit over HTTP, a day after it was written. The
** request names the row by the id in its path, which arrives as text, while
** the stored id is a number: the edit must still be an edit of that row. A
** table with a modify date is edited the way a client that knows record
** versions edits it, with the changed values only and the version it read.
*******************************************************************************/
class SecuredDataHttpEditTest
{
private QApplicationJavalinServer server;
private int port;
private final TimeZone zone = TimeZone.getDefault();
/*******************************************************************************
** These tests run in UTC, whatever the zone of the machine. Core compares the
** version a client sends with the stored modify date in the database, and
** where the default zone of the JVM is not UTC that comparison never matches
** (a version is then refused with 409 for a row nobody changed). That is
** core's to fix. It is kept out of what is tested here, which is that an
** edit is an edit of its row. The test database keeps the zone it first saw,
** so it is told of each change.
*******************************************************************************/
@BeforeEach
void utc()
{
TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
DateTimeUtils.resetCalendar();
}
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
TimeZone.setDefault(zone);
DateTimeUtils.resetCalendar();
if(server != null)
{
server.stop();
}
QJavalinImplementation.setSessionCookieHttpOnly(false);
RetentionTestSupport.close();
}
/*******************************************************************************
** The session of every request: an administrator of the QBit.
*******************************************************************************/
public static class Sessions implements QAuthenticationModuleCustomizerInterface
{
/*******************************************************************************
** Customize session.
*******************************************************************************/
public void customizeSession(QInstance instance, QSession session, Map<String, Object> context)
{
session.setPermissions(Set.of("secured_data.admin", "secured_data.hasAccess", "secured_data.retention", "reading.read", "reading.write"));
}
}
/*******************************************************************************
** Serve the instance, a day after its rows were written.
*******************************************************************************/
private void serveADayLater(QInstance instance) throws Exception
{
instance.getAuthentication().setType(QAuthenticationType.MOCK);
instance.getAuthentication().setCustomizer(new QCodeReference(Sessions.class));
server = new QApplicationJavalinServer(new AbstractQQQApplication()
{
public QInstance defineQInstance()
{
return (instance);
}
}).withPort(0).withServeFrontendNext(false).withServeFrontendMaterialDashboard(false).withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true);
server.start();
var serviceField = QApplicationJavalinServer.class.getDeclaredField("service");
serviceField.setAccessible(true);
port = ((io.javalin.Javalin) serviceField.get(server)).port();
CLOCK.now = NOW.plusSeconds(86400);
}
/*******************************************************************************
** Request.
*******************************************************************************/
private HttpResponse<String> request(String method, String path, String body) throws Exception
{
var builder = HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/qqq/v1" + path)).header("Cookie", "sessionUUID=edit-user");
if(body != null)
{
builder.header("Content-Type", "application/json").method(method, HttpRequest.BodyPublishers.ofString(body));
}
return (HttpClient.newHttpClient().send(builder.build(), HttpResponse.BodyHandlers.ofString()));
}
/*******************************************************************************
** The version a client reads with a row: the value of its modify date.
*******************************************************************************/
private String version(String table, Serializable id, String modifyDate) throws Exception
{
HttpResponse<String> get = request("GET", "/table/" + table + "/" + id, null);
assertThat(get.statusCode()).as(get.body()).isEqualTo(200);
JSONObject body = new JSONObject(get.body());
JSONObject values = body.has("record") ? body.getJSONObject("record").getJSONObject("values") : body.getJSONObject("values");
return (values.isNull(modifyDate) ? null : String.valueOf(values.get(modifyDate)));
}
/*******************************************************************************
** Edit a row: the changed values, and the version that was read where the
** client sends one.
*******************************************************************************/
private HttpResponse<String> patch(String table, Serializable id, String version, String body) throws Exception
{
return (request("PATCH", "/table/" + table + "/" + id + (version == null ? "" : "?expectedVersion=" + URLEncoder.encode(version, StandardCharsets.UTF_8)), body));
}
/*******************************************************************************
** One row as the database holds it, column by column.
*******************************************************************************/
private static Map<String, String> stored(String table, Serializable id) throws Exception
{
try(Connection connection = DriverManager.getConnection(SecuredDataStandardDatesTest.url(), "sa", ""))
{
return (SecuredDataStandardDatesTest.rows(connection, table, false).stream().filter(row -> String.valueOf(id).equals(row.get("id"))).findFirst().orElseThrow());
}
}
/*******************************************************************************
** The columns an edit must leave as they were.
*******************************************************************************/
private static void assertKept(Map<String, String> before, Map<String, String> after, String... columns)
{
for(String column : columns)
{
assertThat(before.get(column)).as(column + " before").isNotNull();
assertThat(after.get(column)).as(column).isEqualTo(before.get(column));
}
}
/*******************************************************************************
** The reason of a hold is edited: only the reason is sent, with the version.
** The hold stays the hold it was, and the version it was edited with is then
** no longer its version.
*******************************************************************************/
@Test
void aPersonEditsTheReasonOfAHold() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("reason", "Preservation")).getValue("id");
serveADayLater(instance);
Map<String, String> placed = stored(SecuredDataTables.LEGAL_HOLD, id);
String version = version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate");
assertThat(version).isNotNull();
HttpResponse<String> edit = patch(SecuredDataTables.LEGAL_HOLD, id, version, "{\"reason\":\"Preservation, extended\"}");
assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200);
Map<String, String> edited = stored(SecuredDataTables.LEGAL_HOLD, id);
assertThat(edited.get("reason")).isEqualTo("Preservation, extended");
assertKept(placed, edited, "table_name", "status", "placed_at", "placed_by", "create_date");
assertThat(edited.get("released_at")).isNull();
assertThat(edited.get("modify_date")).isNotNull().isNotEqualTo(placed.get("modify_date"));
HttpResponse<String> stale = patch(SecuredDataTables.LEGAL_HOLD, id, version, "{\"reason\":\"Preservation, extended again\"}");
assertThat(stale.statusCode()).as(stale.body()).isEqualTo(409);
assertThat(stored(SecuredDataTables.LEGAL_HOLD, id)).isEqualTo(edited);
}
/*******************************************************************************
** A hold is released a day after it was placed: when and by whom it was
** placed, and its create date, are those of the day it was placed.
*******************************************************************************/
@Test
void aHoldReleasedADayLaterKeepsWhenAndByWhomItWasPlaced() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "7").withValue("reason", "Preservation")).getValue("id");
serveADayLater(instance);
Map<String, String> placed = stored(SecuredDataTables.LEGAL_HOLD, id);
HttpResponse<String> release = patch(SecuredDataTables.LEGAL_HOLD, id, version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate"), "{\"status\":\"RELEASED\"}");
assertThat(release.statusCode()).as(release.body()).isEqualTo(200);
Map<String, String> released = stored(SecuredDataTables.LEGAL_HOLD, id);
assertThat(released.get("status")).isEqualTo("RELEASED");
assertKept(placed, released, "table_name", "record_id", "reason", "placed_at", "placed_by", "create_date");
assertThat(released.get("released_at")).isNotNull().isNotEqualTo(placed.get("placed_at"));
assertThat(released.get("released_by")).isNotNull();
assertThat(released.get("modify_date")).isNotNull().isNotEqualTo(placed.get("modify_date"));
}
/*******************************************************************************
** A classification is edited by a client that sends its table and field back
** with the change: the row is not taken for a second classification of the
** same field, and it keeps its create date.
*******************************************************************************/
@Test
void aClassificationIsEditedNamingItsTableAndField() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
Serializable id = insert(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "reading").withValue("field_name", "notes").withValue("classification", "PII")).getValue("id");
serveADayLater(instance);
Map<String, String> written = stored(SecuredDataTables.CLASSIFICATION, id);
HttpResponse<String> edit = patch(SecuredDataTables.CLASSIFICATION, id, null, "{\"table_name\":\"reading\",\"field_name\":\"notes\",\"classification\":\"PHI\"}");
assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200);
Map<String, String> edited = stored(SecuredDataTables.CLASSIFICATION, id);
assertThat(edited.get("classification")).isEqualTo("PHI");
assertKept(written, edited, "table_name", "field_name", "create_date");
assertThat(edited.get("modify_date")).isNotNull().isNotEqualTo(written.get("modify_date"));
}
/*******************************************************************************
** A policy and a retention rule are renamed: each keeps its create date and
** everything else it had, and the rule is validated as the rule it is.
*******************************************************************************/
@Test
void aRenamedPolicyOrRetentionRuleKeepsItsCreateDate() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
Serializable policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE")).getValue("id");
Serializable rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret").getValue("id");
serveADayLater(instance);
Map<String, String> policyWritten = stored(SecuredDataTables.POLICY, policy);
Map<String, String> ruleWritten = stored(SecuredDataTables.RETENTION_RULE, rule);
HttpResponse<String> policyEdit = patch(SecuredDataTables.POLICY, policy, null, "{\"name\":\"renamed\"}");
assertThat(policyEdit.statusCode()).as(policyEdit.body()).isEqualTo(200);
Map<String, String> policyEdited = stored(SecuredDataTables.POLICY, policy);
assertThat(policyEdited.get("name")).isEqualTo("renamed");
assertKept(policyWritten, policyEdited, "permission_name", "classification", "tier", "create_date");
assertThat(policyEdited.get("modify_date")).isNotNull().isNotEqualTo(policyWritten.get("modify_date"));
HttpResponse<String> ruleEdit = patch(SecuredDataTables.RETENTION_RULE, rule, null, "{\"name\":\"renamed\"}");
assertThat(ruleEdit.statusCode()).as(ruleEdit.body()).isEqualTo(200);
Map<String, String> ruleEdited = stored(SecuredDataTables.RETENTION_RULE, rule);
assertThat(ruleEdited.get("name")).isEqualTo("renamed");
assertKept(ruleWritten, ruleEdited, "table_name", "field_names", "age_field_name", "action", "create_date");
assertThat(ruleEdited.get("modify_date")).isNotNull().isNotEqualTo(ruleWritten.get("modify_date"));
}
/*******************************************************************************
** A row that has neither date, as a row is that was written before the two
** columns existed and was not given its own time afterwards: it is read and
** edited like any other, and gets a modify date when it is changed.
*******************************************************************************/
@Test
void aRowWithoutDatesIsReadAndEdited() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
Serializable id = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("reason", "Preservation")).getValue("id");
try(Connection connection = DriverManager.getConnection(SecuredDataStandardDatesTest.url(), "sa", ""))
{
for(String table : List.of(SecuredDataTables.LEGAL_HOLD, SecuredDataTables.ACCESS_LOG, SecuredDataTables.BREAK_GLASS))
{
connection.createStatement().execute("UPDATE " + table + " SET create_date = NULL, modify_date = NULL");
}
}
serveADayLater(instance);
assertThat(version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate")).isNull();
HttpResponse<String> edit = patch(SecuredDataTables.LEGAL_HOLD, id, null, "{\"reason\":\"Preservation, extended\"}");
assertThat(edit.statusCode()).as(edit.body()).isEqualTo(200);
Map<String, String> edited = stored(SecuredDataTables.LEGAL_HOLD, id);
assertThat(edited.get("reason")).isEqualTo("Preservation, extended");
assertThat(edited.get("create_date")).isNull();
assertThat(edited.get("modify_date")).isNotNull();
assertThat(version(SecuredDataTables.LEGAL_HOLD, id, "modifyDate")).isNotNull();
}
}
@@ -109,11 +109,25 @@ class SecuredDataHttpTest
var status = request("/qbit/secured-data/status", null, false);
assertThat(status.statusCode()).as(status.body()).isEqualTo(200);
assertThat(new JSONObject(status.body()).getInt("revealsUsedToday")).isEqualTo(1);
assertThat(request("/qbit/secured-data/reveal", body, false).statusCode()).isEqualTo(429);
assertThat(request("/qbit/secured-data/reveal", body, true).statusCode()).isEqualTo(403);
assertThat(request("/qbit/secured-data/reveal", "{}", false).statusCode()).isEqualTo(400);
assertThat(request("/qbit/secured-data/reveal", "{\"primaryKey\":1}", false).statusCode()).isEqualTo(400);
assertThat(request("/qbit/secured-data/reveal/options?tableName=person&fieldName=missing", null, false).statusCode()).isEqualTo(404);
assertErrorResponse(request("/qbit/secured-data/reveal", body, false), 429);
assertErrorResponse(request("/qbit/secured-data/reveal", body, true), 403);
assertErrorResponse(request("/qbit/secured-data/reveal", "{}", false), 400);
assertErrorResponse(request("/qbit/secured-data/reveal", "{\"primaryKey\":1}", false), 400);
assertErrorResponse(request("/qbit/secured-data/reveal/options?tableName=person&fieldName=missing", null, false), 404);
}
/*******************************************************************************
** Verify standard error bodies without changing endpoint status semantics.
*******************************************************************************/
private void assertErrorResponse(HttpResponse<String> response, int status)
{
assertThat(response.statusCode()).as(response.body()).isEqualTo(status);
JSONObject body = new JSONObject(response.body());
assertThat(body.keySet()).contains("error", "code", "message", "traceId");
assertThat(body.getString("error")).isNotBlank();
assertThat(body.getString("code")).isNotBlank();
assertThat(body.getString("message")).isNotBlank();
assertThat(body.getString("traceId")).isNotBlank();
}
@Test
@@ -0,0 +1,271 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.reveal.RevealResult;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.fail;
/*******************************************************************************
** A reveal, and the start, end and expiry of a break-glass session, hold the
** runtime's monitor from their first check to their last write, so that two of
** them cannot both pass a check (the daily allowance, one active session).
** Every other thread asks this QBit for a classification all the time: core
** asks for every field it decides on and for every filter criterion of every
** query, including the queries made inside this QBit's own writes.
**
** When that question needed the same monitor, a thread that asked it while
** holding another of this QBit's locks could never be answered during a
** reveal that was about to need that lock: the access-log writer holds the
** anomaly counters' monitor while it inserts a row (the insert counts earlier
** rows and writes an audit, both queries with criteria), and a reveal inserts
** its own access-log row. Reveal: runtime, then counters. Writer: counters,
** then runtime. Both waited for ever, and so did every later request, since
** every request asks for a classification.
**
** The rule rows now have a lock of their own that is held only while they are
** read. These tests run on a relational database (H2 in PostgreSQL mode, with
** the audit tables), where an insert really writes its audit. Each parks one
** thread inside the operation, holding the runtime's monitor, and requires
** the access-log writer and a classification question on other threads to
** finish meanwhile; then the parked operation finishes too. Without the fix
** the writer does not finish and the test fails after WAIT_SECONDS.
*******************************************************************************/
@Timeout(180)
class SecuredDataLockOrderTest
{
private static final long WAIT_SECONDS = 20;
private final ExecutorService threads = Executors.newCachedThreadPool(work ->
{
Thread thread = new Thread(work, "lock-order-test");
thread.setDaemon(true);
return (thread);
});
private QInstance instance;
private SecuredDataRuntime runtime;
private QSession person;
private QTableMetaData reading;
private Serializable pk;
private final CountDownLatch parked = new CountDownLatch(1);
private final CountDownLatch resume = new CountDownLatch(1);
private final AtomicReference<Thread> working = new AtomicReference<>();
private final AtomicBoolean heldTheRuntimesMonitor = new AtomicBoolean();
/*******************************************************************************
** Let a parked thread go, then close; a runtime whose threads are stuck was
** already taken out of the list (see abandon).
*******************************************************************************/
@AfterEach
void cleanup()
{
resume.countDown();
threads.shutdownNow();
RetentionTestSupport.close();
}
/*******************************************************************************
** A reveal in progress holds up neither the access-log writer nor a question.
*******************************************************************************/
@Test
void theAccessLogWriterAndAClassificationQuestionFinishWhileARevealIsInProgress() throws Exception
{
setup();
Future<RevealResult> reveal = parkInside(() -> new RevealService().reveal("reading", pk, "notes", null, "Synthetic reveal; lock order test."));
othersFinishMeanwhile();
resume.countDown();
assertThat(reveal.get(WAIT_SECONDS, TimeUnit.SECONDS).value()).isEqualTo("test-notes");
assertThat(logs()).extracting(row -> row.getValueString("event_type")).contains("EXACT_MATCH_FILTER", "REVEAL");
assertThat(runtime.resolver().isRevealActive("reading", "notes", pk)).isTrue();
}
/*******************************************************************************
** The same for the start of a break-glass session, which holds the same monitor.
*******************************************************************************/
@Test
void theAccessLogWriterAndAClassificationQuestionFinishWhileBreakGlassStarts() throws Exception
{
setup();
person.getPermissions().add(BreakGlassService.START + ".hasAccess");
Future<QRecord> started = parkInside(() -> new BreakGlassService().start("Synthetic break-glass session; lock order test.", 5, "reading", DataClassification.PHI));
othersFinishMeanwhile();
resume.countDown();
assertThat(started.get(WAIT_SECONDS, TimeUnit.SECONDS).getValueString("status")).isEqualTo("ACTIVE");
assertThat(logs()).extracting(row -> row.getValueString("event_type")).contains("EXACT_MATCH_FILTER", "BREAK_GLASS_START");
}
/*******************************************************************************
** A relational instance with audit tables; a PII field one policy makes
** revealable to the person; one row; and one search of that field waiting in
** the access-log queue, as it waits after a person filters a list by it.
*******************************************************************************/
private void setup() throws Exception
{
instance = RetentionTestSupport.setup();
runtime = SecuredDataRuntime.current();
person = QContext.getQSession();
reading = instance.getTable("reading");
insert(SecuredDataTables.POLICY, policyRow("secured_data.admin", "PII", "REVEALABLE"));
pk = RetentionTestSupport.reading(120).getValue("id");
runtime.resolver().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH)
.withTableName("reading").withFieldNames(List.of("notes")).withActionName("query").withRecordCount(1)
.withSearchValueFingerprints(List.of(runtime.resolver().fingerprintSearchValue("known"))));
assertThat(logs()).as("the search is queued, not yet written").extracting(row -> row.getValueString("event_type")).doesNotContain("EXACT_MATCH_FILTER");
}
/*******************************************************************************
** Run the operation as the person on another thread and return once it is
** parked at its first reading of the clock, which both operations make
** inside their synchronized block.
*******************************************************************************/
private <T> Future<T> parkInside(Callable<T> operation) throws Exception
{
CLOCK.beforeInstant = () ->
{
if(Thread.currentThread() == working.get() && parked.getCount() == 1)
{
heldTheRuntimesMonitor.set(Thread.holdsLock(runtime));
parked.countDown();
try
{
resume.await();
}
catch(InterruptedException e)
{
Thread.currentThread().interrupt();
}
}
};
Future<T> future = threads.submit(() ->
{
QContext.init(instance, person);
try
{
working.set(Thread.currentThread());
return (operation.call());
}
finally
{
QContext.clear();
}
});
assertThat(parked.await(WAIT_SECONDS, TimeUnit.SECONDS)).as("the operation read the clock").isTrue();
assertThat(heldTheRuntimesMonitor).as("the operation is parked holding the runtime's monitor").isTrue();
return (future);
}
/*******************************************************************************
** While the operation is parked: the access-log writer drains its queue (what
** its worker does on every tick), and a thread that holds no lock is told
** the classification of a field and the active retention rules.
*******************************************************************************/
private void othersFinishMeanwhile() throws Exception
{
Future<?> writer = threads.submit(() -> runtime.logger().flush());
Future<DataClassification> question = threads.submit(() ->
{
QContext.init(instance, person);
try
{
runtime.activeRetentionRules();
return (SecuredDataClassifications.effective(reading, "notes"));
}
finally
{
QContext.clear();
}
});
try
{
writer.get(WAIT_SECONDS, TimeUnit.SECONDS);
assertThat(question.get(WAIT_SECONDS, TimeUnit.SECONDS)).isEqualTo(DataClassification.PII);
}
catch(TimeoutException e)
{
abandon();
fail("A thread waited " + WAIT_SECONDS + " seconds for the monitor the parked operation holds (writer done: " + writer.isDone() + ", question done: " + question.isDone()
+ "): the operation needs the lock that thread holds next, so neither would ever finish");
}
}
/*******************************************************************************
** Threads of this runtime are stuck holding its locks: closing it would wait
** for them, so it is left to the daemon threads instead of being closed.
*******************************************************************************/
private void abandon()
{
SecuredDataTestSupport.RUNTIMES.remove(runtime);
}
}
@@ -26,11 +26,9 @@ import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.FieldsOfTablePossibleValueSource;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataEnums;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.ClientChannelsMetaData;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.standin.TableFeaturePermissionsMetaData;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
@@ -40,11 +38,12 @@ import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSo
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
@@ -89,23 +88,25 @@ class SecuredDataQBitProducerTest
{
QInstance instance = setup();
assertThat(instance.getTables().keySet()).containsAll(SecuredDataTables.ALL);
assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getVersion()).isEqualTo("1.0.0");
assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getVersion()).isEqualTo("0.1.0");
for(String name : SecuredDataTables.ALL)
{
QTableMetaData table = instance.getTable(name);
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(AuditLevel.FIELD);
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(name.equals(SecuredDataTables.AUDIT_KEY) ? AuditLevel.NONE : AuditLevel.FIELD);
assertThat(table.getSections()).isNotEmpty();
assertThat(CoreMetadataBridge.classification(table, null)).isEqualTo(DataClassification.CONFIDENTIAL);
table.getFields().values().forEach(field -> assertThat(field.getBackendName()).isEqualTo(field.getName()));
assertThat(SecuredDataMetadata.classification(table, null)).isEqualTo(DataClassification.CONFIDENTIAL);
// A column has the name of its field. The two standard dates are always the columns create_date and modify_date.
table.getFields().values().forEach(field -> assertThat(field.getBackendName())
.isEqualTo("createDate".equals(field.getName()) ? "create_date" : "modifyDate".equals(field.getName()) ? "modify_date" : field.getName()));
}
assertThat(SecuredDataTables.esbTableNames()).containsExactlyElementsOf(SecuredDataTables.MAIN);
assertThat(SecuredDataEnums.sources()).hasSize(6);
assertThat(ClientChannelsMetaData.ofOrWithNew(instance).getEntries()).containsEntry("assistant", "secured_data_*");
assertThat(TableFeaturePermissionsMetaData.ofOrWithNew(instance).getEntries()).containsKeys("secured_data_full", "secured_data_reveal");
assertThat(ClientChannelsMetaData.ofOrWithNew(instance).getDeniedPatterns().get("assistant")).contains("secured_data_*");
assertThat(TableFeaturePermissionsMetaData.ofOrWithNew(instance).getPermissions()).extracting("suffix").contains("secured_data_full", "secured_data_reveal");
assertThat(instance.getTable(SecuredDataTables.ACCESS_LOG).getDisabledCapabilities()).contains(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
assertThat(instance.getJoin(SecuredDataTables.REVEALS_JOIN)).isNotNull();
assertThat(instance.getWidget(SecuredDataTables.REVEALS_WIDGET)).isNotNull();
assertThat(instance.getApp("secured_data").getChildren()).hasSize(4);
assertThat(instance.getApp("secured_data").getChildren()).hasSize(5);
}
/*******************************************************************************
@@ -23,23 +23,23 @@ package com.kingsrook.qbits.secureddata;
import java.util.List;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.AiTableMetaData;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.standin.FieldAccessTier;
import com.kingsrook.qbits.secureddata.standin.FieldMaskStyle;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
@@ -271,7 +271,7 @@ class SecuredDataResolverTest
{
QFieldMetaData field = new QFieldMetaData(name, QFieldType.valueOf(type)).withMaxLength(255);
person.addField(field);
CoreMetadataBridge.classifyField(field, DataClassification.PII);
SecuredDataMetadata.classifyField(field, DataClassification.PII);
assertThat(resolver.resolve(person, field).exactMatchAllowed()).isFalse();
QRecord row = insert(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "person").withValue("field_name", name)
.withValue("classification", "PII").withValue("low_cardinality", false).withValue("mask_style", "FULL"));
@@ -327,4 +327,22 @@ class SecuredDataResolverTest
CLOCK.now = NOW.plusSeconds(10);
assertThat(resolver.isRevealActive("person", "email", 7)).isFalse();
}
/*******************************************************************************
** A virtual (computed) field is not in getFields, and getField throws for
** it: it resolves through the table classification instead of failing the
** whole read.
*******************************************************************************/
@Test
void virtualFieldsTakeTheTableClassification() throws Exception
{
QVirtualFieldMetaData display = new QVirtualFieldMetaData("displayName", QFieldType.STRING);
person.withVirtualFields(List.of(display));
assertThat(SecuredDataClassifications.effective(person, "displayName")).isNull();
assertThat(resolver.resolve(person, display).tier()).isEqualTo(FieldAccessTier.FULL);
person.withSupplementalMetaData(new AiTableMetaData().withClassification(DataClassification.PII));
assertThat(SecuredDataClassifications.effective(person, "displayName")).isEqualTo(DataClassification.PII);
assertThat(resolver.resolve(person, display).tier()).isEqualTo(FieldAccessTier.MASKED);
}
}
@@ -0,0 +1,261 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.cluster.ClusterBus;
import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
import com.kingsrook.qqq.backend.core.cluster.ClusterEvent;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.delete;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** The rule rows a node keeps (retention rules, policies, classifications) and
** the access fingerprint of a request: read once, cleared by every change on
** every node, never older than the maximum age, and never an empty answer when
** the rules cannot be read.
*******************************************************************************/
class SecuredDataRuleCachesTest
{
private static final List<String> ANNOUNCED = new ArrayList<>();
static
{
ClusterBus.on(ClusterEvent.TYPE_MEMO_INVALIDATE, (instance, event) -> ANNOUNCED.add(event.getName()));
}
QInstance instance;
QTableMetaData person;
SecuredDataRuntime runtime;
/*******************************************************************************
** A person table with a date, so a retention rule on it is valid.
*******************************************************************************/
@BeforeEach
void init() throws Exception
{
instance = setup();
person = instance.getTable("person");
person.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("createDate"));
runtime = SecuredDataRuntime.current();
MemoryRecordStore.setCollectStatistics(true);
MemoryRecordStore.resetStatistics();
ANNOUNCED.clear();
}
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
MemoryRecordStore.setCollectStatistics(false);
close();
}
/*******************************************************************************
** An active rule on one field of the person table.
*******************************************************************************/
private static QRecord rule(String field) throws Exception
{
return (insert(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30 days: " + field).withValue("table_name", "person")
.withValue("field_names", field).withValue("age_field_name", "createDate").withValue("age_days", 30).withValue("action", "NULL_WITH_MARKER")));
}
/*******************************************************************************
** Queries the in-memory store has run since the last reset.
*******************************************************************************/
private static int queries()
{
return (MemoryRecordStore.getStatistics().getOrDefault(MemoryRecordStore.STAT_QUERIES_RAN, 0));
}
/*******************************************************************************
** Core asks for the rules of a field on every audited field and filter
** criterion: many questions are one read, and every committed change (insert,
** update, delete) is seen by the next question.
*******************************************************************************/
@Test
void activeRetentionRulesAreReadOncePerChange() throws Exception
{
QRecord notes = rule("notes");
assertThat(notes.getErrors()).isNullOrEmpty();
MemoryRecordStore.resetStatistics();
for(int i = 0; i < 50; i++)
{
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(RetentionRules.active("person", "email")).isEmpty();
assertThat(RetentionRules.active("anotherTable", "notes")).isEmpty();
assertThat(RetentionRules.changedOnly("person", "notes")).isFalse();
}
assertThat(queries()).isLessThanOrEqualTo(1);
QRecord ssn = rule("ssn");
assertThat(RetentionRules.active("person", "ssn")).hasSize(1);
update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", notes.getValue("id")).withValue("is_active", false));
assertThat(RetentionRules.active("person", "notes")).isEmpty();
delete(SecuredDataTables.RETENTION_RULE, ssn);
assertThat(RetentionRules.active("person", "ssn")).isEmpty();
assertThat(ANNOUNCED).contains(SecuredDataRuntime.RETENTION_CACHE);
}
/*******************************************************************************
** A rule change made on another node arrives as a cluster cache event: each
** of the three kinds of rules is read again here. A change made here is
** announced to the other nodes.
*******************************************************************************/
@Test
void aChangeOnAnyNodeClearsTheRulesOnEveryNode() throws Exception
{
rule("notes");
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
long retention = runtime.retentionVersion();
long policies = runtime.policyVersion();
long classifications = runtime.classificationVersion();
ClusterCaches.invalidate(SecuredDataRuntime.RETENTION_CACHE);
ClusterCaches.invalidate(SecuredDataRuntime.POLICY_CACHE);
ClusterCaches.invalidate(SecuredDataRuntime.CLASSIFICATION_CACHE);
assertThat(runtime.retentionVersion()).isGreaterThan(retention);
assertThat(runtime.policyVersion()).isGreaterThan(policies);
assertThat(runtime.classificationVersion()).isGreaterThan(classifications);
MemoryRecordStore.resetStatistics();
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(queries()).isGreaterThanOrEqualTo(1);
MemoryRecordStore.resetStatistics();
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(queries()).isZero();
ANNOUNCED.clear();
runtime.invalidatePolicies();
runtime.invalidateClassifications();
runtime.invalidateRetentionRules();
assertThat(ANNOUNCED).contains(SecuredDataRuntime.POLICY_CACHE, SecuredDataRuntime.CLASSIFICATION_CACHE, SecuredDataRuntime.RETENTION_CACHE);
}
/*******************************************************************************
** Fail closed: when the rules cannot be read, asking for them throws. The
** rules read before are not used, and the answer is never an empty list.
*******************************************************************************/
@Test
void rulesThatCannotBeReadAreNeverAnEmptyAnswer() throws Exception
{
rule("notes");
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
QTableMetaData ruleTable = instance.getTables().remove(SecuredDataTables.RETENTION_RULE);
try
{
runtime.invalidateRetentionRules();
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(RuntimeException.class);
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(RuntimeException.class);
}
finally
{
instance.getTables().put(SecuredDataTables.RETENTION_RULE, ruleTable);
}
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
}
/*******************************************************************************
** The safety net for a lost cluster event: rows are read again once they are
** older than the maximum age.
*******************************************************************************/
@Test
void ruleRowsAreNotUsedPastTheirMaximumAge()
{
assertThat(SecuredDataRuntime.tooOld(System.nanoTime())).isFalse();
assertThat(SecuredDataRuntime.tooOld(System.nanoTime() - SecuredDataRuntime.MAX_RULE_AGE.plusSeconds(1).toNanos())).isTrue();
assertThat(SecuredDataRuntime.MAX_RULE_AGE.toSeconds()).isLessThanOrEqualTo(60);
}
/*******************************************************************************
** Core asks for the access fingerprint before every field decision. Within a
** request the same value object is returned; a permission that matters, given
** to the same session object, changes it at once.
*******************************************************************************/
@Test
void theAccessFingerprintIsKeptForTheRequestAndTracksThePermissions()
{
SecuredDataResolver resolver = runtime.resolver();
QSession session = QContext.getQSession();
String first = resolver.getAccessFingerprint();
for(int i = 0; i < 20; i++)
{
assertThat(resolver.getAccessFingerprint()).isSameAs(first);
}
session.getPermissions().add("unrelated.permission");
assertThat(resolver.getAccessFingerprint()).isEqualTo(first);
session.getPermissions().add(SecuredDataMetadata.permissionBase(person) + "." + SecuredDataPermissions.FULL);
String granted = resolver.getAccessFingerprint();
assertThat(granted).isNotEqualTo(first);
session.setPermissions(new HashSet<>(List.of("person.read")));
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(granted);
}
}
@@ -0,0 +1,682 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessDecision;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Reading rule rows (classifications, retention rules, policies) is a query on
** a secured-data table, and other code runs inside that query: a virtual field
** value provider or a post-query customizer that another QBit or the host put
** on the table. When that code runs a query with filter criteria, core asks
** this QBit for a classification - which needs the rows being read.
**
** These tests put such code on the rule tables and prove: the read asks for
** stored fields only, so no value provider runs inside it; a question asked
** inside a read is answered from the rows held before and starts no other
** read; only the reading thread, as the system, can be given that answer; and
** a read that fails is tried again by the next question.
*******************************************************************************/
class SecuredDataRuleReloadTest
{
QInstance instance;
QTableMetaData person;
SecuredDataRuntime runtime;
/*******************************************************************************
** Something to do inside a query on a rule table.
*******************************************************************************/
@FunctionalInterface
interface NestedWork
{
/*******************************************************************************
** Do it.
*******************************************************************************/
void run() throws Exception;
}
/*******************************************************************************
** Stands in for another QBit's virtual field on a secured-data table: filling
** it runs a query that has filter criteria.
*******************************************************************************/
public static class NestedQueryProvider implements VirtualFieldValueProviderInterface
{
static final AtomicInteger RUNS = new AtomicInteger();
/*******************************************************************************
** Fill the field from a query with criteria, as a real provider does.
*******************************************************************************/
@Override
public void fill(QTableMetaData table, String fieldName, List<QRecord> records, QueryInput input) throws QException
{
RUNS.incrementAndGet();
queryWithCriteria();
records.forEach(record -> record.setValue(fieldName, "filled"));
}
}
/*******************************************************************************
** A post-query customizer on a secured-data table. Core runs it for every
** query on the table, whatever fields the query names.
*******************************************************************************/
public static class NestedWorkCustomizer implements TableCustomizerInterface
{
static final AtomicInteger RUNS = new AtomicInteger();
static volatile NestedWork work;
/*******************************************************************************
** Run the work the test asked for, inside the query.
*******************************************************************************/
@Override
public List<QRecord> postQuery(QueryOrGetInputInterface queryInput, List<QRecord> records) throws QException
{
NestedWork toRun = work;
if(toRun != null)
{
RUNS.incrementAndGet();
try
{
toRun.run();
}
catch(QException | RuntimeException e)
{
throw (e);
}
catch(Exception e)
{
throw (new QException("Nested work failed", e));
}
}
return (records);
}
}
/*******************************************************************************
** A person table with a date, so a retention rule on it is valid.
*******************************************************************************/
@BeforeEach
void init() throws Exception
{
instance = setup();
person = instance.getTable("person");
person.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBackendName("createDate"));
runtime = SecuredDataRuntime.current();
NestedQueryProvider.RUNS.set(0);
NestedWorkCustomizer.RUNS.set(0);
NestedWorkCustomizer.work = null;
}
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
NestedWorkCustomizer.work = null;
close();
}
/*******************************************************************************
** A query with filter criteria on a host table: core validates the operand
** of a classified field, so it asks for the field's classification - and
** this QBit then asks whether a retention rule names the field.
*******************************************************************************/
private static void queryWithCriteria() throws QException
{
new QueryAction().execute(new QueryInput("person").withFilter(new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, 1))));
}
/*******************************************************************************
** Another QBit's virtual field on a secured-data table.
*******************************************************************************/
private void addProvider(String table)
{
instance.getTable(table).addVirtualField(new QVirtualFieldMetaData("labels", QFieldType.STRING).withIsQuerySelectable(true)
.withValueProvider(new QCodeReference(NestedQueryProvider.class)));
}
/*******************************************************************************
** A post-query customizer on a secured-data table.
*******************************************************************************/
private void addCustomizer(String table)
{
instance.getTable(table).withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(NestedWorkCustomizer.class));
}
/*******************************************************************************
** A row as another node writes it: straight into the store. None of this
** node's customizers run and nothing here reads the rule rows; this node
** learns of the change when its version moves (the cluster event).
*******************************************************************************/
private static QRecord writtenElsewhere(String table, QRecord row) throws QException
{
return (MemoryRecordStore.getInstance().insert(new InsertInput(table).withRecord(row), true).getFirst());
}
/*******************************************************************************
** A classification of a person field, written by another node.
*******************************************************************************/
private QRecord classifiedElsewhere(String field, String classification) throws QException
{
QRecord row = writtenElsewhere(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "person")
.withValue("field_name", field).withValue("classification", classification));
runtime.invalidateClassifications();
return (row);
}
/*******************************************************************************
** A classification row changed by another node.
*******************************************************************************/
private void reclassifiedElsewhere(QRecord row, String classification) throws QException
{
MemoryRecordStore.getInstance().update(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord()
.withValue("id", row.getValue("id")).withValue("classification", classification)), false);
runtime.invalidateClassifications();
}
/*******************************************************************************
** An active retention rule on one field of the person table, written by
** another node.
*******************************************************************************/
private void ruleWrittenElsewhere(String field) throws QException
{
writtenElsewhere(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30 days: " + field).withValue("table_name", "person")
.withValue("field_names", field).withValue("age_field_name", "createDate").withValue("age_days", 30).withValue("action", "NULL_WITH_MARKER")
.withValue("audit_treatment", "SHREDDABLE").withValue("is_active", true));
runtime.invalidateRetentionRules();
}
/*******************************************************************************
** The classification of a person field.
*******************************************************************************/
private DataClassification classificationOf(String field)
{
return (SecuredDataClassifications.effective(person, field));
}
/*******************************************************************************
** The reported failure: a value provider on the classification table whose
** work is a query with criteria, and one classification row. Before the fix
** the next classification question recursed until the stack overflowed, and
** so did every question after it. Now the rule read names its stored fields,
** so the provider does not run inside it at all.
*******************************************************************************/
@Test
void aValueProviderOnTheClassificationTableDoesNotRunInsideTheRuleRead() throws Exception
{
addProvider(SecuredDataTables.CLASSIFICATION);
classification("notes", "PHI");
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
queryWithCriteria();
assertThat(NestedQueryProvider.RUNS).hasValue(0);
assertThat(SystemWrites.query(SecuredDataTables.CLASSIFICATION, new QQueryFilter())).singleElement()
.satisfies(row -> assertThat(row.getValues()).doesNotContainKey("labels").containsEntry("field_name", "notes"));
}
/*******************************************************************************
** The virtual field is still there for a query that asks for every field (a
** screen listing the classifications). Its provider's query with criteria
** asks for a classification while the rows are out of date: the rows are
** read once, inside the provider, and nothing recurses.
*******************************************************************************/
@Test
void anOrdinaryQueryOnTheClassificationTableStillFillsTheVirtualField() throws Exception
{
addProvider(SecuredDataTables.CLASSIFICATION);
classifiedElsewhere("notes", "PHI");
List<QRecord> rows = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.CLASSIFICATION)).getRecords());
assertThat(rows).singleElement().satisfies(row -> assertThat(row.getValueString("labels")).isEqualTo("filled"));
assertThat(NestedQueryProvider.RUNS).hasValue(1);
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
}
/*******************************************************************************
** A post-query customizer runs inside the rule read whatever fields the read
** names. Its query with criteria asks for a classification in the middle of
** the read of the classifications: before the fix that started the read again
** and recursed until the stack overflowed. Now the rows are read once.
*******************************************************************************/
@Test
void aQuestionAskedInsideTheClassificationReadDoesNotStartAnotherRead() throws Exception
{
addCustomizer(SecuredDataTables.CLASSIFICATION);
classifiedElsewhere("notes", "PHI");
NestedWorkCustomizer.work = SecuredDataRuleReloadTest::queryWithCriteria;
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
assertThat(classificationOf("email")).isEqualTo(DataClassification.PII);
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
}
/*******************************************************************************
** What a question asked inside the read is told, and who can ask it.
**
** The nested question gets the rows this node held when the read began (no
** rows, the first time; then the rows of the previous version), in both
** directions of a change. It can only be asked by the thread doing the read,
** which is inside the QBit's system read: the session is the system session
** and the resolver answers FULL there without looking at any classification,
** so no person's field access is decided from that answer. The question that
** started the read gets the new rows.
*******************************************************************************/
@Test
void aQuestionAskedInsideTheReadIsAnsweredFromTheRowsHeldBefore() throws Exception
{
addCustomizer(SecuredDataTables.CLASSIFICATION);
assertThat(classificationOf("notes")).isNull();
List<DataClassification> nestedAnswers = new ArrayList<>();
List<Boolean> nestedAsSystem = new ArrayList<>();
List<FieldAccessDecision> nestedDecisions = new ArrayList<>();
NestedWork ask = () ->
{
nestedAnswers.add(classificationOf("notes"));
nestedAsSystem.add(QContext.getQSession() instanceof QSystemUserSession && SystemWrites.active());
nestedDecisions.add(runtime.resolver().resolve(person, person.getField("notes")));
};
/////////////////////////////////////////////////////////////////////////
// a row raises the field: inside the read it is still unclassified, //
// and the question that started the read is answered PHI //
/////////////////////////////////////////////////////////////////////////
QRecord row = classifiedElsewhere("notes", "PHI");
NestedWorkCustomizer.work = ask;
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
NestedWorkCustomizer.work = null;
assertThat(nestedAnswers).containsExactly((DataClassification) null);
/////////////////////////////////////////////////////////////////////////
// the row is changed to lower the field: inside the read it is still //
// PHI, and the question that started the read is answered PUBLIC //
/////////////////////////////////////////////////////////////////////////
reclassifiedElsewhere(row, "PUBLIC");
NestedWorkCustomizer.work = ask;
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
NestedWorkCustomizer.work = null;
assertThat(nestedAnswers).containsExactly(null, DataClassification.PHI);
assertThat(nestedAsSystem).containsExactly(true, true);
assertThat(nestedDecisions).containsExactly(FieldAccessDecision.FULL, FieldAccessDecision.FULL);
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
}
/*******************************************************************************
** A request on another thread is never given the rows held before: while the
** read is running it waits, and then it is answered from the rows the read
** returned.
*******************************************************************************/
@Test
void aRequestOnAnotherThreadWaitsForTheReadAndIsGivenItsRows() throws Exception
{
addCustomizer(SecuredDataTables.CLASSIFICATION);
QRecord row = classifiedElsewhere("notes", "PUBLIC");
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
reclassifiedElsewhere(row, "PHI");
CountDownLatch readIsRunning = new CountDownLatch(1);
CountDownLatch letReadFinish = new CountDownLatch(1);
NestedWorkCustomizer.work = () ->
{
readIsRunning.countDown();
assertThat(letReadFinish.await(30, TimeUnit.SECONDS)).isTrue();
};
ExecutorService pool = Executors.newFixedThreadPool(2);
try
{
Future<DataClassification> reader = pool.submit(asUser("u2", () -> classificationOf("notes")));
assertThat(readIsRunning.await(30, TimeUnit.SECONDS)).isTrue();
Future<DataClassification> other = pool.submit(asUser("u3", () -> classificationOf("notes")));
assertThatThrownBy(() -> other.get(500, TimeUnit.MILLISECONDS)).isInstanceOf(TimeoutException.class);
letReadFinish.countDown();
assertThat(reader.get(30, TimeUnit.SECONDS)).isEqualTo(DataClassification.PHI);
assertThat(other.get(30, TimeUnit.SECONDS)).isEqualTo(DataClassification.PHI);
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
}
finally
{
letReadFinish.countDown();
pool.shutdownNow();
}
}
/*******************************************************************************
** A person's request on its own thread.
*******************************************************************************/
private <T> Callable<T> asUser(String id, Callable<T> work)
{
return (() ->
{
QContext.init(instance, user(id));
try
{
return (work.call());
}
finally
{
QContext.clear();
}
});
}
/*******************************************************************************
** A read that fails (here, in code running inside it, after that code asked a
** nested question) throws to the request that started it and leaves nothing
** behind: the next request reads again and gets the current rows - not the
** rows held before, and not another failure. An Error, such as the stack
** overflow this defect used to end in, is no different.
*******************************************************************************/
@Test
void aClassificationReadThatFailsIsTriedAgainByTheNextQuestion() throws Exception
{
addCustomizer(SecuredDataTables.CLASSIFICATION);
QRecord row = classifiedElsewhere("notes", "PUBLIC");
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PUBLIC);
reclassifiedElsewhere(row, "PHI");
NestedWorkCustomizer.work = () ->
{
classificationOf("notes");
throw (new QException("The store is unavailable"));
};
assertThatThrownBy(() -> classificationOf("notes")).isInstanceOf(IllegalStateException.class);
NestedWorkCustomizer.work = () ->
{
classificationOf("notes");
throw (new StackOverflowError());
};
assertThatThrownBy(() -> classificationOf("notes")).isInstanceOf(StackOverflowError.class);
NestedWorkCustomizer.work = null;
assertThat(classificationOf("notes")).isEqualTo(DataClassification.PHI);
}
/*******************************************************************************
** The active retention rules have the same shape: core's classification
** question for a host field also asks whether a retention rule names the
** field. A value provider on the retention rule table, with one active rule,
** recursed the same way before the fix; it does not run inside the rule read.
*******************************************************************************/
@Test
void aValueProviderOnTheRetentionRuleTableDoesNotRunInsideTheRuleRead() throws Exception
{
addProvider(SecuredDataTables.RETENTION_RULE);
ruleWrittenElsewhere("notes");
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
queryWithCriteria();
assertThat(NestedQueryProvider.RUNS).hasValue(0);
}
/*******************************************************************************
** A question asked inside the read of the retention rules starts no other
** read and is answered from the rules held before; the question that started
** the read gets the new rules. A read that fails is tried again.
*******************************************************************************/
@Test
void aQuestionAskedInsideTheRetentionRuleReadIsAnsweredFromTheRulesHeldBefore() throws Exception
{
addCustomizer(SecuredDataTables.RETENTION_RULE);
assertThat(RetentionRules.active("person", "notes")).isEmpty();
ruleWrittenElsewhere("notes");
List<Integer> nestedAnswers = new ArrayList<>();
NestedWorkCustomizer.work = () ->
{
queryWithCriteria();
nestedAnswers.add(RetentionRules.active("person", "notes").size());
if(nestedAnswers.size() == 1)
{
throw (new QException("The store is unavailable"));
}
};
assertThatThrownBy(() -> RetentionRules.active("person", "notes")).isInstanceOf(IllegalStateException.class);
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(nestedAnswers).containsExactly(0, 0);
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(NestedWorkCustomizer.RUNS).hasValue(2);
}
/*******************************************************************************
** A transaction that edited a rule reads the rules through itself, not from
** the rules the node keeps. A question asked inside that read is answered
** from the rules every thread shares: before the fix it read the edits again,
** and recursed until the stack overflowed.
*******************************************************************************/
@Test
void aQuestionAskedInsideAReadOfATransactionsOwnEditsDoesNotReadThemAgain() throws Exception
{
addCustomizer(SecuredDataTables.RETENTION_RULE);
ruleWrittenElsewhere("notes");
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
QContext.setTransaction(new QBackendTransaction());
RetentionRules.editedInTransaction();
try
{
NestedWorkCustomizer.work = SecuredDataRuleReloadTest::queryWithCriteria;
assertThat(RetentionRules.active("person", "notes")).hasSize(1);
assertThat(NestedWorkCustomizer.RUNS).hasValue(1);
}
finally
{
////////////////////////////////////////////////////////////////////////
// the next question outside a transaction ends this thread's edits //
////////////////////////////////////////////////////////////////////////
NestedWorkCustomizer.work = null;
QContext.setTransaction(null);
RetentionRules.active("person", "notes");
}
}
/*******************************************************************************
** The policies are read the same way (rows first, version afterwards), so
** they are guarded the same way: code inside the policy read that asks the
** resolver a policy question starts no other read, and a read that fails is
** tried again.
*******************************************************************************/
@Test
void aQuestionAskedInsideThePolicyReadDoesNotStartAnotherRead() throws Exception
{
addCustomizer(SecuredDataTables.POLICY);
addProvider(SecuredDataTables.POLICY);
assertThat(runtime.resolver().revealPolicies(person, "email")).isEmpty();
writtenElsewhere(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE"));
runtime.invalidatePolicies();
List<Integer> nestedAnswers = new ArrayList<>();
NestedWorkCustomizer.work = () ->
{
nestedAnswers.add(runtime.resolver().revealPolicies(person, "email").size());
if(nestedAnswers.size() == 1)
{
throw (new QException("The store is unavailable"));
}
};
assertThatThrownBy(() -> runtime.resolver().revealPolicies(person, "email")).isInstanceOf(IllegalStateException.class);
assertThat(runtime.resolver().revealPolicies(person, "email")).hasSize(1);
assertThat(nestedAnswers).containsExactly(0, 0);
assertThat(NestedQueryProvider.RUNS).hasValue(0);
NestedWorkCustomizer.RUNS.set(0);
assertThat(runtime.resolver().revealPolicies(person, "email")).hasSize(1);
assertThat(NestedWorkCustomizer.RUNS).hasValue(0);
}
/*******************************************************************************
** The rows themselves, without a table: nothing is held before the first
** read, a nested question never reads, a failed read is forgotten, and a new
** version is read once.
*******************************************************************************/
@Test
void ruleRowsAreReadOncePerVersionAndNeverFromInsideTheirOwnRead()
{
RuleRows rows = new RuleRows();
AtomicInteger reads = new AtomicInteger();
List<QRecord> first = List.of(new QRecord().withValue("id", 1));
List<QRecord> second = List.of(new QRecord().withValue("id", 2));
assertThatThrownBy(() -> rows.get(1, () ->
{
reads.incrementAndGet();
assertThat(rows.get(1, () -> fail("a nested question must not read"))).isEmpty();
throw (new IllegalStateException("The store is unavailable"));
})).isInstanceOf(IllegalStateException.class);
assertThat(rows.get(1, () ->
{
reads.incrementAndGet();
assertThat(rows.get(1, () -> fail("a nested question must not read"))).isEmpty();
return (first);
})).isEqualTo(first);
assertThat(rows.get(1, () -> fail("rows of this version are held"))).isEqualTo(first);
assertThat(rows.get(2, () ->
{
reads.incrementAndGet();
assertThat(rows.get(2, () -> fail("a nested question must not read"))).isEqualTo(first);
return (second);
})).isEqualTo(second);
assertThat(reads).hasValue(3);
}
/*******************************************************************************
** A read that must not happen.
*******************************************************************************/
private static List<QRecord> fail(String why)
{
throw (new AssertionError(why));
}
}
@@ -0,0 +1,774 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.IOException;
import java.io.Serializable;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Every table has a create date and a modify date: what the metadata says of
** them, who may read them, every writer of this QBit setting them, what the
** access-log writer runs for a row, the records of a changed policy, and rows
** that were written before the two columns existed.
*******************************************************************************/
class SecuredDataStandardDatesTest
{
static final String DDL = "docs/ddl/secured-data-postgres.sql";
/** The first line of the block of the DDL that an existing database runs to get the two columns. */
static final String DDL_BLOCK = "-- Standard dates:";
/** The tables that had the two dates before the other tables did: they keep the field names they have always had. */
private static final Set<String> SNAKE_CASE_NAMES = Set.of("secured_data_classification", "secured_data_policy", "secured_data_retention_rule");
private static final Instant LATER = NOW.plusSeconds(90);
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
RetentionTestSupport.close();
}
/*******************************************************************************
** The tables the producer put in an instance, whichever class declares them,
** so that a table added later is covered without being named here.
*******************************************************************************/
static List<QTableMetaData> declared(QInstance instance)
{
String qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getName();
List<QTableMetaData> tables = instance.getTables().values().stream().filter(table -> qBit.equals(table.getSourceQBitName())).toList();
assertThat(tables).extracting(QTableMetaData::getName).containsAll(SecuredDataTables.ALL);
return (tables);
}
/*******************************************************************************
** Create date field name.
*******************************************************************************/
private static String createDate(String table)
{
return (SNAKE_CASE_NAMES.contains(table) ? "create_date" : "createDate");
}
/*******************************************************************************
** Modify date field name.
*******************************************************************************/
private static String modifyDate(String table)
{
return (SNAKE_CASE_NAMES.contains(table) ? "modify_date" : "modifyDate");
}
/*******************************************************************************
** What the QBit declares, before core enriches anything: core gives a field
** named createDate or modifyDate its behavior by itself, unless a host turned
** that off, and gives the snake_case names nothing.
*******************************************************************************/
@Test
void everyDeclaredTableHasACreateDateAndAModifyDate() throws Exception
{
QInstance instance = base();
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
instance.addTable(person("memory"));
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withClock(CLOCK).withAccessLogFlushIntervalMillis(60000)).produce(instance);
QContext.init(instance, user("u1"));
RUNTIMES.add(SecuredDataRuntime.current());
for(QTableMetaData table : declared(instance))
{
assertDate(table, createDate(table.getName()), "create_date", DynamicDefaultValueBehavior.CREATE_DATE);
assertDate(table, modifyDate(table.getName()), "modify_date", DynamicDefaultValueBehavior.MODIFY_DATE);
}
}
/*******************************************************************************
** One date field: a date-time nobody edits, kept by core, in a section.
*******************************************************************************/
private static void assertDate(QTableMetaData table, String name, String column, DynamicDefaultValueBehavior behavior)
{
String where = table.getName() + "." + name;
assertThat(table.getFields()).as(where).containsKey(name);
QFieldMetaData field = table.getField(name);
assertThat(field.getType()).as(where).isEqualTo(QFieldType.DATE_TIME);
assertThat(field.getIsEditable()).as(where).isFalse();
assertThat(field.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class)).as(where).isEqualTo(behavior);
assertThat(field.getBackendName()).as(where).isEqualTo(column);
assertThat(table.getFields().values()).as(where).filteredOn(other -> other.getBehaviorOnlyIfSet(DynamicDefaultValueBehavior.class) == behavior).hasSize(1);
assertThat(table.getSections()).as(where).anySatisfy(section -> assertThat(section.getFieldNames()).contains(name));
}
/*******************************************************************************
** The two dates have no classification, policy or assistant exposure of their
** own. Whoever asks, they are given the tier of every other field of their
** table that has none of its own either, and the assistant is given nothing.
*******************************************************************************/
@Test
void theDatesAreReadableExactlyWhereTheOtherFieldsOfTheirTableAre() throws Exception
{
QInstance instance = setup(config -> config.withDefaultTier(DataClassification.CONFIDENTIAL, FieldAccessTier.MASKED));
insert(SecuredDataTables.POLICY, policyRow("hidden", "CONFIDENTIAL", "HIDDEN"));
insert(SecuredDataTables.POLICY, policyRow("full", "CONFIDENTIAL", "FULL"));
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "FULL"));
insert(SecuredDataTables.POLICY, policyRow("capped", "CONFIDENTIAL", "HIDDEN").withValue("client_channel", "web"));
Map<String, QSession> sessions = new LinkedHashMap<>();
sessions.put("the default tier", user("u1"));
sessions.put("a policy that hides", user("u2", "hidden"));
sessions.put("a policy that shows", user("u3", "full"));
sessions.put("a channel cap", SecuredDataTestSupport.channelUser("web", "capped"));
sessions.put("the administrator", user("u4", "secured_data.admin"));
Map<String, FieldAccessTier> expected = Map.of("the default tier", FieldAccessTier.MASKED, "a policy that hides", FieldAccessTier.HIDDEN,
"a policy that shows", FieldAccessTier.FULL, "a channel cap", FieldAccessTier.HIDDEN, "the administrator", FieldAccessTier.FULL);
for(Map.Entry<String, QSession> session : sessions.entrySet())
{
QContext.setQSession(session.getValue());
for(QTableMetaData table : declared(instance))
{
for(String name : List.of(createDate(table.getName()), modifyDate(table.getName())))
{
String where = session.getKey() + ": " + table.getName() + "." + name;
QFieldMetaData date = table.getField(name);
assertThat(AiFieldMetaData.of(date)).as(where).isNull();
assertThat(SecuredDataClassifications.effective(table, name)).as(where).isEqualTo(DataClassification.CONFIDENTIAL);
FieldAccessTier tier = SecuredDataRuntime.current().resolver().resolve(table, date).tier();
assertThat(tier).as(where).isEqualTo(expected.get(session.getKey()));
for(QFieldMetaData other : table.getFields().values())
{
if(!other.getName().equals(table.getPrimaryKeyField()))
{
assertThat(SecuredDataRuntime.current().resolver().resolve(table, other).tier()).as(where + " and " + other.getName()).isEqualTo(tier);
}
}
}
assertThat(AiTableMetaData.of(table).getExposure()).as(table.getName()).isEqualTo(AiExposure.NONE);
}
}
}
/*******************************************************************************
** Every writer of this QBit, on the memory backend: the access-log writer, a
** changed policy and classification with their own log rows, a legal hold, a
** reveal and a break-glass session from its start to its end and its expiry.
** A new row has both dates, equal, from the clock of the QBit. A changed row
** keeps its create date and has a later modify date, which core chooses (the
** modify date is the record version of its table).
*******************************************************************************/
@Test
void everyWriterSetsBothDatesAndOnlyAChangeMovesTheModifyDate() throws Exception
{
setup();
////////////////////////////////////////////////////////////////////////////
// the access-log writer: created_at is when the access happened, the two //
// dates are when the writer's one insert wrote the row //
////////////////////////////////////////////////////////////////////////////
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hmac"));
CLOCK.now = LATER;
SecuredDataRuntime.current().logger().flush();
QRecord logged = logs().getLast();
assertThat(logged.getValueString("event_type")).isEqualTo("EXACT_MATCH_FILTER");
assertThat(logged.getValueInstant("created_at")).isEqualTo(NOW);
assertDates(logged, SecuredDataTables.ACCESS_LOG, LATER, LATER);
assertThatThrownBy(() -> update(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("id", logged.getValue("id")).withValue("outcome", "DENIED"))).isInstanceOf(Exception.class);
assertDates(row(SecuredDataTables.ACCESS_LOG, logged.getValue("id")), SecuredDataTables.ACCESS_LOG, LATER, LATER);
CLOCK.now = NOW;
///////////////////////////////////////////////////////////////////////
// the rows the rules are read from, and the log row of every change //
///////////////////////////////////////////////////////////////////////
QRecord policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("reason_options", "Support").withValue("allow_free_text_reason", false));
QRecord classification = classification("notes", "PII");
QRecord hold = insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "person").withValue("reason", "Preservation"));
Map<String, Serializable> written = new LinkedHashMap<>();
written.put(SecuredDataTables.POLICY, policy.getValue("id"));
written.put(SecuredDataTables.CLASSIFICATION, classification.getValue("id"));
written.put(SecuredDataTables.LEGAL_HOLD, hold.getValue("id"));
for(Map.Entry<String, Serializable> row : written.entrySet())
{
assertDates(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
}
assertDates(log("POLICY_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
assertDates(log("CLASSIFICATION_CHANGE", "INSERT"), SecuredDataTables.ACCESS_LOG, NOW, NOW);
CLOCK.now = LATER;
update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("name", "renamed"));
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", classification.getValue("id")).withValue("notes", "reviewed"));
update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED"));
for(Map.Entry<String, Serializable> row : written.entrySet())
{
assertChanged(row(row.getKey(), row.getValue()), row.getKey(), NOW, NOW);
}
assertDates(log("POLICY_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
assertDates(log("CLASSIFICATION_CHANGE", "UPDATE"), SecuredDataTables.ACCESS_LOG, LATER, LATER);
CLOCK.now = NOW;
////////////////////////////////////////////
// a reveal: its grant and its log row //
////////////////////////////////////////////
Serializable person = insert("person", new QRecord().withValue("email", "someone@example.invalid")).getValue("id");
new RevealService().reveal("person", person, "email", "Support", null);
assertDates(SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()).getFirst(), SecuredDataTables.REVEAL_GRANT, NOW, NOW);
assertDates(log("REVEAL", null), SecuredDataTables.ACCESS_LOG, NOW, NOW);
//////////////////////////////////////////////////////////////////////
// a break-glass session: started, then ended; started, then expired //
//////////////////////////////////////////////////////////////////////
BreakGlassService breaks = new BreakGlassService();
QContext.setQSession(user("u1", "cs_admin", BreakGlassService.START + ".hasAccess"));
Serializable ended = breaks.start("Investigating urgent incident", 60, null, null).getValue("id");
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
CLOCK.now = LATER;
breaks.end(ended);
Instant endedAt = assertChanged(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, NOW);
assertDates(log("BREAK_GLASS_END", null), SecuredDataTables.ACCESS_LOG, LATER, LATER);
Serializable expired = breaks.start("Investigating another incident", 1, null, null).getValue("id");
assertDates(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
CLOCK.now = LATER.plusSeconds(60);
assertThat(breaks.expire()).isEqualTo(1);
assertChanged(breaks.find(expired), SecuredDataTables.BREAK_GLASS, LATER, LATER);
assertDates(breaks.find(ended), SecuredDataTables.BREAK_GLASS, NOW, endedAt);
/////////////////////////////////////////////////////////////////////////
// the access log is only ever inserted, so no row's modify date moved //
/////////////////////////////////////////////////////////////////////////
assertThat(logs()).hasSizeGreaterThan(8).allSatisfy(row ->
{
assertThat(row.getValueInstant("createDate")).isNotNull();
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("createDate"));
});
}
/*******************************************************************************
** The writers of a wipe, in the shipped schema: the audit key, the run, the
** marker (written once, then changed by a later wipe of the same field), the
** log row, and the rule the run is recorded on. A time is compared with what
** the same database gives back for another time written from the same clock,
** not with the clock: the test database shifts a time by the zone of the
** machine it runs on.
*******************************************************************************/
@Test
void aWipeSetsBothDatesInTheShippedSchema() throws Exception
{
RetentionTestSupport.setup();
QRecord rule = RetentionTestSupport.rule("OVERWRITE_TEXT", "secret");
QRecord reading = RetentionTestSupport.reading(120);
assertThat(SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", reading.getValue("id"), true)).isPresent();
QRecord key = RetentionTestSupport.rows(SecuredDataTables.AUDIT_KEY).getFirst();
assertDates(key, SecuredDataTables.AUDIT_KEY, key.getValueInstant("created_at"), key.getValueInstant("created_at"));
Instant ruleWritten = row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")).getValueInstant("create_date");
assertDates(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
CLOCK.now = LATER;
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
Instant wiped = ruleWritten.plusSeconds(90);
QRecord firstRun = row(SecuredDataTables.WIPE_RUN, run.getValue("id"));
assertThat(firstRun.getValueInstant("finished_at")).isEqualTo(wiped);
assertDates(firstRun, SecuredDataTables.WIPE_RUN, wiped, wiped);
QRecord marker = RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER).getFirst();
assertThat(marker.getValueInstant("wiped_at")).isEqualTo(wiped);
assertDates(marker, SecuredDataTables.WIPE_MARKER, wiped, wiped);
assertThat(log("WIPE", "OVERWRITE_TEXT").getValueInstant("created_at")).isEqualTo(wiped);
assertDates(log("WIPE", "OVERWRITE_TEXT"), SecuredDataTables.ACCESS_LOG, wiped, wiped);
Instant ruleRun = assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleWritten);
///////////////////////////////////////////////////////////////////////////
// the field is given a value again and wiped again: the marker of that //
// record and field is changed, and no second marker is written //
///////////////////////////////////////////////////////////////////////////
CLOCK.now = LATER.plusSeconds(90);
Instant again = wiped.plusSeconds(90);
update("reading", new QRecord().withValue("id", reading.getValue("id")).withValue("secret", "another-test-secret"));
QRecord second = new WipeService().batch(rule.getValue("id"), List.of(reading.getValue("id")));
assertThat(second.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(RetentionTestSupport.rows(SecuredDataTables.WIPE_MARKER)).hasSize(1);
QRecord changed = row(SecuredDataTables.WIPE_MARKER, marker.getValue("id"));
assertThat(changed.getValue("run_id")).isEqualTo(second.getValue("id"));
assertThat(changed.getValueInstant("wiped_at")).isEqualTo(again);
assertChanged(changed, SecuredDataTables.WIPE_MARKER, wiped, wiped);
assertDates(row(SecuredDataTables.WIPE_RUN, run.getValue("id")), SecuredDataTables.WIPE_RUN, wiped, wiped);
assertDates(row(SecuredDataTables.WIPE_RUN, second.getValue("id")), SecuredDataTables.WIPE_RUN, again, again);
assertChanged(row(SecuredDataTables.RETENTION_RULE, rule.getValue("id")), SecuredDataTables.RETENTION_RULE, ruleWritten, ruleRun);
}
/*******************************************************************************
** The access-log writer runs concurrently with reveals, so the dates must cost
** it nothing: for each row it still runs one insert, which now names the two
** columns, and no other statement that writes the log.
*******************************************************************************/
@Test
void theAccessLogWriterStillRunsOneInsertForARow() throws Exception
{
RetentionTestSupport.setup();
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
{
connection.createStatement().execute("SET QUERY_STATISTICS TRUE");
for(int i = 0; i < 3; i++)
{
SecuredDataRuntime.current().logger().onAccess(new FieldAccessEvent().withType(FieldAccessEventType.EXACT_MATCH_FILTER).withOutcome(FieldAccessOutcome.MATCH)
.withTableName("reading").withFieldNames(List.of("secret")).withActionName("query").withRecordCount(1).withSearchValueFingerprints(List.of("hmac" + i)));
}
SecuredDataRuntime.current().logger().flush();
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).hasSize(3).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull().isEqualTo(row.get("modify_date")));
Map<String, Integer> writes = new LinkedHashMap<>();
try(ResultSet statistics = connection.createStatement().executeQuery("SELECT SQL_STATEMENT, EXECUTION_COUNT FROM INFORMATION_SCHEMA.QUERY_STATISTICS"))
{
while(statistics.next())
{
String sql = statistics.getString(1).strip();
if(sql.contains(SecuredDataTables.ACCESS_LOG) && !sql.toUpperCase().startsWith("SELECT"))
{
writes.merge(sql, statistics.getInt(2), Integer::sum);
}
}
}
assertThat(writes).hasSize(1);
assertThat(writes.keySet().iterator().next()).startsWithIgnoringCase("INSERT INTO").contains("create_date", "modify_date");
assertThat(writes.values()).containsExactly(3);
}
}
/*******************************************************************************
** A changed policy or classification writes what it wrote before the tables
** had the standard dates: its core audit, field by field, and its
** POLICY_CHANGE or CLASSIFICATION_CHANGE row in the access log.
*******************************************************************************/
@Test
void aChangedPolicyOrClassificationWritesItsAuditAndItsLogRowAsBefore() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
QRecord policy = new InsertAction().execute(new InsertInput(SecuredDataTables.POLICY).withRecord(policyRow("viewer", "PHI", "MASKED"))).getRecords().getFirst();
QRecord classification = new InsertAction().execute(new InsertInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("table_name", "reading")
.withValue("field_name", "notes").withValue("classification", "PHI"))).getRecords().getFirst();
assertThat(policy.getErrors()).isNullOrEmpty();
assertThat(classification.getErrors()).isNullOrEmpty();
CLOCK.now = LATER;
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.POLICY).withRecord(new QRecord().withValue("id", policy.getValue("id")).withValue("tier", "HIDDEN")))
.getRecords().getFirst().getErrors()).isNullOrEmpty();
assertThat(new UpdateAction().execute(new UpdateInput(SecuredDataTables.CLASSIFICATION).withRecord(new QRecord().withValue("id", classification.getValue("id"))
.withValue("classification", "PII"))).getRecords().getFirst().getErrors()).isNullOrEmpty();
new DeleteAction().execute(new DeleteInput(SecuredDataTables.POLICY).withPrimaryKeys(List.of(policy.getValue("id"))));
new DeleteAction().execute(new DeleteInput(SecuredDataTables.CLASSIFICATION).withPrimaryKeys(List.of(classification.getValue("id"))));
///////////////////////////////////////////////////////////////////////////
// the log rows: one per change, with every value they have always had //
///////////////////////////////////////////////////////////////////////////
List<String> changes = new ArrayList<>();
List<QRecord> logged = RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG);
Instant inserted = logged.getFirst().getValueInstant("created_at");
for(QRecord row : logged)
{
changes.add(String.join(" ", row.getValueString("event_type"), row.getValueString("table_name"), row.getValueString("record_id"), row.getValueString("action_name"),
row.getValueString("outcome"), row.getValueString("user_id"), row.getValueString("user_name"),
"after " + Duration.between(inserted, row.getValueInstant("created_at")).toSeconds() + " s", String.valueOf(row.getValueBoolean("anomaly_flag"))));
for(String empty : List.of("field_name", "record_count", "reason", "reason_text", "client_channel", "break_glass_id", "search_value_fingerprints", "rule_id"))
{
assertThat(row.getValue(empty)).as(empty).isNull();
}
}
String policyId = policy.getValueString("id");
String classificationId = classification.getValueString("id");
assertThat(changes).containsExactly(
"POLICY_CHANGE secured_data_policy " + policyId + " INSERT ALLOWED admin admin after 0 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " INSERT ALLOWED admin admin after 0 s false",
"POLICY_CHANGE secured_data_policy " + policyId + " UPDATE ALLOWED admin admin after 90 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " UPDATE ALLOWED admin admin after 90 s false",
"POLICY_CHANGE secured_data_policy " + policyId + " DELETE ALLOWED admin admin after 90 s false",
"CLASSIFICATION_CHANGE secured_data_classification " + classificationId + " DELETE ALLOWED admin admin after 90 s false");
//////////////////////////////////////////////////////////////////////////////
// the core audits of the two rows: the same audits with the same details, //
// the two dates among them under the names they have always been audited by //
//////////////////////////////////////////////////////////////////////////////
assertThat(audits(instance, SecuredDataTables.POLICY, policy.getValue("id"))).containsExactly(
"Record was Inserted: allow_free_text_reason [Set allow free text reason to Yes], classification [Set classification to PHI], create_date,"
+ " distinct_value_alert [Set distinct value alert to 15], exact_match_per_hour [Set exact match per hour to 20], id [Set Id to " + policyId + "],"
+ " is_active [Set is active to Yes], min_entropy_enabled [Set min entropy enabled to Yes], miss_alert_consecutive [Set miss alert consecutive to 5],"
+ " modified_by [Set modified by to admin], modify_date, name [Set name to viewer PHI], permission_name [Set permission name to viewer],"
+ " require_free_text [Set require free text to No], reveal_minutes [Set reveal minutes to 5], tier [Set tier to MASKED]",
"Record was Edited: modify_date, tier [Changed tier from \"MASKED\" to \"HIDDEN\"]",
"Record was Deleted: ");
assertThat(audits(instance, SecuredDataTables.CLASSIFICATION, classification.getValue("id"))).containsExactly(
"Record was Inserted: classification [Set classification to PHI], create_date, field_name [Set field name to notes], id [Set Id to " + classificationId + "],"
+ " modified_by [Set modified by to admin], modify_date, table_name [Set table name to Reading]",
"Record was Edited: classification [Changed classification from \"PHI\" to \"PII\"], modify_date",
"Record was Deleted: ");
}
/*******************************************************************************
** The core audits of one record, oldest first: the message of each and, in
** the order of the field names, the fields it has a detail for and what the
** detail says. The two dates are named only: how a time reads depends on the
** zone of the machine.
*******************************************************************************/
private static List<String> audits(QInstance instance, String table, Serializable id)
{
Serializable auditTable = SystemWrites.query("auditTable", new QQueryFilter().withCriteria(new QFilterCriteria("name", QCriteriaOperator.EQUALS, table))).getFirst().getValue("id");
List<String> audits = new ArrayList<>();
for(QRecord audit : SystemWrites.query("audit", new QQueryFilter().withCriteria(new QFilterCriteria("auditTableId", QCriteriaOperator.EQUALS, auditTable))
.withCriteria(new QFilterCriteria("recordId", QCriteriaOperator.EQUALS, id)).withOrderBy(new QFilterOrderBy("id", true))))
{
List<String> fields = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("auditId", QCriteriaOperator.EQUALS, audit.getValue("id"))))
.stream().map(detail -> detail.getValueString("fieldName") + (detail.getValueString("fieldName").endsWith("_date") ? ""
: " [" + detail.getValueString("message") + "]")).sorted().toList();
audits.add(audit.getValueString("message") + ": " + String.join(", ", fields));
}
assertThat(instance.getTable(table)).isNotNull();
return (audits);
}
/*******************************************************************************
** Rows written before the two columns existed. A reading is audited (its
** audit token is made with a data key of its own, and that key is stored
** wrapped, bound to its record and field), the log has rows, a rule and a
** hold exist. The two columns are then taken away, which leaves the database
** as the release before this one left it, and a key and a log row are
** written the way that release wrote them. The block an existing database
** runs must change nothing but the two columns: the same rows, the same
** wrapped keys, which still unwrap to the same data keys, and the same
** token, which still verifies the value and no other.
*******************************************************************************/
@Test
void rowsWrittenBeforeTheDatesExistedVerifyUnchangedAfterTheyAreAdded() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
QTableMetaData reading = instance.getTable("reading");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = RetentionTestSupport.rule("NULL_WITH_MARKER", "systolic");
insert(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", "404").withValue("reason", "Preservation"));
Serializable id = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 120).withValue("secret", "s")
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
Serializable wiped = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 130).withValue("secret", "s")
.withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst().getValue("id");
new WipeService().batch(rule.getValue("id"), List.of(wiped));
SecuredDataKeySource keys = SecuredDataRuntime.current().keys();
byte[] dataKey = keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key();
List<String> tokens = SystemWrites.query("auditDetail", new QQueryFilter().withCriteria(new QFilterCriteria("fieldName", QCriteriaOperator.EQUALS, "systolic"))).stream()
.map(detail -> detail.getValueString("newValue")).filter(value -> value != null && value.startsWith("shred:v1:")).toList();
String token = tokens.stream().filter(candidate -> ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, candidate)).findFirst().orElseThrow();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
List<String> tables = declared(instance).stream().map(QTableMetaData::getName).filter(name -> !SNAKE_CASE_NAMES.contains(name)).toList();
byte[] earlierKey = new byte[32];
Arrays.fill(earlierKey, (byte) 7);
Instant earlier = NOW.minusSeconds(86400);
try(Connection connection = DriverManager.getConnection(url(), "sa", ""))
{
for(String table : tables)
{
assertThat(rows(connection, table, false)).as(table).allSatisfy(row -> assertThat(row.get("create_date")).isNotNull());
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN create_date");
connection.createStatement().execute("ALTER TABLE " + table + " DROP COLUMN modify_date");
}
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_audit_key (table_name, record_id, field_name, wrapped_key, created_at) VALUES ('reading', '9001', 'systolic', ?, ?)"))
{
insert.setString(1, Base64.getEncoder().encodeToString(SecuredDataKeySource.encrypt(earlierKey, "reading\u001fsystolic\u001f9001")));
insert.setTimestamp(2, Timestamp.from(earlier));
insert.executeUpdate();
}
try(PreparedStatement insert = connection.prepareStatement("INSERT INTO secured_data_access_log (event_type, user_id, table_name, field_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u1', 'reading', 'systolic', 'ALLOWED', false, ?)"))
{
insert.setTimestamp(1, Timestamp.from(earlier));
insert.executeUpdate();
}
Map<String, List<Map<String, String>>> before = new LinkedHashMap<>();
for(String table : tables)
{
before.put(table, rows(connection, table, false));
}
assertThat(before.get(SecuredDataTables.ACCESS_LOG)).hasSizeGreaterThan(3);
assertThat(before.get(SecuredDataTables.AUDIT_KEY)).hasSize(2);
for(String table : List.of(SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD))
{
assertThat(before.get(table)).as(table).hasSize(1);
}
run(connection, ddl(true));
for(String table : tables)
{
assertThat(rows(connection, table, true)).as(table).isEqualTo(before.get(table));
assertThat(rows(connection, table, false)).as(table).allSatisfy(row ->
{
assertThat(row.get("create_date")).isNotNull();
assertThat(row.get("modify_date")).isNotNull();
});
}
assertThat(rows(connection, SecuredDataTables.ACCESS_LOG, false)).allSatisfy(row ->
{
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
});
assertThat(rows(connection, SecuredDataTables.AUDIT_KEY, false)).allSatisfy(row ->
{
assertThat(row.get("create_date")).isEqualTo(row.get("created_at"));
assertThat(row.get("modify_date")).isEqualTo(row.get("created_at"));
});
/////////////////////////////////////////////////////////////////////////////
// the release before this one goes on writing while both run: its insert //
// names neither column, and the row has both, equal, all the same //
/////////////////////////////////////////////////////////////////////////////
connection.createStatement().execute("INSERT INTO secured_data_access_log (event_type, user_id, table_name, outcome, anomaly_flag, created_at) VALUES ('VIEW', 'u2', 'reading', 'ALLOWED', false, CURRENT_TIMESTAMP)");
Map<String, String> concurrent = rows(connection, SecuredDataTables.ACCESS_LOG, false).getLast();
assertThat(concurrent.get("user_id")).isEqualTo("u2");
assertThat(concurrent.get("create_date")).isNotNull().isEqualTo(concurrent.get("modify_date"));
}
///////////////////////////////////////////////////////////////////////////
// nothing kept in memory: every key is unwrapped again from its row //
///////////////////////////////////////////////////////////////////////////
keys.close();
assertThat(keys.keyForRecordField("reading", "systolic", id, false).orElseThrow().key()).isEqualTo(dataKey);
assertThat(keys.keyForRecordField("reading", "systolic", 9001, false).orElseThrow().key()).isEqualTo(earlierKey);
assertThat(keys.keyForRecordField("reading", "systolic", wiped, false)).isEmpty();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isTrue();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 121, token)).isFalse();
////////////////////////////////////////////////////////////////////////////
// and the rows that were there are read and written on as they were: the //
// earlier log row is counted, and the reading is wiped under its rule //
////////////////////////////////////////////////////////////////////////////
assertThat(RetentionTestSupport.rows(SecuredDataTables.ACCESS_LOG)).anySatisfy(row ->
{
assertThat(row.getValueString("user_id")).isEqualTo("u1");
assertThat(row.getValueInstant("createDate")).isNotNull().isEqualTo(row.getValueInstant("created_at"));
assertThat(row.getValueInstant("modifyDate")).isEqualTo(row.getValueInstant("created_at"));
});
CLOCK.now = LATER;
assertThat(new WipeService().batch(rule.getValue("id"), List.of(id)).getValueInteger("wiped_count")).isEqualTo(1);
assertThat(keys.keyForRecordField("reading", "systolic", id, false)).isEmpty();
assertThat(ClassifiedValueTokens.verifyRecordField(reading, reading.getField("systolic"), id, 120, token)).isFalse();
}
/*******************************************************************************
** The database RetentionTestSupport made from the shipped DDL.
*******************************************************************************/
static String url()
{
return ("jdbc:h2:mem:" + RetentionTestSupport.BACKEND + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1");
}
/*******************************************************************************
** The statements of the shipped DDL: those of the block an existing database
** runs to get the two columns, or those before it (the schema without them).
*******************************************************************************/
static List<String> ddl(boolean datesBlock) throws IOException
{
String ddl = Files.readString(Path.of(DDL));
int block = ddl.indexOf(DDL_BLOCK);
assertThat(block).as("the block of " + DDL + " that begins " + DDL_BLOCK).isPositive().isEqualTo(ddl.lastIndexOf(DDL_BLOCK));
return (Arrays.stream((datesBlock ? ddl.substring(block) : ddl.substring(0, block)).split(";")).filter(sql -> !sql.isBlank()).toList());
}
/*******************************************************************************
** Run statements.
*******************************************************************************/
static void run(Connection connection, List<String> statements) throws SQLException
{
for(String sql : statements)
{
connection.createStatement().execute(sql);
}
}
/*******************************************************************************
** Every row of a table as the database holds it, column by column, in id
** order, with or without the two date columns.
*******************************************************************************/
static List<Map<String, String>> rows(Connection connection, String table, boolean withoutDates) throws SQLException
{
List<Map<String, String>> rows = new ArrayList<>();
try(ResultSet results = connection.createStatement().executeQuery("SELECT * FROM " + table + " ORDER BY id"))
{
ResultSetMetaData columns = results.getMetaData();
while(results.next())
{
Map<String, String> row = new LinkedHashMap<>();
for(int i = 1; i <= columns.getColumnCount(); i++)
{
String column = columns.getColumnName(i);
if(!withoutDates || !Set.of("create_date", "modify_date").contains(column))
{
row.put(column, results.getString(i));
}
}
rows.add(row);
}
}
return (rows);
}
/*******************************************************************************
** One stored row.
*******************************************************************************/
private static QRecord row(String table, Serializable id)
{
return (SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id))).getFirst());
}
/*******************************************************************************
** The newest log row of an event, and of an action where one is given.
*******************************************************************************/
private static QRecord log(String event, String action)
{
return (logs().stream().filter(row -> event.equals(row.getValueString("event_type")) && (action == null || action.equals(row.getValueString("action_name"))))
.reduce((first, second) -> second).orElseThrow());
}
/*******************************************************************************
** Both dates of a stored row.
*******************************************************************************/
private static void assertDates(QRecord row, String table, Instant created, Instant modified)
{
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isEqualTo(modified);
}
/*******************************************************************************
** Both dates of a stored row that was changed: the create date it had, and a
** modify date later than the one it had, which is returned.
*******************************************************************************/
private static Instant assertChanged(QRecord row, String table, Instant created, Instant modifiedBefore)
{
assertThat(row.getValueInstant(createDate(table))).as(table + " create date").isEqualTo(created);
assertThat(row.getValueInstant(modifyDate(table))).as(table + " modify date").isAfter(modifiedBefore);
return (row.getValueInstant(modifyDate(table)));
}
}
@@ -30,9 +30,8 @@ import java.util.Arrays;
import java.util.List;
import java.util.function.Consumer;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.standin.ClientChannelSession;
import com.kingsrook.qbits.secureddata.standin.DataClassification;
import com.kingsrook.qbits.secureddata.support.CoreMetadataBridge;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
@@ -124,8 +123,8 @@ final class SecuredDataTestSupport
.withField(new QFieldMetaData("email", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("ssn", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("notes", QFieldType.TEXT));
CoreMetadataBridge.classifyField(table.getField("email"), DataClassification.PII);
CoreMetadataBridge.classifyField(table.getField("ssn"), DataClassification.PHI);
SecuredDataMetadata.classifyField(table.getField("email"), DataClassification.PII);
SecuredDataMetadata.classifyField(table.getField("ssn"), DataClassification.PHI);
table.getFields().values().forEach(field -> field.setBackendName(field.getName()));
return (table);
}
@@ -149,16 +148,7 @@ final class SecuredDataTestSupport
*******************************************************************************/
static QSession channelUser(String channel, String... permissions) throws Exception
{
QSession session;
try
{
session = new QSession();
QSession.class.getMethod("withClientChannel", String.class).invoke(session, channel);
}
catch(NoSuchMethodException expected)
{
session = new ClientChannelSession().withClientChannel(channel);
}
QSession session = new QSession().withClientChannel(channel);
List<String> grants = new ArrayList<>(Arrays.asList(permissions));
grants.add("person.read");
return (session.withUser(new QUser().withIdReference("u1").withFullName("User One")).withPermissions(grants));
@@ -244,6 +234,7 @@ final class SecuredDataTestSupport
*******************************************************************************/
static void close()
{
CLOCK.beforeInstant = null;
RUNTIMES.forEach(SecuredDataRuntime::close);
RUNTIMES.clear();
QContext.clear();
@@ -258,6 +249,9 @@ final class SecuredDataTestSupport
{
Instant now = NOW;
/** Run on the reading thread before every reading of the clock; a test parks a thread at a known point with it. */
volatile Runnable beforeInstant;
/*******************************************************************************
@@ -285,6 +279,11 @@ final class SecuredDataTestSupport
*******************************************************************************/
public Instant instant()
{
Runnable hook = beforeInstant;
if(hook != null)
{
hook.run();
}
return (now);
}
}
@@ -20,24 +20,25 @@
package com.kingsrook.qbits.secureddata;
import java.nio.file.Files;
import java.nio.file.Path;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtension;
import java.nio.file.Files;
import com.kingsrook.qqq.middleware.javalin.specs.v1.MiddlewareVersionV1;
import com.kingsrook.qqq.middleware.javalin.tools.UiApiContract;
import static org.assertj.core.api.Assertions.assertThat;
import com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension;
import org.junit.jupiter.api.Test;
/** STAND-IN(core=QRun-IO/qqq#1128, removeBy=RC.6): supplied worktree exposes UiApiContract, not the documented assertion facade. */
final class UiApiContractAssertions
class SecuredDataUiApiContractTest
{
static void assertExtensionMatchesCommitted(MiddlewareSpecExtension extension, Path path) throws Exception
@Test
void committedContractMatches() throws Exception
{
String generated = UiApiContract.generateExtensionYaml(new MiddlewareVersionV1(), extension);
Path path = Path.of("src/main/resources/openapi/qqq-v1/secured-data.yaml");
String generated = UiApiContract.generateExtensionYaml(new MiddlewareVersionV1(), SecuredDataExtension.defineExtension());
if(Boolean.getBoolean("qqq.uiApi.regenerate"))
{
Files.createDirectories(path.getParent());
Files.writeString(path, generated);
}
assertThat(UiApiContract.check(generated, Files.readString(path), null, 1, "Regenerate secured-data.yaml with -Dqqq.uiApi.regenerate=true")).isEmpty();
assertThat(UiApiContract.check(generated, Files.readString(path), null, 1, "Regenerate secured-data.yaml")).isEmpty();
}
}