Compare commits

...
Author SHA1 Message Date
james.maesandCodex 9f04dd3e91 ci: use the shared branch-aware CodeQL worker
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 10:24:06 -05:00
james.maesandCodex 9091f26dd7 ci: adopt central orb 0.7.12 and stable gates (#24)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 08:53:36 -05:00
james.maesandCodex 4942b7e994 ci: centralize rc release completion in qqq orb (#23)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 02:25:10 -05:00
james.maesandCodex 0875069488 ci: adopt shared orb 0.7.11 (#22)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 00:59:13 -05:00
james.maes bf81c587bc Merge pull request #21 from QRun-IO/feature/central-ci
ci: adopt shared QQQ orb standard
2026-10-20 19:19:22 -05:00
james.maesandCodex 70058c1ef6 ci: adopt verified central QQQ orb 0.7.3
Co-Authored-By: Codex <noreply@openai.com>
2026-10-20 19:08:13 -05:00
james.maes af62471cf6 Merge pull request #19 from QRun-IO/feature/access-log-lifecycle
fix: the access logger no longer keeps its instance alive
2026-10-16 11:41:27 -05:00
james.maesandClaude Opus 5.5 325150a938 fix: the access logger no longer keeps its instance alive
The worker thread's task and the JVM shutdown hook referenced the logger
(and through it the QInstance) strongly for the life of the JVM, so every
instance a host or test built and dropped stayed in memory (kof22's suite
ran out of heap in CI). Both now hold the logger weakly: a dropped
instance can be collected and its worker stops at the next tick.

- withRegisterShutdownHook(false) for hosts that close the runtime
  themselves (default true, unchanged behaviour).
- isWorkerRunning() for hosts and tests.
- README and CHANGELOG (also notes #18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:39:12 -05:00
james.maes 72ec4a08be Merge pull request #18 from QRun-IO/feature/virtual-fields-and-idle-flush
fix: resolve virtual fields; keep the access-log worker off unfinished instances
2026-10-16 11:35:24 -05:00
james.maesandClaude Opus 5.5 ef5c096e84 fix: resolve virtual fields and keep the access-log worker off unfinished instances
- SecuredDataClassifications.effective looked fields up with
  QTableMetaData.getField, which throws for a virtual (computed) field, so
  any user read of a table with one failed (kof22's permalink sitePath and
  siteUrl on every content table, and API validation). Virtual fields now
  take the table classification.
- The access-log worker opened a QContext on every tick, even with an empty
  queue. QContext validates an unvalidated instance, and the worker starts
  while the host is still building it, so the first tick validated (and
  enriched) a half-built instance on another thread; one failure cancelled
  every later tick for good. Idle ticks now return before opening a
  context, and a failed tick is logged and retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:42:05 -05:00
james.maes 75b5f0b64f Merge pull request #17 from QRun-IO/ci/release-rc-secrets-inherit
ci(release-rc): pass secrets so the shared workflow can sign the RC tag
2026-10-16 08:05:16 -05:00
james.maesandClaude Opus 5.5 2cb37a067d ci(release-rc): pass secrets to the shared workflow so it can sign the RC tag
QRun-IO/.github#2 adds a tag-and-release job that signs vX with the
QRUN_TAG_SIGNING_KEY secret after Maven Central confirms the artifact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:18:59 -05:00
james.maes 971ddbed2d Merge pull request #16 from QRun-IO/feature/ui-api-1.6
feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.15)
2026-10-15 16:17:33 -05:00
james.maesandClaude Opus 5.5 e2f5199e06 test(retention): canary leak scan without false positives
The canary was 16 random letters g-v and any 5-letter window counted as a
leak, so ordinary JSON words built from those letters (options, joins)
matched by chance and RetentionProofTest failed intermittently. The canary
now has 24 random letters and a leak is the whole marker or a 12-letter
window of its random part (16^12 values): chance matches are impossible,
truncated or re-encoded fragments of a real leak are still caught. A new
test pins both directions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 16:15:09 -05:00
james.maesandClaude Opus 5.5 214bd75298 chore: target QQQ 4.1.0-RC.15 (UI API 1.6.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:27:07 -05:00
james.maesandClaude Opus 5.5 6bb55f332a feat(ui-api): extension for QQQ UI API 1.6.0 (operationIds, BasicErrorResponseV1, QQQ 4.1.0-RC.14)
Regenerated against UI API 1.6.0: every operation has an operationId and
the standard error responses (BasicErrorResponseV1: error, code, message,
fieldErrors, traceId); the extension's own error schema is kept but
deprecated; hand-written error bodies go through QJavalinUtils so they carry
code/message/traceId. Extension MINOR bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:48:13 -05:00
james.maes 781bae3efe Merge pull request #15 from QRun-IO/feature/remove-standins
build: QQQ 4.1.0-RC.7; remove core stand-ins
2026-10-14 16:33:33 -05:00
james.maesandClaude Opus 5.5 677d611b15 build: start on the 0.1 version line, like the other new qbits
The UI API extension keeps its own semver (secured-data 1.0.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:27:45 -05:00
james.maesandClaude Opus 5.5 527ee4fc96 build: pin QQQ 4.1.0-RC.7
RC.7 ships field access (#1172) and audit value protection (#1182), so the
qbit builds on the released core with no stand-ins. Also repairs a
non-UTF-8 character in the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:25:17 -05:00
james.maes c0a2d67026 Merge remote-tracking branch 'origin/develop' into feature/remove-standins 2026-10-14 16:22:15 -05:00
james.maes f415f2cc8d Merge pull request #14 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:46:31 -05:00
james.maesandClaude Opus 5.5 0dec9463a6 chore: use core field access and audit-token APIs directly; remove stand-ins
QQQ #1172 and #1182 are released, so the standin package and the
reflective core bridges go; the resolver, key source and audit protector
register through core metadata, and the core-dependent tests run
unconditionally.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:10:18 -05:00
james.maesandClaude Opus 5.5 e9b8a05f87 ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:07:46 -05:00
james.maes f2736311da Merge pull request #13 from QRun-IO/chore/standin-rc8
chore: retag stand-ins removeBy=RC.8
2026-10-14 07:57:01 -05:00
james.maesandClaude Opus 5.5 d8e3cb976d chore: stand-ins are removed with the first QQQ RC that ships #1172 and #1182 (RC.8)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:52:48 -05:00
james.maes 9b2d8fd182 Merge pull request #11 from QRun-IO/feature/retention-and-wipe
feat: retention and wipe with crypto-shredding; QQQ RC.6 (phase f)
2026-10-14 07:42:39 -05:00
james.maesandClaude Opus 5.5 470b0bd214 feat: retention and wipe with crypto-shredding; move to QQQ RC.6 (phase f)
Moves the pin to QQQ 4.1.0-RC.6 and removes the stand-ins RC.6 provides
(classification, client channels, table features, the UI API extension
point). Adds retention rules, wipe markers, legal holds, wipe runs and the
per-record-per-field audit key table; the wipe process (null with marker,
overwrite, keyed hash, delete, archive then wipe) runs in one transaction
per batch, destroying the audit keys with the field updates; the audit
protector and key source (AES-GCM wrapped DEKs, AAD-bound); the backups
warning; canary and wipe-proof tests.

Fixes a key-source bug that zeroed each DEK after returning it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:35:45 -05:00
james.maes 9801c6b4d1 Merge pull request #9 from QRun-IO/feature/app-home
feat: app home and anomaly alerts (phase e)
2026-10-14 07:16:04 -05:00
james.maes 0ce1723019 Merge pull request #8 from QRun-IO/feature/reveal-and-break-glass
feat: reveal, break-glass and the secured-data UI API extension (phase c)
2026-10-14 07:15:51 -05:00
james.maesandClaude Opus 5.5 6606bf3e90 test: demo server uses MOCK auth so the demo administrator's permissions apply
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:30:48 -05:00
james.maesandClaude Opus 5.5 3457481841 feat: app home and anomaly alerts (phase e)
One Secured Data home: KPIs, recent reveals, top revealers and reasons,
break-glass events, policy and classification changes and anomaly flags,
as full-width tables. Anomaly rules (3x daily median, distinct records per
hour, exact-match volume, break-glass) raise secured_data.anomaly and flag
the log row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:20:16 -05:00
james.maesandClaude Opus 5.5 e26ba2b4ad feat: reveal, break-glass and the secured-data UI API extension (phase c)
Reveal service (options, reason validation, daily cap, grants, audit),
break-glass start/end/expiry with alerts and admin notification, status,
the secured-data 1.0.0 UI API extension (compiled under the uiapi profile
until RC.6) and a contingent field picker for classification rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:47:31 -05:00
james.maes 76da1c74fc Merge pull request #7 from QRun-IO/feature/policy-and-access-log
feat: classification, policy, tier resolver and access log (phase b)
2026-10-13 20:14:57 -05:00
james.maesandClaude Opus 5.5 3290128de9 chore: point the stand-in markers at QRun-IO/qqq#1172
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:12:43 -05:00
james.maesandClaude Opus 5.5 0963b45467 feat: classification, policy, tier resolver and access log (phase b)
Tables secured_data_classification, _policy, _reveal_grant, _break_glass
and _access_log; SecuredDataClassifications; the tier resolver with
break-glass, table features, policy rows, channel caps and code defaults;
the asynchronous access log; enumeration guardrails (rate limit, miss and
distinct-value alerts, minimum entropy). Builds on QQQ RC.5 with marked
stand-ins for the unreleased fieldaccess API and a reflective bridge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:03:57 -05:00
james.maes 99c97932b6 Merge pull request #5 from QRun-IO/fix/pin-qqq-rc
build: pin QQQ to 4.1.0-RC.5
2026-10-13 19:33:56 -05:00
james.maesandClaude Opus 5.5 3d50612a45 build: pin QQQ to 4.1.0-RC.5
Closes #4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:19:12 -05:00
james.maes 4e74994cc9 Merge pull request #1 from QRun-IO/feature/scaffold
chore: repo-standard scaffold
2026-10-13 18:34:49 -05:00
james.maesandClaude Opus 5.5 cc629078e9 chore: repo-standard scaffold for qbit-secured-data
Replaces the template example with the qbit-secured-data skeleton:
pom on qbit-build-parent with checkstyle and coverage gates, CircleCI,
release workflow, governance files and a minimal producer and config.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:22:17 -05:00
89 changed files with 12535 additions and 303 deletions

No files matched your search

+11 -18
View File
@@ -1,21 +1,14 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.5.0
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
qqq-orb: kingsrook/qqq-orb@0.7.12
workflows:
build:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
setup:
jobs:
- qqq-orb/generate_pipeline:
orb_version: 0.7.12
filters:
branches:
ignore: automation/qbit-rc-pending
tags:
only: /^v\d+\.\d+\.\d+$/
+20
View File
@@ -0,0 +1,20 @@
# QRun-IO qbit .editorconfig: the Kingsrook layout (3-space indentation, never tabs).
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
indent_style = space
indent_size = 3
tab_width = 3
[*.md]
trim_trailing_whitespace = false
[*.{yml,yaml}]
indent_size = 2
[*.{cmd,bat}]
end_of_line = crlf
+3
View File
@@ -0,0 +1,3 @@
* text=auto eol=lf
*.cmd text eol=crlf
mvnw text eol=lf
+25
View File
@@ -0,0 +1,25 @@
# Weekly, grouped dependency updates. Minor and patch updates arrive as one PR per ecosystem;
# major updates stay separate so each gets its own review. At most three version-update PRs per
# ecosystem are open at once. Security updates are enabled in the repository settings and are
# not limited by this schedule.
version: 2
updates:
- package-ecosystem: maven
directory: /
target-branch: develop
schedule:
interval: weekly
open-pull-requests-limit: 3
groups:
maven-minor-and-patch:
update-types: [minor, patch]
- package-ecosystem: github-actions
directory: /
target-branch: develop
schedule:
interval: weekly
open-pull-requests-limit: 3
groups:
github-actions:
update-types: [minor, patch]
+31
View File
@@ -0,0 +1,31 @@
# GitHub Release Configuration
# Auto-generates release notes from PR labels
changelog:
exclude:
labels:
- ignore-for-release
- duplicate
- invalid
authors:
- dependabot
- dependabot[bot]
categories:
- title: Breaking Changes
labels:
- breaking-change
- breaking
- title: New Features
labels:
- enhancement
- feature
- title: Bug Fixes
labels:
- bug
- fix
- title: Dependencies
labels:
- dependencies
- title: Other Changes
labels:
- "*"
+23
View File
@@ -0,0 +1,23 @@
name: CodeQL
'on':
push:
branches:
- main
- develop
- release/**
- hotfix/**
pull_request: {}
schedule:
- cron: 41 16 * * 0
workflow_dispatch: {}
permissions:
contents: read
actions: read
security-events: write
packages: read
concurrency:
group: codeql-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
analyze:
uses: QRun-IO/qqq-orb/.github/workflows/qbit-codeql.yml@c288a8864fe9054f68410e1d78f5863aea4c4d22
+74
View File
@@ -0,0 +1,74 @@
name: Release candidate
'on':
workflow_dispatch:
inputs:
dry_run:
description: Plan a cut without changing release state
type: boolean
default: false
finalize:
description: Retry finalization of an existing cut; do not cut another RC
type: boolean
default: false
recover:
description: Recover the same published RC after a deployment timeout; never
reupload
type: boolean
default: false
circle_build:
description: Original Circle publisher build number (recovery only)
type: string
default: ''
merge_sha:
description: Existing cut merge_sha (finalize retry only)
type: string
default: ''
bump_sha:
description: Existing cut bump_sha (finalize retry only)
type: string
default: ''
branch:
description: Existing cut branch (finalize retry only)
type: string
default: ''
version:
description: Existing cut version (finalize retry only)
type: string
default: ''
pull_request:
types:
- closed
branches:
- develop
repository_dispatch:
types:
- qbit-rc-published
permissions:
contents: write
actions: read
checks: read
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch'
|| (github.event_name == 'pull_request' && github.event.pull_request.merged
== true && github.event.pull_request.base.ref == 'develop' && contains(github.event.pull_request.labels.*.name,
'release:rc'))
uses: QRun-IO/qqq-orb/.github/workflows/qbit-release-rc.yml@cc30f599e63c8e41629c99d23395bac1adb8bc44
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || false
}}
finalize: ${{ github.event_name == 'repository_dispatch' || (github.event_name
== 'workflow_dispatch' && inputs.finalize) }}
recover: ${{ github.event_name == 'workflow_dispatch' && inputs.recover || false
}}
circle_build: ${{ github.event_name == 'workflow_dispatch' && inputs.circle_build
|| '' }}
merge_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.merge_sha
|| inputs.merge_sha || '' }}
bump_sha: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.bump_sha
|| inputs.bump_sha || '' }}
branch: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.branch
|| inputs.branch || '' }}
version: ${{ github.event_name == 'repository_dispatch' && github.event.client_payload.version
|| inputs.version || '' }}
secrets: inherit
+16 -15
View File
@@ -1,20 +1,21 @@
# Maven
target/
.env
.idea
*.iml
*.class
*.log
*.jar
*.war
*.nar
*.ear
# IDE
.idea/
*.iml
.vscode/
.project
.classpath
.settings/
# OS
*.zip
*.tar.gz
*.rar
hs_err_pid*
.DS_Store
Thumbs.db
# Logs
*.log
*.sw*
.flattened-pom.xml
dependency-reduced-pom.xml
/.env.local
/.cache/
.claude/settings.local.json
+3
View File
@@ -0,0 +1,3 @@
wrapperVersion=3.3.4
distributionType=only-script
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.11/apache-maven-3.9.11-bin.zip
+16
View File
@@ -0,0 +1,16 @@
profile: qbit
orb_version: 0.7.12
default_branch: develop
java_version: '21'
browsers: true
api_checks: false
docs_only: false
release_on_main: false
release_tag_policy: stable
test_branch_policy: non-release-no-integration
publish:
snapshot: true
release_candidate: true
release: true
asciidoc: false
release_completion: true
+42
View File
@@ -0,0 +1,42 @@
# AGENTS.md
Guidance for coding agents working in this repository.
## Overview
QBit Secured Data provides FULL/REVEALABLE/MASKED/HIDDEN field tiers by role × DataClassification, reveal grants, break-glass access and an access log. Masking is enforced in QQQ core. This is a scaffold only; domain behavior is not implemented yet.
Design: https://github.com/QRun-IO/qqq/blob/develop/docs/SECURED-DATA.md (§5–§8).
## Build commands
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
./mvnw -q -o verify
```
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`.
## Architecture
Package: `com.kingsrook.qbits.secureddata`. Entry point: `SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredDataQBitConfig>`. It registers QBit metadata and returns it in a `MetaDataProducerMultiOutput`, with no tables yet. Config exposes `tableMetaDataCustomizer` and an initially empty `Map<String, String> defaultTierByClassification`.
Planned tables: `secured_data_classification`, `secured_data_policy`, `secured_data_reveal_grant`, `secured_data_break_glass`, `secured_data_access_log`.
## Invariants and naming
QQQ core owns masking enforcement. Reveal, break-glass and access logging follow the spec; do not imply this scaffold enforces access. Tables use lowercase snake_case with the `secured_data_` prefix; backend names equal table names. Permissions use the full `secured_data` prefix. Processes are table or record actions, not navigation entries.
## Code style
3-space indent, next-line braces, `return (x);`, flower-box Javadoc on classes and methods, fluent `withX()` setters. Follow `checkstyle/` and `CODE_STYLE.md`.
## Dependencies
Parent: `com.kingsrook:qbit-build-parent:2.0.0`. QQQ BOM: `4.1.0-SNAPSHOT`, pending the RC with fieldaccess hooks. Core and Javalin are compile dependencies; memory/RDBMS backends, frontend-next, H2, JUnit and AssertJ support tests. Do not add qbit-alerts yet.
## Release flow
CircleCI uses `kingsrook/qqq-orb@0.6.8`: develop publishes SNAPSHOTs, release/* publishes RCs, and only `vX.Y.Z` GA tags publish releases. Hotfix branches run tests. RC automation uses workflow_dispatch or a merged develop PR labelled `release:rc`. Feature branches start from develop and return by PR. Never publish manually.
+57
View File
@@ -0,0 +1,57 @@
# Changelog
All notable changes are documented here, following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed: access-log lifecycle
- The access-log worker and JVM shutdown hook hold the logger weakly, so an instance the host drops (tests, reloads) can be collected; its worker stops at the next tick.
- `withRegisterShutdownHook(false)` for hosts that call `SecuredDataRuntime.close()` themselves; `SecuredDataAccessLogger.isWorkerRunning()`.
- Virtual (computed) fields resolve through the table classification instead of throwing (#18).
- Idle access-log ticks open no context, so the worker never validates an unfinished host instance, and a failed tick no longer stops the worker (#18).
### Changed: UI API 1.6.0
- UI API extension secured-data 1.1.0 (requires QQQ 4.1.0-RC.15, UI API 1.6.0): operationIds, standard error responses with BasicErrorResponseV1, extension tags, and retained SecuredDataErrorV1 schema deprecated since 1.1.0.
### Changed — direct core field access and audit protection
- Remove all #1172/#1182 stand-ins and reflective bridges; use core field-access, classified-value token, audit protection, wipe flag and shredded-event APIs directly.
- Register the resolver, key source and audit protector through core supplemental metadata.
- Run core field-access integration and retention proof tests unconditionally.
- Pin QQQ to `4.1.0-RC.7`, which ships field access (#1172) and audit value protection (#1182); all core stand-ins and reflective bridges are removed.
### Changed — RC.6 migration
- Pin QQQ to 4.1.0-RC.6; use released classification, client-channel and table-feature APIs directly.
- Fold UI API code and tests into the main build and remove the profile and reflective registration.
- Remove released-API stand-ins and the local contract assertion facade; retain #1172 field-access stand-ins with RC.7 removal markers.
### Added: phase (f) retention and wipe
- Retention rules, wipe markers, legal holds, wipe runs and a hidden, locked audit-key table, with the `secured_data.retention` permission and a "Retention rules" navigation entry.
- Actions NULL_WITH_MARKER, OVERWRITE_TEXT, KEYED_HASH, DELETE_RECORD and ARCHIVE_THEN_WIPE (host `ArchiveSinkInterface`), with validation and low-entropy refusal.
- Atomic wipe batches (fields, markers, key deletions, audits, access log and run row in one rollback-capable transaction), per-record and per-table legal holds, a Review/Result "Run wipe now" process and a daily scheduled run.
- Per-record-field AES-GCM wrapped DEKs (AAD-bound) for crypto-shredding of audit tokens, `CHANGED_ONLY` audit treatment, and the audit-value protector.
- App-home warning when known backup retention exceeds a rule's `age_days`; `withBackupRetentionDays` and `withArchiveSink` options; README retention section and the backups residual risk.
- Tests: all actions, holds, rollback, Review/Result, schedule, permissions, wrap/unwrap, and the section 15 canary and wipe-proof tests.
### Fixed
- The key source returned a DEK array it then zeroed; it now returns copies, with a regression test.
### Added
- Phase (e) configurable reveal-median, distinct-record and exact-match anomaly detection, with system access-log counters, bounded daily median caching, per-user/day alerts, and flags retained without an alert sink.
- Break-glass start and capped-reveal anomaly flags alongside the existing alerts.
- One Secured Data app home with six live KPIs, six full-width responsive lists, filtered View all links, theme tokens, compact dates, clamped text/tooltips and explicit empty states.
- Transaction-aware, value-free policy/classification change audit rows from post-write customizers and two access-log indexes for anomaly/home queries.
- Boundary, alert, home data, rendering and navigation tests plus a synthetic Next demo harness for coordinator screenshot capture.
- Phase (c) HTTP-independent reveal options, reason validation, UTC daily limits, grants, audit links, cap alerts and assistant refusal.
- Five-second per-user grant cache with immediate invalidation and read-time expiration.
- Start/End break-glass table and record actions, streamed Review/Result summaries, owner/admin authorization, scheduled expiration, alerts and optional admin bell notification.
- Caller status and the versioned `secured-data` UI API extension behind the temporary `uiapi` Maven profile, with committed OpenAPI and contract/HTTP tests.
- Contingent classification field picker support for snake_case and camelCase table selections; unknown-table rejection.
- Memory and real-core coverage for reveal, break-glass, record locks and expiry. Reason columns support 500 characters; existing RDBMS installations must widen the three `reason` columns to VARCHAR(500).
- Phase (b) secured-data classification, policy, reveal-grant, break-glass and insert-only access-log tables, metadata, permissions, pickers and app navigation.
- Versioned classification and policy resolution with channel caps, table features, break-glass expiry and access fingerprints.
- Bounded asynchronous value-free logging, HMAC search fingerprints, rate limits, entropy rules and enumeration alerts.
- RC.5-compatible stand-ins and reflective core adapters; conditional core Query/Get integration coverage.
- PostgreSQL DDL, H2 persistence/constraint coverage, and configuration/deployment documentation.
+35 -10
View File
@@ -1,17 +1,42 @@
# qbit-template-extension
# CLAUDE.md
GitHub template repository for scaffolding QQQ "Extension QBits" (infrastructure-level plugins: table customizers, auth, audit, action wrappers). Not a shipping library — consumers click "Use this template" and rename the `com.kingsrook.qbits.example` package.
Guidance for coding agents working in this repository.
## QQQ 4.0 baseline — 2026-09-24
## Overview
The historical review below predates the 4.0 migration. This template targets Java 21 and the public QQQ 4.0.0 release from Maven Central. The corrected APIs, license references and actual template commands are in README.md and docs/00-getting-started.md. Preserve the documented multi-instance and scaffold limitations.
QBit Secured Data provides FULL/REVEALABLE/MASKED/HIDDEN field tiers by role × DataClassification, reveal grants, break-glass access and an access log. Masking is enforced in QQQ core. This is a scaffold only; domain behavior is not implemented yet.
## Knowledge base
Design: https://github.com/QRun-IO/qqq/blob/develop/docs/SECURED-DATA.md (§5–§8).
Reviewed dossiers for this repo and the wider QQQ platform live in the second-brain vault:
## Build commands
- Hub / entry point: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/qqq-hub.md`
- This repo's dossier: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/repos/qbit-template-extension.md`
- QBit mechanics refresher: `R:/Git.Local/KofTwentyTwo/second-brain/knowledge/qqq/architecture/metadata-model.md`
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
./mvnw -q -o verify
```
Reviewed at commit `e7d4b4d672cc` (branch `main`, 2026-05-21). Known drift as of that review: no `qbit-build-parent` parent pom (org convention), qqq pinned at 0.35.0 on main (0.40.0-SNAPSHOT on develop), README still claims AGPL-3.0 while LICENSE/NOTICE are Apache-2.0, and README references `scripts/customize_template.py` / `actions/ExampleActionCustomizer.java` which do not exist. See the dossier's "Maturity & risks" and "v4.0 impact" sections before modifying the scaffold.
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`.
## Architecture
Package: `com.kingsrook.qbits.secureddata`. Entry point: `SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredDataQBitConfig>`. It registers QBit metadata and returns it in a `MetaDataProducerMultiOutput`, with no tables yet. Config exposes `tableMetaDataCustomizer` and an initially empty `Map<String, String> defaultTierByClassification`.
Planned tables: `secured_data_classification`, `secured_data_policy`, `secured_data_reveal_grant`, `secured_data_break_glass`, `secured_data_access_log`.
## Invariants and naming
QQQ core owns masking enforcement. Reveal, break-glass and access logging follow the spec; do not imply this scaffold enforces access. Tables use lowercase snake_case with the `secured_data_` prefix; backend names equal table names. Permissions use the full `secured_data` prefix. Processes are table or record actions, not navigation entries.
## Code style
3-space indent, next-line braces, `return (x);`, flower-box Javadoc on classes and methods, fluent `withX()` setters. Follow `checkstyle/` and `CODE_STYLE.md`.
## Dependencies
Parent: `com.kingsrook:qbit-build-parent:2.0.0`. QQQ BOM: `4.1.0-SNAPSHOT`, pending the RC with fieldaccess hooks. Core and Javalin are compile dependencies; memory/RDBMS backends, frontend-next, H2, JUnit and AssertJ support tests. Do not add qbit-alerts yet.
## Release flow
CircleCI uses `kingsrook/qqq-orb@0.6.8`: develop publishes SNAPSHOTs, release/* publishes RCs, and only `vX.Y.Z` GA tags publish releases. Hotfix branches run tests. RC automation uses workflow_dispatch or a merged develop PR labelled `release:rc`. Feature branches start from develop and return by PR. Never publish manually.
+3
View File
@@ -0,0 +1,3 @@
# Code of Conduct
See Project level [Code of Conduct](https://github.com/QRun-IO/qqq/blob/develop/CODE_OF_CONDUCT.md)
+3
View File
@@ -0,0 +1,3 @@
# Code Style Guide
* See Project [Code Style Guide](https://github.com/QRun-IO/qqq/blob/develop/CODE_STYLE.md)
+6
View File
@@ -0,0 +1,6 @@
# Contributing to QBit Secured Data
* See Project [Contributing](https://github.com/QRun-IO/qqq/blob/develop/CONTRIBUTING.md)
* Branches follow GitFlow: `feature/*` from `develop`, pull requests to `develop`, `release/*` for release candidates, `vX.Y.Z` tags for releases.
* Commits are signed and follow Conventional Commits.
* Build and test locally with `./mvnw verify` (Java 21).
+1 -1
View File
@@ -176,7 +176,7 @@
END OF TERMS AND CONDITIONS
Copyright 2022-2025 QRunIO, LLC
Copyright 2024 Kingsrook, LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+3 -3
View File
@@ -1,6 +1,6 @@
qBit Template - Extension
Copyright 2022-2025 QRunIO, LLC
QBit Secured Data
Copyright 2026 Kingsrook, LLC
This product includes software developed at QRunIO, LLC (https://qrun.io/).
This product includes software developed at Kingsrook, LLC (https://kingsrook.com/).
Licensed under the Apache License, Version 2.0.
+239 -27
View File
@@ -1,42 +1,254 @@
# QBit Template: Extension
# QBit Secured Data
Template repository for creating QQQ Extension QBits - infrastructure extensions, customizers, and action handlers.
Phases (b), (c), (e) and (f) of [SECURED-DATA](https://github.com/QRun-IO/qqq/blob/develop/docs/SECURED-DATA.md): classification overrides, policy resolution, enumeration guardrails, and value-free access logging. QQQ core owns masking and query enforcement. Reveal, break-glass, the app home and anomaly alerts are implemented. Retention, wipe and audit-value protection are implemented (phase f); see Retention and wipe.
## What is an Extension QBit?
## Compatibility
Extension QBits extend or enhance how QQQ operates at the infrastructure/framework level. They provide:
- New backend implementations
- Authentication mechanisms
- Audit capabilities
- Table customizers
- Action handlers
Java 21, Maven wrapper 3.9.11, and **QQQ 4.1.0-RC.15** or later (UI API 1.6.0, field access #1172 and audit value protection #1182).
## Quick Start
Core classifications, AI metadata, client channels, table-feature permissions and UI API extension points are used directly. The UI API and its contract test are part of the main build; the contract test calls `UiApiContract` directly.
Requires **Java 21**, **Maven 3.8+**, and QQQ **4.0.0**. Create your repository, customize its Maven coordinates and Java package/classes, implement the table customizer, then run `mvn clean verify`. See [Getting Started](docs/00-getting-started.md).
Field access (#1172), classified-value tokens and audit-value protection (#1182) use core types directly. The producer registers the resolver, key source and audit protector through core supplemental metadata; QQQ core enforces field tiers.
## Structure
## Install and register
```
src/main/java/com/kingsrook/qbits/example/
├── ExampleExtensionQBitConfig.java # Configuration
├── ExampleExtensionQBitProducer.java # Entry point
└── customizers/
└── ExampleTableCustomizer.java # Table customization
```xml
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-secured-data</artifactId>
<version>0.1.0-RC.1</version>
</dependency>
```
## Key Characteristics
```java
new SecuredDataQBitProducer()
.withQBitConfig(new SecuredDataQBitConfig()
.withBackendName("rdbms")
.withDefaultTier(DataClassification.PII, FieldAccessTier.MASKED))
.produce(qInstance);
```
- **No QAppSection** - Pure infrastructure
- **No user-facing tables** (or minimal operational)
- **Wraps existing tables/processes** with new behavior
- **Validates dependencies** on other QQQ components
Register the host backend and application tables before producing the QBit. Enable core permissions (`qInstance.withEnforcePermissionsInCoreActions(true)`) at application entry points. Only one secured-data runtime is supported per instance. A namespace labels the QBit identity; table names remain fixed.
## Documentation
The host supplies database connection metadata. Physical table names and field backend names are the same snake_case names as the QQQ metadata; an RDBMS host may set `RDBMSTableBackendDetails.withTableName(table.getName())` through `withTableMetaDataCustomizer(...)`. Apply [the PostgreSQL 15+ DDL](docs/ddl/secured-data-postgres.sql) before serving requests. The same SQL is tested with H2 in PostgreSQL mode.
- [Getting Started](docs/00-getting-started.md)
- [Extension Patterns](docs/01-extension-patterns.md)
## Tables, screens and permissions
| Table | Label | User access |
|---|---|---|
| `secured_data_classification` | Classification | Admin management |
| `secured_data_policy` | Policy | Admin management |
| `secured_data_reveal_grant` | Reveal | Admin read; system writes |
| `secured_data_break_glass` | Break-glass session | Admin read; system writes |
| `secured_data_access_log` | Access log | Admin read; system insert only |
Every table has audit rules, explicit record sections and CONFIDENTIAL code classification. The Secured Data app navigates to Classification, Policy, Access log and Break-glass session. Reveal grants are a read-only child list joined by `access_log_id`; no process or home widget appears in navigation. The classification target `(table_name, field_name)` is unique, including the null field used for a table-wide override.
`secured_data.admin` gates configuration editing and activity reads through a standard QQQ custom permission checker. It gives no implicit access to classified fields on application tables. `secured_data.hasAccess` is the catalog base permission; the custom checker requires admin. Activity writes are refused for ordinary users even when they hold admin. Customizers also protect system-only writes, and update/delete capabilities are disabled on the access log, including system calls.
Both table features are registered for **all tables**: `secured_data_full` (Secured data: full) and `secured_data_reveal` (Secured data: reveal). This deliberate catalog choice supports new admin classification rows without a restart. Core's permission-base, READ prerequisite and NOT_PROTECTED semantics are preserved: NOT_PROTECTED application tables grant their features without an explicit permission. Use protected table permission rules when field policies must restrict users. Feature grants have no standard WRITE/admin implication.
The assistant is denied `secured_data_*` through core client-channel metadata. Use `QSession.withClientChannel("assistant/t1")` from trusted request context. The normalized channel `assistant` is stored in logs and used for policy caps.
ESB integration follows webhooks' host-wiring pattern: `SecuredDataTables.esbTableNames()` lists Classification, Policy, Access log and Break-glass. Connect those names to the host ESB table-event producer. No ESB or qbit-alerts dependency is added.
## Classifying fields
Code metadata establishes defaults using the core APIs:
```java
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
email.withSupplementalMetaData(
new AiFieldMetaData().withClassification(DataClassification.PII));
```
Core `AiTableMetaData` sets a table default. Runtime classification rows take precedence over code metadata.
Admin rows in Classification can raise **or lower** the default. Effective order:
1. Field-specific classification row.
2. Table-wide row (`field_name` null).
3. Code field classification.
4. Code table classification.
5. Unclassified.
Rows can also set `mask_style` and `low_cardinality`: true/false override entropy detection in either direction; null selects automatic rules. The tables picker reuses core's PVS, and the fields picker uses the selected `table_name`.
Other QBits use `SecuredDataClassifications.effective(table, fieldName)`, `effective(table)`, `highest(table)`, or `highest(table, fieldNames)` under their QContext. An empty field subset returns null. Rows are cached by classification version; startup and successful post-commit insert/update/delete change the version.
## Policy resolution and guardrails
Unclassified/PUBLIC fields and primary keys are FULL. Active, unexpired break-glass grants cover a table or all tables up to their classification ceiling. Full/reveal table features combine with the least restrictive matching non-channel policy tier. A policy matches an active row, the session's `permission_name`, classification and optional table. If no grant row matches, code defaults are CONFIDENTIAL FULL, PII MASKED and PHI MASKED.
A `client_channel` policy row **only caps** the result. Caps also restrict break-glass and feature grants; a channel-only FULL row cannot elevate a default MASKED tier. This makes the channel a ceiling regardless of how access was granted. The admin shortcut applies only to this QBit's own tables.
The four policy guardrail settings combine most restrictively across applicable rows. Positive numeric limits combine by minimum; zero disables a control only when no matching row enables it. `min_entropy_enabled` is enabled if any matching row enables it. When no policy matches, configuration defaults apply.
- `exact_match_per_hour`: 20 searches per user/table/field, including MATCH, NO_MATCH and REFUSED rows. The next request is refused with `Exact-match search limit reached for [field]. Try again later or ask an administrator.`
- `miss_alert_consecutive`: 5 consecutive NO_MATCH outcomes, including across hour boundaries; other outcomes reset the streak.
- `distinct_value_alert`: more than 15 HMAC fingerprints per hour flags the row.
- `min_entropy_enabled`: disables exact match for boolean, PVS/enum, DATE/DATE_TIME, integer year names/labels, state/region/province/zip3 names/labels, and strings with maxLength under 6. Explicit classification-row true/false takes precedence over detection; disabling the control bypasses both.
Database history plus unflushed rows and local in-flight reservations prevent local concurrent overspending. Failed queries/dropped audit rows retain conservative reservations for an hour. Database history is shared, but another node's pending queue/reservations are not: this is not an atomic distributed quota. Hosts requiring that must serialize exact-search preflight across nodes.
Policy/break-glass caches are bounded per instance and keyed by relevant user permissions and policy version; expiry is rechecked at use time. Fingerprints hash user, held permissions mentioned in any policy or either table feature, policy/classification versions, active break-glass ids, and channel. Unrelated records/permissions and reveal grants do not change them. Multi-node hosts must propagate configuration/break-glass changes to `SecuredDataRuntime.invalidatePolicies()` / `invalidateClassifications()` on other nodes; local customizers only invalidate their own instance.
## Logging and lifecycle
`SecuredDataAccessLogger` captures identifiers, counts, normalized channel and outcomes, never field/search values. VIEW includes only configured classifications (PII/PHI by default), at most 50 primary keys encoded as JSON, and one row per table/action. Exact searches produce one row per affected field with MATCH/NO_MATCH/REFUSED; DENIED events become FILTER_DENIED. Break-glass id and refusal reason are recorded when available.
The extra `search_value_fingerprints` TEXT column persists JSON arrays of HMAC-SHA256 fingerprints for distinct-value detection after restarts and across nodes. Use a shared secret key supplier on multi-node deployments; the default is a random per-JVM key, so historical fingerprints from another JVM/key cannot be compared reliably.
One daemon worker drains a bounded queue as a system session. Enqueueing does not wait for persistence. Full queues and failed writes warn without values and increment `getDroppedCount()`. The app can expose this counter later. Explicitly call the instance's `SecuredDataRuntime.close()` at host shutdown; a JVM shutdown hook also drains accepted rows unless `withRegisterShutdownHook(false)` turns it off (hosts that close the runtime themselves). Neither the worker nor the hook holds the instance strongly: an instance the host drops (tests, reloads) can be collected, and its worker stops at the next tick. An abrupt process crash can lose queued rows.
Flagged rows raise the optional alerts producer signal `secured_data.enumeration`, deduplicated by `secured_data.enumeration:<user>:<table>.<field>:<UTC date>`. RC.5 already provides `QAlerts`; without an installed sink it logs the signal. Its catalog validator rejects underscores in type names, so this mandated type is sent through the producer without catalog registration until that core restriction is reconciled. No values or fingerprints are sent to the alert sink.
## Reveal and break-glass
`new RevealService().options(tableName, fieldName)` describes the reason choices, free-text rules, duration and remaining allowance. `reveal(tableName, primaryKey, fieldName, reason, reasonText)` performs an explicitly permission-enforced Get as the caller before a system read. It returns `value`, core-formatted `displayValue`, `expiresAt`, and `revealId`. Grants and their linked audit rows are system-written; no raw value is copied into the access log.
The effective tier must be REVEALABLE. The chosen granting policy is the row with the highest daily cap (null means unlimited), then the longest duration, then lowest lexical id. Channel-only cap rows never grant reveal settings. A feature/default-based REVEALABLE tier with no granting row uses five minutes and an allowed free-text reason. Limits count grants across all tables for the user in the UTC day. Reason and reason text each have a 500-character limit. A required explanation is accepted alongside a listed reason even if arbitrary free-text reasons are disabled. Capped requests record `REVEAL_DENIED/CAPPED` and raise `secured_data.reveal_capped`; assistant-channel requests record `REVEAL_DENIED/DENIED` and return 403.
Grant lookups cache identifiers and expiration times for at most five seconds per user, check expiration on each use, and invalidate immediately after a new grant. The service serializes cap-check/write and duplicate-session checks within one runtime. Multi-replica deployments must coordinate these operations externally for a globally atomic cap and single active session; the in-process lock is not a distributed lock.
`secured_data_break_glass_start` is a table action, labeled **Start break-glass**, requiring `secured_data_break_glass_start.hasAccess`. Its reason must contain 20-500 characters; duration defaults to 60 minutes and accepts 1-480. Table scope is optional and classification ceiling defaults to PHI. `secured_data_break_glass_end` is a record action with the standard streamed ETL Review/Result screens; ownership/admin authorization is checked at review and again at execution. Neither action is a navigation entry. The current admin-only activity table permissions remain in place; an owner can invoke End using the session id returned by status without obtaining admin access to activity tables.
Start raises `secured_data.break_glass` at core's `HIGH` mapping (`AlertSeverity.ERROR`). End and expiry resolve the same per-session alert. Bell delivery targets holders of `secured_data.admin` through core's optional notification SPI #1125. RC.5 and the installed secure-data snapshot lack that SPI: a clear warning is logged. TODO: switch to a direct notification call when #1125 ships and verify against a notification-capable core.
Configure `.withSchedulerName("yourScheduler")` to register the hidden `secured_data_break_glass_expire` process every minute. Without a scheduler, the host must invoke that process itself. Tier resolution checks expiration independently, so delayed maintenance cannot prolong access. `new StatusService().status()` returns the caller's active session or null, grants used today, and the allowance for each readable REVEALABLE field.
## UI API extension
The `secured-data` extension is version **1.1.0**:
- `POST /qqq/v1/qbit/secured-data/reveal`
- `GET /qqq/v1/qbit/secured-data/reveal/options?tableName=&fieldName=`
- `GET /qqq/v1/qbit/secured-data/status`
The producer registers the UI API extension directly. The committed contract is [secured-data.yaml](src/main/resources/openapi/qqq-v1/secured-data.yaml); regenerate intentionally with `-Dqqq.uiApi.regenerate=true`. Contract and HTTP tests run in the normal build.
The classification field picker uses CUSTOM possible values, reading `table_name` or `tableName` from the form's other values. It returns only the selected readable table's labeled fields, returns none without a selection, and rejects an unknown table. TODO: add `dependsOn("table_name")` when the core metadata API supports it.
## Configuration
| Fluent option | Default / purpose |
|---|---|
| `withBackendName(String)` | `memory`; host backend name |
| `withTableMetaDataCustomizer(...)` | Optional host storage/security customization |
| `withDefaultTier(DataClassification, FieldAccessTier)` | CONFIDENTIAL FULL; PII/PHI MASKED |
| `withDefaultTierByClassification(Map<String,String>)` | Compatibility bulk setter; omitted restricted types fail closed |
| `withNameHeuristics(boolean)` | false; opt-in PII for ssn/dob/date_of_birth/email/phone, PHI for diagnosis* |
| `withViewLogClassifications(Set<DataClassification>)` | PII, PHI |
| `withFingerprintKeySupplier(Supplier<byte[]>)` | Random 32-byte JVM key; supplied key must be at least 16 bytes |
| `withAccessLogQueueCapacity(int)` | 4096 rows |
| `withAccessLogFlushIntervalMillis(long)` | 250 ms |
| `withRegisterShutdownHook(boolean)` | true; JVM hook that drains the access log at shutdown |
| `withExactMatchPerHour(int)` | 20; zero disables |
| `withMissAlertConsecutive(int)` | 5; zero disables |
| `withDistinctValueAlert(int)` | 15; zero disables |
| `withMinEntropyEnabled(boolean)` | true |
| `withSchedulerName(String)` | Host scheduler for the 60-second break-glass expiry job; null means host-managed invocation |
| `withBackupRetentionDays(Integer)` | Unknown; physical backup window in days, used for the retention warning (must not be negative) |
| `withArchiveSink(ArchiveSinkInterface)` | None; required before any `ARCHIVE_THEN_WIPE` rule can run |
| `withClock(Clock)` | UTC system clock; injectable for deterministic expiry tests |
Configure before producing the QBit; runtime infrastructure and HMAC keys are captured at startup. Classification/policy/break-glass lookups always run as system, with a thread-local recursion guard. No reveal values are cached.
## Build and tests
```powershell
./mvnw.cmd -q -o clean verify
```
The suite uses memory and H2. `CoreFieldAccessIT` runs unconditionally through Surefire and exercises Query/Get, policy tiers, masking, exact matches and outcomes, sorting denial, rate/entropy refusal and the assistant wall. `RetentionProofTest` also runs unconditionally. Checkstyle and JaCoCo gates remain 70% instructions / 90% classes.
## Anomaly monitoring (phase e)
Application REVEAL and queued EXACT_MATCH_FILTER audit writes evaluate persisted access-log counters as system, counting the current row once. Successful reveals flag above three times the user's median across the previous 30 complete UTC days (including zero days), with at least ten reveals today. More than 20 distinct `(table, record)` pairs within the last hour also flag. More than 50 exact-match attempts on one restricted field within the last hour flag independently of enumeration controls. The default rate limit of 20 usually refuses searches first; refused attempts still count.
Break-glass starts and capped reveal refusals also flag. Each raises `secured_data.anomaly` with dedup key `secured_data.anomaly:<user>:<UTC day>` through the optional alerts SPI. Existing enumeration, break-glass and cap alerts remain intact. No qbit-alerts dependency is required; flags remain visible without a sink. The sink owns deduplication. End and expiry events remain in the break-glass activity list rather than representing new session anomalies.
Thresholds are positive integers:
| Config method | Default |
| --- | ---: |
| `withAnomalyDailyMedianMultiplier` | 3 |
| `withAnomalyMinimumDailyReveals` | 10 |
| `withAnomalyDistinctRecordsPerHour` | 20 |
| `withAnomalyExactMatchesPerHour` | 50 |
Medians use 30 scalar indexed counts on first qualifying use per user/day, cached for at most 2,048 users per instance and cleared at UTC day rollover. Today's and hourly counts read persisted history; distinct-record queries return at most the threshold plus one grouped rows. Counter evaluation and insertion serialize within an instance; other nodes become visible through persisted rows. This monitors activity rather than providing a distributed enforcement lock. Historical backfills after caching take effect next UTC day or after runtime restart. Existing deployments should apply the two new access-log indexes in `docs/ddl/secured-data-postgres.sql`.
## Secured Data home
The single Secured Data app owns one full-width HTML home widget, following qbit-webhooks' data/view/renderer pattern. It reads as the viewer with normal permissions and record security:
- Six KPIs: reveals today, unexpired grants, active and unexpired break-glass sessions, anomaly flags and denied filters over seven days, and this runtime's access-log queue drops since startup. Denied filters include refused exact-match attempts.
- Six full-width lists: recent reveals (10), top revealers and reasons (30-day backend aggregates, top 10), recent break-glass starts/ends/expiries (10), policy/classification changes (10), and seven-day anomaly flags (10).
- Each list has a filtered access-log `View all` link without the widget's ten-row limit. Configure the host table route with `withTablePathPrefix("/app")`.
Lists order newest first with id tie-breaking; rankings order by count descending then group alphabetically. Narrow screens use wrapping labelled card rows. Dates stay compact on one line, text is ellipsized with full tooltips, and colors inherit the host with `--qqq-grid-border` for borders. There are no inner scrollers. Empty lists explicitly say there is no activity.
Navigation contains Classification, Policy, Access log, Break-glass sessions and Retention rules. Reveal grants remain a child list. A warning line appears under the KPIs for each active retention rule whose known backup retention exceeds its `age_days`. Successful policy/classification inserts, updates and deletes produce value-free change rows with actor, record id and action in the source transaction. No configuration values are copied into these rows.
### Local screenshot harness
`SecuredDataDemoServer` is a test-only memory-backed Next demo adapted from qbit-webhooks. It seeds synthetic activity for all six lists, including long reasons, sessions and anomaly flags. It uses a demo administrator and must only be run locally. After a test run, launch in PowerShell:
```powershell
[xml]$report = Get-Content target/surefire-reports/TEST-com.kingsrook.qbits.secureddata.SecuredDataHomeTest.xml
$demoClasspath = ($report.testsuite.properties.property | Where-Object name -eq 'java.class.path').value
java -cp $demoClasspath com.kingsrook.qbits.secureddata.SecuredDataDemoServer
```
Open `http://localhost:8098/app/secured_data`. Stop the demo before `clean` on Windows. Capture light and dark at 390, 768, 1024, 1440, 1920 and 3200 px under `target/screenshots/`. This session started the server, but screenshot capture was blocked by Chromium `spawn EPERM` in the sandbox and two connected-browser timeouts. The coordinator still needs to capture and inspect the twelve images through Next.
## Retention and wipe
Spec: SECURED-DATA sections 13 to 15. Retention rules live in the **Retention rules** table (permission `secured_data.retention`). Each rule names a host table, the fields to wipe, an age field, `age_days`, a batch size, an action and an audit treatment.
Actions:
- `NULL_WITH_MARKER`: set the fields to null (fields must be nullable) and record a wipe marker.
- `OVERWRITE_TEXT`: replace with the rule's text (STRING/TEXT fields only; the text must fit the field).
- `KEYED_HASH`: replace with an instance-key HMAC token (`hmac:v1:...`) that the host can still verify. Refused for low-entropy fields and for fields shorter than 200 characters.
- `DELETE_RECORD`: delete the record, with markers for every field.
- `ARCHIVE_THEN_WIPE`: write the field values as AES-GCM encrypted JSON through the host `ArchiveSinkInterface` (an adapter to object storage, ideally a WORM bucket), then null them. The wipe is refused if no sink is configured.
Audit treatment: `SHREDDABLE` (default) gives each audited record-field its own data key (DEK), so deleting the key makes the audit tokens unverifiable (crypto-shredding). `CHANGED_ONLY` stores only `changed:v1` in audits and never creates a DEK.
Mechanics:
- Eligibility is evaluated in the database and returns record ids only; values are read only for `KEYED_HASH` and `ARCHIVE_THEN_WIPE`.
- Each batch is one transaction in a rollback-capable backend: field updates, wipe markers, audit-key deletions, audit details, the access-log `WIPE` rows and the run row commit or roll back together. Cross-backend or non-transactional deployments are refused before any write.
- Legal holds (per record, or per table with a blank record id) skip records; age and holds are re-checked inside the batch.
- **Run wipe now** is a standard Review/Result process with counts only; the hidden `secured_data_wipe_daily` process runs daily from the host scheduler (`withSchedulerName`). Neither places a host value in a message, log, run row or marker.
- DEKs are wrapped with AES-GCM under the KEK, with the table, field and record id as additional authenticated data. A wrong KEK or a wrong binding fails the unwrap. The wrapped-key table is hidden, locked, not exportable and audited at level NONE.
### Backups are a residual risk (section 14.3)
A wipe removes the value and its audit key from the live database only. Database backups taken before the wipe still contain the value and the wrapped DEK, and remain recoverable until they expire. Set `withBackupRetentionDays(...)`; when it exceeds a rule's `age_days` the rule and the app home warn that wiped values stay recoverable from backups. Unknown retention is reported as unknown, not as safe. Backups that must be purged earlier have to be handled outside this qbit.
### Keys and environment
| Variable | Purpose |
|---|---|
| `QQQ_AUDIT_HMAC_KEY` | Base64 32-byte instance key for `hmac:v1:` audit tokens and `KEYED_HASH`. Without it audits fall back to `changed:v1`. |
| `QQQ_AUDIT_HMAC_KEY_ID` | Optional id embedded in tokens, default `env1`; change it when rotating the key. |
| `QQQ_SECURED_DATA_KEK` | Base64 32-byte key-encryption key that wraps the per-record DEKs and archive objects. Required for `SHREDDABLE` and archive; a missing or invalid KEK fails closed. |
Each is read from a JVM property or the environment and is never logged. Losing the KEK makes every existing DEK unrecoverable, so protect and back it up separately from the database.
### Core hooks
Audit-value protection, wipe audit flags and shredded audit events use the core #1182 APIs directly. Core tokenises ordinary audits and writes value-free wipe details. The section 15 canary and wipe-proof tests (`RetentionProofTest`) are mandatory in the normal build.
## Remaining phases
Field-access and audit-token stand-ins and reflective bridges have been removed. The coordinator will replace the temporary snapshot pin with the first QQQ RC containing #1172 and #1182.
## License
See [LICENSE](LICENSE) and [NOTICE](NOTICE).
[Apache License 2.0](LICENSE). Copyright Kingsrook, LLC.
+4
View File
@@ -0,0 +1,4 @@
# Security Policy
* See Project [Security Policy](https://github.com/QRun-IO/qqq/blob/develop/SECURITY.md)
* Report suspected vulnerabilities privately through GitHub private vulnerability reporting on this repository, never in a public issue.
+246
View File
@@ -0,0 +1,246 @@
<?xml version="1.0"?>
<!--
~ QQQ - Low-code Application Framework for Engineers.
~ Copyright (C) 2021-2022. Kingsrook, LLC
~ 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
~ contact@kingsrook.com
~ https://github.com/Kingsrook/
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ https://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://checkstyle.sourceforge.net/dtds/configuration_1_3.dtd">
<!--
See reference at https://checkstyle.sourceforge.io/checks.html
-->
<module name = "Checker">
<property name="charset" value="UTF-8"/>
<property name="severity" value="warning"/>
<property name="fileExtensions" value="java, properties, xml"/>
<!-- Checks for whitespace -->
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
<module name="FileTabCharacter">
<property name="eachLine" value="true"/>
</module>
<!--
<module name="LineLength">
<property name="max" value="100"/>
<property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
</module>
-->
<module name="TreeWalker">
<module name="SuppressWarningsHolder"/>
<module name="OuterTypeFilename"/>
<module name="IllegalTokenText">
<property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/>
<property name="format" value="\\u00(09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/>
<property name="message" value="Consider using special escape sequence instead of octal value or Unicode escaped value."/>
</module>
<module name="AvoidEscapedUnicodeCharacters">
<property name="allowEscapesForControlCharacters" value="true"/>
<property name="allowByTailComment" value="true"/>
<property name="allowNonPrintableEscapes" value="true"/>
</module>
<module name="AvoidStarImport"/>
<module name="OneTopLevelClass"/>
<module name="NoLineWrap"/>
<module name="EmptyBlock">
<property name="option" value="TEXT"/>
<property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/>
</module>
<module name="NeedBraces"/>
<module name="LeftCurly">
<property name="option" value="nl"/>
</module>
<module name="RightCurly">
<property name="option" value="alone"/>
</module>
<!--
<module name="WhitespaceAround">
<property name="allowEmptyConstructors" value="true"/>
<property name="allowEmptyMethods" value="true"/>
<property name="allowEmptyTypes" value="true"/>
<property name="allowEmptyLoops" value="true"/>
<message key="ws.notFollowed"
value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may only be represented as '{}' when not part of a multi-block statement (4.1.3)"/>
<message key="ws.notPreceded"
value="WhitespaceAround: ''{0}'' is not preceded with whitespace."/>
</module>
-->
<module name="OneStatementPerLine"/>
<module name="MultipleVariableDeclarations"/>
<module name="ArrayTypeStyle"/>
<module name="MissingSwitchDefault"/>
<module name="FallThrough"/>
<module name="UpperEll"/>
<module name="ModifierOrder"/>
<module name="EmptyLineSeparator">
<property name="allowNoEmptyLineBetweenFields" value="true"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapDot"/>
<property name="tokens" value="DOT"/>
<property name="option" value="nl"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapComma"/>
<property name="tokens" value="COMMA"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<!-- ELLIPSIS is EOL until https://github.com/google/styleguide/issues/258 -->
<property name="id" value="SeparatorWrapEllipsis"/>
<property name="tokens" value="ELLIPSIS"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<!-- ARRAY_DECLARATOR is EOL until https://github.com/google/styleguide/issues/259 -->
<property name="id" value="SeparatorWrapArrayDeclarator"/>
<property name="tokens" value="ARRAY_DECLARATOR"/>
<property name="option" value="EOL"/>
</module>
<module name="SeparatorWrap">
<property name="id" value="SeparatorWrapMethodRef"/>
<property name="tokens" value="METHOD_REF"/>
<property name="option" value="nl"/>
</module>
<module name="PackageName">
<property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/>
<message key="name.invalidPattern"
value="Package name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="TypeName">
<message key="name.invalidPattern"
value="Type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="MemberName">
<property name="format" value="^[_a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Member name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="ParameterName">
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Parameter name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="CatchParameterName">
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Catch parameter name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="LocalVariableName">
<property name="tokens" value="VARIABLE_DEF"/>
<property name="format" value="^[a-zA-Z]([a-z0-9A-Z][a-zA-Z0-9]*)?$"/>
<message key="name.invalidPattern" value="Local variable name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="ClassTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Class type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="MethodTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Method type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="InterfaceTypeParameterName">
<property name="format" value="(^[A-Z][0-9]?)$|([A-Z][_A-Z0-9]*$)"/>
<message key="name.invalidPattern" value="Interface type name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="NoFinalizer"/>
<module name="GenericWhitespace">
<message key="ws.followed" value="GenericWhitespace ''{0}'' is followed by whitespace."/>
<message key="ws.preceded" value="GenericWhitespace ''{0}'' is preceded with whitespace."/>
<message key="ws.illegalFollow" value="GenericWhitespace ''{0}'' should followed by whitespace."/>
<message key="ws.notPreceded" value="GenericWhitespace ''{0}'' is not preceded with whitespace."/>
</module>
<module name="Indentation">
<property name="basicOffset" value="3"/>
<property name="braceAdjustment" value="0"/>
<property name="caseIndent" value="3"/>
<property name="throwsIndent" value="6"/>
<property name="lineWrappingIndentation" value="3"/>
<property name="arrayInitIndent" value="6"/>
</module>
<!--
<module name="AbbreviationAsWordInName">
<property name="ignoreFinal" value="false"/>
<property name="allowedAbbreviationLength" value="1"/>
</module>
<module name="OverloadMethodsDeclarationOrder"/>
<module name="VariableDeclarationUsageDistance"/>
<module name="CustomImportOrder">
<property name="sortImportsInGroupAlphabetically" value="true"/>
<property name="separateLineBetweenGroups" value="true"/>
<property name="customImportOrderRules" value="STATIC###THIRD_PARTY_PACKAGE"/>
</module>
-->
<module name="CustomImportOrder">
<property name="customImportOrderRules" value="SPECIAL_IMPORTS###STANDARD_JAVA_PACKAGE###THIRD_PARTY_PACKAGE###STATIC"/>
<property name="specialImportsRegExp" value="^javax\."/>
<property name="standardPackageRegExp" value="^java\."/>
<property name="sortImportsInGroupAlphabetically" value="true"/>
<property name="separateLineBetweenGroups" value="false"/>
</module>
<module name="MethodParamPad"/>
<module name="NoWhitespaceBefore">
<property name="tokens" value="COMMA, SEMI, POST_INC, POST_DEC, DOT, ELLIPSIS, METHOD_REF"/>
<property name="allowLineBreaks" value="true"/>
</module>
<module name="ParenPad"/>
<module name="OperatorWrap">
<property name="option" value="NL"/>
<property name="tokens" value="BAND, BOR, BSR, BXOR, DIV, EQUAL, GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD, NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR, METHOD_REF "/>
</module>
<module name="AnnotationLocation">
<property name="id" value="AnnotationLocationMostCases"/>
<property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/>
</module>
<module name="AnnotationLocation">
<property name="id" value="AnnotationLocationVariables"/>
<property name="tokens" value="VARIABLE_DEF"/>
<property name="allowSamelineMultipleAnnotations" value="true"/>
</module>
<module name="JavadocMethod">
<property name="allowMissingParamTags" value="true"/>
<property name="allowMissingReturnTag" value="true"/>
<property name="allowedAnnotations" value="Override, Test"/>
</module>
<module name="MissingJavadocMethod">
<property name="scope" value="private"/>
</module>
<module name="MissingJavadocType">
<property name="scope" value="private"/>
</module>
<module name="MethodName">
<property name="format" value="^[a-z][a-zA-Z0-9_]*$"/>
<message key="name.invalidPattern"
value="Method name ''{0}'' must match pattern ''{1}''."/>
</module>
<module name="EmptyCatchBlock">
<property name="exceptionVariableName" value="expected"/>
</module>
<module name="CommentsIndentation"/>
<!-- this would be nice, but requires the the @inheritDoc javadoc tag... -->
<module name="MissingOverride"/>
</module>
<module name="Header">
<property name="headerFile" value="checkstyle/license.txt"/>
<property name="fileExtensions" value="java"/>
<property name="ignoreLines" value="3"/>
</module>
<module name="SuppressWarningsFilter"/>
</module>
+19
View File
@@ -0,0 +1,19 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-28
View File
@@ -1,28 +0,0 @@
# Getting Started
Targets Java 21, Maven 3.8 or later, and QQQ 4.0.0. Maven resolves the framework dependencies from Maven Central.
1. Choose **Use this template** on GitHub, then clone the repository you created.
2. In `pom.xml`, set your own `groupId`, `artifactId`, name, description, and version.
3. Use your IDE's package refactoring to rename `com.kingsrook.qbits.example` throughout `src/`. Rename the `Example*` classes and update their imports and references with the IDE's rename refactoring.
4. Update the QBit producer's `GROUP_ID`, `ARTIFACT_ID`, and `VERSION` constants to match your project. Give the metadata names their own stable names before combining the QBit with other examples.
5. Build the generated project:
```bash
mvn clean verify
```
The repository provides example Java sources; customization uses normal package/class refactoring. Keep example coordinates unpublished. After customizing, add tests for your QBit's behavior before setting up its publishing workflow.
## Register with a host application
The following uses the original example names; substitute the names chosen above:
```java
new ExampleExtensionQBitProducer()
.withConfig(new ExampleExtensionQBitConfig()
.withTargetTableName("order"))
.produce(qInstance, "my-extension");
```
Register the target table before producing the extension, then implement its customizer. See [Extension Patterns](01-extension-patterns.md).
-16
View File
@@ -1,16 +0,0 @@
# Extension Patterns
[ExampleExtensionQBitProducer](../src/main/java/com/kingsrook/qbits/example/ExampleExtensionQBitProducer.java) registers the QBit identity and calls `applyTableCustomizations` for the configured target. That method is the extension point: look up the table, validate that it exists, and call your customized [ExampleTableCustomizer](../src/main/java/com/kingsrook/qbits/example/customizers/ExampleTableCustomizer.java).
```java
QTableMetaData table = qInstance.getTable(tableName);
if(table == null)
{
throw new QException("Target table not found: " + tableName);
}
new ExampleTableCustomizer().customize(table);
```
When implementing this method, add `throws QException` to its declaration. Run the config's `validate(QInstance, List<String>)` before registering metadata, and report validation errors rather than silently ignoring a missing target. The uncustomized template contains placeholders for this behavior.
For record hooks, use QQQ's existing table customizer interfaces or `AbstractPreInsertCustomizer`. Its `apply(List<QRecord>)` method returns the records for the next stage; register the customizer through the table's `TableCustomizers.PRE_INSERT_RECORD` role. Check the actual QQQ interface before adding another hook. Add tests proving that the intended table changes and invalid configurations are handled.
+1
View File
@@ -0,0 +1 @@
The design lives in QRun-IO/qqq docs/SECURED-DATA.md; this repo implements §5–§8.
+140
View File
@@ -0,0 +1,140 @@
-- PostgreSQL 15+ (NULLS NOT DISTINCT makes the table-wide classification unique).
-- Extra value-free fingerprint column supports enumeration detection across restarts/nodes.
CREATE TABLE secured_data_classification (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
table_name VARCHAR(255) NOT NULL,
field_name VARCHAR(255),
classification VARCHAR(255) NOT NULL,
mask_style VARCHAR(255),
low_cardinality BOOLEAN,
notes TEXT,
create_date TIMESTAMP WITH TIME ZONE,
modify_date TIMESTAMP WITH TIME ZONE,
modified_by VARCHAR(255),
CONSTRAINT secured_data_classification_target UNIQUE NULLS NOT DISTINCT (table_name, field_name)
);
CREATE TABLE secured_data_policy (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name VARCHAR(255) NOT NULL,
permission_name VARCHAR(255) NOT NULL,
classification VARCHAR(255) NOT NULL,
table_name VARCHAR(255),
tier VARCHAR(255) NOT NULL,
reveal_minutes INTEGER DEFAULT 5,
daily_reveal_cap INTEGER,
reason_options TEXT,
allow_free_text_reason BOOLEAN DEFAULT true,
require_free_text BOOLEAN DEFAULT false,
client_channel VARCHAR(255),
exact_match_per_hour INTEGER DEFAULT 20,
miss_alert_consecutive INTEGER DEFAULT 5,
distinct_value_alert INTEGER DEFAULT 15,
min_entropy_enabled BOOLEAN DEFAULT true,
is_active BOOLEAN DEFAULT true,
create_date TIMESTAMP WITH TIME ZONE,
modify_date TIMESTAMP WITH TIME ZONE,
modified_by VARCHAR(255),
CHECK (reveal_minutes BETWEEN 1 AND 60),
CHECK (exact_match_per_hour >= 0 AND miss_alert_consecutive >= 0 AND distinct_value_alert >= 0)
);
CREATE TABLE secured_data_reveal_grant (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
user_name VARCHAR(255),
table_name VARCHAR(255) NOT NULL,
record_id VARCHAR(255) NOT NULL,
field_name VARCHAR(255) NOT NULL,
reason VARCHAR(500),
reason_text TEXT,
granted_at TIMESTAMP WITH TIME ZONE NOT NULL,
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
access_log_id INTEGER NOT NULL
);
CREATE TABLE secured_data_break_glass (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
user_name VARCHAR(255),
reason VARCHAR(500),
reason_text TEXT,
classification_ceiling VARCHAR(255) DEFAULT 'PHI',
table_name VARCHAR(255),
started_at TIMESTAMP WITH TIME ZONE NOT NULL,
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
ended_at TIMESTAMP WITH TIME ZONE,
ended_by VARCHAR(255),
status VARCHAR(255) DEFAULT 'ACTIVE'
);
CREATE TABLE secured_data_access_log (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
event_type VARCHAR(255) NOT NULL,
user_id VARCHAR(255),
user_name VARCHAR(255),
table_name VARCHAR(255) NOT NULL,
record_id TEXT,
field_name TEXT,
record_count INTEGER,
action_name VARCHAR(255),
reason VARCHAR(500),
reason_text TEXT,
client_channel VARCHAR(255),
outcome VARCHAR(255),
anomaly_flag BOOLEAN DEFAULT false,
break_glass_id INTEGER,
created_at TIMESTAMP WITH TIME ZONE NOT NULL,
search_value_fingerprints TEXT,
rule_id INTEGER
);
CREATE INDEX secured_data_policy_lookup ON secured_data_policy (is_active, permission_name, classification, table_name);
CREATE INDEX secured_data_break_glass_lookup ON secured_data_break_glass (user_id, status, expires_at);
CREATE INDEX secured_data_reveal_lookup ON secured_data_reveal_grant (user_id, table_name, field_name, record_id, expires_at);
CREATE INDEX secured_data_access_lookup ON secured_data_access_log (user_id, table_name, field_name, event_type, created_at);
-- Phase (e): bounded user counters and home time windows.
CREATE INDEX secured_data_anomaly_lookup ON secured_data_access_log (user_id, event_type, created_at);
CREATE INDEX secured_data_home_lookup ON secured_data_access_log (event_type, created_at);
-- Phase (f): all policy targets and audit tables must share this transactional backend.
CREATE TABLE secured_data_retention_rule (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
name VARCHAR(255) NOT NULL, table_name VARCHAR(255) NOT NULL, field_names TEXT NOT NULL,
age_field_name VARCHAR(255) NOT NULL, age_days INTEGER NOT NULL CHECK (age_days >= 1),
action VARCHAR(40) DEFAULT 'NULL_WITH_MARKER' NOT NULL CHECK (action IN ('NULL_WITH_MARKER','OVERWRITE_TEXT','KEYED_HASH','DELETE_RECORD','ARCHIVE_THEN_WIPE')),
overwrite_text TEXT DEFAULT '[wiped per policy]',
audit_treatment VARCHAR(30) DEFAULT 'SHREDDABLE' NOT NULL CHECK (audit_treatment IN ('SHREDDABLE','CHANGED_ONLY')),
archive_storage_name VARCHAR(255), batch_size INTEGER DEFAULT 500 NOT NULL CHECK (batch_size >= 1),
is_active BOOLEAN DEFAULT true, last_run_at TIMESTAMP WITH TIME ZONE, last_run_count INTEGER,
create_date TIMESTAMP WITH TIME ZONE, modify_date TIMESTAMP WITH TIME ZONE, modified_by VARCHAR(255)
);
CREATE TABLE secured_data_wipe_run (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id),
started_at TIMESTAMP WITH TIME ZONE NOT NULL, finished_at TIMESTAMP WITH TIME ZONE, status VARCHAR(30),
eligible_count INTEGER, held_count INTEGER, wiped_count INTEGER, error_count INTEGER,
record_ids VARCHAR(4000), message VARCHAR(500)
);
CREATE TABLE secured_data_wipe_marker (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255) NOT NULL,
field_name VARCHAR(255) NOT NULL, rule_id INTEGER NOT NULL REFERENCES secured_data_retention_rule(id), action VARCHAR(40) NOT NULL,
wiped_at TIMESTAMP WITH TIME ZONE NOT NULL, run_id INTEGER NOT NULL REFERENCES secured_data_wipe_run(id),
UNIQUE (table_name, record_id, field_name)
);
CREATE TABLE secured_data_legal_hold (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL, record_id VARCHAR(255), reason TEXT NOT NULL,
placed_by VARCHAR(255), placed_at TIMESTAMP WITH TIME ZONE, released_by VARCHAR(255), released_at TIMESTAMP WITH TIME ZONE,
status VARCHAR(20) DEFAULT 'ACTIVE' NOT NULL CHECK (status IN ('ACTIVE','RELEASED'))
);
-- Mutable key material: never register this table in ESB, WORM or long-term logical backups.
CREATE TABLE secured_data_audit_key (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(255) NOT NULL,
record_id VARCHAR(255) NOT NULL, field_name VARCHAR(255) NOT NULL, wrapped_key TEXT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL, UNIQUE (table_name, record_id, field_name)
);
CREATE INDEX secured_data_hold_lookup ON secured_data_legal_hold (table_name, status, record_id);
CREATE INDEX secured_data_retention_lookup ON secured_data_retention_rule (table_name, is_active);
CREATE INDEX secured_data_wipe_rule_lookup ON secured_data_wipe_marker (rule_id);
Vendored Executable
+295
View File
@@ -0,0 +1,295 @@
#!/bin/sh
# ----------------------------------------------------------------------------
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# ----------------------------------------------------------------------------
# ----------------------------------------------------------------------------
# Apache Maven Wrapper startup batch script, version 3.3.4
#
# Optional ENV vars
# -----------------
# JAVA_HOME - location of a JDK home dir, required when download maven via java source
# MVNW_REPOURL - repo url base for downloading maven distribution
# MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
# MVNW_VERBOSE - true: enable verbose log; debug: trace the mvnw script; others: silence the output
# ----------------------------------------------------------------------------
set -euf
[ "${MVNW_VERBOSE-}" != debug ] || set -x
# OS specific support.
native_path() { printf %s\\n "$1"; }
case "$(uname)" in
CYGWIN* | MINGW*)
[ -z "${JAVA_HOME-}" ] || JAVA_HOME="$(cygpath --unix "$JAVA_HOME")"
native_path() { cygpath --path --windows "$1"; }
;;
esac
# set JAVACMD and JAVACCMD
set_java_home() {
# For Cygwin and MinGW, ensure paths are in Unix format before anything is touched
if [ -n "${JAVA_HOME-}" ]; then
if [ -x "$JAVA_HOME/jre/sh/java" ]; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
JAVACCMD="$JAVA_HOME/jre/sh/javac"
else
JAVACMD="$JAVA_HOME/bin/java"
JAVACCMD="$JAVA_HOME/bin/javac"
if [ ! -x "$JAVACMD" ] || [ ! -x "$JAVACCMD" ]; then
echo "The JAVA_HOME environment variable is not defined correctly, so mvnw cannot run." >&2
echo "JAVA_HOME is set to \"$JAVA_HOME\", but \"\$JAVA_HOME/bin/java\" or \"\$JAVA_HOME/bin/javac\" does not exist." >&2
return 1
fi
fi
else
JAVACMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v java
)" || :
JAVACCMD="$(
'set' +e
'unset' -f command 2>/dev/null
'command' -v javac
)" || :
if [ ! -x "${JAVACMD-}" ] || [ ! -x "${JAVACCMD-}" ]; then
echo "The java/javac command does not exist in PATH nor is JAVA_HOME set, so mvnw cannot run." >&2
return 1
fi
fi
}
# hash string like Java String::hashCode
hash_string() {
str="${1:-}" h=0
while [ -n "$str" ]; do
char="${str%"${str#?}"}"
h=$(((h * 31 + $(LC_CTYPE=C printf %d "'$char")) % 4294967296))
str="${str#?}"
done
printf %x\\n $h
}
verbose() { :; }
[ "${MVNW_VERBOSE-}" != true ] || verbose() { printf %s\\n "${1-}"; }
die() {
printf %s\\n "$1" >&2
exit 1
}
trim() {
# MWRAPPER-139:
# Trims trailing and leading whitespace, carriage returns, tabs, and linefeeds.
# Needed for removing poorly interpreted newline sequences when running in more
# exotic environments such as mingw bash on Windows.
printf "%s" "${1}" | tr -d '[:space:]'
}
scriptDir="$(dirname "$0")"
scriptName="$(basename "$0")"
# parse distributionUrl and optional distributionSha256Sum, requires .mvn/wrapper/maven-wrapper.properties
while IFS="=" read -r key value; do
case "${key-}" in
distributionUrl) distributionUrl=$(trim "${value-}") ;;
distributionSha256Sum) distributionSha256Sum=$(trim "${value-}") ;;
esac
done <"$scriptDir/.mvn/wrapper/maven-wrapper.properties"
[ -n "${distributionUrl-}" ] || die "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties"
case "${distributionUrl##*/}" in
maven-mvnd-*bin.*)
MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/
case "${PROCESSOR_ARCHITECTURE-}${PROCESSOR_ARCHITEW6432-}:$(uname -a)" in
*AMD64:CYGWIN* | *AMD64:MINGW*) distributionPlatform=windows-amd64 ;;
:Darwin*x86_64) distributionPlatform=darwin-amd64 ;;
:Darwin*arm64) distributionPlatform=darwin-aarch64 ;;
:Linux*x86_64*) distributionPlatform=linux-amd64 ;;
*)
echo "Cannot detect native platform for mvnd on $(uname)-$(uname -m), use pure java version" >&2
distributionPlatform=linux-amd64
;;
esac
distributionUrl="${distributionUrl%-bin.*}-$distributionPlatform.zip"
;;
maven-mvnd-*) MVN_CMD=mvnd.sh _MVNW_REPO_PATTERN=/maven/mvnd/ ;;
*) MVN_CMD="mvn${scriptName#mvnw}" _MVNW_REPO_PATTERN=/org/apache/maven/ ;;
esac
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
[ -z "${MVNW_REPOURL-}" ] || distributionUrl="$MVNW_REPOURL$_MVNW_REPO_PATTERN${distributionUrl#*"$_MVNW_REPO_PATTERN"}"
distributionUrlName="${distributionUrl##*/}"
distributionUrlNameMain="${distributionUrlName%.*}"
distributionUrlNameMain="${distributionUrlNameMain%-bin}"
MAVEN_USER_HOME="${MAVEN_USER_HOME:-${HOME}/.m2}"
MAVEN_HOME="${MAVEN_USER_HOME}/wrapper/dists/${distributionUrlNameMain-}/$(hash_string "$distributionUrl")"
exec_maven() {
unset MVNW_VERBOSE MVNW_USERNAME MVNW_PASSWORD MVNW_REPOURL || :
exec "$MAVEN_HOME/bin/$MVN_CMD" "$@" || die "cannot exec $MAVEN_HOME/bin/$MVN_CMD"
}
if [ -d "$MAVEN_HOME" ]; then
verbose "found existing MAVEN_HOME at $MAVEN_HOME"
exec_maven "$@"
fi
case "${distributionUrl-}" in
*?-bin.zip | *?maven-mvnd-?*-?*.zip) ;;
*) die "distributionUrl is not valid, must match *-bin.zip or maven-mvnd-*.zip, but found '${distributionUrl-}'" ;;
esac
# prepare tmp dir
if TMP_DOWNLOAD_DIR="$(mktemp -d)" && [ -d "$TMP_DOWNLOAD_DIR" ]; then
clean() { rm -rf -- "$TMP_DOWNLOAD_DIR"; }
trap clean HUP INT TERM EXIT
else
die "cannot create temp dir"
fi
mkdir -p -- "${MAVEN_HOME%/*}"
# Download and Install Apache Maven
verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
verbose "Downloading from: $distributionUrl"
verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
# select .zip or .tar.gz
if ! command -v unzip >/dev/null; then
distributionUrl="${distributionUrl%.zip}.tar.gz"
distributionUrlName="${distributionUrl##*/}"
fi
# verbose opt
__MVNW_QUIET_WGET=--quiet __MVNW_QUIET_CURL=--silent __MVNW_QUIET_UNZIP=-q __MVNW_QUIET_TAR=''
[ "${MVNW_VERBOSE-}" != true ] || __MVNW_QUIET_WGET='' __MVNW_QUIET_CURL='' __MVNW_QUIET_UNZIP='' __MVNW_QUIET_TAR=v
# normalize http auth
case "${MVNW_PASSWORD:+has-password}" in
'') MVNW_USERNAME='' MVNW_PASSWORD='' ;;
has-password) [ -n "${MVNW_USERNAME-}" ] || MVNW_USERNAME='' MVNW_PASSWORD='' ;;
esac
if [ -z "${MVNW_USERNAME-}" ] && command -v wget >/dev/null; then
verbose "Found wget ... using wget"
wget ${__MVNW_QUIET_WGET:+"$__MVNW_QUIET_WGET"} "$distributionUrl" -O "$TMP_DOWNLOAD_DIR/$distributionUrlName" || die "wget: Failed to fetch $distributionUrl"
elif [ -z "${MVNW_USERNAME-}" ] && command -v curl >/dev/null; then
verbose "Found curl ... using curl"
curl ${__MVNW_QUIET_CURL:+"$__MVNW_QUIET_CURL"} -f -L -o "$TMP_DOWNLOAD_DIR/$distributionUrlName" "$distributionUrl" || die "curl: Failed to fetch $distributionUrl"
elif set_java_home; then
verbose "Falling back to use Java to download"
javaSource="$TMP_DOWNLOAD_DIR/Downloader.java"
targetZip="$TMP_DOWNLOAD_DIR/$distributionUrlName"
cat >"$javaSource" <<-END
public class Downloader extends java.net.Authenticator
{
protected java.net.PasswordAuthentication getPasswordAuthentication()
{
return new java.net.PasswordAuthentication( System.getenv( "MVNW_USERNAME" ), System.getenv( "MVNW_PASSWORD" ).toCharArray() );
}
public static void main( String[] args ) throws Exception
{
setDefault( new Downloader() );
java.nio.file.Files.copy( java.net.URI.create( args[0] ).toURL().openStream(), java.nio.file.Paths.get( args[1] ).toAbsolutePath().normalize() );
}
}
END
# For Cygwin/MinGW, switch paths to Windows format before running javac and java
verbose " - Compiling Downloader.java ..."
"$(native_path "$JAVACCMD")" "$(native_path "$javaSource")" || die "Failed to compile Downloader.java"
verbose " - Running Downloader.java ..."
"$(native_path "$JAVACMD")" -cp "$(native_path "$TMP_DOWNLOAD_DIR")" Downloader "$distributionUrl" "$(native_path "$targetZip")"
fi
# If specified, validate the SHA-256 sum of the Maven distribution zip file
if [ -n "${distributionSha256Sum-}" ]; then
distributionSha256Result=false
if [ "$MVN_CMD" = mvnd.sh ]; then
echo "Checksum validation is not supported for maven-mvnd." >&2
echo "Please disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
elif command -v sha256sum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | sha256sum -c - >/dev/null 2>&1; then
distributionSha256Result=true
fi
elif command -v shasum >/dev/null; then
if echo "$distributionSha256Sum $TMP_DOWNLOAD_DIR/$distributionUrlName" | shasum -a 256 -c >/dev/null 2>&1; then
distributionSha256Result=true
fi
else
echo "Checksum validation was requested but neither 'sha256sum' or 'shasum' are available." >&2
echo "Please install either command, or disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties." >&2
exit 1
fi
if [ $distributionSha256Result = false ]; then
echo "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised." >&2
echo "If you updated your Maven version, you need to update the specified distributionSha256Sum property." >&2
exit 1
fi
fi
# unzip and move
if command -v unzip >/dev/null; then
unzip ${__MVNW_QUIET_UNZIP:+"$__MVNW_QUIET_UNZIP"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -d "$TMP_DOWNLOAD_DIR" || die "failed to unzip"
else
tar xzf${__MVNW_QUIET_TAR:+"$__MVNW_QUIET_TAR"} "$TMP_DOWNLOAD_DIR/$distributionUrlName" -C "$TMP_DOWNLOAD_DIR" || die "failed to untar"
fi
# Find the actual extracted directory name (handles snapshots where filename != directory name)
actualDistributionDir=""
# First try the expected directory name (for regular distributions)
if [ -d "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain" ]; then
if [ -f "$TMP_DOWNLOAD_DIR/$distributionUrlNameMain/bin/$MVN_CMD" ]; then
actualDistributionDir="$distributionUrlNameMain"
fi
fi
# If not found, search for any directory with the Maven executable (for snapshots)
if [ -z "$actualDistributionDir" ]; then
# enable globbing to iterate over items
set +f
for dir in "$TMP_DOWNLOAD_DIR"/*; do
if [ -d "$dir" ]; then
if [ -f "$dir/bin/$MVN_CMD" ]; then
actualDistributionDir="$(basename "$dir")"
break
fi
fi
done
set -f
fi
if [ -z "$actualDistributionDir" ]; then
verbose "Contents of $TMP_DOWNLOAD_DIR:"
verbose "$(ls -la "$TMP_DOWNLOAD_DIR")"
die "Could not find Maven distribution directory in extracted archive"
fi
verbose "Found extracted Maven distribution directory: $actualDistributionDir"
printf %s\\n "$distributionUrl" >"$TMP_DOWNLOAD_DIR/$actualDistributionDir/mvnw.url"
mv -- "$TMP_DOWNLOAD_DIR/$actualDistributionDir" "$MAVEN_HOME" || [ -d "$MAVEN_HOME" ] || die "fail to move MAVEN_HOME"
clean || :
exec_maven "$@"
Vendored
+189
View File
@@ -0,0 +1,189 @@
<# : batch portion
@REM ----------------------------------------------------------------------------
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements. See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership. The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License. You may obtain a copy of the License at
@REM
@REM http://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied. See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM ----------------------------------------------------------------------------
@REM ----------------------------------------------------------------------------
@REM Apache Maven Wrapper startup batch script, version 3.3.4
@REM
@REM Optional ENV vars
@REM MVNW_REPOURL - repo url base for downloading maven distribution
@REM MVNW_USERNAME/MVNW_PASSWORD - user and password for downloading maven
@REM MVNW_VERBOSE - true: enable verbose log; others: silence the output
@REM ----------------------------------------------------------------------------
@IF "%__MVNW_ARG0_NAME__%"=="" (SET __MVNW_ARG0_NAME__=%~nx0)
@SET __MVNW_CMD__=
@SET __MVNW_ERROR__=
@SET __MVNW_PSMODULEP_SAVE=%PSModulePath%
@SET PSModulePath=
@FOR /F "usebackq tokens=1* delims==" %%A IN (`powershell -noprofile "& {$scriptDir='%~dp0'; $script='%__MVNW_ARG0_NAME__%'; icm -ScriptBlock ([Scriptblock]::Create((Get-Content -Raw '%~f0'))) -NoNewScope}"`) DO @(
IF "%%A"=="MVN_CMD" (set __MVNW_CMD__=%%B) ELSE IF "%%B"=="" (echo %%A) ELSE (echo %%A=%%B)
)
@SET PSModulePath=%__MVNW_PSMODULEP_SAVE%
@SET __MVNW_PSMODULEP_SAVE=
@SET __MVNW_ARG0_NAME__=
@SET MVNW_USERNAME=
@SET MVNW_PASSWORD=
@IF NOT "%__MVNW_CMD__%"=="" ("%__MVNW_CMD__%" %*)
@echo Cannot start maven from wrapper >&2 && exit /b 1
@GOTO :EOF
: end batch / begin powershell #>
$ErrorActionPreference = "Stop"
if ($env:MVNW_VERBOSE -eq "true") {
$VerbosePreference = "Continue"
}
# calculate distributionUrl, requires .mvn/wrapper/maven-wrapper.properties
$distributionUrl = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionUrl
if (!$distributionUrl) {
Write-Error "cannot read distributionUrl property in $scriptDir/.mvn/wrapper/maven-wrapper.properties"
}
switch -wildcard -casesensitive ( $($distributionUrl -replace '^.*/','') ) {
"maven-mvnd-*" {
$USE_MVND = $true
$distributionUrl = $distributionUrl -replace '-bin\.[^.]*$',"-windows-amd64.zip"
$MVN_CMD = "mvnd.cmd"
break
}
default {
$USE_MVND = $false
$MVN_CMD = $script -replace '^mvnw','mvn'
break
}
}
# apply MVNW_REPOURL and calculate MAVEN_HOME
# maven home pattern: ~/.m2/wrapper/dists/{apache-maven-<version>,maven-mvnd-<version>-<platform>}/<hash>
if ($env:MVNW_REPOURL) {
$MVNW_REPO_PATTERN = if ($USE_MVND -eq $False) { "/org/apache/maven/" } else { "/maven/mvnd/" }
$distributionUrl = "$env:MVNW_REPOURL$MVNW_REPO_PATTERN$($distributionUrl -replace "^.*$MVNW_REPO_PATTERN",'')"
}
$distributionUrlName = $distributionUrl -replace '^.*/',''
$distributionUrlNameMain = $distributionUrlName -replace '\.[^.]*$','' -replace '-bin$',''
$MAVEN_M2_PATH = "$HOME/.m2"
if ($env:MAVEN_USER_HOME) {
$MAVEN_M2_PATH = "$env:MAVEN_USER_HOME"
}
if (-not (Test-Path -Path $MAVEN_M2_PATH)) {
New-Item -Path $MAVEN_M2_PATH -ItemType Directory | Out-Null
}
$MAVEN_WRAPPER_DISTS = $null
if ((Get-Item $MAVEN_M2_PATH).Target[0] -eq $null) {
$MAVEN_WRAPPER_DISTS = "$MAVEN_M2_PATH/wrapper/dists"
} else {
$MAVEN_WRAPPER_DISTS = (Get-Item $MAVEN_M2_PATH).Target[0] + "/wrapper/dists"
}
$MAVEN_HOME_PARENT = "$MAVEN_WRAPPER_DISTS/$distributionUrlNameMain"
$MAVEN_HOME_NAME = ([System.Security.Cryptography.SHA256]::Create().ComputeHash([byte[]][char[]]$distributionUrl) | ForEach-Object {$_.ToString("x2")}) -join ''
$MAVEN_HOME = "$MAVEN_HOME_PARENT/$MAVEN_HOME_NAME"
if (Test-Path -Path "$MAVEN_HOME" -PathType Container) {
Write-Verbose "found existing MAVEN_HOME at $MAVEN_HOME"
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"
exit $?
}
if (! $distributionUrlNameMain -or ($distributionUrlName -eq $distributionUrlNameMain)) {
Write-Error "distributionUrl is not valid, must end with *-bin.zip, but found $distributionUrl"
}
# prepare tmp dir
$TMP_DOWNLOAD_DIR_HOLDER = New-TemporaryFile
$TMP_DOWNLOAD_DIR = New-Item -Itemtype Directory -Path "$TMP_DOWNLOAD_DIR_HOLDER.dir"
$TMP_DOWNLOAD_DIR_HOLDER.Delete() | Out-Null
trap {
if ($TMP_DOWNLOAD_DIR.Exists) {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
}
New-Item -Itemtype Directory -Path "$MAVEN_HOME_PARENT" -Force | Out-Null
# Download and Install Apache Maven
Write-Verbose "Couldn't find MAVEN_HOME, downloading and installing it ..."
Write-Verbose "Downloading from: $distributionUrl"
Write-Verbose "Downloading to: $TMP_DOWNLOAD_DIR/$distributionUrlName"
$webclient = New-Object System.Net.WebClient
if ($env:MVNW_USERNAME -and $env:MVNW_PASSWORD) {
$webclient.Credentials = New-Object System.Net.NetworkCredential($env:MVNW_USERNAME, $env:MVNW_PASSWORD)
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$webclient.DownloadFile($distributionUrl, "$TMP_DOWNLOAD_DIR/$distributionUrlName") | Out-Null
# If specified, validate the SHA-256 sum of the Maven distribution zip file
$distributionSha256Sum = (Get-Content -Raw "$scriptDir/.mvn/wrapper/maven-wrapper.properties" | ConvertFrom-StringData).distributionSha256Sum
if ($distributionSha256Sum) {
if ($USE_MVND) {
Write-Error "Checksum validation is not supported for maven-mvnd. `nPlease disable validation by removing 'distributionSha256Sum' from your maven-wrapper.properties."
}
Import-Module $PSHOME\Modules\Microsoft.PowerShell.Utility -Function Get-FileHash
if ((Get-FileHash "$TMP_DOWNLOAD_DIR/$distributionUrlName" -Algorithm SHA256).Hash.ToLower() -ne $distributionSha256Sum) {
Write-Error "Error: Failed to validate Maven distribution SHA-256, your Maven distribution might be compromised. If you updated your Maven version, you need to update the specified distributionSha256Sum property."
}
}
# unzip and move
Expand-Archive "$TMP_DOWNLOAD_DIR/$distributionUrlName" -DestinationPath "$TMP_DOWNLOAD_DIR" | Out-Null
# Find the actual extracted directory name (handles snapshots where filename != directory name)
$actualDistributionDir = ""
# First try the expected directory name (for regular distributions)
$expectedPath = Join-Path "$TMP_DOWNLOAD_DIR" "$distributionUrlNameMain"
$expectedMvnPath = Join-Path "$expectedPath" "bin/$MVN_CMD"
if ((Test-Path -Path $expectedPath -PathType Container) -and (Test-Path -Path $expectedMvnPath -PathType Leaf)) {
$actualDistributionDir = $distributionUrlNameMain
}
# If not found, search for any directory with the Maven executable (for snapshots)
if (!$actualDistributionDir) {
Get-ChildItem -Path "$TMP_DOWNLOAD_DIR" -Directory | ForEach-Object {
$testPath = Join-Path $_.FullName "bin/$MVN_CMD"
if (Test-Path -Path $testPath -PathType Leaf) {
$actualDistributionDir = $_.Name
}
}
}
if (!$actualDistributionDir) {
Write-Error "Could not find Maven distribution directory in extracted archive"
}
Write-Verbose "Found extracted Maven distribution directory: $actualDistributionDir"
Rename-Item -Path "$TMP_DOWNLOAD_DIR/$actualDistributionDir" -NewName $MAVEN_HOME_NAME | Out-Null
try {
Move-Item -Path "$TMP_DOWNLOAD_DIR/$MAVEN_HOME_NAME" -Destination $MAVEN_HOME_PARENT | Out-Null
} catch {
if (! (Test-Path -Path "$MAVEN_HOME" -PathType Container)) {
Write-Error "fail to move MAVEN_HOME"
}
} finally {
try { Remove-Item $TMP_DOWNLOAD_DIR -Recurse -Force | Out-Null }
catch { Write-Warning "Cannot remove $TMP_DOWNLOAD_DIR" }
}
Write-Output "MVN_CMD=$MAVEN_HOME/bin/$MVN_CMD"
+182 -21
View File
@@ -1,36 +1,109 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<!--
~ QQQ - Low-code Application Framework for Engineers.
~ Copyright (C) 2021-2026. Kingsrook, LLC
~ 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
~ contact@kingsrook.com
~ https://github.com/Kingsrook/
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ https://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-example-extension</artifactId>
<version>0.1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>QBit Example Extension</name>
<description>Example Extension QBit - rename this for your QBit</description>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<!-- QQQ 4.1.0-RC.15 ships UI API 1.6.0, field access and audit value protection -->
<qqq.snapshot.version>4.1.0-RC.15</qqq.snapshot.version>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<qqq.version>4.0.0</qqq.version>
<jacoco.version>0.8.15</jacoco.version>
<h2.version>2.2.224</h2.version>
</properties>
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-secured-data</artifactId>
<version>${revision}</version>
<packaging>jar</packaging>
<name>QBit Secured Data</name>
<description>QBit for role-based field access by data classification, reveal grants, break-glass access and access logging in QQQ applications</description>
<url>https://github.com/QRun-IO/qbit-secured-data</url>
<modelVersion>4.0.0</modelVersion>
<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0</url>
<distribution>repo</distribution>
</license>
</licenses>
<developers>
<developer>
<id>kingsrook</id>
<name>Kingsrook Team</name>
<email>contact@kingsrook.com</email>
<organization>Kingsrook, LLC</organization>
<organizationUrl>https://kingsrook.com</organizationUrl>
</developer>
</developers>
<scm>
<connection>scm:git:git://github.com/QRun-IO/qbit-secured-data.git</connection>
<developerConnection>scm:git:ssh://github.com:QRun-IO/qbit-secured-data.git</developerConnection>
<url>https://github.com/QRun-IO/qbit-secured-data</url>
<tag>HEAD</tag>
</scm>
<dependencies>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-core</artifactId>
<version>${qqq.version}</version>
</dependency>
<!-- The memory backend is included in qqq-backend-core. -->
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-module-rdbms</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-middleware-javalin</artifactId>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-frontend-next</artifactId>
<version>1.0.0-RC.14</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>${h2.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.10.0</version>
<scope>test</scope>
</dependency>
<dependency>
@@ -45,14 +118,102 @@
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.11.0</version>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/*Test.java</include>
<include>**/*IT.java</include>
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.1.2</version>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.3.1</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>10.12.6</version>
</dependency>
</dependencies>
<configuration>
<configLocation>checkstyle/config.xml</configLocation>
<headerLocation>checkstyle/license.txt</headerLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failsOnError>false</failsOnError>
<failOnViolation>true</failOnViolation>
<violationSeverity>warning</violationSeverity>
<includeTestSourceDirectory>false</includeTestSourceDirectory>
</configuration>
<executions>
<execution>
<id>validate</id>
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>${jacoco.version}</version>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>verify</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
<execution>
<id>check</id>
<goals>
<goal>check</goal>
</goals>
<configuration>
<rules>
<rule>
<element>BUNDLE</element>
<limits>
<limit>
<counter>INSTRUCTION</counter>
<value>COVEREDRATIO</value>
<minimum>0.70</minimum>
</limit>
<limit>
<counter>CLASS</counter>
<value>COVEREDRATIO</value>
<minimum>0.90</minimum>
</limit>
</limits>
</rule>
</rules>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-bom-pom</artifactId>
<version>${qqq.snapshot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</project>
@@ -1,75 +0,0 @@
/*******************************************************************************
** Configuration for the Example Extension QBit.
*******************************************************************************/
package com.kingsrook.qbits.example;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
/*******************************************************************************
** Configuration class for the Example Extension QBit.
*******************************************************************************/
public class ExampleExtensionQBitConfig implements QBitConfig
{
private String targetTableName;
private Boolean enableFeatureX = true;
/*******************************************************************************
** Validate configuration before QBit is produced.
*******************************************************************************/
@Override
public void validate(QInstance qInstance, List<String> errors)
{
if(targetTableName != null && qInstance.getTable(targetTableName) == null)
{
errors.add("Target table not found: " + targetTableName);
}
}
/*******************************************************************************
** Getter for targetTableName
*******************************************************************************/
public String getTargetTableName()
{
return targetTableName;
}
/*******************************************************************************
** Fluent setter for targetTableName
*******************************************************************************/
public ExampleExtensionQBitConfig withTargetTableName(String targetTableName)
{
this.targetTableName = targetTableName;
return this;
}
/*******************************************************************************
** Getter for enableFeatureX
*******************************************************************************/
public Boolean getEnableFeatureX()
{
return enableFeatureX;
}
/*******************************************************************************
** Fluent setter for enableFeatureX
*******************************************************************************/
public ExampleExtensionQBitConfig withEnableFeatureX(Boolean enableFeatureX)
{
this.enableFeatureX = enableFeatureX;
return this;
}
}
@@ -1,66 +0,0 @@
/*******************************************************************************
** Producer for the Example Extension QBit.
*******************************************************************************/
package com.kingsrook.qbits.example;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitProducer;
/*******************************************************************************
** Producer class for the Example Extension QBit.
*******************************************************************************/
public class ExampleExtensionQBitProducer implements QBitProducer
{
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-example-extension";
public static final String VERSION = "0.1.0";
private ExampleExtensionQBitConfig config;
/*******************************************************************************
** Produce this QBit into the given QInstance.
*******************************************************************************/
@Override
public void produce(QInstance qInstance, String namespace) throws QException
{
QBitMetaData qBitMetaData = new QBitMetaData()
.withGroupId(GROUP_ID)
.withArtifactId(ARTIFACT_ID)
.withVersion(VERSION)
.withNamespace(namespace)
.withConfig(config);
qInstance.addQBit(qBitMetaData);
if(config != null && config.getTargetTableName() != null)
{
applyTableCustomizations(qInstance, config.getTargetTableName());
}
}
/*******************************************************************************
** Apply customizations to the target table.
*******************************************************************************/
private void applyTableCustomizations(QInstance qInstance, String tableName)
{
// Implement your extension logic here
}
/*******************************************************************************
** Fluent setter for config
*******************************************************************************/
public ExampleExtensionQBitProducer withConfig(ExampleExtensionQBitConfig config)
{
this.config = config;
return this;
}
}
@@ -1,23 +0,0 @@
/*******************************************************************************
** Example table customizer.
*******************************************************************************/
package com.kingsrook.qbits.example.customizers;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Example table customizer that adds behavior to existing tables.
*******************************************************************************/
public class ExampleTableCustomizer
{
/*******************************************************************************
** Apply customizations to the given table.
*******************************************************************************/
public void customize(QTableMetaData table)
{
// Example: Add audit tracking fields
// table.withField(new QFieldMetaData("createdAt", QFieldType.DATE_TIME));
}
}
@@ -0,0 +1,568 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.lang.ref.WeakReference;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Bounded asynchronous value-free audit, with persisted and local enumeration accounting.
*******************************************************************************/
public class SecuredDataAccessLogger implements AutoCloseable
{
private static final QLogger LOG = QLogger.getLogger(SecuredDataAccessLogger.class);
private final QInstance instance;
private final SecuredDataRuntime runtime;
private final ArrayBlockingQueue<PendingLog> queue;
private final ScheduledExecutorService worker;
private final Thread shutdownHook;
private final AtomicLong dropped = new AtomicLong();
private final Object flushLock = new Object();
private final Object lifecycleLock = new Object();
private final ThreadLocal<String> refusalReason = new ThreadLocal<>();
private final Map<String, ArrayDeque<Instant>> reservations = new HashMap<>();
private volatile boolean closed;
/*******************************************************************************
** Settings are captured for the acting user, before the worker becomes system.
*******************************************************************************/
private record PendingLog(QRecord row, SecuredDataResolver.Guardrails guardrails)
{
}
/*******************************************************************************
** One daemon drains this instance's queue; the optional JVM hook flushes on
** shutdown. Neither holds the logger strongly: once the host drops its
** instance (a test, a reloaded instance), the logger, runtime and instance
** can be collected, and the worker stops at its next tick. Hosts that close
** the runtime themselves can turn the hook off
** (SecuredDataQBitConfig.withRegisterShutdownHook(false)).
*******************************************************************************/
public SecuredDataAccessLogger(QInstance instance, SecuredDataRuntime runtime)
{
this.instance = instance;
this.runtime = runtime;
this.queue = new ArrayBlockingQueue<>(runtime.config().getAccessLogQueueCapacity());
this.worker = Executors.newSingleThreadScheduledExecutor(task ->
{
Thread thread = new Thread(task, "secured-data-access-log");
thread.setDaemon(true);
return (thread);
});
WeakReference<SecuredDataAccessLogger> self = new WeakReference<>(this);
long interval = runtime.config().getAccessLogFlushIntervalMillis();
worker.scheduleWithFixedDelay(new WeakTick(self, worker), interval, interval, TimeUnit.MILLISECONDS);
shutdownHook = runtime.config().getRegisterShutdownHook() ? new Thread(new WeakClose(self), "secured-data-access-log-shutdown") : null;
if(shutdownHook != null)
{
Runtime.getRuntime().addShutdownHook(shutdownHook);
}
}
/*******************************************************************************
** The worker's task, holding the logger weakly: when the logger has been
** collected the worker shuts itself down.
*******************************************************************************/
private record WeakTick(WeakReference<SecuredDataAccessLogger> logger, ScheduledExecutorService worker) implements Runnable
{
/*******************************************************************************
** Drain, or stop once the logger is gone.
*******************************************************************************/
@Override
public void run()
{
SecuredDataAccessLogger target = logger.get();
if(target == null)
{
worker.shutdown();
return;
}
target.scheduledFlush();
}
}
/*******************************************************************************
** The JVM shutdown hook, holding the logger weakly.
*******************************************************************************/
private record WeakClose(WeakReference<SecuredDataAccessLogger> logger) implements Runnable
{
/*******************************************************************************
** Drain accepted rows at JVM shutdown, if the logger still exists.
*******************************************************************************/
@Override
public void run()
{
SecuredDataAccessLogger target = logger.get();
if(target != null)
{
target.close();
}
}
}
/*******************************************************************************
** Whether the background worker is still running (false after close).
*******************************************************************************/
public boolean isWorkerRunning()
{
return (!worker.isShutdown());
}
/*******************************************************************************
** Drop count for operational monitoring and the future app home.
*******************************************************************************/
public long getDroppedCount()
{
return (dropped.get());
}
/*******************************************************************************
** Snapshot identifiers only. A failed/full logger must never fail the read.
*******************************************************************************/
public void onAccess(FieldAccessEvent event)
{
if(SystemWrites.active())
{
return;
}
try
{
QTableMetaData table = instance.getTable(event.getTableName());
if(table == null || SecuredDataTables.ALL.contains(table.getName()))
{
return;
}
List<String> fields = event.getFieldNames() == null ? List.of() : event.getFieldNames();
if(event.getType() == FieldAccessEventType.VIEW)
{
fields = fields.stream().filter(field -> runtime.config().getViewLogClassifications().contains(SecuredDataClassifications.effective(table, field))).toList();
if(fields.isEmpty())
{
return;
}
}
if(event.getType() == FieldAccessEventType.EXACT_MATCH_FILTER)
{
for(String field : fields.stream().distinct().toList())
{
offer(row(event, List.of(field)), runtime.resolver().guardrails(table, field));
}
}
else
{
offer(row(event, fields), null);
}
}
catch(Exception e)
{
drop("Unable to queue secured-data access log");
}
}
/*******************************************************************************
** Build a row without copying record values, arbitrary reasons or filter objects.
*******************************************************************************/
private QRecord row(FieldAccessEvent event, List<String> fields)
{
QSession session = QContext.getQSession();
String type = event.getType() == FieldAccessEventType.DENIED ? "FILTER_DENIED" : event.getType().name();
String outcome = event.getOutcome() == null ? (event.getType() == FieldAccessEventType.DENIED ? "DENIED" : "ALLOWED") : event.getOutcome().name();
QRecord row = new QRecord().withValue("event_type", type).withValue("user_id", SecuredDataResolver.userId())
.withValue("user_name", session == null || session.getUser() == null ? null : session.getUser().getFullName())
.withValue("table_name", event.getTableName()).withValue("field_name", String.join(",", fields))
.withValue("record_count", event.getRecordCount()).withValue("action_name", event.getActionName())
.withValue("client_channel", SecuredDataMetadata.channel(session)).withValue("outcome", outcome)
.withValue("anomaly_flag", false).withValue("created_at", runtime.config().getClock().instant());
if(event.getPrimaryKeys() != null)
{
row.setValue("record_id", JsonUtils.toJson(event.getPrimaryKeys().stream().limit(50).toList()));
}
if(event.getSearchValueFingerprints() != null)
{
row.setValue("search_value_fingerprints", JsonUtils.toJson(event.getSearchValueFingerprints()));
}
if(event.getOutcome() == FieldAccessOutcome.REFUSED)
{
row.setValue("reason", Objects.requireNonNullElse(refusalReason.get(), "Exact-match search refused by access policy or enumeration guardrail."));
refusalReason.remove();
}
QTableMetaData table = instance.getTable(event.getTableName());
QRecord breakGlass = runtime.resolver().activeBreakGlass(table.getName(), SecuredDataClassifications.highest(table, fields));
if(breakGlass != null)
{
row.setValue("break_glass_id", breakGlass.getValue("id"));
}
return (row);
}
/*******************************************************************************
** A full queue drops without waiting, warning only with value-free text.
*******************************************************************************/
private void offer(QRecord row, SecuredDataResolver.Guardrails guardrails)
{
synchronized(lifecycleLock)
{
if(closed || !queue.offer(new PendingLog(row, guardrails)))
{
drop("Secured-data access-log queue is full or closed; dropped one row");
}
}
}
/*******************************************************************************
** Keep failed persistence observable without dumping sensitive record objects.
*******************************************************************************/
private void drop(String message)
{
dropped.incrementAndGet();
LOG.warn(message);
}
/*******************************************************************************
** Authoritative persisted count plus queued rows and uncompleted local searches.
** Serializes preflight and drain accounting, never ordinary VIEW enqueues.
*******************************************************************************/
public void checkExactMatch(FieldAccessEvent event) throws QPermissionDeniedException
{
synchronized(flushLock)
{
refusalReason.remove();
String user = SecuredDataResolver.userId();
QTableMetaData table = instance.getTable(event.getTableName());
Instant now = runtime.config().getClock().instant();
Instant since = now.minus(1, ChronoUnit.HOURS);
expireReservations(since);
List<String> fields = event.getFieldNames().stream().distinct().toList();
for(String field : fields)
{
if(!runtime.resolver().resolve(table, table.getField(field)).exactMatchAllowed())
{
refusalReason.set("Exact-match search is disabled for this low-cardinality field.");
throw (new QPermissionDeniedException("Permission denied."));
}
int limit = runtime.resolver().guardrails(table, field).exactMatchPerHour();
if(limit == 0)
{
continue;
}
List<QRecord> persisted = history(user, table.getName(), field, since);
long pending = queue.stream().filter(item -> matches(item.row(), user, table.getName(), field, since)).count();
long reserved = reservations.getOrDefault(key(user, table.getName(), field), new ArrayDeque<>()).size();
// Accepted outcomes in the queue still have their reservation until flush.
long completedReserved = queue.stream().filter(item -> matches(item.row(), user, table.getName(), field, since))
.filter(item -> !"REFUSED".equals(item.row().getValueString("outcome"))).count();
if(persisted.size() + pending + Math.max(0, reserved - completedReserved) >= limit)
{
String reason = "Exact-match search limit reached for [" + field + "]. Try again later or ask an administrator.";
refusalReason.set(reason);
throw (new QPermissionDeniedException(reason));
}
}
for(String field : fields)
{
if(runtime.resolver().guardrails(table, field).exactMatchPerHour() > 0)
{
reservations.computeIfAbsent(key(user, table.getName(), field), ignored -> new ArrayDeque<>()).addLast(now);
}
}
}
}
/*******************************************************************************
** Retire conservative reservations left behind by failed queries or dropped logs.
*******************************************************************************/
private void expireReservations(Instant since)
{
reservations.values().forEach(times -> times.removeIf(time -> !time.isAfter(since)));
reservations.values().removeIf(ArrayDeque::isEmpty);
}
/*******************************************************************************
** Identifier-safe composite key.
*******************************************************************************/
private static String key(String user, String table, String field)
{
return (JsonUtils.toJson(List.of(user, table, field)));
}
/*******************************************************************************
** The queue contains one exact-match row per field, matching the indexed query.
*******************************************************************************/
private static boolean matches(QRecord row, String user, String table, String field, Instant since)
{
return ("EXACT_MATCH_FILTER".equals(row.getValueString("event_type")) && Objects.equals(user, row.getValueString("user_id"))
&& Objects.equals(table, row.getValueString("table_name")) && Objects.equals(field, row.getValueString("field_name"))
&& row.getValueInstant("created_at").isAfter(since));
}
/*******************************************************************************
** Indexed history lookup, always under a system session.
*******************************************************************************/
private List<QRecord> history(String user, String table, String field, Instant since)
{
return (SystemWrites.query(SecuredDataTables.ACCESS_LOG, new QQueryFilter()
.withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, user))
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table))
.withCriteria(new QFilterCriteria("field_name", QCriteriaOperator.EQUALS, field))
.withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.EQUALS, "EXACT_MATCH_FILTER"))
.withCriteria(new QFilterCriteria("created_at", QCriteriaOperator.GREATER_THAN, since))
.withOrderBy(new QFilterOrderBy("created_at", false)).withOrderBy(new QFilterOrderBy("id", false))));
}
/*******************************************************************************
** The worker's tick. An exception would cancel every later run of a
** scheduled task, so a failed drain is logged (value-free) and retried on
** the next tick; the rows stay queued.
*******************************************************************************/
void scheduledFlush()
{
try
{
flush();
}
catch(RuntimeException e)
{
LOG.warn("Unable to drain the secured-data access log; retrying on the next tick");
}
}
/*******************************************************************************
** Drain safely both from the worker and explicit test/host shutdown calls.
** An empty queue opens no context: QContext validates an unvalidated
** instance, and the worker starts while the host is still building it.
*******************************************************************************/
public void flush()
{
synchronized(flushLock)
{
if(queue.isEmpty())
{
return;
}
QContext.withTemporaryContext(new CapturedContext(instance, new QSystemUserSession(instance)), () ->
{
PendingLog pending;
while((pending = queue.peek()) != null)
{
QRecord row = pending.row();
try
{
boolean anomaly = isAnomaly(pending);
row.setValue("anomaly_flag", anomaly);
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, row);
if(anomaly)
{
raiseAlert(row);
}
completeReservation(row);
}
catch(Exception e)
{
drop("Unable to persist secured-data access log; dropped one row");
}
finally
{
queue.poll();
}
}
});
}
}
/*******************************************************************************
** Only persisted successful searches release their in-flight reservation.
*******************************************************************************/
private void completeReservation(QRecord row)
{
if(!"EXACT_MATCH_FILTER".equals(row.getValueString("event_type")) || "REFUSED".equals(row.getValueString("outcome")))
{
return;
}
ArrayDeque<Instant> times = reservations.get(key(row.getValueString("user_id"), row.getValueString("table_name"), row.getValueString("field_name")));
if(times != null)
{
times.pollFirst();
}
}
/*******************************************************************************
** Evaluate each row in drain order against already persisted predecessors.
*******************************************************************************/
private boolean isAnomaly(PendingLog pending) throws java.io.IOException
{
SecuredDataResolver.Guardrails settings = pending.guardrails();
if(settings == null)
{
return (false);
}
QRecord row = pending.row();
List<QRecord> history = history(row.getValueString("user_id"), row.getValueString("table_name"), row.getValueString("field_name"),
row.getValueInstant("created_at").minus(1, ChronoUnit.HOURS));
int misses = "NO_MATCH".equals(row.getValueString("outcome")) ? 1 : 0;
if(misses > 0 && settings.missAlertConsecutive() > 0)
{
List<QRecord> recent = SystemWrites.query(SecuredDataTables.ACCESS_LOG, new QQueryFilter()
.withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, row.getValueString("user_id")))
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, row.getValueString("table_name")))
.withCriteria(new QFilterCriteria("field_name", QCriteriaOperator.EQUALS, row.getValueString("field_name")))
.withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.EQUALS, "EXACT_MATCH_FILTER"))
.withOrderBy(new QFilterOrderBy("created_at", false)).withOrderBy(new QFilterOrderBy("id", false))
.withLimit(Math.max(1, settings.missAlertConsecutive())));
for(QRecord previous : recent)
{
if(!"NO_MATCH".equals(previous.getValueString("outcome")))
{
break;
}
misses++;
}
}
Set<String> distinct = new HashSet<>();
List<QRecord> combined = new ArrayList<>(history);
combined.add(row);
for(QRecord item : combined)
{
String fingerprints = item.getValueString("search_value_fingerprints");
if(fingerprints != null)
{
distinct.addAll(Arrays.asList(JsonUtils.toObject(fingerprints, String[].class)));
}
}
return ((settings.missAlertConsecutive() > 0 && misses >= settings.missAlertConsecutive())
|| (settings.distinctValueAlert() > 0 && distinct.size() > settings.distinctValueAlert()));
}
/*******************************************************************************
** Core #1090 is released in RC.5; its optional sink is supplied by qbit-alerts.
*******************************************************************************/
private static void raiseAlert(QRecord row)
{
String dedup = "secured_data.enumeration:" + row.getValueString("user_id") + ":" + row.getValueString("table_name") + "."
+ row.getValueString("field_name") + ":" + row.getValueInstant("created_at").atOffset(java.time.ZoneOffset.UTC).toLocalDate();
QAlerts.raise(new AlertSignal("secured_data.enumeration").withDedupKey(dedup)
.withTitle("Repeated exact-match searches on secured data").withSummary("Review the secured-data access log for enumeration activity."));
}
/*******************************************************************************
** Idempotent shutdown; stops scheduling and drains accepted rows.
*******************************************************************************/
@Override
public void close()
{
synchronized(lifecycleLock)
{
closed = true;
}
worker.shutdown();
flush();
if(shutdownHook != null && Thread.currentThread() != shutdownHook)
{
try
{
Runtime.getRuntime().removeShutdownHook(shutdownHook);
}
catch(IllegalStateException expected)
{
// JVM shutdown has already begun.
}
}
}
}
@@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.tables.AggregateAction;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.Aggregate;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.GroupBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
/*******************************************************************************
** System-only access-log counters. No protected values or unbounded row reads.
*******************************************************************************/
public class SecuredDataAnomalies
{
private final SecuredDataQBitConfig config;
private final Map<String, Double> medians = new LinkedHashMap<>();
private Instant medianDay;
/*******************************************************************************
** Instance-scoped bounded daily cache.
*******************************************************************************/
public SecuredDataAnomalies(SecuredDataQBitConfig config)
{
this.config = config;
}
/*******************************************************************************
** Evaluate before insertion, including this row exactly once.
*******************************************************************************/
public synchronized boolean evaluate(QRecord row) throws QException
{
String event = row.getValueString("event_type");
if("BREAK_GLASS_START".equals(event) || ("REVEAL_DENIED".equals(event) && "CAPPED".equals(row.getValueString("outcome"))))
{
return (true);
}
Instant now = row.getValueInstant("created_at");
if(!List.of("REVEAL", "EXACT_MATCH_FILTER").contains(event) || now == null)
{
return (false);
}
String user = row.getValueString("user_id");
QQueryFilter hour = filter(user, event, now.minus(1, ChronoUnit.HOURS), now, QCriteriaOperator.LESS_THAN_OR_EQUALS);
hour.getCriteria().stream().filter(criteria -> "created_at".equals(criteria.getFieldName())
&& criteria.getOperator() == QCriteriaOperator.GREATER_THAN_OR_EQUALS).forEach(criteria -> criteria.setOperator(QCriteriaOperator.GREATER_THAN));
if("EXACT_MATCH_FILTER".equals(event))
{
hour.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, row.getValueString("table_name")))
.withCriteria(new QFilterCriteria("field_name", QCriteriaOperator.EQUALS, row.getValueString("field_name")));
return (count(hour) >= config.getAnomalyExactMatchesPerHour());
}
Instant day = now.truncatedTo(ChronoUnit.DAYS);
int today = count(filter(user, event, day, now, QCriteriaOperator.LESS_THAN_OR_EQUALS)) + 1;
if(today >= config.getAnomalyMinimumDailyReveals() && today > config.getAnomalyDailyMedianMultiplier() * median(user, day))
{
return (true);
}
GroupBy table = new GroupBy(QFieldType.STRING, "table_name");
GroupBy record = new GroupBy(QFieldType.STRING, "record_id");
var groups = SystemWrites.call(() -> new AggregateAction().execute(new AggregateInput(SecuredDataTables.ACCESS_LOG)
.withFilter(hour).withGroupBy(table).withGroupBy(record).withAggregate(new Aggregate("id", AggregateOperator.COUNT))
.withLimit(config.getAnomalyDistinctRecordsPerHour() + 1)).getResults());
long distinct = groups.size();
boolean alreadySeen = groups.stream().anyMatch(group -> java.util.Objects.equals(group.getGroupByValue(table), row.getValue("table_name"))
&& java.util.Objects.equals(group.getGroupByValue(record), row.getValue("record_id")));
return (distinct + (alreadySeen ? 0 : 1) > config.getAnomalyDistinctRecordsPerHour());
}
/*******************************************************************************
** Previous thirty complete UTC days, including zeros; even median averages middle days.
** Thirty scalar counts once per user/day; at most 2048 cached users.
*******************************************************************************/
private double median(String user, Instant day) throws QException
{
if(!day.equals(medianDay))
{
medians.clear();
medianDay = day;
}
if(!medians.containsKey(user))
{
int[] counts = new int[30];
for(int i = 0; i < counts.length; i++)
{
counts[i] = count(filter(user, "REVEAL", day.minus(i + 1, ChronoUnit.DAYS), day.minus(i, ChronoUnit.DAYS), QCriteriaOperator.LESS_THAN));
}
Arrays.sort(counts);
if(medians.size() >= 2048)
{
medians.remove(medians.keySet().iterator().next());
}
medians.put(user, (counts[14] + (double) counts[15]) / 2);
}
return (medians.get(user));
}
/*******************************************************************************
** Scalar database count, always as system.
*******************************************************************************/
private static int count(QQueryFilter filter) throws QException
{
return (SystemWrites.call(() -> new CountAction().execute(new CountInput(SecuredDataTables.ACCESS_LOG).withFilter(filter)).getCount()));
}
/*******************************************************************************
** Bounded time window and user, compatible with indexed access-log lookups.
*******************************************************************************/
private static QQueryFilter filter(String user, String event, Instant from, Instant until, QCriteriaOperator endOperator)
{
return (new QQueryFilter().withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, user))
.withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.EQUALS, event))
.withCriteria(new QFilterCriteria("created_at", QCriteriaOperator.GREATER_THAN_OR_EQUALS, from))
.withCriteria(new QFilterCriteria("created_at", endOperator, until)));
}
/*******************************************************************************
** Optional SPI: flags remain useful without a sink. Deduplication belongs to the sink.
*******************************************************************************/
public static void raise(QRecord row)
{
QAlerts.raise(new AlertSignal("secured_data.anomaly")
.withDedupKey("secured_data.anomaly:" + row.getValueString("user_id") + ":" + row.getValueInstant("created_at").toString().substring(0, 10))
.withOccurredAt(row.getValueInstant("created_at")).withTitle("Unusual secured-data access")
.withSummary("Review the flagged secured-data access-log row."));
}
}
@@ -0,0 +1,63 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.audits.AuditValueProtectorInterface;
import com.kingsrook.qqq.backend.core.actions.audits.DMLAuditAction.DMLType;
import com.kingsrook.qqq.backend.core.actions.audits.ProtectedAuditValues;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Classifications and policy determine whether audits retain a keyed derivative.
*******************************************************************************/
public final class SecuredDataAuditProtector implements AuditValueProtectorInterface
{
/*******************************************************************************
** Core also uses this source for runtime classification overrides.
*******************************************************************************/
@Override
public DataClassification classificationOf(QTableMetaData table, QFieldMetaData field)
{
DataClassification classification = SecuredDataClassifications.effective(table, field.getName());
if(com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.ALL.contains(table.getName()))
{
return (classification);
}
return (RetentionRules.active(table.getName(), field.getName()).isEmpty() ? classification : DataClassification.max(classification, DataClassification.PII));
}
/*******************************************************************************
** Null stays absent; CHANGED_ONLY never creates a DEK.
*******************************************************************************/
@Override
public ProtectedAuditValues protect(QTableMetaData table, QFieldMetaData field, Serializable pk, Serializable oldValue, Serializable newValue, DMLType dmlType)
{
boolean changed = RetentionRules.changedOnly(table.getName(), field.getName());
String message = (oldValue == null ? "Set " : newValue == null ? "Removed " : "Changed ") + field.getLabel();
return (new ProtectedAuditValues(oldValue == null ? null : changed ? "changed:v1" : ClassifiedValueTokens.forRecordField(table, field, pk, oldValue),
newValue == null ? null : changed ? "changed:v1" : ClassifiedValueTokens.forRecordField(table, field, pk, newValue), message));
}
}
@@ -0,0 +1,141 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.Collection;
import java.util.Objects;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Public classification API shared by QBits, with runtime rows overriding code metadata.
*******************************************************************************/
public final class SecuredDataClassifications
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataClassifications()
{
}
/*******************************************************************************
** Field row, table row, code field, code table, then unclassified.
*******************************************************************************/
public static DataClassification effective(QTableMetaData table, String fieldName)
{
QRecord row = row(table, fieldName);
if(row != null)
{
return (DataClassification.valueOf(row.getValueString("classification")));
}
////////////////////////////////////////////////////////////////////
// getFields, not getField: a virtual field is not in the map and //
// getField throws, while it still takes the table classification //
////////////////////////////////////////////////////////////////////
QFieldMetaData field = table.getFields() == null || fieldName == null ? null : table.getFields().get(fieldName);
DataClassification classification = field == null ? null : SecuredDataMetadata.classification(table, field);
return (classification == null ? SecuredDataMetadata.classification(table, null) : classification);
}
/*******************************************************************************
** Table-level override or code-level table classification.
*******************************************************************************/
public static DataClassification effective(QTableMetaData table)
{
QRecord row = row(table, null);
return (row == null ? SecuredDataMetadata.classification(table, null) : DataClassification.valueOf(row.getValueString("classification")));
}
/*******************************************************************************
** Most sensitive effective field, including an empty table's classification.
*******************************************************************************/
public static DataClassification highest(QTableMetaData table)
{
return (table.getFields().isEmpty() ? effective(table) : highest(table, table.getFields().keySet()));
}
/*******************************************************************************
** Most sensitive among the requested fields only; an empty subset is null.
*******************************************************************************/
public static DataClassification highest(QTableMetaData table, Collection<String> fieldNames)
{
DataClassification highest = null;
for(String fieldName : fieldNames)
{
highest = DataClassification.max(highest, effective(table, fieldName));
}
return (highest);
}
/*******************************************************************************
** Effective classification row also owns mask and low-cardinality overrides.
*******************************************************************************/
static QRecord row(QTableMetaData table, String fieldName)
{
SecuredDataRuntime runtime = SecuredDataRuntime.current();
if(runtime == null)
{
return (null);
}
QRecord tableRow = null;
for(QRecord row : runtime.classificationRows())
{
if(!Objects.equals(table.getName(), row.getValueString("table_name")))
{
continue;
}
String field = row.getValueString("field_name");
if(Objects.equals(fieldName, field))
{
return (row);
}
if(field == null || field.isBlank())
{
tableRow = row;
}
}
return (tableRow);
}
/*******************************************************************************
** Shared phase-b entropy guardrail, including runtime overrides.
*******************************************************************************/
public static boolean lowEntropy(QTableMetaData table, String field)
{
return (SecuredDataResolver.lowCardinality(table.getField(field), row(table, field)));
}
}
@@ -0,0 +1,291 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import javax.crypto.Cipher;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.Serializable;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.security.SecureRandom;
import java.time.Instant;
import java.util.Arrays;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySource;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Per-instance audit DEKs. Cache hits still require the key row to exist, so
** another node's committed deletion takes effect immediately on the next lookup.
*******************************************************************************/
public final class SecuredDataKeySource implements ClassifiedValueKeySource, AutoCloseable
{
private final ThreadLocal<Boolean> instanceOnly = ThreadLocal.withInitial(() -> false);
private static final SecureRandom RANDOM = new SecureRandom();
private final Map<String, CachedKey> cache = new LinkedHashMap<>();
/*******************************************************************************
** Short-lived cache; never populated by uncommitted work.
*******************************************************************************/
private record CachedKey(byte[] key, Instant expires)
{
}
/*******************************************************************************
** Use core's interpreter directly: convenience environment getters log values.
*******************************************************************************/
public static String environment(String name)
{
return (new QMetaDataVariableInterpreter().interpret("${prop." + name + "}??${env." + name + "}"));
}
/*******************************************************************************
** Instance HMAC version, without retaining or logging configuration text.
*******************************************************************************/
@Override
public Optional<KeyAndId> instanceKey()
{
String encoded = environment("QQQ_AUDIT_HMAC_KEY");
if(encoded == null || encoded.isBlank())
{
return (Optional.empty());
}
try
{
String id = environment("QQQ_AUDIT_HMAC_KEY_ID");
return (Optional.of(new KeyAndId(Base64.getDecoder().decode(encoded), id == null || id.isBlank() ? "env1" : id)));
}
catch(IllegalArgumentException e)
{
throw (new IllegalStateException("Invalid audit HMAC key configuration"));
}
}
/*******************************************************************************
** Policies and entropy share the phase-b classification rules.
*******************************************************************************/
public boolean shreds(String table, String field)
{
var metadata = QContext.getQInstance().getTable(table);
DataClassification classification = SecuredDataClassifications.effective(metadata, field);
return (RetentionRules.active(table, field).stream().anyMatch(rule -> "SHREDDABLE".equals(rule.getValueString("audit_treatment")))
|| ((classification == DataClassification.PII || classification == DataClassification.PHI) && SecuredDataClassifications.lowEntropy(metadata, field)));
}
/*******************************************************************************
** Verification never creates. Missing KEK fails closed rather than falling back
** to an instance HMAC for a field that must be shredded.
*******************************************************************************/
@Override
public synchronized Optional<KeyAndId> keyForRecordField(String table, String field, Serializable pk, boolean create)
{
if(instanceOnly.get())
{
return (Optional.empty());
}
String binding = table + "\u001f" + field + "\u001f" + pk;
try
{
List<QRecord> rows = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)
.withFilter(recordField(table, field, pk)).withShouldOmitHiddenFields(false)
.withTransaction(QContext.getQBackendTransaction())).getRecords());
if(rows.isEmpty())
{
evict(binding);
if(!create || RetentionRules.changedOnly(table, field) || !shreds(table, field))
{
return (Optional.empty());
}
byte[] dek = new byte[32];
RANDOM.nextBytes(dek);
try
{
String wrapped = Base64.getEncoder().encodeToString(encrypt(dek, binding));
QRecord row = SystemWrites.insertOne(SecuredDataTables.AUDIT_KEY, new QRecord()
.withValue("table_name", table).withValue("field_name", field).withValue("record_id", String.valueOf(pk))
.withValue("wrapped_key", wrapped).withValue("created_at", SecuredDataRuntime.current().config().getClock().instant()));
return (Optional.of(new KeyAndId(dek.clone(), "dek" + row.getValueString("id"))));
}
finally
{
Arrays.fill(dek, (byte) 0);
}
}
QRecord row = rows.getFirst();
Instant now = SecuredDataRuntime.current().config().getClock().instant();
CachedKey cached = cache.get(binding);
if(cached != null && cached.expires().isAfter(now) && QContext.getQBackendTransaction() == null)
{
return (Optional.of(new KeyAndId(cached.key().clone(), "dek" + row.getValueString("id"))));
}
evict(binding);
byte[] dek = decrypt(Base64.getDecoder().decode(row.getValueString("wrapped_key")), binding);
try
{
if(QContext.getQBackendTransaction() == null)
{
if(cache.size() >= 128)
{
close();
}
cache.put(binding, new CachedKey(dek.clone(), now.plusSeconds(5)));
}
return (Optional.of(new KeyAndId(dek.clone(), "dek" + row.getValueString("id"))));
}
finally
{
Arrays.fill(dek, (byte) 0);
}
}
catch(QException e)
{
throw (new IllegalStateException("Unable to access audit key storage"));
}
}
/*******************************************************************************
** Exact key identity, shared by deletion and lookups.
*******************************************************************************/
public static QQueryFilter recordField(String table, String field, Serializable pk)
{
return (new QQueryFilter().withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table))
.withCriteria(new QFilterCriteria("field_name", QCriteriaOperator.EQUALS, field))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, String.valueOf(pk))));
}
/*******************************************************************************
** Erase cached key bytes, including on rollback and runtime shutdown.
*******************************************************************************/
private void evict(String binding)
{
CachedKey old = cache.remove(binding);
if(old != null)
{
Arrays.fill(old.key(), (byte) 0);
}
}
/*******************************************************************************
** AES-GCM envelope: version byte, 12-byte random nonce, ciphertext and tag.
** The binding is authenticated additional data and prevents key-row swapping.
*******************************************************************************/
public static byte[] encrypt(byte[] plaintext, String binding)
{
byte[] nonce = new byte[12];
RANDOM.nextBytes(nonce);
byte[] ciphertext = crypt(Cipher.ENCRYPT_MODE, plaintext, nonce, binding);
return (ByteBuffer.allocate(13 + ciphertext.length).put((byte) 1).put(nonce).put(ciphertext).array());
}
/*******************************************************************************
** Authenticate before exposing any plaintext.
*******************************************************************************/
public static byte[] decrypt(byte[] envelope, String binding)
{
if(envelope.length < 29 || envelope[0] != 1)
{
throw (new IllegalArgumentException("Invalid encrypted envelope"));
}
return (crypt(Cipher.DECRYPT_MODE, Arrays.copyOfRange(envelope, 13, envelope.length), Arrays.copyOfRange(envelope, 1, 13), binding));
}
/*******************************************************************************
** Require exactly a 256-bit KEK; never include secret text in failures.
*******************************************************************************/
private static byte[] crypt(int mode, byte[] input, byte[] nonce, String binding)
{
byte[] kek = null;
try
{
String encoded = environment("QQQ_SECURED_DATA_KEK");
kek = encoded == null ? new byte[0] : Base64.getDecoder().decode(encoded);
if(kek.length != 32)
{
throw (new IllegalArgumentException("QQQ_SECURED_DATA_KEK must be base64 for 32 bytes"));
}
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
cipher.init(mode, new SecretKeySpec(kek, "AES"), new GCMParameterSpec(128, nonce));
cipher.updateAAD(binding.getBytes(StandardCharsets.UTF_8));
return (cipher.doFinal(input));
}
catch(GeneralSecurityException | IllegalArgumentException e)
{
throw (new IllegalStateException("Unable to protect audit data; check QQQ_SECURED_DATA_KEK and envelope integrity"));
}
finally
{
if(kek != null)
{
Arrays.fill(kek, (byte) 0);
}
}
}
/*******************************************************************************
** Clear every decrypted cached key.
*******************************************************************************/
@Override
public synchronized void close()
{
cache.values().forEach(key -> Arrays.fill(key.key(), (byte) 0));
cache.clear();
}
/*******************************************************************************
** KEYED_HASH deliberately uses only the instance key and record binding.
*******************************************************************************/
public String instanceToken(com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData table,
com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData field, Serializable pk, Serializable value)
{
if(instanceKey().isEmpty())
{
throw (new IllegalStateException("KEYED_HASH requires QQQ_AUDIT_HMAC_KEY"));
}
instanceOnly.set(true);
try
{
return (ClassifiedValueTokens.forRecordField(table, field, pk, value));
}
finally
{
instanceOnly.remove();
}
}
}
@@ -0,0 +1,571 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.security.SecureRandom;
import java.time.Clock;
import java.util.EnumSet;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Supplier;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Configuration for policy resolution and bounded asynchronous access logging.
*******************************************************************************/
public class SecuredDataQBitConfig implements QBitConfig
{
private static final byte[] JVM_KEY = new SecureRandom().generateSeed(32);
private Map<String, String> defaultTierByClassification = new HashMap<>(Map.of("CONFIDENTIAL", "FULL", "PII", "MASKED", "PHI", "MASKED"));
private int anomalyDailyMedianMultiplier = 3;
private int anomalyMinimumDailyReveals = 10;
private int anomalyDistinctRecordsPerHour = 20;
private int anomalyExactMatchesPerHour = 50;
private String tablePathPrefix = "/app";
private String schedulerName;
private Integer backupRetentionDays;
private int wipeIntervalSeconds = 86400;
private com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface archiveSink;
/*******************************************************************************
** Known CNPG/WAL backup retention; null means unknown.
*******************************************************************************/
public Integer getBackupRetentionDays()
{
return (backupRetentionDays);
}
/*******************************************************************************
** Configure the physical backup recovery window.
*******************************************************************************/
public SecuredDataQBitConfig withBackupRetentionDays(Integer days)
{
if(days != null && days < 0)
{
throw (new IllegalArgumentException("Backup retention cannot be negative"));
}
backupRetentionDays = days;
return (this);
}
/*******************************************************************************
** Optional archive adapter.
*******************************************************************************/
public com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface getArchiveSink()
{
return (archiveSink);
}
/*******************************************************************************
** Supply a durable encrypted-object sink.
*******************************************************************************/
public SecuredDataQBitConfig withArchiveSink(com.kingsrook.qbits.secureddata.retention.ArchiveSinkInterface sink)
{
archiveSink = sink;
return (this);
}
/*******************************************************************************
** Daily by default.
*******************************************************************************/
public int getWipeIntervalSeconds()
{
return (wipeIntervalSeconds);
}
/*******************************************************************************
** Configure the host scheduler interval.
*******************************************************************************/
public SecuredDataQBitConfig withWipeIntervalSeconds(int seconds)
{
if(seconds < 1)
{
throw (new IllegalArgumentException("Wipe interval must be positive"));
}
wipeIntervalSeconds = seconds;
return (this);
}
private String backendName = "memory";
private Clock clock = Clock.systemUTC();
private int accessLogQueueCapacity = 4096;
private long accessLogFlushIntervalMillis = 250;
private boolean registerShutdownHook = true;
private int exactMatchPerHour = 20;
private int missAlertConsecutive = 5;
private int distinctValueAlert = 15;
private boolean minEntropyEnabled = true;
private boolean nameHeuristics = false;
private Set<DataClassification> viewLogClassifications = EnumSet.of(DataClassification.PII, DataClassification.PHI);
private Supplier<byte[]> fingerprintKeySupplier = () -> JVM_KEY.clone();
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
/*******************************************************************************
** Scheduler to run the once-per-minute break-glass expiry job.
*******************************************************************************/
public String getSchedulerName()
{
return (schedulerName);
}
/*******************************************************************************
** Configure the host scheduler; null leaves manual scheduling to the host.
*******************************************************************************/
public SecuredDataQBitConfig withSchedulerName(String schedulerName)
{
this.schedulerName = schedulerName;
return (this);
}
/*******************************************************************************
** Get backendName.
*******************************************************************************/
public String getBackendName()
{
return (backendName);
}
/*******************************************************************************
** Configure backendName.
*******************************************************************************/
public SecuredDataQBitConfig withBackendName(String backendName)
{
this.backendName = backendName;
return (this);
}
/*******************************************************************************
** Get clock.
*******************************************************************************/
public Clock getClock()
{
return (clock);
}
/*******************************************************************************
** Configure clock.
*******************************************************************************/
public SecuredDataQBitConfig withClock(Clock clock)
{
this.clock = clock;
return (this);
}
/*******************************************************************************
** Get accessLogQueueCapacity.
*******************************************************************************/
public int getAccessLogQueueCapacity()
{
return (accessLogQueueCapacity);
}
/*******************************************************************************
** Configure accessLogQueueCapacity.
*******************************************************************************/
public SecuredDataQBitConfig withAccessLogQueueCapacity(int accessLogQueueCapacity)
{
this.accessLogQueueCapacity = accessLogQueueCapacity;
return (this);
}
/*******************************************************************************
** Get accessLogFlushIntervalMillis.
*******************************************************************************/
public long getAccessLogFlushIntervalMillis()
{
return (accessLogFlushIntervalMillis);
}
/*******************************************************************************
** Whether the access logger registers a JVM shutdown hook that drains
** accepted rows (default true). Hosts that call SecuredDataRuntime.close()
** from their own shutdown sequence turn it off.
*******************************************************************************/
public boolean getRegisterShutdownHook()
{
return (registerShutdownHook);
}
/*******************************************************************************
** Configure registerShutdownHook.
*******************************************************************************/
public SecuredDataQBitConfig withRegisterShutdownHook(boolean registerShutdownHook)
{
this.registerShutdownHook = registerShutdownHook;
return (this);
}
/*******************************************************************************
** Configure accessLogFlushIntervalMillis.
*******************************************************************************/
public SecuredDataQBitConfig withAccessLogFlushIntervalMillis(long accessLogFlushIntervalMillis)
{
this.accessLogFlushIntervalMillis = accessLogFlushIntervalMillis;
return (this);
}
/*******************************************************************************
** Get exactMatchPerHour.
*******************************************************************************/
public int getExactMatchPerHour()
{
return (exactMatchPerHour);
}
/*******************************************************************************
** Configure exactMatchPerHour.
*******************************************************************************/
public SecuredDataQBitConfig withExactMatchPerHour(int exactMatchPerHour)
{
this.exactMatchPerHour = exactMatchPerHour;
return (this);
}
/*******************************************************************************
** Get missAlertConsecutive.
*******************************************************************************/
public int getMissAlertConsecutive()
{
return (missAlertConsecutive);
}
/*******************************************************************************
** Configure missAlertConsecutive.
*******************************************************************************/
public SecuredDataQBitConfig withMissAlertConsecutive(int missAlertConsecutive)
{
this.missAlertConsecutive = missAlertConsecutive;
return (this);
}
/*******************************************************************************
** Get distinctValueAlert.
*******************************************************************************/
public int getDistinctValueAlert()
{
return (distinctValueAlert);
}
/*******************************************************************************
** Configure distinctValueAlert.
*******************************************************************************/
public SecuredDataQBitConfig withDistinctValueAlert(int distinctValueAlert)
{
this.distinctValueAlert = distinctValueAlert;
return (this);
}
/*******************************************************************************
** Get minEntropyEnabled.
*******************************************************************************/
public boolean getMinEntropyEnabled()
{
return (minEntropyEnabled);
}
/*******************************************************************************
** Configure minEntropyEnabled.
*******************************************************************************/
public SecuredDataQBitConfig withMinEntropyEnabled(boolean minEntropyEnabled)
{
this.minEntropyEnabled = minEntropyEnabled;
return (this);
}
/*******************************************************************************
** Get nameHeuristics.
*******************************************************************************/
public boolean getNameHeuristics()
{
return (nameHeuristics);
}
/*******************************************************************************
** Configure nameHeuristics.
*******************************************************************************/
public SecuredDataQBitConfig withNameHeuristics(boolean nameHeuristics)
{
this.nameHeuristics = nameHeuristics;
return (this);
}
/*******************************************************************************
** Get viewLogClassifications.
*******************************************************************************/
public Set<DataClassification> getViewLogClassifications()
{
return (viewLogClassifications);
}
/*******************************************************************************
** Configure viewLogClassifications.
*******************************************************************************/
public SecuredDataQBitConfig withViewLogClassifications(Set<DataClassification> viewLogClassifications)
{
this.viewLogClassifications = viewLogClassifications;
return (this);
}
/*******************************************************************************
** Get fingerprintKeySupplier.
*******************************************************************************/
public Supplier<byte[]> getFingerprintKeySupplier()
{
return (fingerprintKeySupplier);
}
/*******************************************************************************
** Configure fingerprintKeySupplier.
*******************************************************************************/
public SecuredDataQBitConfig withFingerprintKeySupplier(Supplier<byte[]> fingerprintKeySupplier)
{
this.fingerprintKeySupplier = fingerprintKeySupplier;
return (this);
}
/*******************************************************************************
** Get tableMetaDataCustomizer.
*******************************************************************************/
@Override
public MetaDataCustomizerInterface<QTableMetaData> getTableMetaDataCustomizer()
{
return (tableMetaDataCustomizer);
}
/*******************************************************************************
** Configure tableMetaDataCustomizer.
*******************************************************************************/
public SecuredDataQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer)
{
this.tableMetaDataCustomizer = tableMetaDataCustomizer;
return (this);
}
/*******************************************************************************
** Default tier for a classification.
*******************************************************************************/
public SecuredDataQBitConfig withDefaultTier(DataClassification classification, FieldAccessTier tier)
{
defaultTierByClassification.put(classification.name(), tier.name());
return (this);
}
/*******************************************************************************
** Compatibility getter for scaffold configuration.
*******************************************************************************/
public Map<String, String> getDefaultTierByClassification()
{
return (Map.copyOf(defaultTierByClassification));
}
/*******************************************************************************
** Replace configured defaults; missing restricted classifications fail closed.
*******************************************************************************/
public SecuredDataQBitConfig withDefaultTierByClassification(Map<String, String> tiers)
{
tiers.forEach((classification, tier) ->
{
DataClassification.valueOf(classification);
FieldAccessTier.valueOf(tier);
});
defaultTierByClassification = new HashMap<>(tiers);
return (this);
}
/*******************************************************************************
** Resolve the configured default.
*******************************************************************************/
public FieldAccessTier defaultTier(DataClassification classification)
{
return (FieldAccessTier.valueOf(defaultTierByClassification.getOrDefault(classification.name(), "MASKED")));
}
/*******************************************************************************
** Reject invalid infrastructure and guardrail settings at startup.
*******************************************************************************/
public void validate()
{
if(backendName == null || backendName.isBlank() || accessLogQueueCapacity < 1 || accessLogFlushIntervalMillis < 1
|| anomalyDailyMedianMultiplier < 1 || anomalyMinimumDailyReveals < 1 || anomalyDistinctRecordsPerHour < 1
|| anomalyExactMatchesPerHour < 1 || tablePathPrefix == null || !tablePathPrefix.startsWith("/") || tablePathPrefix.startsWith("//")
|| exactMatchPerHour < 0 || missAlertConsecutive < 0 || distinctValueAlert < 0)
{
throw (new IllegalArgumentException("Invalid secured-data configuration"));
}
Objects.requireNonNull(clock);
Objects.requireNonNull(viewLogClassifications);
Objects.requireNonNull(fingerprintKeySupplier);
}
/*******************************************************************************
** Get anomalyDailyMedianMultiplier.
*******************************************************************************/
public int getAnomalyDailyMedianMultiplier()
{
return (anomalyDailyMedianMultiplier);
}
/*******************************************************************************
** Configure anomalyDailyMedianMultiplier.
*******************************************************************************/
public SecuredDataQBitConfig withAnomalyDailyMedianMultiplier(int value)
{
this.anomalyDailyMedianMultiplier = value;
return (this);
}
/*******************************************************************************
** Get anomalyMinimumDailyReveals.
*******************************************************************************/
public int getAnomalyMinimumDailyReveals()
{
return (anomalyMinimumDailyReveals);
}
/*******************************************************************************
** Configure anomalyMinimumDailyReveals.
*******************************************************************************/
public SecuredDataQBitConfig withAnomalyMinimumDailyReveals(int value)
{
this.anomalyMinimumDailyReveals = value;
return (this);
}
/*******************************************************************************
** Get anomalyDistinctRecordsPerHour.
*******************************************************************************/
public int getAnomalyDistinctRecordsPerHour()
{
return (anomalyDistinctRecordsPerHour);
}
/*******************************************************************************
** Configure anomalyDistinctRecordsPerHour.
*******************************************************************************/
public SecuredDataQBitConfig withAnomalyDistinctRecordsPerHour(int value)
{
this.anomalyDistinctRecordsPerHour = value;
return (this);
}
/*******************************************************************************
** Get anomalyExactMatchesPerHour.
*******************************************************************************/
public int getAnomalyExactMatchesPerHour()
{
return (anomalyExactMatchesPerHour);
}
/*******************************************************************************
** Configure anomalyExactMatchesPerHour.
*******************************************************************************/
public SecuredDataQBitConfig withAnomalyExactMatchesPerHour(int value)
{
this.anomalyExactMatchesPerHour = value;
return (this);
}
/*******************************************************************************
** Get tablePathPrefix.
*******************************************************************************/
public String getTablePathPrefix()
{
return (tablePathPrefix);
}
/*******************************************************************************
** Configure tablePathPrefix.
*******************************************************************************/
public SecuredDataQBitConfig withTablePathPrefix(String value)
{
this.tablePathPrefix = value;
return (this);
}
}
@@ -0,0 +1,219 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTableCustomizers;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.audits.AuditValueProtectionMetaData;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySourceMetaData;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverMetaData;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Entry point for secured-data policy, classification and access logging.
*******************************************************************************/
public class SecuredDataQBitProducer implements QBitMetaDataProducer<SecuredDataQBitConfig>
{
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-secured-data";
public static final String VERSION = "0.1.0";
private SecuredDataQBitConfig qBitConfig;
private String namespace;
/*******************************************************************************
** Fluent setter for the config.
*******************************************************************************/
public SecuredDataQBitProducer withQBitConfig(SecuredDataQBitConfig config)
{
this.qBitConfig = config;
return (this);
}
/*******************************************************************************
** Getter for the config.
*******************************************************************************/
@Override
public SecuredDataQBitConfig getQBitConfig()
{
return (qBitConfig);
}
/*******************************************************************************
** Fluent setter for the namespace.
*******************************************************************************/
public SecuredDataQBitProducer withNamespace(String namespace)
{
this.namespace = namespace;
return (this);
}
/*******************************************************************************
** Getter for the namespace.
*******************************************************************************/
@Override
public String getNamespace()
{
return (namespace);
}
/*******************************************************************************
** Identity and configuration of this qbit.
*******************************************************************************/
@Override
public QBitMetaData getQBitMetaData()
{
return (new QBitMetaData().withGroupId(GROUP_ID).withArtifactId(ARTIFACT_ID).withVersion(VERSION).withNamespace(namespace).withConfig(qBitConfig));
}
/*******************************************************************************
** Package-scanned producers have no config: skip them.
*******************************************************************************/
@Override
public boolean isEnabled()
{
return (qBitConfig != null);
}
/*******************************************************************************
** Register tables, permissions, runtime and the optional core enforcement bridge.
*******************************************************************************/
@Override
public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException
{
qBitConfig.validate();
if(qInstance.getSupplementalMetaData(SecuredDataRuntime.NAME) != null)
{
throw (new IllegalStateException("Only one secured-data QBit may be registered per instance"));
}
QBitMetaData qBit = getQBitMetaData();
qInstance.addQBit(qBit);
MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();
SecuredDataTables.possibleValueSources(qInstance).forEach(output::add);
Map<String, QTableMetaData> tables = new LinkedHashMap<>();
for(QTableMetaData definition : SecuredDataTables.tables(qBitConfig))
{
QTableMetaData table = qBitConfig.getTableMetaDataCustomizer() == null ? definition
: qBitConfig.getTableMetaDataCustomizer().customizeMetaData(qInstance, definition);
for(TableCustomizers role : List.of(TableCustomizers.PRE_INSERT_RECORD, TableCustomizers.PRE_UPDATE_RECORD, TableCustomizers.PRE_DELETE_RECORD,
TableCustomizers.POST_INSERT_RECORD, TableCustomizers.POST_UPDATE_RECORD, TableCustomizers.POST_DELETE_RECORD))
{
table.withCustomizer(role, new QCodeReference(SecuredDataTableCustomizers.class));
}
table.setSourceQBitName(qBit.getName());
tables.put(table.getName(), table);
output.add(table);
}
output.add(com.kingsrook.qbits.secureddata.app.SecuredDataApp.app(tables));
output.add(com.kingsrook.qbits.secureddata.app.SecuredDataApp.widget());
com.kingsrook.qbits.secureddata.processes.BreakGlassProcesses.processes(qBitConfig).forEach(output::add);
com.kingsrook.qbits.secureddata.retention.WipeProcesses.processes(qBitConfig).forEach(output::add);
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.backups());
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.markersJoin());
output.add(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.markers());
output.add(SecuredDataTables.revealsJoin());
output.add(SecuredDataTables.revealsWidget());
output.add(new com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType().withName(com.kingsrook.qbits.secureddata.metadata.RetentionTables.KEY_LOCK));
output.addSelfToInstance(qInstance);
SecuredDataMetadata.register(qInstance);
com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension.register(qInstance);
if(qBitConfig.getNameHeuristics())
{
seedHeuristics(qInstance);
}
SecuredDataRuntime runtime = new SecuredDataRuntime(qInstance, qBitConfig);
runtime.addSelfToInstance(qInstance);
try
{
FieldAccessResolverMetaData.ofOrWithNew(qInstance).withResolverInstance(runtime.resolver());
ClassifiedValueKeySourceMetaData.ofOrWithNew(qInstance).withKeySourceInstance(runtime.keys());
AuditValueProtectionMetaData.ofOrWithNew(qInstance).withProtectorInstance(new SecuredDataAuditProtector());
}
catch(RuntimeException e)
{
runtime.close();
throw e;
}
return (new MetaDataProducerMultiOutput());
}
/*******************************************************************************
** Opt-in common-name defaults, preserving existing field/table classifications.
*******************************************************************************/
private static void seedHeuristics(QInstance instance)
{
for(QTableMetaData table : instance.getTables().values())
{
if(SecuredDataTables.ALL.contains(table.getName()))
{
continue;
}
for(QFieldMetaData field : table.getFields().values())
{
if(SecuredDataMetadata.classification(table, field) != null || SecuredDataMetadata.classification(table, null) != null)
{
continue;
}
String name = field.getName().toLowerCase(Locale.ROOT);
if(name.startsWith("diagnosis"))
{
SecuredDataMetadata.classifyField(field, DataClassification.PHI);
}
else if(List.of("ssn", "dob", "date_of_birth", "email", "phone").contains(name))
{
SecuredDataMetadata.classifyField(field, DataClassification.PII);
}
}
}
}
}
@@ -0,0 +1,508 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Instant;
import java.util.Base64;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.TreeSet;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessDecision;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessResolverInterface;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Seven-step policy resolution with instance-versioned, bounded per-user state.
*******************************************************************************/
public class SecuredDataResolver implements FieldAccessResolverInterface
{
private final SecuredDataRuntime runtime;
private final byte[] fingerprintKey;
private final ThreadLocal<Boolean> resolving = ThreadLocal.withInitial(() -> false);
private final Map<String, RevealCache> reveals = new HashMap<>();
/*******************************************************************************
** Bounded per-user grant cache, with exact read-time expiry.
*******************************************************************************/
private record RevealCache(Instant until, List<QRecord> rows)
{
}
private long cachedVersion;
private List<QRecord> policies = List.of();
private final Map<String, UserState> users = new HashMap<>();
/*******************************************************************************
** Cached policy matches and break-glass rows. Expiration is checked on every use.
*******************************************************************************/
private record UserState(List<QRecord> policies, List<QRecord> breakGlass)
{
}
/*******************************************************************************
** Zero thresholds disable that guardrail; positive thresholds combine by minimum.
*******************************************************************************/
public record Guardrails(int exactMatchPerHour, int missAlertConsecutive, int distinctValueAlert, boolean minEntropyEnabled)
{
}
/*******************************************************************************
** Capture the supplied key once; default keys are shared within this JVM.
*******************************************************************************/
public SecuredDataResolver(SecuredDataRuntime runtime)
{
this.runtime = runtime;
this.fingerprintKey = Objects.requireNonNull(runtime.config().getFingerprintKeySupplier().get()).clone();
if(fingerprintKey.length < 16)
{
throw (new IllegalArgumentException("Fingerprint keys must contain at least 16 bytes"));
}
}
/*******************************************************************************
** Resolve classification, grants, policies, channel caps and entropy.
*******************************************************************************/
@Override
public FieldAccessDecision resolve(QTableMetaData table, QFieldMetaData field)
{
if(SystemWrites.active() || QContext.getQSession() instanceof QSystemUserSession
|| Objects.equals(table.getPrimaryKeyField(), field.getName()))
{
return (FieldAccessDecision.FULL);
}
if(resolving.get())
{
throw (new IllegalStateException("Recursive secured-data resolution outside a system lookup"));
}
resolving.set(true);
try
{
DataClassification classification = SecuredDataClassifications.effective(table, field.getName());
if(classification == null || classification == DataClassification.PUBLIC)
{
return (FieldAccessDecision.FULL);
}
QSession session = QContext.getQSession();
if(SecuredDataTables.ALL.contains(table.getName()) && session != null && session.hasPermission(SecuredDataPermissions.ADMIN))
{
return (FieldAccessDecision.FULL);
}
List<QRecord> matches = matching(table, classification);
FieldAccessTier tier = null;
for(QRecord row : matches)
{
if(blank(row.getValueString("client_channel")))
{
FieldAccessTier candidate = FieldAccessTier.valueOf(row.getValueString("tier"));
tier = tier == null ? candidate : tier.leastRestrictive(candidate);
}
}
if(tier == null)
{
tier = runtime.config().defaultTier(classification);
}
if(session != null && SecuredDataMetadata.hasFeature(table, SecuredDataPermissions.REVEAL))
{
tier = tier.leastRestrictive(FieldAccessTier.REVEALABLE);
}
if(activeBreakGlass(table.getName(), classification) != null
|| (session != null && SecuredDataMetadata.hasFeature(table, SecuredDataPermissions.FULL)))
{
tier = FieldAccessTier.FULL;
}
for(QRecord row : matches)
{
if(!blank(row.getValueString("client_channel")))
{
tier = tier.mostRestrictive(FieldAccessTier.valueOf(row.getValueString("tier")));
}
}
QRecord override = SecuredDataClassifications.row(table, field.getName());
String style = override == null ? null : override.getValueString("mask_style");
boolean exactAllowed = !guardrails(matches).minEntropyEnabled() || !lowCardinality(field, override);
return (new FieldAccessDecision(tier, blank(style) ? null : FieldMaskStyle.valueOf(style), "Secured-data policy", exactAllowed));
}
finally
{
resolving.remove();
}
}
/*******************************************************************************
** The field's matching policies also drive enumeration controls.
*******************************************************************************/
public Guardrails guardrails(QTableMetaData table, String fieldName)
{
return (guardrails(matching(table, SecuredDataClassifications.effective(table, fieldName))));
}
/*******************************************************************************
** Most restrictive enabled setting across matching rows; config when none match.
*******************************************************************************/
private Guardrails guardrails(List<QRecord> matches)
{
SecuredDataQBitConfig config = runtime.config();
if(matches.isEmpty())
{
return (new Guardrails(config.getExactMatchPerHour(), config.getMissAlertConsecutive(), config.getDistinctValueAlert(), config.getMinEntropyEnabled()));
}
int rate = 0;
int misses = 0;
int distinct = 0;
boolean entropy = false;
for(QRecord row : matches)
{
rate = minimumEnabled(rate, setting(row, "exact_match_per_hour", config.getExactMatchPerHour()));
misses = minimumEnabled(misses, setting(row, "miss_alert_consecutive", config.getMissAlertConsecutive()));
distinct = minimumEnabled(distinct, setting(row, "distinct_value_alert", config.getDistinctValueAlert()));
entropy |= row.getValueBoolean("min_entropy_enabled") == null ? config.getMinEntropyEnabled() : row.getValueBoolean("min_entropy_enabled");
}
return (new Guardrails(rate, misses, distinct, entropy));
}
/*******************************************************************************
** Read an optional numeric setting.
*******************************************************************************/
private static int setting(QRecord row, String name, int fallback)
{
Integer value = row.getValueInteger(name);
return (value == null ? fallback : value);
}
/*******************************************************************************
** Zero means off, so an enabled row always wins over a disabled row.
*******************************************************************************/
private static int minimumEnabled(int left, int right)
{
return (left == 0 ? right : right == 0 ? left : Math.min(left, right));
}
/*******************************************************************************
** Explicit true/false wins over the field type/name heuristic.
*******************************************************************************/
static boolean lowCardinality(QFieldMetaData field, QRecord override)
{
Boolean explicit = override == null ? null : override.getValueBoolean("low_cardinality");
if(explicit != null)
{
return (explicit);
}
QFieldType type = field.getType();
String name = (field.getName() + " " + field.getLabel()).replaceAll("([a-z])([A-Z])", "$1 $2").toLowerCase(Locale.ROOT);
return (type == QFieldType.BOOLEAN || type == QFieldType.DATE || type == QFieldType.DATE_TIME
|| field.getPossibleValueSourceName() != null || field.getInlinePossibleValueSource() != null
|| (type == QFieldType.INTEGER && name.contains("year"))
|| name.matches(".*(?:^|[^a-z])(state|region|province|zip3)(?:[^a-z]|$).*")
|| (type == QFieldType.STRING && field.getMaxLength() != null && field.getMaxLength() < 6));
}
/*******************************************************************************
** Policies for this user, classification, table and current normalized channel.
*******************************************************************************/
private List<QRecord> matching(QTableMetaData table, DataClassification classification)
{
String channel = SecuredDataMetadata.channel(QContext.getQSession());
return (state().policies().stream().filter(row -> classification != null && classification.name().equals(row.getValueString("classification")))
.filter(row -> blank(row.getValueString("table_name")) || table.getName().equals(row.getValueString("table_name")))
.filter(row -> blank(row.getValueString("client_channel")) || Objects.equals(channel, row.getValueString("client_channel"))).toList());
}
/*******************************************************************************
** Granting rows only: channel rows cap tiers and never grant reveal settings.
*******************************************************************************/
public List<QRecord> revealPolicies(QTableMetaData table, String fieldName)
{
return (matching(table, SecuredDataClassifications.effective(table, fieldName)).stream()
.filter(row -> blank(row.getValueString("client_channel")) && "REVEALABLE".equals(row.getValueString("tier"))).toList());
}
/*******************************************************************************
** Refresh policy rows once per version and matched state per relevant permissions.
*******************************************************************************/
private synchronized UserState state()
{
long version = runtime.policyVersion();
if(cachedVersion != version)
{
policies = List.copyOf(SystemWrites.query(SecuredDataTables.POLICY, new QQueryFilter()));
users.clear();
cachedVersion = version;
}
QSession session = QContext.getQSession();
TreeSet<String> held = new TreeSet<>();
for(QRecord policy : policies)
{
String permission = policy.getValueString("permission_name");
if(session != null && session.hasPermission(permission))
{
held.add(permission);
}
}
String key = JsonUtils.toJson(List.of(userId(), held));
UserState state = users.get(key);
if(state == null)
{
if(users.size() >= 1024)
{
users.clear();
}
List<QRecord> matching = policies.stream().filter(row -> Boolean.TRUE.equals(row.getValueBoolean("is_active")))
.filter(row -> held.contains(row.getValueString("permission_name"))).toList();
List<QRecord> breaks = SystemWrites.query(SecuredDataTables.BREAK_GLASS, new QQueryFilter()
.withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, userId())));
state = new UserState(matching, List.copyOf(breaks));
users.put(key, state);
}
return (state);
}
/*******************************************************************************
** Exact expiry is checked at use time, independent of the future expiry job.
*******************************************************************************/
public List<QRecord> activeBreakGlass()
{
Instant now = runtime.config().getClock().instant();
return (state().breakGlass().stream().filter(row -> "ACTIVE".equals(row.getValueString("status")))
.filter(row -> row.getValueInstant("expires_at") != null && row.getValueInstant("expires_at").isAfter(now)).toList());
}
/*******************************************************************************
** A break-glass grant must cover both the table and classification.
*******************************************************************************/
QRecord activeBreakGlass(String table, DataClassification classification)
{
return (activeBreakGlass().stream().filter(row -> blank(row.getValueString("table_name")) || table.equals(row.getValueString("table_name")))
.filter(row -> DataClassification.valueOf(Objects.requireNonNullElse(row.getValueString("classification_ceiling"), "PHI")).isAtLeast(classification))
.findFirst().orElse(null));
}
/*******************************************************************************
** Stable across unrelated permissions/records; includes channel and exact expiry.
*******************************************************************************/
@Override
public synchronized String getAccessFingerprint()
{
state();
TreeSet<String> relevant = new TreeSet<>();
QSession session = QContext.getQSession();
for(QRecord row : policies)
{
String permission = row.getValueString("permission_name");
if(session != null && session.hasPermission(permission))
{
relevant.add(permission);
}
}
for(QTableMetaData table : QContext.getQInstance().getTables().values())
{
String base = SecuredDataMetadata.permissionBase(table);
for(String suffix : List.of(SecuredDataPermissions.FULL, SecuredDataPermissions.REVEAL))
{
if(session != null && session.hasPermission(base + "." + suffix))
{
relevant.add(base + "." + suffix);
}
}
}
List<String> breaks = activeBreakGlass().stream().map(row -> row.getValueString("id")).sorted().toList();
String payload = JsonUtils.toJson(List.of(userId(), relevant, runtime.policyVersion(), runtime.classificationVersion(), breaks,
Objects.requireNonNullElse(SecuredDataMetadata.channel(session), "")));
try
{
return (Base64.getUrlEncoder().withoutPadding().encodeToString(MessageDigest.getInstance("SHA-256").digest(payload.getBytes(StandardCharsets.UTF_8))));
}
catch(java.security.NoSuchAlgorithmException e)
{
throw (new IllegalStateException(e));
}
}
/*******************************************************************************
** Existing grants are read as system; phase (c) owns their creation.
*******************************************************************************/
@Override
public boolean isRevealActive(String tableName, String fieldName, Serializable primaryKey)
{
synchronized(reveals)
{
String user = userId();
Instant now = runtime.config().getClock().instant();
RevealCache cached = reveals.get(user);
if(cached == null || !cached.until().isAfter(now))
{
if(reveals.size() >= 1024)
{
reveals.clear();
}
cached = new RevealCache(now.plusSeconds(5), SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()
.withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, user))
.withCriteria(new QFilterCriteria("expires_at", QCriteriaOperator.GREATER_THAN, now))));
reveals.put(user, cached);
}
return (cached.rows().stream().anyMatch(row -> tableName.equals(row.getValueString("table_name"))
&& fieldName.equals(row.getValueString("field_name")) && String.valueOf(primaryKey).equals(row.getValueString("record_id"))
&& row.getValueInstant("expires_at").isAfter(now)));
}
}
/*******************************************************************************
** Publish a newly created grant to all sessions on this instance immediately.
*******************************************************************************/
public void invalidateReveals()
{
synchronized(reveals)
{
reveals.clear();
}
}
/*******************************************************************************
** Logging never throws into a core read.
*******************************************************************************/
@Override
public void onAccess(FieldAccessEvent event)
{
runtime.logger().onAccess(event);
}
/*******************************************************************************
** Preflight enumeration limits; core reports the REFUSED outcome exactly once.
*******************************************************************************/
@Override
public void checkExactMatch(FieldAccessEvent event) throws QPermissionDeniedException
{
runtime.logger().checkExactMatch(event);
}
/*******************************************************************************
** A keyed, non-reversible search identifier. No raw value leaves this method.
*******************************************************************************/
@Override
public String fingerprintSearchValue(Serializable value)
{
try
{
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(fingerprintKey, "HmacSHA256"));
///////////////////////////////////////////////////////////////////////////////
// keep only 16 bits: enough to count distinct searched values in an hour, //
// far too few to confirm a guessed value (each fingerprint matches ~1 in //
// 65536 of the value space), so a persisted fingerprint is not a derivative //
// of the searched value that could be brute-forced with the key. //
///////////////////////////////////////////////////////////////////////////////
byte[] digest = mac.doFinal(JsonUtils.toJson(value).getBytes(StandardCharsets.UTF_8));
return (String.format("%02x%02x", digest[0], digest[1]));
}
catch(java.security.GeneralSecurityException e)
{
throw (new IllegalStateException("Unable to fingerprint search", e));
}
}
/*******************************************************************************
** Stable user identity for all of that user's sessions.
*******************************************************************************/
public static String userId()
{
QSession session = QContext.getQSession();
return (session == null || session.getUser() == null ? "anonymous" : Objects.requireNonNullElse(session.getUser().getIdReference(), "anonymous"));
}
/*******************************************************************************
** Treat empty optional selectors like null.
*******************************************************************************/
private static boolean blank(String value)
{
return (value == null || value.isBlank());
}
}
@@ -0,0 +1,196 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
/*******************************************************************************
** Instance-scoped services and invalidation clocks, never shared across applications.
*******************************************************************************/
public class SecuredDataRuntime implements QSupplementalInstanceMetaData, AutoCloseable
{
public static final String NAME = "securedDataRuntime";
private static final AtomicLong STARTUP_VERSION = new AtomicLong(System.currentTimeMillis());
private final AtomicLong classificationVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
private final AtomicLong policyVersion = new AtomicLong(STARTUP_VERSION.incrementAndGet());
private final transient SecuredDataQBitConfig config;
private final transient SecuredDataResolver resolver;
private final transient SecuredDataKeySource keys = new SecuredDataKeySource();
private final transient SecuredDataAccessLogger logger;
private final transient SecuredDataAnomalies anomalies;
private transient long cachedClassificationVersion;
private transient List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows = List.of();
/*******************************************************************************
** Construct services only after tables have been registered.
*******************************************************************************/
public SecuredDataRuntime(QInstance instance, SecuredDataQBitConfig config)
{
this.config = config;
this.resolver = new SecuredDataResolver(this);
this.anomalies = new SecuredDataAnomalies(config);
this.logger = new SecuredDataAccessLogger(instance, this);
}
/*******************************************************************************
** Supplemental metadata identity.
*******************************************************************************/
@Override
public String getName()
{
return (NAME);
}
/*******************************************************************************
** Services for the current QContext.
*******************************************************************************/
public static SecuredDataRuntime current()
{
return (QContext.getQInstance() == null ? null : QSupplementalInstanceMetaData.of(QContext.getQInstance(), NAME));
}
/*******************************************************************************
** Code configuration.
*******************************************************************************/
public SecuredDataQBitConfig config()
{
return (config);
}
/*******************************************************************************
** Resolver service.
*******************************************************************************/
public SecuredDataResolver resolver()
{
return (resolver);
}
/*******************************************************************************
** Logger service and drop counter.
*******************************************************************************/
public SecuredDataAccessLogger logger()
{
return (logger);
}
/*******************************************************************************
** Classification invalidation clock.
*******************************************************************************/
public long classificationVersion()
{
return (classificationVersion.get());
}
/*******************************************************************************
** Policy/break-glass invalidation clock.
*******************************************************************************/
public long policyVersion()
{
return (policyVersion.get());
}
/*******************************************************************************
** Post-commit customizers and host distributed invalidation call this.
*******************************************************************************/
public void invalidateClassifications()
{
classificationVersion.incrementAndGet();
}
/*******************************************************************************
** Post-commit customizers and host distributed invalidation call this.
*******************************************************************************/
public void invalidatePolicies()
{
policyVersion.incrementAndGet();
}
/*******************************************************************************
** Read classification rows once per version, always as the system.
*******************************************************************************/
synchronized List<com.kingsrook.qqq.backend.core.model.data.QRecord> classificationRows()
{
long version = classificationVersion();
if(cachedClassificationVersion != version)
{
classificationRows = List.copyOf(com.kingsrook.qbits.secureddata.support.SystemWrites.query(
com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.CLASSIFICATION, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter()));
cachedClassificationVersion = version;
}
return (classificationRows);
}
/*******************************************************************************
** Drain the audit queue during normal host shutdown.
*******************************************************************************/
@Override
public void close()
{
keys.close();
logger.close();
}
/*******************************************************************************
** Anomaly counters shared by synchronous reveals and queued searches.
*******************************************************************************/
public SecuredDataAnomalies anomalies()
{
return (anomalies);
}
/*******************************************************************************
** Instance-scoped audit keys.
*******************************************************************************/
public SecuredDataKeySource keys()
{
return (keys);
}
}
@@ -0,0 +1,90 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.secureddata.reveal.RevealOptions;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Caller-only banner and per-field reveal allowance summary.
*******************************************************************************/
public class StatusService
{
/*******************************************************************************
** Active session projection.
*******************************************************************************/
public record BreakGlassStatus(Serializable id, String reason, Instant expiresAt, String scope, String ceiling)
{
}
/*******************************************************************************
** Allowances differ by granting policy; null remaining means unlimited.
*******************************************************************************/
public record Allowance(String tableName, String fieldName, Integer remainingDailyAllowance)
{
}
/*******************************************************************************
** Status response.
*******************************************************************************/
public record Status(BreakGlassStatus breakGlass, int revealsUsedToday, List<Allowance> revealAllowances)
{
}
/*******************************************************************************
** Only readable revealable fields are included, without any record values.
*******************************************************************************/
public Status status() throws QException
{
List<QRecord> active = SecuredDataRuntime.current().resolver().activeBreakGlass();
QRecord row = active.isEmpty() ? null : active.getFirst();
BreakGlassStatus session = row == null ? null : new BreakGlassStatus(row.getValue("id"), row.getValueString("reason"), row.getValueInstant("expires_at"),
row.getValueString("table_name"), row.getValueString("classification_ceiling"));
RevealService service = new RevealService();
List<Allowance> allowances = new ArrayList<>();
for(QTableMetaData table : QContext.getQInstance().getTables().values())
{
for(QFieldMetaData field : table.getFields().values())
{
try
{
RevealOptions options = service.options(table.getName(), field.getName());
allowances.add(new Allowance(table.getName(), field.getName(), options.remainingDailyAllowance()));
}
catch(QPermissionDeniedException expected)
{
// This field offers no reveal operation to the caller.
}
}
}
return (new Status(session, service.usedToday(), List.copyOf(allowances)));
}
}
@@ -0,0 +1,88 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.app;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** One app and one home. Grants remain a child list; processes stay off the nav.
*******************************************************************************/
public final class SecuredDataApp
{
public static final String HOME_WIDGET = "securedDataHome";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataApp()
{
}
/*******************************************************************************
** Admin-only home spanning the host's full grid.
*******************************************************************************/
public static QWidgetMetaData widget()
{
return (new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType())
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false)
.withPermissionRules(SecuredDataPermissions.rules()));
}
/*******************************************************************************
** Exactly four navigation tables.
*******************************************************************************/
public static QAppMetaData app(Map<String, QTableMetaData> tables)
{
QAppMetaData app = new QAppMetaData().withName("secured_data").withLabel("Secured Data")
.withIcon(new QIcon("shield")).withPermissionRules(SecuredDataPermissions.rules()).withWidgets(List.of(HOME_WIDGET));
SecuredDataTables.MAIN.forEach(name -> app.withChild(tables.get(name)));
return (app);
}
/*******************************************************************************
** Standard HTML widget, following the webhooks home pattern.
*******************************************************************************/
public static class HomeRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
** Render real viewer-scoped data.
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
return (new RenderWidgetOutput(new RawHTML(null, SecuredDataHomeView.render(SecuredDataHomeData.load()))));
}
}
}
@@ -0,0 +1,203 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.app;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.actions.tables.AggregateAction;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.Aggregate;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.GroupBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByAggregate;
import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.QFilterOrderByGroupBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Real home data, read as the viewer. Counts and rankings aggregate in the backend.
*******************************************************************************/
public class SecuredDataHomeData
{
private final Map<String, Long> kpis = new LinkedHashMap<>();
private final List<Section> sections = new ArrayList<>();
/*******************************************************************************
** A full-width table and the exact filter behind its View all link.
*******************************************************************************/
public record Section(String title, String empty, List<String> fields, List<String> labels, List<QRecord> rows, String href)
{
}
/*******************************************************************************
** Six KPIs and six lists; no samples or hidden truncation of aggregate counts.
*******************************************************************************/
public static SecuredDataHomeData load() throws QException
{
SecuredDataHomeData data = new SecuredDataHomeData();
Instant now = SecuredDataRuntime.current().config().getClock().instant();
Instant week = now.minus(7, ChronoUnit.DAYS);
Instant month = now.minus(30, ChronoUnit.DAYS);
data.kpis.put("Reveals today", count(SecuredDataTables.ACCESS_LOG, events(now.truncatedTo(ChronoUnit.DAYS), now, "REVEAL")));
data.kpis.put("Active reveal grants", count(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()
.withCriteria(new QFilterCriteria("expires_at", QCriteriaOperator.GREATER_THAN, now))));
data.kpis.put("Active break-glass sessions", count(SecuredDataTables.BREAK_GLASS, new QQueryFilter()
.withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"))
.withCriteria(new QFilterCriteria("expires_at", QCriteriaOperator.GREATER_THAN, now))));
QQueryFilter anomalies = window(week, now).withCriteria(new QFilterCriteria("anomaly_flag", QCriteriaOperator.EQUALS, true));
data.kpis.put("Anomaly flags (7 days)", count(SecuredDataTables.ACCESS_LOG, anomalies));
QQueryFilter denied = window(week, now).withSubFilter(new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR)
.withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.EQUALS, "FILTER_DENIED"))
.withSubFilter(new QQueryFilter().withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.EQUALS, "EXACT_MATCH_FILTER"))
.withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "REFUSED"))));
data.kpis.put("Denied filters (7 days)", count(SecuredDataTables.ACCESS_LOG, denied));
data.kpis.put("Access-log queue drops", SecuredDataRuntime.current().logger().getDroppedCount());
data.recent("Recent reveals", "No reveals yet", events(null, now, "REVEAL"),
List.of("user_name", "table_name", "record_id", "field_name", "reason", "created_at", "client_channel"),
List.of("User", "Table", "Record", "Field", "Reason", "When", "Channel"));
data.top("Top revealers (30 days)", "No reveals in the last 30 days", "user_id", "User", events(month, now, "REVEAL"));
data.top("Top reasons (30 days)", "No reveal reasons in the last 30 days", "reason", "Reason", events(month, now, "REVEAL"));
data.recent("Break-glass events", "No break-glass events yet", events(null, now, "BREAK_GLASS_START", "BREAK_GLASS_END", "BREAK_GLASS_EXPIRED"),
List.of("user_name", "event_type", "reason", "created_at"), List.of("User", "Event", "Reason", "When"));
data.recent("Policy and classification changes", "No policy or classification changes yet", events(null, now, "POLICY_CHANGE", "CLASSIFICATION_CHANGE"),
List.of("user_name", "table_name", "record_id", "action_name", "created_at"), List.of("User", "Table", "Record", "Action", "When"));
data.recent("Anomaly flags", "No anomaly flags in the last 7 days", anomalies,
List.of("user_name", "event_type", "table_name", "field_name", "created_at"), List.of("User", "Event", "Table", "Field", "When"));
for(QRecord rule : new QueryAction().execute(new QueryInput(SecuredDataTables.RETENTION_RULE)).getRecords())
{
String warning = com.kingsrook.qbits.secureddata.retention.RetentionRules.backupWarning(rule);
if(warning != null)
{
data.backupWarnings.add(rule.getValueString("name") + ": " + warning);
}
}
return (data);
}
/*******************************************************************************
** Scalar counts preserve viewer permissions.
*******************************************************************************/
private static long count(String table, QQueryFilter filter) throws QException
{
return (new CountAction().execute(new CountInput(table).withFilter(filter)).getCount());
}
/*******************************************************************************
** Newest ten, with deterministic id ordering for equal timestamps.
*******************************************************************************/
private void recent(String title, String empty, QQueryFilter filter, List<String> fields, List<String> labels) throws QException
{
filter.withOrderBy(new QFilterOrderBy("created_at", false)).withOrderBy(new QFilterOrderBy("id", false));
String href = link(filter);
List<QRecord> rows = new QueryAction().execute(new QueryInput(SecuredDataTables.ACCESS_LOG).withFilter(filter.withLimit(10))).getRecords();
sections.add(new Section(title, empty, fields, labels, rows, href));
}
/*******************************************************************************
** Top ten complete counts, with stable alphabetical tie ordering.
*******************************************************************************/
private void top(String title, String empty, String field, String label, QQueryFilter filter) throws QException
{
String href = link(filter);
GroupBy group = new GroupBy(QFieldType.STRING, field);
Aggregate count = new Aggregate("id", AggregateOperator.COUNT);
filter.withOrderBy(new QFilterOrderByAggregate(count, false)).withOrderBy(new QFilterOrderByGroupBy(group));
var results = new AggregateAction().execute(new AggregateInput(SecuredDataTables.ACCESS_LOG).withFilter(filter)
.withGroupBy(group).withAggregate(count).withLimit(10)).getResults();
List<QRecord> rows = results.stream().map(result -> new QRecord().withValue(field, result.getGroupByValue(group))
.withValue("count", result.getAggregateValue(count))).toList();
sections.add(new Section(title, empty, List.of(field, "count"), List.of(label, "Reveals"), rows, href));
}
/*******************************************************************************
** Next's table filter wire format, URL encoded; no widget row limit in the link.
*******************************************************************************/
private static String link(QQueryFilter filter)
{
return (SecuredDataRuntime.current().config().getTablePathPrefix() + "/" + SecuredDataTables.ACCESS_LOG
+ "?filter=" + URLEncoder.encode(JsonUtils.toJson(filter), StandardCharsets.UTF_8));
}
/*******************************************************************************
** Event filter scoped to a time window.
*******************************************************************************/
private static QQueryFilter events(Instant from, Instant now, String... events)
{
return (window(from, now).withCriteria(new QFilterCriteria("event_type", QCriteriaOperator.IN, java.util.Arrays.asList(events))));
}
/*******************************************************************************
** Exclude future timestamps from all home activity windows.
*******************************************************************************/
private static QQueryFilter window(Instant from, Instant now)
{
QQueryFilter filter = new QQueryFilter().withCriteria(new QFilterCriteria("created_at", QCriteriaOperator.LESS_THAN_OR_EQUALS, now));
if(from != null)
{
filter.withCriteria(new QFilterCriteria("created_at", QCriteriaOperator.GREATER_THAN_OR_EQUALS, from));
}
return (filter);
}
/*******************************************************************************
** KPI row.
*******************************************************************************/
public Map<String, Long> getKpis()
{
return (kpis);
}
/*******************************************************************************
** Full-width lists in display order.
*******************************************************************************/
public List<Section> getSections()
{
return (sections);
}
private final List<String> backupWarnings = new ArrayList<>();
/*******************************************************************************
** One warning for each policy outlived by known physical backups.
*******************************************************************************/
public List<String> getBackupWarnings()
{
return (backupWarnings);
}
}
@@ -0,0 +1,105 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.app;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.util.Locale;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Inline styles survive Next's sanitizer. Full-width labelled card rows wrap at
** narrow widths; no inner scrollers. Inherited host text and theme border tokens.
*******************************************************************************/
public final class SecuredDataHomeView
{
private static final DateTimeFormatter WHEN = DateTimeFormatter.ofPattern("MMM d, HH:mm 'UTC'", Locale.ENGLISH).withZone(ZoneOffset.UTC);
private static final String GRID = "display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,10rem),1fr));gap:.75rem;min-width:0;";
private static final String CARD = "border:1px solid var(--qqq-grid-border,currentColor);border-radius:.5rem;padding:1rem;min-width:0;";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataHomeView()
{
}
/*******************************************************************************
** KPI row followed by six full-width tables, never side by side.
*******************************************************************************/
public static String render(SecuredDataHomeData data)
{
StringBuilder html = new StringBuilder("<div style=\"display:flex;flex-direction:column;gap:1rem;min-width:0;color:inherit\">");
html.append("<div aria-label=\"Secured data metrics\" style=\"").append(GRID).append("\">");
data.getKpis().forEach((label, value) -> html.append("<div style=\"").append(CARD)
.append("\"><div style=\"font-size:1.6rem;font-weight:600\">").append(value)
.append("</div><div>").append(esc(label)).append("</div></div>"));
html.append("</div>");
for(String warning : data.getBackupWarnings())
{
html.append("<p role='alert'>").append(esc(warning)).append("</p>");
}
for(SecuredDataHomeData.Section section : data.getSections())
{
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0;font-size:1rem\">")
.append(esc(section.title())).append("</h3><p><a style=\"color:inherit\" aria-label=\"View all ")
.append(esc(section.title())).append("\" href=\"")
.append(esc(section.href())).append("\">View all</a></p>");
if(section.rows().isEmpty())
{
html.append("<p>").append(esc(section.empty())).append("</p>");
}
for(QRecord row : section.rows())
{
html.append("<div style=\"").append(GRID).append("border-top:1px solid var(--qqq-grid-border,currentColor);padding:.75rem 0\">");
for(int i = 0; i < section.fields().size(); i++)
{
String field = section.fields().get(i);
String value = row.getValueString(field);
boolean date = "created_at".equals(field);
if(date && row.getValueInstant(field) != null)
{
value = WHEN.format(row.getValueInstant(field));
}
if(value == null || value.isBlank())
{
value = "—";
}
html.append("<div style=\"min-width:0\"><span style=\"display:block;font-size:.75rem;opacity:.7\">")
.append(esc(section.labels().get(i))).append("</span><span title=\"").append(esc(value))
.append("\" style=\"display:block;overflow:hidden;text-overflow:ellipsis;white-space:nowrap\">")
.append(esc(value)).append("</span></div>");
}
html.append("</div>");
}
html.append("</section>");
}
return (html.append("</div>").toString());
}
/*******************************************************************************
** Escape every text, tooltip and URL attribute.
*******************************************************************************/
private static String esc(String value)
{
return (value.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace("\"", "&quot;").replace("'", "&#39;"));
}
}
@@ -0,0 +1,243 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.breakglass;
import java.io.Serializable;
import java.time.Instant;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Explicit, audited emergency access. No implicit admin access to user data.
*******************************************************************************/
public class BreakGlassService
{
public static final String START = "secured_data_break_glass_start";
public static final String END = "secured_data_break_glass_end";
public static final String EXPIRE = "secured_data_break_glass_expire";
private static final QLogger LOG = QLogger.getLogger(BreakGlassService.class);
/*******************************************************************************
** Validate first, then create a system-owned session and publish its policy version.
*******************************************************************************/
public QRecord start(String reason, Integer minutes, String table, DataClassification ceiling) throws QException
{
checkChannel();
if(!QContext.getQSession().hasPermission(START + ".hasAccess"))
{
throw (new QPermissionDeniedException("Permission denied."));
}
int duration = minutes == null ? 60 : minutes;
if(reason == null || reason.trim().length() < 20 || reason.length() > 500 || duration < 1 || duration > 480)
{
throw (new QBadRequestException("Reason must contain 20-500 characters; duration must be 1-480 minutes."));
}
if(table != null && table.isBlank())
{
table = null;
}
if(table != null && QContext.getQInstance().getTable(table) == null)
{
throw (new QBadRequestException("Unknown table scope."));
}
SecuredDataRuntime runtime = SecuredDataRuntime.current();
synchronized(runtime)
{
if(!runtime.resolver().activeBreakGlass().isEmpty())
{
throw (new QBadRequestException("An active break-glass session already exists."));
}
Instant now = runtime.config().getClock().instant();
QRecord row = SystemWrites.insertOne(SecuredDataTables.BREAK_GLASS, new QRecord()
.withValue("user_id", SecuredDataResolver.userId()).withValue("user_name", QContext.getQSession().getUser().getFullName())
.withValue("reason", reason).withValue("table_name", table).withValue("classification_ceiling", Objects.requireNonNullElse(ceiling, DataClassification.PHI).name())
.withValue("started_at", now).withValue("expires_at", now.plusSeconds(duration * 60L)).withValue("status", "ACTIVE"));
runtime.invalidatePolicies();
log(row, "BREAK_GLASS_START");
QAlerts.raise(new AlertSignal("secured_data.break_glass").withSeverity(AlertSeverity.fromLabel("HIGH", AlertSeverity.ERROR)).withDedupKey(dedup(row))
.withTitle("Break-glass access started").withSummary("Review the secured-data break-glass session.").withOccurredAt(now));
notifyAdmins(row);
return (row);
}
}
/*******************************************************************************
** Owner or admin may end an active session, never an assistant channel.
*******************************************************************************/
public QRecord end(Serializable id) throws QException
{
checkChannel();
synchronized(SecuredDataRuntime.current())
{
QRecord row = find(id);
checkOwner(row);
if(!"ACTIVE".equals(row.getValueString("status")))
{
throw (new QBadRequestException("Session is already finished."));
}
return (finish(row, "ENDED", "BREAK_GLASS_END"));
}
}
/*******************************************************************************
** Review uses the same ownership check; load rechecks against persisted state.
*******************************************************************************/
public void checkOwner(QRecord row) throws QPermissionDeniedException
{
checkChannel();
if(!Objects.equals(SecuredDataResolver.userId(), row.getValueString("user_id"))
&& !QContext.getQSession().hasPermission(SecuredDataPermissions.ADMIN))
{
throw (new QPermissionDeniedException("Only the owner or a secured-data administrator can end this session."));
}
}
/*******************************************************************************
** All rows are looked up as system; authorization remains the caller responsibility.
*******************************************************************************/
public QRecord find(Serializable id) throws QNotFoundException
{
List<QRecord> rows = SystemWrites.query(SecuredDataTables.BREAK_GLASS, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id)));
if(rows.isEmpty())
{
throw (new QNotFoundException("No record was found"));
}
return (rows.getFirst());
}
/*******************************************************************************
** Scheduled maintenance always performs its lookups and changes as system.
*******************************************************************************/
public int expire() throws QException
{
synchronized(SecuredDataRuntime.current())
{
return (SystemWrites.call(() ->
{
List<QRecord> rows = SystemWrites.query(SecuredDataTables.BREAK_GLASS, new QQueryFilter()
.withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"))
.withCriteria(new QFilterCriteria("expires_at", QCriteriaOperator.LESS_THAN_OR_EQUALS, SecuredDataRuntime.current().config().getClock().instant())));
for(QRecord row : rows)
{
finish(row, "EXPIRED", "BREAK_GLASS_EXPIRED");
}
return (rows.size());
}));
}
}
/*******************************************************************************
** Update, audit and resolve using one stable per-session alert key.
*******************************************************************************/
private QRecord finish(QRecord row, String status, String event) throws QException
{
QRecord update = new QRecord().withValue("id", row.getValue("id")).withValue("status", status)
.withValue("ended_at", SecuredDataRuntime.current().config().getClock().instant()).withValue("ended_by", SecuredDataResolver.userId());
QRecord result = SystemWrites.call(() -> new UpdateAction().execute(new UpdateInput(SecuredDataTables.BREAK_GLASS).withRecord(update)).getRecords().getFirst());
if(result.getErrors() != null && !result.getErrors().isEmpty())
{
throw (new QException("Unable to finish break-glass session."));
}
SecuredDataRuntime.current().invalidatePolicies();
log(row, event);
QAlerts.resolve("secured_data.break_glass", dedup(row), status);
return (result);
}
/*******************************************************************************
** Audit the actor and affected session without copying protected field values.
*******************************************************************************/
private void log(QRecord row, String event) throws QException
{
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, ActivityLog.row(event, SecuredDataTables.BREAK_GLASS, "ALLOWED")
.withValue("break_glass_id", row.getValue("id")).withValue("record_id", row.getValueString("id")).withValue("reason", row.getValue("reason")));
}
/*******************************************************************************
** Stable alert identity.
*******************************************************************************/
private String dedup(QRecord row)
{
return ("secured_data.break_glass:" + row.getValueString("id"));
}
/*******************************************************************************
** RC.5 lacks #1125; use the notify SPI when installed without a qbit-alerts dependency.
*******************************************************************************/
private void notifyAdmins(QRecord row)
{
try
{
Class<?> signalClass = Class.forName("com.kingsrook.qqq.backend.core.alerts.NotificationSignal");
Class<?> audienceClass = Class.forName("com.kingsrook.qqq.backend.core.alerts.NotificationAudience");
Object signal = signalClass.getConstructor(String.class).newInstance("Break-glass access started");
signalClass.getMethod("withAudience", audienceClass).invoke(signal, audienceClass.getMethod("permission", String.class).invoke(null, SecuredDataPermissions.ADMIN));
signalClass.getMethod("withSourceName", String.class).invoke(signal, "secured_data");
signalClass.getMethod("withSourceKey", String.class).invoke(signal, dedup(row));
QAlerts.class.getMethod("notify", signalClass).invoke(null, signal);
}
catch(ClassNotFoundException | NoSuchMethodException expected)
{
// TODO(core #1125): remove reflection at RC.6 and use the notification SPI directly.
LOG.warn("Break-glass admin bell notification unavailable: core notification SPI #1125 is not installed.");
}
catch(ReflectiveOperationException e)
{
LOG.warn("Unable to deliver break-glass admin bell notification.");
}
}
/*******************************************************************************
** Deny unauthenticated and assistant invocations in direct service calls too.
*******************************************************************************/
private void checkChannel() throws QPermissionDeniedException
{
if(QContext.getQSession() == null || QContext.getQSession().getUser() == null
|| "anonymous".equals(SecuredDataResolver.userId()) || SecuredDataResolver.userId().isBlank()
|| "assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
throw (new QPermissionDeniedException("Permission denied."));
}
}
}
@@ -0,0 +1,81 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.actions.values.QCustomPossibleValueProvider;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Fields of the selected table; table_name is supplied through the dependent picker.
*******************************************************************************/
public class FieldsOfTablePossibleValueSource implements QCustomPossibleValueProvider<String>
{
/*******************************************************************************
** Stored ids are unqualified field names; labels are provided by search.
*******************************************************************************/
@Override
public QPossibleValue<String> getPossibleValue(Serializable id)
{
return (id == null ? null : new QPossibleValue<>(id.toString(), id.toString()));
}
/*******************************************************************************
** Never offer fields from a table the administrator cannot read.
*******************************************************************************/
@Override
public List<QPossibleValue<String>> search(SearchPossibleValueSourceInput input) throws QException
{
Object tableName = input.getOtherValues() == null ? null : input.getOtherValues().get("table_name");
if(tableName == null && input.getOtherValues() != null)
{
tableName = input.getOtherValues().get("tableName");
}
List<QPossibleValue<String>> values = new ArrayList<>();
if(tableName == null || tableName.toString().isBlank())
{
return (values);
}
QTableMetaData table = QContext.getQInstance().getTable(tableName.toString());
if(table == null)
{
throw (new QException("Unknown table."));
}
if(PermissionsHelper.hasTablePermission(new GetInput(table.getName()), table.getName(), TablePermissionSubType.READ))
{
table.getFields().values().forEach(field -> values.add(new QPossibleValue<>(field.getName(), field.getLabel())));
}
return (completeCustomPVSSearch(input, values));
}
}
@@ -0,0 +1,134 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.util.List;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.AUDIT_KEY;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.LEGAL_HOLD;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.RETENTION_RULE;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_MARKER;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.WIPE_RUN;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.base;
import static com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.field;
/*******************************************************************************
** Retention policies, value-free history, legal holds and mutable audit keys.
*******************************************************************************/
public final class RetentionTables
{
public static final String KEY_LOCK = "secured_data_internal_key";
/** A key value no session is ever given, so the lock denies every user (validator requires a non-empty override list). */
public static final String KEY_LOCK_NEVER_GRANTED = "secured_data_never_granted";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionTables()
{
}
/*******************************************************************************
** No backup or WORM registration is made for the key table.
*******************************************************************************/
public static List<QTableMetaData> tables(SecuredDataQBitConfig config)
{
QTableMetaData rule = base(config, RETENTION_RULE, "Retention rules")
.withField(field("name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("field_names", QFieldType.TEXT).withIsRequired(true))
.withField(field("age_field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_fields"))
.withField(field("age_days", QFieldType.INTEGER).withIsRequired(true))
.withField(field("action", QFieldType.STRING).withMaxLength(40).withDefaultValue("NULL_WITH_MARKER"))
.withField(field("overwrite_text", QFieldType.TEXT).withDefaultValue("[wiped per policy]"))
.withField(field("audit_treatment", QFieldType.STRING).withMaxLength(30).withDefaultValue("SHREDDABLE"))
.withField(field("archive_storage_name", QFieldType.STRING).withMaxLength(255))
.withField(field("batch_size", QFieldType.INTEGER).withDefaultValue(500))
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("last_run_at", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("last_run_count", QFieldType.INTEGER).withIsEditable(false))
.withField(field("create_date", QFieldType.DATE_TIME))
.withField(field("modify_date", QFieldType.DATE_TIME))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
rule.withSection(SectionFactory.defaultT1("id", "name", "table_name", "field_names"));
rule.withSection(SectionFactory.defaultT2("age_field_name", "age_days", "action", "overwrite_text", "audit_treatment", "archive_storage_name", "batch_size", "is_active"));
rule.withSection(SectionFactory.defaultT3("last_run_at", "last_run_count", "create_date", "modify_date", "modified_by"));
rule.withSection(SectionFactory.customT2("backups", new com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon("backup"))
.withLabel("Backups").withWidgetName(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.BACKUPS));
rule.withSection(SectionFactory.customT2("markers", new com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon("delete"))
.withLabel("Wipe markers").withWidgetName(com.kingsrook.qbits.secureddata.retention.RetentionWidgets.MARKERS));
QTableMetaData marker = base(config, WIPE_MARKER, "Wipe marker")
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true))
.withField(field("action", QFieldType.STRING).withMaxLength(40).withIsRequired(true))
.withField(field("wiped_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("run_id", QFieldType.INTEGER).withIsRequired(true))
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
marker.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "rule_id", "action", "wiped_at", "run_id"));
QTableMetaData hold = base(config, LEGAL_HOLD, "Legal hold")
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255))
.withField(field("reason", QFieldType.TEXT).withIsRequired(true))
.withField(field("placed_by", QFieldType.STRING).withMaxLength(255))
.withField(field("placed_at", QFieldType.DATE_TIME))
.withField(field("released_by", QFieldType.STRING).withMaxLength(255))
.withField(field("released_at", QFieldType.DATE_TIME))
.withField(field("status", QFieldType.STRING).withMaxLength(20).withDefaultValue("ACTIVE"));
hold.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "reason", "status"));
hold.withSection(SectionFactory.defaultT3("placed_by", "placed_at", "released_by", "released_at"));
QTableMetaData run = base(config, WIPE_RUN, "Wipe run")
.withField(field("rule_id", QFieldType.INTEGER).withIsRequired(true))
.withField(field("started_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("finished_at", QFieldType.DATE_TIME))
.withField(field("status", QFieldType.STRING).withMaxLength(30))
.withField(field("eligible_count", QFieldType.INTEGER))
.withField(field("held_count", QFieldType.INTEGER))
.withField(field("wiped_count", QFieldType.INTEGER))
.withField(field("error_count", QFieldType.INTEGER))
.withField(field("record_ids", QFieldType.STRING).withMaxLength(4000))
.withField(field("message", QFieldType.STRING).withMaxLength(500));
run.withSection(SectionFactory.defaultT1("id", "rule_id", "status", "eligible_count", "held_count", "wiped_count", "error_count"));
run.withSection(SectionFactory.defaultT3("started_at", "finished_at", "record_ids", "message"));
QTableMetaData key = base(config, AUDIT_KEY, "Audit key")
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
.withoutCapabilities(Capability.TABLE_EXPORT)
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("wrapped_key", QFieldType.TEXT).withIsRequired(true).withIsSensitive(true).withIsHidden(true)
.withFieldSecurityLock(new FieldSecurityLock().withSecurityKeyType(KEY_LOCK).withDefaultBehavior(FieldSecurityLock.Behavior.DENY).withOverrideValues(List.of(KEY_LOCK_NEVER_GRANTED))))
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "record_id", "field_name")));
key.withSection(SectionFactory.defaultT1("id", "table_name", "record_id", "field_name", "created_at"));
return (List.of(rule, marker, hold, run, key));
}
}
@@ -0,0 +1,104 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
/*******************************************************************************
** String-valued possible-value sources shared by UI metadata and the DDL.
*******************************************************************************/
public final class SecuredDataEnums
{
/*******************************************************************************
** Activity event names include the future Reveal and break-glass actions.
*******************************************************************************/
public enum EventType
{
VIEW, REVEAL, REVEAL_DENIED, EXACT_MATCH_FILTER, FILTER_DENIED, BREAK_GLASS_START, BREAK_GLASS_END,
BREAK_GLASS_EXPIRED, POLICY_CHANGE, CLASSIFICATION_CHANGE
}
/*******************************************************************************
** Access outcomes.
*******************************************************************************/
public enum Outcome
{
ALLOWED, DENIED, CAPPED, MATCH, NO_MATCH, REFUSED
}
/*******************************************************************************
** Break-glass lifecycle.
*******************************************************************************/
public enum BreakGlassStatus
{
ACTIVE, ENDED, EXPIRED
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataEnums()
{
}
/*******************************************************************************
** All enum sources, keeping the stand-in core types exact copies.
*******************************************************************************/
public static List<QPossibleValueSource> sources()
{
return (List.of(source("classification", DataClassification.values()), source("tier", FieldAccessTier.values()),
source("mask_style", FieldMaskStyle.values()), source("event_type", EventType.values()), source("outcome", Outcome.values()),
source("break_glass_status", BreakGlassStatus.values())));
}
/*******************************************************************************
** Convert enum names to stored string ids.
*******************************************************************************/
private static QPossibleValueSource source(String name, Enum<?>[] values)
{
List<QPossibleValue<?>> options = new ArrayList<>();
for(Enum<?> value : values)
{
options.add(new QPossibleValue<>(value.name(), value.name().replace('_', ' ')));
}
return (new QPossibleValueSource().withName("secured_data_" + name).withType(QPossibleValueSourceType.ENUM)
.withIdType(QFieldType.STRING).withEnumValues(options));
}
}
@@ -0,0 +1,276 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.time.Instant;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
/*******************************************************************************
** Validation, activity-table write protection and post-commit cache invalidation.
*******************************************************************************/
public class SecuredDataTableCustomizers implements TableCustomizerInterface
{
/*******************************************************************************
** Defend system-only activity writes even when a caller skips permission checks.
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
{
String table = input instanceof InsertInput insert ? insert.getTableName() : ((UpdateInput) input).getTableName();
assertWrite(table, input instanceof UpdateInput);
if(!SecuredDataTables.SYSTEM_WRITTEN.contains(table))
{
for(QRecord record : records)
{
QRecord old = oldRecords.orElse(List.of()).stream().filter(row -> Objects.equals(row.getValue("id"), record.getValue("id"))).findFirst().orElse(null);
QRecord effective = old == null ? new QRecord() : new QRecord(old);
record.getValues().forEach(effective::setValue);
validate(table, record, effective);
Instant now = SecuredDataRuntime.current().config().getClock().instant();
if(SecuredDataTables.LEGAL_HOLD.equals(table))
{
if(old == null)
{
record.setValue("placed_at", now);
record.setValue("placed_by", SecuredDataResolver.userId());
}
if("RELEASED".equals(effective.getValueString("status")))
{
record.setValue("released_at", now);
record.setValue("released_by", SecuredDataResolver.userId());
}
continue;
}
if(old == null)
{
record.setValue("create_date", now);
}
record.setValue("modify_date", now);
record.setValue("modified_by", SecuredDataResolver.userId());
}
}
return (records);
}
/*******************************************************************************
** Access logs are append-only even for the system.
*******************************************************************************/
private static void assertWrite(String table, boolean mutation) throws QPermissionDeniedException
{
if((mutation && List.of(SecuredDataTables.ACCESS_LOG, SecuredDataTables.WIPE_RUN).contains(table))
|| (SecuredDataTables.SYSTEM_WRITTEN.contains(table) && !SystemWrites.active() && !(QContext.getQSession() instanceof QSystemUserSession)))
{
throw (new QPermissionDeniedException("Secured-data activity is system-written; access logs are insert-only."));
}
}
/*******************************************************************************
** Validate target names, unique table-wide classifications and guardrail ranges.
*******************************************************************************/
private static void validate(String table, QRecord record, QRecord effective)
{
String target = effective.getValueString("table_name");
QTableMetaData metadata = target == null ? null : QContext.getQInstance().getTable(target);
if(target != null && metadata == null)
{
record.addError(new BadInputStatusMessage("Select an existing table."));
}
if(SecuredDataTables.RETENTION_RULE.equals(table))
{
try
{
com.kingsrook.qbits.secureddata.retention.RetentionRules.validate(effective);
}
catch(IllegalArgumentException e)
{
record.addError(new BadInputStatusMessage(e.getMessage()));
}
}
else if(SecuredDataTables.LEGAL_HOLD.equals(table))
{
if(!List.of("ACTIVE", "RELEASED").contains(effective.getValueString("status")))
{
record.addError(new BadInputStatusMessage("Select ACTIVE or RELEASED."));
}
}
else if(SecuredDataTables.CLASSIFICATION.equals(table))
{
String field = effective.getValueString("field_name");
if(field != null && field.isBlank())
{
field = null;
record.setValue("field_name", null);
}
if(field != null && (metadata == null || !metadata.getFields().containsKey(field)))
{
record.addError(new BadInputStatusMessage("Select a field of the selected table."));
}
if(target != null)
{
for(QRecord row : SystemWrites.query(table, new QQueryFilter().withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, target))))
{
if(Objects.equals(field, row.getValueString("field_name")) && !Objects.equals(row.getValue("id"), effective.getValue("id")))
{
record.addError(new BadInputStatusMessage("A classification for this table and field already exists."));
}
}
}
}
else
{
Integer minutes = effective.getValueInteger("reveal_minutes");
if(minutes != null && (minutes < 1 || minutes > 60))
{
record.addError(new BadInputStatusMessage("Reveal duration must be 1 to 60 minutes."));
}
for(String name : List.of("exact_match_per_hour", "miss_alert_consecutive", "distinct_value_alert", "daily_reveal_cap"))
{
Integer value = effective.getValueInteger(name);
if(value != null && value < 0)
{
record.addError(new BadInputStatusMessage("Guardrail limits cannot be negative."));
}
}
}
}
/*******************************************************************************
** User deletes of activity are forbidden; the log is immutable for everyone.
*******************************************************************************/
@Override
public List<QRecord> preDelete(DeleteInput input, List<QRecord> records, boolean preview) throws QException
{
assertWrite(input.getTableName(), true);
return (records);
}
/*******************************************************************************
** Only successful mutations invalidate; transactions invalidate after commit.
*******************************************************************************/
@Override
public List<QRecord> postInsertOrUpdate(AbstractActionInput input, List<QRecord> records, Optional<List<QRecord>> oldRecords) throws QException
{
if(input instanceof InsertInput insert)
{
changed(insert.getTableName(), records, insert.getTransaction());
audit(insert.getTableName(), records, "INSERT", insert.getTransaction());
}
else if(input instanceof UpdateInput update)
{
changed(update.getTableName(), records, update.getTransaction());
audit(update.getTableName(), records, "UPDATE", update.getTransaction());
}
return (records);
}
/*******************************************************************************
** Deletions change the same versions as inserts and updates.
*******************************************************************************/
@Override
public List<QRecord> postDelete(DeleteInput input, List<QRecord> records) throws QException
{
changed(input.getTableName(), records, input.getTransaction());
audit(input.getTableName(), records, "DELETE", input.getTransaction());
return (records);
}
/*******************************************************************************
** Capture the instance service; callbacks need no current user context.
*******************************************************************************/
private static void changed(String table, List<QRecord> records, QBackendTransaction transaction)
{
if(records.stream().noneMatch(row -> row.getErrors() == null || row.getErrors().isEmpty()))
{
return;
}
SecuredDataRuntime runtime = SecuredDataRuntime.current();
Runnable invalidate = SecuredDataTables.CLASSIFICATION.equals(table) ? runtime::invalidateClassifications
: SecuredDataTables.POLICY.equals(table) || SecuredDataTables.BREAK_GLASS.equals(table) ? runtime::invalidatePolicies : null;
if(invalidate != null)
{
QBackendTransaction active = transaction == null ? QContext.getQBackendTransaction() : transaction;
if(active == null)
{
invalidate.run();
}
else
{
active.addAfterCommitCallback(invalidate);
}
}
}
/*******************************************************************************
** Value-free mutation audit in the same transaction as the changed metadata.
*******************************************************************************/
private static void audit(String table, List<QRecord> records, String action, QBackendTransaction transaction) throws QException
{
if(!List.of(SecuredDataTables.POLICY, SecuredDataTables.CLASSIFICATION).contains(table))
{
return;
}
for(QRecord record : records)
{
if(record.getErrors() == null || record.getErrors().isEmpty())
{
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, com.kingsrook.qbits.secureddata.support.ActivityLog.row(
SecuredDataTables.POLICY.equals(table) ? "POLICY_CHANGE" : "CLASSIFICATION_CHANGE", table, "ALLOWED")
.withValue("record_id", record.getValueString("id")).withValue("action_name", action),
transaction == null ? QContext.getQBackendTransaction() : transaction);
}
}
}
}
@@ -0,0 +1,312 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.metadata;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PVSValueFormatAndFields;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
import com.kingsrook.qqq.backend.core.model.metadata.tables.TablesPossibleValueSourceMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
/*******************************************************************************
** The five audited tables, record sections, possible values and reveal child list.
*******************************************************************************/
public final class SecuredDataTables
{
public static final String CLASSIFICATION = "secured_data_classification";
public static final String POLICY = "secured_data_policy";
public static final String REVEAL_GRANT = "secured_data_reveal_grant";
public static final String BREAK_GLASS = "secured_data_break_glass";
public static final String ACCESS_LOG = "secured_data_access_log";
public static final String RETENTION_RULE = "secured_data_retention_rule";
public static final String WIPE_MARKER = "secured_data_wipe_marker";
public static final String LEGAL_HOLD = "secured_data_legal_hold";
public static final String WIPE_RUN = "secured_data_wipe_run";
public static final String AUDIT_KEY = "secured_data_audit_key";
public static final List<String> ALL = List.of(CLASSIFICATION, POLICY, REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, RETENTION_RULE, WIPE_MARKER, LEGAL_HOLD, WIPE_RUN, AUDIT_KEY);
public static final List<String> MAIN = List.of(CLASSIFICATION, POLICY, ACCESS_LOG, BREAK_GLASS, RETENTION_RULE);
public static final List<String> SYSTEM_WRITTEN = List.of(REVEAL_GRANT, BREAK_GLASS, ACCESS_LOG, WIPE_MARKER, WIPE_RUN, AUDIT_KEY);
public static final String REVEALS_JOIN = "secured_data_access_log_reveals";
public static final String REVEALS_WIDGET = "securedDataReveals";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataTables()
{
}
/*******************************************************************************
** Main table events for the host ESB wiring, following WebhooksIntegrations.
*******************************************************************************/
public static List<String> esbTableNames()
{
return (MAIN);
}
/*******************************************************************************
** Complete table metadata.
*******************************************************************************/
public static List<QTableMetaData> tables(SecuredDataQBitConfig config)
{
List<QTableMetaData> tables = new ArrayList<>(List.of(classification(config), policy(config), revealGrant(config), breakGlass(config), accessLog(config)));
tables.addAll(RetentionTables.tables(config));
return (tables);
}
/*******************************************************************************
** Common identity, audit and administration rules.
*******************************************************************************/
static QTableMetaData base(SecuredDataQBitConfig config, String name, String label)
{
QTableMetaData table = new QTableMetaData().withName(name).withLabel(label).withBackendName(config.getBackendName())
.withPrimaryKeyField("id").withField(new QFieldMetaData("id", QFieldType.INTEGER).withBackendName("id").withIsEditable(false))
.withPermissionRules(SecuredDataPermissions.rules()).withIcon(new QIcon("shield"))
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD));
SecuredDataMetadata.classifyTable(table);
if(ACCESS_LOG.equals(name))
{
table.withoutCapabilities(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
}
return (table);
}
/*******************************************************************************
** Every database column has an explicit snake_case name.
*******************************************************************************/
static QFieldMetaData field(String name, QFieldType type)
{
QFieldMetaData field = new QFieldMetaData(name, type).withBackendName(name).withLabel(name.replace('_', ' '));
// TODO(contingent): dependsOn("table_name") once the core RC has it
return (field);
}
/*******************************************************************************
** Define Classification.
*******************************************************************************/
private static QTableMetaData classification(SecuredDataQBitConfig config)
{
QTableMetaData table = base(config, CLASSIFICATION, "Classification")
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_fields"))
.withField(field("classification", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_classification"))
.withField(field("mask_style", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_mask_style"))
.withField(field("low_cardinality", QFieldType.BOOLEAN))
.withField(field("notes", QFieldType.TEXT))
.withField(field("create_date", QFieldType.DATE_TIME))
.withField(field("modify_date", QFieldType.DATE_TIME))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
table.withSection(SectionFactory.defaultT1("id", "table_name", "field_name", "classification"));
table.withSection(SectionFactory.defaultT2("mask_style", "low_cardinality", "notes"));
table.withSection(SectionFactory.defaultT3("create_date", "modify_date", "modified_by"));
table.withUniqueKey(new UniqueKey().withFieldNames(List.of("table_name", "field_name")));
return (table);
}
/*******************************************************************************
** Define Policy.
*******************************************************************************/
private static QTableMetaData policy(SecuredDataQBitConfig config)
{
QTableMetaData table = base(config, POLICY, "Policy")
.withField(field("name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("permission_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("classification", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_classification"))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("tables"))
.withField(field("tier", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_tier"))
.withField(field("reveal_minutes", QFieldType.INTEGER).withDefaultValue(5))
.withField(field("daily_reveal_cap", QFieldType.INTEGER))
.withField(field("reason_options", QFieldType.TEXT))
.withField(field("allow_free_text_reason", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("require_free_text", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("client_channel", QFieldType.STRING).withMaxLength(255))
.withField(field("exact_match_per_hour", QFieldType.INTEGER).withDefaultValue(20))
.withField(field("miss_alert_consecutive", QFieldType.INTEGER).withDefaultValue(5))
.withField(field("distinct_value_alert", QFieldType.INTEGER).withDefaultValue(15))
.withField(field("min_entropy_enabled", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("is_active", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("create_date", QFieldType.DATE_TIME))
.withField(field("modify_date", QFieldType.DATE_TIME))
.withField(field("modified_by", QFieldType.STRING).withMaxLength(255));
table.withSection(SectionFactory.defaultT1("id", "name", "permission_name", "classification"));
table.withSection(SectionFactory.defaultT2("table_name", "tier", "reveal_minutes", "daily_reveal_cap", "reason_options", "allow_free_text_reason", "require_free_text", "client_channel", "exact_match_per_hour", "miss_alert_consecutive", "distinct_value_alert", "min_entropy_enabled", "is_active"));
table.withSection(SectionFactory.defaultT3("create_date", "modify_date", "modified_by"));
return (table);
}
/*******************************************************************************
** Define Reveal.
*******************************************************************************/
private static QTableMetaData revealGrant(SecuredDataQBitConfig config)
{
QTableMetaData table = base(config, REVEAL_GRANT, "Reveal")
.withField(field("user_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("user_name", QFieldType.STRING).withMaxLength(255))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("record_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("field_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_fields"))
.withField(field("reason", QFieldType.STRING).withMaxLength(500))
.withField(field("reason_text", QFieldType.TEXT))
.withField(field("granted_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("access_log_id", QFieldType.INTEGER).withIsRequired(true));
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "table_name"));
table.withSection(SectionFactory.defaultT2("record_id", "field_name", "reason", "reason_text", "access_log_id"));
table.withSection(SectionFactory.defaultT3("granted_at", "expires_at"));
return (table);
}
/*******************************************************************************
** Define Break-glass session.
*******************************************************************************/
private static QTableMetaData breakGlass(SecuredDataQBitConfig config)
{
QTableMetaData table = base(config, BREAK_GLASS, "Break-glass session")
.withField(field("user_id", QFieldType.STRING).withMaxLength(255).withIsRequired(true))
.withField(field("user_name", QFieldType.STRING).withMaxLength(255))
.withField(field("reason", QFieldType.STRING).withMaxLength(500))
.withField(field("reason_text", QFieldType.TEXT))
.withField(field("classification_ceiling", QFieldType.STRING).withMaxLength(255).withDefaultValue("PHI").withPossibleValueSourceName("secured_data_classification"))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("tables"))
.withField(field("started_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("expires_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("ended_at", QFieldType.DATE_TIME))
.withField(field("ended_by", QFieldType.STRING).withMaxLength(255))
.withField(field("status", QFieldType.STRING).withMaxLength(255).withDefaultValue("ACTIVE").withPossibleValueSourceName("secured_data_break_glass_status"));
table.withSection(SectionFactory.defaultT1("id", "user_id", "user_name", "reason"));
table.withSection(SectionFactory.defaultT2("reason_text", "classification_ceiling", "table_name", "ended_by", "status"));
table.withSection(SectionFactory.defaultT3("started_at", "expires_at", "ended_at"));
return (table);
}
/*******************************************************************************
** Define Access log.
*******************************************************************************/
private static QTableMetaData accessLog(SecuredDataQBitConfig config)
{
QTableMetaData table = base(config, ACCESS_LOG, "Access log")
.withField(field("event_type", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("secured_data_event_type"))
.withField(field("user_id", QFieldType.STRING).withMaxLength(255))
.withField(field("user_name", QFieldType.STRING).withMaxLength(255))
.withField(field("table_name", QFieldType.STRING).withMaxLength(255).withIsRequired(true).withPossibleValueSourceName("tables"))
.withField(field("record_id", QFieldType.TEXT))
.withField(field("field_name", QFieldType.TEXT))
.withField(field("record_count", QFieldType.INTEGER))
.withField(field("action_name", QFieldType.STRING).withMaxLength(255))
.withField(field("reason", QFieldType.STRING).withMaxLength(500))
.withField(field("reason_text", QFieldType.TEXT))
.withField(field("client_channel", QFieldType.STRING).withMaxLength(255))
.withField(field("outcome", QFieldType.STRING).withMaxLength(255).withPossibleValueSourceName("secured_data_outcome"))
.withField(field("anomaly_flag", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("break_glass_id", QFieldType.INTEGER))
.withField(field("created_at", QFieldType.DATE_TIME).withIsRequired(true))
.withField(field("search_value_fingerprints", QFieldType.TEXT))
.withField(field("rule_id", QFieldType.INTEGER));
table.withSection(SectionFactory.defaultT1("id", "event_type", "user_id", "user_name"));
table.withSection(SectionFactory.defaultT2("table_name", "record_id", "field_name", "record_count", "action_name", "reason", "reason_text", "client_channel", "outcome", "anomaly_flag", "break_glass_id", "rule_id", "search_value_fingerprints"));
table.withSection(SectionFactory.defaultT3("created_at"));
table.withSection(SectionFactory.customT2("reveals", new QIcon("visibility")).withLabel("Reveals").withWidgetName(REVEALS_WIDGET));
return (table);
}
/*******************************************************************************
** Core table picker plus a dependent field picker.
*******************************************************************************/
public static List<QPossibleValueSource> possibleValueSources(QInstance instance)
{
List<QPossibleValueSource> sources = new ArrayList<>(SecuredDataEnums.sources());
if(instance.getPossibleValueSource("tables") == null)
{
sources.add(TablesPossibleValueSourceMetaDataProvider.defineTablesPossibleValueSource(instance));
}
sources.add(new QPossibleValueSource().withName("secured_data_fields").withType(QPossibleValueSourceType.CUSTOM)
.withIdType(QFieldType.STRING).withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY)
.withCustomCodeReference(new QCodeReference(FieldsOfTablePossibleValueSource.class)));
return (sources);
}
/*******************************************************************************
** Reveals belong to the access-log record that authorized them.
*******************************************************************************/
public static QJoinMetaData revealsJoin()
{
return (new QJoinMetaData().withName(REVEALS_JOIN).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ACCESS_LOG).withRightTable(REVEAL_GRANT).withJoinOn(new JoinOn("id", "access_log_id")));
}
/*******************************************************************************
** Read-only child list, never a navigation entry.
*******************************************************************************/
public static QWidgetMetaData revealsWidget()
{
return (ChildRecordListRenderer.widgetMetaDataBuilder(revealsJoin()).withName(REVEALS_WIDGET).withLabel("Reveals")
.withCanAddChildRecord(false).withMaxRows(50).getWidgetMetaData().withGridColumns(12)
.withPermissionRules(SecuredDataPermissions.rules()));
}
}
@@ -0,0 +1,55 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.permissions;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
** Admin-only management and read access; activity writes belong to system operations.
*******************************************************************************/
public class SecuredDataPermissionChecker implements CustomPermissionChecker
{
/*******************************************************************************
** Enforce the assistant wall on RC.5 too, when a trusted channel is supplied.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
boolean write = input instanceof InsertInput || input instanceof UpdateInput || input instanceof DeleteInput;
if(QContext.getQSession() == null || "assistant".equals(SecuredDataMetadata.channel(QContext.getQSession()))
|| !QContext.getQSession().hasPermission(SecuredDataPermissions.ADMIN)
|| (write && SecuredDataTables.SYSTEM_WRITTEN.contains(metadata.getName())))
{
throw (new QPermissionDeniedException("Permission denied."));
}
}
}
@@ -0,0 +1,59 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.permissions;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
/*******************************************************************************
** Administrative access is independent of access to classified application data.
*******************************************************************************/
public final class SecuredDataPermissions
{
public static final String ADMIN = "secured_data.admin";
public static final String FULL = "secured_data_full";
public static final String REVEAL = "secured_data_reveal";
public static final List<String> ALL = List.of(ADMIN, "secured_data.hasAccess");
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataPermissions()
{
}
/*******************************************************************************
** Standard QQQ custom checker, shared by tables, app and child widget.
*******************************************************************************/
public static QPermissionRules rules()
{
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION).withPermissionBaseName("secured_data")
.withCustomPermissionChecker(new QCodeReference(SecuredDataPermissionChecker.class)));
}
}
@@ -0,0 +1,233 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.processes;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Table/record actions and the expiry schedule. Never added to navigation.
*******************************************************************************/
public final class BreakGlassProcesses
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BreakGlassProcesses()
{
}
/*******************************************************************************
** A start form, a streamed Review/Result end action, and a hidden scheduled job.
*******************************************************************************/
public static List<QProcessMetaData> processes(SecuredDataQBitConfig config)
{
QProcessMetaData start = new QProcessMetaData().withName(BreakGlassService.START).withLabel("Start break-glass")
.withTableName(SecuredDataTables.BREAK_GLASS).withMinInputRecords(0).withMaxInputRecords(0)
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION))
.withStep(new QFrontendStepMetaData().withName("input").withLabel("Start break-glass")
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM))
.withFormField(new QFieldMetaData("reason", QFieldType.STRING).withLabel("Reason").withIsRequired(true).withMaxLength(500))
.withFormField(new QFieldMetaData("duration_minutes", QFieldType.INTEGER).withLabel("Duration minutes").withDefaultValue(60))
.withFormField(new QFieldMetaData("table_name", QFieldType.STRING).withLabel("Table scope").withPossibleValueSourceName("tables"))
.withFormField(new QFieldMetaData("classification_ceiling", QFieldType.STRING).withLabel("Classification ceiling")
.withDefaultValue("PHI").withPossibleValueSourceName("secured_data_classification")))
.withStep(new QBackendStepMetaData().withName("start").withCode(new QCodeReference(Start.class)))
.withStep(new QFrontendStepMetaData().withName("result").withLabel("Result")
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.VIEW_FORM))
.withViewField(new QFieldMetaData("message", QFieldType.STRING)));
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, SecuredDataTables.BREAK_GLASS);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, SecuredDataTables.BREAK_GLASS);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, "The selected break-glass sessions will be ended.");
QProcessMetaData end = StreamedETLWithFrontendProcess.defineProcessMetaData(EndExtract.class, EndReview.class, EndLoad.class, values)
.withName(BreakGlassService.END).withLabel("End break-glass").withTableName(SecuredDataTables.BREAK_GLASS).withMinInputRecords(1)
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED));
QProcessMetaData expire = new QProcessMetaData().withName(BreakGlassService.EXPIRE).withLabel("Expire break-glass sessions").withIsHidden(true)
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION))
.withStep(new QBackendStepMetaData().withName("expire").withCode(new QCodeReference(Expire.class)));
if(config.getSchedulerName() != null)
{
expire.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(60).withInitialDelaySeconds(30));
}
return (List.of(start, end, expire));
}
/*******************************************************************************
** Start after the form is submitted.
*******************************************************************************/
public static class Start implements BackendStep
{
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
String ceiling = input.getValueString("classification_ceiling");
DataClassification classification;
try
{
classification = ceiling == null ? null : DataClassification.valueOf(ceiling);
}
catch(IllegalArgumentException e)
{
throw (new com.kingsrook.qqq.backend.core.exceptions.QBadRequestException("Unknown classification ceiling."));
}
new BreakGlassService().start(input.getValueString("reason"), input.getValueInteger("duration_minutes"), input.getValueString("table_name"), classification);
output.addValue("message", "Break-glass access is active.");
}
}
/*******************************************************************************
** Extract with system visibility; ownership is checked before any row enters the pipe.
*******************************************************************************/
public static class EndExtract extends ExtractViaQueryStep
{
@Override
public Integer doCount(RunBackendStepInput input)
{
return (null);
}
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord row : SystemWrites.query(SecuredDataTables.BREAK_GLASS, getQueryFilter(input)))
{
new BreakGlassService().checkOwner(row);
getRecordPipe().addRecord(row);
}
}
}
/*******************************************************************************
** Review validates ownership and reports the change in future and past tense.
*******************************************************************************/
public static class EndReview extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine line = summary();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord row : input.getRecords())
{
new BreakGlassService().checkOwner(row);
if("ACTIVE".equals(row.getValueString("status")))
{
line.incrementCountAndAddPrimaryKey(row.getValue("id"));
output.getRecords().add(row);
}
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
line.addSelfToListIfAnyCount(lines);
return (lines);
}
}
/*******************************************************************************
** Apply the reviewed operation, rechecking persisted ownership at execution time.
*******************************************************************************/
public static class EndLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine line = summary();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord row : input.getRecords())
{
output.getRecords().add(new BreakGlassService().end(row.getValue("id")));
line.incrementCountAndAddPrimaryKey(row.getValue("id"));
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
line.addSelfToListIfAnyCount(lines);
return (lines);
}
}
/*******************************************************************************
** Standard ETL summary wording.
*******************************************************************************/
private static ProcessSummaryLine summary()
{
ProcessSummaryLine line = new ProcessSummaryLine(Status.OK, 0, null);
line.setSingularFutureMessage("session will be ended");
line.setPluralFutureMessage("sessions will be ended");
line.setSingularPastMessage("session was ended");
line.setPluralPastMessage("sessions were ended");
return (line);
}
/*******************************************************************************
** Expiry job; system context and writes are established by the service.
*******************************************************************************/
public static class Expire implements BackendStep
{
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
output.addValue("expired", new BreakGlassService().expire());
}
}
}
@@ -0,0 +1,36 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** Host adapter to qbit-object-storage. Receives only AES-GCM ciphertext.
** The host should use a locked WORM bucket when available. Return after durable write.
*******************************************************************************/
@FunctionalInterface
public interface ArchiveSinkInterface
{
/*******************************************************************************
** Store an encrypted JSON record at an opaque, unique object key.
*******************************************************************************/
void put(String storageName, String objectKey, byte[] ciphertext) throws QException;
}
@@ -0,0 +1,150 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.util.Arrays;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.secureddata.SecuredDataClassifications;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Shared validation and effective audit treatment; no record values are queried.
*******************************************************************************/
public final class RetentionRules
{
public static final Set<String> ACTIONS = Set.of("NULL_WITH_MARKER", "OVERWRITE_TEXT", "KEYED_HASH", "DELETE_RECORD", "ARCHIVE_THEN_WIPE");
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionRules()
{
}
/*******************************************************************************
** Validated comma selection; preserve order and reject duplicates.
*******************************************************************************/
public static List<String> fields(QRecord rule)
{
String names = rule.getValueString("field_names");
return (names == null ? List.of() : Arrays.stream(names.split(",", -1)).map(String::trim).toList());
}
/*******************************************************************************
** Fresh rules in the current transaction; edits take effect without a stale cache.
*******************************************************************************/
public static List<QRecord> active(String table, String field)
{
return (SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter()
.withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table)))
.stream().filter(rule -> fields(rule).contains(field)).toList());
}
/*******************************************************************************
** A changed-only policy takes precedence over derivatives.
*******************************************************************************/
public static boolean changedOnly(String table, String field)
{
return (active(table, field).stream().anyMatch(rule -> "CHANGED_ONLY".equals(rule.getValueString("audit_treatment"))));
}
/*******************************************************************************
** Validate both edits and persisted policies before any execution.
*******************************************************************************/
public static void validate(QRecord rule)
{
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
if(table == null || SecuredDataTables.ALL.contains(table.getName()))
{
throw (new IllegalArgumentException("Select a host table for retention."));
}
List<String> fields = fields(rule);
if(fields.isEmpty() || fields.size() != Set.copyOf(fields).size() || fields.stream().anyMatch(f -> table.getField(f) == null || f.equals(table.getPrimaryKeyField())))
{
throw (new IllegalArgumentException("Select unique, non-key fields of the selected table."));
}
var age = table.getField(rule.getValueString("age_field_name"));
if(age == null || !Set.of(QFieldType.DATE, QFieldType.DATE_TIME).contains(age.getType()))
{
throw (new IllegalArgumentException("Age must use a date or date-time field of the selected table."));
}
if(rule.getValueInteger("age_days") == null || rule.getValueInteger("age_days") < 1 || rule.getValueInteger("batch_size") == null || rule.getValueInteger("batch_size") < 1)
{
throw (new IllegalArgumentException("Age days and batch size must be positive."));
}
String action = rule.getValueString("action");
if(!ACTIONS.contains(action) || !Set.of("SHREDDABLE", "CHANGED_ONLY").contains(rule.getValueString("audit_treatment")))
{
throw (new IllegalArgumentException("Select a supported action and audit treatment."));
}
for(String name : fields)
{
var field = table.getField(name);
if(Set.of("OVERWRITE_TEXT", "KEYED_HASH").contains(action) && !Set.of(QFieldType.STRING, QFieldType.TEXT).contains(field.getType()))
{
throw (new IllegalArgumentException("Overwrite and keyed hash require STRING or TEXT fields."));
}
if("KEYED_HASH".equals(action) && SecuredDataClassifications.lowEntropy(table, name))
{
throw (new IllegalArgumentException("Keyed hash is forbidden for low-entropy fields."));
}
if("KEYED_HASH".equals(action) && field.getMaxLength() != null && field.getMaxLength() < 200)
{
throw (new IllegalArgumentException("Keyed hash requires capacity for a 200-character token."));
}
if(Set.of("NULL_WITH_MARKER", "ARCHIVE_THEN_WIPE").contains(action) && field.getIsRequired())
{
throw (new IllegalArgumentException("Null wipe requires nullable fields."));
}
String overwrite = rule.getValueString("overwrite_text");
if("OVERWRITE_TEXT".equals(action) && (overwrite == null || field.getMaxLength() != null && overwrite.length() > field.getMaxLength()))
{
throw (new IllegalArgumentException("Overwrite text must fit every target field."));
}
}
if("ARCHIVE_THEN_WIPE".equals(action) && (rule.getValueString("archive_storage_name") == null || rule.getValueString("archive_storage_name").isBlank()))
{
throw (new IllegalArgumentException("Archive then wipe requires an archive storage name."));
}
}
/*******************************************************************************
** Known backup window; unknown is explicit rather than assumed safe.
*******************************************************************************/
public static String backupWarning(QRecord rule)
{
Integer days = SecuredDataRuntime.current().config().getBackupRetentionDays();
Integer age = rule.getValueInteger("age_days");
return (days != null && age != null && days > age ? "Backups keep data for " + days + " days, longer than this rule's " + age
+ " days. Wiped values remain recoverable from backups until they expire." : null);
}
}
@@ -0,0 +1,163 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.permissions.SecuredDataPermissions;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Value-free policy panels and host-opt-in record marker lists.
*******************************************************************************/
public final class RetentionWidgets
{
public static final String BACKUPS = "securedDataBackups";
public static final String MARKERS = "securedDataRuleMarkers";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private RetentionWidgets()
{
}
/*******************************************************************************
** Markers belong to retention rules and are never editable in child lists.
*******************************************************************************/
public static QJoinMetaData markersJoin()
{
return (new QJoinMetaData().withName("secured_data_rule_markers").withType(JoinType.ONE_TO_MANY)
.withLeftTable(SecuredDataTables.RETENTION_RULE).withRightTable(SecuredDataTables.WIPE_MARKER).withJoinOn(new JoinOn("id", "rule_id")));
}
/*******************************************************************************
** Standard rule child list.
*******************************************************************************/
public static QWidgetMetaData markers()
{
return (ChildRecordListRenderer.widgetMetaDataBuilder(markersJoin()).withName(MARKERS).withLabel("Wipe markers")
.withCanAddChildRecord(false).withMaxRows(50).getWidgetMetaData().withPermissionRules(SecuredDataPermissions.rules()));
}
/*******************************************************************************
** Known physical backup retention and its residual recovery risk.
*******************************************************************************/
public static QWidgetMetaData backups()
{
return (new QWidgetMetaData().withName(BACKUPS).withLabel("Backups").withType(WidgetType.HTML.getType()).withGridColumns(12)
.withCodeReference(new QCodeReference(Backups.class)).withPermissionRules(SecuredDataPermissions.rules()));
}
/*******************************************************************************
** Host adds this metadata and a section referencing its name on its own table.
** The immutable widget configuration selects the table, not request parameters.
*******************************************************************************/
public static QWidgetMetaData wipedFields(QTableMetaData table)
{
return (new QWidgetMetaData().withName("securedDataWipedFields_" + table.getName()).withLabel("Wiped fields").withType(WidgetType.HTML.getType())
.withGridColumns(12).withDefaultValue("hostTable", table.getName()).withCodeReference(new QCodeReference(WipedFields.class)));
}
/*******************************************************************************
** Render the rule through normal user permissions.
*******************************************************************************/
public static class Backups extends AbstractWidgetRenderer
{
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
List<QRecord> rules = new QueryAction().execute(new QueryInput(SecuredDataTables.RETENTION_RULE)
.withFilter(new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, input.getQueryParams().get("id"))))).getRecords();
StringBuilder html = new StringBuilder("<div>");
Integer days = SecuredDataRuntime.current().config().getBackupRetentionDays();
html.append("<p>Physical backup retention: ").append(days == null ? "unknown" : days + " days").append(".</p>");
for(QRecord rule : rules)
{
String warning = RetentionRules.backupWarning(rule);
if(warning != null)
{
html.append("<p role='alert'>").append(escape(warning)).append("</p>");
}
}
html.append("<p>CNPG physical backups and WAL cannot exclude audit keys. A backup and the KEK can recover wiped data until that backup expires.</p></div>");
return (new RenderWidgetOutput(new RawHTML(null, html.toString())));
}
}
/*******************************************************************************
** Authorize the host record with an ID-only query before looking up markers.
*******************************************************************************/
public static class WipedFields extends AbstractWidgetRenderer
{
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
String tableName = (String) input.getWidgetMetaData().getDefaultValues().get("hostTable");
var table = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getTable(tableName);
String id = input.getQueryParams().get("id");
List<QRecord> visible = new QueryAction().execute(new QueryInput(tableName).withFieldNamesToInclude(Set.of(table.getPrimaryKeyField()))
.withFilter(new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.EQUALS, id)))).getRecords();
StringBuilder html = new StringBuilder("<ul aria-label='Wiped fields'>");
if(!visible.isEmpty())
{
for(QRecord marker : SystemWrites.query(SecuredDataTables.WIPE_MARKER, new QQueryFilter()
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, tableName))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, id))))
{
List<QRecord> rules = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, marker.getValue("rule_id"))));
String age = rules.isEmpty() ? "unknown" : rules.getFirst().getValueString("age_days");
html.append("<li>").append(escape(marker.getValueString("field_name"))).append(": Wiped per policy (").append(escape(age)).append(" days)</li>");
}
}
return (new RenderWidgetOutput(new RawHTML(null, html.append("</ul>").toString())));
}
}
/*******************************************************************************
** Escape user-managed names and all text before HTML rendering.
*******************************************************************************/
private static String escape(String text)
{
return (text == null ? "" : text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace("\"", "&quot;").replace("'", "&#39;"));
}
}
@@ -0,0 +1,258 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.SecuredDataQBitConfig;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Standard Review/Result action and a daily system scheduler entry point.
*******************************************************************************/
public final class WipeProcesses
{
public static final String WIPE = "secured_data_wipe";
public static final String DAILY = "secured_data_wipe_daily";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private WipeProcesses()
{
}
/*******************************************************************************
** Processes are table actions or hidden schedules, never navigation entries.
*******************************************************************************/
public static List<QProcessMetaData> processes(SecuredDataQBitConfig config)
{
Map<String, Serializable> values = new LinkedHashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, SecuredDataTables.RETENTION_RULE);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, SecuredDataTables.WIPE_RUN);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, "Eligible records will be wiped; active legal holds are preserved.");
QPermissionRules permissions = new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
.withPermissionBaseName("secured_data").withCustomPermissionChecker(new QCodeReference(Permission.class));
QProcessMetaData wipe = StreamedETLWithFrontendProcess.defineProcessMetaData(Extract.class, Review.class, Load.class, values)
.withName(WIPE).withLabel("Run wipe now").withTableName(SecuredDataTables.RETENTION_RULE).withMinInputRecords(1).withPermissionRules(permissions);
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily retention wipe").withIsHidden(true).withPermissionRules(permissions)
.withStep(new QBackendStepMetaData().withName("wipe").withCode(new QCodeReference(Daily.class)));
if(config.getSchedulerName() != null)
{
daily.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(config.getWipeIntervalSeconds()).withInitialDelaySeconds(30));
}
return (List.of(wipe, daily));
}
/*******************************************************************************
** Explicit retention permission is independent of classified-value access.
*******************************************************************************/
public static class Permission implements CustomPermissionChecker
{
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
WipeService.checkPermission();
}
}
/*******************************************************************************
** Select policies using the standard table-selection filter, then stream IDs.
*******************************************************************************/
public static class Extract extends ExtractViaQueryStep
{
@Override
public Integer doCount(RunBackendStepInput input)
{
return (null);
}
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
for(QRecord rule : SystemWrites.query(SecuredDataTables.RETENTION_RULE, getQueryFilter(input)))
{
new WipeService().extract(rule, packets ->
{
for(QRecord packet : packets)
{
getRecordPipe().addRecord(packet);
}
});
}
}
}
/*******************************************************************************
** Counts per policy and hold status; no record or field values in summaries.
*******************************************************************************/
public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final Map<String, ProcessSummaryLine> lines = new LinkedHashMap<>();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
for(QRecord packet : input.getRecords())
{
String id = packet.getValueString("rule_id");
boolean held = Boolean.TRUE.equals(packet.getValueBoolean("held"));
String label = held ? "held" : "to wipe";
lines.computeIfAbsent(id + label, key -> line(id, held, false)).incrementCountAndAddPrimaryKey(null);
output.getRecords().add(packet);
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
return (new ArrayList<>(lines.values()));
}
}
/*******************************************************************************
** Group only the current ETL page, splitting again by each policy's batch size.
*******************************************************************************/
public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final Map<String, ProcessSummaryLine> lines = new LinkedHashMap<>();
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
WipeService.checkPermission();
Map<Serializable, List<Serializable>> rules = new LinkedHashMap<>();
for(QRecord packet : input.getRecords())
{
rules.computeIfAbsent(packet.getValue("rule_id"), ignored -> new ArrayList<>()).add(packet.getValue("id"));
}
for(var entry : rules.entrySet())
{
int size = WipeService.rule(entry.getKey()).getValueInteger("batch_size");
for(int offset = 0; offset < entry.getValue().size(); offset += size)
{
QRecord run = new WipeService().batch(entry.getKey(), entry.getValue().subList(offset, Math.min(offset + size, entry.getValue().size())));
output.getRecords().add(run);
for(boolean held : List.of(false, true))
{
String id = String.valueOf(entry.getKey());
ProcessSummaryLine line = lines.computeIfAbsent(id + held, ignored -> line(id, held, true));
for(int n = 0; n < run.getValueInteger(held ? "held_count" : "wiped_count"); n++)
{
line.incrementCountAndAddPrimaryKey(null);
}
}
}
}
}
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
return (new ArrayList<>(lines.values()));
}
}
/*******************************************************************************
** Scheduler identity must already be system; never elevate a user invocation.
*******************************************************************************/
public static class Daily implements BackendStep
{
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
if(!(QContext.getQSession() instanceof QSystemUserSession))
{
throw (new QPermissionDeniedException("Daily retention requires the system session."));
}
for(QRecord rule : SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("is_active", QCriteriaOperator.EQUALS, true))))
{
List<QException> errors = new ArrayList<>();
new WipeService().extract(rule, packets ->
{
try
{
output.getRecords().add(new WipeService().batch(rule.getValue("id"), packets.stream().map(row -> row.getValue("id")).toList()));
}
catch(QException e)
{
errors.add(e);
}
});
if(!errors.isEmpty())
{
throw (new QException("Daily retention had rolled-back batches."));
}
}
}
}
/*******************************************************************************
** Standard future/past summary text.
*******************************************************************************/
private static ProcessSummaryLine line(String id, boolean held, boolean result)
{
ProcessSummaryLine line = new ProcessSummaryLine(held ? Status.INFO : Status.OK, 0, null);
String future = "eligible records for rule " + id + (held ? " held by legal hold" : " to wipe");
String past = "records for rule " + id + (held ? " held by legal hold" : " wiped");
line.setSingularFutureMessage(future);
line.setPluralFutureMessage(future);
line.setSingularPastMessage(past);
line.setPluralPastMessage(past);
return (line);
}
}
@@ -0,0 +1,421 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.retention;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
import com.kingsrook.qbits.secureddata.SecuredDataKeySource;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.audits.AuditShreddedEvent;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.tables.WipeAuditFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** ID-only extraction and atomic, system-owned batches. No host values are placed
** in process records, run messages, access logs or exceptions.
*******************************************************************************/
public final class WipeService
{
/*******************************************************************************
** Checked-exception-aware streaming boundary.
*******************************************************************************/
@FunctionalInterface
public interface PageConsumer
{
/*******************************************************************************
** Consume one bounded page.
*******************************************************************************/
void accept(List<QRecord> records) throws QException;
}
public static final String PERMISSION = "secured_data.retention";
/*******************************************************************************
** Enforce at every callable boundary, including process resumption.
*******************************************************************************/
public static void checkPermission() throws QPermissionDeniedException
{
if(!(QContext.getQSession() instanceof QSystemUserSession)
&& (QContext.getQSession() == null || !QContext.getQSession().hasPermission(PERMISSION) || "assistant".equals(QContext.getQSession().getClientChannelName())))
{
throw (new QPermissionDeniedException("Permission denied: secured_data.retention is required."));
}
}
/*******************************************************************************
** Reload the rule instead of trusting process-state configuration.
*******************************************************************************/
public static QRecord rule(Serializable id) throws QException
{
List<QRecord> rows = SystemWrites.query(SecuredDataTables.RETENTION_RULE, new QQueryFilter().withCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id)));
if(rows.isEmpty())
{
throw (new QException("Retention rule no longer exists."));
}
QRecord rule = rows.getFirst();
RetentionRules.validate(rule);
return (rule);
}
/*******************************************************************************
** A bounded keyset scan. Per-field predicates return IDs only, even for PHI.
** Packets carry IDs, field names and a hold indicator, never stored values.
*******************************************************************************/
public void extract(QRecord rule, PageConsumer consumer) throws QException
{
checkPermission();
RetentionRules.validate(rule);
if(!Boolean.TRUE.equals(rule.getValueBoolean("is_active")))
{
return;
}
String tableName = rule.getValueString("table_name");
QTableMetaData table = QContext.getQInstance().getTable(tableName);
String pkName = table.getPrimaryKeyField();
Serializable after = null;
while(true)
{
QQueryFilter filter = eligible(rule, null).withOrderBy(new QFilterOrderBy(pkName)).withLimit(rule.getValueInteger("batch_size"));
if(after != null)
{
filter.withCriteria(new QFilterCriteria(pkName, QCriteriaOperator.GREATER_THAN, after));
}
List<QRecord> ids = query(tableName, filter, Set.of(pkName));
if(ids.isEmpty())
{
return;
}
List<QRecord> packets = new ArrayList<>();
for(QRecord id : ids)
{
Serializable pk = id.getValue(pkName);
packets.add(new QRecord().withValue("id", pk).withValue("rule_id", rule.getValue("id"))
.withValue("held", held(tableName, pk)));
}
consumer.accept(packets);
after = ids.getLast().getValue(pkName);
}
}
/*******************************************************************************
** Candidate fields are tested in SQL, not fetched.
*******************************************************************************/
private static QQueryFilter eligible(QRecord rule, String field)
{
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
Instant cutoff = SecuredDataRuntime.current().config().getClock().instant().minus(rule.getValueInteger("age_days"), ChronoUnit.DAYS);
Serializable age = table.getField(rule.getValueString("age_field_name")).getType() == QFieldType.DATE ? cutoff.atZone(ZoneOffset.UTC).toLocalDate() : cutoff;
QQueryFilter filter = new QQueryFilter().withCriteria(new QFilterCriteria(rule.getValueString("age_field_name"), QCriteriaOperator.LESS_THAN, age));
QQueryFilter nonnull = new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR);
for(String name : field == null ? RetentionRules.fields(rule) : List.of(field))
{
nonnull.withCriteria(new QFilterCriteria(name, QCriteriaOperator.IS_NOT_BLANK));
}
return (filter.withSubFilter(nonnull));
}
/*******************************************************************************
** Active record and whole-table holds pause all policies.
*******************************************************************************/
public static boolean held(String table, Serializable pk)
{
return (!SystemWrites.query(SecuredDataTables.LEGAL_HOLD, new QQueryFilter()
.withCriteria(new QFilterCriteria("table_name", QCriteriaOperator.EQUALS, table))
.withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"))
.withSubFilter(new QQueryFilter().withBooleanOperator(QQueryFilter.BooleanOperator.OR)
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.IS_BLANK))
.withCriteria(new QFilterCriteria("record_id", QCriteriaOperator.EQUALS, String.valueOf(pk))))).isEmpty());
}
/*******************************************************************************
** Explicit projection and transaction for every host read.
*******************************************************************************/
private static List<QRecord> query(String table, QQueryFilter filter, Set<String> fields) throws QException
{
return (SystemWrites.call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withFieldNamesToInclude(fields)
.withShouldFetchHeavyFields(true).withTransaction(QContext.getQBackendTransaction())).getRecords()));
}
/*******************************************************************************
** Revalidate eligibility and holds inside the batch, then commit all effects.
** Cross-backend and nontransactional deployments are refused before any writes.
*******************************************************************************/
public QRecord batch(Serializable ruleId, List<Serializable> ids) throws QException
{
checkPermission();
QRecord rule = rule(ruleId);
QTableMetaData table = QContext.getQInstance().getTable(rule.getValueString("table_name"));
if(ids.size() > rule.getValueInteger("batch_size"))
{
throw (new QException("Wipe batch exceeds the policy batch size."));
}
if(QContext.getQBackendTransaction() != null)
{
throw (new QException("Wipe batches must own their transaction."));
}
for(String name : List.of(SecuredDataTables.RETENTION_RULE, SecuredDataTables.WIPE_MARKER, SecuredDataTables.LEGAL_HOLD,
SecuredDataTables.WIPE_RUN, SecuredDataTables.AUDIT_KEY, SecuredDataTables.ACCESS_LOG, "audit", "auditDetail"))
{
QTableMetaData metadata = QContext.getQInstance().getTable(name);
if(metadata != null && !Objects.equals(metadata.getBackendName(), table.getBackendName()))
{
throw (new QException("Wipe, audit and secured-data tables must share one transactional backend."));
}
}
if("ARCHIVE_THEN_WIPE".equals(rule.getValueString("action")) && SecuredDataRuntime.current().config().getArchiveSink() == null)
{
throw (new QException("ARCHIVE_THEN_WIPE requires a host ArchiveSinkInterface backed by object storage."));
}
try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(table.getName())))
{
if(transaction == null || transaction.getClass() == QBackendTransaction.class)
{
throw (new QException("Wipe requires a backend with rollback-capable write transactions."));
}
QContext.setTransaction(transaction);
try
{
QRecord result = SystemWrites.call(() -> apply(rule, table, ids, transaction));
transaction.commit();
return (result);
}
catch(Exception e)
{
transaction.rollback();
throw (new QException("Wipe batch failed and was rolled back; no values are included in this message."));
}
finally
{
SecuredDataRuntime.current().keys().close();
QContext.clearTransaction();
}
}
}
/*******************************************************************************
** Batch body. Markers, key deletion, audit, access log and run row share tx.
*******************************************************************************/
private QRecord apply(QRecord initialRule, QTableMetaData table, List<Serializable> ids, QBackendTransaction transaction) throws QException
{
QRecord rule = rule(initialRule.getValue("id"));
Instant started = SecuredDataRuntime.current().config().getClock().instant();
List<Serializable> wiped = new ArrayList<>();
int held = 0;
Map<Serializable, List<String>> fieldsById = new LinkedHashMap<>();
if(Boolean.TRUE.equals(rule.getValueBoolean("is_active")))
{
for(String field : RetentionRules.fields(rule))
{
QQueryFilter filter = eligible(rule, field).withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.IN, ids));
for(QRecord row : query(table.getName(), filter, Set.of(table.getPrimaryKeyField())))
{
fieldsById.computeIfAbsent(row.getValue(table.getPrimaryKeyField()), ignored -> new ArrayList<>()).add(field);
}
}
}
List<QRecord> markers = new ArrayList<>();
Map<String, List<Serializable>> logged = new LinkedHashMap<>();
for(var entry : fieldsById.entrySet())
{
Serializable pk = entry.getKey();
if(held(table.getName(), pk))
{
held++;
continue;
}
List<String> fields = entry.getValue();
String action = rule.getValueString("action");
if("DELETE_RECORD".equals(action))
{
fields = new ArrayList<>(table.getFields().keySet());
fields.remove(table.getPrimaryKeyField());
}
change(rule, table, pk, fields, transaction);
for(String field : fields)
{
List<QRecord> keys = query(SecuredDataTables.AUDIT_KEY, SecuredDataKeySource.recordField(table.getName(), field, pk), Set.of("id"));
for(QRecord key : keys)
{
var deleted = new DeleteAction().execute(new DeleteInput(SecuredDataTables.AUDIT_KEY).withPrimaryKeys(List.of(key.getValue("id"))).withTransaction(transaction));
if(deleted.getDeletedRecordCount() != 1)
{
throw (new QException("Audit key deletion failed."));
}
AuditShreddedEvent.shreddedDetail(table, pk, table.getField(field), rule.getValueString("id"), transaction);
}
markers.add(new QRecord().withValue("table_name", table.getName()).withValue("record_id", String.valueOf(pk)).withValue("field_name", field)
.withValue("rule_id", rule.getValue("id")).withValue("action", action).withValue("wiped_at", started));
logged.computeIfAbsent(field, ignored -> new ArrayList<>()).add(pk);
}
wiped.add(pk);
}
QRecord run = SystemWrites.insertOne(SecuredDataTables.WIPE_RUN, new QRecord().withValue("rule_id", rule.getValue("id"))
.withValue("started_at", started).withValue("finished_at", SecuredDataRuntime.current().config().getClock().instant())
.withValue("status", "COMPLETE").withValue("eligible_count", fieldsById.size()).withValue("held_count", held)
.withValue("wiped_count", wiped.size()).withValue("error_count", 0).withValue("record_ids", boundedIds(wiped))
.withValue("message", "Policy batch completed."), transaction);
for(QRecord marker : markers)
{
marker.setValue("run_id", run.getValue("id"));
List<QRecord> previous = SystemWrites.query(SecuredDataTables.WIPE_MARKER, SecuredDataKeySource.recordField(marker.getValueString("table_name"), marker.getValueString("field_name"), marker.getValue("record_id")));
if(previous.isEmpty())
{
SystemWrites.insertOne(SecuredDataTables.WIPE_MARKER, marker, transaction);
}
else
{
marker.setValue("id", previous.getFirst().getValue("id"));
update(SecuredDataTables.WIPE_MARKER, marker, transaction);
}
}
for(var entry : logged.entrySet())
{
for(int offset = 0; offset < entry.getValue().size(); offset += 50)
{
List<Serializable> chunk = entry.getValue().subList(offset, Math.min(offset + 50, entry.getValue().size()));
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, ActivityLog.row("WIPE", table.getName(), "ALLOWED")
.withValue("rule_id", rule.getValue("id")).withValue("field_name", entry.getKey()).withValue("record_id", boundedIds(chunk))
.withValue("record_count", chunk.size()).withValue("action_name", rule.getValue("action")), transaction);
}
}
update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", rule.getValue("id")).withValue("last_run_at", started).withValue("last_run_count", wiped.size()), transaction);
return (run);
}
/*******************************************************************************
** The two value transformations read only the fields they transform; all
** other operations work entirely by ID. Archive receives encrypted JSON only.
*******************************************************************************/
private static void change(QRecord rule, QTableMetaData table, Serializable pk, List<String> fields, QBackendTransaction transaction) throws QException
{
String action = rule.getValueString("action");
if("DELETE_RECORD".equals(action))
{
if(new DeleteAction().execute(new DeleteInput(table.getName()).withPrimaryKeys(List.of(pk)).withTransaction(transaction)).getDeletedRecordCount() != 1)
{
throw (new QException("Record deletion failed."));
}
return;
}
QRecord stored = null;
if(Set.of("KEYED_HASH", "ARCHIVE_THEN_WIPE").contains(action))
{
stored = query(table.getName(), new QQueryFilter().withCriteria(new QFilterCriteria(table.getPrimaryKeyField(), QCriteriaOperator.EQUALS, pk)), new LinkedHashSet<>(fields)).getFirst();
}
if("ARCHIVE_THEN_WIPE".equals(action))
{
String objectKey = "secured-data/" + UUID.randomUUID() + ".aesgcm";
Map<String, Serializable> values = new LinkedHashMap<>();
for(String field : fields)
{
values.put(field, stored.getValue(field));
}
byte[] json = JsonUtils.toJson(values).getBytes(StandardCharsets.UTF_8);
try
{
byte[] encrypted = SecuredDataKeySource.encrypt(json, objectKey);
SecuredDataRuntime.current().config().getArchiveSink().put(rule.getValueString("archive_storage_name"), objectKey, encrypted);
}
finally
{
Arrays.fill(json, (byte) 0);
values.clear();
}
}
QRecord patch = new QRecord().withValue(table.getPrimaryKeyField(), pk);
for(String field : fields)
{
Serializable value = switch(action)
{
case "OVERWRITE_TEXT" -> rule.getValue("overwrite_text");
case "KEYED_HASH" -> SecuredDataRuntime.current().keys().instanceToken(table, table.getField(field), pk, stored.getValue(field));
default -> null;
};
patch.setValue(field, value);
}
UpdateInput input = new UpdateInput(table.getName()).withRecord(patch).withTransaction(transaction);
input.setFlags(Set.of(new WipeAuditFlag(rule.getValueString("id"))));
check(new UpdateAction().execute(input).getRecords());
}
/*******************************************************************************
** Check record-level backend failures, which need not throw exceptions.
*******************************************************************************/
private static void check(List<QRecord> records) throws QException
{
if(records.isEmpty() || records.stream().anyMatch(row -> row.getErrors() != null && !row.getErrors().isEmpty()))
{
throw (new QException("Wipe write failed."));
}
}
/*******************************************************************************
** System metadata updates still validate in the transaction.
*******************************************************************************/
private static void update(String table, QRecord row, QBackendTransaction transaction) throws QException
{
check(new UpdateAction().execute(new UpdateInput(table).withRecord(row).withTransaction(transaction)).getRecords());
}
/*******************************************************************************
** Bounded identifiers, never values. Full record evidence is in audit rows.
*******************************************************************************/
private static String boundedIds(List<Serializable> ids)
{
String joined = ids.stream().limit(50).map(String::valueOf).collect(Collectors.joining(","));
return (joined.length() > 4000 ? joined.substring(0, 4000) : joined);
}
}
@@ -0,0 +1,37 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.reveal;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** Transport-independent refusal, mapped to 429 by the UI API.
*******************************************************************************/
public class RevealCappedException extends QException
{
/*******************************************************************************
** Daily allowance exhausted.
*******************************************************************************/
public RevealCappedException()
{
super("Daily reveal allowance exhausted.");
}
}
@@ -0,0 +1,31 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.reveal;
import java.util.List;
/*******************************************************************************
** Policy-driven reveal dialog. Null remainingDailyAllowance means unlimited.
*******************************************************************************/
public record RevealOptions(List<String> reasonOptions, boolean allowFreeTextReason, boolean requireFreeText,
int revealMinutes, Integer remainingDailyAllowance)
{
}
@@ -0,0 +1,31 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.reveal;
import java.io.Serializable;
import java.time.Instant;
/*******************************************************************************
** The authorized value and its grant; never persist this response in an audit row.
*******************************************************************************/
public record RevealResult(Serializable value, String displayValue, Instant expiresAt, Serializable revealId)
{
}
@@ -0,0 +1,215 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.reveal;
import java.io.Serializable;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Objects;
import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Reveal authorization and grant creation independent of any HTTP transport.
*******************************************************************************/
public class RevealService
{
/*******************************************************************************
** Use the current instance runtime, including its injectable clock.
*******************************************************************************/
public RevealOptions options(String tableName, String fieldName) throws QException
{
requireActor();
QRecord policy = policy(tableName, fieldName);
String reasons = policy.getValueString("reason_options");
Integer cap = policy.getValueInteger("daily_reveal_cap");
return (new RevealOptions(reasons == null ? List.of() : Arrays.stream(reasons.split("\\R")).map(String::trim).filter(s -> !s.isEmpty()).toList(),
!Boolean.FALSE.equals(policy.getValueBoolean("allow_free_text_reason")), Boolean.TRUE.equals(policy.getValueBoolean("require_free_text")),
Objects.requireNonNullElse(policy.getValueInteger("reveal_minutes"), 5), cap == null ? null : Math.max(0, cap - usedToday())));
}
/*******************************************************************************
** A single coherent policy: unlimited/highest cap, then longest duration, then id.
*******************************************************************************/
private QRecord policy(String tableName, String fieldName) throws QException
{
requireTarget(tableName, fieldName);
if("assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
throw (new QPermissionDeniedException("Reveal is unavailable on the assistant channel."));
}
QTableMetaData table = QContext.getQInstance().getTable(tableName);
if(table == null || !table.getFields().containsKey(fieldName))
{
throw (new QNotFoundException("Table or field not found."));
}
GetInput input = new GetInput(tableName);
input.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
CorePermissionEnforcer.checkTablePermissions(input, TablePermissionSubType.READ);
if(SecuredDataRuntime.current().resolver().resolve(table, table.getField(fieldName)).tier() != FieldAccessTier.REVEALABLE)
{
throw (new QPermissionDeniedException("Field is not revealable."));
}
return (SecuredDataRuntime.current().resolver().revealPolicies(table, fieldName).stream()
.max(Comparator.comparingInt((QRecord row) -> Objects.requireNonNullElse(row.getValueInteger("daily_reveal_cap"), Integer.MAX_VALUE))
.thenComparingInt(row -> Objects.requireNonNullElse(row.getValueInteger("reveal_minutes"), 5))
.thenComparing(row -> row.getValueString("id"), Comparator.reverseOrder()))
.orElse(new QRecord().withValue("reveal_minutes", 5).withValue("allow_free_text_reason", true)));
}
/*******************************************************************************
** UTC daily count across all tables, including expired grants.
*******************************************************************************/
public int usedToday()
{
Instant start = SecuredDataRuntime.current().config().getClock().instant().atOffset(ZoneOffset.UTC).toLocalDate().atStartOfDay().toInstant(ZoneOffset.UTC);
return (SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()
.withCriteria(new QFilterCriteria("user_id", QCriteriaOperator.EQUALS, SecuredDataResolver.userId()))
.withCriteria(new QFilterCriteria("granted_at", QCriteriaOperator.GREATER_THAN_OR_EQUALS, start))).size());
}
/*******************************************************************************
** Serialize the cap check and write within this runtime; first fetch always as caller.
*******************************************************************************/
public RevealResult reveal(String tableName, Serializable primaryKey, String fieldName, String reason, String reasonText) throws QException
{
requireActor();
SecuredDataRuntime runtime = SecuredDataRuntime.current();
synchronized(runtime)
{
if("assistant".equals(SecuredDataMetadata.channel(QContext.getQSession())))
{
denied(tableName, primaryKey, fieldName, "DENIED");
throw (new QPermissionDeniedException("Reveal is unavailable on the assistant channel."));
}
if(primaryKey == null)
{
throw (new QBadRequestException("primaryKey is required."));
}
requireTarget(tableName, fieldName);
GetInput input = new GetInput(tableName).withPrimaryKey(primaryKey).withShouldFetchHeavyFields(true);
input.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
if(new GetAction().execute(input).getRecord() == null)
{
throw (new QNotFoundException("No record was found"));
}
RevealOptions options = options(tableName, fieldName);
validateReason(options, reason, reasonText);
if(options.remainingDailyAllowance() != null && options.remainingDailyAllowance() == 0)
{
denied(tableName, primaryKey, fieldName, "CAPPED");
QAlerts.raise(new AlertSignal("secured_data.reveal_capped").withDedupKey("secured_data.reveal_capped:" + SecuredDataResolver.userId())
.withTitle("Daily reveal allowance exhausted").withSummary("Review the secured-data access log."));
throw (new RevealCappedException());
}
QRecord raw = SystemWrites.call(() -> new GetAction().execute(new GetInput(tableName).withPrimaryKey(primaryKey)
.withShouldFetchHeavyFields(true).withShouldTranslatePossibleValues(true)).getRecord());
if(raw == null)
{
throw (new QNotFoundException("No record was found"));
}
com.kingsrook.qqq.backend.core.actions.values.QValueFormatter.setDisplayValuesInRecords(QContext.getQInstance().getTable(tableName), List.of(raw));
Instant now = runtime.config().getClock().instant();
Instant expiry = now.plusSeconds(options.revealMinutes() * 60L);
QRecord audit = SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, ActivityLog.row("REVEAL", tableName, "ALLOWED")
.withValue("record_id", String.valueOf(primaryKey)).withValue("field_name", fieldName).withValue("reason", reason).withValue("reason_text", reasonText));
QRecord grant = SystemWrites.insertOne(SecuredDataTables.REVEAL_GRANT, new QRecord()
.withValue("user_id", SecuredDataResolver.userId()).withValue("user_name", QContext.getQSession().getUser().getFullName())
.withValue("table_name", tableName).withValue("field_name", fieldName).withValue("record_id", String.valueOf(primaryKey))
.withValue("reason", reason).withValue("reason_text", reasonText).withValue("granted_at", now).withValue("expires_at", expiry)
.withValue("access_log_id", audit.getValue("id")));
runtime.resolver().invalidateReveals();
return (new RevealResult(raw.getValue(fieldName), raw.getDisplayValue(fieldName), expiry, grant.getValue("id")));
}
}
/*******************************************************************************
** Grants require a stable authenticated user identity across sessions.
*******************************************************************************/
private void requireActor() throws QPermissionDeniedException
{
if("anonymous".equals(SecuredDataResolver.userId()) || SecuredDataResolver.userId().isBlank())
{
throw (new QPermissionDeniedException("Reveal requires an authenticated user."));
}
}
/*******************************************************************************
** Missing target identifiers are malformed input, not an internal lookup error.
*******************************************************************************/
private void requireTarget(String table, String field) throws QBadRequestException
{
if(table == null || table.isBlank() || field == null || field.isBlank())
{
throw (new QBadRequestException("tableName and fieldName are required."));
}
}
/*******************************************************************************
** Persist refusals without user supplied text or protected values.
*******************************************************************************/
private void denied(String table, Serializable pk, String field, String outcome) throws QException
{
SystemWrites.insertOne(SecuredDataTables.ACCESS_LOG, ActivityLog.row("REVEAL_DENIED", table, outcome)
.withValue("record_id", String.valueOf(pk)).withValue("field_name", field));
}
/*******************************************************************************
** Reasons are a listed option or allowed free text; both inputs are bounded.
*******************************************************************************/
private void validateReason(RevealOptions options, String reason, String text) throws QBadRequestException
{
boolean hasText = text != null && !text.isBlank();
boolean listed = reason != null && options.reasonOptions().contains(reason);
if((reason != null && reason.length() > 500) || (text != null && text.length() > 500)
|| (options.requireFreeText() && !hasText)
|| (!options.allowFreeTextReason() && hasText && !options.requireFreeText())
|| (!listed && !(options.allowFreeTextReason() && (hasText || (reason != null && !reason.isBlank())))))
{
throw (new QBadRequestException("Supply an allowed reason and required explanation, each at most 500 characters."));
}
}
}
@@ -0,0 +1,52 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import com.kingsrook.qbits.secureddata.SecuredDataResolver;
import com.kingsrook.qbits.secureddata.SecuredDataRuntime;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Synchronous activity audit rows for state-changing operations.
*******************************************************************************/
public final class ActivityLog
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private ActivityLog()
{
}
/*******************************************************************************
** Capture the actor before entering the system session. No field values.
*******************************************************************************/
public static QRecord row(String event, String table, String outcome)
{
return (new QRecord().withValue("event_type", event).withValue("table_name", table)
.withValue("user_id", SecuredDataResolver.userId())
.withValue("user_name", QContext.getQSession().getUser() == null ? "system" : QContext.getQSession().getUser().getFullName())
.withValue("client_channel", SecuredDataMetadata.channel(QContext.getQSession()))
.withValue("created_at", SecuredDataRuntime.current().config().getClock().instant())
.withValue("outcome", outcome).withValue("anomaly_flag", false));
}
}
@@ -0,0 +1,124 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermission;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Secured-data defaults and registration using the released core metadata API.
*******************************************************************************/
public final class SecuredDataMetadata
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataMetadata()
{
}
/*******************************************************************************
** Normalized trusted client channel.
*******************************************************************************/
public static String channel(QSession session)
{
return (session == null ? null : session.getClientChannelName());
}
/*******************************************************************************
** Read code classification without applying field/table precedence.
*******************************************************************************/
public static DataClassification classification(QTableMetaData table, QFieldMetaData field)
{
if(field == null)
{
AiTableMetaData ai = AiTableMetaData.of(table);
return (ai == null ? null : ai.getClassification());
}
AiFieldMetaData ai = AiFieldMetaData.of(field);
return (ai == null ? null : ai.getClassification());
}
/*******************************************************************************
** Protect the QBit's own AI routing metadata.
*******************************************************************************/
public static void classifyTable(QTableMetaData table)
{
AiTableMetaData.ofOrWithNew(table).withExposure(AiExposure.NONE).withClassification(DataClassification.CONFIDENTIAL);
}
/*******************************************************************************
** Seed opt-in name heuristics without replacing other AI field metadata.
*******************************************************************************/
public static void classifyField(QFieldMetaData field, DataClassification classification)
{
AiFieldMetaData ai = AiFieldMetaData.of(field);
if(ai == null)
{
ai = new AiFieldMetaData();
field.withSupplementalMetaData(ai);
}
ai.withClassification(classification);
}
/*******************************************************************************
** Register the assistant wall and both table features.
*******************************************************************************/
public static void register(QInstance instance)
{
ClientChannelsMetaData.ofOrWithNew(instance).withDeniedPattern("assistant", "secured_data_*");
for(String suffix : new String[] {"secured_data_full", "secured_data_reveal"})
{
TableFeaturePermissionsMetaData.ofOrWithNew(instance).withPermission(new TableFeaturePermission()
.withSuffix(suffix).withLabel("Secured data: " + suffix.substring("secured_data_".length())));
}
}
/*******************************************************************************
** Use core feature semantics, including table READ and permission base names.
*******************************************************************************/
public static boolean hasFeature(QTableMetaData table, String suffix)
{
return (PermissionsHelper.hasTableFeaturePermission(table, suffix));
}
/*******************************************************************************
** Effective permission base name.
*******************************************************************************/
public static String permissionBase(QTableMetaData table)
{
QPermissionRules rules = PermissionsHelper.getEffectivePermissionRules(table, QContext.getQInstance());
String base = rules.getPermissionBaseName();
return (base == null || base.isBlank() ? table.getName() : base);
}
}
@@ -0,0 +1,173 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.support;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
/*******************************************************************************
** System-session lookups and writes; restoring the caller even after a failure.
*******************************************************************************/
public final class SystemWrites
{
private static final ThreadLocal<Integer> DEPTH = ThreadLocal.withInitial(() -> 0);
/*******************************************************************************
** Work inside a system session.
*******************************************************************************/
@FunctionalInterface
public interface Work<T>
{
/*******************************************************************************
** Execute.
*******************************************************************************/
T run() throws QException;
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SystemWrites()
{
}
/*******************************************************************************
** Preserve the transaction and instance, replacing only the session.
*******************************************************************************/
public static <T> T call(Work<T> work) throws QException
{
QSession previous = QContext.getQSession();
DEPTH.set(DEPTH.get() + 1);
QContext.setQSession(new QSystemUserSession(QContext.getQInstance()));
try(CorePermissionEnforcer.AuthorizedScope ignored = CorePermissionEnforcer.openAuthorizedScope())
{
return (work.run());
}
finally
{
QContext.setQSession(previous);
DEPTH.set(DEPTH.get() - 1);
}
}
/*******************************************************************************
** Reentrancy guard used by the resolver.
*******************************************************************************/
public static boolean active()
{
return (DEPTH.get() > 0);
}
/*******************************************************************************
** Query without user locks or field access recursion.
*******************************************************************************/
public static List<QRecord> query(String table, QQueryFilter filter)
{
try
{
return (call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withTransaction(QContext.getQBackendTransaction())).getRecords()));
}
catch(QException e)
{
throw (new IllegalStateException("Unable to read secured-data state", e));
}
}
/*******************************************************************************
** Insert and reject backend record errors.
*******************************************************************************/
public static QRecord insertOne(String table, QRecord record) throws QException
{
return (insertOne(table, record, QContext.getQBackendTransaction()));
}
/*******************************************************************************
** Explicit transaction for post-write customizers; audit rolls back with its source.
*******************************************************************************/
public static QRecord insertOne(String table, QRecord record, QBackendTransaction transaction) throws QException
{
if(com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.ACCESS_LOG.equals(table))
{
synchronized(com.kingsrook.qbits.secureddata.SecuredDataRuntime.current().anomalies())
{
return (insertAndAlert(table, record, transaction));
}
}
return (insertAndAlert(table, record, transaction));
}
/*******************************************************************************
** Serialize same-node counter evaluation and insertion. Preserve enumeration flags.
*******************************************************************************/
private static QRecord insertAndAlert(String table, QRecord record, QBackendTransaction transaction) throws QException
{
boolean anomaly = com.kingsrook.qbits.secureddata.metadata.SecuredDataTables.ACCESS_LOG.equals(table)
&& com.kingsrook.qbits.secureddata.SecuredDataRuntime.current().anomalies().evaluate(record);
if(anomaly)
{
record.setValue("anomaly_flag", true);
}
QRecord inserted = call(() -> new InsertAction().execute(new InsertInput(table).withRecord(record).withTransaction(transaction)).getRecords().getFirst());
if(inserted.getErrors() != null && !inserted.getErrors().isEmpty())
{
throw (new QException("Secured-data system insert failed"));
}
if(anomaly)
{
if(transaction == null)
{
com.kingsrook.qbits.secureddata.SecuredDataAnomalies.raise(inserted);
}
else
{
var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(QContext.getQInstance(), QContext.getQSession());
transaction.addAfterCommitCallback(() -> QContext.withTemporaryContext(context,
() -> com.kingsrook.qbits.secureddata.SecuredDataAnomalies.raise(inserted)));
}
}
return (inserted);
}
}
@@ -0,0 +1,56 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.uiapi;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtension;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtensionsMetaData;
/*******************************************************************************
** Secured-data extension for the released UI API.
*******************************************************************************/
public final class SecuredDataExtension
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataExtension()
{
}
/*******************************************************************************
** The secured-data UI API, version 1.1.0.
*******************************************************************************/
public static MiddlewareSpecExtension defineExtension()
{
return (new MiddlewareSpecExtension().withName("secured-data").withSemanticVersion("1.1.0")
.withDescription("Reveal grants and break-glass status")
.withSpec(SecuredDataRevealSpecV1.class).withSpec(SecuredDataRevealOptionsSpecV1.class).withSpec(SecuredDataStatusSpecV1.class));
}
/*******************************************************************************
** Register directly with the middleware extension catalog.
*******************************************************************************/
public static void register(QInstance instance)
{
MiddlewareSpecExtensionsMetaData.ofOrWithNew(instance).withExtension(defineExtension());
}
}
@@ -0,0 +1,84 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.uiapi;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import com.kingsrook.qqq.openapi.model.Parameter;
import com.kingsrook.qqq.openapi.model.Schema;
/*******************************************************************************
** GET secured-data/reveal/options.
*******************************************************************************/
public class SecuredDataRevealOptionsSpecV1 extends AbstractJsonEndpointSpec
{
/*******************************************************************************
** Operation metadata.
*******************************************************************************/
@Override
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/reveal/options").withHttpMethod(HttpMethod.GET)
.withTag(() -> "secured-data").withShortSummary("RevealOptions"));
}
/*******************************************************************************
** Request query parameters.
*******************************************************************************/
@Override
public List<Parameter> defineRequestParameters()
{
return (List.of(new Parameter().withName("tableName").withIn(com.kingsrook.qqq.openapi.model.In.QUERY).withRequired(true).withSchema(new Schema().withType(com.kingsrook.qqq.openapi.model.Type.STRING)),
new Parameter().withName("fieldName").withIn(com.kingsrook.qqq.openapi.model.In.QUERY).withRequired(true).withSchema(new Schema().withType(com.kingsrook.qqq.openapi.model.Type.STRING))));
}
/*******************************************************************************
** Published component schemas.
*******************************************************************************/
@Override
public Map<String, Schema> defineComponentSchemas()
{
Map<String, Schema> schemas = SecuredDataSchemas.all();
return (Map.of("SecuredDataRevealOptionsResponseV1", schemas.get("SecuredDataRevealOptionsResponseV1")));
}
/*******************************************************************************
** Response component.
*******************************************************************************/
@Override
protected String getResponseSchemaName()
{
return ("SecuredDataRevealOptionsResponseV1");
}
/*******************************************************************************
** Execute with the caller session supplied by the core middleware.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new RevealService().options(input.getQueryParam("tableName"), input.getQueryParam("fieldName")));
}
}
@@ -0,0 +1,144 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.uiapi;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.reveal.RevealCappedException;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
import com.kingsrook.qqq.middleware.javalin.specs.v1.responses.BasicErrorResponseV1;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import com.kingsrook.qqq.openapi.model.Parameter;
import com.kingsrook.qqq.openapi.model.Schema;
/*******************************************************************************
** POST secured-data/reveal.
*******************************************************************************/
public class SecuredDataRevealSpecV1 extends AbstractJsonEndpointSpec
{
/*******************************************************************************
** Operation metadata.
*******************************************************************************/
@Override
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/reveal").withHttpMethod(HttpMethod.POST)
.withTag(() -> "secured-data").withShortSummary("Reveal"));
}
/*******************************************************************************
** Request query parameters.
*******************************************************************************/
@Override
public List<Parameter> defineRequestParameters()
{
return (List.of());
}
/*******************************************************************************
** Published component schemas.
*******************************************************************************/
@Override
public Map<String, Schema> defineComponentSchemas()
{
Map<String, Schema> schemas = SecuredDataSchemas.all();
return (Map.of("SecuredDataRevealRequestV1", schemas.get("SecuredDataRevealRequestV1"), "SecuredDataRevealResponseV1", schemas.get("SecuredDataRevealResponseV1"), "SecuredDataErrorV1", schemas.get("SecuredDataErrorV1")));
}
/*******************************************************************************
** Response component.
*******************************************************************************/
@Override
protected String getResponseSchemaName()
{
return ("SecuredDataRevealResponseV1");
}
/*******************************************************************************
** Standard error responses, including the daily-limit refusal.
*******************************************************************************/
@Override
public List<com.kingsrook.qqq.middleware.javalin.specs.BasicResponse> defineAdditionalBasicResponses()
{
return (List.of(new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.BAD_REQUEST, "Invalid reveal request", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.FORBIDDEN, "Reveal denied", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.NOT_FOUND, "Record not found or locked", BasicErrorResponseV1.class.getSimpleName()),
new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.TOO_MANY_REQUESTS, "Daily reveal cap reached", BasicErrorResponseV1.class.getSimpleName())));
}
/*******************************************************************************
** Required JSON request.
*******************************************************************************/
@Override
protected Schema defineRequestBodySchema()
{
return (new Schema().withRefToSchema("SecuredDataRevealRequestV1"));
}
/*******************************************************************************
** Reject wrong JSON types with the standard 400 response.
*******************************************************************************/
private String string(Map<String, Object> body, String key) throws QBadRequestException
{
Object value = body.get(key);
if(value != null && !(value instanceof String))
{
throw (new QBadRequestException(key + " must be a string."));
}
return ((String) value);
}
/*******************************************************************************
** Execute with the caller session supplied by the core middleware.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
Map<String, Object> body = input.getBody();
Object pk = body.get("primaryKey");
if(!(pk instanceof String || pk instanceof Number))
{
throw (new QBadRequestException("primaryKey must be a string or number."));
}
return (new RevealService().reveal(string(body, "tableName"), (Serializable) pk, string(body, "fieldName"),
string(body, "reason"), string(body, "reasonText")));
}
/*******************************************************************************
** Preserve core error bodies and use 429 for exhausted daily allowances.
*******************************************************************************/
@Override
protected void handleException(io.javalin.http.Context context, Exception e)
{
if(e instanceof RevealCappedException)
{
com.kingsrook.qqq.middleware.javalin.QJavalinUtils.respondWithError(context, org.eclipse.jetty.http.HttpStatus.Code.TOO_MANY_REQUESTS, e.getMessage());
}
else
{
super.handleException(context, e);
}
}
}
@@ -0,0 +1,86 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.uiapi;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.openapi.model.Schema;
import com.kingsrook.qqq.openapi.model.Type;
/*******************************************************************************
** Published shapes of service responses; nullable allowances mean unlimited.
*******************************************************************************/
public final class SecuredDataSchemas
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private SecuredDataSchemas()
{
}
/*******************************************************************************
** Every request and response component carries the extension prefix and version.
*******************************************************************************/
public static Map<String, Schema> all()
{
Map<String, Schema> schemas = new LinkedHashMap<>();
schemas.put("SecuredDataRevealRequestV1", object("tableName", type(Type.STRING), "fieldName", type(Type.STRING),
"primaryKey", new Schema().withAnyOf(List.of(type(Type.STRING), type(Type.NUMBER))).withDescription("Record primary key (string or number)."),
"reason", type(Type.STRING).withNullable(true).withMaxLength(500), "reasonText", type(Type.STRING).withNullable(true).withMaxLength(500))
.withDescription("tableName, primaryKey and fieldName are required. Supply a policy reason or allowed free text; maximum 500 characters."));
schemas.put("SecuredDataRevealResponseV1", object("value", new Schema().withNullable(true).withDescription("Authorized raw field value."),
"displayValue", type(Type.STRING).withNullable(true), "expiresAt", type(Type.STRING).withFormat("date-time"), "revealId", type(Type.INTEGER)));
schemas.put("SecuredDataRevealOptionsResponseV1", object("reasonOptions", type(Type.ARRAY).withItems(type(Type.STRING)),
"allowFreeTextReason", type(Type.BOOLEAN), "requireFreeText", type(Type.BOOLEAN), "revealMinutes", type(Type.INTEGER),
"remainingDailyAllowance", type(Type.INTEGER).withNullable(true)));
schemas.put("SecuredDataBreakGlassV1", object("id", type(Type.INTEGER), "reason", type(Type.STRING), "expiresAt", type(Type.STRING).withFormat("date-time"),
"scope", type(Type.STRING).withNullable(true), "ceiling", type(Type.STRING)));
schemas.put("SecuredDataRevealAllowanceV1", object("tableName", type(Type.STRING), "fieldName", type(Type.STRING), "remainingDailyAllowance", type(Type.INTEGER).withNullable(true)));
schemas.put("SecuredDataStatusResponseV1", object("breakGlass", new Schema().withRefToSchema("SecuredDataBreakGlassV1").withNullable(true),
"revealsUsedToday", type(Type.INTEGER), "revealAllowances", type(Type.ARRAY).withItems(new Schema().withRefToSchema("SecuredDataRevealAllowanceV1"))));
schemas.put("SecuredDataErrorV1", object("error", type(Type.STRING))
.withDescription("Deprecated since 1.1.0: errors use BasicErrorResponseV1."));
return (schemas);
}
/*******************************************************************************
** Object properties in stable order.
*******************************************************************************/
private static Schema object(Object... pairs)
{
Map<String, Schema> properties = new LinkedHashMap<>();
for(int i = 0; i < pairs.length; i += 2)
{
properties.put((String) pairs[i], (Schema) pairs[i + 1]);
}
return (type(Type.OBJECT).withProperties(properties));
}
/*******************************************************************************
** Typed schema.
*******************************************************************************/
private static Schema type(Type type)
{
return (new Schema().withType(type));
}
}
@@ -0,0 +1,83 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata.uiapi;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.StatusService;
import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import com.kingsrook.qqq.openapi.model.Parameter;
import com.kingsrook.qqq.openapi.model.Schema;
/*******************************************************************************
** GET secured-data/status.
*******************************************************************************/
public class SecuredDataStatusSpecV1 extends AbstractJsonEndpointSpec
{
/*******************************************************************************
** Operation metadata.
*******************************************************************************/
@Override
public BasicOperation defineBasicOperation()
{
return (new BasicOperation().withPath("/qbit/secured-data/status").withHttpMethod(HttpMethod.GET)
.withTag(() -> "secured-data").withShortSummary("Status"));
}
/*******************************************************************************
** Request query parameters.
*******************************************************************************/
@Override
public List<Parameter> defineRequestParameters()
{
return (List.of());
}
/*******************************************************************************
** Published component schemas.
*******************************************************************************/
@Override
public Map<String, Schema> defineComponentSchemas()
{
Map<String, Schema> schemas = SecuredDataSchemas.all();
return (Map.of("SecuredDataStatusResponseV1", schemas.get("SecuredDataStatusResponseV1"), "SecuredDataBreakGlassV1", schemas.get("SecuredDataBreakGlassV1"), "SecuredDataRevealAllowanceV1", schemas.get("SecuredDataRevealAllowanceV1")));
}
/*******************************************************************************
** Response component.
*******************************************************************************/
@Override
protected String getResponseSchemaName()
{
return ("SecuredDataStatusResponseV1");
}
/*******************************************************************************
** Execute with the caller session supplied by the core middleware.
*******************************************************************************/
@Override
protected Object handle(JsonInput input) throws Exception
{
return (new StatusService().status());
}
}
@@ -0,0 +1,389 @@
components:
schemas:
BasicErrorResponseV1:
description: "The error body of every UI API operation (core and extensions)."
properties:
error:
description: "The error message (the same as message; kept for clients of\
\ UI API 1.0 to 1.5)."
type: "string"
code:
description: "A stable, machine-readable code. Well-known values: BAD_REQUEST\
\ (400), UNAUTHENTICATED (401),\nPERMISSION_DENIED (403), NOT_FOUND (404),\
\ CONFLICT (409), RATE_LIMITED (429), INTERNAL_ERROR (500),\nERROR (other).\
\ Extensions may add codes; treat an unknown code by the HTTP status.\
\ Since 1.6.0."
type: "string"
message:
description: "A message to show the user. Since 1.6.0."
type: "string"
fieldErrors:
description: "Errors about particular request fields, when known. Since\
\ 1.6.0."
items:
properties:
field:
description: "The field (or parameter) name."
type: "string"
message:
description: "The error about that field."
type: "string"
code:
description: "An optional machine-readable code, e.g. REQUIRED, TOO_LONG,\
\ PATTERN."
type: "string"
type: "object"
type: "array"
traceId:
description: "Identifies this request in the server logs (also the X-QQQ-Trace-Id\
\ response header); quote it when reporting a problem. Since 1.6.0."
type: "string"
type: "object"
SecuredDataBreakGlassV1:
properties:
id:
type: "integer"
reason:
type: "string"
expiresAt:
format: "date-time"
type: "string"
scope:
nullable: true
type: "string"
ceiling:
type: "string"
type: "object"
SecuredDataErrorV1:
description: "Deprecated since 1.1.0: errors use BasicErrorResponseV1."
properties:
error:
type: "string"
type: "object"
SecuredDataRevealAllowanceV1:
properties:
tableName:
type: "string"
fieldName:
type: "string"
remainingDailyAllowance:
nullable: true
type: "integer"
type: "object"
SecuredDataRevealOptionsResponseV1:
properties:
reasonOptions:
items:
type: "string"
type: "array"
allowFreeTextReason:
type: "boolean"
requireFreeText:
type: "boolean"
revealMinutes:
type: "integer"
remainingDailyAllowance:
nullable: true
type: "integer"
type: "object"
SecuredDataRevealRequestV1:
description: "tableName, primaryKey and fieldName are required. Supply a policy\
\ reason or allowed free text; maximum 500 characters."
properties:
tableName:
type: "string"
fieldName:
type: "string"
primaryKey:
anyOf:
- type: "string"
- type: "number"
description: "Record primary key (string or number)."
reason:
maxLength: 500
nullable: true
type: "string"
reasonText:
maxLength: 500
nullable: true
type: "string"
type: "object"
SecuredDataRevealResponseV1:
properties:
value:
description: "Authorized raw field value."
nullable: true
displayValue:
nullable: true
type: "string"
expiresAt:
format: "date-time"
type: "string"
revealId:
type: "integer"
type: "object"
SecuredDataStatusResponseV1:
properties:
breakGlass:
$ref: "#/components/schemas/SecuredDataBreakGlassV1"
nullable: true
revealsUsedToday:
type: "integer"
revealAllowances:
items:
$ref: "#/components/schemas/SecuredDataRevealAllowanceV1"
type: "array"
type: "object"
securitySchemes:
sessionUuidCookie:
description: "The session cookie set by POST /qqq/v1/manageSession (web clients)."
in: "cookie"
name: "sessionUUID"
type: "apiKey"
bearerAuth:
bearerFormat: "JWT or session UUID"
description: "`Authorization: Bearer <token>`, where the token is either\n-\
\ an OAuth2 access token (a signed JWT) issued by the app's identity provider\
\ to the app's client,\n or to its native client (`values.nativeClientId`\
\ in GET /qqq/v1/metaData/authentication), for\n example from the device-authorization\
\ grant or auth-code + PKCE. It is verified against the\n provider's keys\
\ (issuer, audience, expiry, signature) and gives the same user and permissions\n\
\ as signing in on the web. When it expires (401), refresh it at the provider\
\ and retry; or\n- a session UUID returned by POST /qqq/v1/manageSession,\
\ which the server keeps refreshed."
scheme: "bearer"
type: "http"
oauth2:
description: "The app's OAuth2 / OpenID Connect provider. The real endpoints\
\ come from the provider's discovery\ndocument: `{values.baseUrl}.well-known/openid-configuration`,\
\ with values from\nGET /qqq/v1/metaData/authentication. Native clients use\
\ `values.nativeClientId` (a public client)\nwith auth-code + PKCE (loopback\
\ or custom-scheme redirect URI registered at the provider) or the\ndevice-authorization\
\ grant (`device_authorization_endpoint`, RFC 8628), then send the access\
\ token\nas `Authorization: Bearer`."
flows:
authorizationCode:
authorizationUrl: "/authorize"
scopes:
email: "Email address"
offline_access: "A refresh token, to renew the access token without signing\
\ in again"
openid: "OpenID Connect sign-in"
profile: "Name, username and groups (permissions come from the groups)"
tokenUrl: "/token"
type: "oauth2"
info:
contact:
email: "contact@kingsrook.com"
description: "Reveal grants and break-glass status"
title: "QQQ Middleware API extension: secured-data"
version: "1.1.0"
openapi: "3.0.3"
paths:
/qqq/v1/qbit/secured-data/reveal:
post:
operationId: "postQbitSecuredDataReveal"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataRevealRequestV1"
required: true
responses:
400:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Invalid reveal request"
403:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Reveal denied"
404:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Record not found or locked"
200:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataRevealResponseV1"
description: "Successful Response"
429:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Daily reveal cap reached"
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "Reveal"
tags:
- "secured-data"
/qqq/v1/qbit/secured-data/reveal/options:
get:
operationId: "getQbitSecuredDataRevealOptions"
parameters:
- in: "query"
name: "tableName"
required: true
schema:
type: "string"
- in: "query"
name: "fieldName"
required: true
schema:
type: "string"
responses:
200:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataRevealOptionsResponseV1"
description: "Successful Response"
400:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Bad request: a parameter or the body is invalid (code BAD_REQUEST);\
\ fieldErrors may name the fields."
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
403:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "The user (or the credential's scope) may not do this (code\
\ PERMISSION_DENIED)."
404:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not found, or not visible to this user (code NOT_FOUND)."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
429:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Too many requests (code RATE_LIMITED); retry later."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "RevealOptions"
tags:
- "secured-data"
/qqq/v1/qbit/secured-data/status:
get:
operationId: "getQbitSecuredDataStatus"
responses:
200:
content:
application/json:
schema:
$ref: "#/components/schemas/SecuredDataStatusResponseV1"
description: "Successful Response"
400:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Bad request: a parameter or the body is invalid (code BAD_REQUEST);\
\ fieldErrors may name the fields."
401:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not authenticated, or the session or token is invalid or expired\
\ (code UNAUTHENTICATED). Sign in again, or refresh the token."
403:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "The user (or the credential's scope) may not do this (code\
\ PERMISSION_DENIED)."
404:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Not found, or not visible to this user (code NOT_FOUND)."
409:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Conflict with the current state, e.g. an edit conflict or\
\ an invalid transition (code CONFLICT)."
429:
content:
application/json:
schema:
$ref: "#/components/schemas/BasicErrorResponseV1"
description: "Too many requests (code RATE_LIMITED); retry later."
security:
- sessionUuidCookie:
- "N/A"
- bearerAuth:
- "N/A"
- oauth2:
- "openid"
- "profile"
- "email"
summary: "Status"
tags:
- "secured-data"
servers:
- description: "The QQQ server that serves this document (paths include /qqq/v1)."
url: "/"
tags:
- description: "Operations of the secured-data extension."
name: "secured-data"
@@ -0,0 +1,186 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.List;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policy;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Core field access integration tests.
*******************************************************************************/
class CoreFieldAccessIT
{
static final String EMAIL = "CANARY-EMAIL-918273@example.invalid";
static final String SSN = "CANARY-PHI-192837465";
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
close();
}
/*******************************************************************************
** End to end query get exact match and denial matrix.
*******************************************************************************/
@Test
void endToEndQueryGetExactMatchAndDenialMatrix() throws Exception
{
QInstance instance = setup();
for(String classification : List.of("PII", "PHI"))
{
policy("cs_rep", classification, "MASKED");
policy("cs_admin", classification, "REVEALABLE");
policy("admin", classification, "FULL");
}
QRecord person = insert("person", new QRecord().withValue("email", EMAIL).withValue("ssn", SSN).withValue("notes", "ordinary"));
for(String role : List.of("cs_rep", "cs_admin", "admin"))
{
QContext.setQSession(user(role, role));
QRecord query = new QueryAction().execute(new QueryInput("person")).getRecords().getFirst();
QRecord get = new GetAction().execute(new GetInput("person").withPrimaryKey(person.getValue("id"))).getRecord();
for(QRecord row : List.of(query, get))
{
assertThat(row.getValueString("notes")).isEqualTo("ordinary");
if(role.equals("admin"))
{
assertThat(row.getValueString("email")).isEqualTo(EMAIL);
assertThat(row.getValueString("ssn")).isEqualTo(SSN);
}
else
{
assertThat(row.getValueString("email")).isNotNull().isNotEqualTo(EMAIL);
assertThat(row.getValueString("ssn")).isNotNull().isNotEqualTo(SSN);
}
}
}
QContext.setQSession(user("cs_admin", "cs_admin"));
assertThat(search(EMAIL)).hasSize(1).allMatch(row -> !EMAIL.equals(row.getValueString("email")));
assertThat(search("NOT-IN-TABLE")).isEmpty();
assertThatThrownBy(() -> new QueryAction().execute(new QueryInput("person").withFilter(new QQueryFilter().withOrderBy(new QFilterOrderBy("email"))))).hasMessageContaining("Permission denied");
QContext.setQSession(user("cs_rep", "cs_rep"));
assertThatThrownBy(() -> search(EMAIL)).hasMessageContaining("Permission denied");
QContext.setQSession(user("admin", "admin"));
assertThat(search(EMAIL)).hasSize(1).allMatch(row -> EMAIL.equals(row.getValueString("email")));
SecuredDataRuntime.current().logger().flush();
assertThat(logs().stream().filter(row -> "EXACT_MATCH_FILTER".equals(row.getValueString("event_type"))).map(row -> row.getValueString("outcome"))).contains("MATCH", "NO_MATCH");
assertThat(logs().stream().filter(row -> "FILTER_DENIED".equals(row.getValueString("event_type")))).isNotEmpty();
assertThat(com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(logs())).doesNotContain(EMAIL, SSN, "NOT-IN-TABLE");
QContext.setQSession(channelUser("assistant/t1", "secured_data.admin"));
assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.ACCESS_LOG))).hasMessageContaining("Permission denied");
}
/*******************************************************************************
** Core preflight reports exactly one refused log and honors entropy.
*******************************************************************************/
@Test
void corePreflightReportsExactlyOneRefusedLogAndHonorsEntropy() throws Exception
{
setup();
policy("cs_admin", "PII", "REVEALABLE");
for(int i = 0; i < 20; i++)
{
assertThat(search("known" + i)).isEmpty();
}
assertThatThrownBy(() -> search("known21")).hasMessageContaining("Exact-match search limit reached for [email]");
SecuredDataRuntime.current().logger().flush();
assertThat(logs().stream().filter(row -> "EXACT_MATCH_FILTER".equals(row.getValueString("event_type")))) .hasSize(21);
assertThat(logs().stream().filter(row -> "REFUSED".equals(row.getValueString("outcome")))) .hasSize(1);
classification("email", "PII");
QRecord row = com.kingsrook.qbits.secureddata.support.SystemWrites.query(SecuredDataTables.CLASSIFICATION, new QQueryFilter()).getFirst();
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", row.getValue("id")).withValue("low_cardinality", true));
assertThatThrownBy(() -> search("known")).hasMessageContaining("Permission denied");
}
@Test
void revealUnmasksOnlyGrantedRecordAndExpiresInCoreGet() throws Exception
{
setup();
policy("cs_admin", "PII", "REVEALABLE");
QRecord row = insert("person", new QRecord().withValue("email", EMAIL));
QRecord other = insert("person", new QRecord().withValue("email", "OTHER-CANARY"));
GetInput get = new GetInput("person").withPrimaryKey(row.getValue("id"));
assertThat(new GetAction().execute(get).getRecord().getValueString("email")).isNotEqualTo(EMAIL);
var reveal = new com.kingsrook.qbits.secureddata.reveal.RevealService().reveal("person", row.getValue("id"), "email", "Investigating", null);
assertThat(reveal.value()).isEqualTo(EMAIL);
assertThat(new GetAction().execute(get).getRecord().getValueString("email")).isEqualTo(EMAIL);
assertThat(new GetAction().execute(new GetInput("person").withPrimaryKey(other.getValue("id"))).getRecord().getValueString("email")).isNotEqualTo("OTHER-CANARY");
SecuredDataTestSupport.CLOCK.now = SecuredDataTestSupport.NOW.plusSeconds(300);
assertThat(new GetAction().execute(get).getRecord().getValueString("email")).isNotEqualTo(EMAIL);
}
@Test
void revealPreservesCoreRecordLockDenial() throws Exception
{
QInstance instance = setup();
policy("cs_admin", "PII", "REVEALABLE");
QRecord row = insert("person", new QRecord().withValue("email", EMAIL).withValue("notes", "tenant-a"));
instance.addSecurityKeyType(new com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType().withName("tenant"));
instance.getTable("person").withRecordSecurityLock(new com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock().withSecurityKeyType("tenant").withFieldName("notes"));
QContext.getQSession().withSecurityKeyValue("tenant", "tenant-b");
assertThat(new GetAction().execute(new GetInput("person").withPrimaryKey(row.getValue("id"))).getRecord()).isNull();
assertThatThrownBy(() -> new com.kingsrook.qbits.secureddata.reveal.RevealService().reveal("person", row.getValue("id"), "email", "Investigating", null))
.isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QNotFoundException.class);
assertThat(com.kingsrook.qbits.secureddata.support.SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter())).isEmpty();
}
/*******************************************************************************
** Search.
*******************************************************************************/
private List<QRecord> search(String email) throws Exception
{
return (new QueryAction().execute(new QueryInput("person").withFilter(new QQueryFilter()
.withCriteria(new QFilterCriteria("email", QCriteriaOperator.EQUALS, email)))).getRecords());
}
}
@@ -0,0 +1,119 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.core.Appender;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.LoggerContext;
import org.apache.logging.log4j.core.appender.AbstractAppender;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.LoggerConfig;
import org.apache.logging.log4j.core.config.Property;
/*******************************************************************************
** Captures every log event, at every level, from every logger while open.
** Used by the canary test to prove no protected value reaches a log line.
*******************************************************************************/
final class LogCapture implements AutoCloseable
{
private final List<String> lines = new ArrayList<>();
private final Map<String, Level> originalLevels = new HashMap<>();
private final LoggerContext context;
private final Appender appender;
/*******************************************************************************
** Attach to the root logger and force all configured loggers to ALL.
*******************************************************************************/
LogCapture()
{
context = (LoggerContext) LogManager.getContext(false);
Configuration configuration = context.getConfiguration();
appender = new AbstractAppender("secured-data-log-capture", null, null, true, Property.EMPTY_ARRAY)
{
@Override
public void append(LogEvent event)
{
StringBuilder line = new StringBuilder(event.getLoggerName()).append(' ').append(event.getMessage().getFormattedMessage());
for(Throwable t = event.getThrown(); t != null; t = t.getCause())
{
line.append(' ').append(t.getClass().getName()).append(' ').append(t.getMessage());
}
synchronized(lines)
{
lines.add(line.toString());
}
}
};
appender.start();
configuration.addAppender(appender);
List<LoggerConfig> all = new ArrayList<>(configuration.getLoggers().values());
all.add(configuration.getRootLogger());
for(LoggerConfig config : all)
{
originalLevels.put(config.getName(), config.getLevel());
config.setLevel(Level.ALL);
}
configuration.getRootLogger().addAppender(appender, Level.ALL, null);
context.updateLoggers();
}
/*******************************************************************************
** Every captured line so far.
*******************************************************************************/
List<String> lines()
{
synchronized(lines)
{
return (new ArrayList<>(lines));
}
}
/*******************************************************************************
** Detach and restore levels.
*******************************************************************************/
@Override
public void close()
{
Configuration configuration = context.getConfiguration();
configuration.getRootLogger().removeAppender(appender.getName());
for(LoggerConfig config : configuration.getLoggers().values())
{
config.setLevel(originalLevels.get(config.getName()));
}
configuration.getRootLogger().setLevel(originalLevels.get(configuration.getRootLogger().getName()));
context.updateLoggers();
appender.stop();
}
}
@@ -0,0 +1,109 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Runs a streamed-ETL table or record action the way the UI does: the
** review screen, then the result.
*******************************************************************************/
public final class ProcessRunner
{
/*******************************************************************************
** Review and result lines, as shown.
*******************************************************************************/
public record Lines(List<String> review, List<String> result)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private ProcessRunner()
{
}
/*******************************************************************************
** Run on some record ids.
*******************************************************************************/
@SuppressWarnings("unchecked")
public static Lines run(String process, Serializable... ids) throws Exception
{
RunProcessInput input = new RunProcessInput();
input.setProcessName(process);
input.addValue("recordsParam", "recordIds");
input.addValue("recordIds", Arrays.stream(ids).map(String::valueOf).collect(Collectors.joining(",")));
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK);
RunProcessOutput review = new RunProcessAction().execute(input);
if(review.getException().isPresent())
{
throw (review.getException().get());
}
List<ProcessSummaryLineInterface> reviewLines = (List<ProcessSummaryLineInterface>) review.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY);
RunProcessInput go = new RunProcessInput();
go.setProcessName(process);
go.setProcessUUID(review.getProcessUUID());
go.setStartAfterStep(StreamedETLWithFrontendProcess.STEP_NAME_REVIEW);
go.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
RunProcessOutput result = new RunProcessAction().execute(go);
if(result.getException().isPresent())
{
throw (result.getException().get());
}
List<ProcessSummaryLineInterface> resultLines = (List<ProcessSummaryLineInterface>) result.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY);
return (new Lines(text(reviewLines, false), text(resultLines, true)));
}
/*******************************************************************************
** Lines as text.
*******************************************************************************/
private static List<String> text(List<ProcessSummaryLineInterface> lines, boolean past)
{
return (lines == null ? List.of() : lines.stream().map(line ->
{
if(line instanceof ProcessSummaryLine summary)
{
summary.pickMessage(past);
return ((summary.getCount() == null ? "" : summary.getCount() + " ") + summary.getMessage());
}
return (String.valueOf(line));
}).toList());
}
}
@@ -0,0 +1,388 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.ByteArrayOutputStream;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.Base64;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueKeySource.KeyAndId;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.reporting.ExportAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination;
import com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.rows;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.rule;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** SECURED-DATA section 15 proofs: a planted canary never leaves the host table,
** and shredded audit tokens cannot be brute-forced once the data key is gone.
*******************************************************************************/
class RetentionProofTest
{
private static final SecureRandom RANDOM = new SecureRandom();
private static final String OTHER_KEY = "/v79vLu6ubi3trW0s7KxsK+uraypqKempaSjoqGgn56dnJuamZiXlpU=";
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
RetentionTestSupport.close();
}
private static final String CANARY_PREFIX = "CANARY-PHI-";
private static final int CANARY_RANDOM = 24;
private static final int MIN_LEAK_FRAGMENT = 12;
/*******************************************************************************
** A fixed prefix and 24 random letters g-v (so it never coincides with hex
** digests). Leaks are matched on the whole marker or on 12-letter windows of
** its random part: 16^12 possibilities, so ordinary text, JSON words or
** base64 tokens cannot match by chance, while a truncated or re-encoded
** fragment of a real leak still does.
*******************************************************************************/
private static String canary()
{
StringBuilder marker = new StringBuilder(CANARY_PREFIX);
for(int i = 0; i < CANARY_RANDOM; i++)
{
marker.append((char) ('g' + RANDOM.nextInt(16)));
}
return (marker.toString());
}
/*******************************************************************************
** The whole marker, or any MIN_LEAK_FRAGMENT-letter window of its random part.
*******************************************************************************/
private static List<String> leaks(String haystack, String... markers)
{
List<String> found = new ArrayList<>();
for(String marker : markers)
{
if(haystack.contains(marker))
{
found.add("whole marker");
continue;
}
String random = marker.substring(CANARY_PREFIX.length());
for(int start = 0; start + MIN_LEAK_FRAGMENT <= random.length(); start++)
{
if(haystack.contains(random.substring(start, start + MIN_LEAK_FRAGMENT)))
{
found.add("fragment at " + start);
}
}
}
return (found);
}
/*******************************************************************************
** The leak scan: whole markers and 12-letter fragments are found; the shared
** prefix and ordinary words made of the same letters are not.
*******************************************************************************/
@Test
void leakScanHasNoFalsePositives()
{
String marker = canary();
String random = marker.substring(CANARY_PREFIX.length());
assertThat(leaks("x" + marker + "x", marker)).containsExactly("whole marker");
assertThat(leaks("{\"v\":\"" + random.substring(5, 5 + MIN_LEAK_FRAGMENT) + "\"}", marker)).isNotEmpty();
assertThat(leaks(CANARY_PREFIX + " options joins strings rows points", marker)).isEmpty();
assertThat(leaks(random.substring(0, MIN_LEAK_FRAGMENT - 1), marker)).isEmpty();
}
/*******************************************************************************
** Canary through CRUD, a failing validation, a wipe run, a masked export and
** the access log: it may appear only in the host table itself.
*******************************************************************************/
@Test
void canaryNeverLeavesTheHostTable() throws Exception
{
String first = canary();
String second = canary();
String rejected = canary();
try(LogCapture logs = new LogCapture())
{
QInstance instance = RetentionTestSupport.setup();
QLogger.getLogger(RetentionProofTest.class).info("log-capture-sentinel");
assertThat(logs.lines()).anyMatch(line -> line.contains("log-capture-sentinel"));
SecuredDataTestSupport.policy("viewer", "PHI", "MASKED");
SecuredDataTestSupport.policy("viewer", "PII", "MASKED");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = rule("NULL_WITH_MARKER", "secret,notes");
// CRUD with audits.
List<String> errors = new ArrayList<>();
QRecord inserted = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("secret", first).withValue("notes", first)
.withValue("systolic", 120).withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst();
assertThat(inserted.getErrors()).isNullOrEmpty();
Serializable id = inserted.getValue("id");
QRecord updated = new UpdateAction().execute(new UpdateInput("reading").withRecord(new QRecord().withValue("id", id).withValue("secret", second)
.withValue("notes", second))).getRecords().getFirst();
assertThat(updated.getErrors()).isNullOrEmpty();
// A failing validation: the secret value is over the field's maximum length.
QRecord invalid = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("secret", rejected + "x".repeat(300))
.withValue("notes", rejected).withValue("reading_time", RetentionTestSupport.OLD))).getRecords().getFirst();
assertThat(invalid.getErrors()).as("the invalid insert must be reported as an error").isNotEmpty();
invalid.getErrors().forEach(error -> errors.add(String.valueOf(error.getMessage())));
// A masked user reads and exports; the access log records the views.
QContext.setQSession(SecuredDataTestSupport.user("viewer", "viewer", "reading.read"));
assertThat(new QueryAction().execute(new QueryInput("reading")).getRecords()).hasSize(1);
ByteArrayOutputStream export = new ByteArrayOutputStream();
ExportInput exportInput = new ExportInput();
exportInput.setTableName("reading");
new ExportAction().execute(exportInput.withReportDestination(new ReportDestination().withReportFormat(ReportFormat.CSV).withReportOutputStream(export)));
String exported = export.toString(StandardCharsets.UTF_8);
assertThat(exported).isNotBlank();
SecuredDataRuntime.current().logger().flush();
// Positive control: the scan does detect the planted value where it legitimately lives.
assertThat(leaks(JsonUtils.toJson(rows("reading")), second)).isNotEmpty();
// The wipe run, as the retention administrator.
QContext.setQSession(SecuredDataTestSupport.user("admin", "secured_data.admin", "secured_data.retention", "reading.read", "reading.write"));
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(id));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(rows("reading").getFirst().getValue("secret")).isNull();
assertThat(leaks(JsonUtils.toJson(rows("reading")), first, second)).as("the wiped host table").isEmpty();
List<String> places = new ArrayList<>();
Map<String, String> haystacks = new LinkedHashMap<>();
haystacks.put("export", exported);
haystacks.put("errors", String.join(" ", errors));
haystacks.put("logs", String.join("\n", logs.lines()));
for(String table : List.of("audit", "auditDetail", SecuredDataTables.ACCESS_LOG, SecuredDataTables.WIPE_RUN, SecuredDataTables.WIPE_MARKER,
SecuredDataTables.AUDIT_KEY, SecuredDataTables.RETENTION_RULE, SecuredDataTables.LEGAL_HOLD, SecuredDataTables.POLICY, SecuredDataTables.CLASSIFICATION))
{
haystacks.put(table, JsonUtils.toJson(rows(table)));
assertThat(instance.getTable(table)).isNotNull();
}
haystacks.forEach((name, text) ->
{
if(!leaks(text, first, second, rejected).isEmpty())
{
places.add(name);
}
});
assertThat(places).as("places where the canary or a long fragment of it appeared").isEmpty();
assertThat(haystacks.get("audit")).isNotBlank();
assertThat(rows("auditDetail")).isNotEmpty();
assertThat(rows(SecuredDataTables.ACCESS_LOG)).isNotEmpty();
}
}
/*******************************************************************************
** HMAC-SHA256 over the agreed record binding, exactly as the token format does.
*******************************************************************************/
private static String digest(byte[] key, QTableMetaData table, String field, Serializable pk, int value) throws Exception
{
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(key, "HmacSHA256"));
byte[] out = mac.doFinal((String.join("\u001F", table.getName(), field, String.valueOf(pk)) + "\u001F" + value).getBytes(StandardCharsets.UTF_8));
return (Base64.getUrlEncoder().withoutPadding().encodeToString(out));
}
/*******************************************************************************
** After the wipe nothing can recover a systolic reading from its audit tokens,
** not even with every key still reachable, while before the wipe the same
** procedure does find the value (so the search genuinely detects a match).
*******************************************************************************/
@Test
void shreddedAuditTokensCannotBeBruteForcedAfterTheWipe() throws Exception
{
QInstance instance = RetentionTestSupport.setup();
System.setProperty("QQQ_AUDIT_HMAC_KEY", OTHER_KEY);
QTableMetaData table = instance.getTable("reading");
SecuredDataTestSupport.policy("secured_data.admin", "PHI", "FULL");
SecuredDataTestSupport.policy("secured_data.admin", "PII", "FULL");
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
List<QRecord> toInsert = new ArrayList<>();
for(int systolic = 60; systolic <= 250; systolic++)
{
toInsert.add(new QRecord().withValue("systolic", systolic).withValue("secret", "s").withValue("reading_time", RetentionTestSupport.OLD));
}
List<QRecord> inserted = new InsertAction().execute(new InsertInput("reading").withRecords(toInsert)).getRecords();
assertThat(inserted).hasSize(191).allSatisfy(r -> assertThat(r.getErrors()).isNullOrEmpty());
Map<Serializable, Integer> truth = new LinkedHashMap<>();
inserted.forEach(r -> truth.put(r.getValue("id"), r.getValueInteger("systolic")));
// A control record the wipe will not touch: its data key stays reachable.
QRecord control = new InsertAction().execute(new InsertInput("reading").withRecord(new QRecord().withValue("systolic", 111)
.withValue("reading_time", SecuredDataTestSupport.NOW))).getRecords().getFirst();
Map<Serializable, List<String>> tokens = auditTokens(truth.keySet());
assertThat(tokens).hasSize(191).allSatisfy((id, list) -> assertThat(list).isNotEmpty().allMatch(t -> t.startsWith("shred:v1:")));
// Before the wipe, the DEK verifies the true value and no other value.
var keys = SecuredDataRuntime.current().keys();
for(var entry : truth.entrySet())
{
Optional<KeyAndId> dek = keys.keyForRecordField("reading", "systolic", entry.getKey(), false);
assertThat(dek).isPresent();
for(String token : tokens.get(entry.getKey()))
{
assertThat(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), entry.getValue(), token)).isTrue();
assertThat(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), entry.getValue() + 1, token)).isFalse();
assertThat(token).endsWith(digest(dek.get().key(), table, "systolic", entry.getKey(), entry.getValue()));
}
}
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(192);
// Wipe the old readings in policy-sized batches.
List<Serializable> ids = new ArrayList<>(truth.keySet());
for(int offset = 0; offset < ids.size(); offset += 100)
{
new WipeService().batch(rule.getValue("id"), ids.subList(offset, Math.min(offset + 100, ids.size())));
}
assertThat(rows("reading").stream().filter(r -> truth.containsKey(r.getValue("id")))).allSatisfy(r -> assertThat(r.getValue("systolic")).isNull());
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(1);
assertThat(rows(SecuredDataTables.AUDIT_KEY).getFirst().getValueString("record_id")).isEqualTo(String.valueOf(control.getValue("id")));
// Every key the test can still reach.
List<byte[]> reachable = new ArrayList<>();
reachable.add(Base64.getDecoder().decode(OTHER_KEY));
reachable.add(Base64.getDecoder().decode(RetentionTestSupport.KEY));
reachable.add(keys.keyForRecordField("reading", "systolic", control.getValue("id"), false).orElseThrow().key());
reachable.add(new byte[32]);
assertThat(reachable.get(2)).isNotEqualTo(new byte[32]);
Map<Serializable, List<String>> after = auditTokens(truth.keySet());
int matches = 0;
for(var entry : truth.entrySet())
{
assertThat(after.get(entry.getKey())).containsAll(tokens.get(entry.getKey()));
for(String token : after.get(entry.getKey()))
{
String tail = token.substring(token.lastIndexOf(':') + 1);
for(int guess = 60; guess <= 250; guess++)
{
if(ClassifiedValueTokens.verifyRecordField(table, table.getField("systolic"), entry.getKey(), guess, token))
{
matches++;
}
for(byte[] key : reachable)
{
if(digest(key, table, "systolic", entry.getKey(), guess).equals(tail))
{
matches++;
}
}
}
}
}
assertThat(matches).as("brute-force matches after the wipe").isZero();
assertThat(rows("auditDetail")).anySatisfy(row -> assertThat(row.getValueString("newValue")).startsWith("shredded:v1:"));
}
/*******************************************************************************
** Every systolic audit-detail value for the given records, by record id.
*******************************************************************************/
private static Map<Serializable, List<String>> auditTokens(Iterable<Serializable> ids)
{
Map<Serializable, String> recordByAudit = new LinkedHashMap<>();
for(QRecord audit : SystemWrites.query("audit", new QQueryFilter()))
{
recordByAudit.put(audit.getValue("id"), audit.getValueString("recordId"));
}
Map<Serializable, List<String>> result = new LinkedHashMap<>();
for(Serializable id : ids)
{
result.put(id, new ArrayList<>());
}
for(QRecord detail : SystemWrites.query("auditDetail", new QQueryFilter()))
{
if(!"systolic".equals(detail.getValueString("fieldName")))
{
continue;
}
String record = recordByAudit.get(detail.getValue("auditId"));
for(Serializable id : ids)
{
if(String.valueOf(id).equals(record))
{
for(String value : new String[] {detail.getValueString("oldValue"), detail.getValueString("newValue")})
{
if(value != null && !value.isBlank())
{
result.get(id).add(value);
}
}
}
}
}
return (result);
}
}
@@ -0,0 +1,147 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.time.Instant;
import java.util.List;
import java.util.UUID;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
final class RetentionTestSupport
{
static String BACKEND = "retention_rdbms";
static final String KEY = "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=";
static final Instant OLD = SecuredDataTestSupport.NOW.minusSeconds(60L * 86400);
static QInstance setup() throws Exception
{
SecuredDataTestSupport.close();
System.setProperty("QQQ_SECURED_DATA_KEK", KEY);
System.setProperty("QQQ_AUDIT_HMAC_KEY", KEY);
SecuredDataTestSupport.CLOCK.now = SecuredDataTestSupport.NOW;
String db = "retention_" + UUID.randomUUID().toString().replace("-", "");
BACKEND = db; // pools are cached by backend name, so each instance needs its own
String url = "jdbc:h2:mem:" + db + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1";
RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName(BACKEND).withVendor("h2").withHostName("mem").withDatabaseName(db).withUsername("sa");
backend.setJdbcUrl(url);
QInstance instance = SecuredDataTestSupport.base();
instance.addBackend(backend);
QTableMetaData table = new QTableMetaData().withName("reading").withLabel("Reading").withBackendName(BACKEND).withPrimaryKeyField("id")
.withBackendDetails(new RDBMSTableBackendDetails().withTableName("reading"))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS))
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("reading_time", QFieldType.DATE_TIME))
.withField(new QFieldMetaData("reading_date", QFieldType.DATE))
.withField(new QFieldMetaData("systolic", QFieldType.INTEGER).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)))
.withField(new QFieldMetaData("secret", QFieldType.STRING).withMaxLength(255).withBehavior(com.kingsrook.qqq.backend.core.model.metadata.fields.ValueTooLongBehavior.ERROR).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)))
.withField(new QFieldMetaData("notes", QFieldType.TEXT).withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)));
table.getFields().values().forEach(f -> f.setBackendName(f.getName()));
instance.addTable(table);
new AuditsMetaDataProvider().defineAll(instance, BACKEND, t -> t.withBackendDetails(new RDBMSTableBackendDetails().withTableName(t.getName())));
try(Connection connection = DriverManager.getConnection(url, "sa", ""))
{
for(QTableMetaData t : instance.getTables().values())
{
StringBuilder sql = new StringBuilder("CREATE TABLE ").append(t.getName()).append(" (");
for(QFieldMetaData field : t.getFields().values())
{
field.setBackendName(field.getName());
sql.append(field.getName()).append(" ");
if(field.getName().equals(t.getPrimaryKeyField()))
{
sql.append("INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY");
}
else
{
sql.append(switch(field.getType())
{
case INTEGER -> "INTEGER";
case LONG -> "BIGINT";
case DATE_TIME -> "TIMESTAMP WITH TIME ZONE";
case DATE -> "DATE";
case BOOLEAN -> "BOOLEAN";
case DECIMAL -> "DECIMAL";
default -> "TEXT";
});
}
sql.append(",");
}
sql.setLength(sql.length() - 1);
connection.createStatement().execute(sql.append(")").toString());
}
for(String sql : Files.readString(Path.of("docs/ddl/secured-data-postgres.sql")).split(";"))
{
if(!sql.isBlank()) connection.createStatement().execute(sql);
}
}
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withBackendName(BACKEND).withClock(SecuredDataTestSupport.CLOCK)
.withBackupRetentionDays(35).withAccessLogFlushIntervalMillis(60000)
.withTableMetaDataCustomizer((i,t) -> t.withBackendDetails(new RDBMSTableBackendDetails().withTableName(t.getName())))).produce(instance);
QContext.init(instance, SecuredDataTestSupport.user("admin", "secured_data.admin", "secured_data.retention", "reading.read", "reading.write"));
SecuredDataTestSupport.RUNTIMES.add(SecuredDataRuntime.current());
new QInstanceValidator().validate(instance);
return instance;
}
static QRecord rule(String action, String fields) throws Exception
{
return SystemWrites.insertOne(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("name", "30-day policy").withValue("table_name", "reading")
.withValue("field_names", fields).withValue("age_field_name", "reading_time").withValue("age_days", 30).withValue("action", action)
.withValue("archive_storage_name", "archive"));
}
static QRecord reading(Object value) throws Exception
{
return SystemWrites.insertOne("reading", new QRecord().withValue("systolic", (java.io.Serializable)value).withValue("secret", "test-secret")
.withValue("notes", "test-notes").withValue("reading_time", OLD).withValue("reading_date", OLD.atZone(java.time.ZoneOffset.UTC).toLocalDate()));
}
static List<QRecord> rows(String table)
{
return SystemWrites.query(table, new QQueryFilter());
}
static void close()
{
SecuredDataTestSupport.close();
System.clearProperty("QQQ_SECURED_DATA_KEK");
System.clearProperty("QQQ_AUDIT_HMAC_KEY");
System.clearProperty("QQQ_AUDIT_HMAC_KEY_ID");
}
}
@@ -0,0 +1,265 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.retention.RetentionRules;
import com.kingsrook.qbits.secureddata.retention.RetentionWidgets;
import com.kingsrook.qbits.secureddata.retention.WipeProcesses;
import com.kingsrook.qbits.secureddata.retention.WipeService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.audits.DMLAuditAction.DMLType;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.ClassifiedValueTokens;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static com.kingsrook.qbits.secureddata.RetentionTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class RetentionWipeTest
{
@AfterEach void cleanup() { close(); }
@ParameterizedTest
@ValueSource(strings={"NULL_WITH_MARKER", "OVERWRITE_TEXT", "KEYED_HASH", "DELETE_RECORD", "ARCHIVE_THEN_WIPE"})
void eachActionIsTransactionalAndValueFree(String action) throws Exception
{
var instance = setup();
List<String> archiveJson = new ArrayList<>();
SecuredDataRuntime.current().config().withArchiveSink((store, key, bytes) -> {
assertThat(store).isEqualTo("archive");
assertThat(new String(bytes, StandardCharsets.UTF_8)).doesNotContain("test-secret");
archiveJson.add(new String(SecuredDataKeySource.decrypt(bytes, key), StandardCharsets.UTF_8));
});
QRecord rule = rule(action, "secret");
QRecord record = reading(120);
var keys = SecuredDataRuntime.current().keys();
keys.keyForRecordField("reading", "secret", record.getValue("id"), true);
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isNotEmpty();
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")));
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isEmpty();
assertThat(keys.keyForRecordField("reading", "secret", record.getValue("id"), false)).isEmpty();
if(action.equals("DELETE_RECORD")) assertThat(rows("reading")).isEmpty();
else {
QRecord remaining = rows("reading").getFirst();
if(action.equals("OVERWRITE_TEXT")) assertThat(remaining.getValue("secret")).isEqualTo("[wiped per policy]");
else if(action.equals("KEYED_HASH")) {
assertThat(remaining.getValueString("secret")).startsWith("hmac:v1:");
assertThat(ClassifiedValueTokens.verifyRecordField(instance.getTable("reading"), instance.getTable("reading").getField("secret"), record.getValue("id"), "test-secret", remaining.getValueString("secret"))).isTrue();
}
else assertThat(remaining.getValue("secret")).isNull();
}
if(action.equals("ARCHIVE_THEN_WIPE")) assertThat(archiveJson).containsExactly("{\"secret\":\"test-secret\"}");
else assertThat(archiveJson).isEmpty();
assertThat(rows(SecuredDataTables.WIPE_MARKER)).isNotEmpty();
assertThat(rows(SecuredDataTables.ACCESS_LOG).stream().filter(r -> "WIPE".equals(r.getValueString("event_type"))).toList()).isNotEmpty()
.allSatisfy(r -> { assertThat(r.getValueString("record_id")).isEqualTo(record.getValueString("id")); assertThat(r.getValueInteger("record_count")).isEqualTo(1); });
assertThat(rows("auditDetail")).anySatisfy(r -> assertThat(r.getValueString("newValue")).isEqualTo("shredded:v1:" + rule.getValueString("id")));
}
@Test void recordAndTableLegalHoldsAndAgeAreRechecked() throws Exception
{
setup();
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
QRecord one = reading(120), two = reading(130);
QRecord hold = SystemWrites.insertOne(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", one.getValueString("id")).withValue("reason", "Preservation"));
QRecord run = new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"), two.getValue("id")));
assertThat(run.getValueInteger("held_count")).isEqualTo(1);
assertThat(run.getValueInteger("wiped_count")).isEqualTo(1);
SecuredDataTestSupport.update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("record_id", null));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"))).getValueInteger("held_count")).isEqualTo(1);
SecuredDataTestSupport.update(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("id", hold.getValue("id")).withValue("status", "RELEASED"));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(one.getValue("id"))).getValueInteger("wiped_count")).isEqualTo(1);
QRecord young = reading(140);
SecuredDataTestSupport.update("reading", new QRecord().withValue("id", young.getValue("id")).withValue("reading_time", SecuredDataTestSupport.NOW));
assertThat(new WipeService().batch(rule.getValue("id"), List.of(young.getValue("id"))).getValueInteger("eligible_count")).isZero();
}
@Test void failureAfterFirstRecordRollsBackUpdatesKeysMarkersAuditsAndRun() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret");
QRecord first = reading(120), second = reading(130);
for(QRecord row : List.of(first, second)) SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", row.getValue("id"), true);
int audits = rows("auditDetail").size();
FailSecond.id = second.getValue("id");
instance.getTable("reading").withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(FailSecond.class));
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(first.getValue("id"), second.getValue("id")))).hasMessageContaining("rolled back");
assertThat(rows("reading")).allSatisfy(r -> assertThat(r.getValue("secret")).isEqualTo("test-secret"));
assertThat(rows(SecuredDataTables.AUDIT_KEY)).hasSize(2);
assertThat(rows(SecuredDataTables.WIPE_MARKER)).isEmpty();
assertThat(rows(SecuredDataTables.WIPE_RUN)).isEmpty();
assertThat(rows("auditDetail")).hasSize(audits);
}
public static class FailSecond implements TableCustomizerInterface
{
static java.io.Serializable id;
@Override public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> old) throws QException
{
if(records.stream().anyMatch(r -> id.equals(r.getValue("id")))) throw new QException("Injected failure");
return records;
}
}
@Test void reviewResultAndDailySchedule() throws Exception
{
setup();
QRecord rule = rule("NULL_WITH_MARKER", "systolic");
reading(120);
QRecord held = reading(130);
SystemWrites.insertOne(SecuredDataTables.LEGAL_HOLD, new QRecord().withValue("table_name", "reading").withValue("record_id", held.getValueString("id")).withValue("reason", "Preserve"));
var lines = ProcessRunner.run(WipeProcesses.WIPE, rule.getValue("id"));
assertThat(lines.review()).anyMatch(s -> s.contains("to wipe")).anyMatch(s -> s.contains("held"));
assertThat(lines.result()).anyMatch(s -> s.contains("wiped")).anyMatch(s -> s.contains("held"));
reading(150);
assertThatThrownBy(() -> new WipeProcesses.Daily().run(new RunBackendStepInput(), new RunBackendStepOutput())).hasMessageContaining("system session");
QContext.setQSession(new QSystemUserSession(QContext.getQInstance()));
var output = new RunBackendStepOutput();
new WipeProcesses.Daily().run(new RunBackendStepInput(), output);
assertThat(output.getRecords()).anySatisfy(r -> assertThat(r.getValueInteger("wiped_count")).isEqualTo(1));
var scheduled = WipeProcesses.processes(new SecuredDataQBitConfig().withSchedulerName("test")).getLast();
assertThat(scheduled.getSchedule().getRepeatSeconds()).isEqualTo(86400);
}
@Test void permissionsKeyLockAndBackupsPanels() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret"), record = reading(120);
SecuredDataRuntime.current().keys().keyForRecordField("reading", "secret", record.getValue("id"), true);
var keyTable = instance.getTable(SecuredDataTables.AUDIT_KEY);
assertThat(keyTable.getField("wrapped_key").getFieldSecurityLock().getOverrideValues()).containsExactly(com.kingsrook.qbits.secureddata.metadata.RetentionTables.KEY_LOCK_NEVER_GRANTED);
assertThat(keyTable.getDisabledCapabilities()).contains(com.kingsrook.qqq.backend.core.model.metadata.tables.Capability.TABLE_EXPORT);
assertThat(SecuredDataTables.MAIN).doesNotContain(SecuredDataTables.AUDIT_KEY);
assertThat(new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)).getRecords()).allSatisfy(r -> assertThat(r.getValue("wrapped_key")).isNull());
assertThat(RetentionRules.backupWarning(rule)).contains("35 days", "30 days", "recoverable");
assertThat(com.kingsrook.qbits.secureddata.app.SecuredDataHomeView.render(com.kingsrook.qbits.secureddata.app.SecuredDataHomeData.load())).contains("Backups keep data for 35 days");
RenderWidgetInput input = new RenderWidgetInput().withWidgetMetaData(RetentionWidgets.backups());
input.getQueryParams().put("id", rule.getValueString("id"));
assertThat(new RetentionWidgets.Backups().render(input)).isNotNull();
new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")));
var widget = RetentionWidgets.wipedFields(instance.getTable("reading"));
input = new RenderWidgetInput().withWidgetMetaData(widget); input.getQueryParams().put("id", record.getValueString("id"));
assertThat(new RetentionWidgets.WipedFields().render(input)).isNotNull();
QContext.setQSession(SecuredDataTestSupport.user("ordinary"));
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")))).hasMessageContaining("secured_data.retention");
}
@Test void changedOnlyAndWrapUnwrap() throws Exception
{
var instance = setup();
QRecord rule = rule("NULL_WITH_MARKER", "secret");
SecuredDataTestSupport.update(SecuredDataTables.RETENTION_RULE, new QRecord().withValue("id", rule.getValue("id")).withValue("audit_treatment", "CHANGED_ONLY"));
var table = instance.getTable("reading");
var protectedValues = new SecuredDataAuditProtector().protect(table, table.getField("secret"), 1, "old", "new", DMLType.UPDATE);
assertThat(protectedValues.oldToken()).isEqualTo("changed:v1"); assertThat(protectedValues.newToken()).isEqualTo("changed:v1");
assertThat(rows(SecuredDataTables.AUDIT_KEY)).isEmpty();
byte[] plain = Base64.getDecoder().decode(KEY);
byte[] encrypted = SecuredDataKeySource.encrypt(plain, "binding");
assertThat(SecuredDataKeySource.decrypt(encrypted, "binding")).isEqualTo(plain);
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(encrypted, "wrong-binding")).isInstanceOf(IllegalStateException.class);
encrypted[15] ^= 1;
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(encrypted, "binding")).isInstanceOf(IllegalStateException.class);
System.setProperty("QQQ_SECURED_DATA_KEK", "bad");
assertThatThrownBy(() -> SecuredDataKeySource.encrypt(plain, "binding")).hasMessageContaining("QQQ_SECURED_DATA_KEK");
}
@Test void validatesPolicyAndRefusesMissingArchive() throws Exception
{
setup();
assertThatThrownBy(() -> rule("KEYED_HASH", "systolic")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> rule("NULL_WITH_MARKER", "missing")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> rule("NULL_WITH_MARKER", "secret,secret")).hasMessageContaining("insert failed");
var rule = rule("ARCHIVE_THEN_WIPE", "secret"); var record = reading(120);
assertThatThrownBy(() -> new WipeService().batch(rule.getValue("id"), List.of(record.getValue("id")))).hasMessageContaining("ArchiveSinkInterface");
assertThat(rows("reading").getFirst().getValue("secret")).isEqualTo("test-secret");
assertThatThrownBy(() -> new SecuredDataQBitConfig().withBackupRetentionDays(-1)).isInstanceOf(IllegalArgumentException.class);
}
@Test void wrongKekCannotUnwrapAndKeyReturnedIsNeverZeroed() throws Exception
{
setup();
rule("NULL_WITH_MARKER", "secret");
QRecord record = reading(120);
var keys = SecuredDataRuntime.current().keys();
Object pk = record.getValue("id");
String binding = "readingsecret" + pk;
// Regression: the source once returned the array it then zeroed in finally.
byte[] created = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, true).orElseThrow().key();
assertThat(created).hasSize(32).isNotEqualTo(new byte[32]);
QRecord stored = SystemWrites.call(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.AUDIT_KEY)
.withFilter(SecuredDataKeySource.recordField("reading", "secret", (java.io.Serializable) pk)).withShouldOmitHiddenFields(false)).getRecords()).getFirst();
byte[] wrapped = Base64.getDecoder().decode(stored.getValueString("wrapped_key"));
assertThat(created).isEqualTo(SecuredDataKeySource.decrypt(wrapped, binding));
// Lookups (uncached then cached) agree with the unwrapped DEK, and callers zeroing their copy cannot poison the cache.
byte[] looked = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, false).orElseThrow().key();
assertThat(looked).isNotEqualTo(new byte[32]).isEqualTo(created);
java.util.Arrays.fill(looked, (byte) 0);
byte[] cached = keys.keyForRecordField("reading", "secret", (java.io.Serializable) pk, false).orElseThrow().key();
assertThat(cached).isEqualTo(created).isNotEqualTo(new byte[32]);
// A different KEK, or a different binding, cannot unwrap.
System.setProperty("QQQ_SECURED_DATA_KEK", "/v79vLu6ubi3trW0s7KxsK+uraypqKempaSjoqGgn56dnJuamZiXlpU=");
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(wrapped, binding)).isInstanceOf(IllegalStateException.class);
System.setProperty("QQQ_SECURED_DATA_KEK", KEY);
assertThatThrownBy(() -> SecuredDataKeySource.decrypt(wrapped, binding + "x")).isInstanceOf(IllegalStateException.class);
assertThat(SecuredDataKeySource.decrypt(wrapped, binding)).isEqualTo(created);
}
@Test void keyedHashIsRefusedOnLowEntropyFields() throws Exception
{
setup();
assertThat(rule("KEYED_HASH", "secret").getValue("id")).isNotNull();
SystemWrites.insertOne(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "reading").withValue("field_name", "secret")
.withValue("classification", "PHI").withValue("low_cardinality", true));
assertThatThrownBy(() -> rule("KEYED_HASH", "secret")).hasMessageContaining("insert failed");
assertThatThrownBy(() -> RetentionRules.validate(new QRecord().withValue("table_name", "reading").withValue("field_names", "secret")
.withValue("age_field_name", "reading_time").withValue("age_days", 30).withValue("batch_size", 10).withValue("action", "KEYED_HASH")
.withValue("audit_treatment", "SHREDDABLE"))).hasMessageContaining("low-entropy");
assertThat(rule("OVERWRITE_TEXT", "secret").getValue("id")).isNotNull();
}
}
@@ -0,0 +1,319 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.FieldsOfTablePossibleValueSource;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.processes.BreakGlassProcesses;
import com.kingsrook.qbits.secureddata.reveal.RevealCappedException;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class RevealAndBreakGlassTest
{
RevealService reveals = new RevealService();
BreakGlassService breaks = new BreakGlassService();
Serializable pk;
QRecord policy;
@BeforeEach
void init() throws Exception
{
var instance = setup();
policy = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE")
.withValue("reason_options", "Support\nInvestigation").withValue("daily_reveal_cap", 2).withValue("allow_free_text_reason", false));
pk = insert("person", new QRecord().withValue("email", "secret@example.invalid").withValue("ssn", "123456789")).getValue("id");
Sink.signals.clear(); Sink.resolutions.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(Sink.class));
}
@AfterEach
void cleanup() { close(); }
@Test
void revealValueAuditLinkCacheAndExactExpiry() throws Exception
{
var resolver = SecuredDataRuntime.current().resolver();
assertThat(resolver.isRevealActive("person", "email", pk)).isFalse();
var result = reveals.reveal("person", pk, "email", "Support", null);
assertThat(result.value()).isEqualTo("secret@example.invalid");
assertThat(result.displayValue()).isEqualTo("secret@example.invalid");
assertThat(result.expiresAt()).isEqualTo(NOW.plusSeconds(300));
assertThat(resolver.isRevealActive("person", "email", pk)).isTrue();
assertThat(resolver.isRevealActive("person", "ssn", pk)).isFalse();
assertThat(resolver.isRevealActive("person", "email", 999)).isFalse();
QRecord grant = SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter()).getFirst();
QRecord log = logs().stream().filter(r -> "REVEAL".equals(r.getValueString("event_type"))).findFirst().orElseThrow();
assertThat(grant.getValue("access_log_id")).isEqualTo(log.getValue("id"));
assertThat(log.getValueString("reason")).isEqualTo("Support");
QContext.setQSession(user("u2", "cs_admin"));
assertThat(resolver.isRevealActive("person", "email", pk)).isFalse();
QContext.setQSession(user("u1", "cs_admin"));
assertThat(resolver.isRevealActive("person", "email", pk)).isTrue();
CLOCK.now = NOW.plusSeconds(300);
assertThat(resolver.isRevealActive("person", "email", pk)).isFalse();
}
@Test
void optionsSelectOneMostPermissiveGrantingPolicy() throws Exception
{
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("daily_reveal_cap", 10)
.withValue("reveal_minutes", 15).withValue("reason_options", "Other policy").withValue("require_free_text", true));
var options = reveals.options("person", "email");
assertThat(options.remainingDailyAllowance()).isEqualTo(10);
assertThat(options.revealMinutes()).isEqualTo(15);
assertThat(options.reasonOptions()).containsExactly("Other policy");
assertThat(options.requireFreeText()).isTrue();
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("client_channel", "web").withValue("reveal_minutes", 60));
QContext.setQSession(channelUser("web", "cs_admin"));
assertThat(reveals.options("person", "email").revealMinutes()).isEqualTo(15);
}
@Test
void wrongTierAndReadPermissionAreRefused() throws Exception
{
assertThatThrownBy(() -> reveals.reveal("person", pk, "ssn", "Support", null)).isInstanceOf(QPermissionDeniedException.class);
QContext.setQSession(new QSession().withUser(user("u1").getUser()).withPermissions(List.of("cs_admin")));
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Support", null)).isInstanceOf(QPermissionDeniedException.class);
assertThat(SystemWrites.query(SecuredDataTables.REVEAL_GRANT, new QQueryFilter())).isEmpty();
}
@Test
void missingRecordAndFieldAreNotFound() throws Exception
{
assertThatThrownBy(() -> reveals.reveal("person", 999, "email", "Support", null)).isInstanceOf(QNotFoundException.class);
assertThatThrownBy(() -> reveals.options("person", "bogus")).isInstanceOf(QNotFoundException.class);
assertThatThrownBy(() -> reveals.reveal("person", null, "email", "Support", null)).isInstanceOf(QBadRequestException.class);
}
@Test
void invalidReasonsAndLengthAreRejected() throws Exception
{
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Bogus", null)).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Support", "x".repeat(501))).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "x".repeat(501), null)).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", null, null)).isInstanceOf(QBadRequestException.class);
}
@Test
void freeTextRequiredAndFiveHundredCharactersAccepted() throws Exception
{
update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("allow_free_text_reason", true).withValue("require_free_text", true));
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Support", null)).isInstanceOf(QBadRequestException.class);
assertThat(reveals.reveal("person", pk, "email", null, "x".repeat(500)).value()).isEqualTo("secret@example.invalid");
}
@Test
void dailyCapAcrossTablesAlertsAndResetsAtUtcMidnight() throws Exception
{
reveals.reveal("person", pk, "email", "Support", null);
QContext.getQInstance().addTable(person("memory").withName("customer"));
QContext.setQSession(user("u1", "cs_admin", "customer.read"));
Serializable other = insert("customer", new QRecord().withValue("email", "second@example.invalid")).getValue("id");
reveals.reveal("customer", other, "email", "Support", null);
assertThat(reveals.options("person", "email").remainingDailyAllowance()).isZero();
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Support", null)).isInstanceOf(RevealCappedException.class);
assertThat(logs()).anyMatch(r -> "CAPPED".equals(r.getValueString("outcome")) && "REVEAL_DENIED".equals(r.getValueString("event_type")));
assertThat(Sink.signals).anyMatch(s -> "secured_data.reveal_capped".equals(s.getAlertTypeName()));
CLOCK.now = NOW.plusSeconds(86400);
assertThat(reveals.options("person", "email").remainingDailyAllowance()).isEqualTo(2);
}
@Test
void assistantRefusedAndLogged() throws Exception
{
QContext.setQSession(channelUser("assistant/thread", "cs_admin"));
assertThatThrownBy(() -> reveals.reveal("person", pk, "email", "Support", null)).isInstanceOf(QPermissionDeniedException.class);
assertThat(logs()).anyMatch(r -> "REVEAL_DENIED".equals(r.getValueString("event_type")) && "assistant".equals(r.getValueString("client_channel")));
}
void enableStart() { QContext.setQSession(user("u1", "cs_admin", BreakGlassService.START + ".hasAccess")); }
@Test
void breakGlassStartsImmediatelyAlertsAndReportsStatus() throws Exception
{
enableStart();
String fingerprint = SecuredDataRuntime.current().resolver().getAccessFingerprint();
QRecord row = breaks.start("Investigating urgent incident", null, null, null);
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.FULL);
assertThat(SecuredDataRuntime.current().resolver().getAccessFingerprint()).isNotEqualTo(fingerprint);
assertThat(Sink.signals).filteredOn(s -> "secured_data.break_glass".equals(s.getAlertTypeName())).singleElement().satisfies(s -> { assertThat(s.getSeverity()).isEqualTo(AlertSeverity.ERROR); assertThat(s.getAlertTypeName()).isEqualTo("secured_data.break_glass"); });
var status = new StatusService().status();
assertThat(status.breakGlass().id()).isEqualTo(row.getValue("id"));
assertThat(status.breakGlass().expiresAt()).isEqualTo(NOW.plusSeconds(3600));
assertThat(status.breakGlass().ceiling()).isEqualTo("PHI");
assertThatThrownBy(() -> breaks.start("Another urgent investigation", 60, null, null)).isInstanceOf(QBadRequestException.class);
}
FieldAccessTier tier(String field) { var table = QContext.getQInstance().getTable("person"); return SecuredDataRuntime.current().resolver().resolve(table, table.getField(field)).tier(); }
@Test
void ceilingScopeAndReadTimeExpiry() throws Exception
{
enableStart();
breaks.start("Investigating urgent incident", 1, "person", DataClassification.PII);
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.MASKED);
CLOCK.now = NOW.plusSeconds(60);
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
assertThat(new StatusService().status().breakGlass()).isNull();
breaks.start("Investigating another incident", 1, SecuredDataTables.POLICY, null);
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
}
@Test
void ownerAndAdminEndButOthersCannot() throws Exception
{
enableStart(); QRecord row = breaks.start("Investigating urgent incident", 60, null, null);
Serializable firstId = row.getValue("id");
QContext.setQSession(user("u2"));
assertThatThrownBy(() -> breaks.end(firstId)).isInstanceOf(QPermissionDeniedException.class);
QContext.setQSession(user("u2", "secured_data.admin"));
breaks.end(row.getValue("id"));
assertThat(breaks.find(row.getValue("id")).getValueString("ended_by")).isEqualTo("u2");
assertThat(Sink.resolutions).hasSize(1);
enableStart(); row = breaks.start("Investigating urgent incident", 60, null, null);
breaks.end(row.getValue("id"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
assertThat(logs()).anyMatch(r -> "BREAK_GLASS_END".equals(r.getValueString("event_type")));
}
@Test
void expiryJobIsIdempotentAndResolvesAlert() throws Exception
{
enableStart(); QRecord row = breaks.start("Investigating urgent incident", 1, null, null);
CLOCK.now = NOW.plusSeconds(60);
var output = new RunBackendStepOutput();
new BreakGlassProcesses.Expire().run(new RunBackendStepInput(), output);
assertThat(breaks.find(row.getValue("id")).getValueString("status")).isEqualTo("EXPIRED");
assertThat(breaks.expire()).isZero();
assertThat(Sink.resolutions).hasSize(1);
assertThat(logs()).anyMatch(r -> "BREAK_GLASS_EXPIRED".equals(r.getValueString("event_type")));
}
@Test
void breakGlassValidationAndPermission() throws Exception
{
assertThatThrownBy(() -> breaks.start("Investigating urgent incident", 60, null, null)).isInstanceOf(QPermissionDeniedException.class);
enableStart();
assertThatThrownBy(() -> breaks.start("short", 60, null, null)).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> breaks.start("Investigating urgent incident", 481, null, null)).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> breaks.start("Investigating urgent incident", 0, null, null)).isInstanceOf(QBadRequestException.class);
assertThatThrownBy(() -> breaks.start("Investigating urgent incident", 60, "missing", null)).isInstanceOf(QBadRequestException.class);
QContext.setQSession(channelUser("assistant", BreakGlassService.START + ".hasAccess"));
assertThatThrownBy(() -> breaks.start("Investigating urgent incident", 60, null, null)).isInstanceOf(QPermissionDeniedException.class);
}
@Test
void statusAllowanceAndPicker() throws Exception
{
var status = new StatusService().status();
assertThat(status.breakGlass()).isNull();
assertThat(status.revealAllowances()).singleElement().satisfies(a -> { assertThat(a.tableName()).isEqualTo("person"); assertThat(a.fieldName()).isEqualTo("email"); assertThat(a.remainingDailyAllowance()).isEqualTo(2); });
var provider = new FieldsOfTablePossibleValueSource();
assertThat(provider.search(new SearchPossibleValueSourceInput())).isEmpty();
assertThat(provider.search(new SearchPossibleValueSourceInput().withOtherValues(Map.of("table_name", "")))).isEmpty();
var fields = provider.search(new SearchPossibleValueSourceInput().withOtherValues(Map.of("table_name", "person")));
assertThat(fields).extracting(v -> v.getId()).containsExactlyInAnyOrder("id", "email", "ssn", "notes");
assertThat(fields).allMatch(v -> v.getLabel() != null);
assertThat(provider.search(new SearchPossibleValueSourceInput().withOtherValues(Map.of("tableName", "person")))).hasSize(4);
assertThatThrownBy(() -> provider.search(new SearchPossibleValueSourceInput().withOtherValues(Map.of("tableName", "unknown")))).hasMessageContaining("Unknown table");
}
@Test
void endReviewAndResultRecheckOwnership() throws Exception
{
enableStart(); QRecord row = breaks.start("Investigating urgent incident", 60, null, null);
var input = new RunBackendStepInput(); input.setRecords(List.of(row));
var output = new RunBackendStepOutput();
var review = new BreakGlassProcesses.EndReview(); review.runOnePage(input, output);
assertThat(review.getProcessSummary(output, false)).hasSize(1);
var load = new BreakGlassProcesses.EndLoad(); load.runOnePage(input, new RunBackendStepOutput());
assertThat(load.getProcessSummary(output, true)).hasSize(1);
assertThat(breaks.find(row.getValue("id")).getValueString("status")).isEqualTo("ENDED");
}
@Test
void startAndEndThroughProcessEngine() throws Exception
{
enableStart();
var input = new RunBackendStepInput();
input.addValue("reason", "Investigating urgent incident");
input.addValue("duration_minutes", 60);
new BreakGlassProcesses.Start().run(input, new RunBackendStepOutput());
QRecord row = SecuredDataRuntime.current().resolver().activeBreakGlass().getFirst();
var lines = ProcessRunner.run(BreakGlassService.END, row.getValue("id"));
assertThat(lines.review()).anyMatch(line -> line.contains("will be ended"));
assertThat(lines.result()).anyMatch(line -> line.contains("was ended"));
assertThat(breaks.find(row.getValue("id")).getValueString("status")).isEqualTo("ENDED");
}
@Test
void expiryScheduleUsesHostSchedulerAndSixtySeconds()
{
var config = new SecuredDataQBitConfig().withSchedulerName("host");
var process = BreakGlassProcesses.processes(config).getLast();
assertThat(process.getSchedule().getRepeatSeconds()).isEqualTo(60);
assertThat(process.getSchedule().getSchedulerName()).isEqualTo("host");
assertThat(process.getIsHidden()).isTrue();
}
public static class Sink implements AlertSinkInterface
{
static final List<AlertSignal> signals = new ArrayList<>();
static final List<AlertResolution> resolutions = new ArrayList<>();
public void raise(AlertSignal signal) { signals.add(signal); }
public void resolve(AlertResolution resolution) { resolutions.add(resolution); }
public void heartbeat(String key, Instant at) { }
}
}
@@ -0,0 +1,452 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.io.Serializable;
import java.lang.ref.WeakReference;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEvent;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.context.CapturedContext;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policy;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Secured data access logger test.
*******************************************************************************/
class SecuredDataAccessLoggerTest
{
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
close();
}
static FieldAccessEvent event(FieldAccessEventType type, FieldAccessOutcome outcome, String fingerprint)
{
return (new FieldAccessEvent().withType(type).withOutcome(outcome).withTableName("person").withFieldNames(List.of("email"))
.withActionName("query").withRecordCount(outcome == FieldAccessOutcome.MATCH ? 1 : 0)
.withSearchValueFingerprints(fingerprint == null ? null : List.of(fingerprint)));
}
/*******************************************************************************
** Twenty first search refused including unflushed rows and persisted history.
*******************************************************************************/
@Test
void twentyFirstSearchRefusedIncludingUnflushedRowsAndPersistedHistory() throws Exception
{
setup();
policy("cs_admin", "PII", "REVEALABLE");
var resolver = SecuredDataRuntime.current().resolver();
var logger = SecuredDataRuntime.current().logger();
for(int i = 0; i < 20; i++)
{
var event = event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, resolver.fingerprintSearchValue("known"));
resolver.checkExactMatch(event);
resolver.onAccess(event);
if(i == 9)
{
logger.flush();
}
}
var refused = event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.REFUSED, resolver.fingerprintSearchValue("sentinel"));
assertThatThrownBy(() -> resolver.checkExactMatch(refused)).isInstanceOf(QPermissionDeniedException.class)
.hasMessage("Exact-match search limit reached for [email]. Try again later or ask an administrator.");
resolver.onAccess(refused);
logger.flush();
assertThat(logs()).filteredOn(row -> "EXACT_MATCH_FILTER".equals(row.getValueString("event_type"))).hasSize(21);
assertThat(logs().stream().filter(row -> "REFUSED".equals(row.getValueString("outcome")))).hasSize(1);
assertThat(JsonUtils.toJson(logs())).doesNotContain("sentinel").doesNotContain("known");
CLOCK.now = NOW.plusSeconds(3601);
resolver.checkExactMatch(event(FieldAccessEventType.EXACT_MATCH_FILTER, null, null));
}
/*******************************************************************************
** Misses and distinct values flag and raise through real alerts spi.
*******************************************************************************/
@Test
void missesAndDistinctValuesFlagAndRaiseThroughRealAlertsSpi() throws Exception
{
QInstance instance = setup();
AlertsSink.signals.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(AlertsSink.class));
var resolver = SecuredDataRuntime.current().resolver();
for(int i = 0; i < 5; i++)
{
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, resolver.fingerprintSearchValue("same")));
}
SecuredDataRuntime.current().logger().flush();
assertThat(logs().stream().filter(row -> Boolean.TRUE.equals(row.getValueBoolean("anomaly_flag")))) .hasSize(1);
assertThat(AlertsSink.signals).hasSize(1);
assertThat(AlertsSink.signals.getFirst().getDedupKey()).isEqualTo("secured_data.enumeration:u1:person.email:2026-10-13");
assertThat(AlertsSink.signals.getFirst().getAlertTypeName()).isEqualTo("secured_data.enumeration");
QContext.setQSession(user("different", "cs_admin"));
for(int i = 0; i < 16; i++)
{
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, resolver.fingerprintSearchValue("value-" + i)));
}
SecuredDataRuntime.current().logger().flush();
assertThat(logs().stream().filter(row -> Boolean.TRUE.equals(row.getValueBoolean("anomaly_flag")))).hasSize(2);
assertThat(AlertsSink.signals).hasSize(2);
}
/*******************************************************************************
** Match resets miss streak and policy can disable all controls.
*******************************************************************************/
@Test
void matchResetsMissStreakAndPolicyCanDisableAllControls() throws Exception
{
setup();
var resolver = SecuredDataRuntime.current().resolver();
for(int i = 0; i < 4; i++)
{
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, "hash"));
}
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hash"));
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, "hash"));
SecuredDataRuntime.current().logger().flush();
assertThat(logs()).allMatch(row -> !row.getValueBoolean("anomaly_flag"));
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("exact_match_per_hour", 0)
.withValue("miss_alert_consecutive", 0).withValue("distinct_value_alert", 0).withValue("min_entropy_enabled", false));
for(int i = 0; i < 25; i++)
{
var event = event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, resolver.fingerprintSearchValue(i));
resolver.checkExactMatch(event);
resolver.onAccess(event);
}
SecuredDataRuntime.current().logger().flush();
assertThat(logs()).allMatch(row -> !row.getValueBoolean("anomaly_flag"));
}
/*******************************************************************************
** Views only log configured classifications with fifty ids and channel.
*******************************************************************************/
@Test
void viewsOnlyLogConfiguredClassificationsWithFiftyIdsAndChannel() throws Exception
{
setup();
QContext.setQSession(channelUser("assistant/t1", "cs_admin"));
var logger = SecuredDataRuntime.current().logger();
logger.onAccess(event(FieldAccessEventType.VIEW, null, null).withFieldNames(List.of("notes")));
List<Serializable> ids = IntStream.rangeClosed(1, 80).mapToObj(i -> (Serializable) i).toList();
logger.onAccess(event(FieldAccessEventType.VIEW, null, null).withFieldNames(List.of("notes", "email", "ssn")).withPrimaryKeys(ids).withRecordCount(80));
logger.onAccess(event(FieldAccessEventType.DENIED, null, null));
logger.flush();
assertThat(logs()).hasSize(2);
QRecord view = logs().stream().filter(row -> "VIEW".equals(row.getValueString("event_type"))).findFirst().orElseThrow();
assertThat(view.getValueString("field_name")).isEqualTo("email,ssn");
assertThat(view.getValueString("client_channel")).isEqualTo("assistant");
assertThat(JsonUtils.toObject(view.getValueString("record_id"), Integer[].class)).hasSize(50);
assertThat(logs().stream().anyMatch(row -> "FILTER_DENIED".equals(row.getValueString("event_type")))) .isTrue();
SecuredDataRuntime.current().config().withViewLogClassifications(Set.of(DataClassification.CONFIDENTIAL));
classification("notes", "CONFIDENTIAL");
logger.onAccess(event(FieldAccessEventType.VIEW, null, null).withFieldNames(List.of("notes", "email")));
logger.flush();
assertThat(logs()).filteredOn(row -> !"CLASSIFICATION_CHANGE".equals(row.getValueString("event_type"))).hasSize(3);
}
/*******************************************************************************
** Queue overflow and shutdown are observable without failing access.
*******************************************************************************/
@Test
void queueOverflowAndShutdownAreObservableWithoutFailingAccess() throws Exception
{
setup(config -> config.withAccessLogQueueCapacity(1));
var logger = SecuredDataRuntime.current().logger();
logger.onAccess(event(FieldAccessEventType.VIEW, null, null));
logger.onAccess(event(FieldAccessEventType.VIEW, null, null));
assertThat(logger.getDroppedCount()).isEqualTo(1);
logger.close();
assertThat(logs()).hasSize(1);
logger.onAccess(event(FieldAccessEventType.VIEW, null, null));
assertThat(logger.getDroppedCount()).isEqualTo(2);
}
/*******************************************************************************
** Asynchronous worker drains in system context.
*******************************************************************************/
@Test
void asynchronousWorkerDrainsInSystemContext() throws Exception
{
setup(config -> config.withAccessLogFlushIntervalMillis(10));
SecuredDataRuntime.current().logger().onAccess(event(FieldAccessEventType.VIEW, null, null));
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
while(logs().isEmpty() && System.nanoTime() < deadline)
{
Thread.sleep(10);
}
assertThat(logs()).hasSize(1);
assertThat(QContext.getQSession().getUser().getIdReference()).isEqualTo("u1");
}
/*******************************************************************************
** Concurrent preflight cannot spend the same last slot.
*******************************************************************************/
@Test
void concurrentPreflightCannotSpendTheSameLastSlot() throws Exception
{
QInstance instance = setup();
var resolver = SecuredDataRuntime.current().resolver();
for(int i = 0; i < 19; i++)
{
resolver.checkExactMatch(event(FieldAccessEventType.EXACT_MATCH_FILTER, null, null));
}
AtomicInteger accepted = new AtomicInteger();
CountDownLatch start = new CountDownLatch(1);
try(var pool = Executors.newFixedThreadPool(8))
{
List<java.util.concurrent.Future<?>> futures = new ArrayList<>();
for(int i = 0; i < 8; i++)
{
futures.add(pool.submit(() ->
{
QContext.withTemporaryContext(new CapturedContext(instance, user("u1", "cs_admin")), () ->
{
start.await();
try
{
resolver.checkExactMatch(event(FieldAccessEventType.EXACT_MATCH_FILTER, null, null));
accepted.incrementAndGet();
}
catch(QPermissionDeniedException expected)
{
}
});
return (null);
}));
}
start.countDown();
for(var future : futures)
{
future.get(5, TimeUnit.SECONDS);
}
}
assertThat(accepted).hasValue(1);
}
/*******************************************************************************
** Consecutive misses span an hour but distinct values do not.
*******************************************************************************/
@Test
void consecutiveMissesSpanAnHourButDistinctValuesDoNot() throws Exception
{
QInstance instance = setup();
AlertsSink.signals.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(AlertsSink.class));
var resolver = SecuredDataRuntime.current().resolver();
for(int i = 0; i < 4; i++)
{
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, resolver.fingerprintSearchValue("same")));
}
SecuredDataRuntime.current().logger().flush();
CLOCK.now = NOW.plusSeconds(3601);
resolver.onAccess(event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.NO_MATCH, resolver.fingerprintSearchValue("same")));
SecuredDataRuntime.current().logger().flush();
assertThat(AlertsSink.signals).hasSize(1);
assertThat(logs().stream().filter(row -> Boolean.TRUE.equals(row.getValueBoolean("anomaly_flag")))) .hasSize(1);
}
/*******************************************************************************
** Break glass identity is captured before worker switches to system.
*******************************************************************************/
@Test
void breakGlassIdentityIsCapturedBeforeWorkerSwitchesToSystem() throws Exception
{
setup();
QRecord grant = insert(SecuredDataTables.BREAK_GLASS, new QRecord().withValue("user_id", "u1").withValue("status", "ACTIVE")
.withValue("classification_ceiling", "PHI").withValue("started_at", NOW).withValue("expires_at", NOW.plusSeconds(60)));
SecuredDataRuntime.current().logger().onAccess(event(FieldAccessEventType.VIEW, null, null));
SecuredDataRuntime.current().logger().flush();
assertThat(logs().getFirst().getValue("break_glass_id")).isEqualTo(grant.getValue("id"));
}
/*******************************************************************************
** Alerts sink.
*******************************************************************************/
public static class AlertsSink implements AlertSinkInterface
{
static final List<AlertSignal> signals = new ArrayList<>();
/*******************************************************************************
** Raise.
*******************************************************************************/
public void raise(AlertSignal signal)
{
signals.add(signal);
}
/*******************************************************************************
** Resolve.
*******************************************************************************/
public void resolve(AlertResolution resolution)
{
}
/*******************************************************************************
** Heartbeat.
*******************************************************************************/
public void heartbeat(String key, Instant at)
{
}
}
/*******************************************************************************
** The worker starts while the host is still building its instance. An idle
** tick opens no context, so it never validates (or enriches) an unfinished
** instance on the worker thread, and a failing tick is logged instead of
** cancelling every later tick.
*******************************************************************************/
@Test
void idleTicksNeverValidateAnUnfinishedInstance() throws Exception
{
setup();
SecuredDataRuntime runtime = SecuredDataRuntime.current();
QInstance unfinished = SecuredDataTestSupport.base();
unfinished.addTable(new QTableMetaData().withName("unfinished").withBackendName("not_registered_yet"));
SecuredDataAccessLogger idle = new SecuredDataAccessLogger(unfinished, runtime);
try
{
idle.flush();
idle.scheduledFlush();
assertThat(unfinished.getHasBeenValidated()).isFalse();
}
finally
{
idle.close();
}
}
/*******************************************************************************
** Neither the worker thread nor the JVM shutdown hook keeps a discarded
** instance alive: a host or test that drops its instance without closing
** the runtime lets it be collected.
*******************************************************************************/
@Test
void aDiscardedInstanceCanBeCollected() throws Exception
{
close();
WeakReference<QInstance> discarded = discardedInstance();
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
List<byte[]> pressure = new ArrayList<>();
while(discarded.get() != null && System.nanoTime() < deadline)
{
pressure.add(new byte[1 << 20]);
if(pressure.size() > 64)
{
pressure.clear();
}
System.gc();
Thread.sleep(20);
}
assertThat(discarded.get()).as("instance still reachable from the access logger").isNull();
}
/*******************************************************************************
** Build, use and drop an instance whose worker ticks every 10 ms.
*******************************************************************************/
private static WeakReference<QInstance> discardedInstance() throws Exception
{
QInstance instance = SecuredDataTestSupport.base();
instance.addBackend(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
instance.addTable(person("memory"));
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withAccessLogFlushIntervalMillis(10)).produce(instance);
Thread.sleep(50);
QContext.clear();
return (new WeakReference<>(instance));
}
/*******************************************************************************
** Hosts that close the runtime themselves can skip the JVM hook; close
** still drains and stops the worker.
*******************************************************************************/
@Test
void shutdownHookIsOptional() throws Exception
{
setup(config -> config.withRegisterShutdownHook(false));
var logger = SecuredDataRuntime.current().logger();
assertThat(logger.isWorkerRunning()).isTrue();
logger.onAccess(event(FieldAccessEventType.VIEW, null, null));
logger.close();
assertThat(logger.isWorkerRunning()).isFalse();
assertThat(logs()).hasSize(1);
}
}
@@ -0,0 +1,186 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.time.temporal.ChronoUnit;
import java.util.List;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.reveal.RevealService;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class SecuredDataAnomaliesTest
{
@AfterEach void cleanup() { close(); }
void init(java.util.function.Consumer<SecuredDataQBitConfig> config) throws Exception
{
var instance = setup(config);
SecuredDataAccessLoggerTest.AlertsSink.signals.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(SecuredDataAccessLoggerTest.AlertsSink.class));
}
QRecord reveal(int record) throws Exception
{
return insert(SecuredDataTables.ACCESS_LOG, ActivityLog.row("REVEAL", "person", "ALLOWED")
.withValue("record_id", String.valueOf(record)).withValue("field_name", "email"));
}
static void seed(QRecord row) throws Exception
{
SystemWrites.call(() -> new InsertAction().execute(new InsertInput(SecuredDataTables.ACCESS_LOG).withRecord(row)));
}
void alertCount(int count)
{
assertThat(SecuredDataAccessLoggerTest.AlertsSink.signals).filteredOn(s -> "secured_data.anomaly".equals(s.getAlertTypeName())).hasSize(count)
.allSatisfy(s -> assertThat(s.getDedupKey()).isEqualTo("secured_data.anomaly:u1:2026-10-13"));
}
@Test void minimumTenAndZeroDaysMedian() throws Exception
{
init(c -> {});
for(int i = 0; i < 9; i++) assertThat(reveal(1).getValueBoolean("anomaly_flag")).isFalse();
alertCount(0);
assertThat(reveal(1).getValueBoolean("anomaly_flag")).isTrue();
alertCount(1);
}
@Test void strictlyAboveThreeTimesMedianAndDailyCacheRollover() throws Exception
{
init(c -> {});
for(int day = 1; day <= 30; day++)
for(int i = 0; i < 4; i++) seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("record_id", "1")
.withValue("created_at", NOW.truncatedTo(ChronoUnit.DAYS).minus(day, ChronoUnit.DAYS).plusSeconds(i)));
for(int i = 0; i < 12; i++) assertThat(reveal(1).getValueBoolean("anomaly_flag")).isFalse();
alertCount(0);
assertThat(reveal(1).getValueBoolean("anomaly_flag")).isTrue();
alertCount(1);
CLOCK.now = NOW.plus(1, ChronoUnit.DAYS);
assertThat(reveal(1).getValueBoolean("anomaly_flag")).isFalse();
}
@Test void distinctRecordBoundaryDuplicatesOtherUsersAndExpiredRows() throws Exception
{
init(c -> c.withAnomalyMinimumDailyReveals(1000));
for(int i = 0; i < 20; i++) assertThat(reveal(i).getValueBoolean("anomaly_flag")).isFalse();
assertThat(reveal(0).getValueBoolean("anomaly_flag")).isFalse();
seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("user_id", "other").withValue("record_id", "999"));
seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("record_id", "998").withValue("created_at", NOW.minusSeconds(3600)));
alertCount(0);
assertThat(reveal(20).getValueBoolean("anomaly_flag")).isTrue();
alertCount(1);
CLOCK.now = NOW.plusSeconds(3601);
assertThat(reveal(21).getValueBoolean("anomaly_flag")).isFalse();
}
@Test void exactMatchFiftyFirstQueuedSearchFlagsWithoutEnumerationAlert() throws Exception
{
init(c -> c.withExactMatchPerHour(0).withMissAlertConsecutive(0).withDistinctValueAlert(0));
var logger = SecuredDataRuntime.current().logger();
for(int i = 0; i < 50; i++) logger.onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hash"));
logger.flush();
assertThat(logs()).allSatisfy(row -> assertThat(row.getValueBoolean("anomaly_flag")).isFalse());
alertCount(0);
logger.onAccess(SecuredDataAccessLoggerTest.event(FieldAccessEventType.EXACT_MATCH_FILTER, FieldAccessOutcome.MATCH, "hash"));
logger.flush();
assertThat(logs().getLast().getValueBoolean("anomaly_flag")).isTrue();
alertCount(1);
}
@Test void configurableThresholdsAndMissingAlertSink() throws Exception
{
setup(c -> c.withAnomalyDailyMedianMultiplier(2).withAnomalyMinimumDailyReveals(3).withAnomalyDistinctRecordsPerHour(2).withAnomalyExactMatchesPerHour(2));
assertThat(reveal(1).getValueBoolean("anomaly_flag")).isFalse();
assertThat(reveal(2).getValueBoolean("anomaly_flag")).isFalse();
assertThat(reveal(3).getValueBoolean("anomaly_flag")).isTrue();
QContext.setQSession(user("u2"));
for(int i = 0; i < 3; i++)
{
var row = insert(SecuredDataTables.ACCESS_LOG, ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH").withValue("field_name", "email"));
assertThat(row.getValueBoolean("anomaly_flag")).isEqualTo(i == 2);
}
}
@Test void actualRevealServiceBreakGlassAndCapFlagAndAlert() throws Exception
{
init(c -> c.withAnomalyMinimumDailyReveals(1));
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("daily_reveal_cap", 1));
var pk = insert("person", new QRecord().withValue("email", "private@example.invalid")).getValue("id");
new RevealService().reveal("person", pk, "email", "Support", null);
assertThat(logs()).filteredOn(r -> "REVEAL".equals(r.getValueString("event_type"))).singleElement()
.satisfies(r -> assertThat(r.getValueBoolean("anomaly_flag")).isTrue());
alertCount(1);
assertThatThrownBy(() -> new RevealService().reveal("person", pk, "email", "Support", null)).isInstanceOf(com.kingsrook.qbits.secureddata.reveal.RevealCappedException.class);
alertCount(2);
QContext.setQSession(user("u1", BreakGlassService.START + ".hasAccess"));
new BreakGlassService().start("Investigating urgent customer incident", 5, null, null);
alertCount(3);
assertThat(logs()).filteredOn(r -> "BREAK_GLASS_START".equals(r.getValueString("event_type"))).singleElement()
.satisfies(r -> assertThat(r.getValueBoolean("anomaly_flag")).isTrue());
}
@Test void invalidThresholdsRejected() throws Exception
{
for(var config : List.of(new SecuredDataQBitConfig().withAnomalyDailyMedianMultiplier(0), new SecuredDataQBitConfig().withAnomalyMinimumDailyReveals(0),
new SecuredDataQBitConfig().withAnomalyDistinctRecordsPerHour(0), new SecuredDataQBitConfig().withAnomalyExactMatchesPerHour(0)))
assertThatThrownBy(config::validate).isInstanceOf(IllegalArgumentException.class);
}
@Test void exactMatchCounterIsScopedToFieldTableUserAndHour() throws Exception
{
init(c -> c.withAnomalyExactMatchesPerHour(2));
seed(ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH").withValue("field_name", "email").withValue("created_at", NOW.minusSeconds(3600)));
seed(ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH").withValue("field_name", "ssn"));
seed(ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH").withValue("field_name", "email").withValue("user_id", "u2"));
for(int i = 0; i < 3; i++)
{
QRecord row = insert(SecuredDataTables.ACCESS_LOG, ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH").withValue("field_name", "email"));
assertThat(row.getValueBoolean("anomaly_flag")).isEqualTo(i == 2);
}
alertCount(1);
}
@Test void configurableMedianMultiplierUsesHistoricalCounts() throws Exception
{
init(c -> c.withAnomalyDailyMedianMultiplier(2).withAnomalyMinimumDailyReveals(3));
for(int day = 1; day <= 30; day++)
for(int i = 0; i < 2; i++) seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("record_id", "1")
.withValue("created_at", NOW.minus(day, ChronoUnit.DAYS)));
for(int i = 0; i < 4; i++) assertThat(reveal(1).getValueBoolean("anomaly_flag")).isFalse();
alertCount(0);
assertThat(reveal(1).getValueBoolean("anomaly_flag")).isTrue();
alertCount(1);
}
}
@@ -0,0 +1,123 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policy;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Secured data ddl test.
*******************************************************************************/
class SecuredDataDdlTest
{
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
close();
}
/*******************************************************************************
** Postgres ddl runs in h2and supports actual qqq writes.
*******************************************************************************/
@Test
void postgresDdlRunsInH2AndSupportsActualQqqWrites() throws Exception
{
String url = "jdbc:h2:mem:secured_data;MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;DB_CLOSE_DELAY=-1";
try(Connection connection = DriverManager.getConnection(url, "sa", ""))
{
String ddl = Files.readString(Path.of("docs/ddl/secured-data-postgres.sql"));
for(String sql : ddl.split(";"))
{
if(!sql.isBlank())
{
connection.createStatement().execute(sql);
}
}
RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName("rdbms").withVendor("h2").withHostName("mem")
.withDatabaseName("secured_data").withUsername("sa");
backend.setJdbcUrl(url);
QInstance instance = base();
instance.addBackend(backend);
instance.addTable(person("rdbms").withBackendDetails(new RDBMSTableBackendDetails().withTableName("person")));
connection.createStatement().execute("CREATE TABLE person (id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, email VARCHAR(255), ssn VARCHAR(255), notes TEXT)");
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withBackendName("rdbms").withClock(CLOCK)
.withAccessLogFlushIntervalMillis(60000).withTableMetaDataCustomizer((i, t) -> t.withBackendDetails(new RDBMSTableBackendDetails().withTableName(t.getName())))).produce(instance);
QContext.init(instance, user("u1", "cs_admin"));
RUNTIMES.add(SecuredDataRuntime.current());
classification(null, "PII");
policy("cs_admin", "PII", "REVEALABLE");
assertThat(SecuredDataRuntime.current().resolver().resolve(instance.getTable("person"), instance.getTable("person").getField("email")).tier().name()).isEqualTo("REVEALABLE");
SecuredDataRuntime.current().logger().onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.EXACT_MATCH_FILTER,
com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessOutcome.MATCH, "hmac"));
SecuredDataRuntime.current().logger().flush();
assertThat(logs()).extracting(row -> row.getValueString("event_type")).containsExactly("CLASSIFICATION_CHANGE", "POLICY_CHANGE", "EXACT_MATCH_FILTER");
// Equal timestamps must be counted by SQL as well as the memory backend.
SecuredDataRuntime.current().config().withAnomalyMinimumDailyReveals(1000).withAnomalyDistinctRecordsPerHour(2).withAnomalyExactMatchesPerHour(2);
for(int i = 0; i < 3; i++)
{
QRecord audit = insert(SecuredDataTables.ACCESS_LOG, com.kingsrook.qbits.secureddata.support.ActivityLog.row("REVEAL", "person", "ALLOWED")
.withValue("record_id", String.valueOf(i)).withValue("field_name", "email"));
assertThat(audit.getValueBoolean("anomaly_flag")).isEqualTo(i == 2);
}
for(int i = 0; i < 2; i++)
{
QRecord audit = insert(SecuredDataTables.ACCESS_LOG, com.kingsrook.qbits.secureddata.support.ActivityLog.row("EXACT_MATCH_FILTER", "person", "MATCH")
.withValue("field_name", "email"));
assertThat(audit.getValueBoolean("anomaly_flag")).isEqualTo(i == 1);
}
assertThatThrownBy(() -> connection.createStatement().execute("INSERT INTO secured_data_classification (table_name, classification) VALUES ('person', 'PHI')"))
.isInstanceOf(java.sql.SQLException.class);
assertThatThrownBy(() -> connection.createStatement().execute("INSERT INTO secured_data_policy (name, permission_name, classification, tier, reveal_minutes) VALUES ('x','x','PII','FULL',61)"))
.isInstanceOf(java.sql.SQLException.class);
}
}
}
@@ -0,0 +1,117 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QUser;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
/*******************************************************************************
** Test-only local Next demo, following WebhooksDemoServer. Synthetic identifiers
** and reasons only; never shipped. Run on port 8098 after test compilation.
*******************************************************************************/
public class SecuredDataDemoServer
{
/*******************************************************************************
** Local demo administrator; not an application authentication implementation.
*******************************************************************************/
public static class Sessions implements QAuthenticationModuleCustomizerInterface
{
/*******************************************************************************
** Give the local demo access to its home and tables.
*******************************************************************************/
public void customizeSession(QInstance instance, QSession session, Map<String, Object> context)
{
session.setUser(new QUser().withIdReference("demo-admin").withFullName("Demo Administrator"));
session.setPermissions(Set.of("secured_data.admin", "secured_data.hasAccess", "person.read"));
}
}
/*******************************************************************************
** Seed every home section with realistic synthetic rows.
*******************************************************************************/
public static void seedDemo() throws Exception
{
String[] names = {"Avery Morgan", "Jordan Patel", "Sam Rivera", "Taylor Chen"};
String[] reasons = {"Customer identity verification", "Support case follow-up", "Account recovery investigation", "Review of an escalated access request with the customer support team"};
for(int i = 0; i < 48; i++)
{
SecuredDataAnomaliesTest.seed(ActivityLog.row("REVEAL", "person", "ALLOWED")
.withValue("user_id", "operator-" + i % 4).withValue("user_name", names[i % 4])
.withValue("record_id", String.valueOf(4200 + i)).withValue("field_name", i % 2 == 0 ? "email" : "ssn")
.withValue("reason", reasons[i % 4]).withValue("client_channel", i % 3 == 0 ? "api" : "web")
.withValue("created_at", NOW.minusSeconds(i < 15 ? i * 180L : i * 36000L)).withValue("anomaly_flag", i < 2));
}
for(int i = 0; i < 3; i++)
{
String event = new String[] {"BREAK_GLASS_START", "BREAK_GLASS_END", "BREAK_GLASS_EXPIRED"}[i];
SecuredDataAnomaliesTest.seed(ActivityLog.row(event, SecuredDataTables.BREAK_GLASS, "ALLOWED")
.withValue("user_name", names[i]).withValue("reason", "Urgent investigation of customer support incident SD-2026-1042")
.withValue("created_at", NOW.minusSeconds(i * 1200L)).withValue("anomaly_flag", i == 0));
}
for(int i = 0; i < 4; i++)
{
insert(SecuredDataTables.REVEAL_GRANT, new QRecord().withValue("user_id", "operator-" + i).withValue("user_name", names[i])
.withValue("table_name", "person").withValue("record_id", "4200").withValue("field_name", "email")
.withValue("granted_at", NOW.minusSeconds(60)).withValue("expires_at", NOW.plusSeconds(240)).withValue("access_log_id", i + 1));
}
insert(SecuredDataTables.BREAK_GLASS, new QRecord().withValue("user_id", "operator-0").withValue("user_name", names[0])
.withValue("started_at", NOW.minusSeconds(300)).withValue("expires_at", NOW.plusSeconds(3300)).withValue("status", "ACTIVE"));
for(int i = 0; i < 6; i++)
SecuredDataAnomaliesTest.seed(ActivityLog.row("FILTER_DENIED", "person", "DENIED").withValue("user_name", names[i % 4]));
policy("support_specialist", "PII", "REVEALABLE");
classification("email", "PII");
}
/*******************************************************************************
** Run the real bundled Next frontend and middleware against memory data.
*******************************************************************************/
public static void main(String[] args) throws Exception
{
QInstance instance = setup();
instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("mock").withType(QAuthenticationType.MOCK));
instance.getAuthentication().setCustomizer(new QCodeReference(Sessions.class));
instance.setHomeAppName("secured_data");
QContext.setQSession(user("demo-admin", "secured_data.admin"));
seedDemo();
new QApplicationJavalinServer(new AbstractQQQApplication()
{
public QInstance defineQInstance()
{
return (instance);
}
}).withPort(args.length == 0 ? 8098 : Integer.parseInt(args[0])).withServeFrontendNext(true).start();
System.out.println("Secured Data demo: http://localhost:8098/app/secured_data");
}
}
@@ -0,0 +1,143 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.util.List;
import com.kingsrook.qbits.secureddata.app.SecuredDataApp;
import com.kingsrook.qbits.secureddata.app.SecuredDataHomeData;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.ActivityLog;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class SecuredDataHomeTest
{
@AfterEach void cleanup() { close(); }
String home() throws Exception
{
return ((RawHTML) new SecuredDataApp.HomeRenderer().render(new RenderWidgetInput()).getWidgetData()).getHtml();
}
@Test void emptyStateAndNav() throws Exception
{
var instance = setup();
QContext.setQSession(user("admin", "secured_data.admin"));
var data = SecuredDataHomeData.load();
assertThat(data.getKpis().values()).containsOnly(0L);
assertThat(data.getSections()).hasSize(6).allSatisfy(s -> assertThat(s.rows()).isEmpty());
assertThat(home()).contains("No reveals yet", "No reveals in the last 30 days", "No reveal reasons in the last 30 days",
"No break-glass events yet", "No policy or classification changes yet", "No anomaly flags in the last 7 days");
var app = instance.getApp("secured_data");
assertThat(app.getWidgets()).containsExactly(SecuredDataApp.HOME_WIDGET);
assertThat(app.getChildren()).extracting(c -> c.getName()).containsExactly(SecuredDataTables.CLASSIFICATION, SecuredDataTables.POLICY,
SecuredDataTables.ACCESS_LOG, SecuredDataTables.BREAK_GLASS, SecuredDataTables.RETENTION_RULE);
assertThat(instance.getTable(SecuredDataTables.ACCESS_LOG).getSections()).anyMatch(s -> SecuredDataTables.REVEALS_WIDGET.equals(s.getWidgetName()));
}
@Test void seededKpisOrderingLimitsLinksAndEveryWidget() throws Exception
{
setup(c -> c.withTablePathPrefix("/admin/data").withAccessLogQueueCapacity(1));
QContext.setQSession(user("admin", "secured_data.admin"));
for(int i = 0; i < 15; i++)
SecuredDataAnomaliesTest.seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("user_id", i < 12 ? "alice" : "bob")
.withValue("user_name", i < 12 ? "Alice <b>" : "Bob").withValue("reason", i < 11 ? "Customer support" : "Case review")
.withValue("record_id", String.valueOf(i)).withValue("field_name", "email").withValue("client_channel", "web")
.withValue("created_at", NOW.minusSeconds(i)).withValue("anomaly_flag", i == 0));
SecuredDataAnomaliesTest.seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("created_at", NOW.minusSeconds(31 * 86400L)));
SecuredDataAnomaliesTest.seed(ActivityLog.row("REVEAL", "person", "ALLOWED").withValue("created_at", NOW.plusSeconds(60)));
for(String type : List.of("BREAK_GLASS_START", "BREAK_GLASS_END", "BREAK_GLASS_EXPIRED", "FILTER_DENIED"))
SecuredDataAnomaliesTest.seed(ActivityLog.row(type, "person", "ALLOWED"));
SecuredDataAnomaliesTest.seed(ActivityLog.row("EXACT_MATCH_FILTER", "person", "REFUSED"));
SecuredDataAnomaliesTest.seed(ActivityLog.row("FILTER_DENIED", "person", "DENIED").withValue("created_at", NOW.minusSeconds(8 * 86400L)));
for(int i = 0; i < 2; i++)
insert(SecuredDataTables.REVEAL_GRANT, new QRecord().withValue("user_id", "alice").withValue("table_name", "person")
.withValue("record_id", "1").withValue("field_name", "email").withValue("granted_at", NOW.minusSeconds(60))
.withValue("expires_at", NOW.plusSeconds(i == 0 ? 60 : 0)).withValue("access_log_id", 1));
for(int i = 0; i < 3; i++)
insert(SecuredDataTables.BREAK_GLASS, new QRecord().withValue("user_id", "alice").withValue("started_at", NOW.minusSeconds(60))
.withValue("expires_at", NOW.plusSeconds(i == 0 ? 60 : 0)).withValue("status", i == 2 ? "ENDED" : "ACTIVE"));
var policy = policy("role", "PII", "MASKED");
update(SecuredDataTables.POLICY, new QRecord().withValue("id", policy.getValue("id")).withValue("name", "Changed"));
delete(SecuredDataTables.POLICY, policy);
classification("email", "PII");
var logger = SecuredDataRuntime.current().logger();
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.VIEW, null, null));
logger.onAccess(SecuredDataAccessLoggerTest.event(com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessEventType.VIEW, null, null));
var data = SecuredDataHomeData.load();
assertThat(data.getKpis()).containsEntry("Reveals today", 15L).containsEntry("Active reveal grants", 1L)
.containsEntry("Active break-glass sessions", 1L).containsEntry("Anomaly flags (7 days)", 1L)
.containsEntry("Denied filters (7 days)", 2L).containsEntry("Access-log queue drops", 1L);
assertThat(data.getSections().get(0).rows()).hasSize(10).extracting(r -> r.getValueString("record_id"))
.containsExactly("0", "1", "2", "3", "4", "5", "6", "7", "8", "9");
assertThat(data.getSections().get(1).rows()).extracting(r -> r.getValueString("user_id")).containsExactly("alice", "bob");
assertThat(data.getSections().get(1).rows().getFirst().getValueInteger("count")).isEqualTo(12);
assertThat(data.getSections().get(2).rows().getFirst().getValueString("reason")).isEqualTo("Customer support");
assertThat(data.getSections().get(3).rows()).hasSize(3);
assertThat(data.getSections().get(4).rows()).hasSize(4).extracting(r -> r.getValueString("action_name")).containsExactly("INSERT", "DELETE", "UPDATE", "INSERT");
assertThat(data.getSections().get(5).rows()).hasSize(1);
for(var section : data.getSections())
{
assertThat(section.href()).startsWith("/admin/data/secured_data_access_log?filter=");
String filter = URLDecoder.decode(section.href().split("filter=")[1], StandardCharsets.UTF_8);
assertThat(filter).contains("created_at").doesNotContain("\"limit\":10");
}
String html = home();
assertThat(html).contains("Recent reveals", "Top revealers (30 days)", "Top reasons (30 days)", "Break-glass events", "Policy and classification changes", "Anomaly flags")
.contains("Alice &lt;b&gt;", "Customer support", "BREAK_GLASS_EXPIRED", "CLASSIFICATION_CHANGE", "View all", "UTC", "title=", "text-overflow:ellipsis", "white-space:nowrap", "minmax(min(100%", "var(--qqq-grid-border")
.doesNotContain("Alice <b>", "overflow:auto", "overflow-x", "<style", "#ffffff", "rgba(");
assertThat(html.split("<section ", -1)).hasSize(7);
}
@Test void viewerWithoutAdminCannotReadHome() throws Exception
{
setup();
assertThatThrownBy(SecuredDataHomeData::load).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class);
}
@Test void classificationMutationsAreValueFreeAndAttributed() throws Exception
{
setup();
QContext.setQSession(user("admin", "secured_data.admin"));
var input = new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput(SecuredDataTables.CLASSIFICATION)
.withRecord(new QRecord().withValue("table_name", "person").withValue("field_name", "email").withValue("classification", "PII"));
var row = new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(input).getRecords().getFirst();
new com.kingsrook.qqq.backend.core.actions.tables.UpdateAction().execute(new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(SecuredDataTables.CLASSIFICATION)
.withRecord(new QRecord().withValue("id", row.getValue("id")).withValue("classification", "PHI")));
new com.kingsrook.qqq.backend.core.actions.tables.DeleteAction().execute(new com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput(SecuredDataTables.CLASSIFICATION)
.withPrimaryKeys(List.of(row.getValue("id"))));
assertThat(logs()).hasSize(3).allSatisfy(r -> {
assertThat(r.getValueString("user_id")).isEqualTo("admin");
assertThat(r.getValueString("record_id")).isEqualTo(row.getValueString("id"));
assertThat(r.getValueString("event_type")).isEqualTo("CLASSIFICATION_CHANGE");
assertThat(r.getValue("reason")).isNull();
});
assertThat(logs()).extracting(r -> r.getValueString("action_name")).containsExactly("INSERT", "UPDATE", "DELETE");
}
}
@@ -0,0 +1,161 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.breakglass.BreakGlassService;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
import com.kingsrook.qqq.middleware.javalin.QJavalinImplementation;
import org.json.JSONObject;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.*;
import static org.assertj.core.api.Assertions.*;
class SecuredDataHttpTest
{
QApplicationJavalinServer server;
int port;
@AfterEach
void cleanup()
{
if(server != null) server.stop();
QJavalinImplementation.setSessionCookieHttpOnly(false);
close();
}
public static class Sessions implements QAuthenticationModuleCustomizerInterface
{
public void customizeSession(QInstance instance, QSession session, Map<String, Object> context)
{
session.setPermissions(java.util.Set.of("person.read", "cs_admin"));
}
}
void start() throws Exception
{
QInstance instance = setup();
instance.getAuthentication().setType(QAuthenticationType.MOCK);
instance.getAuthentication().setCustomizer(new QCodeReference(Sessions.class));
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("daily_reveal_cap", 1));
insert("person", new QRecord().withValue("id", 1).withValue("email", "HTTP-CANARY@example.invalid"));
server = new QApplicationJavalinServer(new AbstractQQQApplication()
{
public QInstance defineQInstance() { return instance; }
}).withPort(0).withServeFrontendNext(false).withServeFrontendMaterialDashboard(false).withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true);
server.start();
var serviceField = QApplicationJavalinServer.class.getDeclaredField("service");
serviceField.setAccessible(true);
port = ((io.javalin.Javalin) serviceField.get(server)).port();
}
HttpResponse<String> request(String path, String body, boolean assistant) throws Exception
{
var builder = HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/qqq/v1" + path)).header("Cookie", "sessionUUID=phase-c-user");
if(assistant) builder.header("X-QQQ-Client", "assistant");
if(body != null) builder.header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString(body));
return HttpClient.newHttpClient().send(builder.build(), HttpResponse.BodyHandlers.ofString());
}
@Test
void optionsRevealStatusAndCapThroughRealMiddleware() throws Exception
{
start();
var options = request("/qbit/secured-data/reveal/options?tableName=person&fieldName=email", null, false);
assertThat(options.statusCode()).as(options.body()).isEqualTo(200);
assertThat(new JSONObject(options.body()).getInt("remainingDailyAllowance")).isEqualTo(1);
String body = "{\"tableName\":\"person\",\"primaryKey\":1,\"fieldName\":\"email\",\"reason\":\"Investigation\"}";
var reveal = request("/qbit/secured-data/reveal", body, false);
assertThat(reveal.statusCode()).as(reveal.body()).isEqualTo(200);
assertThat(new JSONObject(reveal.body()).getString("value")).isEqualTo("HTTP-CANARY@example.invalid");
var status = request("/qbit/secured-data/status", null, false);
assertThat(status.statusCode()).as(status.body()).isEqualTo(200);
assertThat(new JSONObject(status.body()).getInt("revealsUsedToday")).isEqualTo(1);
assertErrorResponse(request("/qbit/secured-data/reveal", body, false), 429);
assertErrorResponse(request("/qbit/secured-data/reveal", body, true), 403);
assertErrorResponse(request("/qbit/secured-data/reveal", "{}", false), 400);
assertErrorResponse(request("/qbit/secured-data/reveal", "{\"primaryKey\":1}", false), 400);
assertErrorResponse(request("/qbit/secured-data/reveal/options?tableName=person&fieldName=missing", null, false), 404);
}
/*******************************************************************************
** Verify standard error bodies without changing endpoint status semantics.
*******************************************************************************/
private void assertErrorResponse(HttpResponse<String> response, int status)
{
assertThat(response.statusCode()).as(response.body()).isEqualTo(status);
JSONObject body = new JSONObject(response.body());
assertThat(body.keySet()).contains("error", "code", "message", "traceId");
assertThat(body.getString("error")).isNotBlank();
assertThat(body.getString("code")).isNotBlank();
assertThat(body.getString("message")).isNotBlank();
assertThat(body.getString("traceId")).isNotBlank();
}
@Test
void notificationTargetsOnlySecuredDataAdmins() throws Exception
{
Class<?> signalType;
try { signalType = Class.forName("com.kingsrook.qqq.backend.core.alerts.NotificationSignal"); }
catch(ClassNotFoundException expected) { org.junit.jupiter.api.Assumptions.assumeTrue(false, "Installed core lacks notification SPI #1125"); return; }
var instance = setup();
Bell.signal = null;
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(Bell.class));
var metadata = AlertsInstanceMetaData.ofOrWithNew(instance);
var setter = metadata.getClass().getDeclaredMethod("setLoadedSink", AlertSinkInterface.class);
setter.setAccessible(true);
setter.invoke(metadata, java.lang.reflect.Proxy.newProxyInstance(AlertSinkInterface.class.getClassLoader(), new Class<?>[] {AlertSinkInterface.class},
(proxy, method, args) -> { if(method.getName().equals("notify")) Bell.signal = args[0]; return null; }));
QContext.setQSession(user("u1", BreakGlassService.START + ".hasAccess"));
new BreakGlassService().start("Urgent incident investigation", 60, null, null);
assertThat(Bell.signal).isNotNull();
Object audience = signalType.getMethod("getAudience").invoke(Bell.signal);
assertThat((List<?>) audience.getClass().getMethod("getValues").invoke(audience)).isEqualTo(List.of("secured_data.admin"));
}
public static class Bell implements AlertSinkInterface
{
static Object signal;
public void raise(AlertSignal ignored) { }
public void resolve(AlertResolution ignored) { }
public void heartbeat(String key, Instant at) { }
}
}
@@ -0,0 +1,224 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.secureddata.metadata.FieldsOfTablePossibleValueSource;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataEnums;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.TableFeaturePermissionsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.RUNTIMES;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.base;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.delete;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.logs;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Secured data qbit producer test.
*******************************************************************************/
class SecuredDataQBitProducerTest
{
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
close();
}
/*******************************************************************************
** Produces complete metadata.
*******************************************************************************/
@Test
void producesCompleteMetadata() throws Exception
{
QInstance instance = setup();
assertThat(instance.getTables().keySet()).containsAll(SecuredDataTables.ALL);
assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-secured-data").getVersion()).isEqualTo("0.1.0");
for(String name : SecuredDataTables.ALL)
{
QTableMetaData table = instance.getTable(name);
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(name.equals(SecuredDataTables.AUDIT_KEY) ? AuditLevel.NONE : AuditLevel.FIELD);
assertThat(table.getSections()).isNotEmpty();
assertThat(SecuredDataMetadata.classification(table, null)).isEqualTo(DataClassification.CONFIDENTIAL);
table.getFields().values().forEach(field -> assertThat(field.getBackendName()).isEqualTo(field.getName()));
}
assertThat(SecuredDataTables.esbTableNames()).containsExactlyElementsOf(SecuredDataTables.MAIN);
assertThat(SecuredDataEnums.sources()).hasSize(6);
assertThat(ClientChannelsMetaData.ofOrWithNew(instance).getDeniedPatterns().get("assistant")).contains("secured_data_*");
assertThat(TableFeaturePermissionsMetaData.ofOrWithNew(instance).getPermissions()).extracting("suffix").contains("secured_data_full", "secured_data_reveal");
assertThat(instance.getTable(SecuredDataTables.ACCESS_LOG).getDisabledCapabilities()).contains(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
assertThat(instance.getJoin(SecuredDataTables.REVEALS_JOIN)).isNotNull();
assertThat(instance.getWidget(SecuredDataTables.REVEALS_WIDGET)).isNotNull();
assertThat(instance.getApp("secured_data").getChildren()).hasSize(5);
}
/*******************************************************************************
** Config and producer contract.
*******************************************************************************/
@Test
void configAndProducerContract() throws Exception
{
SecuredDataQBitConfig config = new SecuredDataQBitConfig().withBackendName("memory")
.withDefaultTier(DataClassification.PHI, FieldAccessTier.HIDDEN).withDefaultTierByClassification(Map.of("PHI", "FULL"))
.withNameHeuristics(true).withTableMetaDataCustomizer((instance, table) -> table);
assertThat(config.defaultTier(DataClassification.PHI)).isEqualTo(FieldAccessTier.FULL);
assertThat(config.defaultTier(DataClassification.PII)).isEqualTo(FieldAccessTier.MASKED);
SecuredDataQBitProducer producer = new SecuredDataQBitProducer().withQBitConfig(config).withNamespace("test");
assertThat(producer.getNamespace()).isEqualTo("test");
assertThat(producer.getQBitConfig()).isSameAs(config);
assertThat(producer.isEnabled()).isTrue();
assertThat(new SecuredDataQBitProducer().isEnabled()).isFalse();
assertThatThrownBy(() -> new SecuredDataQBitConfig().withAccessLogQueueCapacity(0).validate()).isInstanceOf(IllegalArgumentException.class);
setup();
assertThatThrownBy(() -> new SecuredDataQBitProducer().withQBitConfig(config).produce(QContext.getQInstance())).isInstanceOf(IllegalStateException.class);
}
/*******************************************************************************
** Admin can read but cannot forge activity and assistant cannot read.
*******************************************************************************/
@Test
void adminCanReadButCannotForgeActivityAndAssistantCannotRead() throws Exception
{
QInstance instance = setup();
QContext.setQSession(user("admin", "secured_data.admin"));
assertThat(new QueryAction().execute(new QueryInput(SecuredDataTables.POLICY)).getRecords()).isEmpty();
for(String table : SecuredDataTables.SYSTEM_WRITTEN)
{
assertThatThrownBy(() -> new InsertAction().execute(new InsertInput(table).withRecord(new QRecord()))).hasMessageContaining("Permission denied");
}
QContext.setQSession(channelUser("assistant/t1", "secured_data.admin"));
for(String table : SecuredDataTables.ALL)
{
assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(table))).hasMessageContaining("Permission denied");
}
QContext.setQSession(user("ordinary"));
assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(SecuredDataTables.CLASSIFICATION))).hasMessageContaining("Permission denied");
}
/*******************************************************************************
** Fields picker and classification uniqueness.
*******************************************************************************/
@Test
void fieldsPickerAndClassificationUniqueness() throws Exception
{
setup();
FieldsOfTablePossibleValueSource picker = new FieldsOfTablePossibleValueSource();
assertThat(picker.search(new SearchPossibleValueSourceInput().withOtherValues(Map.of("table_name", "person")))).hasSize(4);
assertThat(picker.search(new SearchPossibleValueSourceInput())).isEmpty();
assertThat(picker.getPossibleValue("email").getId()).isEqualTo("email");
assertThat(picker.getPossibleValue(null)).isNull();
classification(null, "CONFIDENTIAL");
assertThatThrownBy(() -> classification(null, "PHI")).hasMessageContaining("system insert failed");
assertThatThrownBy(() -> classification("no_such_field", "PHI")).hasMessageContaining("system insert failed");
assertThatThrownBy(() -> insert(SecuredDataTables.POLICY, policyRow("x", "PII", "FULL").withValue("reveal_minutes", 61))).hasMessageContaining("system insert failed");
}
/*******************************************************************************
** System cannot update or delete an access log.
*******************************************************************************/
@Test
void systemCannotUpdateOrDeleteAnAccessLog() throws Exception
{
setup();
QRecord row = insert(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("event_type", "VIEW").withValue("table_name", "person").withValue("created_at", NOW));
assertThatThrownBy(() -> update(SecuredDataTables.ACCESS_LOG, new QRecord().withValue("id", row.getValue("id")).withValue("outcome", "DENIED"))).isInstanceOf(Exception.class);
assertThatThrownBy(() -> delete(SecuredDataTables.ACCESS_LOG, row)).isInstanceOf(Exception.class);
assertThat(logs()).hasSize(1);
}
/*******************************************************************************
** Versions invalidate after commit and never for rollback.
*******************************************************************************/
@Test
void versionsInvalidateAfterCommitAndNeverForRollback() throws Exception
{
setup();
var runtime = SecuredDataRuntime.current();
var customizer = new com.kingsrook.qbits.secureddata.metadata.SecuredDataTableCustomizers();
var transaction = new com.kingsrook.qqq.backend.core.actions.QBackendTransaction();
var input = new InsertInput(SecuredDataTables.CLASSIFICATION).withTransaction(transaction);
long before = runtime.classificationVersion();
customizer.postInsert(input, List.of(new QRecord().withValue("id", 42)));
assertThat(runtime.classificationVersion()).isEqualTo(before);
transaction.rollback();
assertThat(runtime.classificationVersion()).isEqualTo(before);
customizer.postInsert(input, List.of(new QRecord().withValue("id", 43)));
transaction.commit();
assertThat(runtime.classificationVersion()).isGreaterThan(before);
}
/*******************************************************************************
** Code heuristics are opt in and preserve explicit classifications.
*******************************************************************************/
@Test
void codeHeuristicsAreOptInAndPreserveExplicitClassifications() throws Exception
{
QInstance instance = base();
instance.addBackend(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
QTableMetaData table = person("memory");
table.withField(new com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData("diagnosis_code", com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType.STRING));
instance.addTable(table);
new SecuredDataQBitProducer().withQBitConfig(new SecuredDataQBitConfig().withNameHeuristics(true)).produce(instance);
QContext.init(instance, user("u1"));
RUNTIMES.add(SecuredDataRuntime.current());
assertThat(SecuredDataClassifications.effective(table, "diagnosis_code")).isEqualTo(DataClassification.PHI);
assertThat(SecuredDataClassifications.effective(table, "email")).isEqualTo(DataClassification.PII);
}
}
@@ -0,0 +1,348 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.util.List;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldAccessTier;
import com.kingsrook.qqq.backend.core.actions.permissions.fieldaccess.FieldMaskStyle;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.CLOCK;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.NOW;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.channelUser;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.classification;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.close;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.delete;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.insert;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.person;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policy;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.policyRow;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.setup;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.update;
import static com.kingsrook.qbits.secureddata.SecuredDataTestSupport.user;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Secured data resolver test.
*******************************************************************************/
class SecuredDataResolverTest
{
QTableMetaData person;
SecuredDataResolver resolver;
/*******************************************************************************
** Init.
*******************************************************************************/
@BeforeEach
void init() throws Exception
{
person = setup().getTable("person");
resolver = SecuredDataRuntime.current().resolver();
}
/*******************************************************************************
** Cleanup.
*******************************************************************************/
@AfterEach
void cleanup()
{
close();
}
/*******************************************************************************
** Tier.
*******************************************************************************/
FieldAccessTier tier(String field)
{
return (resolver.resolve(person, person.getField(field)).tier());
}
/*******************************************************************************
** Classification order and highest.
*******************************************************************************/
@Test
void classificationOrderAndHighest() throws Exception
{
assertThat(SecuredDataClassifications.effective(person)).isNull();
assertThat(SecuredDataClassifications.effective(person, "email")).isEqualTo(DataClassification.PII);
person.withSupplementalMetaData(new AiTableMetaData().withClassification(DataClassification.CONFIDENTIAL));
assertThat(SecuredDataClassifications.effective(person, "notes")).isEqualTo(DataClassification.CONFIDENTIAL);
assertThat(SecuredDataClassifications.effective(person, "email")).isEqualTo(DataClassification.PII);
QRecord tableRow = classification(null, "PHI");
assertThat(SecuredDataClassifications.effective(person, "email")).isEqualTo(DataClassification.PHI);
assertThat(SecuredDataClassifications.effective(person)).isEqualTo(DataClassification.PHI);
QRecord fieldRow = classification("email", "PUBLIC");
assertThat(SecuredDataClassifications.effective(person, "email")).isEqualTo(DataClassification.PUBLIC);
assertThat(SecuredDataClassifications.highest(person)).isEqualTo(DataClassification.PHI);
assertThat(SecuredDataClassifications.highest(person, List.of("email"))).isEqualTo(DataClassification.PUBLIC);
assertThat(SecuredDataClassifications.highest(person, List.of())).isNull();
delete(SecuredDataTables.CLASSIFICATION, fieldRow);
delete(SecuredDataTables.CLASSIFICATION, tableRow);
assertThat(SecuredDataClassifications.effective(person, "email")).isEqualTo(DataClassification.PII);
}
/*******************************************************************************
** Defaults public primary key and admin separation.
*******************************************************************************/
@Test
void defaultsPublicPrimaryKeyAndAdminSeparation() throws Exception
{
assertThat(tier("id")).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("notes")).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("email")).isEqualTo(FieldAccessTier.MASKED);
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.MASKED);
classification("notes", "CONFIDENTIAL");
assertThat(tier("notes")).isEqualTo(FieldAccessTier.FULL);
QContext.setQSession(user("admin", "secured_data.admin"));
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.MASKED);
QTableMetaData own = QContext.getQInstance().getTable(SecuredDataTables.POLICY);
assertThat(resolver.resolve(own, own.getField("name")).tier()).isEqualTo(FieldAccessTier.FULL);
assertThat(SystemWrites.call(() -> tier("ssn"))).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.MASKED);
}
/*******************************************************************************
** Least restrictive active matching policy and table scope.
*******************************************************************************/
@Test
void leastRestrictiveActiveMatchingPolicyAndTableScope() throws Exception
{
policy("cs_admin", "PII", "REVEALABLE");
policy("other_role", "PII", "FULL");
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
QRecord full = policy("cs_admin", "PII", "FULL");
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
update(SecuredDataTables.POLICY, new QRecord().withValue("id", full.getValue("id")).withValue("is_active", false));
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
QTableMetaData another = new QTableMetaData().withName("another").withBackendName("memory").withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER));
QContext.getQInstance().addTable(another);
update(SecuredDataTables.POLICY, new QRecord().withValue("id", full.getValue("id")).withValue("is_active", true).withValue("table_name", "another"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
}
/*******************************************************************************
** Channel rows only cap including features and break glass.
*******************************************************************************/
@Test
void channelRowsOnlyCapIncludingFeaturesAndBreakGlass() throws Exception
{
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "FULL").withValue("client_channel", "assistant"));
QContext.setQSession(channelUser("assistant/t1", "cs_admin"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.MASKED);
policy("cs_admin", "PII", "FULL");
QRecord cap = insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "HIDDEN").withValue("client_channel", "assistant"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.HIDDEN);
QContext.setQSession(user("u1", "cs_admin"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
QContext.setQSession(channelUser("assistant/t2", "cs_admin", "person.secured_data_full"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.HIDDEN);
breakGlass("PHI", null);
assertThat(tier("email")).isEqualTo(FieldAccessTier.HIDDEN);
}
/*******************************************************************************
** Feature grants have no admin shortcut.
*******************************************************************************/
@Test
void featureGrantsHaveNoAdminShortcut() throws Exception
{
policy("cs_admin", "PII", "HIDDEN");
assertThat(tier("email")).isEqualTo(FieldAccessTier.HIDDEN);
QContext.setQSession(user("u1", "cs_admin", "person.secured_data_reveal"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.REVEALABLE);
QContext.setQSession(user("u1", "person.secured_data_full"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
}
/*******************************************************************************
** Break glass.
*******************************************************************************/
QRecord breakGlass(String ceiling, String scope) throws Exception
{
return (insert(SecuredDataTables.BREAK_GLASS, new QRecord().withValue("user_id", "u1").withValue("classification_ceiling", ceiling)
.withValue("table_name", scope).withValue("status", "ACTIVE").withValue("started_at", NOW).withValue("expires_at", NOW.plusSeconds(60))));
}
/*******************************************************************************
** Break glass ceiling user and exact expiry.
*******************************************************************************/
@Test
void breakGlassCeilingUserAndExactExpiry() throws Exception
{
QRecord grant = breakGlass("PII", "person");
assertThat(tier("email")).isEqualTo(FieldAccessTier.FULL);
assertThat(tier("ssn")).isEqualTo(FieldAccessTier.MASKED);
QContext.setQSession(user("other"));
assertThat(tier("email")).isEqualTo(FieldAccessTier.MASKED);
QContext.setQSession(user("u1"));
String before = resolver.getAccessFingerprint();
CLOCK.now = NOW.plusSeconds(60);
assertThat(tier("email")).isEqualTo(FieldAccessTier.MASKED);
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(before);
}
/*******************************************************************************
** Fingerprint tracks only relevant state.
*******************************************************************************/
@Test
void fingerprintTracksOnlyRelevantState() throws Exception
{
String initial = resolver.getAccessFingerprint();
assertThat(resolver.getAccessFingerprint()).isEqualTo(initial);
QContext.setQSession(user("u1", "cs_admin", "unrelated.permission"));
assertThat(resolver.getAccessFingerprint()).isEqualTo(initial);
insert("person", new QRecord().withValue("notes", "unrelated"));
assertThat(resolver.getAccessFingerprint()).isEqualTo(initial);
QRecord policy = policy("cs_admin", "PII", "REVEALABLE");
String afterPolicy = resolver.getAccessFingerprint();
assertThat(afterPolicy).isNotEqualTo(initial);
QRecord classification = classification("email", "PHI");
String afterClassification = resolver.getAccessFingerprint();
assertThat(afterClassification).isNotEqualTo(afterPolicy);
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", classification.getValue("id")).withValue("classification", "PII"));
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(afterClassification);
String beforeBreak = resolver.getAccessFingerprint();
QRecord grant = breakGlass("PHI", null);
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(beforeBreak);
update(SecuredDataTables.BREAK_GLASS, new QRecord().withValue("id", grant.getValue("id")).withValue("status", "ENDED"));
String ended = resolver.getAccessFingerprint();
delete(SecuredDataTables.BREAK_GLASS, grant);
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(ended);
String beforeDelete = resolver.getAccessFingerprint();
delete(SecuredDataTables.POLICY, policy);
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(beforeDelete);
String beforeChannel = resolver.getAccessFingerprint();
QContext.setQSession(channelUser("assistant/t1", "cs_admin"));
assertThat(resolver.getAccessFingerprint()).isNotEqualTo(beforeChannel);
}
/*******************************************************************************
** Entropy auto and both overrides.
*******************************************************************************/
@ParameterizedTest
@CsvSource({"BOOLEAN, flag", "DATE, dob", "DATE_TIME, timestamp", "INTEGER, birthYear", "STRING, state", "STRING, region", "STRING, province", "STRING, zip3"})
void entropyAutoAndBothOverrides(String type, String name) throws Exception
{
QFieldMetaData field = new QFieldMetaData(name, QFieldType.valueOf(type)).withMaxLength(255);
person.addField(field);
SecuredDataMetadata.classifyField(field, DataClassification.PII);
assertThat(resolver.resolve(person, field).exactMatchAllowed()).isFalse();
QRecord row = insert(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "person").withValue("field_name", name)
.withValue("classification", "PII").withValue("low_cardinality", false).withValue("mask_style", "FULL"));
assertThat(resolver.resolve(person, field).exactMatchAllowed()).isTrue();
assertThat(resolver.resolve(person, field).maskStyle()).isEqualTo(FieldMaskStyle.FULL);
update(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("id", row.getValue("id")).withValue("low_cardinality", true));
assertThat(resolver.resolve(person, field).exactMatchAllowed()).isFalse();
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("min_entropy_enabled", false));
assertThat(resolver.resolve(person, field).exactMatchAllowed()).isTrue();
}
/*******************************************************************************
** Enum short string and year label auto rules.
*******************************************************************************/
@Test
void enumShortStringAndYearLabelAutoRules()
{
assertThat(SecuredDataResolver.lowCardinality(new QFieldMetaData("kind", QFieldType.STRING).withPossibleValueSourceName("enum"), null)).isTrue();
assertThat(SecuredDataResolver.lowCardinality(new QFieldMetaData("code", QFieldType.STRING).withMaxLength(5), null)).isTrue();
assertThat(SecuredDataResolver.lowCardinality(new QFieldMetaData("y", QFieldType.INTEGER).withLabel("Birth year"), null)).isTrue();
assertThat(SecuredDataResolver.lowCardinality(new QFieldMetaData("code", QFieldType.STRING).withMaxLength(6), null)).isFalse();
}
/*******************************************************************************
** Most restrictive guardrails and hmac.
*******************************************************************************/
@Test
void mostRestrictiveGuardrailsAndHmac() throws Exception
{
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "FULL").withValue("exact_match_per_hour", 0).withValue("min_entropy_enabled", false));
insert(SecuredDataTables.POLICY, policyRow("cs_admin", "PII", "REVEALABLE").withValue("exact_match_per_hour", 7).withValue("miss_alert_consecutive", 3));
var guardrails = resolver.guardrails(person, "email");
assertThat(guardrails.exactMatchPerHour()).isEqualTo(7);
assertThat(guardrails.missAlertConsecutive()).isEqualTo(3);
assertThat(guardrails.minEntropyEnabled()).isTrue();
String hash = resolver.fingerprintSearchValue("CANARY-SEARCH");
assertThat(hash).isEqualTo(resolver.fingerprintSearchValue("CANARY-SEARCH")).doesNotContain("CANARY");
assertThat(hash).isNotEqualTo(resolver.fingerprintSearchValue("different"));
}
/*******************************************************************************
** Reveal grant scope and expiration.
*******************************************************************************/
@Test
void revealGrantScopeAndExpiration() throws Exception
{
insert(SecuredDataTables.REVEAL_GRANT, new QRecord().withValue("user_id", "u1").withValue("table_name", "person")
.withValue("record_id", "7").withValue("field_name", "email").withValue("granted_at", NOW).withValue("expires_at", NOW.plusSeconds(10))
.withValue("access_log_id", 1));
assertThat(resolver.isRevealActive("person", "email", 7)).isTrue();
assertThat(resolver.isRevealActive("person", "ssn", 7)).isFalse();
assertThat(resolver.isRevealActive("person", "email", 8)).isFalse();
CLOCK.now = NOW.plusSeconds(10);
assertThat(resolver.isRevealActive("person", "email", 7)).isFalse();
}
/*******************************************************************************
** A virtual (computed) field is not in getFields, and getField throws for
** it: it resolves through the table classification instead of failing the
** whole read.
*******************************************************************************/
@Test
void virtualFieldsTakeTheTableClassification() throws Exception
{
QVirtualFieldMetaData display = new QVirtualFieldMetaData("displayName", QFieldType.STRING);
person.withVirtualFields(List.of(display));
assertThat(SecuredDataClassifications.effective(person, "displayName")).isNull();
assertThat(resolver.resolve(person, display).tier()).isEqualTo(FieldAccessTier.FULL);
person.withSupplementalMetaData(new AiTableMetaData().withClassification(DataClassification.PII));
assertThat(SecuredDataClassifications.effective(person, "displayName")).isEqualTo(DataClassification.PII);
assertThat(resolver.resolve(person, display).tier()).isEqualTo(FieldAccessTier.MASKED);
}
}
@@ -0,0 +1,281 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.function.Consumer;
import com.kingsrook.qbits.secureddata.metadata.SecuredDataTables;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qbits.secureddata.support.SecuredDataMetadata;
import com.kingsrook.qbits.secureddata.support.SystemWrites;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QUser;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
/*******************************************************************************
** Memory-backed, permission-enforced fixtures shared by direct and bridge tests.
*******************************************************************************/
final class SecuredDataTestSupport
{
static final Instant NOW = Instant.parse("2026-10-13T12:00:00Z");
static final MutableClock CLOCK = new MutableClock();
static final List<SecuredDataRuntime> RUNTIMES = new ArrayList<>();
/*******************************************************************************
** Setup.
*******************************************************************************/
static QInstance setup() throws QException
{
return (setup(config ->
{
}));
}
/*******************************************************************************
** Setup.
*******************************************************************************/
static QInstance setup(Consumer<SecuredDataQBitConfig> tweak) throws QException
{
close();
MemoryRecordStore.fullReset();
CLOCK.now = NOW;
QInstance instance = base();
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
instance.addTable(person("memory"));
SecuredDataQBitConfig config = new SecuredDataQBitConfig().withClock(CLOCK).withAccessLogFlushIntervalMillis(60000);
tweak.accept(config);
new SecuredDataQBitProducer().withQBitConfig(config).produce(instance);
QContext.init(instance, user("u1", "cs_admin"));
RUNTIMES.add(SecuredDataRuntime.current());
new QInstanceValidator().validate(instance);
return (instance);
}
/*******************************************************************************
** Base.
*******************************************************************************/
static QInstance base()
{
return (new QInstance().withEnforcePermissionsInCoreActions(true)
.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS)));
}
/*******************************************************************************
** Person.
*******************************************************************************/
static QTableMetaData person(String backend)
{
QTableMetaData table = new QTableMetaData().withName("person").withBackendName(backend).withPrimaryKeyField("id")
.withLabel("Person").withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS))
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("email", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("ssn", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("notes", QFieldType.TEXT));
SecuredDataMetadata.classifyField(table.getField("email"), DataClassification.PII);
SecuredDataMetadata.classifyField(table.getField("ssn"), DataClassification.PHI);
table.getFields().values().forEach(field -> field.setBackendName(field.getName()));
return (table);
}
/*******************************************************************************
** User.
*******************************************************************************/
static QSession user(String id, String... permissions)
{
List<String> grants = new ArrayList<>(Arrays.asList(permissions));
grants.add("person.read");
return (new QSession().withUser(new QUser().withIdReference(id).withFullName(id)).withPermissions(grants));
}
/*******************************************************************************
** Channel user.
*******************************************************************************/
static QSession channelUser(String channel, String... permissions) throws Exception
{
QSession session = new QSession().withClientChannel(channel);
List<String> grants = new ArrayList<>(Arrays.asList(permissions));
grants.add("person.read");
return (session.withUser(new QUser().withIdReference("u1").withFullName("User One")).withPermissions(grants));
}
/*******************************************************************************
** Policy.
*******************************************************************************/
static QRecord policy(String permission, String classification, String tier) throws QException
{
return (insert(SecuredDataTables.POLICY, policyRow(permission, classification, tier)));
}
/*******************************************************************************
** Policy row.
*******************************************************************************/
static QRecord policyRow(String permission, String classification, String tier)
{
return (new QRecord().withValue("name", permission + " " + classification).withValue("permission_name", permission)
.withValue("classification", classification).withValue("tier", tier).withValue("is_active", true));
}
/*******************************************************************************
** Classification.
*******************************************************************************/
static QRecord classification(String field, String classification) throws QException
{
return (insert(SecuredDataTables.CLASSIFICATION, new QRecord().withValue("table_name", "person")
.withValue("field_name", field).withValue("classification", classification)));
}
/*******************************************************************************
** Insert.
*******************************************************************************/
static QRecord insert(String table, QRecord row) throws QException
{
return (SystemWrites.insertOne(table, row));
}
/*******************************************************************************
** Update.
*******************************************************************************/
static void update(String table, QRecord row) throws QException
{
List<QRecord> records = SystemWrites.call(() -> new UpdateAction().execute(new UpdateInput(table).withRecord(row)).getRecords());
org.assertj.core.api.Assertions.assertThat(records.getFirst().getErrors()).isNullOrEmpty();
}
/*******************************************************************************
** Delete.
*******************************************************************************/
static void delete(String table, QRecord row) throws QException
{
SystemWrites.call(() -> new DeleteAction().execute(new DeleteInput(table).withPrimaryKeys(List.of(row.getValue("id")))));
}
/*******************************************************************************
** Logs.
*******************************************************************************/
static List<QRecord> logs()
{
return (SystemWrites.query(SecuredDataTables.ACCESS_LOG, new QQueryFilter()));
}
/*******************************************************************************
** Close.
*******************************************************************************/
static void close()
{
RUNTIMES.forEach(SecuredDataRuntime::close);
RUNTIMES.clear();
QContext.clear();
}
/*******************************************************************************
** Mutable clock.
*******************************************************************************/
static class MutableClock extends Clock
{
Instant now = NOW;
/*******************************************************************************
** Get zone.
*******************************************************************************/
public ZoneId getZone()
{
return (ZoneOffset.UTC);
}
/*******************************************************************************
** With zone.
*******************************************************************************/
public Clock withZone(ZoneId zone)
{
return (this);
}
/*******************************************************************************
** Instant.
*******************************************************************************/
public Instant instant()
{
return (now);
}
}
}
@@ -0,0 +1,44 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.secureddata;
import java.nio.file.Path;
import java.nio.file.Files;
import com.kingsrook.qqq.middleware.javalin.specs.v1.MiddlewareVersionV1;
import com.kingsrook.qqq.middleware.javalin.tools.UiApiContract;
import static org.assertj.core.api.Assertions.assertThat;
import com.kingsrook.qbits.secureddata.uiapi.SecuredDataExtension;
import org.junit.jupiter.api.Test;
class SecuredDataUiApiContractTest
{
@Test
void committedContractMatches() throws Exception
{
Path path = Path.of("src/main/resources/openapi/qqq-v1/secured-data.yaml");
String generated = UiApiContract.generateExtensionYaml(new MiddlewareVersionV1(), SecuredDataExtension.defineExtension());
if(Boolean.getBoolean("qqq.uiApi.regenerate"))
{
Files.writeString(path, generated);
}
assertThat(UiApiContract.check(generated, Files.readString(path), null, 1, "Regenerate secured-data.yaml")).isEmpty();
}
}