Compare commits

..
Author SHA1 Message Date
james.maesandClaude Opus 5.5 f14a24fdbf feat: in-app WORM Audit Settings and GOVERNANCE lock mode for sealed exports
Owner decisions 2026-10-14:
- worm_audit_settings: one audited, editable row (changeset worm-audit:10,
  seeded insert-only) holding audit retention years, export lock mode
  (GOVERNANCE default, COMPLIANCE), export retention days (30), checkpoint
  cadence, chain interval, settle horizon, late-detail lookback and the
  export/verify schedules; edits are FIELD-audited and therefore chained;
  schedules apply after restart, the rest on the next run (30 s cache);
- exporter and verifier honour the recorded lock mode and retain-until;
  the home shows the mode and warns while GOVERNANCE is in use or the
  export retention is shorter than audit retention;
- README: Settings and Switching sealed exports to COMPLIANCE; DESIGN
  addendum; screenshots of the home and the Settings record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:23:52 -05:00
james.maes 0fb9e8912c Merge pull request #9 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:46:49 -05:00
james.maesandClaude Opus 5.5 626e153841 ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:08:11 -05:00
james.maes 85de1daf79 Merge pull request #8 from QRun-IO/feature/app-home
feat: WORM Audit app home, declared alerts and record screens
2026-10-14 07:09:35 -05:00
james.maes be26a370ac Merge pull request #7 from QRun-IO/feature/sealed-exports
feat: sealed exports to S3 Object Lock and the export verifier
2026-10-14 07:09:21 -05:00
james.maes 3c6e7d57a2 Merge pull request #6 from QRun-IO/feature/chain
feat: append-only audit tables, hash chain, checkpoints and verification
2026-10-14 07:09:09 -05:00
james.maesandClaude Opus 5.5 4b6717ebdd feat: WORM Audit app home, setup warnings, declared alerts and record screens
Phase (c) of docs/DESIGN.md:
- the WORM Audit app and home: chain status, export status, retention,
  recent purges and setup warnings (missing triggers, owner role, signing
  and trusted keys, bucket without Object Lock, stale verification), plus
  a summary card for host dashboards; no audit content is rendered;
- alert types renamed to wormaudit.* (core name pattern) and declared,
  with daily heartbeat monitors;
- full verification holds one page plus one checkpoint range;
- plain field labels, record labels, full-width sections and child lists;
- a test-scope demo server and screenshots (home and records, light and
  dark, 390/1440/1920) with overflow checks;
- DESIGN.md consistency edits (alert names, object_versions,
  s3:GetObjectVersion, snake_case columns).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 22:05:32 -05:00
james.maesandClaude Opus 5.5 9c2766e324 feat: sealed exports to S3 Object Lock and the sealed export verifier
Phase (b) of docs/DESIGN.md:
- daily and on-demand exports of contiguous chain ranges up to the latest
  signed checkpoint: links.jsonl, checkpoints.jsonl and a manifest that
  chains to the previous export, each put through qbit-object-storage with
  COMPLIANCE retention (default 6 years) and checked by head; sealed
  object version ids are recorded;
- refuses and alerts (worm_audit.export.failed) on buckets without Object
  Lock or without versions; heartbeat on scheduled runs;
- Verify sealed export: manifest hashes, lock and retention, the manifest
  chain, link and checkpoint recomputation, overwrite detection against the
  pinned versions, and comparison with the live chain and audit rows;
- MinIO Testcontainers round trip, retention, delete refusal, overwrite and
  live tamper detection, and the unsupported-bucket refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:58:14 -05:00
73 changed files with 6298 additions and 95 deletions

No files matched your search

+1 -1
View File
@@ -1,7 +1,7 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
+2 -2
View File
@@ -29,8 +29,8 @@ Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer im
PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM.
Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set.
Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata;
signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration.
Core RC.5 has no assistant deny API (reflective registration). Alert types use the
`wormaudit.*` namespace and are declared through core alert-type metadata.
## Invariants and naming
+15 -1
View File
@@ -6,9 +6,23 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
### Added
- RC.2 settings: editable, FIELD-audited singleton with insert-only defaults, validation, separate edit permission and a 30-second effective-settings cache. Schedules load at startup.
- GOVERNANCE exports with 30-day default retention; per-export retention verification, administrator-bypass diagnostics, home warnings and documented COMPLIANCE transition.
- New `worm-audit:10` settings migration; historical changesets remain unchanged.
- Phase (c): configurable WORM Audit app home, full-width summary widget and optional host health card; permission-aware navigation and content-free, responsive light/dark summaries.
- Chain backlog/settling counts, verification and export record links, sealed-range gaps/lag, retention eligibility and recent purge/refusal history.
- PostgreSQL guard/ownership/purge-function diagnostics, signing/trust warnings, tri-state Object Lock capability and stale daily-verification warnings.
- Test-scope Next UI demo with synthetic tokenized patient notes, genuine verification failure/recovery, signed checkpoints, sealed memory exports and documented simulated purges.
- Home privacy/permission/empty-state tests, memory/H2 summary tests and Docker-gated PostgreSQL setup-warning acceptance.
- Phase (a): PostgreSQL append-only tables/triggers, safe DDL renderer, retention purge with immutable refusal/anchor records, and runtime grants.
- Canonical audit v1 documents, contiguous hash chain with late-detail coverage, pinned Ed25519 checkpoints and binary Merkle roots.
- Snapshot verification, streamed ETL Review/Result action, scheduled chaining/daily verification, and core QAlerts raise/resolve/heartbeat integration.
- Read-only metadata, WORM Audit table navigation, record range references and child lists; reflective RC.6 assistant deny registration.
- Offline memory/H2 acceptance, real pre-insert PHI canary hook, and PostgreSQL Testcontainers acceptance with CI-strict Docker availability.
- Adoption and operations documentation. Export execution and app-home widgets remain later phases.
- Adoption, app-home, demo and operations documentation.
### Fixed
- Keep runtime signing/trust objects and storage suppliers out of serialized metadata so signed-chain record child lists render successfully.
+2 -2
View File
@@ -29,8 +29,8 @@ Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer im
PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM.
Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set.
Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata;
signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration.
Core RC.5 has no assistant deny API (reflective registration). Alert types use the
`wormaudit.*` namespace and are declared through core alert-type metadata.
## Invariants and naming
+207 -19
View File
@@ -1,10 +1,10 @@
# QBit WORM Audit
Phase (a) provides PostgreSQL append-only enforcement, an asynchronous audit hash chain,
Ed25519 checkpoints, a verification action, and daily verification with core QAlerts signals.
Phases (a)–(c) provide PostgreSQL append-only enforcement, an asynchronous audit hash chain,
Ed25519 checkpoints, live and sealed-export verification actions, and daily jobs with core QAlerts signals.
It hashes exactly the audit values core stored; it does not read application records or
secured-data keys. Sealed exports and the app home are later phases. The export table and
range references are present as integration hooks; no export job or home widgets are installed.
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and configurable
Object Lock (GOVERNANCE for 30 days by default). The WORM Audit app home summarizes chain health, exports, retention and setup.
Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement;
memory/H2 support tamper-evidence tests without database enforcement.
@@ -12,27 +12,31 @@ memory/H2 support tamper-evidence tests without database enforcement.
## Tables and permissions
All sidecar tables use the configured host backend, snake_case fields, BIGINT identifiers,
record sections, and full table-name permissions. They do not audit themselves. The app lists
Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that order.
record sections, and full table-name permissions. Immutable sidecars do not audit themselves.
The editable Settings singleton uses an INTEGER ID and FIELD-level core audit. The app lists
Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges, and Settings in that order.
| Table | Purpose | Permission |
|---|---|---|
| `worm_audit_verification` | Immutable USER/SCHEDULE results | `worm_audit_verification.read` |
| `worm_audit_export` | Reserved sealed-export catalog | `worm_audit_export.read` |
| `worm_audit_export` | Successfully sealed export catalog | `worm_audit_export.read` |
| `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` |
| `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` |
| `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` |
| `worm_audit_settings` | Editable singleton; every edit is audited and chained | `worm_audit_settings.read`; editing needs `worm_audit_settings.edit` **and** `worm_audit.settings` |
`worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action.
Processes are attached to the chain table or scheduled; none is a navigation entry.
Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities
It also authorizes **Verify sealed export** on selected export records. `worm_audit.export`
authorizes **Export now** on the export table. Processes are table/record actions or schedules;
none is a navigation entry.
Insert/update/delete capabilities are absent on immutable sidecar tables; Settings allows updates only; update/delete capabilities
are removed from all four core audit tables. Internal appends use an explicit permission bypass.
Checkpoint, export and verification record screens have a bounded chain-link child list;
range endpoints and export/checkpoint references link to their records.
The table metadata customizer can configure host ESB publications. Quick Search registration
is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables.
Verification inserts use QQQ's record-change listener SPI (as will phase-(b) export inserts),
is host-owned through `QuickSearchQBitConfig.searchableTables`; add the desired tables.
Verification and export inserts use QQQ's record-change listener SPI,
so an installed ESB publisher receives those events when the host configures its destinations
on these two tables. The qbit defines no destination and configures no chain-link publication.
@@ -41,11 +45,18 @@ on these two tables. The qbit defines no destination and configures no chain-lin
| Field | Default / contract |
|---|---|
| `backendName` | Required; core and sidecar audit tables must use this backend |
| `appName` / `appLabel` | `worm_audit` / `WORM Audit` |
| `parentAppName` | Optional host app, for example `admin` |
| `tablePathPrefix` | `/app`; standard QQQ table/record route prefix |
| `sealedStoreSupplier` | Optional host/test storage boundary; defaults to qbit-object-storage |
| `retentionYears` | 6; compiled into the migration, not changed by a runtime flag |
| `signingKey` | Optional `Supplier<SigningKey>`; `SigningKey(keyId, PrivateKey)` |
| `trustedPublicKeys` | `Map<String, PublicKey>`; must contain the signing key's public key |
| `chainIntervalSeconds` | 30 |
| `settleHorizonSeconds` | 60 (zero allowed) |
| `exportBucketAlias` | `worm_audit`; qbit-object-storage BACKUP bucket alias |
| `exportPrefix` | `worm-audit`; slash-separated letters, digits, underscores or hyphens |
| `exportCron` | `0 0 2 * * ?`; 02:00 UTC on the configured scheduler |
| `batchSize` | 1,000 headers per tick / links per insert batch |
| `lateDetailLookbackHours` | 24; positive hours, older omissions checked by full verification |
| `checkpointEveryLinks` | 10,000; maximum links in each checkpoint |
@@ -61,6 +72,102 @@ to verification. It does not itself break the chain. Signing requires a matching
Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints.
Never store the private key in a database row or checked-in configuration.
## Settings
Open **WORM Audit → Settings**, record **1** (`worm_audit_settings`, labelled WORM Audit Settings).
This is a standard editable QQQ table. New records and deletion are disabled; a pre-insert
check and database constraint require ID 1. Every edit uses core FIELD-level auditing, so
the change enters the audit chain. `modify_date` and `modified_by` are stamped by the server.
Keep notes free of sensitive data; notes are audited too.
| Field | Default | Validation / effect |
|---|---|---|
| `audit_retention_years` | 6 | 1–100; home next eligible purge date |
| `export_lock_mode` | GOVERNANCE | GOVERNANCE or COMPLIANCE; new exports |
| `export_retention_days` | 30 | 1–36500; new exports |
| `checkpoint_every_links` | 10000 | Positive integer; new checkpoint ranges |
| `checkpoint_every_minutes` | 60 | Positive integer; checkpoint cadence |
| `chain_interval_seconds` | 30 | Positive integer; **takes effect after restart** |
| `settle_horizon_seconds` | 60 | Nonnegative integer; chaining and verification |
| `late_detail_lookback_hours` | 24 | Positive integer; late-detail polling |
| `export_cron` | `0 0 2 * * ?` | Quartz cron, UTC; **takes effect after restart** |
| `verify_cron` | `0 0 3 * * ?` | Quartz cron, UTC; **takes effect after restart** |
| `notes` | Empty | Administrative notes |
| `modify_date`, `modified_by` | Server supplied | Last edit time and user |
Config values supply defaults for missing rows or null fields; a missing settings table also
falls back to config. Other database failures fail closed. `exportLockMode` and
`exportRetentionDays` are the corresponding config defaults. Runtime reads cache the row
for at most 30 seconds per application instance; local QQQ edits invalidate that cache.
Each run reads effective settings. Startup schedules read the existing row, or config when
absent. Historical checkpoints remain valid after changing their creation cadence.
Changeset `worm-audit:10` creates and seeds the table with `INSERT ... SELECT ... WHERE NOT
EXISTS`; rerunning it never overwrites an administrator change. The renderer seeds from the
host config. Memory hosts and the demo can call `WormAuditSettings.seed(config)` for the same
insert-only behavior during single-threaded startup.
`audit_retention_years` does **not** rewrite the database purge function. The home warns if
it differs from the compiled interval; changing that interval requires a reviewed migration.
The home also warns when export retention is shorter than audit retention and while
GOVERNANCE permits administrator bypass deletion. The runtime role receives SELECT, INSERT
and UPDATE on Settings, with no DELETE or TRUNCATE grant and no append-only triggers.
## Switching sealed exports to COMPLIANCE
Edit `export_lock_mode` to **COMPLIANCE** in Settings. New exports use COMPLIANCE once the
settings cache refreshes (within 30 seconds); existing GOVERNANCE objects keep their mode
unless an administrator explicitly extends their retention and upgrades their mode. The
verifier continues to compare each object with its export row's recorded mode and date, so
an out-of-band mode change is reported as a mismatch. The bucket default retention should
match the chosen mode and retention days, although the exporter always sends explicit values.
The settings change is audited and chained. **COMPLIANCE cannot be undone**: protected
versions cannot be deleted or have retention shortened until expiry, including by administrators.
## App home
Open `/app/worm_audit` (or `/app/admin/worm_audit` with `parentAppName="admin"`).
The full-width `wormAuditHome` HTML widget uses the standard `/qqq/v1/widget` API.
Hosts can add `wormAuditHealthCard` to their own dashboard. Both widgets and the app
require read permission on at least one WORM table, using the host's table permission rules.
The aggregate home shows only identifiers, counts, times, shortened hashes and reasons;
it never renders audit messages or field values. Record links retain the target table's permissions.
- Chain status: live link count, latest link time, settled unchained backlog and settling
count at the configured horizon, latest verification and first open chain break.
A subsequent full scheduled CHAIN PASS clears a reported break; a selected USER PASS does not.
- Export status: current lock mode and retention days, latest sealed range and its recorded retention, link/time lag, coverage holes
between the purge anchor and sealed head, and bucket Object Lock capability. The unsealed
trailing suffix is lag, not a gap. Provider failures show UNKNOWN with a value-free reason.
- Retention: configured years, purge anchor, oldest live audit and its next eligible purge date.
The date indicates retention eligibility; the database purge function still checks the sealed prefix.
- Recent purges: the last ten requests, including refusals.
- Setup warnings: missing signing/trusted keys, unsupported or unknown Object Lock,
daily CHAIN verification never run or older than 26 hours. PostgreSQL also checks every
protected table's row/truncate guards, runtime ownership and the purge function.
Sections and definition-list rows wrap at narrow widths and inherit the Next UI's light/dark colors.
Empty states explicitly distinguish missing history from a passing check.
### Local demo
The test-scope server serves the API and bundled Next UI with a memory backend:
```powershell
./mvnw.cmd -o test-compile exec:java "-Dexec.classpathScope=test" "-Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer" "-Dexec.args=8099"
```
On macOS/Linux use `./mvnw` with the same arguments. Open <http://localhost:8099/app/worm_audit>.
No Docker or credentials are required. This anonymous demo uses synthetic `patient_note`
records audited at FIELD level through the `PhiCanaryTest` HMAC stand-in. It seeds two weeks
of actual chainer ticks, signed checkpoints, daily verifier results and immutable memory exports.
One temporary detail tamper produces a genuine FAIL before restoration and later PASSes.
The PURGED/REFUSED receipts and memory-only prefix removal are explicitly marked demo data;
PostgreSQL enforcement and real provider durability are not simulated production guarantees.
The demo includes a small unsealed suffix and unchained backlog, with no invented export gap.
The final settling count naturally ages into the backlog while the demo is open.
## Adopting in another app
Add `com.kingsrook.qbits:qbit-worm-audit:0.1.0-SNAPSHOT`. Define the host's core audit tables
@@ -95,10 +202,11 @@ String migration = WormAuditDdl.postgres(config);
`sealed_at`, `finished_at`, and `requested_at` respectively (these tables have no `modifyDate`).
The RC.6 assistant deny API is registered reflectively when present; absence in RC.5 is a no-op.
RC.5's alert-type metadata validator disallows underscores, so the qbit sends the specified
`worm_audit.chain.break` signal without declaring incompatible alert-type metadata. `QAlerts`
still dispatches it to the sink (or its logging fallback). Every scheduled verification sends
`worm_audit.verify.daily`; FAIL raises with the first break and verification record reference;
The qbit declares `wormaudit.chain.break` (CRITICAL), `wormaudit.export.failed` (ERROR),
and the daily verification/export heartbeat types through core alert-type metadata.
Daily heartbeat monitors expect a 24-hour interval with 2 hours of grace (overdue at 26 hours).
`QAlerts` dispatches to the installed sink or its logging fallback. Every scheduled verification sends
`wormaudit.verify.daily`; FAIL raises with the first break and verification record reference;
PASS resolves dedup key `chain`.
## Chain and verification format
@@ -119,7 +227,7 @@ PASS resolves dedup key `chain`.
an odd node is promoted unchanged. Ed25519 signs JCS of the seven specified range/trust fields.
- SQL verification uses a repeatable-read snapshot. Full runs check settled header/detail
completeness; selected or filtered runs check their links, predecessor boundaries and
intersecting checkpoints. Full verification currently materializes link metadata in memory.
intersecting checkpoints.
## Retention operations
@@ -143,10 +251,73 @@ rights. The qbit maps those errors to a value-free user-facing message.
Verification starts from the latest successful purge anchor. Checkpoint signatures still verify
after purge, but Merkle recomputation of a purged/partly purged checkpoint requires its sealed
export and is reported as a warning. Sealed exports are phase (b); preserve external evidence
before exercising retention removal in production. Superusers/owners can disable triggers;
export and is reported as a warning. Seal and verify exports before exercising retention
removal in production. Superusers/owners can disable triggers;
protect signing keys and keep public-key pins outside the database.
## Sealed exports
Install qbit-object-storage and configure `exportBucketAlias` (default `worm_audit`) as a
`BucketRole.BACKUP` bucket. **Object Lock must be enabled when the bucket is created**.
The exporter requires the bucket's `OBJECT_LOCK` capability to be `SUPPORTED`; run the
object-storage read-only capability probe during setup. Unknown, partial and unsupported
capabilities all refuse sealing. Every object receives the effective `export_lock_mode` and retention until sealing time plus
`export_retention_days` (GOVERNANCE and 30 days by default), checked by HEAD after PUT. The
export row records both values. Verification checks that historical mode and retain-until,
regardless of subsequent settings changes. Missing GOVERNANCE objects are reported as breaks
with a possible administrator bypass-removal explanation.
The runtime IAM policy needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
`s3:GetObjectRetention`, `s3:PutObjectRetention`, and
`s3:GetBucketObjectLockConfiguration`. Grant no delete or governance-bypass access.
The storage bucket's upload size limit must accommodate the largest daily JSONL file.
Retention protects object versions; protect the storage volumes and maintain off-site copies too.
At 02:00 UTC, or through **Export now**, the exporter takes the chainer's writer lock and
seals links after the previous export through the latest checkpoint. The first export starts
after the purge anchor, or at genesis. No new checkpoint produces an explicit "Nothing to
seal" Result. If retention has removed an unexported gap, or a range cannot be proven by
complete signed checkpoints, sealing refuses rather than recording incomplete evidence.
Other database backends require a single writer JVM, as in phase (a).
Keys use `<exportPrefix>/<yyyy>/<MM>/<dd>/<from>-<to>/` with a UTC date:
- `links.jsonl`: JCS lines containing `link` identifiers/hashes and the embedded canonical
`content` document exactly as hashed.
- `checkpoints.jsonl`: JCS checkpoint envelopes including signatures and public keys.
- `manifest.json`: JCS `worm-audit-export/1` with `fromLinkId`, `toLinkId`, `linkCount`,
`checkpointCount`, `files` (filename to SHA-256), `headHash`, `prevManifestSha256`,
`qbitVersion`, and `createdAt`.
Files are streamed through temporary disk storage with 1,000-link pages and batched content
fetches, then removed. Only ids, hashes and object metadata enter the export catalog.
The `object_versions` text column pins each upload's returned version id in a JCS JSON
object keyed by `checkpoints.jsonl`, `links.jsonl`, and `manifest.json`. Sealing refuses
missing version ids or a different version in the retention HEAD response, leaving no
catalog row and raising the export failure alert.
Partial upload failures leave no catalog row; retained orphan objects may remain. Retrying
can create new locked versions under the same keys. Failures raise `wormaudit.export.failed`
with dedup key `export`; success resolves it. Every scheduled run emits
`wormaudit.export.daily`.
**Verify sealed export** accepts one or several selected records. It downloads and checks
the object hashes, retention, manifest chain, link hashes, pinned checkpoint signatures and
Merkle roots, then compares the sealed content and link metadata against a live database
snapshot. Differences name exact link/audit ids without printing audit values. Each export
gets its own `SEALED_EXPORT` verification row. Purged prefixes verify from the sealed copy
alone and receive an INFO line. Verification does not write restored audit rows.
Verification compares every current HEAD version to the catalog and reports an ERROR for
any overwrite, even when the replacement bytes are identical. It then reads the recorded
sealed versions, so retained evidence remains verifiable after an overwrite; the overall
verification still fails because the overwrite is a break. Missing catalog versions fail
closed. RC.2's `StorageBucket.get` has no version parameter, so the adapter uses the public
`ObjectStorageRuntime.connection(...).client()` accessor for an explicit S3 version GET,
reusing its connection, key normalization and rate limiter and checking the response version.
No object-storage change is required; this adapter depends on RC.2's public internal runtime
accessors. The storage credentials need `s3:GetObjectVersion` in addition to the existing
upload, HEAD and listing permissions.
## Build and acceptance
```powershell
@@ -170,7 +341,21 @@ $env:CI = 'true'
./mvnw.cmd -o verify
```
POSIX equivalents: `CI=true ./mvnw -o -Dtest=PostgresAcceptanceTest test` and
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks GOVERNANCE
with 30-day retention and a COMPLIANCE export, denied version deletion (without bypass for GOVERNANCE), overwrite version preservation, export verification,
exact live-tamper findings through the record action, and refusal of a bucket without lock:
```powershell
$env:CI = 'true'
./mvnw.cmd -o '-Dtest=MinioExportAcceptanceTest' test
```
The default MinIO image is pinned to the same Chainguard digest as object-storage's tests.
Override it with `-Dqqq.test.minioImage=<image>` when necessary. Images must already be
available for a completely network-free Docker run.
POSIX equivalents: `CI=true ./mvnw -o -Dtest=MinioExportAcceptanceTest test`,
`CI=true ./mvnw -o -Dtest=PostgresAcceptanceTest test` and
`CI=true ./mvnw -o verify`. No commits, publishing or deployment are part of these commands.
## License
@@ -194,6 +379,9 @@ that window. Initial header details are always chained, including overflow
chunks. Checkpoints contain at most `checkpointEveryLinks` links; multiple full
ranges can be signed per tick. Verification uses 1,000-link keyset pages.
Full verification holds at most one 1,000-link page and one checkpoint range of hashes.
SQL completeness uses bounded offender queries; other persistent backends use repeated bounded
scans. Only the memory backend uses a full coverage index.
Full verification retains at most 1,000 exact break diagnostics and 100 warnings,
while persisting the total break count. SQL completeness counts all offenders
and reports the first 100 header ids and first 100 detail ids. This keeps a badly
+48 -9
View File
@@ -104,7 +104,9 @@ runtime role owns the audit tables (checked through `pg_tables.tableowner = curr
The qbit ships `src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql`,
a Liquibase formatted-SQL template with placeholders for the four core audit table names, the
app role and the retention interval, plus a renderer (`WormAuditDdl.postgres(config)`) and the
rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Hosts copy the rendered
rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Core audit physical
columns must use snake_case (for example `audit_table_id`, `audit_id`, `new_value`); hosts map
QQQ field names to those columns. Hosts copy the rendered
file into their changelog (kof22: `changesets/features/YYYY-MM-DD-worm-audit.sql`). It creates the
qbit tables, the functions, the triggers, the purger role (`CREATE ROLE ... NOLOGIN` when it is
missing) and the grants. It is idempotent (`CREATE OR REPLACE`, `DROP TRIGGER IF EXISTS`).
@@ -217,15 +219,19 @@ standard streamed ETL:
does not verify".
**Daily verification**: a scheduled job runs the same verifier at 03:00. A failure raises
`QAlerts.raise(new AlertSignal("worm_audit.chain.break").withDedupKey("chain")...)` with the first
`QAlerts.raise(new AlertSignal("wormaudit.chain.break").withDedupKey("chain")...)` with the first
break in the title and the verification record as the record ref; a later pass resolves it. Every
run sends `QAlerts.heartbeat("worm_audit.verify.daily")`. `QAlerts` is the core producer SPI
run sends `QAlerts.heartbeat("wormaudit.verify.daily")`. `QAlerts` is the core producer SPI
(QQQ 4.1.0-RC.5), so the qbit needs no dependency on qbit-alerts. With no alerts implementation
installed the alerts go to the log.
## 6. Sealed exports
### 6.1 Export job
**Default superseded by Owner decisions (2026-10-14) below:** lock mode and export retention
now come from Settings, defaulting to GOVERNANCE and 30 days. The original decision text is
retained for history.
Daily at 02:00 (configurable), and by the table action "Export now" on `worm_audit_export`:
1. Range: links after the last export's `to_link_id`, up to the latest checkpoint. Exports are
contiguous by construction. The app home flags a gap (a range not covered) and lag.
@@ -236,14 +242,17 @@ Daily at 02:00 (configurable), and by the table action "Export now" on `worm_aud
the previous export's manifest SHA-256 (the exports form their own chain), qbit version.
3. Each object is put through qbit-object-storage (`SystemStorage`, bucket alias from config,
default `worm_audit`) with `LockMode.COMPLIANCE` and `retainUntil = now + retentionYears`. The
export refuses to run, and alerts `worm_audit.export.failed`, when the bucket does not report
export refuses to run, and alerts `wormaudit.export.failed`, when the bucket does not report
the `OBJECT_LOCK` capability or the put does not echo the lock.
4. On success it inserts one `worm_audit_export` row (insert-only; a failed export leaves no row,
and its alert carries the reason).
and its alert carries the reason). Every scheduled run sends `wormaudit.export.daily`.
### 6.2 Restore and verify a sealed export
"Verify sealed export", a record action on `worm_audit_export` (standard ETL), downloads the
objects. It checks the manifest hashes, the lock mode and retain-until of each object, the link
exact object versions pinned in the catalog's `object_versions` map. A new version under an
existing key is an overwrite even when bytes and retention match: verification checks the latest
version against the pin and rejects replacements, then reads and checks the pinned version.
It checks the manifest hashes, the lock mode and retain-until of each object, the link
hashes recomputed from the exported content, the checkpoint signatures, and the export's link to
the previous manifest. Then it compares each exported link and content document with the live
chain and the live audit rows. Result lines name each difference ("link 88,412: live content
@@ -265,7 +274,7 @@ events integrations care about); chain links are high-volume and publish none.
- **`worm_audit_checkpoint`** [Checkpoint]: `id`, `from_link_id`, `to_link_id`, `link_count`,
`head_hash`, `merkle_root`, `algorithm`, `key_id`, `public_key`, `signature`, `created_at`.
- **`worm_audit_export`** [Sealed Export]: `id`, `from_link_id`, `to_link_id`,
`from_checkpoint_id`, `to_checkpoint_id`, `bucket_alias`, `object_prefix`, `manifest_key`,
`from_checkpoint_id`, `to_checkpoint_id`, `bucket_alias`, `object_prefix`, `object_versions`, `manifest_key`,
`manifest_sha256`, `prev_manifest_sha256`, `object_count`, `byte_count`, `lock_mode`,
`retain_until`, `sealed_at`.
- **`worm_audit_verification`** [Verification]: `id`, `kind` (CHAIN, SEALED_EXPORT), `trigger`
@@ -341,7 +350,7 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
qbit's tests until the core hook ships), the audit header, the details (old, new, message),
the canonical content and the export JSONL contain neither the canary nor any substring of it
longer than 4 characters.
6. **Alerts**: a break raises `worm_audit.chain.break` through `QAlerts` (captured by a test
6. **Alerts**: a break raises `wormaudit.chain.break` through `QAlerts` (captured by a test
sink); a later pass resolves it.
7. Coverage gates: 70% instructions, 90% classes.
@@ -367,7 +376,7 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
- **Bucket:** `kof22-worm-audit` on the kof22 MinIO, created by `minio-bucket-init` with
`mc mb --with-lock --ignore-existing` and `mc retention set --default COMPLIANCE 6y`. Object
Lock can only be enabled at bucket creation.
- **Policy:** `kof22-app` gets `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
- **Policy:** `kof22-app` gets `s3:PutObject`, `s3:GetObject`, `s3:GetObjectVersion`, `s3:ListBucket`,
`s3:GetObjectRetention`, `s3:PutObjectRetention` (needed to set the lock on each put),
`s3:GetBucketObjectLockConfiguration` on that bucket. No delete, and no
`s3:BypassGovernanceRetention` (irrelevant under COMPLIANCE anyway).
@@ -381,3 +390,33 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
These are owner-run (Praetor and cluster-gitops are applied by the owner). The qbit only
documents them.
## Owner decisions (2026-10-14)
The 2026-10-12 decisions above remain the historical design. These decisions supersede the
COMPLIANCE/six-year export default in §6.1 and the corresponding default assumptions in
§§2, 4.5, 9, 11, 12 and the §14 infrastructure proposal.
- Settings live in the application as the standard QQQ `worm_audit_settings` table, record
ID 1, labelled WORM Audit Settings and navigated as Settings after Purges. It allows edits,
has no append-only triggers, and uses FIELD-level core audit so every change is chained.
Reading needs `worm_audit_settings.read`; editing additionally needs
`worm_audit_settings.edit` and `worm_audit.settings`. New IDs and deletion are refused.
- Defaults are seeded insert-only in new changeset `worm-audit:10`; existing changesets are
unchanged. An administrator's edits survive subsequent seeding. Config supplies defaults
when a row/field/table is absent. Runtime reads cache at most 30 seconds, scoped to the
QInstance. Chain interval and UTC Quartz export/verify schedules take effect after restart.
- kof22 starts with **GOVERNANCE, 30 days** for sealed exports. An administrator with
`s3:BypassGovernanceRetention` can remove those copies. This is deliberately reversible;
it does not provide COMPLIANCE's irreversible retention. Runtime credentials still receive
no delete or bypass permission. Bucket defaults should match the selected settings.
- Changing Settings to COMPLIANCE affects new exports. Existing GOVERNANCE versions retain
their original mode unless an administrator extends/upgrades them. Verification checks
the catalog's recorded mode and date, so external mode changes are reported as mismatches.
COMPLIANCE cannot be undone. The settings change itself is audited and chained.
- Audit retention defaults to six years independently of export retention. The home warns
about a shorter export horizon and a mismatch with the purge function's compiled interval.
Updating the setting does not modify the purge function; that still requires a migration.
The README Settings table specifies all fields, defaults, bounds and timing. Signing keys,
public-key pins, bucket alias, prefix and backend remain host configuration, never settings.
+28
View File
@@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS public.worm_audit_export (
to_checkpoint_id bigint,
bucket_alias text,
object_prefix text,
object_versions text,
manifest_key text,
manifest_sha256 text,
prev_manifest_sha256 text,
@@ -323,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
-- changeset worm-audit:10
-- Editable singleton: deliberately excluded from append-only triggers.
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
id integer PRIMARY KEY CHECK (id = 1),
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
notes text,
modify_date timestamptz NOT NULL DEFAULT now(),
modified_by text NOT NULL DEFAULT current_user
);
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM "app";
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO "app";
INSERT INTO public.worm_audit_settings
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
export_cron, verify_cron)
SELECT 1, 6, 'GOVERNANCE', 30, 10000, 60, 30, 60, 24, '0 0 2 * * ?', '0 0 3 * * ?'
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
+80
View File
@@ -0,0 +1,80 @@
# WORM Audit screenshots
## RC.2 settings captures (2026-10-14)
Home (GOVERNANCE, 30-day exports) and the WORM Audit Settings record at 390 x 844 and 1440 x 1000,
light and dark, captured with Playwright against `WormAuditDemoServer` on port 8099:
`worm-audit-home-{390,1440}-{light,dark}.jpg` (replaced) and
`worm-audit-settings-{390,1440}-{light,dark}.jpg` (new). Overflow check on all eight: no page
horizontal overflow, no inner horizontal scrollers, no double scrollbars.
## Earlier captures
Refreshed after record-screen polish **2026-10-13 (America/Chicago)** from the rebuilt test-scope demo using
packaged Next UI 1.0.0-RC.14 and connected Edge through the browser screenshot
and viewport tools. All files are JPEG viewport captures, below 300 KB each.
```powershell
./mvnw.cmd -o test-compile exec:java "-Dexec.classpathScope=test" "-Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer" "-Dexec.args=8105"
```
Open <http://localhost:8105/app/worm_audit/>. Port 8105 isolates this rebuilt instance from earlier demos. The seed creates genuine verification
results and exports over synthetic data. Records captured: FAIL verification #8,
PASS verification #15, sealed export #15, and chain link #329.
Required matrix: home at **390, 1440, 1920 px**, and all four records at **390,
1440 px**, each in **light and dark**. Heights are 844, 1000 and 1080 px respectively.
The existing supplementary 390 px light purge-history capture was refreshed too.
Theme changes used Preferences and were restored to light; the temporary viewport override was reset afterward.
Every image was visually inspected. At each capture, the live DOM was checked after responsive layout settled for
page-level horizontal overflow (`document.documentElement.scrollWidth > innerWidth`)
and all rendered inner horizontal scrollers (`scrollWidth > clientWidth + 1` with
computed `overflow-x` of `auto` or `scroll`). The scan included content below the
viewport, including loaded child lists. Mobile verification accordions were also explicitly expanded and checked after their child rows loaded. There were **zero** page overflow failures
and **zero** inner horizontal scrollers. Each home/verification/export view had
one vertical scroll container (main); chain records had none. No double scrollbars
were found. Long hashes, object versions and failure diagnostics wrap on mobile.
No CSS or DOM overrides were injected for the captures.
All five tables have explicit field labels and meaningful record-label formats.
Verification uses full-width Result, Scope, Run, and Chain links sections. Every
child-list widget and section declares 12 grid columns. Kind and outcome use
readable possible values; child queries translate those values too. Hashes stay
complete because RC.5 has no separate list-only field display format. Numeric
chain-reference labels keep the export title concise.
Core RC.5 shares the table label with its sidebar tree node. Existing plural
navigation labels are retained, with singular record titles and record-section
labels. Separate singular table labels and plural sidebar labels require core
support; no application-wide metadata customizer was replaced to work around it.
`./mvnw.cmd -o verify` passed: 79 tests, no failures or errors, and 9
Docker-dependent skips. Checkstyle and JaCoCo coverage gates passed. Metadata
tests cover every field label, record-label formats and fields, verification
section coverage, and 12-column sections and child widgets.
| Screenshot | Viewport | Size | Page horizontal overflow | Inner horizontal scrollers | Double scrollbars |
|---|---|---|---|---|---|
| [worm-audit-chain-link-1440-dark](worm-audit-chain-link-1440-dark.jpg) | 1440 x 1000 | 41.1 KB | None | None | None |
| [worm-audit-chain-link-1440-light](worm-audit-chain-link-1440-light.jpg) | 1440 x 1000 | 41.4 KB | None | None | None |
| [worm-audit-chain-link-390-dark](worm-audit-chain-link-390-dark.jpg) | 390 x 844 | 21.8 KB | None | None | None |
| [worm-audit-chain-link-390-light](worm-audit-chain-link-390-light.jpg) | 390 x 844 | 21.8 KB | None | None | None |
| [worm-audit-home-1440-dark](worm-audit-home-1440-dark.jpg) | 1440 x 1000 | 73.6 KB | None | None | None |
| [worm-audit-home-1440-light](worm-audit-home-1440-light.jpg) | 1440 x 1000 | 72.4 KB | None | None | None |
| [worm-audit-home-1920-dark](worm-audit-home-1920-dark.jpg) | 1920 x 1080 | 84.7 KB | None | None | None |
| [worm-audit-home-1920-light](worm-audit-home-1920-light.jpg) | 1920 x 1080 | 83.6 KB | None | None | None |
| [worm-audit-home-390-dark](worm-audit-home-390-dark.jpg) | 390 x 844 | 30.0 KB | None | None | None |
| [worm-audit-home-390-light](worm-audit-home-390-light.jpg) | 390 x 844 | 28.8 KB | None | None | None |
| [worm-audit-home-390-purges-light](worm-audit-home-390-purges-light.jpg) | 390 x 844 | 29.5 KB | None | None | None |
| [worm-audit-sealed-export-1440-dark](worm-audit-sealed-export-1440-dark.jpg) | 1440 x 1000 | 117.6 KB | None | None | None |
| [worm-audit-sealed-export-1440-light](worm-audit-sealed-export-1440-light.jpg) | 1440 x 1000 | 118.4 KB | None | None | None |
| [worm-audit-sealed-export-390-dark](worm-audit-sealed-export-390-dark.jpg) | 390 x 844 | 38.4 KB | None | None | None |
| [worm-audit-sealed-export-390-light](worm-audit-sealed-export-390-light.jpg) | 390 x 844 | 37.3 KB | None | None | None |
| [worm-audit-verification-fail-1440-dark](worm-audit-verification-fail-1440-dark.jpg) | 1440 x 1000 | 53.6 KB | None | None | None |
| [worm-audit-verification-fail-1440-light](worm-audit-verification-fail-1440-light.jpg) | 1440 x 1000 | 54.1 KB | None | None | None |
| [worm-audit-verification-fail-390-dark](worm-audit-verification-fail-390-dark.jpg) | 390 x 844 | 28.1 KB | None | None | None |
| [worm-audit-verification-fail-390-light](worm-audit-verification-fail-390-light.jpg) | 390 x 844 | 27.8 KB | None | None | None |
| [worm-audit-verification-pass-1440-dark](worm-audit-verification-pass-1440-dark.jpg) | 1440 x 1000 | 49.0 KB | None | None | None |
| [worm-audit-verification-pass-1440-light](worm-audit-verification-pass-1440-light.jpg) | 1440 x 1000 | 49.7 KB | None | None | None |
| [worm-audit-verification-pass-390-dark](worm-audit-verification-pass-390-dark.jpg) | 390 x 844 | 23.5 KB | None | None | None |
| [worm-audit-verification-pass-390-light](worm-audit-verification-pass-390-light.jpg) | 390 x 844 | 23.4 KB | None | None | None |
Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 22 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

+5
View File
@@ -83,6 +83,11 @@
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-module-rdbms</artifactId>
</dependency>
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-object-storage</artifactId>
<version>0.1.0-RC.2</version>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-middleware-javalin</artifactId>
@@ -25,6 +25,7 @@ import java.security.PrivateKey;
import java.security.PublicKey;
import java.util.Map;
import java.util.function.Supplier;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -44,6 +45,14 @@ public class WormAuditQBitConfig implements QBitConfig
{
}
private String exportLockMode = "GOVERNANCE";
private int exportRetentionDays = 30;
private final WormAuditSettings settings = new WormAuditSettings(this);
private String appName = "worm_audit";
private String appLabel = "WORM Audit";
private String parentAppName;
private String tablePathPrefix = "/app";
private Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> sealedStoreSupplier;
private String backendName = null;
private int retentionYears = 6;
private Supplier<SigningKey> signingKey = null;
@@ -55,12 +64,168 @@ public class WormAuditQBitConfig implements QBitConfig
private int checkpointEveryLinks = 10000;
private int checkpointEveryMinutes = 60;
private String verifyCron = "0 0 3 * * ?";
private String exportBucketAlias = "worm_audit";
private String exportPrefix = "worm-audit";
private String exportCron = "0 0 2 * * ?";
private String schedulerName = "worm_audit";
private Map<String, String> coreAuditTableNames = Map.of();
private String appRole = "app";
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
/*******************************************************************************
** Read exportLockMode.
*******************************************************************************/
public String getExportLockMode()
{
return (exportLockMode);
}
/*******************************************************************************
** Configure exportLockMode.
*******************************************************************************/
public WormAuditQBitConfig withExportLockMode(String exportLockMode)
{
this.exportLockMode = exportLockMode;
return (this);
}
/*******************************************************************************
** Read exportRetentionDays.
*******************************************************************************/
public int getExportRetentionDays()
{
return (exportRetentionDays);
}
/*******************************************************************************
** Configure exportRetentionDays.
*******************************************************************************/
public WormAuditQBitConfig withExportRetentionDays(int exportRetentionDays)
{
this.exportRetentionDays = exportRetentionDays;
return (this);
}
/*******************************************************************************
** Read effective settings through the instance-scoped 30-second cache.
*******************************************************************************/
@JsonIgnore
public WormAuditSettings.Values effectiveSettings()
{
return (settings.read());
}
/*******************************************************************************
** Discard this instance's cached row after a successful local settings write.
*******************************************************************************/
public void invalidateSettings()
{
settings.invalidate();
}
/*******************************************************************************
** Read the app name.
*******************************************************************************/
public String getAppName()
{
return (appName);
}
/*******************************************************************************
** Configure the app name.
*******************************************************************************/
public WormAuditQBitConfig withAppName(String appName)
{
this.appName = appName;
return (this);
}
/*******************************************************************************
** Read the app label.
*******************************************************************************/
public String getAppLabel()
{
return (appLabel);
}
/*******************************************************************************
** Configure the app label.
*******************************************************************************/
public WormAuditQBitConfig withAppLabel(String appLabel)
{
this.appLabel = appLabel;
return (this);
}
/*******************************************************************************
** Read the optional parent app.
*******************************************************************************/
public String getParentAppName()
{
return (parentAppName);
}
/*******************************************************************************
** Configure the optional parent app.
*******************************************************************************/
public WormAuditQBitConfig withParentAppName(String parentAppName)
{
this.parentAppName = parentAppName;
return (this);
}
/*******************************************************************************
** Read the table route prefix.
*******************************************************************************/
public String getTablePathPrefix()
{
return (tablePathPrefix);
}
/*******************************************************************************
** Configure the table route prefix.
*******************************************************************************/
public WormAuditQBitConfig withTablePathPrefix(String tablePathPrefix)
{
this.tablePathPrefix = tablePathPrefix;
return (this);
}
/*******************************************************************************
** Read the optional storage boundary (defaults to qbit-object-storage).
*******************************************************************************/
@JsonIgnore
public Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> getSealedStoreSupplier()
{
return (sealedStoreSupplier);
}
/*******************************************************************************
** Configure the optional storage boundary (defaults to qbit-object-storage).
*******************************************************************************/
public WormAuditQBitConfig withSealedStoreSupplier(Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> sealedStoreSupplier)
{
this.sealedStoreSupplier = sealedStoreSupplier;
return (this);
}
/*******************************************************************************
** Read backendName.
*******************************************************************************/
@@ -102,6 +267,7 @@ public class WormAuditQBitConfig implements QBitConfig
/*******************************************************************************
** Read signingKey.
*******************************************************************************/
@JsonIgnore
public Supplier<SigningKey> getSigningKey()
{
return (signingKey);
@@ -121,6 +287,7 @@ public class WormAuditQBitConfig implements QBitConfig
/*******************************************************************************
** Read trustedPublicKeys.
*******************************************************************************/
@JsonIgnore
public Map<String, PublicKey> getTrustedPublicKeys()
{
return (trustedPublicKeys);
@@ -260,6 +427,63 @@ public class WormAuditQBitConfig implements QBitConfig
}
/*******************************************************************************
** Read the Object Lock bucket alias.
*******************************************************************************/
public String getExportBucketAlias()
{
return (exportBucketAlias);
}
/*******************************************************************************
** Configure the Object Lock bucket alias.
*******************************************************************************/
public WormAuditQBitConfig withExportBucketAlias(String exportBucketAlias)
{
this.exportBucketAlias = exportBucketAlias;
return (this);
}
/*******************************************************************************
** Read the export key prefix.
*******************************************************************************/
public String getExportPrefix()
{
return (exportPrefix);
}
/*******************************************************************************
** Configure the export key prefix.
*******************************************************************************/
public WormAuditQBitConfig withExportPrefix(String exportPrefix)
{
this.exportPrefix = exportPrefix;
return (this);
}
/*******************************************************************************
** Read the UTC export schedule.
*******************************************************************************/
public String getExportCron()
{
return (exportCron);
}
/*******************************************************************************
** Configure the UTC export schedule.
*******************************************************************************/
public WormAuditQBitConfig withExportCron(String exportCron)
{
this.exportCron = exportCron;
return (this);
}
/*******************************************************************************
** Configure verifyCron.
*******************************************************************************/
@@ -352,6 +576,18 @@ public class WormAuditQBitConfig implements QBitConfig
*******************************************************************************/
public void validate()
{
if(appName == null || !appName.matches("[A-Za-z0-9_-]+") || appLabel == null || appLabel.isBlank()
|| (parentAppName != null && !parentAppName.matches("[A-Za-z0-9_-]+"))
|| tablePathPrefix == null || !tablePathPrefix.matches("/[A-Za-z0-9_/-]*") || tablePathPrefix.endsWith("/"))
{
throw (new IllegalArgumentException("App names and a local table path prefix are required"));
}
if(exportBucketAlias == null || exportBucketAlias.isBlank() || exportCron == null || exportCron.isBlank()
|| exportPrefix == null || !exportPrefix.matches("[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*"))
{
throw (new IllegalArgumentException("Export bucket, prefix and schedule must be configured"));
}
WormAuditSettings.validate(WormAuditSettings.defaults(this));
if(backendName == null || backendName.isBlank() || retentionYears < 1 || chainIntervalSeconds < 1
|| lateDetailLookbackHours < 1 || settleHorizonSeconds < 0 || batchSize < 1 || checkpointEveryLinks < 1 || checkpointEveryMinutes < 1)
{
@@ -21,19 +21,23 @@
package com.kingsrook.qbits.wormaudit;
import java.util.List;
import com.kingsrook.qbits.wormaudit.app.WormAuditApp;
import com.kingsrook.qbits.wormaudit.ddl.WormAuditDdl;
import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
import com.kingsrook.qbits.wormaudit.model.LinkRangeWidget;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
import com.kingsrook.qqq.backend.core.alerts.AlertDefaultMonitor;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
@@ -122,7 +126,7 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
/*******************************************************************************
** Register phase (a) tables, processes and schedules on the host backend.
** Register audit tables, verification/export actions and schedules.
*******************************************************************************/
@Override
public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException
@@ -149,6 +153,14 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
{
table = qBitConfig.getTableMetaDataCustomizer().customizeMetaData(qInstance, table);
}
for(var field : table.getFields().values())
{
if(field.getInlinePossibleValueSource() != null)
{
output.add(field.getInlinePossibleValueSource());
field.withPossibleValueSourceName(field.getInlinePossibleValueSource().getName()).withInlinePossibleValueSource(null);
}
}
guard.customizeMetaData(qInstance, table);
table.setSourceQBitName(qBit.getName());
output.add(table);
@@ -159,37 +171,88 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
}
for(String name : WormAuditTables.ALL)
{
output.add(QPossibleValueSource.newForTable(name));
QPossibleValueSource source = QPossibleValueSource.newForTable(name);
if(WormAuditTables.CHAIN.equals(name))
{
source.withValueFormat("%s").withValueFields(java.util.List.of("id"));
}
output.add(source);
}
for(String name : java.util.List.of(WormAuditTables.CHECKPOINT, WormAuditTables.EXPORT, WormAuditTables.VERIFICATION))
{
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList")
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList").withGridColumns(12)
.withCodeReference(new QCodeReference(LinkRangeWidget.class)).withDefaultValue("parentTable", name));
}
for(var process : WormAuditProcesses.processes(qBitConfig))
List<QTableMetaData> producedTables = output.getEach(QTableMetaData.class);
output.addSelfToInstance(qInstance);
output = new MetaDataProducerMultiOutput();
var previousInstance = QContext.getQInstance();
var previousSession = QContext.getQSession();
var previousTransaction = QContext.getQBackendTransaction();
try
{
process.setSourceQBitName(qBit.getName());
output.add(process);
// Production is still assembling the host metadata. Do not validate it mid-build.
QContext.setQInstance(qInstance);
QContext.setQSession(new com.kingsrook.qqq.backend.core.model.session.QSession());
QContext.setTransaction(null);
addProcesses(output, qBit);
}
finally
{
QContext.setQInstance(previousInstance);
QContext.setQSession(previousSession);
QContext.setTransaction(previousTransaction);
}
if(qInstance.getSchedulers() == null || !qInstance.getSchedulers().containsKey(qBitConfig.getSchedulerName()))
{
qInstance.addScheduler(new QuartzSchedulerMetaData().withName(qBitConfig.getSchedulerName()));
}
QAppMetaData app = new QAppMetaData().withName("worm_audit").withLabel("WORM Audit").withIcon(new QIcon("policy"));
for(String name : WormAuditTables.NAV)
output.add(WormAuditApp.app(qBitConfig, producedTables));
for(QWidgetMetaData widget : WormAuditApp.widgets())
{
QTableMetaData table = output.getEach(QTableMetaData.class).stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table);
output.add(widget);
}
output.add(new QAlertTypeMetaData().withName("wormaudit.chain.break").withLabel("Audit chain break")
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.CRITICAL)
.withDescription("Audit chain verification found a break in content, continuity, coverage or checkpoint trust.")
.withSubjectTableName(WormAuditTables.VERIFICATION).withVisibilityPermission("worm_audit_verification.read"));
output.add(new QAlertTypeMetaData().withName("wormaudit.export.failed").withLabel("Sealed audit export failed")
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.ERROR)
.withDescription("The audit export could not be sealed with verified retention and object versions.")
.withSubjectTableName(WormAuditTables.EXPORT).withVisibilityPermission("worm_audit_export.read"));
for(String job : List.of("verify", "export"))
{
String name = "wormaudit." + job + ".daily";
output.add(new QAlertTypeMetaData().withName(name).withLabel("Daily audit " + job + " overdue")
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.ERROR)
.withDescription("The daily audit " + job + " job has not sent a heartbeat within 26 hours.")
.withVisibilityPermission(job.equals("verify") ? "worm_audit_verification.read" : "worm_audit_export.read")
.withDefaultMonitor(AlertDefaultMonitor.heartbeat(name, name, 24 * 3600, 2 * 3600)));
}
output.add(app);
// RC.5 rejects underscores in declared alert type names. QAlerts still
// dispatches the specified worm_audit.chain.break signal to installed sinks.
output.addSelfToInstance(qInstance);
denyAssistant(qInstance);
return (new MetaDataProducerMultiOutput());
}
/*******************************************************************************
** Build schedules from settings available at producer startup.
*******************************************************************************/
private void addProcesses(MetaDataProducerMultiOutput output, QBitMetaData qBit)
{
for(var process : WormAuditProcesses.processes(qBitConfig))
{
process.setSourceQBitName(qBit.getName());
output.add(process);
}
for(var process : ExportProcesses.processes(qBitConfig))
{
process.setSourceQBitName(qBit.getName());
output.add(process);
}
}
/*******************************************************************************
** RC.5 compatibility: replace with a direct call when the qbit builds on RC.6 (qqq#1141, #1163).
*******************************************************************************/
@@ -0,0 +1,246 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.sql.SQLException;
import java.time.Clock;
import java.time.Instant;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import org.quartz.CronExpression;
/*******************************************************************************
** Singleton settings merged over host defaults. Only absent rows/tables fall
** back; database outages and invalid persisted settings fail closed.
*******************************************************************************/
public class WormAuditSettings
{
private final WormAuditQBitConfig config;
private final Clock clock;
private QInstance instance;
private QRecord cached;
private Instant expires = Instant.MIN;
/*******************************************************************************
** An immutable snapshot for one run.
*******************************************************************************/
public record Values(int audit_retention_years,
String export_lock_mode,
int export_retention_days,
int checkpoint_every_links,
int checkpoint_every_minutes,
int chain_interval_seconds,
int settle_horizon_seconds,
int late_detail_lookback_hours,
String export_cron,
String verify_cron)
{
}
/*******************************************************************************
** Use the production clock.
*******************************************************************************/
public WormAuditSettings(WormAuditQBitConfig config)
{
this(config, Clock.systemUTC());
}
/*******************************************************************************
** Inject a clock for cache boundary tests.
*******************************************************************************/
public WormAuditSettings(WormAuditQBitConfig config, Clock clock)
{
this.config = config;
this.clock = clock;
}
/*******************************************************************************
** Invalidate after a local edit; other application instances refresh within 30 seconds.
*******************************************************************************/
public synchronized void invalidate()
{
expires = Instant.MIN;
}
/*******************************************************************************
** Seed the singleton only when absent; never overwrite an administrator edit.
*******************************************************************************/
public static void seed(WormAuditQBitConfig config) throws com.kingsrook.qqq.backend.core.exceptions.QException
{
var store = new AuditStore(null);
if(store.get(WormAuditTables.SETTINGS, 1) == null)
{
store.insert(WormAuditTables.SETTINGS, java.util.List.of(defaults(config)));
}
config.invalidateSettings();
}
/*******************************************************************************
** Read at most once per 30 seconds, isolated by QInstance identity.
*******************************************************************************/
public synchronized Values read()
{
QInstance current = QContext.getQInstance();
if(current != instance || !clock.instant().isBefore(expires))
{
QRecord row = null;
if(current != null && current.getTable(WormAuditTables.SETTINGS) != null)
{
try
{
row = new AuditStore(null).get(WormAuditTables.SETTINGS, 1);
}
catch(Exception e)
{
if(!missingTable(e))
{
throw (new IllegalStateException("Cannot read WORM Audit Settings", e));
}
}
}
cached = row;
instance = current;
expires = row == null ? Instant.MIN : clock.instant().plusSeconds(30);
}
QRecord row = defaults(config);
if(cached != null)
{
for(String field : row.getValues().keySet())
{
if(cached.getValue(field) != null)
{
row.setValue(field, cached.getValue(field));
}
}
}
validate(row);
return (new Values(row.getValueInteger("audit_retention_years"),
row.getValueString("export_lock_mode"),
row.getValueInteger("export_retention_days"),
row.getValueInteger("checkpoint_every_links"),
row.getValueInteger("checkpoint_every_minutes"),
row.getValueInteger("chain_interval_seconds"),
row.getValueInteger("settle_horizon_seconds"),
row.getValueInteger("late_detail_lookback_hours"),
row.getValueString("export_cron"),
row.getValueString("verify_cron")));
}
/*******************************************************************************
** Build insert-only seed values from the host defaults.
*******************************************************************************/
public static QRecord defaults(WormAuditQBitConfig config)
{
return (new QRecord().withValue("id", 1)
.withValue("audit_retention_years", config.getRetentionYears())
.withValue("export_lock_mode", config.getExportLockMode())
.withValue("export_retention_days", config.getExportRetentionDays())
.withValue("checkpoint_every_links", config.getCheckpointEveryLinks())
.withValue("checkpoint_every_minutes", config.getCheckpointEveryMinutes())
.withValue("chain_interval_seconds", config.getChainIntervalSeconds())
.withValue("settle_horizon_seconds", config.getSettleHorizonSeconds())
.withValue("late_detail_lookback_hours", config.getLateDetailLookbackHours())
.withValue("export_cron", config.getExportCron())
.withValue("verify_cron", config.getVerifyCron()));
}
/*******************************************************************************
** Validate supplied fields; partial updates leave omitted fields untouched.
*******************************************************************************/
public static void validate(QRecord row)
{
if(row.getValue("id") != null && !Integer.valueOf(1).equals(row.getValueInteger("id")))
{
throw (new IllegalArgumentException("WORM Audit Settings has one row only (ID 1)."));
}
range(row, "audit_retention_years", "Audit retention years", 1, 100);
range(row, "export_retention_days", "Export retention days", 1, 36500);
range(row, "checkpoint_every_links", "Checkpoint every links", 1, 2147483647);
range(row, "checkpoint_every_minutes", "Checkpoint every minutes", 1, 2147483647);
range(row, "chain_interval_seconds", "Chain interval seconds", 1, 2147483647);
range(row, "settle_horizon_seconds", "Settle horizon seconds", 0, 2147483647);
range(row, "late_detail_lookback_hours", "Late detail lookback hours", 1, 2147483647);
if(row.getValues().containsKey("export_lock_mode") && !java.util.Arrays.asList("GOVERNANCE", "COMPLIANCE").contains(row.getValueString("export_lock_mode")))
{
throw (new IllegalArgumentException("Export lock mode must be GOVERNANCE or COMPLIANCE."));
}
for(String field : java.util.List.of("export_cron", "verify_cron"))
{
if(row.getValues().containsKey(field) && (row.getValueString(field) == null || !CronExpression.isValidExpression(row.getValueString(field))))
{
throw (new IllegalArgumentException((field.equals("export_cron") ? "Export" : "Verify") + " schedule must be a valid Quartz cron expression (UTC)."));
}
}
}
/*******************************************************************************
** Reject null, fractional and out-of-range numbers with a plain field label.
*******************************************************************************/
private static void range(QRecord row, String field, String label, int minimum, int maximum)
{
if(!row.getValues().containsKey(field))
{
return;
}
try
{
long value = Long.parseLong(String.valueOf(row.getValue(field)));
if(value >= minimum && value <= maximum)
{
return;
}
}
catch(NumberFormatException ignored)
{
// Use the same clear message for invalid numbers and invalid ranges.
}
throw (new IllegalArgumentException(label + " must be a whole number from " + minimum + " to " + maximum + "."));
}
/*******************************************************************************
** Recognize PostgreSQL/H2 missing-table errors without hiding other failures.
*******************************************************************************/
private static boolean missingTable(Throwable error)
{
for(Throwable cause = error; cause != null; cause = cause.getCause())
{
if(cause instanceof SQLException sql && ("42P01".equals(sql.getSQLState()) || "42S02".equals(sql.getSQLState()) || "42S04".equals(sql.getSQLState())))
{
return (true);
}
}
return (false);
}
}
@@ -0,0 +1,129 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.app;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.ddl.WormAuditDdl;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
/*******************************************************************************
** Read-only PostgreSQL catalog probes; errors become value-free diagnostics.
*******************************************************************************/
public final class PostgresSetupWarnings
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private PostgresSetupWarnings()
{
}
/*******************************************************************************
** Probe the runtime connection, never the configured migration role.
*******************************************************************************/
public static List<String> load(QBackendTransaction transaction, WormAuditQBitConfig config)
{
List<String> warnings = new ArrayList<>();
int years = config.effectiveSettings().audit_retention_years();
if(years != config.getRetentionYears())
{
warnings.add("Audit retention setting differs from the configured database purge interval (" + config.getRetentionYears() + " years). A reviewed migration is required.");
}
if(!(transaction instanceof RDBMSTransaction rdbms))
{
return (warnings);
}
try
{
var connection = rdbms.getConnection();
if(!"PostgreSQL".equals(connection.getMetaData().getDatabaseProductName()))
{
return (warnings);
}
// The live function is authoritative on PostgreSQL, rather than the renderer defaults.
warnings.clear();
List<String> tables = new ArrayList<>(config.getCoreAuditTableNames().values());
tables.addAll(WormAuditTables.ALL.stream().filter(t -> !WormAuditTables.SETTINGS.equals(t)).toList());
for(String table : tables)
{
for(String trigger : List.of("worm_audit_row_guard", "worm_audit_truncate_guard"))
{
try(var statement = connection.prepareStatement("SELECT EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid=to_regclass(?) AND tgname=? AND NOT tgisinternal AND tgenabled IN ('O','A'))"))
{
statement.setString(1, WormAuditDdl.identifier(table));
statement.setString(2, trigger);
try(var result = statement.executeQuery())
{
result.next();
if(!result.getBoolean(1))
{
warnings.add("Missing or disabled " + trigger + " on " + table);
}
}
}
}
try(var statement = connection.prepareStatement("SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tableowner=current_user AND to_regclass(quote_ident(schemaname)||'.'||quote_ident(tablename))=to_regclass(?))"))
{
statement.setString(1, WormAuditDdl.identifier(table));
try(var result = statement.executeQuery())
{
result.next();
if(result.getBoolean(1))
{
warnings.add("Runtime role owns audit table " + table);
}
}
}
}
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT pg_get_functiondef(to_regprocedure('public.worm_audit_purge(timestamp with time zone)'))"))
{
if(result.next() && result.getString(1) != null)
{
var interval = java.util.regex.Pattern.compile("(?i)interval\\s+'([0-9]+) years'").matcher(result.getString(1));
if(interval.find() && Integer.parseInt(interval.group(1)) != years)
{
warnings.add("Audit retention setting differs from the purge function's compiled interval (" + interval.group(1) + " years). A reviewed migration is required.");
}
}
}
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT to_regprocedure('public.worm_audit_purge(timestamp with time zone)') IS NOT NULL"))
{
result.next();
if(!result.getBoolean(1))
{
warnings.add("Retention purge function is missing");
}
}
}
catch(Exception e)
{
warnings.add("PostgreSQL setup checks unavailable");
}
return (warnings);
}
}
@@ -0,0 +1,170 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.app;
import java.util.List;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** WORM Audit navigation and standard HTML widgets, scoped to this QInstance.
*******************************************************************************/
public final class WormAuditApp
{
public static final String HOME_WIDGET = "wormAuditHome";
public static final String HEALTH_CARD_WIDGET = "wormAuditHealthCard";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private WormAuditApp()
{
}
/*******************************************************************************
** The home and optional host-dashboard summary share the app's permissions.
*******************************************************************************/
public static List<QWidgetMetaData> widgets()
{
return (List.of(
new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType())
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false).withPermissionRules(permissions()),
new QWidgetMetaData().withName(HEALTH_CARD_WIDGET).withLabel("WORM Audit").withType(WidgetType.HTML.getType())
.withCodeReference(new QCodeReference(HealthCardRenderer.class)).withGridColumns(12).withIsCard(true).withPermissionRules(permissions())));
}
/*******************************************************************************
** Home, Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges.
*******************************************************************************/
public static QAppMetaData app(WormAuditQBitConfig config, List<QTableMetaData> tables)
{
QAppMetaData app = new QAppMetaData().withName(config.getAppName()).withLabel(config.getAppLabel())
.withIcon(new QIcon("verified_user")).withPermissionRules(permissions()).withWidgets(List.of(HOME_WIDGET));
app.setParentAppName(config.getParentAppName());
for(String name : WormAuditTables.NAV)
{
QTableMetaData table = tables.stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(WormAuditTables.navigationLabel(name)),
WormAuditTables.SETTINGS.equals(name) ? table.clone().withLabel("Settings") : table);
}
return (app);
}
/*******************************************************************************
** Read on any WORM table grants the value-free aggregate overview.
*******************************************************************************/
public static QPermissionRules permissions()
{
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
.withCustomPermissionChecker(new QCodeReference(HomePermissionChecker.class)));
}
/*******************************************************************************
** Honor the host's actual table permission rules, including custom checkers.
*******************************************************************************/
public static class HomePermissionChecker implements CustomPermissionChecker
{
/*******************************************************************************
** Reject direct widget requests unless at least one table is readable.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
for(String table : WormAuditTables.ALL)
{
try
{
PermissionsHelper.checkTablePermissionThrowing(new QueryInput(table), TablePermissionSubType.READ);
return;
}
catch(QPermissionDeniedException expected)
{
// Another WORM table may grant access to the aggregate overview.
}
}
throw (new QPermissionDeniedException("Read access to a WORM audit table is required"));
}
}
/*******************************************************************************
** Full-width home renderer.
*******************************************************************************/
public static class HomeRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
** Render only metadata, counts and value-free reasons.
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
new HomePermissionChecker().checkPermissionsThrowing(input, null);
return (new RenderWidgetOutput(new RawHTML(null, WormAuditHomeView.render(WormAuditHomeData.load(), VerificationRuns.config().getTablePathPrefix()))));
}
}
/*******************************************************************************
** Optional compact host dashboard renderer.
*******************************************************************************/
public static class HealthCardRenderer extends AbstractWidgetRenderer
{
/*******************************************************************************
** Link the status summary to the owning app home.
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
new HomePermissionChecker().checkPermissionsThrowing(input, null);
WormAuditQBitConfig config = VerificationRuns.config();
String path = config.getTablePathPrefix() + "/" + (config.getParentAppName() == null ? "" : config.getParentAppName() + "/") + config.getAppName();
return (new RenderWidgetOutput(new RawHTML(config.getAppLabel(), WormAuditHomeView.renderHealthCard(WormAuditHomeData.load(), path))));
}
}
}
@@ -0,0 +1,213 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.app;
import java.sql.Connection;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore;
import com.kingsrook.qbits.wormaudit.export.SealedStore;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
/*******************************************************************************
** Read-only home snapshot. No audit messages, detail values or canonical content
** enter this model. Unexported suffixes are lag; holes before the sealed head
** are gaps. Only a later full scheduled chain PASS clears an open chain break.
*******************************************************************************/
public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, long settling, int settleSeconds,
QRecord lastVerification, QRecord openBreak, QRecord lastExport, long exportLag, Long exportAgeHours,
List<String> exportGaps, String bucketAlias, String lockCapability, String lockReason,
String exportLockMode, int exportRetentionDays, int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
List<QRecord> recentPurges, List<String> warnings)
{
/*******************************************************************************
** Load using the current instance's configuration and production clock.
*******************************************************************************/
public static WormAuditHomeData load() throws QException
{
return (load(VerificationRuns.config(), null, Instant.now()));
}
/*******************************************************************************
** The storage and time seams also serve the memory demo and deterministic tests.
*******************************************************************************/
public static WormAuditHomeData load(WormAuditQBitConfig config, SealedStore storage, Instant now) throws QException
{
var settings = config.effectiveSettings();
List<String> warnings = new ArrayList<>();
if("GOVERNANCE".equals(settings.export_lock_mode()))
{
warnings.add("Sealed exports use GOVERNANCE mode: an administrator with s3:BypassGovernanceRetention can delete them. Switch to COMPLIANCE for write-once copies.");
}
if(now.plus(Duration.ofDays(settings.export_retention_days())).isBefore(now.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant()))
{
warnings.add("Export retention is shorter than audit retention.");
}
String capability = "UNKNOWN";
String reason = "Object Lock capability could not be confirmed";
try
{
SealedStore resolved = storage != null ? storage : config.getSealedStoreSupplier() == null
? new ObjectStorageSealedStore(config.getExportBucketAlias(), "home") : config.getSealedStoreSupplier().get();
CapabilityStatus status = resolved.lockCapability();
if(status == CapabilityStatus.SUPPORTED || status == CapabilityStatus.UNSUPPORTED)
{
capability = status.name();
reason = status == CapabilityStatus.SUPPORTED ? null : "Export bucket does not support Object Lock";
}
}
catch(Exception e)
{
// Provider exception messages can contain credentials or object content.
reason = "Object Lock capability check failed";
}
if(reason != null)
{
warnings.add(reason);
}
if(config.getSigningKey() == null)
{
warnings.add("No signing key configured");
}
if(config.getTrustedPublicKeys() == null || config.getTrustedPublicKeys().isEmpty())
{
warnings.add("No trusted public keys configured");
}
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
{
try
{
if(transaction instanceof RDBMSTransaction rdbms)
{
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
}
AuditStore store = new AuditStore(transaction);
QRecord lastLink = store.last(WormAuditTables.CHAIN);
long links = store.count(WormAuditTables.CHAIN, new QQueryFilter());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
long backlog = store.unchainedCount(cutoff, true);
long settling = store.unchainedCount(cutoff, false);
QRecord verification = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(), "finished_at");
QQueryFilter dailyFilter = new QQueryFilter(new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"),
new QFilterCriteria("trigger", QCriteriaOperator.EQUALS, "SCHEDULE"));
QRecord daily = latest(store, WormAuditTables.VERIFICATION, dailyFilter, "finished_at");
if(daily == null || daily.getValueInstant("finished_at") == null)
{
warnings.add("Daily verification has never run");
}
else if(daily.getValueInstant("finished_at").isBefore(now.minus(Duration.ofHours(26))))
{
warnings.add("Daily verification last ran more than 26 hours ago");
}
QRecord failure = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(
new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"), new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "FAIL")), "id");
QRecord fullPass = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(
new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"), new QFilterCriteria("trigger", QCriteriaOperator.EQUALS, "SCHEDULE"),
new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "PASS")), "id");
QRecord openBreak = failure != null && (fullPass == null || failure.getValueLong("id") > fullPass.getValueLong("id")) ? failure : null;
QRecord export = store.last(WormAuditTables.EXPORT);
QRecord anchor = store.anchor();
long anchorId = anchor.getValueLong("anchor_link_id");
long exported = export == null ? anchorId : export.getValueLong("to_link_id");
long lag = store.count(WormAuditTables.CHAIN, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, exported)));
List<String> gaps = gaps(store, anchorId);
List<QRecord> oldest = store.summary("audit", new QQueryFilter().withOrderBy(new QFilterOrderBy("timestamp", true)).withLimit(1), Set.of("timestamp"));
Instant oldestAt = oldest.isEmpty() ? null : oldest.getFirst().getValueInstant("timestamp");
List<QRecord> purges = store.query(WormAuditTables.PURGE, new QQueryFilter()
.withOrderBy(new QFilterOrderBy("requested_at", false)).withOrderBy(new QFilterOrderBy("id", false)).withLimit(10));
warnings.addAll(PostgresSetupWarnings.load(transaction, config));
transaction.commit();
return (new WormAuditHomeData(links, lastLink == null ? null : lastLink.getValueInstant("chained_at"), backlog, settling,
settings.settle_horizon_seconds(), verification, openBreak, export, lag,
export == null || export.getValueInstant("sealed_at") == null ? null : Math.max(0, Duration.between(export.getValueInstant("sealed_at"), now).toHours()),
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, settings.export_lock_mode(), settings.export_retention_days(), settings.audit_retention_years(), anchorId == 0 ? null : anchor,
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
}
catch(Exception e)
{
transaction.rollback();
throw (new QException("Cannot load WORM audit home summary"));
}
}
}
/*******************************************************************************
** Read one result, using id to break equal-time ties deterministically.
*******************************************************************************/
private static QRecord latest(AuditStore store, String table, QQueryFilter filter, String time) throws QException
{
List<QRecord> rows = store.query(table, filter.withOrderBy(new QFilterOrderBy(time, false))
.withOrderBy(new QFilterOrderBy("id", false)).withLimit(1));
return (rows.isEmpty() ? null : rows.getFirst());
}
/*******************************************************************************
** Sweep export intervals in bounded pages; never enumerate links or objects.
** The writer appends ranges in id order. Overlapping ranges extend coverage.
*******************************************************************************/
private static List<String> gaps(AuditStore store, long anchor) throws QException
{
List<String> gaps = new ArrayList<>();
long next = anchor + 1;
long cursor = 0;
while(true)
{
List<QRecord> page = store.page(WormAuditTables.EXPORT, cursor, 1000);
if(page.isEmpty())
{
return (gaps);
}
for(QRecord export : page)
{
long from = export.getValueLong("from_link_id");
long to = export.getValueLong("to_link_id");
if(from > next)
{
gaps.add(next + "–" + (from - 1));
}
next = Math.max(next, to + 1);
}
cursor = page.getLast().getValueLong("id");
}
}
}
@@ -0,0 +1,225 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.app;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Stacked, wrapping definition lists. Colors follow BackupsHomeView exactly:
** inherited text/link colors, translucent neutral borders, muted opacity. Only
** allowlisted summary fields are rendered; no audit content is ever inspected.
*******************************************************************************/
public final class WormAuditHomeView
{
private static final DateTimeFormatter WHEN = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC);
private static final String CARD = "border:1px solid rgba(127,127,127,.3);border-radius:8px;padding:1rem;min-width:0;overflow-wrap:anywhere;";
private static final String LINK = "color:inherit;text-decoration:underline;text-underline-offset:2px;";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private WormAuditHomeView()
{
}
/*******************************************************************************
** Full home with explicit empty states and no fixed-width content.
*******************************************************************************/
public static String render(WormAuditHomeData data, String prefix)
{
StringBuilder html = new StringBuilder("<div class=\"worm-audit-home\" style=\"display:grid;gap:1rem;min-width:0;max-width:100%;\">");
open(html, "Chain status");
row(html, "Links chained", data.links());
row(html, "Last link", when(data.lastLinkAt(), "No links chained yet"));
row(html, "Unchained backlog", data.backlog() + " settled; " + data.settling() + " settling (" + data.settleSeconds() + " s horizon)");
QRecord verification = data.lastVerification();
if(verification == null)
{
row(html, "Last verification", "No verification has run yet");
}
else
{
linkedRow(html, "Last verification", recordLink(prefix, WormAuditTables.VERIFICATION, verification.getValueLong("id"), "Verification #" + verification.getValueLong("id")));
row(html, "Verified at", when(verification.getValueInstant("finished_at"), "Not recorded"));
row(html, "Result", verification.getValueString("outcome") + "; " + verification.getValueLong("links_checked") + " links checked");
}
QRecord broken = data.openBreak();
if(broken == null)
{
row(html, "Open chain breaks", verification == null ? "Unknown until verification runs" : "None reported");
}
else
{
linkedRow(html, "Open chain breaks", recordLink(prefix, WormAuditTables.VERIFICATION, broken.getValueLong("id"), broken.getValueLong("break_count") + " reported"));
linkedRow(html, "First break link", recordLink(prefix, WormAuditTables.CHAIN, broken.getValueLong("first_break_link_id"), "Link #" + broken.getValueLong("first_break_link_id")));
row(html, "First break audit", broken.getValueLong("first_break_audit_id"));
row(html, "Reason", broken.getValueString("first_break_reason"));
}
close(html);
open(html, "Export status");
row(html, "Export lock mode", data.exportLockMode());
row(html, "Export retention", data.exportRetentionDays() + " days");
QRecord export = data.lastExport();
if(export == null)
{
row(html, "Last sealed export", "No sealed exports yet");
}
else
{
linkedRow(html, "Last sealed export", recordLink(prefix, WormAuditTables.EXPORT, export.getValueLong("id"), "Export #" + export.getValueLong("id")));
row(html, "Link range", export.getValueLong("from_link_id") + "–" + export.getValueLong("to_link_id"));
row(html, "Sealed at", when(export.getValueInstant("sealed_at"), "Not recorded"));
row(html, "Retain until", when(export.getValueInstant("retain_until"), "Not recorded"));
row(html, "Last export lock mode", export.getValueString("lock_mode"));
}
row(html, "Export lag", data.exportLag() + " links; " + (data.exportAgeHours() == null ? "no previous export time" : data.exportAgeHours() + " h since last export"));
row(html, "Gaps in sealed ranges", data.exportGaps().isEmpty() ? (export == null ? "No sealed ranges to check" : "None") : String.join(", ", data.exportGaps()));
row(html, "Bucket alias", data.bucketAlias());
row(html, "Object Lock capability", data.lockCapability() + (data.lockReason() == null ? "" : ": " + data.lockReason()));
close(html);
open(html, "Retention");
row(html, "Configured retention", data.retentionYears() + " years");
QRecord anchor = data.anchor();
row(html, "Purge anchor", anchor == null ? "None" : "Link #" + anchor.getValueLong("anchor_link_id") + "; hash " + shortHash(anchor.getValueString("anchor_hash")));
row(html, "Oldest live audit", when(data.oldestAudit(), "No live audits"));
row(html, "Next eligible purge", when(data.nextPurge(), "No live audits to purge"));
close(html);
open(html, "Recent purges");
if(data.recentPurges().isEmpty())
{
row(html, "History", "No purge requests yet");
}
for(QRecord purge : data.recentPurges())
{
linkedRow(html, "Purge", recordLink(prefix, WormAuditTables.PURGE, purge.getValueLong("id"), "Purge #" + purge.getValueLong("id")));
row(html, "Requested at", when(purge.getValueInstant("requested_at"), "Not recorded"));
row(html, "Requested by", purge.getValueString("requested_by"));
row(html, "Outcome", purge.getValueString("outcome"));
row(html, "Cutoff", when(purge.getValueInstant("cutoff"), "Not recorded"));
row(html, "Deleted", purge.getValueLong("audits_deleted") + " audits; " + purge.getValueLong("details_deleted") + " details; " + purge.getValueLong("links_deleted") + " links");
row(html, "Reason", purge.getValueString("reason"));
}
close(html);
if(!data.warnings().isEmpty())
{
open(html, "Setup warnings");
for(String warning : data.warnings())
{
row(html, "Warning", warning);
}
close(html);
}
return (html.append("</div>").toString());
}
/*******************************************************************************
** Honest compact status, owned by and linked to the full home.
*******************************************************************************/
public static String renderHealthCard(WormAuditHomeData data, String homePath)
{
String status = data.openBreak() != null ? "Chain breaks reported" : data.lastVerification() == null ? "Not verified yet"
: data.warnings().isEmpty() ? "No open chain breaks reported" : "Setup needs attention";
return ("<div style=\"" + CARD + "\"><strong>" + esc(status) + "</strong><p>" + data.links() + " links; " + data.backlog()
+ " settled audits waiting; " + data.exportLag() + " links awaiting export.</p><a style=\"" + LINK + "\" href=\"" + esc(homePath) + "\">Open WORM Audit</a></div>");
}
/*******************************************************************************
** Section and definition list, with fluid wrapping label/value rows.
*******************************************************************************/
private static void open(StringBuilder html, String title)
{
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0 0 .75rem;font-size:1rem;font-weight:600;\">")
.append(esc(title)).append("</h3><dl style=\"margin:0;font-size:.9rem;\">");
}
/*******************************************************************************
** Close a section.
*******************************************************************************/
private static void close(StringBuilder html)
{
html.append("</dl></section>");
}
/*******************************************************************************
** Escape every value, including reasons and runtime identities.
*******************************************************************************/
private static void row(StringBuilder html, String label, Object value)
{
linkedRow(html, label, esc(value == null ? "Not recorded" : value.toString()));
}
/*******************************************************************************
** Only internally built and escaped links may supply HTML here.
*******************************************************************************/
private static void linkedRow(StringBuilder html, String label, String value)
{
html.append("<div style=\"display:flex;flex-wrap:wrap;gap:.25rem .75rem;margin:.4rem 0;min-width:0;\"><dt style=\"opacity:.75;flex:0 1 12rem;\">")
.append(esc(label)).append("</dt><dd style=\"margin:0;flex:1 1 14rem;min-width:0;overflow-wrap:anywhere;\">").append(value).append("</dd></div>");
}
/*******************************************************************************
** Standard QQQ record route; identifiers are numeric, never content.
*******************************************************************************/
private static String recordLink(String prefix, String table, Long id, String text)
{
return (id == null ? "Not recorded" : "<a style=\"" + LINK + "\" href=\"" + esc(prefix + "/" + table + "/" + id) + "\">" + esc(text) + "</a>");
}
/*******************************************************************************
** UTC time with an explicit empty state.
*******************************************************************************/
private static String when(Instant time, String empty)
{
return (time == null ? empty : WHEN.format(time));
}
/*******************************************************************************
** Keep hashes compact even on a narrow screen.
*******************************************************************************/
private static String shortHash(String hash)
{
return (hash == null ? "Not recorded" : hash.substring(0, Math.min(12, hash.length())) + "…");
}
/*******************************************************************************
** HTML text and quoted attribute escaping.
*******************************************************************************/
private static String esc(String text)
{
return (text == null ? "" : text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace("\"", "&quot;").replace("'", "&#39;"));
}
}
@@ -44,10 +44,12 @@ import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
@@ -91,6 +93,57 @@ public class AuditStore
}
/*******************************************************************************
** Project only summary fields; home reads never fetch audit content.
*******************************************************************************/
public List<QRecord> summary(String table, QQueryFilter filter, Set<String> fields) throws QException
{
QueryInput input = new QueryInput(table).withFilter(filter).withTransaction(transaction)
.withFieldNamesToInclude(fields).withShouldFetchHeavyFields(true);
input.withPermissionBypass("WORM audit home summary after widget permission check");
return (new QueryAction().execute(input).getRecords());
}
/*******************************************************************************
** Aggregate counts avoid loading the live chain for the home.
*******************************************************************************/
public long count(String table, QQueryFilter filter) throws QException
{
CountInput input = new CountInput(table).withFilter(filter).withTransaction(transaction);
input.withPermissionBypass("WORM audit home count after widget permission check");
return (new CountAction().execute(input).getCount());
}
/*******************************************************************************
** Count the same AUDIT anti-join as the writer, split at the settle horizon.
*******************************************************************************/
public long unchainedCount(Instant cutoff, boolean settled) throws QException
{
if(isRdbms())
{
String sql = "SELECT count(*) FROM " + physical("audit") + " a WHERE a." + column("audit", "timestamp")
+ (settled ? " < ?" : " >= ?") + unchainedPredicate();
return (sqlIds(sql, cutoff).getFirst());
}
Set<Long> covered = summary(WormAuditTables.CHAIN, new QQueryFilter(new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "AUDIT")), Set.of("audit_id"))
.stream().map(r -> r.getValueLong("audit_id")).collect(Collectors.toSet());
return (summary("audit", new QQueryFilter(), Set.of("id", "timestamp")).stream()
.filter(r -> r.getValueInstant("timestamp") != null && r.getValueInstant("timestamp").isBefore(cutoff) == settled
&& !covered.contains(r.getValueLong("id"))).count());
}
/*******************************************************************************
** Shared SQL coverage predicate for both writer selection and home counts.
*******************************************************************************/
private String unchainedPredicate()
{
return (" AND NOT EXISTS (SELECT 1 FROM worm_audit_chain c WHERE c.audit_id=a." + column("audit", "id") + " AND c.kind='AUDIT')");
}
/*******************************************************************************
** All records ordered by id.
*******************************************************************************/
@@ -219,8 +272,7 @@ public class AuditStore
if(isRdbms())
{
String sql = "SELECT a." + column("audit", "id") + " FROM " + physical("audit") + " a WHERE a." + column("audit", "timestamp")
+ " < ? AND NOT EXISTS (SELECT 1 FROM worm_audit_chain c WHERE c.audit_id=a." + column("audit", "id")
+ " AND c.kind='AUDIT') ORDER BY a." + column("audit", "timestamp") + ", a." + column("audit", "id") + " LIMIT " + limit;
+ " < ?" + unchainedPredicate() + " ORDER BY a." + column("audit", "timestamp") + ", a." + column("audit", "id") + " LIMIT " + limit;
Map<Long, QRecord> rows = index(in("audit", "id", sqlIds(sql, cutoff)));
return (rows.values().stream().sorted(Comparator.comparing((QRecord r) -> r.getValueInstant("timestamp"))
.thenComparing(r -> r.getValueLong("id"))).toList());
@@ -251,6 +303,15 @@ public class AuditStore
}
/*******************************************************************************
** Only the in-memory backend may use an unbounded completeness index.
*******************************************************************************/
public boolean isMemory()
{
return ("memory".equals(QContext.getQInstance().getBackendForTable(WormAuditTables.CHAIN).getBackendType()));
}
/*******************************************************************************
** Fetch one keyset page, never using an offset or a whole-chain read.
*******************************************************************************/
@@ -31,6 +31,7 @@ import java.util.TreeMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
@@ -99,7 +100,7 @@ public class WormAuditChainer
}
}
}
Counts counts = append(new AuditStore(transaction), now);
Counts counts = append(new AuditStore(transaction), now, config.effectiveSettings());
transaction.commit();
return (counts);
}
@@ -119,14 +120,14 @@ public class WormAuditChainer
/*******************************************************************************
** Append settled headers and uncovered detail chunks, then checkpoint.
*******************************************************************************/
private Counts append(AuditStore store, Instant now) throws Exception
private Counts append(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
{
QRecord tail = store.last(WormAuditTables.CHAIN);
QRecord anchor = store.anchor();
long nextId = tail == null ? anchor.getValueLong("anchor_link_id") + 1 : tail.getValueLong("id") + 1;
String previous = tail == null ? anchor.getValueString("anchor_hash") : tail.getValueString("link_hash");
List<QRecord> appended = new ArrayList<>();
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
List<QRecord> audits = store.unchained(cutoff, config.getBatchSize());
Map<Long, QRecord> details = AuditStore.index(store.in("auditDetail", "auditId", audits.stream().map(r -> r.getValueLong("id")).toList()));
Map<Long, List<QRecord>> grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId")));
@@ -149,7 +150,7 @@ public class WormAuditChainer
insertLinks(store, appended);
int linkCount = appended.size();
appended.clear();
details = AuditStore.index(store.lateDetails(cutoff, config.getLateDetailLookbackHours(), config.getBatchSize()));
details = AuditStore.index(store.lateDetails(cutoff, settings.late_detail_lookback_hours(), config.getBatchSize()));
grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId"), TreeMap::new, Collectors.toList()));
Map<Long, QRecord> headers = AuditStore.index(store.in("audit", "id", grouped.keySet()));
for(var entry : grouped.entrySet())
@@ -165,7 +166,7 @@ public class WormAuditChainer
}
}
insertLinks(store, appended);
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now), false));
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now, settings), false));
}
@@ -195,7 +196,7 @@ public class WormAuditChainer
/*******************************************************************************
** Checkpoint the uncheckpointed suffix by count or elapsed time.
*******************************************************************************/
private int checkpoint(AuditStore store, Instant now) throws Exception
private int checkpoint(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
{
if(config.getSigningKey() == null)
{
@@ -206,13 +207,13 @@ public class WormAuditChainer
int count = 0;
while(true)
{
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, config.getCheckpointEveryLinks());
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, settings.checkpoint_every_links());
if(pending.isEmpty())
{
return (count);
}
Instant since = latest == null ? pending.getFirst().getValueInstant("chained_at") : latest.getValueInstant("created_at");
if(pending.size() < config.getCheckpointEveryLinks() && now.isBefore(since.plusSeconds(config.getCheckpointEveryMinutes() * 60L)))
if(pending.size() < settings.checkpoint_every_links() && now.isBefore(since.plusSeconds(settings.checkpoint_every_minutes() * 60L)))
{
return (count);
}
@@ -102,6 +102,13 @@ public final class WormAuditDdl
{
sql = sql.replace("${" + entry.getValue() + "}", identifier(names.get(entry.getKey())));
}
var defaults = com.kingsrook.qbits.wormaudit.WormAuditSettings.defaults(config);
com.kingsrook.qbits.wormaudit.WormAuditSettings.validate(defaults);
String values = java.util.List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days", "checkpoint_every_links",
"checkpoint_every_minutes", "chain_interval_seconds", "settle_horizon_seconds", "late_detail_lookback_hours", "export_cron", "verify_cron")
.stream().map(field -> defaults.getValue(field) instanceof Number ? defaults.getValue(field).toString()
: "'" + defaults.getValueString(field).replace("'", "''") + "'").collect(java.util.stream.Collectors.joining(", "));
sql = sql.replace("${settings_values}", values);
return (sql.replace("${app_role}", identifier(config.getAppRole())).replace("${retention}", config.getRetentionYears() + " years"));
}
catch(IOException e)
@@ -0,0 +1,220 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.export;
import java.io.IOException;
import java.io.InputStream;
import java.io.Serializable;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.objectstorage.api.StoredObject;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Version-one export encoding and temporary files. JSON errors never echo input.
*******************************************************************************/
public class ExportFiles implements AutoCloseable
{
public static final List<String> NAMES = List.of("links.jsonl", "checkpoints.jsonl", "manifest.json");
private static final ObjectMapper JSON = new ObjectMapper().enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION)
.enable(DeserializationFeature.USE_LONG_FOR_INTS).enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
private final Path directory;
/*******************************************************************************
** Create a private temporary directory, removed on all normal failure paths.
*******************************************************************************/
public ExportFiles() throws IOException
{
directory = Files.createTempDirectory("worm-audit-");
}
/*******************************************************************************
** Resolve only one of the fixed filenames, never a manifest-supplied path.
*******************************************************************************/
public Path file(String name)
{
if(!NAMES.contains(name))
{
throw (new IllegalArgumentException("Unknown export file"));
}
return (directory.resolve(name));
}
/*******************************************************************************
** Parse the JCS schema with strict duplicate and trailing-token rejection.
*******************************************************************************/
public static Map<String, Object> parse(byte[] bytes) throws IOException
{
try
{
return (JSON.readValue(bytes, new TypeReference<Map<String, Object>>() {}));
}
catch(Exception e)
{
throw (new IOException("Invalid sealed JSON"));
}
}
/*******************************************************************************
** Wire representation of a chain row, excluding database-only timestamps.
*******************************************************************************/
public static Map<String, Object> link(QRecord row)
{
return (Map.of("id", row.getValueLong("id"), "kind", row.getValueString("kind"),
"auditId", row.getValueLong("audit_id"), "detailIds", AuditStore.detailIds(row),
"prevHash", row.getValueString("prev_hash"), "linkHash", row.getValueString("link_hash")));
}
/*******************************************************************************
** Preserve every checkpoint field, representing the timestamp as UTC text.
*******************************************************************************/
public static Map<String, Object> checkpoint(QRecord row)
{
Map<String, Object> fields = new LinkedHashMap<>();
for(String key : List.of("id", "from_link_id", "to_link_id", "link_count"))
{
fields.put(key, row.getValueLong(key));
}
for(String key : List.of("head_hash", "merkle_root", "algorithm", "key_id", "public_key", "signature"))
{
fields.put(key, row.getValueString(key));
}
fields.put("created_at", row.getValueInstant("created_at").toString());
return (fields);
}
/*******************************************************************************
** Convert a checkpoint envelope back to the phase (a) signature input.
*******************************************************************************/
public static QRecord record(Map<String, Object> fields)
{
QRecord row = new QRecord();
fields.forEach((key, value) -> row.setValue(key, (Serializable) value));
return (row);
}
/*******************************************************************************
** Compute a digest without materializing a file.
*******************************************************************************/
public static String sha256(Path path) throws Exception
{
MessageDigest digest = MessageDigest.getInstance("SHA-256");
try(InputStream input = Files.newInputStream(path))
{
byte[] buffer = new byte[65536];
int read;
while((read = input.read(buffer)) != -1)
{
digest.update(buffer, 0, read);
}
}
return (HexFormat.of().formatHex(digest.digest()));
}
/*******************************************************************************
** A lock echo is required even if a provider advertises the capability.
*******************************************************************************/
public static boolean locked(StoredObject object, String mode, java.time.Instant until)
{
return (object != null && java.util.Objects.equals(mode, object.getLockMode()) && object.getLockRetainUntil() != null
&& !object.getLockRetainUntil().isBefore(until));
}
/*******************************************************************************
** A null version belongs to unversioned storage and cannot be sealed.
*******************************************************************************/
public static boolean validVersion(String versionId)
{
return (versionId != null && !versionId.isBlank() && !"null".equals(versionId));
}
/*******************************************************************************
** Fetch covered detail rows in bounded IN batches.
*******************************************************************************/
public static Map<Long, QRecord> details(AuditStore store, List<QRecord> links) throws Exception
{
List<Long> ids = new ArrayList<>();
for(QRecord link : links)
{
List<Long> covered = AuditStore.detailIds(link);
if(covered.size() > 250)
{
throw (new IOException("Invalid detail coverage"));
}
ids.addAll(covered);
}
Map<Long, QRecord> rows = new LinkedHashMap<>();
for(int start = 0; start < ids.size(); start += 1000)
{
rows.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + 1000, ids.size())))));
}
return (rows);
}
/*******************************************************************************
** Remove only the three owned files and their directory.
*******************************************************************************/
@Override
public void close() throws IOException
{
IOException failure = null;
for(String name : NAMES)
{
try
{
Files.deleteIfExists(file(name));
}
catch(IOException e)
{
failure = new IOException("Cannot remove temporary export files");
}
}
Files.deleteIfExists(directory);
if(failure != null)
{
throw (failure);
}
}
}
@@ -0,0 +1,141 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.export;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import com.kingsrook.qbits.objectstorage.api.Capability;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
import com.kingsrook.qbits.objectstorage.api.Page;
import com.kingsrook.qbits.objectstorage.api.PutRequest;
import com.kingsrook.qbits.objectstorage.api.StorageBucket;
import com.kingsrook.qbits.objectstorage.api.StorageKeys;
import com.kingsrook.qbits.objectstorage.api.StoredObject;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
import com.kingsrook.qbits.objectstorage.internal.SecretRefs;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
/*******************************************************************************
** Thin system-caller adapter to qbit-object-storage.
*******************************************************************************/
public class ObjectStorageSealedStore implements SealedStore
{
private final StorageBucket bucket;
/*******************************************************************************
** Bind the recorded alias, so old exports survive configuration changes.
*******************************************************************************/
public ObjectStorageSealedStore(String alias, String runId)
{
bucket = ObjectStorage.asSystem("worm_audit", runId).bucket(alias);
}
/*******************************************************************************
** Refuse unknown, partial, or unsupported lock capabilities.
*******************************************************************************/
@Override
public boolean supportsLock()
{
return (bucket.config().getRole() == BucketRole.BACKUP
&& bucket.capability(Capability.OBJECT_LOCK) == CapabilityStatus.SUPPORTED);
}
/*******************************************************************************
** Report the provider capability without collapsing UNKNOWN into UNSUPPORTED.
*******************************************************************************/
@Override
public CapabilityStatus lockCapability()
{
return (bucket.capability(Capability.OBJECT_LOCK));
}
/*******************************************************************************
** Stream a bounded upload and close the source even when the provider fails.
*******************************************************************************/
@Override
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception
{
try(InputStream input = Files.newInputStream(file))
{
return (bucket.put(PutRequest.of(key, input).withContentLength(Files.size(file))
.withObjectLock(mode, retainUntil)).getVersionId());
}
}
/*******************************************************************************
** Read retention metadata.
*******************************************************************************/
@Override
public StoredObject head(String key)
{
return (bucket.head(key).orElse(null));
}
/*******************************************************************************
** RC.2 has no version GET facade. Reuse its configured system connection,
** key normalization and limiter; closing the stream closes the S3 response.
*******************************************************************************/
@Override
public InputStream getVersion(String key, String versionId) throws IOException
{
if(!ExportFiles.validVersion(versionId))
{
throw (new IOException("Missing sealed object version"));
}
var connection = ObjectStorageRuntime.connection(bucket.config().getConnection());
try(var ignored = connection.getRateLimiter().acquire())
{
var input = connection.client().getObject(GetObjectRequest.builder()
.bucket(SecretRefs.resolveSetting(bucket.config().getBucketName())).key(StorageKeys.normalize(key)).versionId(versionId).build());
if(!versionId.equals(input.response().versionId()))
{
input.close();
throw (new IOException("Storage returned a different object version"));
}
return (input);
}
}
/*******************************************************************************
** Forward bounded listing.
*******************************************************************************/
@Override
public Page<ObjectSummary> list(ListRequest request)
{
return (bucket.list(request));
}
}
@@ -0,0 +1,424 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.export;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.sql.Connection;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.chain.Hashes;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
/*******************************************************************************
** Verify the sealed evidence before comparing it to a repeatable live snapshot.
** No exported content or provider/parser exception text enters diagnostics.
*******************************************************************************/
public class SealedExportVerifier
{
private final WormAuditQBitConfig config;
private final SealedStore suppliedStorage;
/*******************************************************************************
** Findings plus the number verified without live rows after a retention purge.
*******************************************************************************/
public record Report(VerificationResult result, long purged)
{
}
/*******************************************************************************
** Production entry point.
*******************************************************************************/
public SealedExportVerifier(WormAuditQBitConfig config)
{
this(config, null);
}
/*******************************************************************************
** Testable storage boundary.
*******************************************************************************/
public SealedExportVerifier(WormAuditQBitConfig config, SealedStore storage)
{
this.config = config;
this.suppliedStorage = storage;
}
/*******************************************************************************
** Persist exactly one verification row for this selected export, even on failure.
*******************************************************************************/
public Report verify(QRecord export, String trigger) throws QException
{
config.effectiveSettings();
VerificationResult failure = new VerificationResult();
failure.scope(0, export.getValueLong("from_link_id"), export.getValueLong("to_link_id"));
Report report;
WormAuditChainer.JVM_LOCK.lock();
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN));
ExportFiles files = new ExportFiles())
{
try
{
if(transaction instanceof RDBMSTransaction rdbms)
{
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
}
SealedStore storage = suppliedStorage == null
? config.getSealedStoreSupplier() == null || !config.getExportBucketAlias().equals(export.getValueString("bucket_alias"))
? new ObjectStorageSealedStore(export.getValueString("bucket_alias"), "verify-" + export.getValueLong("id"))
: config.getSealedStoreSupplier().get() : suppliedStorage;
Map<String, Object> versions = ExportFiles.parse(export.getValueString("object_versions").getBytes(StandardCharsets.UTF_8));
for(String name : ExportFiles.NAMES)
{
String key = export.getValueString("object_prefix") + name;
if(!(versions.get(name) instanceof String version) || !ExportFiles.validVersion(version))
{
failure.add(null, null, "object " + name + " has no recorded sealed version id");
throw (new IOException("Missing sealed object version"));
}
var head = storage.head(key);
if(head == null)
{
failure.add(null, null, "object " + key + " is missing" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
? "; possible GOVERNANCE bypass removal by an administrator" : ""));
}
if(head != null && !version.equals(head.getVersionId()))
{
failure.add(null, null, "object " + name + " was overwritten after sealing (current version " + head.getVersionId()
+ ", sealed version " + version + " is still retained under Object Lock)");
}
if(!ExportFiles.locked(head, export.getValueString("lock_mode"), export.getValueInstant("retain_until")))
{
failure.add(null, null, "object " + name + " lacks the recorded " + export.getValueString("lock_mode") + " retention");
}
}
for(String name : ExportFiles.NAMES)
{
String key = export.getValueString("object_prefix") + name;
try(InputStream input = storage.getVersion(key, (String) versions.get(name)))
{
Files.copy(input, files.file(name));
}
catch(Exception e)
{
failure.add(null, null, "object " + key + " sealed version cannot be read" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
? "; if removed, possible GOVERNANCE bypass removal by an administrator" : ""));
throw (new IOException("Cannot read sealed object version"));
}
}
report = inspect(files, export, new AuditStore(transaction));
for(var item : failure.getBreaks())
{
report.result().add(item.linkId(), item.auditId(), item.reason());
}
transaction.commit();
}
catch(Exception e)
{
transaction.rollback();
throw (e);
}
}
catch(Exception e)
{
failure.add(null, null, "Sealed export cannot be read or has invalid structure");
report = new Report(failure, 0);
}
finally
{
WormAuditChainer.JVM_LOCK.unlock();
}
VerificationRuns.save(report.result(), trigger, export.getValueLong("id"));
return (report);
}
/*******************************************************************************
** Inspect local files, also used by the exporter before committing any objects.
*******************************************************************************/
public Report inspect(ExportFiles files, QRecord export, AuditStore store) throws Exception
{
VerificationResult result = new VerificationResult();
long from = export.getValueLong("from_link_id");
long to = export.getValueLong("to_link_id");
result.scope(0, from, to);
if(!Objects.equals(ExportFiles.sha256(files.file("manifest.json")), export.getValueString("manifest_sha256")))
{
result.add(null, null, "object manifest.json hash does not match the export row");
return (new Report(result, 0));
}
if(Files.size(files.file("manifest.json")) > 65536)
{
result.add(null, null, "Manifest exceeds the format size limit");
return (new Report(result, 0));
}
Map<String, Object> manifest = ExportFiles.parse(Files.readAllBytes(files.file("manifest.json")));
if(!"worm-audit-export/1".equals(manifest.get("format")) || from < 1 || to < from
|| !Objects.equals(manifest.get("fromLinkId"), from) || !Objects.equals(manifest.get("toLinkId"), to)
|| !Objects.equals(manifest.get("linkCount"), to - from + 1)
|| !Objects.equals(manifest.get("createdAt"), export.getValueInstant("sealed_at").toString())
|| !java.util.Set.of("GOVERNANCE", "COMPLIANCE").contains(export.getValueString("lock_mode"))
|| !Objects.equals(export.getValueString("manifest_key"), export.getValueString("object_prefix") + "manifest.json"))
{
result.add(null, null, "Manifest metadata does not match the export row");
}
QQueryFilter previousFilter = new QQueryFilter().withOrderBy(new QFilterOrderBy("id", false)).withLimit(1);
if(export.getValueLong("id") != null)
{
previousFilter.withCriteria(new QFilterCriteria("id", QCriteriaOperator.LESS_THAN, export.getValueLong("id")));
}
List<QRecord> previousRows = store.query(WormAuditTables.EXPORT, previousFilter);
QRecord previous = previousRows.isEmpty() ? null : previousRows.getFirst();
String previousHash = previous == null ? null : previous.getValueString("manifest_sha256");
if(!Objects.equals(manifest.get("prevManifestSha256"), previousHash)
|| !Objects.equals(export.getValueString("prev_manifest_sha256"), previousHash)
|| (previous != null && previous.getValueLong("to_link_id") != from - 1))
{
result.add(null, null, "prevManifestSha256 does not match the previous export row");
}
Map<?, ?> hashes = (Map<?, ?>) manifest.get("files");
for(String name : List.of("links.jsonl", "checkpoints.jsonl"))
{
if(!Objects.equals(ExportFiles.sha256(files.file(name)), hashes.get(name)))
{
result.add(null, null, "object " + name + " hash does not match the manifest");
}
}
if(!result.passed())
{
return (new Report(result, 0));
}
return (checkLines(files, export, store, manifest, result));
}
/*******************************************************************************
** Consume links alongside their checkpoint envelopes. Only one checkpoint's
** hashes and one content page are retained, never the whole export.
*******************************************************************************/
private Report checkLines(ExportFiles files, QRecord export, AuditStore store, Map<String, Object> manifest, VerificationResult result) throws Exception
{
long from = export.getValueLong("from_link_id");
long to = export.getValueLong("to_link_id");
long cursor = from - 1;
long checkpointCount = 0;
long checked = 0;
long purged = 0;
long anchor = store.anchor().getValueLong("anchor_link_id");
String previousHash = from == 1 ? Hashes.GENESIS : null;
if(cursor == anchor)
{
previousHash = store.anchor().getValueString("anchor_hash");
}
long lastCheckpoint = 0;
try(BufferedReader links = Files.newBufferedReader(files.file("links.jsonl"), StandardCharsets.UTF_8);
BufferedReader checkpoints = Files.newBufferedReader(files.file("checkpoints.jsonl"), StandardCharsets.UTF_8))
{
String line;
while((line = checkpoints.readLine()) != null)
{
QRecord checkpoint = ExportFiles.record(ExportFiles.parse(line.getBytes(StandardCharsets.UTF_8)));
long checkpointId = checkpoint.getValueLong("id");
String label = "checkpoint " + WormAuditVerifier.number(checkpointId) + ": ";
long end = checkpoint.getValueLong("to_link_id");
long start = checkpoint.getValueLong("from_link_id");
long count = checkpoint.getValueLong("link_count");
if(start != cursor + 1 || end < start || end > to || count != end - start + 1
|| count > Integer.MAX_VALUE || checkpointId <= lastCheckpoint)
{
result.add(null, null, label + "invalid or incomplete checkpoint range");
break;
}
if(!Checkpoints.verify(checkpoint, config.getTrustedPublicKeys().get(checkpoint.getValueString("key_id"))))
{
result.add(null, null, label + "signature does not verify");
}
if(checkpointCount == 0 && !Objects.equals(export.getValueLong("from_checkpoint_id"), checkpointId))
{
result.add(null, null, label + "first checkpoint does not match the export row");
}
List<String> hashes = new ArrayList<>();
while(cursor < end)
{
List<Map<String, Object>> page = new ArrayList<>();
for(int i = 0, size = (int) Math.min(1000, end - cursor); i < size; i++)
{
String linkLine = links.readLine();
if(linkLine == null)
{
result.add(cursor + 1, null, "Sealed links are missing from the checkpoint range");
result.scope(checked, from, to);
return (new Report(result, purged));
}
page.add(ExportFiles.parse(linkLine.getBytes(StandardCharsets.UTF_8)));
}
compareLive(page, store, export, anchor, result);
for(Map<String, Object> document : page)
{
Map<?, ?> link = (Map<?, ?>) document.get("link");
long id = (Long) link.get("id");
long auditId = (Long) link.get("auditId");
String prefix = "link " + WormAuditVerifier.number(id) + " (audit " + WormAuditVerifier.number(auditId) + "): ";
String prev = (String) link.get("prevHash");
String hash = (String) link.get("linkHash");
if(id != cursor + 1 || (previousHash != null && !Objects.equals(prev, previousHash)))
{
result.add(id, auditId, prefix + "sealed chain continuity differs");
}
Map<?, ?> content = (Map<?, ?>) document.get("content");
Map<?, ?> audit = (Map<?, ?>) content.get("audit");
List<?> detailIds = (List<?>) link.get("detailIds");
List<?> details = (List<?>) content.get("details");
List<?> contentIds = details.stream().map(d -> ((Map<?, ?>) d).get("id")).toList();
if(!Objects.equals(audit.get("id"), auditId) || !Objects.equals(content.get("v"), 1L)
|| !List.of("AUDIT", "LATE_DETAIL").contains(link.get("kind"))
|| detailIds.size() > 250 || !detailIds.equals(contentIds)
|| detailIds.stream().distinct().count() != detailIds.size()
|| ("LATE_DETAIL".equals(link.get("kind")) && detailIds.isEmpty()))
{
result.add(id, auditId, prefix + "sealed content identifiers or coverage differ");
}
if(!Hashes.link(prev, Jcs.bytes(content)).equals(hash))
{
result.add(id, auditId, prefix + "sealed content differs from the chained hash");
}
hashes.add(hash);
previousHash = hash;
cursor++;
checked++;
if(id <= anchor)
{
purged++;
}
}
}
if(!Objects.equals(Hashes.merkle(hashes), checkpoint.getValueString("merkle_root"))
|| !Objects.equals(previousHash, checkpoint.getValueString("head_hash")))
{
result.add(null, null, label + "Merkle root or head hash differs");
}
lastCheckpoint = checkpointId;
checkpointCount++;
}
if(cursor != to || links.readLine() != null || !Objects.equals(manifest.get("checkpointCount"), checkpointCount)
|| !Objects.equals(export.getValueLong("to_checkpoint_id"), lastCheckpoint)
|| !Objects.equals(manifest.get("headHash"), previousHash))
{
result.add(null, null, "Sealed range, counts or head hash differ from the manifest");
}
}
result.scope(checked, from, to);
return (new Report(result, purged));
}
/*******************************************************************************
** Compare a bounded page by exported ids, so missing live rows cannot vanish
** from the verification scope. Purged ids are checked from signed copies only.
*******************************************************************************/
private void compareLive(List<Map<String, Object>> page, AuditStore store, QRecord export, long anchor, VerificationResult result) throws Exception
{
List<QRecord> sealed = page.stream().map(document ->
{
Map<?, ?> link = (Map<?, ?>) document.get("link");
List<?> ids = (List<?>) link.get("detailIds");
return (new QRecord().withValue("id", (Long) link.get("id")).withValue("audit_id", (Long) link.get("auditId"))
.withValue("kind", (String) link.get("kind")).withValue("prev_hash", (String) link.get("prevHash"))
.withValue("link_hash", (String) link.get("linkHash"))
.withValue("detail_ids", ids.stream().map(Object::toString).collect(java.util.stream.Collectors.joining(","))));
}).toList();
List<QRecord> liveScope = sealed.stream().filter(r -> r.getValueLong("id") > anchor).toList();
Map<Long, QRecord> live = AuditStore.index(store.in(WormAuditTables.CHAIN, "id", liveScope.stream().map(r -> r.getValueLong("id")).toList()));
Map<Long, QRecord> audits = AuditStore.index(store.in("audit", "id", liveScope.stream().map(r -> r.getValueLong("audit_id")).toList()));
Map<Long, QRecord> details = ExportFiles.details(store, liveScope);
for(int i = 0; i < sealed.size(); i++)
{
QRecord link = sealed.get(i);
long id = link.getValueLong("id");
long auditId = link.getValueLong("audit_id");
if(id <= anchor)
{
continue;
}
String prefix = "link " + WormAuditVerifier.number(id) + " (audit " + WormAuditVerifier.number(auditId) + "): ";
QRecord current = live.get(id);
if(current == null)
{
result.add(id, auditId, "link " + WormAuditVerifier.number(id) + " is missing from the live chain");
}
else
{
for(String field : List.of("kind", "audit_id", "detail_ids", "prev_hash", "link_hash"))
{
if(!Objects.equals(link.getValue(field), current.getValue(field)))
{
result.add(id, auditId, prefix + "live chain " + field + " differs from the sealed copy");
}
}
}
try
{
QRecord audit = audits.get(auditId);
if(audit == null)
{
result.add(id, auditId, prefix + "live audit header is missing");
}
else if(!Arrays.equals(store.canonical(audit, AuditStore.detailIds(link), details), Jcs.bytes(page.get(i).get("content"))))
{
result.add(id, auditId, prefix + "live content differs from the sealed copy (sealed "
+ export.getValueInstant("sealed_at").atZone(ZoneOffset.UTC).toLocalDate() + ")");
}
}
catch(Exception e)
{
result.add(id, auditId, prefix + "live content is missing or invalid");
}
}
}
}
@@ -0,0 +1,392 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.export;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.sql.ResultSet;
import java.sql.Statement;
import java.time.Instant;
import java.time.ZoneOffset;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoUnit;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditQBitProducer;
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.chain.Hashes;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
/*******************************************************************************
** Seal the checkpointed suffix under the chainer/purger lock. Only successful
** uploads with verified lock echoes produce an export row.
*******************************************************************************/
public class SealedExporter
{
private final WormAuditQBitConfig config;
private final SealedStore suppliedStorage;
/*******************************************************************************
** A review scope; execution always recomputes it under the writer lock.
*******************************************************************************/
public record Scope(long from, long to, long checkpoints, QRecord previous)
{
/*******************************************************************************
** Number of links, including the empty checkpoint suffix.
*******************************************************************************/
public long count()
{
return (Math.max(0, to - from + 1));
}
/*******************************************************************************
** Value-free Review text.
*******************************************************************************/
public String review()
{
return (count() == 0 ? "Nothing to seal: no new checkpoint" : WormAuditVerifier.number(count())
+ " chain links will be sealed (links " + WormAuditVerifier.number(from) + "–" + WormAuditVerifier.number(to)
+ ", " + WormAuditVerifier.number(checkpoints) + " checkpoints)");
}
}
/*******************************************************************************
** A completed, empty, or failed run suitable for a Result line.
*******************************************************************************/
public record Result(QRecord row, String message, boolean failed)
{
}
/*******************************************************************************
** Production storage is resolved lazily, so an empty run needs no bucket.
*******************************************************************************/
public SealedExporter(WormAuditQBitConfig config)
{
this(config, null);
}
/*******************************************************************************
** Inject the tiny storage boundary for deterministic tests.
*******************************************************************************/
public SealedExporter(WormAuditQBitConfig config, SealedStore storage)
{
this.config = config;
this.suppliedStorage = storage;
}
/*******************************************************************************
** Select the contiguous suffix, never silently skipping an unexported purge.
*******************************************************************************/
public static Scope scope(AuditStore store) throws QException
{
QRecord previous = store.last(WormAuditTables.EXPORT);
long anchor = store.anchor().getValueLong("anchor_link_id");
long after = previous == null ? anchor : previous.getValueLong("to_link_id");
if(after < anchor)
{
throw (new QException("Unexported links were purged"));
}
QRecord checkpoint = store.last(WormAuditTables.CHECKPOINT);
long to = checkpoint == null ? after : Math.max(after, checkpoint.getValueLong("to_link_id"));
long count = 0;
long cursor = 0;
while(true)
{
List<QRecord> page = checkpoints(store, after + 1, to, cursor);
if(page.isEmpty())
{
break;
}
count += page.size();
cursor = page.getLast().getValueLong("id");
}
return (new Scope(after + 1, to, count, previous));
}
/*******************************************************************************
** Select checkpoint envelopes wholly contained in the export range.
*******************************************************************************/
private static List<QRecord> checkpoints(AuditStore store, long from, long to, long after) throws QException
{
return (store.query(WormAuditTables.CHECKPOINT, new QQueryFilter(
new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, after),
new QFilterCriteria("from_link_id", QCriteriaOperator.GREATER_THAN_OR_EQUALS, from),
new QFilterCriteria("to_link_id", QCriteriaOperator.LESS_THAN_OR_EQUALS, to))
.withOrderBy(new QFilterOrderBy("id", true)).withLimit(1000)));
}
/*******************************************************************************
** Execute with alerts on user and scheduled failures; schedules always beat.
*******************************************************************************/
public Result run(String trigger, Instant now)
{
try
{
config.validate();
Result result = lockedRun(now);
QAlerts.resolve("wormaudit.export.failed", "export", "Sealed audit export completed");
return (result);
}
catch(Exception e)
{
// All exposed failures originate here or use a fixed, value-free code.
String reason = e instanceof ExportFailure ? e.getMessage() : "Sealed export failed: storage or database operation failed";
QAlerts.raise(new AlertSignal("wormaudit.export.failed").withDedupKey("export").withTitle(reason));
return (new Result(null, reason, true));
}
finally
{
if("SCHEDULE".equals(trigger))
{
QAlerts.heartbeat("wormaudit.export.daily");
}
}
}
/*******************************************************************************
** Serialize with writers in this JVM and PostgreSQL writers on other hosts.
*******************************************************************************/
private Result lockedRun(Instant now) throws Exception
{
WormAuditChainer.JVM_LOCK.lock();
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
{
try
{
if(transaction instanceof RDBMSTransaction rdbms && "PostgreSQL".equals(rdbms.getConnection().getMetaData().getDatabaseProductName()))
{
try(Statement statement = rdbms.getConnection().createStatement();
ResultSet result = statement.executeQuery("SELECT pg_advisory_xact_lock(" + WormAuditChainer.ADVISORY_LOCK + ")"))
{
result.next();
}
}
AuditStore store = new AuditStore(transaction);
Scope scope = scope(store);
Result result;
if(scope.count() == 0)
{
result = new Result(null, scope.review(), false);
}
else
{
SealedStore storage = suppliedStorage == null
? config.getSealedStoreSupplier() == null ? new ObjectStorageSealedStore(config.getExportBucketAlias(), java.util.UUID.randomUUID().toString())
: config.getSealedStoreSupplier().get() : suppliedStorage;
result = seal(store, storage, scope, now.truncatedTo(ChronoUnit.MILLIS));
}
transaction.commit();
return (result);
}
catch(Exception e)
{
transaction.rollback();
throw (e);
}
}
finally
{
WormAuditChainer.JVM_LOCK.unlock();
}
}
/*******************************************************************************
** Build locally before uploading. Partial remote uploads have no database row.
*******************************************************************************/
private Result seal(AuditStore store, SealedStore storage, Scope scope, Instant now) throws Exception
{
if(!storage.supportsLock())
{
throw (new ExportFailure("Export refused: bucket does not support Object Lock"));
}
var settings = config.effectiveSettings();
var mode = com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(settings.export_lock_mode());
Instant until = now.plus(settings.export_retention_days(), ChronoUnit.DAYS);
String prefix = config.getExportPrefix() + "/" + DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(now)
+ "/" + scope.from() + "-" + scope.to() + "/";
QRecord row;
try(ExportFiles files = new ExportFiles())
{
String head = writeLinks(files, store, scope);
long[] checkpointIds = writeCheckpoints(files, store, scope);
Map<String, Object> manifest = new LinkedHashMap<>();
manifest.put("format", "worm-audit-export/1");
manifest.put("fromLinkId", scope.from());
manifest.put("toLinkId", scope.to());
manifest.put("linkCount", scope.count());
manifest.put("checkpointCount", scope.checkpoints());
manifest.put("files", Map.of("links.jsonl", ExportFiles.sha256(files.file("links.jsonl")),
"checkpoints.jsonl", ExportFiles.sha256(files.file("checkpoints.jsonl"))));
manifest.put("headHash", head);
String previous = scope.previous() == null ? null : scope.previous().getValueString("manifest_sha256");
manifest.put("prevManifestSha256", previous);
manifest.put("qbitVersion", WormAuditQBitProducer.VERSION);
manifest.put("createdAt", now.toString());
Files.write(files.file("manifest.json"), Jcs.bytes(manifest));
row = new QRecord().withValue("from_link_id", scope.from()).withValue("to_link_id", scope.to())
.withValue("from_checkpoint_id", checkpointIds[0]).withValue("to_checkpoint_id", checkpointIds[1])
.withValue("bucket_alias", config.getExportBucketAlias()).withValue("object_prefix", prefix)
.withValue("manifest_key", prefix + "manifest.json").withValue("manifest_sha256", ExportFiles.sha256(files.file("manifest.json")))
.withValue("prev_manifest_sha256", previous).withValue("object_count", 3L)
.withValue("lock_mode", mode.name()).withValue("retain_until", until).withValue("sealed_at", now);
if(!new SealedExportVerifier(config, storage).inspect(files, row, store).result().passed())
{
throw (new ExportFailure("Export refused: source does not match the signed chain"));
}
long bytes = 0;
Map<String, String> versions = new LinkedHashMap<>();
for(String name : ExportFiles.NAMES)
{
String version = storage.put(prefix + name, files.file(name), mode, until);
if(!ExportFiles.validVersion(version))
{
throw (new ExportFailure("Export refused: object " + name + " returned no version id"));
}
var headObject = storage.head(prefix + name);
if(!ExportFiles.locked(headObject, mode.name(), until))
{
throw (new ExportFailure("Export refused: object " + name + " did not echo " + mode.name() + " retention"));
}
if(!version.equals(headObject.getVersionId()))
{
throw (new ExportFailure("Export refused: object " + name + " version changed after upload"));
}
versions.put(name, version);
bytes += Files.size(files.file(name));
}
row.withValue("byte_count", bytes).withValue("object_versions", Jcs.string(versions));
}
row = store.insert(WormAuditTables.EXPORT, List.of(row)).getFirst();
return (new Result(row, WormAuditVerifier.number(scope.count()) + " links sealed in export " + row.getValueLong("id")
+ " (retained until " + until + ")", false));
}
/*******************************************************************************
** Stream canonical lines in pages of 1000 with batched content lookups.
*******************************************************************************/
private String writeLinks(ExportFiles files, AuditStore store, Scope scope) throws Exception
{
long cursor = scope.from() - 1;
String head = null;
try(BufferedWriter writer = Files.newBufferedWriter(files.file("links.jsonl"), StandardCharsets.UTF_8))
{
while(cursor < scope.to())
{
List<QRecord> links = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(1000, scope.to() - cursor));
if(links.isEmpty())
{
throw (new ExportFailure("Export refused: chain range is incomplete"));
}
Map<Long, QRecord> audits = AuditStore.index(store.in("audit", "id", links.stream().map(r -> r.getValueLong("audit_id")).toList()));
Map<Long, QRecord> details = ExportFiles.details(store, links);
for(QRecord link : links)
{
if(link.getValueLong("id") != cursor + 1 || link.getValueLong("id") > scope.to())
{
throw (new ExportFailure("Export refused: chain range is incomplete"));
}
byte[] content = store.canonical(audits.get(link.getValueLong("audit_id")), AuditStore.detailIds(link), details);
if(!Hashes.link(link.getValueString("prev_hash"), content).equals(link.getValueString("link_hash")))
{
throw (new ExportFailure("Export refused: content differs at link " + link.getValueLong("id")));
}
writer.write(Jcs.string(Map.of("link", ExportFiles.link(link), "content", ExportFiles.parse(content))));
writer.write('\n');
head = link.getValueString("link_hash");
cursor = link.getValueLong("id");
}
}
}
return (head);
}
/*******************************************************************************
** Stream complete signed checkpoint envelopes in keyset pages.
*******************************************************************************/
private long[] writeCheckpoints(ExportFiles files, AuditStore store, Scope scope) throws Exception
{
long first = 0;
long cursor = 0;
try(BufferedWriter writer = Files.newBufferedWriter(files.file("checkpoints.jsonl"), StandardCharsets.UTF_8))
{
while(true)
{
List<QRecord> page = checkpoints(store, scope.from(), scope.to(), cursor);
if(page.isEmpty())
{
break;
}
for(QRecord row : page)
{
first = first == 0 ? row.getValueLong("id") : first;
cursor = row.getValueLong("id");
writer.write(Jcs.string(ExportFiles.checkpoint(row)));
writer.write('\n');
}
}
}
return (new long[] { first, cursor });
}
/*******************************************************************************
** Only fixed messages constructed here may reach an alert or Result screen.
*******************************************************************************/
private static class ExportFailure extends IOException
{
/*******************************************************************************
** Carry a value-free exporter reason without a provider exception cause.
*******************************************************************************/
ExportFailure(String reason)
{
super(reason);
}
}
}
@@ -0,0 +1,72 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.export;
import java.io.InputStream;
import java.nio.file.Path;
import java.time.Instant;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
import com.kingsrook.qbits.objectstorage.api.Page;
import com.kingsrook.qbits.objectstorage.api.StoredObject;
/*******************************************************************************
** Minimal storage boundary. The application never deletes sealed objects.
*******************************************************************************/
public interface SealedStore
{
/*******************************************************************************
** Only positively confirmed Object Lock support is acceptable.
*******************************************************************************/
boolean supportsLock();
/*******************************************************************************
** Preserve unknown provider capabilities for read-only setup diagnostics.
*******************************************************************************/
default CapabilityStatus lockCapability()
{
return (supportsLock() ? CapabilityStatus.SUPPORTED : CapabilityStatus.UNSUPPORTED);
}
/*******************************************************************************
** Upload a file with the selected Object Lock mode and retention.
*******************************************************************************/
String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception;
/*******************************************************************************
** Read current object metadata, or null for an absent object.
*******************************************************************************/
StoredObject head(String key) throws Exception;
/*******************************************************************************
** Open the exact sealed version; the caller owns and closes the stream.
*******************************************************************************/
InputStream getVersion(String key, String versionId) throws Exception;
/*******************************************************************************
** List one bounded page.
*******************************************************************************/
Page<ObjectSummary> list(ListRequest request) throws Exception;
}
@@ -45,6 +45,13 @@ public class AppendOnlyGuard implements MetaDataCustomizerInterface<QTableMetaDa
@Override
public QTableMetaData customizeMetaData(QInstance instance, QTableMetaData table)
{
if(com.kingsrook.qbits.wormaudit.model.WormAuditTables.SETTINGS.equals(table.getName()))
{
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE);
table.withAuditRules(new com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules()
.withAuditLevel(com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel.FIELD));
return (table);
}
if(CORE_TABLES.contains(table.getName()) || table.getName().startsWith("worm_audit_"))
{
table.withoutCapabilities(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
@@ -81,6 +81,7 @@ public class LinkRangeWidget extends AbstractWidgetRenderer
String path = QContext.getQInstance().getTablePath(WormAuditTables.CHAIN);
String viewAll = path + "?filter=" + URLEncoder.encode(JsonUtils.toJson(filter), StandardCharsets.UTF_8);
QueryInput query = new QueryInput(WormAuditTables.CHAIN).withFilter(filter.clone().withLimit(100)).withInputSource(QInputSource.USER);
query.withShouldTranslatePossibleValues(true);
query.withPermissionEnforcement(PermissionEnforcement.ENFORCE);
return (new RenderWidgetOutput(new ChildRecordListData("Chain links", new QueryAction().execute(query),
QContext.getQInstance().getTable(WormAuditTables.CHAIN), path, viewAll, total)));
@@ -0,0 +1,116 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.model;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Validate the editable singleton, including internal insert-only seeds.
*******************************************************************************/
public class SettingsCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Only the fixed singleton may be inserted by the internal seed path.
*******************************************************************************/
@Override
public List<QRecord> preInsert(InsertInput input, List<QRecord> records, boolean preview) throws QException
{
if(records.size() != 1 || !Integer.valueOf(1).equals(records.getFirst().getValueInteger("id")))
{
throw (new QUserFacingException("WORM Audit Settings has one row only (ID 1)."));
}
if(new com.kingsrook.qbits.wormaudit.chain.AuditStore(input.getTransaction()).get(WormAuditTables.SETTINGS, 1) != null)
{
throw (new QUserFacingException("WORM Audit Settings already exists. Edit record 1 instead of adding another record."));
}
return (validate(records));
}
/*******************************************************************************
** Validate partial edits without replacing the existing settings.
*******************************************************************************/
@Override
public List<QRecord> preUpdate(UpdateInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
{
PermissionsHelper.checkTablePermissionThrowing(input, TablePermissionSubType.EDIT);
return (validate(records));
}
/*******************************************************************************
** Reject deletion even for direct core calls where capability checks are off.
*******************************************************************************/
@Override
public List<QRecord> preDelete(DeleteInput input, List<QRecord> records, boolean preview) throws QException
{
throw (new QUserFacingException("WORM Audit Settings cannot be deleted."));
}
/*******************************************************************************
** Refresh local settings after persistence; remote JVMs retain the bounded cache.
*******************************************************************************/
@Override
public List<QRecord> postUpdate(UpdateInput input, List<QRecord> records, Optional<List<QRecord>> oldRecords) throws QException
{
com.kingsrook.qbits.wormaudit.processes.VerificationRuns.config().invalidateSettings();
return (records);
}
/*******************************************************************************
** Stamp the editor and time on every valid write; core writes FIELD audits.
*******************************************************************************/
private List<QRecord> validate(List<QRecord> records) throws QException
{
for(QRecord row : records)
{
try
{
WormAuditSettings.validate(row);
}
catch(IllegalArgumentException e)
{
throw (new QUserFacingException(e.getMessage()));
}
row.setValue("modify_date", Instant.now());
row.setValue("modified_by", QContext.getQSession() == null || QContext.getQSession().getUser() == null
? "system" : QContext.getQSession().getUser().getIdReference());
}
return (records);
}
}
@@ -0,0 +1,55 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.model;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
** Reading uses the full table name; editing additionally needs settings access.
*******************************************************************************/
public class SettingsPermissions implements CustomPermissionChecker
{
/*******************************************************************************
** Enforce table permission and the separate settings permission.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
var session = QContext.getQSession();
boolean edit = input instanceof UpdateInput;
if(input instanceof InsertInput || input instanceof DeleteInput || session == null
|| !session.hasPermission(WormAuditTables.SETTINGS + (edit ? ".edit" : ".read"))
|| (edit && !session.hasPermission("worm_audit.settings")))
{
throw (new QPermissionDeniedException(edit ? "Editing WORM Audit Settings requires worm_audit_settings.edit and worm_audit.settings."
: "WORM Audit Settings requires read access; adding or deleting settings is not allowed."));
}
}
}
@@ -24,13 +24,21 @@ package com.kingsrook.qbits.wormaudit.model;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
@@ -39,7 +47,7 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDe
/*******************************************************************************
** Append-only sidecar tables. Snake-case fields map directly to DDL columns.
** Export metadata is the integration hook for phase (b); no export runs here.
** Labels also supply the standard QQQ app navigation wording.
*******************************************************************************/
public final class WormAuditTables
{
@@ -48,8 +56,9 @@ public final class WormAuditTables
public static final String EXPORT = "worm_audit_export";
public static final String VERIFICATION = "worm_audit_verification";
public static final String PURGE = "worm_audit_purge";
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE);
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE);
public static final String SETTINGS = "worm_audit_settings";
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE, SETTINGS);
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE, SETTINGS);
/*******************************************************************************
** Utility class.
@@ -67,21 +76,83 @@ public final class WormAuditTables
List<QTableMetaData> tables = new ArrayList<>();
tables.add(table(config, CHAIN, "Audit Chain Link", "id:L kind:S audit_id:L detail_ids:T prev_hash:S link_hash:S chained_at:D"));
tables.add(table(config, CHECKPOINT, "Checkpoint", "id:L from_link_id:L to_link_id:L link_count:L head_hash:S merkle_root:S algorithm:S key_id:S public_key:T signature:T created_at:D"));
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T object_versions:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
tables.add(table(config, VERIFICATION, "Verification", "id:L kind:S trigger:S started_at:D finished_at:D from_link_id:L to_link_id:L links_checked:L outcome:S break_count:L first_break_link_id:L first_break_audit_id:L first_break_reason:T export_id:L run_by:S"));
tables.add(table(config, PURGE, "Purge", "id:L requested_at:D requested_by:S cutoff:D retention_interval:S outcome:S reason:T audits_deleted:L details_deleted:L links_deleted:L anchor_link_id:L anchor_hash:S"));
tables.add(settings(config));
return (tables);
}
/*******************************************************************************
** Editable, FIELD-audited singleton with standard QQQ record sections.
*******************************************************************************/
public static QTableMetaData settings(WormAuditQBitConfig config)
{
QTableMetaData table = new QTableMetaData().withName(SETTINGS).withLabel("WORM Audit Settings")
.withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(SETTINGS))
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("WORM Audit Settings")
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS)
.withPermissionBaseName(SETTINGS).withCustomPermissionChecker(new QCodeReference(SettingsPermissions.class)))
.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE)
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.PRE_DELETE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withLabel("ID").withBackendName("id").withIsEditable(false).withDefaultValue(1));
var defaults = WormAuditSettings.defaults(config);
table.withField(new QFieldMetaData("audit_retention_years", QFieldType.INTEGER).withBackendName("audit_retention_years").withLabel("Audit retention years")
.withIsRequired(true).withDefaultValue(defaults.getValue("audit_retention_years")));
table.withField(new QFieldMetaData("export_lock_mode", QFieldType.STRING).withBackendName("export_lock_mode").withLabel("Export lock mode")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_lock_mode")));
table.withField(new QFieldMetaData("export_retention_days", QFieldType.INTEGER).withBackendName("export_retention_days").withLabel("Export retention days")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_retention_days")));
table.withField(new QFieldMetaData("checkpoint_every_links", QFieldType.INTEGER).withBackendName("checkpoint_every_links").withLabel("Checkpoint every links")
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_links")));
table.withField(new QFieldMetaData("checkpoint_every_minutes", QFieldType.INTEGER).withBackendName("checkpoint_every_minutes").withLabel("Checkpoint every minutes")
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_minutes")));
table.withField(new QFieldMetaData("chain_interval_seconds", QFieldType.INTEGER).withBackendName("chain_interval_seconds").withLabel("Chain interval seconds")
.withIsRequired(true).withDefaultValue(defaults.getValue("chain_interval_seconds")));
table.withField(new QFieldMetaData("settle_horizon_seconds", QFieldType.INTEGER).withBackendName("settle_horizon_seconds").withLabel("Settle horizon seconds")
.withIsRequired(true).withDefaultValue(defaults.getValue("settle_horizon_seconds")));
table.withField(new QFieldMetaData("late_detail_lookback_hours", QFieldType.INTEGER).withBackendName("late_detail_lookback_hours").withLabel("Late detail lookback hours")
.withIsRequired(true).withDefaultValue(defaults.getValue("late_detail_lookback_hours")));
table.withField(new QFieldMetaData("export_cron", QFieldType.STRING).withBackendName("export_cron").withLabel("Export schedule (UTC)")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_cron")));
table.withField(new QFieldMetaData("verify_cron", QFieldType.STRING).withBackendName("verify_cron").withLabel("Verify schedule (UTC)")
.withIsRequired(true).withDefaultValue(defaults.getValue("verify_cron")));
table.getField("export_lock_mode").withInlinePossibleValueSource(new QPossibleValueSource().withName(SETTINGS + "_lock_mode")
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(
new QPossibleValue<>("GOVERNANCE", "GOVERNANCE"), new QPossibleValue<>("COMPLIANCE", "COMPLIANCE"))));
for(String field : List.of("export_cron", "verify_cron", "chain_interval_seconds"))
{
table.getField(field).withHelpContent(new QHelpContent("Takes effect after restart. Cron schedules use UTC and Quartz syntax."));
}
table.getField("audit_retention_years").withHelpContent(new QHelpContent("Used for the next eligible purge date. The database purge interval changes only through a reviewed migration."));
table.withField(new QFieldMetaData("notes", QFieldType.TEXT).withLabel("Notes").withBackendName("notes"));
table.withField(new QFieldMetaData("modify_date", QFieldType.DATE_TIME).withLabel("Modified").withBackendName("modify_date").withIsEditable(false));
table.withField(new QFieldMetaData("modified_by", QFieldType.STRING).withLabel("Modified by").withBackendName("modified_by").withIsEditable(false));
table.withSection(new QFieldSection("retention", "Retention", new QIcon("lock"), Tier.T1,
List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days")).withGridColumns(12));
table.withSection(new QFieldSection("cadence", "Chain and checkpoints", new QIcon("link"), Tier.T2,
List.of("checkpoint_every_links", "checkpoint_every_minutes", "settle_horizon_seconds", "late_detail_lookback_hours")).withGridColumns(12));
table.withSection(new QFieldSection("schedules", "Schedules (after restart)", new QIcon("schedule"), Tier.T2,
List.of("chain_interval_seconds", "export_cron", "verify_cron")).withGridColumns(12));
table.withSection(new QFieldSection("changes", "Changes", new QIcon("edit_note"), Tier.T2,
List.of("notes", "modify_date", "modified_by")).withGridColumns(12));
return (table);
}
/*******************************************************************************
** Common metadata for a process-written table.
*******************************************************************************/
private static QTableMetaData table(WormAuditQBitConfig config, String name, String label, String fields)
{
QTableMetaData table = new QTableMetaData().withName(name).withLabel(label)
QTableMetaData table = new QTableMetaData().withName(name).withLabel(navigationLabel(name))
.withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(name))
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("%s")
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat(label + " #%s")
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS).withPermissionBaseName(name));
for(String spec : fields.split(" "))
@@ -94,7 +165,7 @@ public final class WormAuditTables
case "T" -> QFieldType.TEXT;
default -> QFieldType.STRING;
};
table.withField(new QFieldMetaData(parts[0], type).withBackendName(parts[0]));
table.withField(new QFieldMetaData(parts[0], type).withBackendName(parts[0]).withLabel(fieldLabel(parts[0])));
}
for(String reference : List.of("from_link_id", "to_link_id", "first_break_link_id", "anchor_link_id"))
{
@@ -114,11 +185,127 @@ public final class WormAuditTables
{
table.getField("export_id").withPossibleValueSourceName(EXPORT);
}
table.withSection(new QFieldSection("record", label, new QIcon("fact_check"), Tier.T1, new ArrayList<>(table.getFields().keySet())));
switch(name)
{
case CHAIN -> table.withRecordLabelFormat("Link #%s");
case EXPORT -> table.withRecordLabelFormatAndFields("Export #%s (links %s–%s)", "id", "from_link_id", "to_link_id");
case VERIFICATION, PURGE -> table.withRecordLabelFormatAndFields(label + " #%s (%s)", "id", "outcome");
default ->
{
}
}
if(table.getFields().containsKey("kind"))
{
table.getField("kind").withInlinePossibleValueSource(new QPossibleValueSource().withName(name + "_kind")
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(new QPossibleValue<>("AUDIT", "Audit"),
new QPossibleValue<>("LATE_DETAIL", "Late detail"), new QPossibleValue<>("CHAIN", "Chain"),
new QPossibleValue<>("SEALED_EXPORT", "Sealed export"))));
}
if(table.getFields().containsKey("outcome"))
{
table.getField("outcome").withInlinePossibleValueSource(new QPossibleValueSource().withName(name + "_outcome")
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(new QPossibleValue<>("PASS", "Pass"),
new QPossibleValue<>("FAIL", "Fail"), new QPossibleValue<>("PURGED", "Purged"), new QPossibleValue<>("REFUSED", "Refused"))));
}
if(VERIFICATION.equals(name))
{
table.withSection(new QFieldSection("result", "Result", new QIcon("fact_check"), Tier.T2,
List.of("id", "outcome", "break_count", "first_break_link_id", "first_break_audit_id", "first_break_reason")).withGridColumns(12));
table.withSection(new QFieldSection("scope", "Scope", new QIcon("link"), Tier.T2,
List.of("kind", "trigger", "from_link_id", "to_link_id", "links_checked", "export_id")).withGridColumns(12));
table.withSection(new QFieldSection("run", "Run", new QIcon("schedule"), Tier.T2,
List.of("started_at", "finished_at", "run_by")).withGridColumns(12));
}
else
{
table.withSection(new QFieldSection("record", label, new QIcon("fact_check"), Tier.T1,
new ArrayList<>(table.getFields().keySet())).withGridColumns(12));
}
if(List.of(CHECKPOINT, EXPORT, VERIFICATION).contains(name))
{
table.withSection(new QFieldSection("links", new QIcon("link"), Tier.T2).withLabel("Chain links").withWidgetName(name + "_links"));
table.withSection(new QFieldSection("links", new QIcon("link"), Tier.T2).withLabel("Chain links")
.withWidgetName(name + "_links").withGridColumns(12));
}
return (table);
}
/*******************************************************************************
** Core RC.5 shares the table label with its navigation tree node.
*******************************************************************************/
public static String navigationLabel(String name)
{
return (switch(name)
{
case SETTINGS -> "Settings";
case CHAIN -> "Audit Chain";
case CHECKPOINT -> "Checkpoints";
case EXPORT -> "Sealed Exports";
case VERIFICATION -> "Verifications";
default -> "Purges";
});
}
/*******************************************************************************
** Plain record labels independent of physical column names.
*******************************************************************************/
private static String fieldLabel(String name)
{
return (switch(name)
{
case "id" -> "ID";
case "kind" -> "Kind";
case "audit_id" -> "Audit";
case "detail_ids" -> "Detail ids";
case "prev_hash" -> "Previous hash";
case "link_hash" -> "Link hash";
case "chained_at" -> "Chained";
case "from_link_id" -> "From link";
case "to_link_id" -> "To link";
case "link_count" -> "Links";
case "head_hash" -> "Head hash";
case "merkle_root" -> "Merkle root";
case "algorithm" -> "Algorithm";
case "key_id" -> "Key ID";
case "public_key" -> "Public key";
case "signature" -> "Signature";
case "created_at" -> "Created";
case "from_checkpoint_id" -> "From checkpoint";
case "to_checkpoint_id" -> "To checkpoint";
case "bucket_alias" -> "Bucket alias";
case "object_prefix" -> "Object prefix";
case "object_versions" -> "Object versions";
case "manifest_key" -> "Manifest key";
case "manifest_sha256" -> "Manifest SHA-256";
case "prev_manifest_sha256" -> "Previous manifest SHA-256";
case "object_count" -> "Objects";
case "byte_count" -> "Bytes";
case "lock_mode" -> "Lock mode";
case "retain_until" -> "Retain until";
case "sealed_at" -> "Sealed";
case "trigger" -> "Trigger";
case "started_at" -> "Started";
case "finished_at" -> "Finished";
case "links_checked" -> "Links checked";
case "outcome" -> "Outcome";
case "break_count" -> "Breaks";
case "first_break_link_id" -> "First break link";
case "first_break_audit_id" -> "First break audit";
case "first_break_reason" -> "First break";
case "export_id" -> "Sealed export";
case "run_by" -> "Run by";
case "requested_at" -> "Requested";
case "requested_by" -> "Requested by";
case "cutoff" -> "Cutoff";
case "retention_interval" -> "Retention interval";
case "reason" -> "Reason";
case "audits_deleted" -> "Audits deleted";
case "details_deleted" -> "Details deleted";
case "links_deleted" -> "Links deleted";
case "anchor_link_id" -> "Anchor link";
case "anchor_hash" -> "Anchor hash";
default -> throw new IllegalArgumentException("Missing field label: " + name);
});
}
}
@@ -0,0 +1,377 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.processes;
import java.io.Serializable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.export.SealedStore;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.permissions.UseOtherPermissionNameCustomPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunInput;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunOutput;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Standard Review/Result actions for sealing and verifying exports.
*******************************************************************************/
public final class ExportProcesses
{
public static final String EXPORT = "worm_audit_export_now";
public static final String VERIFY = "worm_audit_verify_export";
public static final String DAILY = "worm_audit_export_daily";
public static final String PERMISSION = "worm_audit.export";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private ExportProcesses()
{
}
/*******************************************************************************
** Export is a table action; verification requires one or more selected exports.
*******************************************************************************/
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
{
var settings = config.effectiveSettings();
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.EXPORT);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
QProcessMetaData export = StreamedETLWithFrontendProcess.defineProcessMetaData(ExportExtract.class, ExportReview.class, ExportLoad.class, new HashMap<>(values))
.withName(EXPORT).withLabel("Export now").withTableName(WormAuditTables.EXPORT).withPermissionRules(permission(PERMISSION));
QProcessMetaData verify = StreamedETLWithFrontendProcess.defineProcessMetaData(VerifyExtract.class, VerifyReview.class, VerifyLoad.class, new HashMap<>(values))
.withName(VERIFY).withLabel("Verify sealed export").withTableName(WormAuditTables.EXPORT).withMinInputRecords(1)
.withPermissionRules(permission(WormAuditProcesses.PERMISSION));
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily sealed audit export")
.withStep(new QBackendStepMetaData().withName("export").withCode(new QCodeReference(Daily.class)))
.withPermissionRules(permission(PERMISSION)).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withCronExpression(settings.export_cron()).withCronTimeZoneId("UTC"));
return (List.of(export, verify, daily));
}
/*******************************************************************************
** Reuse the phase (a) explicit permission-name checker.
*******************************************************************************/
private static QPermissionRules permission(String name)
{
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
.withCustomPermissionChecker(UseOtherPermissionNameCustomPermissionChecker.build(name)));
}
/*******************************************************************************
** Plain text summaries must not acquire the framework's default zero count.
*******************************************************************************/
private static ProcessSummaryLine text(Status status, String message)
{
ProcessSummaryLine line = new ProcessSummaryLine(status, message);
line.setCount(null);
return (line);
}
/*******************************************************************************
** One synthetic work item allows the first export on an empty export table.
*******************************************************************************/
public static class ExportExtract extends AbstractExtractStep
{
/*******************************************************************************
** Count the single export operation, not previous export rows.
*******************************************************************************/
@Override
public Integer doCount(RunBackendStepInput input)
{
return (1);
}
/*******************************************************************************
** Feed one work item through the standard stream.
*******************************************************************************/
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
getRecordPipe().addRecord(new QRecord().withValue("id", 0L));
}
}
/*******************************************************************************
** Review is read-only; the writer reselects the suffix after confirmation.
*******************************************************************************/
public static class ExportReview extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private String message;
private boolean failed;
/*******************************************************************************
** Read the planned range once without uploading or inserting.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
try
{
message = SealedExporter.scope(new AuditStore(null)).review();
failed = false;
}
catch(QException e)
{
message = "Cannot select a contiguous export range";
failed = true;
}
output.getRecords().addAll(input.getRecords());
}
/*******************************************************************************
** Show the range, including the explicit no-checkpoint case.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
return (new ArrayList<>(List.of(text(failed ? Status.ERROR : Status.INFO, message == null ? "Nothing to seal: no new checkpoint" : message))));
}
}
/*******************************************************************************
** Seal once after the stream is complete.
*******************************************************************************/
public static class ExportLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final SealedStore storage;
private SealedExporter.Result result;
/*******************************************************************************
** Framework constructor.
*******************************************************************************/
public ExportLoad()
{
this(null);
}
/*******************************************************************************
** Storage-injected action for unit tests.
*******************************************************************************/
public ExportLoad(SealedStore storage)
{
this.storage = storage;
}
/*******************************************************************************
** Forward the work item without retaining content.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output)
{
output.getRecords().addAll(input.getRecords());
}
/*******************************************************************************
** The same execution path as the daily schedule.
*******************************************************************************/
@Override
public void postRun(BackendStepPostRunInput input, BackendStepPostRunOutput output)
{
result = new SealedExporter(VerificationRuns.config(), storage).run("USER", Instant.now());
}
/*******************************************************************************
** Result text is value-free even when the provider fails.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
return (result == null ? new ArrayList<>() : new ArrayList<>(List.of(text(result.failed() ? Status.ERROR : Status.OK, result.message()))));
}
}
/*******************************************************************************
** Respect core's selected-record/filter interpretation; no implicit all-export run.
*******************************************************************************/
public static class VerifyExtract extends ExtractViaQueryStep
{
/*******************************************************************************
** Deterministic selection with full text fields from the export table.
*******************************************************************************/
@Override
public QQueryFilter getQueryFilter(RunBackendStepInput input) throws QException
{
input.addValue(FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
QQueryFilter filter = super.getQueryFilter(input).clone();
filter.setOrderBys(List.of(new QFilterOrderBy("id", true)));
return (filter);
}
}
/*******************************************************************************
** Review selected exports without reading their content.
*******************************************************************************/
public static class VerifyReview extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private long count;
/*******************************************************************************
** Count and forward the selected export records.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output)
{
count += input.getRecords().size();
output.getRecords().addAll(input.getRecords());
}
/*******************************************************************************
** One Review summary.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
return (new ArrayList<>(List.of(text(Status.INFO, WormAuditVerifier.number(count) + " sealed exports will be verified"))));
}
}
/*******************************************************************************
** Verify each selected export and persist its independent result.
*******************************************************************************/
public static class VerifyLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final SealedStore storage;
private final ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
/*******************************************************************************
** Framework constructor.
*******************************************************************************/
public VerifyLoad()
{
this(null);
}
/*******************************************************************************
** Storage-injected action for unit tests.
*******************************************************************************/
public VerifyLoad(SealedStore storage)
{
this.storage = storage;
}
/*******************************************************************************
** Process the current page without retaining exported content.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord selected : input.getRecords())
{
QRecord row = new AuditStore(null).get(WormAuditTables.EXPORT, selected.getValueLong("id"));
if(row == null)
{
lines.add(text(Status.ERROR, "Selected export " + selected.getValueLong("id") + " is missing"));
continue;
}
var report = new SealedExportVerifier(VerificationRuns.config(), storage).verify(row, "USER");
if(report.result().passed())
{
lines.add(text(Status.OK, WormAuditVerifier.number(report.result().getLinksChecked()) + " links verified in export " + row.getValueLong("id")));
}
report.result().getBreaks().forEach(b -> lines.add(text(Status.ERROR, b.reason())));
if(report.purged() > 0 && report.result().passed())
{
lines.add(text(Status.INFO, WormAuditVerifier.number(report.purged()) + " links verified from the sealed copy only (purged from the live database)"));
}
}
}
/*******************************************************************************
** Exact per-link differences and purged-copy information.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
return (lines);
}
}
/*******************************************************************************
** Daily export sends heartbeat even when sealing fails.
*******************************************************************************/
public static class Daily implements BackendStep
{
/*******************************************************************************
** Run the exporter with the installed host configuration.
*******************************************************************************/
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output)
{
var result = new SealedExporter(VerificationRuns.config()).run("SCHEDULE", Instant.now());
output.addValue("outcome", result.failed() ? "FAIL" : "PASS");
output.addValue("result", result.message());
}
}
}
@@ -75,11 +75,11 @@ public final class VerificationRuns
{
if(result.passed())
{
QAlerts.resolve("worm_audit.chain.break", "chain", "Audit chain verification passed");
QAlerts.resolve("wormaudit.chain.break", "chain", "Audit chain verification passed");
}
else
{
QAlerts.raise(new AlertSignal("worm_audit.chain.break").withDedupKey("chain")
QAlerts.raise(new AlertSignal("wormaudit.chain.break").withDedupKey("chain")
.withTitle(result.getBreaks().getFirst().reason()).withRecordRef(new AlertRecordRef(WormAuditTables.VERIFICATION, row.getValue("id"))));
}
}
@@ -89,7 +89,7 @@ public final class VerificationRuns
{
if(scheduled)
{
QAlerts.heartbeat("worm_audit.verify.daily");
QAlerts.heartbeat("wormaudit.verify.daily");
}
}
}
@@ -99,10 +99,28 @@ public final class VerificationRuns
** Persist identifiers and a value-free first reason, never audit content.
*******************************************************************************/
public static QRecord save(VerificationResult result, String trigger) throws QException
{
return (save(result, trigger, null));
}
/*******************************************************************************
** Append a chain or sealed-export result using the same immutable schema.
*******************************************************************************/
public static QRecord save(VerificationResult result, String trigger, Long exportId) throws QException
{
return (save(result, trigger, exportId, Instant.now()));
}
/*******************************************************************************
** Clock-injected persistence for genuine historical verification runs.
*******************************************************************************/
public static QRecord save(VerificationResult result, String trigger, Long exportId, Instant finishedAt) throws QException
{
String user = QContext.getQSession() == null || QContext.getQSession().getUser() == null ? "system" : QContext.getQSession().getUser().getIdReference();
QRecord row = new QRecord().withValue("kind", "CHAIN").withValue("trigger", trigger)
.withValue("started_at", result.getStartedAt()).withValue("finished_at", Instant.now())
QRecord row = new QRecord().withValue("kind", exportId == null ? "CHAIN" : "SEALED_EXPORT").withValue("export_id", exportId).withValue("trigger", trigger)
.withValue("started_at", result.getStartedAt()).withValue("finished_at", finishedAt)
.withValue("from_link_id", result.getFromLinkId()).withValue("to_link_id", result.getToLinkId())
.withValue("links_checked", result.getLinksChecked()).withValue("outcome", result.passed() ? "PASS" : "FAIL")
.withValue("break_count", result.getBreakCount()).withValue("run_by", user);
@@ -89,6 +89,7 @@ public final class WormAuditProcesses
*******************************************************************************/
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
{
var settings = config.effectiveSettings();
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.CHAIN);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.CHAIN);
@@ -100,11 +101,11 @@ public final class WormAuditProcesses
QProcessMetaData chain = new QProcessMetaData().withName(CHAIN).withLabel("Chain settled audit records")
.withStep(new QBackendStepMetaData().withName("chain").withCode(new QCodeReference(ChainTick.class)))
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withRepeatSeconds(config.getChainIntervalSeconds()).withInitialDelaySeconds(config.getChainIntervalSeconds()));
.withRepeatSeconds(settings.chain_interval_seconds()).withInitialDelaySeconds(settings.chain_interval_seconds()));
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily audit chain verification")
.withStep(new QBackendStepMetaData().withName("verify").withCode(new QCodeReference(Daily.class)))
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withCronExpression(config.getVerifyCron()).withCronTimeZoneId("UTC"));
.withCronExpression(settings.verify_cron()).withCronTimeZoneId("UTC"));
return (List.of(verify, chain, daily));
}
@@ -41,12 +41,30 @@ public class VerificationResult
private final List<Break> breaks = new ArrayList<>();
private final List<String> warnings = new ArrayList<>();
private final Instant startedAt = Instant.now();
private final Instant startedAt;
private long breakCount;
private long linksChecked;
private Long fromLinkId;
private Long toLinkId;
/*******************************************************************************
** Use the production clock by default.
*******************************************************************************/
public VerificationResult()
{
this(Instant.now());
}
/*******************************************************************************
** Bind the verification start to the caller's clock.
*******************************************************************************/
public VerificationResult(Instant startedAt)
{
this.startedAt = startedAt;
}
/*******************************************************************************
** Add a finding in deterministic verification order.
*******************************************************************************/
@@ -54,13 +54,28 @@ public class WormAuditVerifier
{
public static final int PAGE_SIZE = 1000;
private final WormAuditQBitConfig config;
private final int pageSize;
/*******************************************************************************
** Bind trust pins and settle horizon.
*******************************************************************************/
public WormAuditVerifier(WormAuditQBitConfig config)
{
this(config, PAGE_SIZE);
}
/*******************************************************************************
** Smaller pages support constrained hosts and boundary regression tests.
*******************************************************************************/
public WormAuditVerifier(WormAuditQBitConfig config, int pageSize)
{
if(pageSize < 1 || pageSize > PAGE_SIZE)
{
throw (new IllegalArgumentException("Verification page size must be between 1 and 1000"));
}
this.config = config;
this.pageSize = pageSize;
}
@@ -77,6 +92,15 @@ public class WormAuditVerifier
** Null means full verification; an empty set is an empty selection.
*******************************************************************************/
public VerificationResult verify(Set<Long> selection) throws QException
{
return (verify(selection, Instant.now()));
}
/*******************************************************************************
** Clock-injected verification uses the same horizon and snapshot as production.
*******************************************************************************/
public VerificationResult verify(Set<Long> selection, Instant now) throws QException
{
WormAuditChainer.JVM_LOCK.lock();
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
@@ -87,7 +111,7 @@ public class WormAuditVerifier
{
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
}
VerificationResult result = verify(new AuditStore(transaction), selection, Instant.now());
VerificationResult result = verify(new AuditStore(transaction), selection, now);
transaction.commit();
return (result);
}
@@ -107,9 +131,10 @@ public class WormAuditVerifier
/*******************************************************************************
** Verify one consistent storage snapshot.
*******************************************************************************/
private VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
{
VerificationResult result = new VerificationResult();
var settings = config.effectiveSettings();
VerificationResult result = new VerificationResult(now);
if(config.getSigningKey() == null)
{
result.warn("no signing key configured");
@@ -125,7 +150,7 @@ public class WormAuditVerifier
CheckpointCursor checkpoints = new CheckpointCursor(store, selection, anchor, result);
while(true)
{
List<QRecord> page = store.page(WormAuditTables.CHAIN, lastId, PAGE_SIZE);
List<QRecord> page = store.page(WormAuditTables.CHAIN, lastId, pageSize);
if(page.isEmpty())
{
break;
@@ -150,9 +175,9 @@ public class WormAuditVerifier
}
Map<Long, QRecord> details = new HashMap<>();
List<Long> ids = new ArrayList<>(detailIds);
for(int start = 0; start < ids.size(); start += PAGE_SIZE)
for(int start = 0; start < ids.size(); start += pageSize)
{
details.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + PAGE_SIZE, ids.size())))));
details.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + pageSize, ids.size())))));
}
for(QRecord link : page)
{
@@ -180,8 +205,8 @@ public class WormAuditVerifier
previous = link.getValueString("link_hash");
lastId = id;
}
checkpoints.through(lastId);
}
// The main scan's page is out of scope before checkpoint pages are read.
checkpoints.through(Long.MAX_VALUE);
result.scope(checked, first, last);
if(missing != null)
@@ -190,15 +215,19 @@ public class WormAuditVerifier
}
if(selection == null)
{
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
if(store.isRdbms())
{
store.completeness(cutoff, result);
}
else
else if(store.isMemory())
{
completeness(store, store.all(WormAuditTables.CHAIN), cutoff, result);
}
else
{
pagedCompleteness(store, cutoff, result);
}
}
return (result);
}
@@ -263,6 +292,73 @@ public class WormAuditVerifier
}
/*******************************************************************************
** Portable completeness for non-SQL backends. Trade repeated scans for bounded
** memory; only the memory backend uses the whole-chain coverage index.
*******************************************************************************/
private void pagedCompleteness(AuditStore store, Instant cutoff, VerificationResult result) throws QException
{
for(String table : List.of("audit", "auditDetail"))
{
long after = 0;
while(true)
{
List<QRecord> rows = store.page(table, after, pageSize);
if(rows.isEmpty())
{
break;
}
Map<Long, Integer> counts = new HashMap<>();
for(QRecord row : rows)
{
QRecord audit = table.equals("audit") ? row : store.get("audit", row.getValueLong("auditId"));
if(audit == null || audit.getValueInstant("timestamp") == null || audit.getValueInstant("timestamp").isBefore(cutoff))
{
counts.put(row.getValueLong("id"), 0);
if(table.equals("audit") && audit.getValueInstant("timestamp") == null)
{
result.add(null, row.getValueLong("id"), "audit " + number(row.getValueLong("id")) + " has no timestamp");
}
}
}
long linkAfter = 0;
while(!counts.isEmpty())
{
List<QRecord> links = store.page(WormAuditTables.CHAIN, linkAfter, pageSize);
if(links.isEmpty())
{
break;
}
for(QRecord link : links)
{
if(table.equals("audit"))
{
if("AUDIT".equals(link.getValueString("kind")))
{
counts.computeIfPresent(link.getValueLong("audit_id"), (id, count) -> count + 1);
}
}
else
{
try
{
AuditStore.detailIds(link).forEach(id -> counts.computeIfPresent(id, (key, count) -> count + 1));
}
catch(IllegalArgumentException e)
{
// Malformed coverage was reported by content verification.
}
}
}
linkAfter = links.getLast().getValueLong("id");
}
counts.forEach((id, count) -> coverage(table.equals("audit") ? "audit" : "audit_detail", id, count, result));
after = rows.getLast().getValueLong("id");
}
}
}
/*******************************************************************************
** Every settled header and detail must occur once; orphan details also fail.
*******************************************************************************/
@@ -405,7 +501,7 @@ public class WormAuditVerifier
result.add(null, null, prefix + "signature does not verify");
}
if(from < 1 || to < from || checkpoint.getValueLong("link_count") != to - from + 1
|| to - from + 1 > config.getCheckpointEveryLinks())
|| to - from + 1 > Integer.MAX_VALUE)
{
result.add(null, null, prefix + "invalid link range or count");
return;
@@ -429,7 +525,7 @@ public class WormAuditVerifier
String head = null;
while(cursor < to)
{
List<QRecord> page = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(PAGE_SIZE, to - cursor));
List<QRecord> page = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(pageSize, to - cursor));
if(page.isEmpty())
{
break;
@@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS public.worm_audit_export (
to_checkpoint_id bigint,
bucket_alias text,
object_prefix text,
object_versions text,
manifest_key text,
manifest_sha256 text,
prev_manifest_sha256 text,
@@ -323,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
-- changeset worm-audit:10
-- Editable singleton: deliberately excluded from append-only triggers.
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
id integer PRIMARY KEY CHECK (id = 1),
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
notes text,
modify_date timestamptz NOT NULL DEFAULT now(),
modified_by text NOT NULL DEFAULT current_user
);
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM ${app_role};
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO ${app_role};
INSERT INTO public.worm_audit_settings
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
export_cron, verify_cron)
SELECT ${settings_values}
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
@@ -0,0 +1,198 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.net.URI;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.Capability;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
import com.kingsrook.qbits.objectstorage.config.AddressingStyle;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.ChecksumMode;
import com.kingsrook.qbits.objectstorage.config.ProviderType;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
import com.kingsrook.qbits.objectstorage.internal.CapabilityProbe;
import com.kingsrook.qbits.objectstorage.internal.CapabilityStore;
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.export.ExportFiles;
import com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.S3Exception;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Real Object Lock acceptance. Missing Docker skips locally and fails in CI.
*******************************************************************************/
@ExtendWith(RequiresDockerCondition.class)
class MinioExportAcceptanceTest
{
/*******************************************************************************
** Production adapter, GOVERNANCE and COMPLIANCE retention, version preservation and UI findings.
*******************************************************************************/
@Test
void lockedRoundTripVersionRetentionLiveTamperAndUnsupportedBucket() throws Exception
{
String image = System.getProperty("qqq.test.minioImage",
"cgr.dev/chainguard/minio@sha256:dc455cdd4db6e390b1b26514f15ff37681a777bf8dab41aac445a0259f751e56");
try(GenericContainer<?> minio = new GenericContainer<>(image)
.withEnv("MINIO_ROOT_USER", "worm-test-access").withEnv("MINIO_ROOT_PASSWORD", "worm-test-secret")
.withCommand("server", "/tmp/minio-data").withExposedPorts(9000)
.waitingFor(Wait.forHttp("/minio/health/live").forPort(9000).withStartupTimeout(Duration.ofMinutes(2))))
{
minio.start();
String endpoint = "http://" + minio.getHost() + ":" + minio.getMappedPort(9000);
try(S3Client s3 = S3Client.builder().endpointOverride(URI.create(endpoint)).region(Region.US_EAST_1).forcePathStyle(true)
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("worm-test-access", "worm-test-secret")))
.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
.responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED).build())
{
s3.createBucket(b -> b.bucket("worm-locked").objectLockEnabledForBucket(true));
s3.createBucket(b -> b.bucket("worm-plain"));
var config = ChainVerificationTest.signed();
var instance = init(config);
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(ProcessesAndAlertsTest.Sink.class));
ProcessesAndAlertsTest.Sink.raised.clear();
System.setProperty("worm.test.access", "worm-test-access");
System.setProperty("worm.test.secret", "worm-test-secret");
ObjectStorage.configureStandalone(new ObjectStorageQBitConfig().withBackendName("memory")
.withAllowedSecretRefPatterns(List.of("prop.worm.test.*"))
.withConnection(new StorageConnectionConfig("minio").withProviderType(ProviderType.MINIO).withEndpoint(endpoint)
.withRegion("us-east-1").withAddressingStyle(AddressingStyle.PATH).withChecksumMode(ChecksumMode.WHEN_REQUIRED)
.withAccessKeyRef("${prop.worm.test.access}").withSecretKeyRef("${prop.worm.test.secret}"))
.withBucket(new StorageBucketConfig("worm_audit").withConnection("minio").withBucketName("worm-locked").withRole(BucketRole.BACKUP).withIndexed(false))
.withBucket(new StorageBucketConfig("plain").withConnection("minio").withBucketName("worm-plain").withRole(BucketRole.BACKUP).withIndexed(false)));
CapabilityProbe.probeConnection("minio", false);
assertThat(ObjectStorage.asSystem("test", "1").bucket("worm_audit").capability(Capability.OBJECT_LOCK)).isEqualTo(CapabilityStatus.SUPPORTED);
assertThat(ObjectStorage.asSystem("test", "1").bucket("plain").capability(Capability.OBJECT_LOCK)).isEqualTo(CapabilityStatus.UNSUPPORTED);
audit(1);
detail(1, 1);
new WormAuditChainer(config).chainOnce();
Instant now = Instant.now();
var exported = new SealedExporter(config).run("USER", now);
assertThat(exported.failed()).as(exported.message()).isFalse();
var row = exported.row();
var storage = new ObjectStorageSealedStore("worm_audit", "acceptance");
assertThat(storage.list(ListRequest.prefix(row.getValueString("object_prefix"))).getItems()).hasSize(3);
var verifier = new SealedExportVerifier(config);
assertThat(verifier.verify(row, "USER").result().passed()).isTrue();
for(String name : ExportFiles.NAMES)
{
String key = row.getValueString("object_prefix") + name;
var head = storage.head(key);
assertThat(head.getLockMode()).isEqualTo("GOVERNANCE");
assertThat(Duration.between(now.plus(Duration.ofDays(30)), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
String originalVersion = head.getVersionId();
assertThat(originalVersion).isNotBlank();
assertThat(ExportFiles.parse(row.getValueString("object_versions").getBytes(java.nio.charset.StandardCharsets.UTF_8)))
.containsEntry(name, originalVersion);
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion).bypassGovernanceRetention(false)))
.isInstanceOf(S3Exception.class);
byte[] original = s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray();
// S3 overwrites create new versions; they cannot replace the locked version.
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("GOVERNANCE")
.objectLockRetainUntilDate(row.getValueInstant("retain_until")), RequestBody.fromString("different replacement bytes"));
assertThat(replacement.versionId()).isNotEqualTo(originalVersion);
assertThat(s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray()).isEqualTo(original);
try(var sealed = storage.getVersion(key, originalVersion))
{
assertThat(sealed.readAllBytes()).isEqualTo(original);
}
var report = verifier.verify(row, "USER");
assertThat(report.result().passed()).isFalse();
assertThat(report.result().getLinksChecked()).isEqualTo(1);
assertThat(report.result().getBreaks()).allSatisfy(b -> assertThat(b.reason()).contains("was overwritten after sealing"));
assertThat(report.result().getBreaks().toString()).contains("object " + name + " was overwritten after sealing (current version "
+ replacement.versionId() + ", sealed version " + originalVersion + " is still retained under Object Lock)");
}
tamper("auditDetail", 1, "newValue", "tampered live value");
var load = new ExportProcesses.VerifyLoad();
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString())
.contains("ERROR", "link 1 (audit 1): live content differs from the sealed copy");
tamper("auditDetail", 1, "newValue", "new");
audit(2);
new WormAuditChainer(config).chainOnce();
config.withExportLockMode("COMPLIANCE");
var compliance = new SealedExporter(config).run("USER", Instant.now());
assertThat(compliance.failed()).isFalse();
assertThat(compliance.row().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
assertThat(verifier.verify(compliance.row(), "USER").result().passed()).isTrue();
for(String name : ExportFiles.NAMES)
{
String key = compliance.row().getValueString("object_prefix") + name;
var head = storage.head(key);
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(head.getVersionId())))
.isInstanceOf(S3Exception.class);
}
audit(3);
new WormAuditChainer(config).chainOnce();
config.withExportBucketAlias("plain");
var failed = new SealedExporter(config).run("SCHEDULE", Instant.now());
assertThat(failed.failed()).isTrue();
assertThat(failed.message()).contains("does not support Object Lock");
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(2);
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(1);
}
finally
{
ObjectStorageRuntime.reset();
CapabilityStore.clearCache();
System.clearProperty("worm.test.access");
System.clearProperty("worm.test.secret");
}
}
}
}
@@ -55,7 +55,7 @@ import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Tests and fixtures for phi canary test.
*******************************************************************************/
class PhiCanaryTest
public class PhiCanaryTest
{
// RC.5 has AiFieldMetaData but not DataClassification; use a test-local marker.
/*******************************************************************************
@@ -93,6 +93,10 @@ class PhiCanaryTest
{
for(String column : List.of("oldValue", "newValue"))
{
if(detail.getValueString(column) == null)
{
continue;
}
try
{
Mac mac = Mac.getInstance("HmacSHA256");
@@ -113,7 +113,6 @@ class PostgresAcceptanceTest
void database() throws Exception
{
config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
try(var c = owner())
{
sql(
@@ -124,6 +123,7 @@ class PostgresAcceptanceTest
c,
"DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='app') THEN"
+ " CREATE ROLE app LOGIN PASSWORD 'fixture'; END IF; END $$");
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
RdbmsTestSupport.createTables(c, instance, true);
String ddl = WormAuditDdl.postgres(config);
sql(c, ddl);
@@ -133,6 +133,39 @@ class PostgresAcceptanceTest
lookups();
}
/*******************************************************************************
** Home setup probes see dropped/disabled guards, missing purge and runtime ownership.
** This test uses the same Docker condition as the existing PostgreSQL suite.
*******************************************************************************/
@Test
void homeSetupWarnings() throws Exception
{
var storage = new com.kingsrook.qbits.wormaudit.demo.MemorySealedStore();
var clean = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
assertThat(clean.warnings()).contains("Daily verification has never run").noneMatch(w -> w.contains("guard") && w.contains("settings"));
try(var connection = owner())
{
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=7 WHERE id=1");
config.invalidateSettings();
assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings())
.anyMatch(w -> w.contains("purge function's compiled interval (6 years)"));
sql(connection, WormAuditDdl.postgres(config));
assertThat(count(connection, "SELECT audit_retention_years FROM worm_audit_settings WHERE id=1")).isEqualTo(7);
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=6 WHERE id=1");
config.invalidateSettings();
sql(connection, "DROP TRIGGER worm_audit_row_guard ON audit_detail");
sql(connection, "ALTER TABLE worm_audit_chain DISABLE TRIGGER worm_audit_truncate_guard");
sql(connection, "DROP FUNCTION public.worm_audit_purge(timestamptz)");
}
var missing = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
assertThat(missing.warnings()).contains("Missing or disabled worm_audit_row_guard on audit_detail",
"Missing or disabled worm_audit_truncate_guard on worm_audit_chain", "Retention purge function is missing");
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword(), config, true);
QContext.init(instance, new QSession());
assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings())
.contains("Runtime role owns audit table audit", "Runtime role owns audit table worm_audit_purge");
}
/*******************************************************************************
** Verify grants triggers lookup rules and spoofed flag.
*******************************************************************************/
@@ -313,7 +346,7 @@ class PostgresAcceptanceTest
void prefixCannotSplitAnAuditAndAdvisoryLockSkipsTick() throws Exception
{
// This retention fixture deliberately polls a detail for a six-year-old header.
config.withLateDetailLookbackHours(24 * 365 * 10);
setting("late_detail_lookback_hours", 24 * 365 * 10);
audit(1);
new WormAuditChainer(config).chainOnce();
try(var c = owner())
@@ -487,7 +520,7 @@ class PostgresAcceptanceTest
void chainerUsesUtcCutoffInChicago() throws Exception
{
var original = java.util.TimeZone.getDefault();
config.withSettleHorizonSeconds(60);
setting("settle_horizon_seconds", 60);
try(var connection = owner())
{
sql(connection, "ALTER TABLE audit ALTER COLUMN timestamp TYPE timestamp without time zone USING timestamp AT TIME ZONE 'UTC'");
@@ -518,7 +551,7 @@ class PostgresAcceptanceTest
@Test
void postgresLateDetailWindowAndCompleteness() throws Exception
{
config.withSettleHorizonSeconds(60);
setting("settle_horizon_seconds", 60);
var now = java.time.Instant.now().truncatedTo(java.time.temporal.ChronoUnit.SECONDS);
var cutoff = now.minusSeconds(60);
try(var connection = owner(); var statement = connection.prepareStatement(
@@ -549,4 +582,16 @@ class PostgresAcceptanceTest
assertThat(new WormAuditVerifier(config).verify().getBreaks()).anyMatch(b -> b.reason().contains("audit_detail 1 is not covered"));
}
/*******************************************************************************
** The settings row (seeded by the DDL) overrides config: change it there.
*******************************************************************************/
private void setting(String column, int value) throws Exception
{
try(var connection = owner())
{
sql(connection, "UPDATE worm_audit_settings SET " + column + "=" + value + " WHERE id=1");
}
config.invalidateSettings();
}
}
@@ -84,6 +84,38 @@ class ProcessesAndAlertsTest
}
}
/*******************************************************************************
** RC.5 validates declared severities, names and daily heartbeat expectations.
*******************************************************************************/
@Test
void alertTypesAndDailyMonitors() throws Exception
{
var instance = init(new WormAuditQBitConfig());
new com.kingsrook.qqq.backend.core.instances.QInstanceValidator().validate(instance);
assertThat(new com.kingsrook.qbits.wormaudit.chain.AuditStore(null).isMemory()).isTrue();
var types = AlertsInstanceMetaData.of(instance).getAlertTypes();
assertThat(types).containsOnlyKeys("wormaudit.chain.break", "wormaudit.export.failed", "wormaudit.verify.daily", "wormaudit.export.daily");
assertThat(types.get("wormaudit.chain.break").getDefaultSeverity())
.isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertSeverity.CRITICAL);
assertThat(types.get("wormaudit.export.failed").getDefaultSeverity())
.isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertSeverity.ERROR);
types.values().forEach(type ->
{
assertThat(type.getLabel()).isNotBlank();
assertThat(type.getDescription()).isNotBlank();
assertThat(AlertsInstanceMetaData.ALERT_TYPE_NAME_PATTERN.matcher(type.getName()).matches()).isTrue();
});
for(String name : List.of("wormaudit.verify.daily", "wormaudit.export.daily"))
{
var monitors = types.get(name).getDefaultMonitors();
assertThat(monitors).hasSize(1);
assertThat(monitors.getFirst().getKind()).isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertMonitorKind.HEARTBEAT);
assertThat(monitors.getFirst().getDefinition()).containsEntry("heartbeatKey", name)
.containsEntry("expectedEverySeconds", 86400).containsEntry("graceSeconds", 7200);
}
}
/*******************************************************************************
** Verify daily raise resolve heartbeat and user result.
*******************************************************************************/
@@ -106,7 +138,7 @@ class ProcessesAndAlertsTest
daily.run(new RunBackendStepInput(), output);
assertThat(output.getValueString("outcome")).isEqualTo("FAIL");
assertThat(Sink.raised).hasSize(1);
assertThat(Sink.raised.getFirst().getAlertTypeName()).isEqualTo("worm_audit.chain.break");
assertThat(Sink.raised.getFirst().getAlertTypeName()).isEqualTo("wormaudit.chain.break");
assertThat(Sink.raised.getFirst().getDedupKey()).isEqualTo("chain");
assertThat(Sink.raised.getFirst().getRecordRefs().getFirst().getTableName())
.isEqualTo(WormAuditTables.VERIFICATION);
@@ -119,7 +151,7 @@ class ProcessesAndAlertsTest
daily.run(new RunBackendStepInput(), new RunBackendStepOutput());
assertThat(Sink.resolved).hasSize(1);
assertThat(Sink.beats)
.containsExactly("worm_audit.verify.daily", "worm_audit.verify.daily");
.containsExactly("wormaudit.verify.daily", "wormaudit.verify.daily");
assertThat(STORE.all(WormAuditTables.VERIFICATION)).hasSize(2);
assertThat(VerificationRuns.run(null, "USER").passed()).isTrue();
assertThat(Sink.beats).hasSize(2);
@@ -45,6 +45,45 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
*******************************************************************************/
class RecordIntegrationTest
{
/*******************************************************************************
** All record screens have readable labels and full-width sections and lists.
*******************************************************************************/
@Test
void recordScreensHaveReadableLabelsAndFullWidthLists() throws Exception
{
var instance = init(new WormAuditQBitConfig());
Map<String, String> formats = Map.of(
WormAuditTables.CHAIN, "Link #%s", WormAuditTables.CHECKPOINT, "Checkpoint #%s",
WormAuditTables.EXPORT, "Export #%s (links %s–%s)", WormAuditTables.VERIFICATION, "Verification #%s (%s)",
WormAuditTables.PURGE, "Purge #%s (%s)", WormAuditTables.SETTINGS, "WORM Audit Settings");
for(String name : WormAuditTables.ALL)
{
var table = instance.getTable(name);
assertThat(table.getRecordLabelFormat()).isEqualTo(formats.get(name));
assertThat(table.getRecordLabelFields()).allMatch(table.getFields()::containsKey);
table.getFields().forEach((fieldName, field) ->
assertThat(field.getLabel()).as(name + "." + fieldName).isNotBlank().isNotEqualTo(fieldName).doesNotContain("_"));
table.getSections().forEach(section ->
{
assertThat(section.getGridColumns()).isEqualTo(12);
if(section.getWidgetName() != null)
{
assertThat(instance.getWidget(section.getWidgetName()).getGridColumns()).isEqualTo(12);
}
});
}
assertThat(instance.getTable(WormAuditTables.EXPORT).getRecordLabelFields()).containsExactly("id", "from_link_id", "to_link_id");
for(String name : List.of(WormAuditTables.VERIFICATION, WormAuditTables.PURGE))
{
assertThat(instance.getTable(name).getRecordLabelFields()).containsExactly("id", "outcome");
}
var verification = instance.getTable(WormAuditTables.VERIFICATION);
assertThat(verification.getSections()).extracting(section -> section.getLabel()).containsExactly("Result", "Scope", "Run", "Chain links");
assertThat(verification.getSections().stream().filter(section -> section.getWidgetName() == null)
.flatMap(section -> section.getFieldNames().stream()).toList()).containsExactlyInAnyOrderElementsOf(verification.getFields().keySet());
}
/*******************************************************************************
** Tests and fixtures for event sink.
*******************************************************************************/
@@ -93,6 +132,8 @@ class RecordIntegrationTest
assertThat(data.getTotalRows()).isEqualTo(2);
assertThat(data.getQueryOutput().getRecords()).hasSize(2);
assertThat(data.getViewAllLink()).contains("filter=");
String json = com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(data);
assertThat(json).contains("Chain links").doesNotContain("signingKey", "trustedPublicKeys", "sealedStoreSupplier", "privateKey");
input.setQueryParams(Map.of("id", "999"));
assertThatThrownBy(() -> new LinkRangeWidget().render(input))
.hasMessageContaining("was not found");
@@ -0,0 +1,710 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
import com.kingsrook.qbits.objectstorage.api.Page;
import com.kingsrook.qbits.objectstorage.api.StoredObject;
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
import com.kingsrook.qbits.wormaudit.chain.Hashes;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.export.ExportFiles;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.export.SealedStore;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunInput;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunOutput;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.remove;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Offline sealed-export contract, corruption, retention, batching and UI tests.
*******************************************************************************/
class SealedExportTest
{
static final Instant NOW = Instant.parse("2026-10-14T02:00:00Z");
WormAuditQBitConfig config;
MemorySealedStore storage;
/*******************************************************************************
** Tiny object boundary; bytes exist only in tests.
*******************************************************************************/
static class MemorySealedStore implements SealedStore
{
final Map<String, byte[]> objects = new LinkedHashMap<>();
final Map<String, StoredObject> heads = new LinkedHashMap<>();
final Map<String, Map<String, byte[]>> history = new LinkedHashMap<>();
int nextVersion;
String missingVersion;
boolean omitVersion;
boolean racePut;
boolean supported = true;
String echoMode;
long shortenSeconds;
int failPut;
@Override
public boolean supportsLock()
{
return (supported);
}
@Override
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
{
if(failPut > 0 && objects.size() + 1 == failPut)
{
throw (new IllegalStateException("CANARY-PRIVATE-PROVIDER-TEXT"));
}
if(objects.containsKey(key))
{
history.computeIfAbsent(key, k -> new LinkedHashMap<>()).put(heads.get(key).getVersionId(), objects.get(key));
}
String version = omitVersion ? missingVersion : "v" + ++nextVersion;
objects.put(key, Files.readAllBytes(file));
heads.put(key, new StoredObject().withKey(key).withVersionId(racePut ? "replacement" : version)
.withLockMode(echoMode == null ? mode.name() : echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
return (version);
}
@Override
public StoredObject head(String key)
{
return (heads.get(key));
}
@Override
public InputStream getVersion(String key, String versionId)
{
byte[] bytes = versionId.equals(heads.get(key).getVersionId()) ? objects.get(key) : history.get(key).get(versionId);
return (new ByteArrayInputStream(bytes));
}
@Override
public Page<ObjectSummary> list(ListRequest request)
{
return (new Page<ObjectSummary>().withItems(objects.keySet().stream().filter(k -> k.startsWith(request.getPrefix()))
.map(k -> new ObjectSummary().withKey(k)).toList()));
}
}
/*******************************************************************************
** Exercise setUp.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
config = ChainVerificationTest.signed();
var instance = init(config);
storage = new MemorySealedStore();
ProcessesAndAlertsTest.Sink.raised.clear();
ProcessesAndAlertsTest.Sink.resolved.clear();
ProcessesAndAlertsTest.Sink.beats.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(ProcessesAndAlertsTest.Sink.class));
}
QRecord seal() throws Exception
{
new WormAuditChainer(config).chainOnce(NOW);
var result = new SealedExporter(config, storage).run("USER", NOW);
assertThat(result.failed()).as(result.message()).isFalse();
assertThat(result.row()).isNotNull();
return (result.row());
}
SealedExportVerifier.Report verify(QRecord row) throws Exception
{
return (new SealedExportVerifier(config, storage).verify(row, "USER"));
}
String findings(QRecord row) throws Exception
{
return (verify(row).result().getBreaks().toString());
}
/*******************************************************************************
** Exercise canonicalGoldenFilesAndRoundTrip.
*******************************************************************************/
@Test
void canonicalGoldenFilesAndRoundTrip() throws Exception
{
audit(1);
detail(1, 1);
QRecord row = seal();
String prefix = "worm-audit/2026/10/14/1-1/";
assertThat(row.getValueString("object_prefix")).isEqualTo(prefix);
String content = "{\"audit\":{\"clientChannel\":null,\"id\":1,\"message\":\"Record was Edited\",\"recordId\":\"17\",\"table\":\"cms_page\",\"timestamp\":\"2020-01-02T03:04:06.123456Z\",\"user\":\"james\"},\"details\":[{\"fieldName\":\"title\",\"id\":1,\"message\":\"Changed Title\",\"newValue\":\"new\",\"oldValue\":\"old\"}],\"v\":1}";
String hash = Hashes.link(Hashes.GENESIS, content.getBytes(StandardCharsets.UTF_8));
String golden = "{\"content\":" + content + ",\"link\":{\"auditId\":1,\"detailIds\":[1],\"id\":1,\"kind\":\"AUDIT\",\"linkHash\":\"" + hash
+ "\",\"prevHash\":\"" + Hashes.GENESIS + "\"}}\n";
assertThat(new String(storage.objects.get(prefix + "links.jsonl"), StandardCharsets.UTF_8)).isEqualTo(golden);
var manifest = ExportFiles.parse(storage.objects.get(prefix + "manifest.json"));
assertThat(manifest.keySet()).containsExactly("checkpointCount", "createdAt", "files", "format", "fromLinkId", "headHash", "linkCount",
"prevManifestSha256", "qbitVersion", "toLinkId");
assertThat(manifest).containsEntry("format", "worm-audit-export/1").containsEntry("linkCount", 1L).containsEntry("checkpointCount", 1L)
.containsEntry("prevManifestSha256", null).containsEntry("headHash", hash).containsEntry("createdAt", NOW.toString());
assertThat(storage.objects.get(prefix + "manifest.json")).isEqualTo(Jcs.bytes(manifest));
assertThat(row.getValues()).containsKeys("from_checkpoint_id", "to_checkpoint_id", "bucket_alias", "manifest_key", "manifest_sha256",
"prev_manifest_sha256", "object_count", "byte_count", "lock_mode", "retain_until", "sealed_at");
assertThat(row.getValueLong("byte_count")).isEqualTo(storage.objects.values().stream().mapToLong(b -> b.length).sum());
assertThat(row.getValueInstant("retain_until")).isEqualTo(NOW.plusSeconds(30 * 86400L));
assertThat(verify(row).result().passed()).isTrue();
QRecord verification = STORE.last(WormAuditTables.VERIFICATION);
assertThat(verification.getValueString("kind")).isEqualTo("SEALED_EXPORT");
assertThat(verification.getValueLong("export_id")).isEqualTo(row.getValueLong("id"));
assertThat(verification.getValueString("outcome")).isEqualTo("PASS");
assertThat(ProcessesAndAlertsTest.Sink.resolved).hasSize(1);
}
/*******************************************************************************
** Every overwrite is a break while verification still uses the sealed bytes.
*******************************************************************************/
@Test
void detectsOverwritesAndReadsSealedVersions() throws Exception
{
audit(1);
QRecord row = seal();
Map<String, Object> versions = ExportFiles.parse(row.getValueString("object_versions").getBytes(StandardCharsets.UTF_8));
assertThat(versions.keySet()).containsExactly("checkpoints.jsonl", "links.jsonl", "manifest.json");
assertThat(row.getValueString("object_versions")).isEqualTo(Jcs.string(versions));
assertThat(STORE.get(WormAuditTables.EXPORT, row.getValueLong("id")).getValueString("object_versions"))
.isEqualTo(row.getValueString("object_versions"));
try(var files = new ExportFiles())
{
String linksKey = row.getValueString("object_prefix") + "links.jsonl";
Files.write(files.file("links.jsonl"), storage.objects.get(linksKey));
storage.put(linksKey, files.file("links.jsonl"), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
var identical = verify(row);
assertThat(identical.result().passed()).isFalse();
assertThat(identical.result().getLinksChecked()).isEqualTo(1);
assertThat(identical.result().getBreaks()).singleElement().satisfies(b ->
assertThat(b.reason()).contains("object links.jsonl was overwritten after sealing"));
for(String name : ExportFiles.NAMES)
{
String key = row.getValueString("object_prefix") + name;
Files.writeString(files.file(name), "different bytes");
String current = storage.put(key, files.file(name), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
var report = verify(row);
assertThat(report.result().passed()).isFalse();
assertThat(report.result().getLinksChecked()).isEqualTo(1);
assertThat(report.result().getBreaks()).allSatisfy(b -> assertThat(b.reason()).contains("was overwritten after sealing"));
assertThat(report.result().getBreaks().toString()).contains("object " + name + " was overwritten after sealing (current version "
+ current + ", sealed version " + versions.get(name) + " is still retained under Object Lock)");
}
}
var load = new ExportProcesses.VerifyLoad(storage);
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString()).contains("ERROR", "was overwritten after sealing");
}
/*******************************************************************************
** Unversioned uploads and upload/HEAD races cannot create a sealed catalog row.
*******************************************************************************/
@Test
void refusesMissingVersionsAndUploadRaces() throws Exception
{
audit(1);
new WormAuditChainer(config).chainOnce(NOW);
storage.omitVersion = true;
for(String version : java.util.Arrays.asList(null, "", " ", "null"))
{
storage.missingVersion = version;
var result = new SealedExporter(config, storage).run("USER", NOW);
assertThat(result.failed()).isTrue();
assertThat(result.message()).contains("returned no version id");
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
}
storage.omitVersion = false;
storage.racePut = true;
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("version changed after upload");
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(5);
}
/*******************************************************************************
** Missing catalog versions must never silently fall back to current content.
*******************************************************************************/
@Test
void refusesMissingRecordedVersion() throws Exception
{
audit(1);
QRecord row = seal();
row.setValue("object_versions", "{}");
assertThat(findings(row)).contains("object links.jsonl has no recorded sealed version id");
row.setValue("object_versions", null);
assertThat(verify(row).result().passed()).isFalse();
}
/*******************************************************************************
** Exercise rangeSelectionNoCheckpointContiguousAndPurgeAnchor.
*******************************************************************************/
@Test
void rangeSelectionNoCheckpointContiguousAndPurgeAnchor() throws Exception
{
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).isEqualTo("Nothing to seal: no new checkpoint");
assertThat(ProcessesAndAlertsTest.Sink.beats).containsExactly("wormaudit.export.daily");
audit(1);
QRecord first = seal();
assertThat(new SealedExporter(config, storage).run("USER", NOW).row()).isNull();
audit(2);
QRecord second = seal();
assertThat(second.getValueLong("from_link_id")).isEqualTo(2L);
assertThat(second.getValueString("prev_manifest_sha256")).isEqualTo(first.getValueString("manifest_sha256"));
assertThat(verify(second).result().passed()).isTrue();
init(config);
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 10L).withValue("anchor_hash", Hashes.GENESIS)));
audit(11);
QRecord anchored = seal();
assertThat(anchored.getValueLong("from_link_id")).isEqualTo(11L);
assertThat(anchored.getValueString("prev_manifest_sha256")).isNull();
assertThat(verify(anchored).result().passed()).isTrue();
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 20L).withValue("anchor_hash", Hashes.GENESIS)));
assertThatThrownBy(() -> SealedExporter.scope(STORE)).hasMessage("Unexported links were purged");
}
/*******************************************************************************
** Exercise refusesUnsupportedAndIncorrectLockEchoAndPartialUpload.
*******************************************************************************/
@Test
void refusesUnsupportedAndIncorrectLockEchoAndPartialUpload() throws Exception
{
audit(1);
new WormAuditChainer(config).chainOnce(NOW);
storage.supported = false;
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).contains("does not support Object Lock");
assertThat(storage.objects).isEmpty();
storage.supported = true;
storage.echoMode = "COMPLIANCE";
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo GOVERNANCE");
storage.echoMode = null;
storage.shortenSeconds = 1;
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
storage.shortenSeconds = 0;
storage.objects.clear();
storage.failPut = 2;
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).doesNotContain("CANARY").contains("storage or database");
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(4);
assertThat(ProcessesAndAlertsTest.Sink.beats).hasSize(2);
}
/*******************************************************************************
** Exercise tamperedObjectsBrokenManifestChainAndLockRemoval.
*******************************************************************************/
@Test
void tamperedObjectsBrokenManifestChainAndLockRemoval() throws Exception
{
audit(1);
QRecord first = seal();
String key = first.getValueString("object_prefix") + "links.jsonl";
byte[] original = storage.objects.get(key);
storage.objects.put(key, "tampered".getBytes(StandardCharsets.UTF_8));
assertThat(findings(first)).contains("object links.jsonl hash does not match the manifest");
storage.objects.put(key, original);
storage.heads.get(key).withLockMode("COMPLIANCE");
assertThat(findings(first)).contains("lacks the recorded GOVERNANCE retention");
storage.heads.get(key).withLockMode(first.getValueString("lock_mode"));
audit(2);
QRecord second = seal();
tamper(WormAuditTables.EXPORT, first.getValueLong("id"), "manifest_sha256", "f".repeat(64));
assertThat(findings(second)).contains("prevManifestSha256 does not match");
assertThat(findings(STORE.get(WormAuditTables.EXPORT, first.getValueLong("id")))).contains("manifest.json hash does not match");
}
/*******************************************************************************
** Exercise exactLiveDifferencesMissingRowsAndPurgedCopies.
*******************************************************************************/
@Test
void exactLiveDifferencesMissingRowsAndPurgedCopies() throws Exception
{
audit(1);
detail(1, 1);
audit(2);
QRecord row = seal();
tamper("auditDetail", 1, "newValue", "CANARY-PRIVATE");
assertThat(findings(row)).contains("link 1 (audit 1): live content differs from the sealed copy (sealed 2026-10-14)").doesNotContain("CANARY");
remove(WormAuditTables.CHAIN, 2);
assertThat(findings(row)).contains("link 2 is missing from the live chain");
tamper(WormAuditTables.CHAIN, 1, "kind", "LATE_DETAIL");
assertThat(findings(row)).contains("live chain kind differs");
remove("audit", 1);
assertThat(findings(row)).contains("live audit header is missing");
String head = (String) ExportFiles.parse(storage.objects.get(row.getValueString("manifest_key"))).get("headHash");
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 2L).withValue("anchor_hash", head)));
remove(WormAuditTables.CHAIN, 1);
remove("audit", 2);
var report = verify(row);
assertThat(report.result().passed()).isTrue();
assertThat(report.purged()).isEqualTo(2);
var load = new ExportProcesses.VerifyLoad(storage);
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString()).contains("INFO", "2 links verified from the sealed copy only");
}
/*******************************************************************************
** Exercise refusesBadSourceAndDetectsMissingLiveDetails.
*******************************************************************************/
@Test
void refusesBadSourceAndDetectsMissingLiveDetails() throws Exception
{
audit(1);
detail(1, 1);
new WormAuditChainer(config).chainOnce(NOW);
tamper("auditDetail", 1, "newValue", "changed");
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("content differs at link 1");
assertThat(storage.objects).isEmpty();
tamper("auditDetail", 1, "newValue", "new");
QRecord row = seal();
remove("auditDetail", 1);
assertThat(findings(row)).contains("link 1 (audit 1): live content is missing or invalid");
}
/*******************************************************************************
** Exercise spansPagesLateDetailsAndCheckpointBoundaries.
*******************************************************************************/
@Test
void spansPagesLateDetailsAndCheckpointBoundaries() throws Exception
{
config.withBatchSize(1100).withCheckpointEveryLinks(500);
for(long id = 1; id <= 1100; id++)
{
audit(id);
}
new WormAuditChainer(config).chainOnce(NOW);
// The final 100 links need the hourly checkpoint.
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(3601));
QRecord row = seal();
assertThat(row.getValueLong("to_link_id")).isEqualTo(1100);
assertThat(verify(row).result().getLinksChecked()).isEqualTo(1100);
detail(1, 1);
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(7202));
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(10803));
QRecord late = seal();
assertThat(late.getValueLong("from_link_id")).isEqualTo(1101);
assertThat(verify(late).result().passed()).isTrue();
}
/*******************************************************************************
** Exercise reviewLoadSelectionAndSchedulesUseStandardEtl.
*******************************************************************************/
@Test
void reviewLoadSelectionAndSchedulesUseStandardEtl() throws Exception
{
var instance = com.kingsrook.qqq.backend.core.context.QContext.getQInstance();
assertThat(config.getExportBucketAlias()).isEqualTo("worm_audit");
assertThat(config.getExportPrefix()).isEqualTo("worm-audit");
assertThat(instance.getProcess(ExportProcesses.EXPORT).getTableName()).isEqualTo(WormAuditTables.EXPORT);
assertThat(instance.getProcess(ExportProcesses.VERIFY).getMinInputRecords()).isEqualTo(1);
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC");
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 0 2 * * ?");
var output = new RunBackendStepOutput();
var input = new RunBackendStepInput().withRecords(List.of(new QRecord().withValue("id", 0L)));
var extract = new ExportProcesses.ExportExtract();
assertThat(extract.doCount(input)).isEqualTo(1);
extract.setRecordPipe(new com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe());
extract.run(input, output);
var review = new ExportProcesses.ExportReview();
review.runOnePage(input, output);
assertThat(review.getProcessSummary(output, false).toString()).contains("Nothing to seal");
audit(1);
new WormAuditChainer(config).chainOnce(NOW);
review.runOnePage(input, output);
assertThat(review.getProcessSummary(output, false).toString()).contains("1 chain links will be sealed", "1 checkpoints");
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
var load = new ExportProcesses.ExportLoad(storage);
assertThat(load.getProcessSummary(output, true)).isEmpty();
load.runOnePage(input, output);
load.postRun(new BackendStepPostRunInput(input), new BackendStepPostRunOutput(output));
assertThat(load.getProcessSummary(output, true).toString()).contains("1 links sealed in export");
var rows = STORE.all(WormAuditTables.EXPORT);
var verifyInput = new RunBackendStepInput().withRecords(rows).addValue("recordIds", rows.getFirst().getValueString("id"));
assertThat(new ExportProcesses.VerifyExtract().getQueryFilter(verifyInput).getCriteria()).isNotEmpty();
var verifyReview = new ExportProcesses.VerifyReview();
verifyReview.runOnePage(verifyInput, output);
assertThat(verifyReview.getProcessSummary(output, false).toString()).contains("1 sealed exports");
var verifyLoad = new ExportProcesses.VerifyLoad(storage);
verifyLoad.runOnePage(verifyInput, output);
assertThat(verifyLoad.getProcessSummary(output, true).toString()).contains("1 links verified");
new ExportProcesses.Daily().run(input, output);
assertThat(output.getValueString("outcome")).isEqualTo("PASS");
audit(2);
new WormAuditChainer(config).chainOnce(NOW);
new ExportProcesses.Daily().run(input, output);
assertThat(output.getValueString("outcome")).isEqualTo("FAIL");
assertThat(ProcessesAndAlertsTest.Sink.beats).contains("wormaudit.export.daily");
new ExportProcesses.ExportLoad();
new ExportProcesses.VerifyLoad();
}
/*******************************************************************************
** Exercise malformedJsonIsValueFreeAndConfigRejectsUnsafePrefixes.
*******************************************************************************/
@Test
void malformedJsonIsValueFreeAndConfigRejectsUnsafePrefixes() throws Exception
{
assertThatThrownBy(() -> ExportFiles.parse("{\"x\":\"CANARY\",\"x\":2}".getBytes(StandardCharsets.UTF_8)))
.hasMessage("Invalid sealed JSON").hasNoCause();
assertThatThrownBy(() -> config.withExportPrefix("../escape").validate()).isInstanceOf(IllegalArgumentException.class);
config.withExportPrefix("evidence/audit").withExportBucketAlias("backup").withExportCron("0 5 2 * * ?").validate();
assertThat(config.getExportBucketAlias()).isEqualTo("backup");
try(var files = new ExportFiles())
{
assertThatThrownBy(() -> files.file("../escape")).isInstanceOf(IllegalArgumentException.class);
}
}
/*******************************************************************************
** Re-hash a tampered object and catalog to reach the independent signature checks.
*******************************************************************************/
QRecord rewrite(QRecord row, String name, byte[] bytes) throws Exception
{
String prefix = row.getValueString("object_prefix");
storage.objects.put(prefix + name, bytes);
Map<String, Object> manifest = ExportFiles.parse(storage.objects.get(prefix + "manifest.json"));
Map<String, Object> hashes = new LinkedHashMap<>((Map<String, Object>) manifest.get("files"));
hashes.put(name, java.util.HexFormat.of().formatHex(Hashes.digest(bytes)));
manifest.put("files", hashes);
byte[] encoded = Jcs.bytes(manifest);
storage.objects.put(prefix + "manifest.json", encoded);
tamper(WormAuditTables.EXPORT, row.getValueLong("id"), "manifest_sha256", java.util.HexFormat.of().formatHex(Hashes.digest(encoded)));
return (STORE.get(WormAuditTables.EXPORT, row.getValueLong("id")));
}
/*******************************************************************************
** Exercise independentSignaturesMerkleAndSealedContentChecks.
*******************************************************************************/
@Test
void independentSignaturesMerkleAndSealedContentChecks() throws Exception
{
audit(1);
QRecord row = seal();
String prefix = row.getValueString("object_prefix");
byte[] originalCheckpoint = storage.objects.get(prefix + "checkpoints.jsonl");
Map<String, Object> checkpoint = ExportFiles.parse(originalCheckpoint);
checkpoint.put("signature", java.util.Base64.getEncoder().encodeToString(new byte[64]));
row = rewrite(row, "checkpoints.jsonl", (Jcs.string(checkpoint) + "\n").getBytes(StandardCharsets.UTF_8));
assertThat(findings(row)).contains("checkpoint 1: signature does not verify");
checkpoint = ExportFiles.parse(originalCheckpoint);
checkpoint.put("merkle_root", "a".repeat(64));
row = rewrite(row, "checkpoints.jsonl", (Jcs.string(checkpoint) + "\n").getBytes(StandardCharsets.UTF_8));
assertThat(findings(row)).contains("Merkle root or head hash differs");
row = rewrite(row, "checkpoints.jsonl", originalCheckpoint);
Map<String, Object> document = ExportFiles.parse(storage.objects.get(prefix + "links.jsonl"));
Map<String, Object> content = (Map<String, Object>) document.get("content");
((Map<String, Object>) content.get("audit")).put("message", "CANARY-SEALED");
row = rewrite(row, "links.jsonl", (Jcs.string(document) + "\n").getBytes(StandardCharsets.UTF_8));
assertThat(findings(row)).contains("sealed content differs from the chained hash").doesNotContain("CANARY");
storage.objects.remove(prefix + "links.jsonl");
assertThat(findings(row)).contains("cannot be read or has invalid structure").doesNotContain("CANARY");
}
/*******************************************************************************
** Exercise concurrentExportersDoNotOverlapAndMultipleSelectionsPersistSeparately.
*******************************************************************************/
@Test
void concurrentExportersDoNotOverlapAndMultipleSelectionsPersistSeparately() throws Exception
{
audit(1);
new WormAuditChainer(config).chainOnce(NOW);
var instance = com.kingsrook.qqq.backend.core.context.QContext.getQInstance();
var workers = java.util.concurrent.Executors.newFixedThreadPool(2);
try
{
var jobs = new java.util.ArrayList<java.util.concurrent.Future<SealedExporter.Result>>();
for(int i = 0; i < 2; i++)
{
jobs.add(workers.submit(() ->
{
com.kingsrook.qqq.backend.core.context.QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSession());
try
{
return (new SealedExporter(config, storage).run("USER", NOW));
}
finally
{
com.kingsrook.qqq.backend.core.context.QContext.clear();
}
}));
}
int sealed = 0;
for(var job : jobs)
{
var result = job.get(10, java.util.concurrent.TimeUnit.SECONDS);
assertThat(result.failed()).as(result.message()).isFalse();
sealed += result.row() == null ? 0 : 1;
}
assertThat(sealed).isEqualTo(1);
}
finally
{
workers.shutdownNow();
}
audit(2);
seal();
var load = new ExportProcesses.VerifyLoad(storage);
load.runOnePage(new RunBackendStepInput().withRecords(STORE.all(WormAuditTables.EXPORT)), new RunBackendStepOutput());
assertThat(STORE.all(WormAuditTables.VERIFICATION)).hasSize(2)
.allMatch(r -> "PASS".equals(r.getValueString("outcome")));
assertThat(STORE.all(WormAuditTables.VERIFICATION)).extracting(r -> r.getValueLong("export_id")).doesNotHaveDuplicates();
}
/*******************************************************************************
** Exercise h2TransactionRoundTripAndFailedPutHasNoCatalogRow.
*******************************************************************************/
@Test
void h2TransactionRoundTripAndFailedPutHasNoCatalogRow() throws Exception
{
String url = "jdbc:h2:mem:exports_" + java.util.UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE";
var instance = RdbmsTestSupport.instance(url, "sa", "", config, false);
try(var connection = java.sql.DriverManager.getConnection(url, "sa", ""))
{
RdbmsTestSupport.createTables(connection, instance, false);
com.kingsrook.qqq.backend.core.context.QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSession());
WormAuditTestSupport.lookups();
audit(1);
detail(1, 1);
QRecord row = seal();
assertThat(verify(row).result().passed()).isTrue();
assertThat(verify(STORE.get(WormAuditTables.EXPORT, row.getValueLong("id"))).result().passed()).isTrue();
connection.createStatement().execute("UPDATE audit_detail SET new_value='changed' WHERE id=1");
assertThat(findings(row)).contains("link 1 (audit 1): live content differs");
audit(2);
new WormAuditChainer(config).chainOnce(NOW);
storage.failPut = storage.objects.size() + 1;
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1);
}
}
/*******************************************************************************
** Exercise realAdapterRequiresBackupRoleAndPositiveCapability.
*******************************************************************************/
@Test
void realAdapterRequiresBackupRoleAndPositiveCapability() throws Exception
{
var bucket = new com.kingsrook.qbits.objectstorage.config.StorageBucketConfig("worm_audit")
.withConnection("test").withBucketName("test-bucket").withIndexed(false)
.withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.BACKUP);
System.setProperty("worm.unit.access", "unit-access");
System.setProperty("worm.unit.secret", "unit-secret");
var server = com.sun.net.httpserver.HttpServer.create(new java.net.InetSocketAddress("127.0.0.1", 0), 0);
var responseVersion = new java.util.concurrent.atomic.AtomicReference<>("sealed-version");
var requested = new java.util.concurrent.CopyOnWriteArrayList<String>();
server.createContext("/test-bucket/links.jsonl", exchange ->
{
requested.add(exchange.getRequestURI().getRawQuery());
exchange.getResponseHeaders().add("x-amz-version-id", responseVersion.get());
byte[] bytes = "sealed fixture".getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(200, bytes.length);
try(var output = exchange.getResponseBody())
{
output.write(bytes);
}
});
server.start();
try
{
com.kingsrook.qbits.objectstorage.api.ObjectStorage.configureStandalone(
new com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig().withBackendName("memory")
.withAllowedSecretRefPatterns(List.of("prop.worm.unit.*"))
.withConnection(new com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig("test")
.withEndpoint("http://127.0.0.1:" + server.getAddress().getPort()).withRegion("us-east-1")
.withAddressingStyle(com.kingsrook.qbits.objectstorage.config.AddressingStyle.PATH)
.withProviderType(com.kingsrook.qbits.objectstorage.config.ProviderType.MINIO)
.withAccessKeyRef("${prop.worm.unit.access}").withSecretKeyRef("${prop.worm.unit.secret}"))
.withBucket(bucket));
var adapter = new com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore("worm_audit", "unit");
assertThat(adapter.supportsLock()).isFalse();
com.kingsrook.qbits.objectstorage.internal.CapabilityStore.record(bucket,
com.kingsrook.qbits.objectstorage.api.Capability.OBJECT_LOCK,
com.kingsrook.qbits.objectstorage.api.CapabilityStatus.SUPPORTED, "unit fixture");
assertThat(adapter.supportsLock()).isTrue();
try(var input = adapter.getVersion("links.jsonl", "sealed-version"))
{
assertThat(new String(input.readAllBytes(), StandardCharsets.UTF_8)).isEqualTo("sealed fixture");
}
responseVersion.set("different-version");
assertThatThrownBy(() -> adapter.getVersion("links.jsonl", "sealed-version"))
.hasMessage("Storage returned a different object version");
assertThatThrownBy(() -> adapter.getVersion("links.jsonl", null)).hasMessage("Missing sealed object version");
assertThat(requested).containsExactly("versionId=sealed-version", "versionId=sealed-version");
bucket.withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.PRIVATE);
assertThat(adapter.supportsLock()).isFalse();
}
finally
{
server.stop(0);
com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime.reset();
System.clearProperty("worm.unit.access");
System.clearProperty("worm.unit.secret");
}
}
}
@@ -0,0 +1,343 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.sql.DriverManager;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.wormaudit.app.WormAuditApp;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.demo.MemorySealedStore;
import com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.OLD;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.lookups;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Home acceptance: truthful summaries, permissions, privacy and real demo data.
*******************************************************************************/
class WormAuditHomeTest
{
/*******************************************************************************
** Empty tables do not imply successful verification or export coverage.
*******************************************************************************/
@Test
void emptyStatesAndWarnings() throws Exception
{
var config = new WormAuditQBitConfig();
init(config);
var data = WormAuditHomeData.load(config, new MemorySealedStore(), OLD);
assertThat(data.links()).isZero();
assertThat(data.backlog()).isZero();
assertThat(data.settling()).isZero();
assertThat(data.anchor()).isNull();
assertThat(data.oldestAudit()).isNull();
assertThat(data.nextPurge()).isNull();
assertThat(data.exportAgeHours()).isNull();
assertThat(data.warnings()).contains("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
String html = WormAuditHomeView.render(data, "/app");
assertThat(html).contains("Chain status", "Export status", "Retention", "Recent purges", "Setup warnings",
"No links chained yet", "No verification has run yet", "Unknown until verification runs", "No sealed exports yet",
"No sealed ranges to check", "No live audits", "No purge requests yet");
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Not verified yet");
}
/*******************************************************************************
** Strict settle boundary and AUDIT anti-join, including late-detail links.
*******************************************************************************/
@Test
void chainBacklogRetentionAndRealFailureRecovery() throws Exception
{
var config = ChainVerificationTest.signed();
init(config);
audit(1);
detail(1, 1);
Instant now = OLD.plusSeconds(120);
new WormAuditChainer(config).chainOnce(now);
audit(2);
audit(3);
tamper("audit", 3, "timestamp", now.minusSeconds(60));
var data = WormAuditHomeData.load(config, new MemorySealedStore(), now);
assertThat(data.links()).isEqualTo(1);
assertThat(data.lastLinkAt()).isEqualTo(now);
assertThat(data.backlog()).isEqualTo(1);
assertThat(data.settling()).isEqualTo(1);
assertThat(data.oldestAudit()).isEqualTo(OLD.plusSeconds(1));
assertThat(data.nextPurge()).isEqualTo(OLD.plusSeconds(1).atZone(ZoneOffset.UTC).plusYears(6).toInstant());
new WormAuditChainer(config).chainOnce(now.plusSeconds(61));
tamper("auditDetail", 1, "newValue", "forced mismatch");
var failure = new WormAuditVerifier(config).verify(null, now.plusSeconds(61));
QRecord failed = VerificationRuns.save(failure, "SCHEDULE", null, now.plusSeconds(62));
assertThat(failure.passed()).isFalse();
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(63));
assertThat(data.openBreak().getValueLong("id")).isEqualTo(failed.getValueLong("id"));
String html = WormAuditHomeView.render(data, "/app/admin/worm");
assertThat(html).contains("/app/admin/worm/worm_audit_verification/1", "/app/admin/worm/worm_audit_chain/1", "content differs from the chained hash");
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Chain breaks reported");
// A passing selected run must not clear a full-chain failure.
VerificationRuns.save(new WormAuditVerifier(config).verify(Set.of(2L), now.plusSeconds(64)), "USER", null, now.plusSeconds(65));
assertThat(WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(66)).openBreak()).isNotNull();
tamper("auditDetail", 1, "newValue", "new");
VerificationRuns.save(new WormAuditVerifier(config).verify(null, now.plusSeconds(67)), "SCHEDULE", null, now.plusSeconds(68));
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(69));
assertThat(data.openBreak()).isNull();
assertThat(data.warnings()).hasSize(2);
assertThat(WormAuditHomeView.render(data, "/app")).contains("Setup warnings", "GOVERNANCE", "30 days");
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Setup needs attention");
}
/*******************************************************************************
** Genuine exports, trailing lag and a corruption fixture for a coverage hole.
*******************************************************************************/
@Test
void sealedRangesLagGapsAndPurges() throws Exception
{
var storage = new MemorySealedStore();
var config = ChainVerificationTest.signed().withSealedStoreSupplier(() -> storage);
init(config);
for(long id = 1; id <= 4; id++)
{
audit(id);
}
Instant now = OLD.plusSeconds(180);
new WormAuditChainer(config).chainOnce(now);
var export = new SealedExporter(config).run("USER", now);
assertThat(export.failed()).isFalse();
assertThat(new SealedExportVerifier(config).verify(export.row(), "USER").result().passed()).isTrue();
audit(5);
new WormAuditChainer(config).chainOnce(now.plusSeconds(120));
var data = WormAuditHomeData.load(config, null, now.plus(Duration.ofHours(3)));
assertThat(data.exportLag()).isEqualTo(1);
assertThat(data.exportAgeHours()).isEqualTo(3);
assertThat(data.exportGaps()).isEmpty();
assertThat(data.lockCapability()).isEqualTo("SUPPORTED");
assertThat(data.bucketAlias()).isEqualTo("worm_audit");
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("GOVERNANCE");
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "GOVERNANCE", "Retain until");
// Corruption fixture only: the real exporter never creates this hole.
tamper(WormAuditTables.EXPORT, 1, "from_link_id", 3L);
assertThat(WormAuditHomeData.load(config, storage, now).exportGaps()).containsExactly("1–2");
for(long id = 1; id <= 12; id++)
{
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("id", id).withValue("requested_at", now.plusSeconds(id))
.withValue("requested_by", "<operator>").withValue("cutoff", OLD).withValue("outcome", id == 1 ? "PURGED" : "REFUSED")
.withValue("reason", "Retention not elapsed").withValue("anchor_link_id", 2L).withValue("anchor_hash", "abcd".repeat(16))
.withValue("audits_deleted", 0L).withValue("details_deleted", 0L).withValue("links_deleted", 0L)));
}
data = WormAuditHomeData.load(config, storage, now.plusSeconds(20));
assertThat(data.anchor().getValueLong("anchor_link_id")).isEqualTo(2);
assertThat(data.recentPurges()).hasSize(10).extracting(r -> r.getValueLong("id")).containsExactly(12L, 11L, 10L, 9L, 8L, 7L, 6L, 5L, 4L, 3L);
assertThat(data.exportGaps()).isEmpty();
assertThat(WormAuditHomeView.render(data, "/app")).contains("abcdabcdabcd…", "&lt;operator&gt;", "REFUSED").doesNotContain("abcd".repeat(16));
}
/*******************************************************************************
** Unknown/unsupported capability and stale schedule diagnostics stay value-free.
*******************************************************************************/
@Test
void capabilityAndDailyWarnings() throws Exception
{
var config = ChainVerificationTest.signed();
init(config);
var unsupported = new MemorySealedStore()
{
@Override
public boolean supportsLock()
{
return (false);
}
};
var unknown = new MemorySealedStore()
{
@Override
public CapabilityStatus lockCapability()
{
return (CapabilityStatus.UNKNOWN);
}
};
var error = new MemorySealedStore()
{
@Override
public boolean supportsLock()
{
throw (new IllegalStateException("CANARY-provider-secret"));
}
};
assertThat(WormAuditHomeData.load(config, unsupported, OLD).lockCapability()).isEqualTo("UNSUPPORTED");
assertThat(WormAuditHomeData.load(config, unknown, OLD).lockCapability()).isEqualTo("UNKNOWN");
String html = WormAuditHomeView.render(WormAuditHomeData.load(config, error, OLD), "/app");
assertThat(html).contains("UNKNOWN", "Object Lock capability check failed").doesNotContain("CANARY", "provider-secret");
VerificationRuns.save(new VerificationResult(OLD), "SCHEDULE", null, OLD);
assertThat(WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofHours(26))).warnings()).doesNotContain("Daily verification last ran more than 26 hours ago");
var stale = WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofHours(26)).plusSeconds(1));
assertThat(stale.warnings()).contains("Daily verification last ran more than 26 hours ago");
assertThat(WormAuditHomeView.renderHealthCard(stale, "/app/worm_audit")).contains("Setup needs attention");
config.withTrustedPublicKeys(Map.of());
assertThat(WormAuditHomeData.load(config, unsupported, OLD).warnings()).contains("No trusted public keys configured");
// USER and SEALED_EXPORT runs cannot refresh the daily CHAIN heartbeat.
VerificationRuns.save(new VerificationResult(OLD.plus(Duration.ofDays(2))), "USER", null, OLD.plus(Duration.ofDays(2)));
VerificationRuns.save(new VerificationResult(OLD.plus(Duration.ofDays(2))), "SCHEDULE", 1L, OLD.plus(Duration.ofDays(2)));
assertThat(WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofDays(2))).warnings()).contains("Daily verification last ran more than 26 hours ago");
}
/*******************************************************************************
** No portion of raw audit content belongs in either HTML widget.
*******************************************************************************/
@Test
void canaryNeverAppearsAndWidgetsRequireReadPermission() throws Exception
{
var config = ChainVerificationTest.signed().withAppName("evidence").withAppLabel("Evidence").withTablePathPrefix("/app");
var instance = init(config);
instance.addApp(new QAppMetaData().withName("admin").withLabel("Admin"));
config.withParentAppName("admin");
instance.getApp("evidence").setParentAppName("admin");
String canary = "CANARY-PHI-" + UUID.randomUUID();
audit(1);
detail(1, 1);
tamper("audit", 1, "message", canary);
tamper("auditDetail", 1, "message", canary);
tamper("auditDetail", 1, "oldValue", canary);
tamper("auditDetail", 1, "newValue", canary);
new WormAuditChainer(config).chainOnce();
var input = new RenderWidgetInput().withWidgetMetaData(instance.getWidget(WormAuditApp.HOME_WIDGET));
assertThatThrownBy(() -> new WormAuditApp.HomeRenderer().render(input)).isInstanceOf(QPermissionDeniedException.class);
assertThatThrownBy(() -> new WormAuditApp.HealthCardRenderer().render(input)).isInstanceOf(QPermissionDeniedException.class);
QContext.getQSession().withPermissions(WormAuditTables.PURGE + ".read");
String home = ((RawHTML) new WormAuditApp.HomeRenderer().render(input).getWidgetData()).getHtml();
String card = ((RawHTML) new WormAuditApp.HealthCardRenderer().render(input).getWidgetData()).getHtml();
for(int start = 0; start <= canary.length() - 5; start++)
{
assertThat(home + card).doesNotContain(canary.substring(start, start + 5));
}
assertThat(home + card).contains("overflow-wrap:anywhere", "color:inherit").doesNotContain("overflow-x", "<table", "#fff");
assertThat(card).contains("href=\"/app/admin/evidence\"");
assertThat(instance.getApp("evidence").getLabel()).isEqualTo("Evidence");
assertThat(instance.getApp("evidence").getParentAppName()).isEqualTo("admin");
assertThat(instance.getApp("evidence").getWidgets()).containsExactly(WormAuditApp.HOME_WIDGET);
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getName()).containsExactlyElementsOf(WormAuditTables.NAV);
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getLabel())
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges", "Settings");
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getGridColumns()).isEqualTo(12);
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getIsCard()).isFalse();
assertThat(instance.getWidget(WormAuditApp.HEALTH_CARD_WIDGET).getIsCard()).isTrue();
new QInstanceValidator().validate(instance);
}
/*******************************************************************************
** Exercise the real SQL anti-join without Docker, including its exact boundary.
*******************************************************************************/
@Test
void sqlSummaryUsesTheWriterAntiJoin() throws Exception
{
String url = "jdbc:h2:mem:home_" + UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;INIT=SET TIME ZONE 'UTC'";
var config = ChainVerificationTest.signed();
var instance = RdbmsTestSupport.instance(url, "sa", "", config, false);
try(var connection = DriverManager.getConnection(url, "sa", ""))
{
RdbmsTestSupport.createTables(connection, instance, false);
QContext.init(instance, new QSession());
lookups();
audit(1);
new WormAuditChainer(config).chainOnce(OLD.plusSeconds(100));
audit(2);
audit(3);
var data = WormAuditHomeData.load(config, new MemorySealedStore(), OLD.plusSeconds(63));
assertThat(data.links()).isEqualTo(1);
assertThat(data.backlog()).isEqualTo(1);
assertThat(data.settling()).isEqualTo(1);
assertThat(data.warnings()).contains("Daily verification has never run");
}
}
/*******************************************************************************
** The demo's history comes from real engines, including FAIL, recovery and PHI tokens.
*******************************************************************************/
@Test
void demoSeedsRealHistory() throws Exception
{
MemoryRecordStore.fullReset();
var application = new WormAuditDemoServer();
var instance = application.defineQInstance();
new QInstanceValidator().validate(instance);
application.seed(instance);
QContext.init(instance, new QSession());
var data = WormAuditHomeData.load();
assertThat(data.links()).isGreaterThanOrEqualTo(350);
assertThat(data.lastVerification().getValueString("outcome")).isEqualTo("PASS");
assertThat(data.openBreak()).isNull();
assertThat(data.exportGaps()).isEmpty();
assertThat(data.exportLag()).isEqualTo(5);
assertThat(data.backlog()).isEqualTo(2);
assertThat(data.settling()).isEqualTo(1);
assertThat(data.warnings()).hasSize(2);
assertThat(data.recentPurges()).extracting(r -> r.getValueString("outcome")).containsExactly("REFUSED", "PURGED");
assertThat(STORE.where(WormAuditTables.VERIFICATION, "outcome", "FAIL")).hasSize(1).allSatisfy(row ->
{
assertThat(row.getValueLong("first_break_link_id")).isPositive();
assertThat(row.getValueLong("first_break_audit_id")).isPositive();
});
assertThat(STORE.all("auditDetail")).filteredOn(r -> "diagnosis".equals(r.getValueString("fieldName")))
.isNotEmpty().allSatisfy(row -> assertThat(row.getValueString("newValue")).startsWith("hmac:v1:test:"));
var config = VerificationRuns.config();
// Settle and chain the final backlog before checking the entire live suffix.
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(120));
assertThat(new WormAuditVerifier(config).verify().passed()).isTrue();
assertThat(new SealedExportVerifier(config).verify(STORE.last(WormAuditTables.EXPORT), "USER").result().passed()).isTrue();
}
}
@@ -0,0 +1,307 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.model.SettingsCustomizer;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.*;
import static org.assertj.core.api.Assertions.*;
/*******************************************************************************
** Editable settings, bounded cache, audited changes and historical lock modes.
*******************************************************************************/
class WormAuditSettingsTest
{
static final String TABLE = WormAuditTables.SETTINGS;
/*******************************************************************************
** Deterministic clock for the exact 30-second cache boundary.
*******************************************************************************/
static class MutableClock extends Clock
{
Instant now = Instant.parse("2026-10-14T02:00:00Z");
/*******************************************************************************
** Verify get zone.
*******************************************************************************/
public ZoneId getZone()
{
return (ZoneOffset.UTC);
}
/*******************************************************************************
** Verify with zone.
*******************************************************************************/
public Clock withZone(ZoneId zone)
{
return (this);
}
/*******************************************************************************
** Verify instant.
*******************************************************************************/
public Instant instant()
{
return (now);
}
}
/*******************************************************************************
** Verify editor.
*******************************************************************************/
static void editor()
{
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit", "worm_audit.settings"));
}
/*******************************************************************************
** Verify update.
*******************************************************************************/
static void update(String field, java.io.Serializable value) throws Exception
{
var result = new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue(field, value))));
assertThat(result.getRecords()).allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty());
}
/*******************************************************************************
** Verify fallback overrides cache and insert only seed.
*******************************************************************************/
@Test
void fallbackOverridesCacheAndInsertOnlySeed() throws Exception
{
var config = new WormAuditQBitConfig().withRetentionYears(8).withExportRetentionDays(45).withCheckpointEveryLinks(12);
var instance = init(config);
var clock = new MutableClock();
var reader = new WormAuditSettings(config, clock);
assertThat(reader.read().audit_retention_years()).isEqualTo(8);
assertThat(reader.read().export_retention_days()).isEqualTo(45);
WormAuditSettings.seed(config);
editor();
update("export_retention_days", 90);
assertThat(reader.read().export_retention_days()).isEqualTo(90);
update("export_retention_days", 120);
clock.now = clock.now.plusSeconds(29);
assertThat(reader.read().export_retention_days()).isEqualTo(90);
clock.now = clock.now.plusSeconds(1);
assertThat(reader.read().export_retention_days()).isEqualTo(120);
WormAuditSettings.seed(config);
assertThat(STORE.get(TABLE, 1).getValueInteger("export_retention_days")).isEqualTo(120);
assertThat(reader.read().checkpoint_every_links()).isEqualTo(12);
instance.getTables().remove(TABLE);
reader.invalidate();
assertThat(reader.read().export_retention_days()).isEqualTo(45);
}
/*******************************************************************************
** Verify partial null row uses config and invalid persisted settings fail closed.
*******************************************************************************/
@Test
void partialNullRowUsesConfigAndInvalidPersistedSettingsFailClosed() throws Exception
{
var config = new WormAuditQBitConfig().withExportRetentionDays(42);
init(config);
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().insert(
new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 1).withValue("export_lock_mode", "COMPLIANCE")), false);
var reader = new WormAuditSettings(config);
assertThat(reader.read().export_retention_days()).isEqualTo(42);
assertThat(reader.read().export_lock_mode()).isEqualTo("COMPLIANCE");
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().update(
new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_retention_days", 0))), false);
reader.invalidate();
assertThatThrownBy(reader::read).hasMessageContaining("Export retention days");
}
/*******************************************************************************
** Verify missing physical table falls back.
*******************************************************************************/
@Test
void missingPhysicalTableFallsBack() throws Exception
{
var config = new WormAuditQBitConfig().withExportRetentionDays(81);
var instance = RdbmsTestSupport.instance("jdbc:h2:mem:settings_missing;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", "", config, false);
QContext.init(instance, new QSession());
assertThat(config.effectiveSettings().export_retention_days()).isEqualTo(81);
}
/*******************************************************************************
** Verify validation and capabilities.
*******************************************************************************/
@Test
void validationAndCapabilities() throws Exception
{
var config = new WormAuditQBitConfig();
var instance = init(config);
var table = instance.getTable(TABLE);
assertThat(table.getLabel()).isEqualTo("WORM Audit Settings");
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(AuditLevel.FIELD);
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_UPDATE)).isTrue();
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_INSERT)).isFalse();
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_DELETE)).isFalse();
WormAuditSettings.seed(config);
editor();
assertThatThrownBy(() -> new DeleteAction().execute(new DeleteInput(TABLE).withPrimaryKeys(List.of(1)))).isInstanceOf(Exception.class);
assertThatThrownBy(() -> new InsertAction().execute(new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 2)))).isInstanceOf(Exception.class);
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(new QRecord().withValue("id", 2)), false)).hasMessageContaining("one row only (ID 1)");
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(WormAuditSettings.defaults(config)), false))
.hasMessageContaining("already exists");
assertThatThrownBy(() -> new SettingsCustomizer().preDelete(new DeleteInput(TABLE), List.of(STORE.get(TABLE, 1)), false))
.hasMessageContaining("cannot be deleted");
for(String field : List.of("audit_retention_years", "export_retention_days", "checkpoint_every_links", "checkpoint_every_minutes", "chain_interval_seconds", "late_detail_lookback_hours"))
{
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue(field, 0))).hasMessageContaining("whole number");
}
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("settle_horizon_seconds", -1))).hasMessageContaining("Settle horizon seconds");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("audit_retention_years", 101))).hasMessageContaining("100");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", 36501))).hasMessageContaining("36500");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", "1.5"))).hasMessageContaining("whole number");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", "NONE"))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", null))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
for(String field : List.of("export_cron", "verify_cron"))
{
assertThatThrownBy(() -> new SettingsCustomizer().preUpdate(new UpdateInput(TABLE), List.of(new QRecord().withValue("id", 1).withValue(field, "bad")), false, java.util.Optional.empty()))
.hasMessageContaining("valid Quartz cron");
}
assertThatThrownBy(() -> new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_cron", "bad")))))
.hasMessageContaining("valid Quartz cron");
assertThat(STORE.get(TABLE, 1).getValueString("export_cron")).isEqualTo(config.getExportCron());
}
/*******************************************************************************
** Verify edit requires both permissions and produces field audit that chains.
*******************************************************************************/
@Test
void editRequiresBothPermissionsAndProducesFieldAuditThatChains() throws Exception
{
var config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
init(config);
WormAuditSettings.seed(config);
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit"));
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit.settings");
QContext.getQSession().setPermissions(new java.util.HashSet<>(Set.of("worm_audit.settings")));
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit_settings.edit");
editor();
update("export_lock_mode", "COMPLIANCE");
assertThat(STORE.all("audit")).hasSize(1);
assertThat(STORE.where("auditDetail", "fieldName", "export_lock_mode")).hasSize(1);
assertThat(STORE.get(TABLE, 1).getValueInstant("modify_date")).isNotNull();
assertThat(STORE.get(TABLE, 1).getValueString("modified_by")).isEqualTo("system");
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(1));
assertThat(STORE.all(WormAuditTables.CHAIN)).hasSize(1);
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(1)).passed()).isTrue();
}
/*******************************************************************************
** Verify startup schedules and home use effective settings.
*******************************************************************************/
@Test
void startupSchedulesAndHomeUseEffectiveSettings() throws Exception
{
var config = ChainVerificationTest.signed();
var instance = init(config);
WormAuditSettings.seed(config);
editor();
update("chain_interval_seconds", 45);
update("export_cron", "0 15 4 * * ?");
update("verify_cron", "0 30 5 * * ?");
update("audit_retention_years", 7);
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(30);
var restarted = WormAuditTestSupport.core(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
new WormAuditQBitProducer().withQBitConfig(config).produce(restarted);
assertThat(QContext.getQInstance()).isSameAs(instance);
instance = restarted;
QContext.init(instance, new QSession());
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(45);
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 15 4 * * ?");
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 30 5 * * ?");
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC");
audit(50);
var home = WormAuditHomeData.load(config, new SealedExportTest.MemorySealedStore(), Instant.now());
assertThat(home.nextPurge()).isEqualTo(OLD.plusSeconds(50).atZone(ZoneOffset.UTC).plusYears(7).toInstant());
assertThat(home.warnings()).anyMatch(w -> w.contains("differs from") && w.contains("purge interval"));
assertThat(WormAuditHomeView.render(home, "/app")).contains("GOVERNANCE", "s3:BypassGovernanceRetention", "Switch to COMPLIANCE", "30 days", "Export retention is shorter");
}
/*******************************************************************************
** Verify switch affects new exports only and old checkpoint cadence remains valid.
*******************************************************************************/
@Test
void switchAffectsNewExportsOnlyAndOldCheckpointCadenceRemainsValid() throws Exception
{
var config = ChainVerificationTest.signed().withCheckpointEveryLinks(2).withSettleHorizonSeconds(0);
init(config);
WormAuditSettings.seed(config);
var storage = new SealedExportTest.MemorySealedStore();
audit(1);
audit(2);
Instant now = Instant.now().plusSeconds(1);
new WormAuditChainer(config).chainOnce(now);
var exporter = new SealedExporter(config, storage);
var first = exporter.run("USER", now).row();
assertThat(first).isNotNull();
assertThat(first.getValueString("lock_mode")).isEqualTo("GOVERNANCE");
assertThat(first.getValueInstant("retain_until")).isEqualTo(now.truncatedTo(java.time.temporal.ChronoUnit.MILLIS).plusSeconds(30 * 86400L));
assertThat(storage.heads.values()).allSatisfy(h -> assertThat(h.getLockMode()).isEqualTo("GOVERNANCE"));
var verifier = new SealedExportVerifier(config, storage);
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
editor();
update("export_lock_mode", "COMPLIANCE");
update("checkpoint_every_links", 1);
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(2));
var second = exporter.run("USER", Instant.now().plusSeconds(2)).row();
assertThat(second.getValueString("lock_mode")).isEqualTo("COMPLIANCE");
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
assertThat(verifier.verify(second, "USER").result().passed()).isTrue();
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(2)).passed()).isTrue();
String key = first.getValueString("object_prefix") + "links.jsonl";
storage.heads.get(key).withLockMode("COMPLIANCE");
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains("lacks the recorded GOVERNANCE retention"));
storage.heads.remove(key);
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains(key) && b.reason().contains("GOVERNANCE bypass removal"));
}
}
@@ -0,0 +1,106 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.demo;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.Map;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
import com.kingsrook.qbits.objectstorage.api.Page;
import com.kingsrook.qbits.objectstorage.api.StoredObject;
import com.kingsrook.qbits.wormaudit.export.SealedStore;
/*******************************************************************************
** Tiny test-scope immutable store with version and COMPLIANCE retention echoes.
*******************************************************************************/
public class MemorySealedStore implements SealedStore
{
private final Map<String, byte[]> objects = new LinkedHashMap<>();
private final Map<String, StoredObject> heads = new LinkedHashMap<>();
/*******************************************************************************
** Demo capability.
*******************************************************************************/
@Override
public boolean supportsLock()
{
return (true);
}
/*******************************************************************************
** Refuse overwrites and retain the exact bytes.
*******************************************************************************/
@Override
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
{
if(objects.containsKey(key))
{
throw (new IllegalStateException("Demo sealed object already exists"));
}
String version = "v" + (objects.size() + 1);
objects.put(key, Files.readAllBytes(file));
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode(mode.name()).withLockRetainUntil(until));
return (version);
}
/*******************************************************************************
** Exact retention metadata.
*******************************************************************************/
@Override
public StoredObject head(String key)
{
return (heads.get(key));
}
/*******************************************************************************
** Exact version read.
*******************************************************************************/
@Override
public InputStream getVersion(String key, String versionId)
{
if(!heads.containsKey(key) || !heads.get(key).getVersionId().equals(versionId))
{
throw (new IllegalArgumentException("Demo sealed version is missing"));
}
return (new ByteArrayInputStream(objects.get(key)));
}
/*******************************************************************************
** The small demo fits one page.
*******************************************************************************/
@Override
public Page<ObjectSummary> list(ListRequest request)
{
return (new Page<ObjectSummary>().withItems(objects.keySet().stream().filter(k -> k.startsWith(request.getPrefix()))
.map(k -> new ObjectSummary().withKey(k)).toList()));
}
}
@@ -0,0 +1,287 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.demo;
import java.security.KeyPairGenerator;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.wormaudit.PhiCanaryTest;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditQBitProducer;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.audits.AuditActionCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditSingleInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
/*******************************************************************************
** Local demo: synthetic notes, test HMAC tokens, real chain, checkpoints,
** verifications and sealed exports over a moving clock. Memory storage and its
** simulated purge are not production enforcement.
**
** <pre>
** ./mvnw -o test-compile exec:java -Dexec.classpathScope=test -Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer -Dexec.args=8099
** </pre>
*******************************************************************************/
public class WormAuditDemoServer extends AbstractQQQApplication
{
private static Clock clock = Clock.systemUTC();
private final MemorySealedStore storage = new MemorySealedStore();
private WormAuditQBitConfig config;
/*******************************************************************************
** The PhiCanaryTest stand-in, with a moving timestamp before audit insertion.
*******************************************************************************/
public static class DemoTokenHook extends PhiCanaryTest.TokenHook
{
/*******************************************************************************
** Tokenize through the same core customization point as the canary test.
*******************************************************************************/
@Override
public void customizeInput(AuditSingleInput input)
{
input.withTimestamp(clock.instant());
super.customizeInput(input);
}
}
/*******************************************************************************
** Fully anonymous demo with a FIELD-audited synthetic note table.
*******************************************************************************/
@Override
public QInstance defineQInstance() throws QException
{
try
{
var keys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair();
config = new WormAuditQBitConfig().withBackendName("memory").withCheckpointEveryLinks(20).withCheckpointEveryMinutes(1)
.withSigningKey(() -> new WormAuditQBitConfig.SigningKey("demo", keys.getPrivate())).withTrustedPublicKeys(Map.of("demo", keys.getPublic()))
.withSealedStoreSupplier(() -> storage)
// Synthetic local demo only: anonymous readers can view its tables.
.withTableMetaDataCustomizer((instance, table) -> table.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED)));
QInstance instance = new QInstance().withInstanceDefaultAuthentication(new QAuthenticationMetaData()
.withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
new AuditsMetaDataProvider().defineAll(instance, "memory", null);
instance.addTable(new QTableMetaData().withName("patient_note").withLabel("Synthetic patient note").withBackendName("memory")
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("Note #%s")
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("diagnosis", QFieldType.STRING).withLabel("Diagnosis").withSupplementalMetaData(new PhiCanaryTest.Phi()))
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)));
instance.addSupplementalCustomizer(AuditActionCustomizerInterface.CUSTOMIZER_TYPE, new QCodeReference(DemoTokenHook.class));
new WormAuditQBitProducer().withQBitConfig(config).produce(instance);
instance.setHomeAppName(config.getAppName());
return (instance);
}
catch(Exception e)
{
throw (new QException("Cannot define WORM Audit demo", e));
}
}
/*******************************************************************************
** Write through core DML and the test token hook at the current demo time.
*******************************************************************************/
private void notes(int count) throws QException
{
for(int i = 0; i < count; i++)
{
new InsertAction().execute(new InsertInput("patient_note").withRecord(new QRecord().withValue("diagnosis", "SYNTHETIC-DEMO-NOTE-" + i)));
}
}
/*******************************************************************************
** Persist a real verifier result with the historical run time.
*******************************************************************************/
private void verify() throws QException
{
var result = new WormAuditVerifier(config).verify(null, clock.instant());
VerificationRuns.save(result, "SCHEDULE", null, clock.instant());
}
/*******************************************************************************
** Use the production exporter and configured storage boundary.
*******************************************************************************/
private void seal() throws QException
{
var result = new SealedExporter(config).run("SCHEDULE", clock.instant());
if(result.failed() || result.row() == null)
{
throw (new QException("Demo seeding failed to seal its checkpointed range"));
}
}
/*******************************************************************************
** A fortnight of activity and a genuinely expired, sealed prefix.
*******************************************************************************/
public void seed(QInstance instance) throws Exception
{
QContext.init(instance, new QSession());
com.kingsrook.qbits.wormaudit.WormAuditSettings.seed(config);
AuditStore store = new AuditStore(null);
Instant today = Instant.now().minus(Duration.ofHours(2));
try
{
Instant expired = today.atZone(ZoneOffset.UTC).minusYears(7).toInstant();
clock = Clock.fixed(expired, ZoneOffset.UTC);
notes(3);
clock = Clock.fixed(expired.plusSeconds(120), ZoneOffset.UTC);
new WormAuditChainer(config).chainOnce(clock.instant());
clock = Clock.fixed(expired.plusSeconds(240), ZoneOffset.UTC);
new WormAuditChainer(config).chainOnce(clock.instant());
verify();
seal();
QRecord anchor = store.last(WormAuditTables.CHAIN);
long auditCount = store.count("audit", new QQueryFilter());
long detailCount = store.count("auditDetail", new QQueryFilter());
// Demo data: memory has no purge function. Remove the expired, sealed
// prefix and append the receipt the PostgreSQL function would write.
for(String table : List.of("auditDetail", "audit", WormAuditTables.CHAIN))
{
for(QRecord row : store.all(table))
{
MemoryRecordStore.getInstance().delete(new DeleteInput(table).withPrimaryKeys(List.of(row.getValue("id"))));
}
}
store.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("requested_at", today.minus(Duration.ofDays(15)))
.withValue("requested_by", "demo-purger").withValue("cutoff", expired.plusSeconds(1)).withValue("retention_interval", "6 years")
.withValue("outcome", "PURGED").withValue("reason", "Expired sealed prefix removed (demo memory purge)")
.withValue("audits_deleted", auditCount).withValue("details_deleted", detailCount).withValue("links_deleted", anchor.getValueLong("id"))
.withValue("anchor_link_id", anchor.getValueLong("id")).withValue("anchor_hash", anchor.getValueString("link_hash"))));
for(int day = 13; day >= 0; day--)
{
Instant time = today.minus(Duration.ofDays(day));
clock = Clock.fixed(time, ZoneOffset.UTC);
notes(25);
clock = Clock.fixed(time.plusSeconds(120), ZoneOffset.UTC);
new WormAuditChainer(config).chainOnce(clock.instant());
clock = Clock.fixed(time.plusSeconds(240), ZoneOffset.UTC);
new WormAuditChainer(config).chainOnce(clock.instant());
if(day == 7)
{
// Demo seeding only: temporarily tamper a memory detail, persist a
// genuine FAIL naming that link, then restore before later PASSes.
QRecord detail = store.all("auditDetail").getFirst();
var original = detail.getValue("newValue");
memoryValue(detail, "hmac:v1:demo:tampered");
verify();
memoryValue(detail, original);
}
else
{
verify();
}
seal();
}
// Demo REFUSED receipt: too-recent cutoff; no rows removed.
store.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("requested_at", today.plusSeconds(300))
.withValue("requested_by", "demo-purger").withValue("cutoff", today).withValue("retention_interval", "6 years")
.withValue("outcome", "REFUSED").withValue("reason", "Cutoff is inside the retention interval")
.withValue("audits_deleted", 0L).withValue("details_deleted", 0L).withValue("links_deleted", 0L)));
// Real unsealed suffix and unchained backlog; no invented export gap.
clock = Clock.fixed(today.plusSeconds(600), ZoneOffset.UTC);
notes(5);
new WormAuditChainer(config).chainOnce(today.plusSeconds(720));
clock = Clock.fixed(Instant.now().minusSeconds(120), ZoneOffset.UTC);
notes(2);
clock = Clock.systemUTC();
notes(1);
}
finally
{
clock = Clock.systemUTC();
QContext.clear();
}
}
/*******************************************************************************
** Deliberate memory-only tamper used solely while seeding the FAIL receipt.
*******************************************************************************/
private static void memoryValue(QRecord detail, java.io.Serializable value) throws QException
{
MemoryRecordStore.getInstance().update(new UpdateInput("auditDetail").withRecords(List.of(new QRecord()
.withValue("id", detail.getValue("id")).withValue("newValue", value))), false);
}
/*******************************************************************************
** Serve the standard widget API and packaged Next UI, reusing the seeded keys.
*******************************************************************************/
public static void main(String[] args) throws Exception
{
int port = args.length == 0 ? 8099 : Integer.parseInt(args[0]);
WormAuditDemoServer application = new WormAuditDemoServer();
QInstance instance = application.defineQInstance();
new QInstanceValidator().validate(instance);
application.seed(instance);
new QApplicationJavalinServer(new AbstractQQQApplication()
{
@Override
public QInstance defineQInstance()
{
return (instance);
}
}).withPort(port).withServeFrontendNext(true).start();
System.out.println("WORM Audit demo on http://localhost:" + port + "/app/worm_audit");
}
}
@@ -0,0 +1,272 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.verify;
import java.security.KeyPairGenerator;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.canonical.AuditCanonical;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.chain.Hashes;
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Instrument the storage boundary independently of the memory backend: no
** unbounded reads are allowed, including portable completeness checks.
*******************************************************************************/
class PagedVerificationTest
{
private static final int LINKS = 25000;
private static final int PAGE = 137;
private static final int RANGE = 997;
private static final Instant NOW = Instant.parse("2026-10-13T00:00:00Z");
/*******************************************************************************
** Cross many non-aligned page/checkpoint boundaries with genuine signatures.
*******************************************************************************/
@ParameterizedTest
@ValueSource(booleans = { true, false })
void fullRunBoundsPagesAndCheckpointRanges(boolean sql) throws Exception
{
var keys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair();
var config = new WormAuditQBitConfig().withCheckpointEveryLinks(RANGE)
.withSigningKey(() -> new WormAuditQBitConfig.SigningKey("test", keys.getPrivate()))
.withTrustedPublicKeys(Map.of("test", keys.getPublic()));
var store = new InstrumentedStore(config, sql);
var result = new WormAuditVerifier(config, PAGE).verify(store, null, NOW);
assertThat(result.getBreaks()).isEmpty();
assertThat(result.getLinksChecked()).isEqualTo(LINKS);
assertThat(result.getFromLinkId()).isEqualTo(1);
assertThat(result.getToLinkId()).isEqualTo(LINKS);
assertThat(store.maxPage).isEqualTo(PAGE);
assertThat(store.maxRange).isEqualTo(RANGE);
assertThat(store.checkpointsRead).isEqualTo(26);
assertThat(store.completenessVisited).isTrue();
if(!sql)
{
store.links.get(0).withValue("detail_ids", "1,1,2");
store.links.get(1).withValue("detail_ids", "2");
store.links.get(2).withValue("detail_ids", "malformed");
store.links.get(3).withValue("audit_id", 1L);
store.audits.get(2).withValue("timestamp", null);
store.audits.get(4).withValue("timestamp", NOW);
store.details.addAll(List.of(new QRecord().withValue("id", 1L).withValue("auditId", 1L),
new QRecord().withValue("id", 2L).withValue("auditId", 2L),
new QRecord().withValue("id", 3L).withValue("auditId", 99999L),
new QRecord().withValue("id", 4L).withValue("auditId", 3L)));
var broken = new WormAuditVerifier(config, PAGE).verify(store, null, NOW);
assertThat(broken.getBreaks()).extracting(finding -> finding.reason()).contains(
"audit 1 is covered twice or more by the chain", "audit 4 is not covered by the chain",
"audit 3 has no timestamp", "audit_detail 1 is covered twice or more by the chain",
"audit_detail 2 is covered twice or more by the chain", "audit_detail 3 is not covered by the chain",
"audit_detail 4 is not covered by the chain");
}
assertThatThrownBy(() -> new WormAuditVerifier(config, 0)).isInstanceOf(IllegalArgumentException.class);
assertThatThrownBy(() -> new WormAuditVerifier(config, 1001)).isInstanceOf(IllegalArgumentException.class);
}
/*******************************************************************************
** Fixture storage may retain its data; the verifier can only fetch bounded
** pages. Track the total hashes read between successive checkpoint requests.
*******************************************************************************/
private static class InstrumentedStore extends AuditStore
{
private final List<QRecord> audits = new ArrayList<>();
private final List<QRecord> details = new ArrayList<>();
private final List<QRecord> links = new ArrayList<>();
private final List<QRecord> checkpoints = new ArrayList<>();
private final boolean sql;
private int maxPage;
private int maxRange;
private int rangeRead;
private int checkpointsRead;
private boolean inCheckpoints;
private boolean completenessVisited;
private boolean mainScanFinished;
/*******************************************************************************
** Generate signed, valid content without involving the backend under test.
*******************************************************************************/
InstrumentedStore(WormAuditQBitConfig config, boolean sql) throws Exception
{
super(null);
this.sql = sql;
String previous = Hashes.GENESIS;
for(long id = 1; id <= LINKS; id++)
{
QRecord audit = new QRecord().withValue("id", id).withValue("timestamp", NOW.minusSeconds(120))
.withValue("auditTableId", 1).withValue("auditUserId", 1).withValue("recordId", id);
audits.add(audit);
String hash = Hashes.link(previous, AuditCanonical.bytes(audit, "fixture", "fixture", List.of()));
links.add(new QRecord().withValue("id", id).withValue("audit_id", id).withValue("kind", "AUDIT")
.withValue("detail_ids", "").withValue("prev_hash", previous).withValue("link_hash", hash).withValue("chained_at", NOW));
previous = hash;
}
for(int start = 0; start < LINKS; start += RANGE)
{
checkpoints.add(Checkpoints.sign(links.subList(start, Math.min(start + RANGE, LINKS)), config, NOW)
.withValue("id", (long) checkpoints.size() + 1));
}
}
/*******************************************************************************
** No full-table read is ever permitted through this boundary.
*******************************************************************************/
@Override
public List<QRecord> all(String table)
{
throw (new AssertionError("Unbounded verifier read: " + table));
}
/*******************************************************************************
** A fresh copied page prevents subList from retaining the fixture chain.
*******************************************************************************/
@Override
public List<QRecord> page(String table, long after, int limit)
{
assertThat(limit).isBetween(1, PAGE);
maxPage = Math.max(maxPage, limit);
List<QRecord> source;
if(table.equals(WormAuditTables.CHECKPOINT))
{
assertThat(mainScanFinished).isTrue();
assertThat(limit).isEqualTo(1);
maxRange = Math.max(maxRange, rangeRead);
assertThat(rangeRead).isLessThanOrEqualTo(RANGE);
rangeRead = 0;
inCheckpoints = after < checkpoints.size();
if(inCheckpoints)
{
checkpointsRead++;
}
source = checkpoints;
}
else if(table.equals(WormAuditTables.CHAIN))
{
source = links;
if(after == LINKS && !inCheckpoints)
{
mainScanFinished = true;
}
}
else
{
completenessVisited = true;
source = table.equals("audit") ? audits : details;
}
int start = (int) Math.min(after, source.size());
var page = new ArrayList<>(source.subList(start, Math.min(start + limit, source.size())));
if(inCheckpoints && table.equals(WormAuditTables.CHAIN))
{
rangeRead += page.size();
}
return (page);
}
/*******************************************************************************
** Batch only the audit headers belonging to the current chain page.
*******************************************************************************/
@Override
public List<QRecord> in(String table, String field, Collection<Long> ids)
{
assertThat(ids.size()).isLessThanOrEqualTo(PAGE);
return (ids.stream().distinct().map(id -> (table.equals("audit") ? audits : details).get(id.intValue() - 1)).toList());
}
/*******************************************************************************
** Resolve detail parents without collecting audit history.
*******************************************************************************/
@Override
public QRecord get(String table, long id)
{
return (id > audits.size() ? null : audits.get((int) id - 1));
}
/*******************************************************************************
** Stable lookup values match the fixture canonical documents.
*******************************************************************************/
@Override
public String name(String table, Long id)
{
return ("fixture");
}
/*******************************************************************************
** Start at genesis.
*******************************************************************************/
@Override
public QRecord anchor()
{
return (new QRecord().withValue("anchor_link_id", 0L).withValue("anchor_hash", Hashes.GENESIS));
}
/*******************************************************************************
** Exercise both SQL dispatch and portable non-memory completeness.
*******************************************************************************/
@Override
public boolean isRdbms()
{
return (sql);
}
/*******************************************************************************
** Never permit the memory fallback in this test.
*******************************************************************************/
@Override
public boolean isMemory()
{
return (false);
}
/*******************************************************************************
** SQL completeness has separate acceptance tests; verify dispatch here.
*******************************************************************************/
@Override
public void completeness(Instant cutoff, VerificationResult result)
{
completenessVisited = true;
}
}
}