From f380b9b2a1f5b6dbf7c7071807eeabf6e2f87f79 Mon Sep 17 00:00:00 2001 From: James Maes Date: Tue, 13 Oct 2026 20:28:00 -0500 Subject: [PATCH 1/2] feat: append-only audit tables, hash chain, signed checkpoints and verification Phase (a) of docs/DESIGN.md: - Postgres DDL template and renderer: write-once triggers on the core audit tables and worm_audit_* tables, lookup id/name protection, the SECURITY DEFINER retention purge (call-stack guarded, chain prefix only, recorded) and the INSERT/SELECT-only grant model; - capability guard and a clear write-once error; - RFC 8785 canonical audit content, a single-writer chainer (advisory lock, settle horizon, contiguous ids, late arrivals and late details), Ed25519 checkpoints with Merkle roots; - a paged verifier naming the exact break, the Verify audit chain table action, and the daily verification with QAlerts break/heartbeat; - the assistant deny pattern for worm_audit_* (reflective until RC.6); - tests: Postgres Testcontainers (triggers, grants, purge, tamper, concurrency, time zone), PHI canary, chain and checkpoint units. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 14 +- CHANGELOG.md | 7 +- CLAUDE.md | 14 +- README.md | 203 ++++++- docs/DESIGN.md | 383 ++++++++++++ docs/ddl/worm-audit-postgres.sql | 325 +++++++++++ pom.xml | 26 +- .../qbits/wormaudit/WormAuditQBitConfig.java | 322 +++++++++- .../wormaudit/WormAuditQBitProducer.java | 103 +++- .../wormaudit/canonical/AuditCanonical.java | 74 +++ .../qbits/wormaudit/canonical/Jcs.java | 130 +++++ .../qbits/wormaudit/chain/AuditStore.java | 479 +++++++++++++++ .../qbits/wormaudit/chain/Hashes.java | 99 ++++ .../wormaudit/chain/WormAuditChainer.java | 224 +++++++ .../wormaudit/checkpoint/Checkpoints.java | 120 ++++ .../qbits/wormaudit/ddl/WormAuditDdl.java | 112 ++++ .../wormaudit/guard/AppendOnlyGuard.java | 81 +++ .../wormaudit/model/LinkRangeWidget.java | 88 +++ .../wormaudit/model/WormAuditTables.java | 124 ++++ .../wormaudit/processes/VerificationRuns.java | 116 ++++ .../processes/WormAuditProcesses.java | 312 ++++++++++ .../wormaudit/verify/VerificationResult.java | 165 ++++++ .../wormaudit/verify/WormAuditVerifier.java | 477 +++++++++++++++ .../wormaudit/ddl/postgres/worm-audit.sql | 325 +++++++++++ .../wormaudit/CanonicalAndCryptoTest.java | 204 +++++++ .../wormaudit/ChainVerificationTest.java | 347 +++++++++++ .../qbits/wormaudit/DockerConditionTest.java | 43 ++ .../qbits/wormaudit/PhiCanaryTest.java | 206 +++++++ .../wormaudit/PostgresAcceptanceTest.java | 552 ++++++++++++++++++ .../wormaudit/ProcessesAndAlertsTest.java | 234 ++++++++ .../qbits/wormaudit/RdbmsChainTest.java | 212 +++++++ .../qbits/wormaudit/RdbmsTestSupport.java | 157 +++++ .../wormaudit/RecordIntegrationTest.java | 125 ++++ .../wormaudit/RequiresDockerCondition.java | 67 +++ .../wormaudit/WormAuditQBitProducerTest.java | 159 +++-- .../qbits/wormaudit/WormAuditTestSupport.java | 136 +++++ src/test/resources/log4j2-test.xml | 17 + 37 files changed, 6683 insertions(+), 99 deletions(-) create mode 100644 docs/DESIGN.md create mode 100644 docs/ddl/worm-audit-postgres.sql create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/canonical/AuditCanonical.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/canonical/Jcs.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/chain/AuditStore.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/chain/Hashes.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/chain/WormAuditChainer.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/checkpoint/Checkpoints.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/ddl/WormAuditDdl.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/guard/AppendOnlyGuard.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/model/LinkRangeWidget.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/model/WormAuditTables.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/processes/VerificationRuns.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/processes/WormAuditProcesses.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/verify/VerificationResult.java create mode 100644 src/main/java/com/kingsrook/qbits/wormaudit/verify/WormAuditVerifier.java create mode 100644 src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/CanonicalAndCryptoTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/ChainVerificationTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/DockerConditionTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/PhiCanaryTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/PostgresAcceptanceTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/ProcessesAndAlertsTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/RdbmsChainTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/RdbmsTestSupport.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/RecordIntegrationTest.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/RequiresDockerCondition.java create mode 100644 src/test/java/com/kingsrook/qbits/wormaudit/WormAuditTestSupport.java create mode 100644 src/test/resources/log4j2-test.xml diff --git a/AGENTS.md b/AGENTS.md index 7efc88c..8f17595 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,7 +18,19 @@ Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. O ## Architecture -Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer implements QBitMetaDataProducer`. Config exposes `tableMetaDataCustomizer` and `retentionYears` (default 6). +Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer implements QBitMetaDataProducer`. Config requires the host `backendName`; core audit tables must exist before production. + +- `ddl`: reviewed Liquibase PostgreSQL template and safe renderer; public sidecar tables and owner-only retention purge. +- `guard`: immutable QQQ capabilities and value-free SQLSTATE mapping. +- `canonical`, `chain`, `checkpoint`: strict JCS v1, transaction/advisory-locked writer, explicit detail coverage, pinned Ed25519 ranges. +- `verify`: snapshot checks of content, continuity, coverage and checkpoints, starting at the purge anchor. +- `processes`: standard streamed ETL action, scheduled chainer and daily QAlerts producer; never app navigation entries. +- `model`: read-only tables, range record references and standard record child lists. No app-home widgets or export execution in phase (a). + +PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM. +Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set. +Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata; +signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration. ## Invariants and naming diff --git a/CHANGELOG.md b/CHANGELOG.md index 89a967a..20c6b5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,4 +6,9 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ### Added -- Initial scaffold +- Phase (a): PostgreSQL append-only tables/triggers, safe DDL renderer, retention purge with immutable refusal/anchor records, and runtime grants. +- Canonical audit v1 documents, contiguous hash chain with late-detail coverage, pinned Ed25519 checkpoints and binary Merkle roots. +- Snapshot verification, streamed ETL Review/Result action, scheduled chaining/daily verification, and core QAlerts raise/resolve/heartbeat integration. +- Read-only metadata, WORM Audit table navigation, record range references and child lists; reflective RC.6 assistant deny registration. +- Offline memory/H2 acceptance, real pre-insert PHI canary hook, and PostgreSQL Testcontainers acceptance with CI-strict Docker availability. +- Adoption and operations documentation. Export execution and app-home widgets remain later phases. diff --git a/CLAUDE.md b/CLAUDE.md index 4ca02cc..f020596 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,7 +18,19 @@ Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. O ## Architecture -Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer implements QBitMetaDataProducer`. Config exposes `tableMetaDataCustomizer` and `retentionYears` (default 6). +Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer implements QBitMetaDataProducer`. Config requires the host `backendName`; core audit tables must exist before production. + +- `ddl`: reviewed Liquibase PostgreSQL template and safe renderer; public sidecar tables and owner-only retention purge. +- `guard`: immutable QQQ capabilities and value-free SQLSTATE mapping. +- `canonical`, `chain`, `checkpoint`: strict JCS v1, transaction/advisory-locked writer, explicit detail coverage, pinned Ed25519 ranges. +- `verify`: snapshot checks of content, continuity, coverage and checkpoints, starting at the purge anchor. +- `processes`: standard streamed ETL action, scheduled chainer and daily QAlerts producer; never app navigation entries. +- `model`: read-only tables, range record references and standard record child lists. No app-home widgets or export execution in phase (a). + +PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM. +Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set. +Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata; +signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration. ## Invariants and naming diff --git a/README.md b/README.md index 1322288..2cb699d 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,200 @@ # QBit WORM Audit -Write-once, read-many (WORM) audit for QQQ applications. The QQQ core audit tables (`audit`, `audit_detail`) and this qbit's own tables become append-only at the database level, every audit row is linked into a hash chain with periodic signed checkpoints, a verification action and a daily job detect tampering, and new audit rows are sealed into an S3 bucket with Object Lock in COMPLIANCE mode (default retention 6 years, the HIPAA audit-record retention). Audit content is hashed and sealed exactly as core writes it; with qbit-secured-data, PHI field values in audit details are already HMACs, so no raw PHI reaches the chain or the sealed copies. This repository currently contains the compiling scaffold; the design is in [docs/DESIGN.md](docs/DESIGN.md) once the spec PR lands. +Phase (a) provides PostgreSQL append-only enforcement, an asynchronous audit hash chain, +Ed25519 checkpoints, a verification action, and daily verification with core QAlerts signals. +It hashes exactly the audit values core stored; it does not read application records or +secured-data keys. Sealed exports and the app home are later phases. The export table and +range references are present as integration hooks; no export job or home widgets are installed. -## Requirements - -Java 21 and QQQ 4.1.0-RC.5 or later. PostgreSQL is the supported database for the append-only enforcement. The Maven wrapper pins Maven 3.9.11. - -## Install - -```xml - - com.kingsrook.qbits - qbit-worm-audit - 0.1.0-SNAPSHOT - -``` +Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement; +memory/H2 support tamper-evidence tests without database enforcement. ## Tables and permissions -Planned tables (none are produced yet): `worm_audit_chain`, `worm_audit_checkpoint`, `worm_audit_export`, `worm_audit_verification`, `worm_audit_purge`. Backend names match table names. Permissions use the full `worm_audit` prefix. +All sidecar tables use the configured host backend, snake_case fields, BIGINT identifiers, +record sections, and full table-name permissions. They do not audit themselves. The app lists +Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that order. + +| Table | Purpose | Permission | +|---|---|---| +| `worm_audit_verification` | Immutable USER/SCHEDULE results | `worm_audit_verification.read` | +| `worm_audit_export` | Reserved sealed-export catalog | `worm_audit_export.read` | +| `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` | +| `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` | +| `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` | + +`worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action. +Processes are attached to the chain table or scheduled; none is a navigation entry. +Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities +are removed from all four core audit tables. Internal appends use an explicit permission bypass. +Checkpoint, export and verification record screens have a bounded chain-link child list; +range endpoints and export/checkpoint references link to their records. + +The table metadata customizer can configure host ESB publications. Quick Search registration +is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables. +Verification inserts use QQQ's record-change listener SPI (as will phase-(b) export inserts), +so an installed ESB publisher receives those events when the host configures its destinations +on these two tables. The qbit defines no destination and configures no chain-link publication. ## Configuration -`WormAuditQBitConfig` exposes an optional `tableMetaDataCustomizer` and `retentionYears` (default 6). +| Field | Default / contract | +|---|---| +| `backendName` | Required; core and sidecar audit tables must use this backend | +| `retentionYears` | 6; compiled into the migration, not changed by a runtime flag | +| `signingKey` | Optional `Supplier`; `SigningKey(keyId, PrivateKey)` | +| `trustedPublicKeys` | `Map`; must contain the signing key's public key | +| `chainIntervalSeconds` | 30 | +| `settleHorizonSeconds` | 60 (zero allowed) | +| `batchSize` | 1,000 headers per tick / links per insert batch | +| `lateDetailLookbackHours` | 24; positive hours, older omissions checked by full verification | +| `checkpointEveryLinks` | 10,000; maximum links in each checkpoint | +| `checkpointEveryMinutes` | 60; requires at least one uncheckpointed link | +| `verifyCron` | `0 0 3 * * ?` (03:00 UTC, Quartz syntax) | +| `schedulerName` | `worm_audit`; uses the host scheduler if present, otherwise registers Quartz | +| `coreAuditTableNames` | Overrides keyed by `audit`, `auditDetail`, `auditTable`, `auditUser` | +| `appRole` | `app` | +| `tableMetaDataCustomizer` | Optional; append-only capability guard runs afterward | + +A missing signing key produces no checkpoints and adds the warning **no signing key configured** +to verification. It does not itself break the chain. Signing requires a matching public-key pin. +Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints. +Never store the private key in a database row or checked-in configuration. ## Adopting in another app +Add `com.kingsrook.qbits:qbit-worm-audit:0.1.0-SNAPSHOT`. Define the host's core audit tables +with `AuditsMetaDataProvider` first, then install the qbit: + ```java -new WormAuditQBitProducer() - .withQBitConfig(new WormAuditQBitConfig()) - .produce(qInstance); +WormAuditQBitConfig config = new WormAuditQBitConfig() + .withBackendName("main") + .withAppRole("app") + .withSigningKey(() -> new WormAuditQBitConfig.SigningKey("2026-10", privateKey)) + .withTrustedPublicKeys(Map.of("2026-10", publicKey)); +new WormAuditQBitProducer().withQBitConfig(config).produce(qInstance); +String migration = WormAuditDdl.postgres(config); ``` -Production currently registers the QBit identity and config only. Database changesets, the object-storage bucket and the alert wiring are documented with each phase. +1. Render and review the migration, then copy it into the host Liquibase changelog. + The [default rendering](docs/ddl/worm-audit-postgres.sql) uses `audit`, `audit_detail`, + `audit_table`, `audit_user`, runtime role `app`, and six years. The renderer resolves table + backend names during production; explicit overrides win. Standalone rendering uses those + defaults. Identifiers are validated and quoted; schema-qualified core table names are supported. +2. Apply the DDL as the schema owner. It creates the five sidecar tables in `public`, functions, + triggers, the NOLOGIN `worm_audit_purger` role, and runtime grants. Core column names must use + the usual snake_case mappings (`audit_id`, `audit_table_id`, etc.). The migration is idempotent. Provision the purger role beforehand or give the migration role permission to create it. +3. Run as a separate runtime role with SELECT/INSERT and sequence USAGE. Do not make it a member + of the purger role or a schema/table owner. Use the host's PostgreSQL RDBMS strategy (RC.5's + base strategy uses MySQL quoting; see the test host strategy in `RdbmsTestSupport`). +4. Start QQQ's standard scheduler with its system session and grant users only the required read + and verification permissions. Point the existing alerts implementation at core `QAlerts`. +5. Configure ESB publications on verification/export in the table customizer. Register all five + tables in the host Quick Search config with `SearchableTableConfig` / `SearchableFieldConfig`. + Index identifiers and status fields; set basepull timestamps to `chained_at`, `created_at`, + `sealed_at`, `finished_at`, and `requested_at` respectively (these tables have no `modifyDate`). -## Build +The RC.6 assistant deny API is registered reflectively when present; absence in RC.5 is a no-op. +RC.5's alert-type metadata validator disallows underscores, so the qbit sends the specified +`worm_audit.chain.break` signal without declaring incompatible alert-type metadata. `QAlerts` +still dispatches it to the sink (or its logging fallback). Every scheduled verification sends +`worm_audit.verify.daily`; FAIL raises with the first break and verification record reference; +PASS resolves dedup key `chain`. -```bash -./mvnw compile -./mvnw test -./mvnw verify +## Chain and verification format + +- JCS v1 uses UTF-16 key ordering, strict surrogate validation and UTC microsecond timestamps. + IDs are integer longs except canonical `recordId`, which is a string. No floating-point values. +- Link hashes concatenate the predecessor's **64 lowercase ASCII hex bytes**, one NUL byte, + and canonical UTF-8 bytes. Genesis is 64 ASCII zeroes. +- Every link names the detail IDs it covered, at most 250. A PostgreSQL insert guard and GIN + overlap index reject duplicate coverage under the same advisory lock. Overflow and later details use + `LATE_DETAIL` links with the same canonical header-and-detail-subset document. The canonical + JSON is never persisted. A v1 content mismatch can identify the link, audit and covered detail + IDs, but cannot identify one changed field among multiple details without original content. +- PostgreSQL chaining uses transaction-scoped advisory lock `86753090421` shared with purge. + Other backends use a JVM-wide lock and require one writer JVM. SQL backends select settled + unchained headers with an anti-join; late commits are picked up on a later tick. +- Merkle leaves are decoded 32-byte link hashes; parents hash the concatenated binary children; + an odd node is promoted unchanged. Ed25519 signs JCS of the seven specified range/trust fields. +- SQL verification uses a repeatable-read snapshot. Full runs check settled header/detail + completeness; selected or filtered runs check their links, predecessor boundaries and + intersecting checkpoints. Full verification currently materializes link metadata in memory. + +## Retention operations + +Only the ops login in `worm_audit_purger` may execute: + +```sql +SELECT public.worm_audit_purge(now() - interval '6 years'); ``` -Verification includes Checkstyle and JaCoCo gates of 70% instructions and 90% classes. On Windows, use `./mvnw.cmd`. +A recent/null cutoff or an incomplete expired prefix records REFUSED and returns normally. +Purge locks the audit tables against concurrent inserts, removes only a prefix closed over +all links and details of its audits, and records its last link id/hash. Detail and chain deletion +is allowed only under the function-owner identity, transaction-local purge flag, and compiled +retention cutoff. An app session setting that flag gets no deletion authority. + +Lookups cannot change id/name or be deleted/truncated. The runtime role may update only the +lookup label/dates through column grants; table-level UPDATE stays revoked. With normal grants, +PostgreSQL rejects prohibited runtime writes as SQLSTATE 42501 before executing a trigger. +The second layer (triggers) returns `P0001` / `worm_audit:` even if someone grants wider DML +rights. The qbit maps those errors to a value-free user-facing message. + +Verification starts from the latest successful purge anchor. Checkpoint signatures still verify +after purge, but Merkle recomputation of a purged/partly purged checkpoint requires its sealed +export and is reported as a warning. Sealed exports are phase (b); preserve external evidence +before exercising retention removal in production. Superusers/owners can disable triggers; +protect signing keys and keep public-key pins outside the database. + +## Build and acceptance + +```powershell +./mvnw.cmd -o verify +``` + +Checkstyle and JaCoCo gates (70% instructions, 90% classes) run without Docker. Memory and H2 +tests cover canonical vectors, signatures, rollback, exact coverage, late arrivals/details, +verification findings, PHI tokenization, process summaries, permissions, and the alerts SPI. +The PHI stand-in uses RC.5 `AuditActionCustomizerInterface.customizeInput` before insertion; +classification is test-local because RC.5 lacks `DataClassification`. + +PostgreSQL acceptance uses `postgres:16-alpine`, applies the DDL twice and tests grants, +triggers, purge, real QQQ RDBMS chaining, superuser tamper, and 8 writers / 3 chainers. +It skips automatically without Docker locally and fails if **any** `CI` environment value is set. +On a Docker-enabled coordinator (image already available if running without network): + +```powershell +$env:CI = 'true' +./mvnw.cmd -o '-Dtest=PostgresAcceptanceTest' test +./mvnw.cmd -o verify +``` + +POSIX equivalents: `CI=true ./mvnw -o -Dtest=PostgresAcceptanceTest test` and +`CI=true ./mvnw -o verify`. No commits, publishing or deployment are part of these commands. ## License [Apache License 2.0](LICENSE). Copyright Kingsrook, LLC. + +### Physical columns and bounded polling + +`WormAuditDdl.postgres(config)` renders table names only. Core physical columns +must be snake_case: `audit.id`, `audit.timestamp`, `audit_detail.id`, +`audit_detail.audit_id`, `audit_table.id/name/label/create_date/modify_date`, and +`audit_user.id/name/create_date/modify_date`. QInstance field backend-name +customizations do not change this migration; adapt the reviewed SQL before +applying it if your schema differs. UTC values in `timestamp without time zone` +and `timestamptz` are both supported. + +`lateDetailLookbackHours` defaults to **24** (positive hours). Each PostgreSQL +poll checks uncovered details whose chained header timestamp lies in +`[cutoff - lookback, cutoff)`. The daily full verifier reports omissions outside +that window. Initial header details are always chained, including overflow +chunks. Checkpoints contain at most `checkpointEveryLinks` links; multiple full +ranges can be signed per tick. Verification uses 1,000-link keyset pages. + +Full verification retains at most 1,000 exact break diagnostics and 100 warnings, +while persisting the total break count. SQL completeness counts all offenders +and reports the first 100 header ids and first 100 detail ids. This keeps a badly +corrupted chain from exhausting verifier memory. diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000..d43ab55 --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,383 @@ +# QBit WORM Audit: design + +Status: approved. **Owner decisions (2026-10-12): all recommendations accepted.** Every open +question in section 12 is decided as recommended. + +## 1. Owner request + +> "we also need to ensure that our auditing qbit supports WORM audits as well. and the phi/secure +> data qbit needs to ensure that we don't leak phi into audits" + +Context: HIPAA §164.312(b) audit controls, §164.312(c)(1) integrity, §164.316(b)(2)(i) six-year +retention of required documentation (audit records are kept to the same horizon), and a +HITRUST-style review that asks three things: (1) can anyone change or delete an audit record, +(2) would we know if they did, and (3) is there a copy nobody can alter? + +## 2. Goals and non-goals + +Goals: +1. **Append-only at the database level.** No application path, and no ordinary database login, + can UPDATE, DELETE or TRUNCATE audit rows. The only removal is an audited retention purge, + after the retention date, through a separate database role. +2. **Tamper evidence.** Every audit row is linked into a hash chain. Signed checkpoints anchor the + chain. A verification action and a daily job report the exact first broken row. +3. **Sealed copies.** New audit rows and checkpoints are exported on a schedule to an S3 bucket + with Object Lock in COMPLIANCE mode (default six years), and can be verified against the live + chain. +4. **No PHI in the WORM layer.** The chain and the exports seal exactly what core wrote. With + qbit-secured-data that means HMACs for classified values, never plaintext. + +Non-goals: changing what core audits (that is core's and qbit-secured-data's job); protecting +against a Postgres superuser *preventing* a change (a superuser can disable triggers; the +chain, the signatures and the Object Lock copies make that change *detectable*, see 4.5). + +## 3. Scope: which tables + +| Table (QQQ name / kof22 physical) | Treatment | +|---|---| +| `audit` / `audit` | Append-only (UPDATE, DELETE, TRUNCATE rejected). Chained. | +| `auditDetail` / `audit_detail` | Append-only. Chained with its header. | +| `auditTable` / `audit_table` | Lookup. DELETE and TRUNCATE rejected; UPDATE of `id` or `name` rejected (label and dates may change). Names are resolved into the chained content, so a later rename is detected. | +| `auditUser` / `audit_user` | As `audit_table`. | +| `worm_audit_chain`, `worm_audit_checkpoint`, `worm_audit_export`, `worm_audit_verification`, `worm_audit_purge` | The qbit's own tables: all append-only. | + +Physical names come from the host's backend (kof22 maps camelCase to snake_case). The DDL is a +template rendered for the host's names (section 5.4). + +## 4. Append-only enforcement (Postgres first) + +### 4.1 Triggers +One function, `worm_audit_reject()`, attached as: +- `BEFORE UPDATE OR DELETE ... FOR EACH ROW` on every append-only table; +- `BEFORE TRUNCATE ... FOR EACH STATEMENT` on every append-only table; +- on the two lookups, a row trigger that rejects DELETE and any UPDATE that changes `id` or + `name`. + +The error is explicit and greppable: + +``` +ERROR: worm_audit: audit rows are write-once; UPDATE of "audit" is not allowed (id=4211). Expired rows are removed only by worm_audit_purge(). +SQLSTATE: P0001 (HINT: see qbit-worm-audit docs/DESIGN.md section 4) +``` + +In the application the qbit also removes the update and delete capabilities from these tables' +QQQ metadata, so the admin UI and both APIs never offer edit or delete, and a direct call gets +QQQ's standard "table does not support update/delete" error before reaching the database. The +qbit maps a `P0001` with the `worm_audit:` prefix to a `QUserFacingException`: "Audit records are +write-once and cannot be changed or deleted." + +### 4.2 The retention purge (the only removal path) +`worm_audit_purge(p_cutoff timestamptz)` is a `SECURITY DEFINER` function owned by the schema +owner. `EXECUTE` is revoked from `PUBLIC` and granted only to the role `worm_audit_purger`, which +the app role is not a member of. + +1. It refuses (and records a `REFUSED` row in `worm_audit_purge`, then returns; no exception, so + the record survives) when `p_cutoff > now() - retention`. The retention interval is a constant + compiled into the function by the changeset (default `6 years`), so shortening it is a schema + migration, reviewed and itself audited. +2. It purges only a **chain prefix**: the audits whose chain links all lie at or before the last + link whose audit timestamp is older than the cutoff, together with their details and links. +3. It sets a transaction-local flag (`set_config('worm_audit.purging', 'on', true)`). The reject + trigger allows DELETE only when that flag is on **and** `current_user` is the function owner + **and** the row is older than the retention interval. Setting the flag from another session + does nothing, because `current_user` is not the owner. +4. It writes a `PURGED` row: who (session_user), cutoff, retention, counts, and the **anchor** + (the last purged link's id and hash). Verification continues from the anchor. + +The purge runs from an ops runbook (`psql` as `worm_audit_purger`). Version 0.1 has no UI action; +the app home lists purge rows, including refusals. + +### 4.3 GRANT model + +| Role | Owns | Grants on append-only tables | +|---|---|---| +| `_owner` (Liquibase) | tables, functions, triggers | all (owner) | +| `` (runtime) | nothing | `SELECT, INSERT`; `USAGE` on their sequences. No UPDATE, DELETE or TRUNCATE. | +| `worm_audit_purger` (ops) | nothing | `EXECUTE worm_audit_purge`; nothing else | + +When a host uses one role for both migrations and runtime (kof22 today), the triggers still stop +every UPDATE, DELETE and TRUNCATE from the application, but the role can `ALTER TABLE ... DISABLE +TRIGGER`. The docs recommend splitting the roles, and the app home shows a warning when the +runtime role owns the audit tables (checked through `pg_tables.tableowner = current_user`). + +### 4.4 Changesets +The qbit ships `src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql`, +a Liquibase formatted-SQL template with placeholders for the four core audit table names, the +app role and the retention interval, plus a renderer (`WormAuditDdl.postgres(config)`) and the +rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Hosts copy the rendered +file into their changelog (kof22: `changesets/features/YYYY-MM-DD-worm-audit.sql`). It creates the +qbit tables, the functions, the triggers, the purger role (`CREATE ROLE ... NOLOGIN` when it is +missing) and the grants. It is idempotent (`CREATE OR REPLACE`, `DROP TRIGGER IF EXISTS`). + +### 4.5 Threat model summary + +| Actor | Can modify audit rows? | Detected by | +|---|---|---| +| App user, admin UI, QQQ APIs, assistant | No (capabilities removed; triggers) | n/a | +| SQL injection or a bug as the runtime role | No (triggers; no grant) | n/a | +| Owner role (migrations) | Only by disabling triggers | chain verification (daily), alert | +| Postgres superuser rewriting rows and recomputing the chain | Yes | checkpoint signatures (no signing key in the DB); sealed exports in Object Lock | +| Anyone deleting the sealed copies | No (COMPLIANCE mode: not even the bucket root, until expiry) | n/a | + +### 4.6 Other databases (documented, not shipped in 0.1) +- **MySQL/MariaDB:** `BEFORE UPDATE`/`BEFORE DELETE` triggers with `SIGNAL SQLSTATE '45000'`; + no TRUNCATE trigger, so revoke `DROP` from the app user; purge through a `SQL SECURITY DEFINER` + procedure. +- **SQL Server:** `INSTEAD OF UPDATE, DELETE` triggers plus `DENY UPDATE, DELETE` to the app user. +- **H2** (tests only): no enforcement; the chain and the exports work on any QQQ RDBMS or memory + backend, so tamper *evidence* is database-independent. + +## 5. Tamper evidence + +### 5.1 Canonical content (agreed with qbit-secured-data) +One canonical document per audit header: the header plus all of its detail rows, as RFC 8785 JCS +JSON (sorted keys, no insignificant whitespace, UTF-8): + +```json +{"audit":{"clientChannel":null,"id":4211,"message":"Record was Edited","recordId":"17","table":"cms_page","timestamp":"2026-10-13T21:04:05.123456Z","user":"james"}, + "details":[{"fieldName":"title","id":9001,"message":"Changed Title from ...","newValue":"...","oldValue":"..."}], + "v":1} +``` + +- Values are exactly as stored: no formatting, no re-masking, no lookups other than resolving + `auditTableId`/`auditUserId` to their names. `details` are sorted by id. +- With qbit-secured-data's core hook (QRun-IO/qqq#1161, `docs/SECURED-DATA.md` §14.2), PII and + PHI values are tokenised inside `DMLAuditAction` before `AuditAction` inserts, in the same + transaction as the data change, so the token is the only form ever written. Stored forms (ASCII, + at most 200 characters) in `oldValue`/`newValue` are + `hmac:v1::`, `shred:v1::`, `changed:v1`, + `wiped:v1:` and `shredded:v1:` (the last is its own detail row, written when + keys are destroyed, and is chained like any other row). Detail messages for those fields + are replaced with value-free ones ("Changed