diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000..d43ab55 --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,383 @@ +# QBit WORM Audit: design + +Status: approved. **Owner decisions (2026-10-12): all recommendations accepted.** Every open +question in section 12 is decided as recommended. + +## 1. Owner request + +> "we also need to ensure that our auditing qbit supports WORM audits as well. and the phi/secure +> data qbit needs to ensure that we don't leak phi into audits" + +Context: HIPAA §164.312(b) audit controls, §164.312(c)(1) integrity, §164.316(b)(2)(i) six-year +retention of required documentation (audit records are kept to the same horizon), and a +HITRUST-style review that asks three things: (1) can anyone change or delete an audit record, +(2) would we know if they did, and (3) is there a copy nobody can alter? + +## 2. Goals and non-goals + +Goals: +1. **Append-only at the database level.** No application path, and no ordinary database login, + can UPDATE, DELETE or TRUNCATE audit rows. The only removal is an audited retention purge, + after the retention date, through a separate database role. +2. **Tamper evidence.** Every audit row is linked into a hash chain. Signed checkpoints anchor the + chain. A verification action and a daily job report the exact first broken row. +3. **Sealed copies.** New audit rows and checkpoints are exported on a schedule to an S3 bucket + with Object Lock in COMPLIANCE mode (default six years), and can be verified against the live + chain. +4. **No PHI in the WORM layer.** The chain and the exports seal exactly what core wrote. With + qbit-secured-data that means HMACs for classified values, never plaintext. + +Non-goals: changing what core audits (that is core's and qbit-secured-data's job); protecting +against a Postgres superuser *preventing* a change (a superuser can disable triggers; the +chain, the signatures and the Object Lock copies make that change *detectable*, see 4.5). + +## 3. Scope: which tables + +| Table (QQQ name / kof22 physical) | Treatment | +|---|---| +| `audit` / `audit` | Append-only (UPDATE, DELETE, TRUNCATE rejected). Chained. | +| `auditDetail` / `audit_detail` | Append-only. Chained with its header. | +| `auditTable` / `audit_table` | Lookup. DELETE and TRUNCATE rejected; UPDATE of `id` or `name` rejected (label and dates may change). Names are resolved into the chained content, so a later rename is detected. | +| `auditUser` / `audit_user` | As `audit_table`. | +| `worm_audit_chain`, `worm_audit_checkpoint`, `worm_audit_export`, `worm_audit_verification`, `worm_audit_purge` | The qbit's own tables: all append-only. | + +Physical names come from the host's backend (kof22 maps camelCase to snake_case). The DDL is a +template rendered for the host's names (section 5.4). + +## 4. Append-only enforcement (Postgres first) + +### 4.1 Triggers +One function, `worm_audit_reject()`, attached as: +- `BEFORE UPDATE OR DELETE ... FOR EACH ROW` on every append-only table; +- `BEFORE TRUNCATE ... FOR EACH STATEMENT` on every append-only table; +- on the two lookups, a row trigger that rejects DELETE and any UPDATE that changes `id` or + `name`. + +The error is explicit and greppable: + +``` +ERROR: worm_audit: audit rows are write-once; UPDATE of "audit" is not allowed (id=4211). Expired rows are removed only by worm_audit_purge(). +SQLSTATE: P0001 (HINT: see qbit-worm-audit docs/DESIGN.md section 4) +``` + +In the application the qbit also removes the update and delete capabilities from these tables' +QQQ metadata, so the admin UI and both APIs never offer edit or delete, and a direct call gets +QQQ's standard "table does not support update/delete" error before reaching the database. The +qbit maps a `P0001` with the `worm_audit:` prefix to a `QUserFacingException`: "Audit records are +write-once and cannot be changed or deleted." + +### 4.2 The retention purge (the only removal path) +`worm_audit_purge(p_cutoff timestamptz)` is a `SECURITY DEFINER` function owned by the schema +owner. `EXECUTE` is revoked from `PUBLIC` and granted only to the role `worm_audit_purger`, which +the app role is not a member of. + +1. It refuses (and records a `REFUSED` row in `worm_audit_purge`, then returns; no exception, so + the record survives) when `p_cutoff > now() - retention`. The retention interval is a constant + compiled into the function by the changeset (default `6 years`), so shortening it is a schema + migration, reviewed and itself audited. +2. It purges only a **chain prefix**: the audits whose chain links all lie at or before the last + link whose audit timestamp is older than the cutoff, together with their details and links. +3. It sets a transaction-local flag (`set_config('worm_audit.purging', 'on', true)`). The reject + trigger allows DELETE only when that flag is on **and** `current_user` is the function owner + **and** the row is older than the retention interval. Setting the flag from another session + does nothing, because `current_user` is not the owner. +4. It writes a `PURGED` row: who (session_user), cutoff, retention, counts, and the **anchor** + (the last purged link's id and hash). Verification continues from the anchor. + +The purge runs from an ops runbook (`psql` as `worm_audit_purger`). Version 0.1 has no UI action; +the app home lists purge rows, including refusals. + +### 4.3 GRANT model + +| Role | Owns | Grants on append-only tables | +|---|---|---| +| `_owner` (Liquibase) | tables, functions, triggers | all (owner) | +| `` (runtime) | nothing | `SELECT, INSERT`; `USAGE` on their sequences. No UPDATE, DELETE or TRUNCATE. | +| `worm_audit_purger` (ops) | nothing | `EXECUTE worm_audit_purge`; nothing else | + +When a host uses one role for both migrations and runtime (kof22 today), the triggers still stop +every UPDATE, DELETE and TRUNCATE from the application, but the role can `ALTER TABLE ... DISABLE +TRIGGER`. The docs recommend splitting the roles, and the app home shows a warning when the +runtime role owns the audit tables (checked through `pg_tables.tableowner = current_user`). + +### 4.4 Changesets +The qbit ships `src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql`, +a Liquibase formatted-SQL template with placeholders for the four core audit table names, the +app role and the retention interval, plus a renderer (`WormAuditDdl.postgres(config)`) and the +rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Hosts copy the rendered +file into their changelog (kof22: `changesets/features/YYYY-MM-DD-worm-audit.sql`). It creates the +qbit tables, the functions, the triggers, the purger role (`CREATE ROLE ... NOLOGIN` when it is +missing) and the grants. It is idempotent (`CREATE OR REPLACE`, `DROP TRIGGER IF EXISTS`). + +### 4.5 Threat model summary + +| Actor | Can modify audit rows? | Detected by | +|---|---|---| +| App user, admin UI, QQQ APIs, assistant | No (capabilities removed; triggers) | n/a | +| SQL injection or a bug as the runtime role | No (triggers; no grant) | n/a | +| Owner role (migrations) | Only by disabling triggers | chain verification (daily), alert | +| Postgres superuser rewriting rows and recomputing the chain | Yes | checkpoint signatures (no signing key in the DB); sealed exports in Object Lock | +| Anyone deleting the sealed copies | No (COMPLIANCE mode: not even the bucket root, until expiry) | n/a | + +### 4.6 Other databases (documented, not shipped in 0.1) +- **MySQL/MariaDB:** `BEFORE UPDATE`/`BEFORE DELETE` triggers with `SIGNAL SQLSTATE '45000'`; + no TRUNCATE trigger, so revoke `DROP` from the app user; purge through a `SQL SECURITY DEFINER` + procedure. +- **SQL Server:** `INSTEAD OF UPDATE, DELETE` triggers plus `DENY UPDATE, DELETE` to the app user. +- **H2** (tests only): no enforcement; the chain and the exports work on any QQQ RDBMS or memory + backend, so tamper *evidence* is database-independent. + +## 5. Tamper evidence + +### 5.1 Canonical content (agreed with qbit-secured-data) +One canonical document per audit header: the header plus all of its detail rows, as RFC 8785 JCS +JSON (sorted keys, no insignificant whitespace, UTF-8): + +```json +{"audit":{"clientChannel":null,"id":4211,"message":"Record was Edited","recordId":"17","table":"cms_page","timestamp":"2026-10-13T21:04:05.123456Z","user":"james"}, + "details":[{"fieldName":"title","id":9001,"message":"Changed Title from ...","newValue":"...","oldValue":"..."}], + "v":1} +``` + +- Values are exactly as stored: no formatting, no re-masking, no lookups other than resolving + `auditTableId`/`auditUserId` to their names. `details` are sorted by id. +- With qbit-secured-data's core hook (QRun-IO/qqq#1161, `docs/SECURED-DATA.md` §14.2), PII and + PHI values are tokenised inside `DMLAuditAction` before `AuditAction` inserts, in the same + transaction as the data change, so the token is the only form ever written. Stored forms (ASCII, + at most 200 characters) in `oldValue`/`newValue` are + `hmac:v1::`, `shred:v1::`, `changed:v1`, + `wiped:v1:` and `shredded:v1:` (the last is its own detail row, written when + keys are destroyed, and is chained like any other row). Detail messages for those fields + are replaced with value-free ones ("Changed