feat: retention and wipe with crypto-shredding; QQQ RC.6 (phase f) #11

2026-10-14T12:36:37Z

KofTwentyTwo commented

2026-10-14 12:36:37 +00:00

(Migrated from github.com)

Summary

Phase (f) of secured data (spec §13, §14.3, §15), plus the move to QQQ RC.6.

RC.6. The pin moves to 4.1.0-RC.6, and every stand-in RC.6 provides is removed: DataClassification, client channels, table features, and the UI API extension point (the uiapi profile is folded into the main build). Two stand-ins remain until RC.7: fieldaccess for #1172 and the audit tokens for #1182, each marked STAND-IN(core=#…, removeBy=RC.7).
Tables: secured_data_retention_rule (in the nav), secured_data_wipe_marker, secured_data_legal_hold, secured_data_wipe_run and secured_data_audit_key.
- secured_data_audit_key holds DEKs wrapped with AES-GCM under QQQ_SECURED_DATA_KEK, with table, field and pk bound as AAD.
- Its wrapped_key field is HIDDEN to everyone behind a lock no one is ever granted. The table is system-written only and is never sent to ESB or WORM.
Wipe process secured_data_wipe: a standard streamed ETL, run daily as the system and available as the "Run wipe now" table action (secured_data.retention).
- Actions: NULL_WITH_MARKER (the default), OVERWRITE_TEXT, KEYED_HASH (refused on low-entropy fields), DELETE_RECORD, and ARCHIVE_THEN_WIPE (through an ArchiveSinkInterface).
- Each batch runs in one transaction, which nulls the fields, writes wiped:v1, deletes the record-field DEKs, writes shredded:v1, writes markers and wipe-run rows, and logs WIPE rows to the access log.
- Legal holds pause wiping. Review/Result lines show counts only, never values.
Audit protector and key source: a field is shredded when it is under a SHREDDABLE rule, or when it is PII/PHI and low-entropy. CHANGED_ONLY rules write changed:v1 with no derivative.
Backups panel and home warning: shown when known backup retention exceeds a rule's age. The README documents the residual risk of keys kept in physical backups.
Fix: the key source used to zero each DEK after returning it, so callers received an all-zero key. It now returns a copy, covered by a regression test.

Tests

RC.6 (CI): 87 tests, 0 failures, 6 skipped. The skips are the core-hook integration, canary and wipe-proof tests.
Against the core branches (#1172 + #1182): 87 tests, 0 failures, 0 skipped.
Canary (§15):
- Plant: CANARY-PHI-* in PHI fields.
- Exercise: insert, update, a failing validation, a MASKED query and CSV export, an access-log flush and a wipe.
- Scan: audit and auditDetail, the access log, wipe runs and markers, the key and policy tables, the export, error messages and captured logs.
- Result: no marker and no substring of 5 or more characters anywhere. A positive control proves the scan works.
Wipe-proof (§15):
- Setup: 191 audited systolic readings (60–250) under a SHREDDABLE rule.
- Before the wipe: each token verifies its true value only.
- After the wipe: 0 matches for every value 60–250 against the instance HMAC key, the KEK, the remaining DEK, and an all-zero key. shred: tokens no longer verify, and the key rows are gone.

Known core issue

Core's integer conversion error echoes the raw value. A string inserted into an INTEGER PHI field produces Value [...] could not be converted. This is being fixed in core #1182.

🤖 Generated with Claude Code

## Summary Phase (f) of secured data (spec §13, §14.3, §15), plus the move to QQQ RC.6. - **RC.6.** The pin moves to `4.1.0-RC.6`, and every stand-in RC.6 provides is removed: DataClassification, client channels, table features, and the UI API extension point (the `uiapi` profile is folded into the main build). Two stand-ins remain until RC.7: fieldaccess for #1172 and the audit tokens for #1182, each marked `STAND-IN(core=#…, removeBy=RC.7)`. - **Tables:** `secured_data_retention_rule` (in the nav), `secured_data_wipe_marker`, `secured_data_legal_hold`, `secured_data_wipe_run` and `secured_data_audit_key`. - `secured_data_audit_key` holds DEKs wrapped with AES-GCM under `QQQ_SECURED_DATA_KEK`, with table, field and pk bound as AAD. - Its `wrapped_key` field is HIDDEN to everyone behind a lock no one is ever granted. The table is system-written only and is never sent to ESB or WORM. - **Wipe process `secured_data_wipe`:** a standard streamed ETL, run daily as the system and available as the "Run wipe now" table action (`secured_data.retention`). - Actions: NULL_WITH_MARKER (the default), OVERWRITE_TEXT, KEYED_HASH (refused on low-entropy fields), DELETE_RECORD, and ARCHIVE_THEN_WIPE (through an `ArchiveSinkInterface`). - Each batch runs in one transaction, which nulls the fields, writes `wiped:v1`, deletes the record-field DEKs, writes `shredded:v1`, writes markers and wipe-run rows, and logs WIPE rows to the access log. - Legal holds pause wiping. Review/Result lines show counts only, never values. - **Audit protector and key source:** a field is shredded when it is under a SHREDDABLE rule, or when it is PII/PHI and low-entropy. CHANGED_ONLY rules write `changed:v1` with no derivative. - **Backups panel and home warning:** shown when known backup retention exceeds a rule's age. The README documents the residual risk of keys kept in physical backups. - **Fix:** the key source used to zero each DEK after returning it, so callers received an all-zero key. It now returns a copy, covered by a regression test. ## Tests - **RC.6 (CI):** 87 tests, 0 failures, 6 skipped. The skips are the core-hook integration, canary and wipe-proof tests. - **Against the core branches (#1172 + #1182):** 87 tests, 0 failures, 0 skipped. - **Canary (§15):** - **Plant:** `CANARY-PHI-*` in PHI fields. - **Exercise:** insert, update, a failing validation, a MASKED query and CSV export, an access-log flush and a wipe. - **Scan:** audit and auditDetail, the access log, wipe runs and markers, the key and policy tables, the export, error messages and captured logs. - **Result:** no marker and no substring of 5 or more characters anywhere. A positive control proves the scan works. - **Wipe-proof (§15):** - **Setup:** 191 audited systolic readings (60–250) under a SHREDDABLE rule. - **Before the wipe:** each token verifies its true value only. - **After the wipe:** 0 matches for every value 60–250 against the instance HMAC key, the KEK, the remaining DEK, and an all-zero key. `shred:` tokens no longer verify, and the key rows are gone. ## Known core issue Core's integer conversion error echoes the raw value. A string inserted into an INTEGER PHI field produces `Value [...] could not be converted`. This is being fixed in core #1182. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-secured-data#11