diff --git a/CHANGELOG.md b/CHANGELOG.md index eb3ede2..7e2a94b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ### Added +- RC.2 settings: editable, FIELD-audited singleton with insert-only defaults, validation, separate edit permission and a 30-second effective-settings cache. Schedules load at startup. +- GOVERNANCE exports with 30-day default retention; per-export retention verification, administrator-bypass diagnostics, home warnings and documented COMPLIANCE transition. +- New `worm-audit:10` settings migration; historical changesets remain unchanged. + - Phase (c): configurable WORM Audit app home, full-width summary widget and optional host health card; permission-aware navigation and content-free, responsive light/dark summaries. - Chain backlog/settling counts, verification and export record links, sealed-range gaps/lag, retention eligibility and recent purge/refusal history. - PostgreSQL guard/ownership/purge-function diagnostics, signing/trust warnings, tri-state Object Lock capability and stale daily-verification warnings. diff --git a/README.md b/README.md index 3ecdc9b..082e7de 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ Phases (a)–(c) provide PostgreSQL append-only enforcement, an asynchronous audit hash chain, Ed25519 checkpoints, live and sealed-export verification actions, and daily jobs with core QAlerts signals. It hashes exactly the audit values core stored; it does not read application records or -secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and COMPLIANCE -Object Lock. The WORM Audit app home summarizes chain health, exports, retention and setup. +secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and configurable +Object Lock (GOVERNANCE for 30 days by default). The WORM Audit app home summarizes chain health, exports, retention and setup. Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement; memory/H2 support tamper-evidence tests without database enforcement. @@ -12,8 +12,9 @@ memory/H2 support tamper-evidence tests without database enforcement. ## Tables and permissions All sidecar tables use the configured host backend, snake_case fields, BIGINT identifiers, -record sections, and full table-name permissions. They do not audit themselves. The app lists -Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that order. +record sections, and full table-name permissions. Immutable sidecars do not audit themselves. +The editable Settings singleton uses an INTEGER ID and FIELD-level core audit. The app lists +Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges, and Settings in that order. | Table | Purpose | Permission | |---|---|---| @@ -22,18 +23,19 @@ Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that orde | `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` | | `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` | | `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` | +| `worm_audit_settings` | Editable singleton; every edit is audited and chained | `worm_audit_settings.read`; editing needs `worm_audit_settings.edit` **and** `worm_audit.settings` | `worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action. It also authorizes **Verify sealed export** on selected export records. `worm_audit.export` authorizes **Export now** on the export table. Processes are table/record actions or schedules; none is a navigation entry. -Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities +Insert/update/delete capabilities are absent on immutable sidecar tables; Settings allows updates only; update/delete capabilities are removed from all four core audit tables. Internal appends use an explicit permission bypass. Checkpoint, export and verification record screens have a bounded chain-link child list; range endpoints and export/checkpoint references link to their records. The table metadata customizer can configure host ESB publications. Quick Search registration -is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables. +is host-owned through `QuickSearchQBitConfig.searchableTables`; add the desired tables. Verification and export inserts use QQQ's record-change listener SPI, so an installed ESB publisher receives those events when the host configures its destinations on these two tables. The qbit defines no destination and configures no chain-link publication. @@ -70,6 +72,58 @@ to verification. It does not itself break the chain. Signing requires a matching Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints. Never store the private key in a database row or checked-in configuration. +## Settings + +Open **WORM Audit → Settings**, record **1** (`worm_audit_settings`, labelled WORM Audit Settings). +This is a standard editable QQQ table. New records and deletion are disabled; a pre-insert +check and database constraint require ID 1. Every edit uses core FIELD-level auditing, so +the change enters the audit chain. `modify_date` and `modified_by` are stamped by the server. +Keep notes free of sensitive data; notes are audited too. + +| Field | Default | Validation / effect | +|---|---|---| +| `audit_retention_years` | 6 | 1–100; home next eligible purge date | +| `export_lock_mode` | GOVERNANCE | GOVERNANCE or COMPLIANCE; new exports | +| `export_retention_days` | 30 | 1–36500; new exports | +| `checkpoint_every_links` | 10000 | Positive integer; new checkpoint ranges | +| `checkpoint_every_minutes` | 60 | Positive integer; checkpoint cadence | +| `chain_interval_seconds` | 30 | Positive integer; **takes effect after restart** | +| `settle_horizon_seconds` | 60 | Nonnegative integer; chaining and verification | +| `late_detail_lookback_hours` | 24 | Positive integer; late-detail polling | +| `export_cron` | `0 0 2 * * ?` | Quartz cron, UTC; **takes effect after restart** | +| `verify_cron` | `0 0 3 * * ?` | Quartz cron, UTC; **takes effect after restart** | +| `notes` | Empty | Administrative notes | +| `modify_date`, `modified_by` | Server supplied | Last edit time and user | + +Config values supply defaults for missing rows or null fields; a missing settings table also +falls back to config. Other database failures fail closed. `exportLockMode` and +`exportRetentionDays` are the corresponding config defaults. Runtime reads cache the row +for at most 30 seconds per application instance; local QQQ edits invalidate that cache. +Each run reads effective settings. Startup schedules read the existing row, or config when +absent. Historical checkpoints remain valid after changing their creation cadence. + +Changeset `worm-audit:10` creates and seeds the table with `INSERT ... SELECT ... WHERE NOT +EXISTS`; rerunning it never overwrites an administrator change. The renderer seeds from the +host config. Memory hosts and the demo can call `WormAuditSettings.seed(config)` for the same +insert-only behavior during single-threaded startup. + +`audit_retention_years` does **not** rewrite the database purge function. The home warns if +it differs from the compiled interval; changing that interval requires a reviewed migration. +The home also warns when export retention is shorter than audit retention and while +GOVERNANCE permits administrator bypass deletion. The runtime role receives SELECT, INSERT +and UPDATE on Settings, with no DELETE or TRUNCATE grant and no append-only triggers. + +## Switching sealed exports to COMPLIANCE + +Edit `export_lock_mode` to **COMPLIANCE** in Settings. New exports use COMPLIANCE once the +settings cache refreshes (within 30 seconds); existing GOVERNANCE objects keep their mode +unless an administrator explicitly extends their retention and upgrades their mode. The +verifier continues to compare each object with its export row's recorded mode and date, so +an out-of-band mode change is reported as a mismatch. The bucket default retention should +match the chosen mode and retention days, although the exporter always sends explicit values. +The settings change is audited and chained. **COMPLIANCE cannot be undone**: protected +versions cannot be deleted or have retention shortened until expiry, including by administrators. + ## App home Open `/app/worm_audit` (or `/app/admin/worm_audit` with `parentAppName="admin"`). @@ -82,7 +136,7 @@ it never renders audit messages or field values. Record links retain the target - Chain status: live link count, latest link time, settled unchained backlog and settling count at the configured horizon, latest verification and first open chain break. A subsequent full scheduled CHAIN PASS clears a reported break; a selected USER PASS does not. -- Export status: latest sealed range, COMPLIANCE retention, link/time lag, coverage holes +- Export status: current lock mode and retention days, latest sealed range and its recorded retention, link/time lag, coverage holes between the purge anchor and sealed head, and bucket Object Lock capability. The unsealed trailing suffix is lag, not a gap. Provider failures show UNKNOWN with a value-free reason. - Retention: configured years, purge anchor, oldest live audit and its next eligible purge date. @@ -207,8 +261,11 @@ Install qbit-object-storage and configure `exportBucketAlias` (default `worm_aud `BucketRole.BACKUP` bucket. **Object Lock must be enabled when the bucket is created**. The exporter requires the bucket's `OBJECT_LOCK` capability to be `SUPPORTED`; run the object-storage read-only capability probe during setup. Unknown, partial and unsupported -capabilities all refuse sealing. Every object receives explicit **COMPLIANCE** retention -until the sealing time plus `retentionYears` (six by default), checked by HEAD after PUT. +capabilities all refuse sealing. Every object receives the effective `export_lock_mode` and retention until sealing time plus +`export_retention_days` (GOVERNANCE and 30 days by default), checked by HEAD after PUT. The +export row records both values. Verification checks that historical mode and retain-until, +regardless of subsequent settings changes. Missing GOVERNANCE objects are reported as breaks +with a possible administrator bypass-removal explanation. The runtime IAM policy needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:GetObjectRetention`, `s3:PutObjectRetention`, and @@ -284,8 +341,8 @@ $env:CI = 'true' ./mvnw.cmd -o verify ``` -MinIO export acceptance uses the same `RequiresDockerCondition`. It checks COMPLIANCE -retention, denied version deletion, overwrite version preservation, export verification, +MinIO export acceptance uses the same `RequiresDockerCondition`. It checks GOVERNANCE +with 30-day retention and a COMPLIANCE export, denied version deletion (without bypass for GOVERNANCE), overwrite version preservation, export verification, exact live-tamper findings through the record action, and refusal of a bucket without lock: ```powershell diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 756ba49..0cde182 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -228,6 +228,10 @@ installed the alerts go to the log. ## 6. Sealed exports ### 6.1 Export job + +**Default superseded by Owner decisions (2026-10-14) below:** lock mode and export retention +now come from Settings, defaulting to GOVERNANCE and 30 days. The original decision text is +retained for history. Daily at 02:00 (configurable), and by the table action "Export now" on `worm_audit_export`: 1. Range: links after the last export's `to_link_id`, up to the latest checkpoint. Exports are contiguous by construction. The app home flags a gap (a range not covered) and lag. @@ -386,3 +390,33 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...` These are owner-run (Praetor and cluster-gitops are applied by the owner). The qbit only documents them. + +## Owner decisions (2026-10-14) + +The 2026-10-12 decisions above remain the historical design. These decisions supersede the +COMPLIANCE/six-year export default in §6.1 and the corresponding default assumptions in +§§2, 4.5, 9, 11, 12 and the §14 infrastructure proposal. + +- Settings live in the application as the standard QQQ `worm_audit_settings` table, record + ID 1, labelled WORM Audit Settings and navigated as Settings after Purges. It allows edits, + has no append-only triggers, and uses FIELD-level core audit so every change is chained. + Reading needs `worm_audit_settings.read`; editing additionally needs + `worm_audit_settings.edit` and `worm_audit.settings`. New IDs and deletion are refused. +- Defaults are seeded insert-only in new changeset `worm-audit:10`; existing changesets are + unchanged. An administrator's edits survive subsequent seeding. Config supplies defaults + when a row/field/table is absent. Runtime reads cache at most 30 seconds, scoped to the + QInstance. Chain interval and UTC Quartz export/verify schedules take effect after restart. +- kof22 starts with **GOVERNANCE, 30 days** for sealed exports. An administrator with + `s3:BypassGovernanceRetention` can remove those copies. This is deliberately reversible; + it does not provide COMPLIANCE's irreversible retention. Runtime credentials still receive + no delete or bypass permission. Bucket defaults should match the selected settings. +- Changing Settings to COMPLIANCE affects new exports. Existing GOVERNANCE versions retain + their original mode unless an administrator extends/upgrades them. Verification checks + the catalog's recorded mode and date, so external mode changes are reported as mismatches. + COMPLIANCE cannot be undone. The settings change itself is audited and chained. +- Audit retention defaults to six years independently of export retention. The home warns + about a shorter export horizon and a mismatch with the purge function's compiled interval. + Updating the setting does not modify the purge function; that still requires a migration. + +The README Settings table specifies all fields, defaults, bounds and timing. Signing keys, +public-key pins, bucket alias, prefix and backend remain host configuration, never settings. diff --git a/docs/ddl/worm-audit-postgres.sql b/docs/ddl/worm-audit-postgres.sql index fca11e9..1561309 100644 --- a/docs/ddl/worm-audit-postgres.sql +++ b/docs/ddl/worm-audit-postgres.sql @@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain; CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage(); + +-- changeset worm-audit:10 +-- Editable singleton: deliberately excluded from append-only triggers. +CREATE TABLE IF NOT EXISTS public.worm_audit_settings ( + id integer PRIMARY KEY CHECK (id = 1), + audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100), + export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')), + export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500), + checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0), + checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0), + chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0), + settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0), + late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0), + export_cron text NOT NULL DEFAULT '0 0 2 * * ?', + verify_cron text NOT NULL DEFAULT '0 0 3 * * ?', + notes text, + modify_date timestamptz NOT NULL DEFAULT now(), + modified_by text NOT NULL DEFAULT current_user +); +REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM "app"; +GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO "app"; +INSERT INTO public.worm_audit_settings + (id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links, + checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours, + export_cron, verify_cron) +SELECT 1, 6, 'GOVERNANCE', 30, 10000, 60, 30, 60, 24, '0 0 2 * * ?', '0 0 3 * * ?' +WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1); diff --git a/docs/screenshots/README.md b/docs/screenshots/README.md index 8398c72..9197150 100644 --- a/docs/screenshots/README.md +++ b/docs/screenshots/README.md @@ -1,5 +1,14 @@ # WORM Audit screenshots +## RC.2 settings captures (2026-10-14) +Home (GOVERNANCE, 30-day exports) and the WORM Audit Settings record at 390 x 844 and 1440 x 1000, +light and dark, captured with Playwright against `WormAuditDemoServer` on port 8099: +`worm-audit-home-{390,1440}-{light,dark}.jpg` (replaced) and +`worm-audit-settings-{390,1440}-{light,dark}.jpg` (new). Overflow check on all eight: no page +horizontal overflow, no inner horizontal scrollers, no double scrollbars. + +## Earlier captures + Refreshed after record-screen polish **2026-10-13 (America/Chicago)** from the rebuilt test-scope demo using packaged Next UI 1.0.0-RC.14 and connected Edge through the browser screenshot and viewport tools. All files are JPEG viewport captures, below 300 KB each. diff --git a/docs/screenshots/worm-audit-home-1440-dark.jpg b/docs/screenshots/worm-audit-home-1440-dark.jpg index 988242e..acfc5df 100644 Binary files a/docs/screenshots/worm-audit-home-1440-dark.jpg and b/docs/screenshots/worm-audit-home-1440-dark.jpg differ diff --git a/docs/screenshots/worm-audit-home-1440-light.jpg b/docs/screenshots/worm-audit-home-1440-light.jpg index 1e30816..484ba5e 100644 Binary files a/docs/screenshots/worm-audit-home-1440-light.jpg and b/docs/screenshots/worm-audit-home-1440-light.jpg differ diff --git a/docs/screenshots/worm-audit-home-390-dark.jpg b/docs/screenshots/worm-audit-home-390-dark.jpg index bf50542..16de6d4 100644 Binary files a/docs/screenshots/worm-audit-home-390-dark.jpg and b/docs/screenshots/worm-audit-home-390-dark.jpg differ diff --git a/docs/screenshots/worm-audit-home-390-light.jpg b/docs/screenshots/worm-audit-home-390-light.jpg index 0ca0e47..25a4763 100644 Binary files a/docs/screenshots/worm-audit-home-390-light.jpg and b/docs/screenshots/worm-audit-home-390-light.jpg differ diff --git a/docs/screenshots/worm-audit-settings-1440-dark.jpg b/docs/screenshots/worm-audit-settings-1440-dark.jpg new file mode 100644 index 0000000..3fcf976 Binary files /dev/null and b/docs/screenshots/worm-audit-settings-1440-dark.jpg differ diff --git a/docs/screenshots/worm-audit-settings-1440-light.jpg b/docs/screenshots/worm-audit-settings-1440-light.jpg new file mode 100644 index 0000000..39292fc Binary files /dev/null and b/docs/screenshots/worm-audit-settings-1440-light.jpg differ diff --git a/docs/screenshots/worm-audit-settings-390-dark.jpg b/docs/screenshots/worm-audit-settings-390-dark.jpg new file mode 100644 index 0000000..1fab827 Binary files /dev/null and b/docs/screenshots/worm-audit-settings-390-dark.jpg differ diff --git a/docs/screenshots/worm-audit-settings-390-light.jpg b/docs/screenshots/worm-audit-settings-390-light.jpg new file mode 100644 index 0000000..aeeb074 Binary files /dev/null and b/docs/screenshots/worm-audit-settings-390-light.jpg differ diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitConfig.java b/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitConfig.java index 6318507..2dc4230 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitConfig.java @@ -45,6 +45,9 @@ public class WormAuditQBitConfig implements QBitConfig { } + private String exportLockMode = "GOVERNANCE"; + private int exportRetentionDays = 30; + private final WormAuditSettings settings = new WormAuditSettings(this); private String appName = "worm_audit"; private String appLabel = "WORM Audit"; private String parentAppName; @@ -70,6 +73,63 @@ public class WormAuditQBitConfig implements QBitConfig private MetaDataCustomizerInterface tableMetaDataCustomizer = null; + /******************************************************************************* + ** Read exportLockMode. + *******************************************************************************/ + public String getExportLockMode() + { + return (exportLockMode); + } + + + /******************************************************************************* + ** Configure exportLockMode. + *******************************************************************************/ + public WormAuditQBitConfig withExportLockMode(String exportLockMode) + { + this.exportLockMode = exportLockMode; + return (this); + } + + + /******************************************************************************* + ** Read exportRetentionDays. + *******************************************************************************/ + public int getExportRetentionDays() + { + return (exportRetentionDays); + } + + + /******************************************************************************* + ** Configure exportRetentionDays. + *******************************************************************************/ + public WormAuditQBitConfig withExportRetentionDays(int exportRetentionDays) + { + this.exportRetentionDays = exportRetentionDays; + return (this); + } + + + /******************************************************************************* + ** Read effective settings through the instance-scoped 30-second cache. + *******************************************************************************/ + @JsonIgnore + public WormAuditSettings.Values effectiveSettings() + { + return (settings.read()); + } + + + /******************************************************************************* + ** Discard this instance's cached row after a successful local settings write. + *******************************************************************************/ + public void invalidateSettings() + { + settings.invalidate(); + } + + /******************************************************************************* ** Read the app name. *******************************************************************************/ @@ -527,6 +587,7 @@ public class WormAuditQBitConfig implements QBitConfig { throw (new IllegalArgumentException("Export bucket, prefix and schedule must be configured")); } + WormAuditSettings.validate(WormAuditSettings.defaults(this)); if(backendName == null || backendName.isBlank() || retentionYears < 1 || chainIntervalSeconds < 1 || lateDetailLookbackHours < 1 || settleHorizonSeconds < 0 || batchSize < 1 || checkpointEveryLinks < 1 || checkpointEveryMinutes < 1) { diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitProducer.java b/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitProducer.java index a16dc59..9ab916a 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/WormAuditQBitProducer.java @@ -32,6 +32,7 @@ import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses; import com.kingsrook.qqq.backend.core.alerts.AlertDefaultMonitor; import com.kingsrook.qqq.backend.core.alerts.AlertSeverity; import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData; +import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; @@ -182,21 +183,31 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer producedTables = output.getEach(QTableMetaData.class); + output.addSelfToInstance(qInstance); + output = new MetaDataProducerMultiOutput(); + var previousInstance = QContext.getQInstance(); + var previousSession = QContext.getQSession(); + var previousTransaction = QContext.getQBackendTransaction(); + try { - process.setSourceQBitName(qBit.getName()); - output.add(process); + // Production is still assembling the host metadata. Do not validate it mid-build. + QContext.setQInstance(qInstance); + QContext.setQSession(new com.kingsrook.qqq.backend.core.model.session.QSession()); + QContext.setTransaction(null); + addProcesses(output, qBit); } - for(var process : ExportProcesses.processes(qBitConfig)) + finally { - process.setSourceQBitName(qBit.getName()); - output.add(process); + QContext.setQInstance(previousInstance); + QContext.setQSession(previousSession); + QContext.setTransaction(previousTransaction); } if(qInstance.getSchedulers() == null || !qInstance.getSchedulers().containsKey(qBitConfig.getSchedulerName())) { qInstance.addScheduler(new QuartzSchedulerMetaData().withName(qBitConfig.getSchedulerName())); } - output.add(WormAuditApp.app(qBitConfig, output.getEach(QTableMetaData.class))); + output.add(WormAuditApp.app(qBitConfig, producedTables)); for(QWidgetMetaData widget : WormAuditApp.widgets()) { output.add(widget); @@ -224,6 +235,24 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer= minimum && value <= maximum) + { + return; + } + } + catch(NumberFormatException ignored) + { + // Use the same clear message for invalid numbers and invalid ranges. + } + throw (new IllegalArgumentException(label + " must be a whole number from " + minimum + " to " + maximum + ".")); + } + + + /******************************************************************************* + ** Recognize PostgreSQL/H2 missing-table errors without hiding other failures. + *******************************************************************************/ + private static boolean missingTable(Throwable error) + { + for(Throwable cause = error; cause != null; cause = cause.getCause()) + { + if(cause instanceof SQLException sql && ("42P01".equals(sql.getSQLState()) || "42S02".equals(sql.getSQLState()) || "42S04".equals(sql.getSQLState()))) + { + return (true); + } + } + return (false); + } +} diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/app/PostgresSetupWarnings.java b/src/main/java/com/kingsrook/qbits/wormaudit/app/PostgresSetupWarnings.java index 4089f60..fa9a7ee 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/app/PostgresSetupWarnings.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/app/PostgresSetupWarnings.java @@ -49,6 +49,11 @@ public final class PostgresSetupWarnings public static List load(QBackendTransaction transaction, WormAuditQBitConfig config) { List warnings = new ArrayList<>(); + int years = config.effectiveSettings().audit_retention_years(); + if(years != config.getRetentionYears()) + { + warnings.add("Audit retention setting differs from the configured database purge interval (" + config.getRetentionYears() + " years). A reviewed migration is required."); + } if(!(transaction instanceof RDBMSTransaction rdbms)) { return (warnings); @@ -60,8 +65,10 @@ public final class PostgresSetupWarnings { return (warnings); } + // The live function is authoritative on PostgreSQL, rather than the renderer defaults. + warnings.clear(); List tables = new ArrayList<>(config.getCoreAuditTableNames().values()); - tables.addAll(WormAuditTables.ALL); + tables.addAll(WormAuditTables.ALL.stream().filter(t -> !WormAuditTables.SETTINGS.equals(t)).toList()); for(String table : tables) { for(String trigger : List.of("worm_audit_row_guard", "worm_audit_truncate_guard")) @@ -93,6 +100,17 @@ public final class PostgresSetupWarnings } } } + try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT pg_get_functiondef(to_regprocedure('public.worm_audit_purge(timestamp with time zone)'))")) + { + if(result.next() && result.getString(1) != null) + { + var interval = java.util.regex.Pattern.compile("(?i)interval\\s+'([0-9]+) years'").matcher(result.getString(1)); + if(interval.find() && Integer.parseInt(interval.group(1)) != years) + { + warnings.add("Audit retention setting differs from the purge function's compiled interval (" + interval.group(1) + " years). A reviewed migration is required."); + } + } + } try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT to_regprocedure('public.worm_audit_purge(timestamp with time zone)') IS NOT NULL")) { result.next(); diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditApp.java b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditApp.java index 356de5a..eb71596 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditApp.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditApp.java @@ -88,7 +88,8 @@ public final class WormAuditApp for(String name : WormAuditTables.NAV) { QTableMetaData table = tables.stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow(); - app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table); + app.withSectionOfChildren(new QAppSection().withName(name).withLabel(WormAuditTables.navigationLabel(name)), + WormAuditTables.SETTINGS.equals(name) ? table.clone().withLabel("Settings") : table); } return (app); } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeData.java b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeData.java index 1b3856e..6cc927e 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeData.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeData.java @@ -54,7 +54,7 @@ import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, long settling, int settleSeconds, QRecord lastVerification, QRecord openBreak, QRecord lastExport, long exportLag, Long exportAgeHours, List exportGaps, String bucketAlias, String lockCapability, String lockReason, - int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge, + String exportLockMode, int exportRetentionDays, int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge, List recentPurges, List warnings) { /******************************************************************************* @@ -71,7 +71,16 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo *******************************************************************************/ public static WormAuditHomeData load(WormAuditQBitConfig config, SealedStore storage, Instant now) throws QException { + var settings = config.effectiveSettings(); List warnings = new ArrayList<>(); + if("GOVERNANCE".equals(settings.export_lock_mode())) + { + warnings.add("Sealed exports use GOVERNANCE mode: an administrator with s3:BypassGovernanceRetention can delete them. Switch to COMPLIANCE for write-once copies."); + } + if(now.plus(Duration.ofDays(settings.export_retention_days())).isBefore(now.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant())) + { + warnings.add("Export retention is shorter than audit retention."); + } String capability = "UNKNOWN"; String reason = "Object Lock capability could not be confirmed"; try @@ -113,7 +122,7 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo AuditStore store = new AuditStore(transaction); QRecord lastLink = store.last(WormAuditTables.CHAIN); long links = store.count(WormAuditTables.CHAIN, new QQueryFilter()); - Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds()); + Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds()); long backlog = store.unchainedCount(cutoff, true); long settling = store.unchainedCount(cutoff, false); QRecord verification = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(), "finished_at"); @@ -147,10 +156,10 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo warnings.addAll(PostgresSetupWarnings.load(transaction, config)); transaction.commit(); return (new WormAuditHomeData(links, lastLink == null ? null : lastLink.getValueInstant("chained_at"), backlog, settling, - config.getSettleHorizonSeconds(), verification, openBreak, export, lag, + settings.settle_horizon_seconds(), verification, openBreak, export, lag, export == null || export.getValueInstant("sealed_at") == null ? null : Math.max(0, Duration.between(export.getValueInstant("sealed_at"), now).toHours()), - List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, config.getRetentionYears(), anchorId == 0 ? null : anchor, - oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant(), List.copyOf(purges), List.copyOf(warnings))); + List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, settings.export_lock_mode(), settings.export_retention_days(), settings.audit_retention_years(), anchorId == 0 ? null : anchor, + oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant(), List.copyOf(purges), List.copyOf(warnings))); } catch(Exception e) { diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeView.java b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeView.java index 1310c06..6a2ed25 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeView.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/app/WormAuditHomeView.java @@ -82,6 +82,8 @@ public final class WormAuditHomeView } close(html); open(html, "Export status"); + row(html, "Export lock mode", data.exportLockMode()); + row(html, "Export retention", data.exportRetentionDays() + " days"); QRecord export = data.lastExport(); if(export == null) { @@ -93,7 +95,7 @@ public final class WormAuditHomeView row(html, "Link range", export.getValueLong("from_link_id") + "–" + export.getValueLong("to_link_id")); row(html, "Sealed at", when(export.getValueInstant("sealed_at"), "Not recorded")); row(html, "Retain until", when(export.getValueInstant("retain_until"), "Not recorded")); - row(html, "Lock mode", export.getValueString("lock_mode")); + row(html, "Last export lock mode", export.getValueString("lock_mode")); } row(html, "Export lag", data.exportLag() + " links; " + (data.exportAgeHours() == null ? "no previous export time" : data.exportAgeHours() + " h since last export")); row(html, "Gaps in sealed ranges", data.exportGaps().isEmpty() ? (export == null ? "No sealed ranges to check" : "None") : String.join(", ", data.exportGaps())); diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/chain/WormAuditChainer.java b/src/main/java/com/kingsrook/qbits/wormaudit/chain/WormAuditChainer.java index 94b1760..b7106ca 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/chain/WormAuditChainer.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/chain/WormAuditChainer.java @@ -31,6 +31,7 @@ import java.util.TreeMap; import java.util.concurrent.locks.ReentrantLock; import java.util.stream.Collectors; import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig; +import com.kingsrook.qbits.wormaudit.WormAuditSettings; import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints; import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard; import com.kingsrook.qbits.wormaudit.model.WormAuditTables; @@ -99,7 +100,7 @@ public class WormAuditChainer } } } - Counts counts = append(new AuditStore(transaction), now); + Counts counts = append(new AuditStore(transaction), now, config.effectiveSettings()); transaction.commit(); return (counts); } @@ -119,14 +120,14 @@ public class WormAuditChainer /******************************************************************************* ** Append settled headers and uncovered detail chunks, then checkpoint. *******************************************************************************/ - private Counts append(AuditStore store, Instant now) throws Exception + private Counts append(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception { QRecord tail = store.last(WormAuditTables.CHAIN); QRecord anchor = store.anchor(); long nextId = tail == null ? anchor.getValueLong("anchor_link_id") + 1 : tail.getValueLong("id") + 1; String previous = tail == null ? anchor.getValueString("anchor_hash") : tail.getValueString("link_hash"); List appended = new ArrayList<>(); - Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds()); + Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds()); List audits = store.unchained(cutoff, config.getBatchSize()); Map details = AuditStore.index(store.in("auditDetail", "auditId", audits.stream().map(r -> r.getValueLong("id")).toList())); Map> grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId"))); @@ -149,7 +150,7 @@ public class WormAuditChainer insertLinks(store, appended); int linkCount = appended.size(); appended.clear(); - details = AuditStore.index(store.lateDetails(cutoff, config.getLateDetailLookbackHours(), config.getBatchSize())); + details = AuditStore.index(store.lateDetails(cutoff, settings.late_detail_lookback_hours(), config.getBatchSize())); grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId"), TreeMap::new, Collectors.toList())); Map headers = AuditStore.index(store.in("audit", "id", grouped.keySet())); for(var entry : grouped.entrySet()) @@ -165,7 +166,7 @@ public class WormAuditChainer } } insertLinks(store, appended); - return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now), false)); + return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now, settings), false)); } @@ -195,7 +196,7 @@ public class WormAuditChainer /******************************************************************************* ** Checkpoint the uncheckpointed suffix by count or elapsed time. *******************************************************************************/ - private int checkpoint(AuditStore store, Instant now) throws Exception + private int checkpoint(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception { if(config.getSigningKey() == null) { @@ -206,13 +207,13 @@ public class WormAuditChainer int count = 0; while(true) { - List pending = store.page(WormAuditTables.CHAIN, lastId, config.getCheckpointEveryLinks()); + List pending = store.page(WormAuditTables.CHAIN, lastId, settings.checkpoint_every_links()); if(pending.isEmpty()) { return (count); } Instant since = latest == null ? pending.getFirst().getValueInstant("chained_at") : latest.getValueInstant("created_at"); - if(pending.size() < config.getCheckpointEveryLinks() && now.isBefore(since.plusSeconds(config.getCheckpointEveryMinutes() * 60L))) + if(pending.size() < settings.checkpoint_every_links() && now.isBefore(since.plusSeconds(settings.checkpoint_every_minutes() * 60L))) { return (count); } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/ddl/WormAuditDdl.java b/src/main/java/com/kingsrook/qbits/wormaudit/ddl/WormAuditDdl.java index b098268..5f83a5d 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/ddl/WormAuditDdl.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/ddl/WormAuditDdl.java @@ -102,6 +102,13 @@ public final class WormAuditDdl { sql = sql.replace("${" + entry.getValue() + "}", identifier(names.get(entry.getKey()))); } + var defaults = com.kingsrook.qbits.wormaudit.WormAuditSettings.defaults(config); + com.kingsrook.qbits.wormaudit.WormAuditSettings.validate(defaults); + String values = java.util.List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days", "checkpoint_every_links", + "checkpoint_every_minutes", "chain_interval_seconds", "settle_horizon_seconds", "late_detail_lookback_hours", "export_cron", "verify_cron") + .stream().map(field -> defaults.getValue(field) instanceof Number ? defaults.getValue(field).toString() + : "'" + defaults.getValueString(field).replace("'", "''") + "'").collect(java.util.stream.Collectors.joining(", ")); + sql = sql.replace("${settings_values}", values); return (sql.replace("${app_role}", identifier(config.getAppRole())).replace("${retention}", config.getRetentionYears() + " years")); } catch(IOException e) diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/export/ExportFiles.java b/src/main/java/com/kingsrook/qbits/wormaudit/export/ExportFiles.java index 93f9f1c..96a97d4 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/export/ExportFiles.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/export/ExportFiles.java @@ -152,9 +152,9 @@ public class ExportFiles implements AutoCloseable /******************************************************************************* ** A lock echo is required even if a provider advertises the capability. *******************************************************************************/ - public static boolean locked(StoredObject object, java.time.Instant until) + public static boolean locked(StoredObject object, String mode, java.time.Instant until) { - return (object != null && "COMPLIANCE".equals(object.getLockMode()) && object.getLockRetainUntil() != null + return (object != null && java.util.Objects.equals(mode, object.getLockMode()) && object.getLockRetainUntil() != null && !object.getLockRetainUntil().isBefore(until)); } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/export/ObjectStorageSealedStore.java b/src/main/java/com/kingsrook/qbits/wormaudit/export/ObjectStorageSealedStore.java index 492f633..58ed2ef 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/export/ObjectStorageSealedStore.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/export/ObjectStorageSealedStore.java @@ -29,7 +29,6 @@ import java.time.Instant; import com.kingsrook.qbits.objectstorage.api.Capability; import com.kingsrook.qbits.objectstorage.api.CapabilityStatus; import com.kingsrook.qbits.objectstorage.api.ListRequest; -import com.kingsrook.qbits.objectstorage.api.LockMode; import com.kingsrook.qbits.objectstorage.api.ObjectStorage; import com.kingsrook.qbits.objectstorage.api.ObjectSummary; import com.kingsrook.qbits.objectstorage.api.Page; @@ -84,12 +83,12 @@ public class ObjectStorageSealedStore implements SealedStore ** Stream a bounded upload and close the source even when the provider fails. *******************************************************************************/ @Override - public String put(String key, Path file, Instant retainUntil) throws Exception + public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception { try(InputStream input = Files.newInputStream(file)) { return (bucket.put(PutRequest.of(key, input).withContentLength(Files.size(file)) - .withObjectLock(LockMode.COMPLIANCE, retainUntil)).getVersionId()); + .withObjectLock(mode, retainUntil)).getVersionId()); } } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExportVerifier.java b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExportVerifier.java index ecea6f6..9d22948 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExportVerifier.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExportVerifier.java @@ -94,6 +94,7 @@ public class SealedExportVerifier *******************************************************************************/ public Report verify(QRecord export, String trigger) throws QException { + config.effectiveSettings(); VerificationResult failure = new VerificationResult(); failure.scope(0, export.getValueLong("from_link_id"), export.getValueLong("to_link_id")); Report report; @@ -121,22 +122,34 @@ public class SealedExportVerifier throw (new IOException("Missing sealed object version")); } var head = storage.head(key); + if(head == null) + { + failure.add(null, null, "object " + key + " is missing" + ("GOVERNANCE".equals(export.getValueString("lock_mode")) + ? "; possible GOVERNANCE bypass removal by an administrator" : "")); + } if(head != null && !version.equals(head.getVersionId())) { failure.add(null, null, "object " + name + " was overwritten after sealing (current version " + head.getVersionId() + ", sealed version " + version + " is still retained under Object Lock)"); } - if(!ExportFiles.locked(head, export.getValueInstant("retain_until"))) + if(!ExportFiles.locked(head, export.getValueString("lock_mode"), export.getValueInstant("retain_until"))) { - failure.add(null, null, "object " + name + " lacks the recorded COMPLIANCE retention"); + failure.add(null, null, "object " + name + " lacks the recorded " + export.getValueString("lock_mode") + " retention"); } } for(String name : ExportFiles.NAMES) { - try(InputStream input = storage.getVersion(export.getValueString("object_prefix") + name, (String) versions.get(name))) + String key = export.getValueString("object_prefix") + name; + try(InputStream input = storage.getVersion(key, (String) versions.get(name))) { Files.copy(input, files.file(name)); } + catch(Exception e) + { + failure.add(null, null, "object " + key + " sealed version cannot be read" + ("GOVERNANCE".equals(export.getValueString("lock_mode")) + ? "; if removed, possible GOVERNANCE bypass removal by an administrator" : "")); + throw (new IOException("Cannot read sealed object version")); + } } report = inspect(files, export, new AuditStore(transaction)); for(var item : failure.getBreaks()) @@ -189,7 +202,7 @@ public class SealedExportVerifier || !Objects.equals(manifest.get("fromLinkId"), from) || !Objects.equals(manifest.get("toLinkId"), to) || !Objects.equals(manifest.get("linkCount"), to - from + 1) || !Objects.equals(manifest.get("createdAt"), export.getValueInstant("sealed_at").toString()) - || !"COMPLIANCE".equals(export.getValueString("lock_mode")) + || !java.util.Set.of("GOVERNANCE", "COMPLIANCE").contains(export.getValueString("lock_mode")) || !Objects.equals(export.getValueString("manifest_key"), export.getValueString("object_prefix") + "manifest.json")) { result.add(null, null, "Manifest metadata does not match the export row"); @@ -256,7 +269,7 @@ public class SealedExportVerifier long start = checkpoint.getValueLong("from_link_id"); long count = checkpoint.getValueLong("link_count"); if(start != cursor + 1 || end < start || end > to || count != end - start + 1 - || count > config.getCheckpointEveryLinks() || checkpointId <= lastCheckpoint) + || count > Integer.MAX_VALUE || checkpointId <= lastCheckpoint) { result.add(null, null, label + "invalid or incomplete checkpoint range"); break; diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExporter.java b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExporter.java index ecf00c2..076eeea 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExporter.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedExporter.java @@ -243,7 +243,9 @@ public class SealedExporter { throw (new ExportFailure("Export refused: bucket does not support Object Lock")); } - Instant until = now.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant(); + var settings = config.effectiveSettings(); + var mode = com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(settings.export_lock_mode()); + Instant until = now.plus(settings.export_retention_days(), ChronoUnit.DAYS); String prefix = config.getExportPrefix() + "/" + DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(now) + "/" + scope.from() + "-" + scope.to() + "/"; QRecord row; @@ -270,7 +272,7 @@ public class SealedExporter .withValue("bucket_alias", config.getExportBucketAlias()).withValue("object_prefix", prefix) .withValue("manifest_key", prefix + "manifest.json").withValue("manifest_sha256", ExportFiles.sha256(files.file("manifest.json"))) .withValue("prev_manifest_sha256", previous).withValue("object_count", 3L) - .withValue("lock_mode", "COMPLIANCE").withValue("retain_until", until).withValue("sealed_at", now); + .withValue("lock_mode", mode.name()).withValue("retain_until", until).withValue("sealed_at", now); if(!new SealedExportVerifier(config, storage).inspect(files, row, store).result().passed()) { throw (new ExportFailure("Export refused: source does not match the signed chain")); @@ -279,15 +281,15 @@ public class SealedExporter Map versions = new LinkedHashMap<>(); for(String name : ExportFiles.NAMES) { - String version = storage.put(prefix + name, files.file(name), until); + String version = storage.put(prefix + name, files.file(name), mode, until); if(!ExportFiles.validVersion(version)) { throw (new ExportFailure("Export refused: object " + name + " returned no version id")); } var headObject = storage.head(prefix + name); - if(!ExportFiles.locked(headObject, until)) + if(!ExportFiles.locked(headObject, mode.name(), until)) { - throw (new ExportFailure("Export refused: object " + name + " did not echo COMPLIANCE retention")); + throw (new ExportFailure("Export refused: object " + name + " did not echo " + mode.name() + " retention")); } if(!version.equals(headObject.getVersionId())) { diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedStore.java b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedStore.java index 055ebd2..9e3672f 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedStore.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/export/SealedStore.java @@ -50,9 +50,9 @@ public interface SealedStore } /******************************************************************************* - ** Upload a file with COMPLIANCE retention. + ** Upload a file with the selected Object Lock mode and retention. *******************************************************************************/ - String put(String key, Path file, Instant retainUntil) throws Exception; + String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception; /******************************************************************************* ** Read current object metadata, or null for an absent object. diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/guard/AppendOnlyGuard.java b/src/main/java/com/kingsrook/qbits/wormaudit/guard/AppendOnlyGuard.java index 20e2a6c..9d89e05 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/guard/AppendOnlyGuard.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/guard/AppendOnlyGuard.java @@ -45,6 +45,13 @@ public class AppendOnlyGuard implements MetaDataCustomizerInterface preInsert(InsertInput input, List records, boolean preview) throws QException + { + if(records.size() != 1 || !Integer.valueOf(1).equals(records.getFirst().getValueInteger("id"))) + { + throw (new QUserFacingException("WORM Audit Settings has one row only (ID 1).")); + } + if(new com.kingsrook.qbits.wormaudit.chain.AuditStore(input.getTransaction()).get(WormAuditTables.SETTINGS, 1) != null) + { + throw (new QUserFacingException("WORM Audit Settings already exists. Edit record 1 instead of adding another record.")); + } + return (validate(records)); + } + + + /******************************************************************************* + ** Validate partial edits without replacing the existing settings. + *******************************************************************************/ + @Override + public List preUpdate(UpdateInput input, List records, boolean preview, Optional> oldRecords) throws QException + { + PermissionsHelper.checkTablePermissionThrowing(input, TablePermissionSubType.EDIT); + return (validate(records)); + } + + + /******************************************************************************* + ** Reject deletion even for direct core calls where capability checks are off. + *******************************************************************************/ + @Override + public List preDelete(DeleteInput input, List records, boolean preview) throws QException + { + throw (new QUserFacingException("WORM Audit Settings cannot be deleted.")); + } + + + /******************************************************************************* + ** Refresh local settings after persistence; remote JVMs retain the bounded cache. + *******************************************************************************/ + @Override + public List postUpdate(UpdateInput input, List records, Optional> oldRecords) throws QException + { + com.kingsrook.qbits.wormaudit.processes.VerificationRuns.config().invalidateSettings(); + return (records); + } + + + /******************************************************************************* + ** Stamp the editor and time on every valid write; core writes FIELD audits. + *******************************************************************************/ + private List validate(List records) throws QException + { + for(QRecord row : records) + { + try + { + WormAuditSettings.validate(row); + } + catch(IllegalArgumentException e) + { + throw (new QUserFacingException(e.getMessage())); + } + row.setValue("modify_date", Instant.now()); + row.setValue("modified_by", QContext.getQSession() == null || QContext.getQSession().getUser() == null + ? "system" : QContext.getQSession().getUser().getIdReference()); + } + return (records); + } +} diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/model/SettingsPermissions.java b/src/main/java/com/kingsrook/qbits/wormaudit/model/SettingsPermissions.java new file mode 100644 index 0000000..19084ef --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/wormaudit/model/SettingsPermissions.java @@ -0,0 +1,55 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.wormaudit.model; + + +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; + + +/******************************************************************************* + ** Reading uses the full table name; editing additionally needs settings access. + *******************************************************************************/ +public class SettingsPermissions implements CustomPermissionChecker +{ + /******************************************************************************* + ** Enforce table permission and the separate settings permission. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + var session = QContext.getQSession(); + boolean edit = input instanceof UpdateInput; + if(input instanceof InsertInput || input instanceof DeleteInput || session == null + || !session.hasPermission(WormAuditTables.SETTINGS + (edit ? ".edit" : ".read")) + || (edit && !session.hasPermission("worm_audit.settings"))) + { + throw (new QPermissionDeniedException(edit ? "Editing WORM Audit Settings requires worm_audit_settings.edit and worm_audit.settings." + : "WORM Audit Settings requires read access; adding or deleting settings is not allowed.")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/model/WormAuditTables.java b/src/main/java/com/kingsrook/qbits/wormaudit/model/WormAuditTables.java index 046afb5..c3659ec 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/model/WormAuditTables.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/model/WormAuditTables.java @@ -24,16 +24,21 @@ package com.kingsrook.qbits.wormaudit.model; import java.util.ArrayList; import java.util.List; import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig; +import com.kingsrook.qbits.wormaudit.WormAuditSettings; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers; import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel; import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent; import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue; import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource; import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier; @@ -51,8 +56,9 @@ public final class WormAuditTables public static final String EXPORT = "worm_audit_export"; public static final String VERIFICATION = "worm_audit_verification"; public static final String PURGE = "worm_audit_purge"; - public static final List ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE); - public static final List NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE); + public static final String SETTINGS = "worm_audit_settings"; + public static final List ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE, SETTINGS); + public static final List NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE, SETTINGS); /******************************************************************************* ** Utility class. @@ -73,10 +79,72 @@ public final class WormAuditTables tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T object_versions:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D")); tables.add(table(config, VERIFICATION, "Verification", "id:L kind:S trigger:S started_at:D finished_at:D from_link_id:L to_link_id:L links_checked:L outcome:S break_count:L first_break_link_id:L first_break_audit_id:L first_break_reason:T export_id:L run_by:S")); tables.add(table(config, PURGE, "Purge", "id:L requested_at:D requested_by:S cutoff:D retention_interval:S outcome:S reason:T audits_deleted:L details_deleted:L links_deleted:L anchor_link_id:L anchor_hash:S")); + tables.add(settings(config)); return (tables); } + /******************************************************************************* + ** Editable, FIELD-audited singleton with standard QQQ record sections. + *******************************************************************************/ + public static QTableMetaData settings(WormAuditQBitConfig config) + { + QTableMetaData table = new QTableMetaData().withName(SETTINGS).withLabel("WORM Audit Settings") + .withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(SETTINGS)) + .withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("WORM Audit Settings") + .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS) + .withPermissionBaseName(SETTINGS).withCustomPermissionChecker(new QCodeReference(SettingsPermissions.class))) + .withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE) + .withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SettingsCustomizer.class)) + .withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class)) + .withCustomizer(TableCustomizers.POST_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class)) + .withCustomizer(TableCustomizers.PRE_DELETE_RECORD, new QCodeReference(SettingsCustomizer.class)) + .withField(new QFieldMetaData("id", QFieldType.INTEGER).withLabel("ID").withBackendName("id").withIsEditable(false).withDefaultValue(1)); + var defaults = WormAuditSettings.defaults(config); + table.withField(new QFieldMetaData("audit_retention_years", QFieldType.INTEGER).withBackendName("audit_retention_years").withLabel("Audit retention years") + .withIsRequired(true).withDefaultValue(defaults.getValue("audit_retention_years"))); + table.withField(new QFieldMetaData("export_lock_mode", QFieldType.STRING).withBackendName("export_lock_mode").withLabel("Export lock mode") + .withIsRequired(true).withDefaultValue(defaults.getValue("export_lock_mode"))); + table.withField(new QFieldMetaData("export_retention_days", QFieldType.INTEGER).withBackendName("export_retention_days").withLabel("Export retention days") + .withIsRequired(true).withDefaultValue(defaults.getValue("export_retention_days"))); + table.withField(new QFieldMetaData("checkpoint_every_links", QFieldType.INTEGER).withBackendName("checkpoint_every_links").withLabel("Checkpoint every links") + .withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_links"))); + table.withField(new QFieldMetaData("checkpoint_every_minutes", QFieldType.INTEGER).withBackendName("checkpoint_every_minutes").withLabel("Checkpoint every minutes") + .withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_minutes"))); + table.withField(new QFieldMetaData("chain_interval_seconds", QFieldType.INTEGER).withBackendName("chain_interval_seconds").withLabel("Chain interval seconds") + .withIsRequired(true).withDefaultValue(defaults.getValue("chain_interval_seconds"))); + table.withField(new QFieldMetaData("settle_horizon_seconds", QFieldType.INTEGER).withBackendName("settle_horizon_seconds").withLabel("Settle horizon seconds") + .withIsRequired(true).withDefaultValue(defaults.getValue("settle_horizon_seconds"))); + table.withField(new QFieldMetaData("late_detail_lookback_hours", QFieldType.INTEGER).withBackendName("late_detail_lookback_hours").withLabel("Late detail lookback hours") + .withIsRequired(true).withDefaultValue(defaults.getValue("late_detail_lookback_hours"))); + table.withField(new QFieldMetaData("export_cron", QFieldType.STRING).withBackendName("export_cron").withLabel("Export schedule (UTC)") + .withIsRequired(true).withDefaultValue(defaults.getValue("export_cron"))); + table.withField(new QFieldMetaData("verify_cron", QFieldType.STRING).withBackendName("verify_cron").withLabel("Verify schedule (UTC)") + .withIsRequired(true).withDefaultValue(defaults.getValue("verify_cron"))); + table.getField("export_lock_mode").withInlinePossibleValueSource(new QPossibleValueSource().withName(SETTINGS + "_lock_mode") + .withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of( + new QPossibleValue<>("GOVERNANCE", "GOVERNANCE"), new QPossibleValue<>("COMPLIANCE", "COMPLIANCE")))); + for(String field : List.of("export_cron", "verify_cron", "chain_interval_seconds")) + { + table.getField(field).withHelpContent(new QHelpContent("Takes effect after restart. Cron schedules use UTC and Quartz syntax.")); + } + table.getField("audit_retention_years").withHelpContent(new QHelpContent("Used for the next eligible purge date. The database purge interval changes only through a reviewed migration.")); + table.withField(new QFieldMetaData("notes", QFieldType.TEXT).withLabel("Notes").withBackendName("notes")); + table.withField(new QFieldMetaData("modify_date", QFieldType.DATE_TIME).withLabel("Modified").withBackendName("modify_date").withIsEditable(false)); + table.withField(new QFieldMetaData("modified_by", QFieldType.STRING).withLabel("Modified by").withBackendName("modified_by").withIsEditable(false)); + table.withSection(new QFieldSection("retention", "Retention", new QIcon("lock"), Tier.T1, + List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days")).withGridColumns(12)); + table.withSection(new QFieldSection("cadence", "Chain and checkpoints", new QIcon("link"), Tier.T2, + List.of("checkpoint_every_links", "checkpoint_every_minutes", "settle_horizon_seconds", "late_detail_lookback_hours")).withGridColumns(12)); + table.withSection(new QFieldSection("schedules", "Schedules (after restart)", new QIcon("schedule"), Tier.T2, + List.of("chain_interval_seconds", "export_cron", "verify_cron")).withGridColumns(12)); + table.withSection(new QFieldSection("changes", "Changes", new QIcon("edit_note"), Tier.T2, + List.of("notes", "modify_date", "modified_by")).withGridColumns(12)); + return (table); + } + + /******************************************************************************* ** Common metadata for a process-written table. *******************************************************************************/ @@ -169,6 +237,7 @@ public final class WormAuditTables { return (switch(name) { + case SETTINGS -> "Settings"; case CHAIN -> "Audit Chain"; case CHECKPOINT -> "Checkpoints"; case EXPORT -> "Sealed Exports"; diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/processes/ExportProcesses.java b/src/main/java/com/kingsrook/qbits/wormaudit/processes/ExportProcesses.java index 8e760bc..fc03daf 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/processes/ExportProcesses.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/processes/ExportProcesses.java @@ -84,6 +84,7 @@ public final class ExportProcesses *******************************************************************************/ public static List processes(WormAuditQBitConfig config) { + var settings = config.effectiveSettings(); Map values = new HashMap<>(); values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.EXPORT); values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.EXPORT); @@ -97,7 +98,7 @@ public final class ExportProcesses QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily sealed audit export") .withStep(new QBackendStepMetaData().withName("export").withCode(new QCodeReference(Daily.class))) .withPermissionRules(permission(PERMISSION)).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()) - .withCronExpression(config.getExportCron()).withCronTimeZoneId("UTC")); + .withCronExpression(settings.export_cron()).withCronTimeZoneId("UTC")); return (List.of(export, verify, daily)); } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/processes/WormAuditProcesses.java b/src/main/java/com/kingsrook/qbits/wormaudit/processes/WormAuditProcesses.java index c610e37..f39defc 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/processes/WormAuditProcesses.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/processes/WormAuditProcesses.java @@ -89,6 +89,7 @@ public final class WormAuditProcesses *******************************************************************************/ public static List processes(WormAuditQBitConfig config) { + var settings = config.effectiveSettings(); Map values = new HashMap<>(); values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.CHAIN); values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.CHAIN); @@ -100,11 +101,11 @@ public final class WormAuditProcesses QProcessMetaData chain = new QProcessMetaData().withName(CHAIN).withLabel("Chain settled audit records") .withStep(new QBackendStepMetaData().withName("chain").withCode(new QCodeReference(ChainTick.class))) .withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()) - .withRepeatSeconds(config.getChainIntervalSeconds()).withInitialDelaySeconds(config.getChainIntervalSeconds())); + .withRepeatSeconds(settings.chain_interval_seconds()).withInitialDelaySeconds(settings.chain_interval_seconds())); QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily audit chain verification") .withStep(new QBackendStepMetaData().withName("verify").withCode(new QCodeReference(Daily.class))) .withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()) - .withCronExpression(config.getVerifyCron()).withCronTimeZoneId("UTC")); + .withCronExpression(settings.verify_cron()).withCronTimeZoneId("UTC")); return (List.of(verify, chain, daily)); } diff --git a/src/main/java/com/kingsrook/qbits/wormaudit/verify/WormAuditVerifier.java b/src/main/java/com/kingsrook/qbits/wormaudit/verify/WormAuditVerifier.java index 1d56da4..7da1a85 100644 --- a/src/main/java/com/kingsrook/qbits/wormaudit/verify/WormAuditVerifier.java +++ b/src/main/java/com/kingsrook/qbits/wormaudit/verify/WormAuditVerifier.java @@ -133,6 +133,7 @@ public class WormAuditVerifier *******************************************************************************/ VerificationResult verify(AuditStore store, Set selection, Instant now) throws QException { + var settings = config.effectiveSettings(); VerificationResult result = new VerificationResult(now); if(config.getSigningKey() == null) { @@ -214,7 +215,7 @@ public class WormAuditVerifier } if(selection == null) { - Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds()); + Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds()); if(store.isRdbms()) { store.completeness(cutoff, result); @@ -500,7 +501,7 @@ public class WormAuditVerifier result.add(null, null, prefix + "signature does not verify"); } if(from < 1 || to < from || checkpoint.getValueLong("link_count") != to - from + 1 - || to - from + 1 > config.getCheckpointEveryLinks()) + || to - from + 1 > Integer.MAX_VALUE) { result.add(null, null, prefix + "invalid link range or count"); return; diff --git a/src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql b/src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql index 343da5e..7b85a92 100644 --- a/src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql +++ b/src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql @@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain; CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage(); + +-- changeset worm-audit:10 +-- Editable singleton: deliberately excluded from append-only triggers. +CREATE TABLE IF NOT EXISTS public.worm_audit_settings ( + id integer PRIMARY KEY CHECK (id = 1), + audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100), + export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')), + export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500), + checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0), + checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0), + chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0), + settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0), + late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0), + export_cron text NOT NULL DEFAULT '0 0 2 * * ?', + verify_cron text NOT NULL DEFAULT '0 0 3 * * ?', + notes text, + modify_date timestamptz NOT NULL DEFAULT now(), + modified_by text NOT NULL DEFAULT current_user +); +REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM ${app_role}; +GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO ${app_role}; +INSERT INTO public.worm_audit_settings + (id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links, + checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours, + export_cron, verify_cron) +SELECT ${settings_values} +WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1); diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/MinioExportAcceptanceTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/MinioExportAcceptanceTest.java index 85c4ba0..d0c4f0a 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/MinioExportAcceptanceTest.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/MinioExportAcceptanceTest.java @@ -78,7 +78,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy; class MinioExportAcceptanceTest { /******************************************************************************* - ** Production adapter, COMPLIANCE retention, version preservation and UI findings. + ** Production adapter, GOVERNANCE and COMPLIANCE retention, version preservation and UI findings. *******************************************************************************/ @Test void lockedRoundTripVersionRetentionLiveTamperAndUnsupportedBucket() throws Exception @@ -130,17 +130,17 @@ class MinioExportAcceptanceTest { String key = row.getValueString("object_prefix") + name; var head = storage.head(key); - assertThat(head.getLockMode()).isEqualTo("COMPLIANCE"); - assertThat(Duration.between(now.atZone(java.time.ZoneOffset.UTC).plusYears(6).toInstant(), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2)); + assertThat(head.getLockMode()).isEqualTo("GOVERNANCE"); + assertThat(Duration.between(now.plus(Duration.ofDays(30)), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2)); String originalVersion = head.getVersionId(); assertThat(originalVersion).isNotBlank(); assertThat(ExportFiles.parse(row.getValueString("object_versions").getBytes(java.nio.charset.StandardCharsets.UTF_8))) .containsEntry(name, originalVersion); - assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion))) + assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion).bypassGovernanceRetention(false))) .isInstanceOf(S3Exception.class); byte[] original = s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray(); // S3 overwrites create new versions; they cannot replace the locked version. - var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("COMPLIANCE") + var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("GOVERNANCE") .objectLockRetainUntilDate(row.getValueInstant("retain_until")), RequestBody.fromString("different replacement bytes")); assertThat(replacement.versionId()).isNotEqualTo(originalVersion); assertThat(s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray()).isEqualTo(original); @@ -163,11 +163,26 @@ class MinioExportAcceptanceTest tamper("auditDetail", 1, "newValue", "new"); audit(2); new WormAuditChainer(config).chainOnce(); + config.withExportLockMode("COMPLIANCE"); + var compliance = new SealedExporter(config).run("USER", Instant.now()); + assertThat(compliance.failed()).isFalse(); + assertThat(compliance.row().getValueString("lock_mode")).isEqualTo("COMPLIANCE"); + assertThat(verifier.verify(compliance.row(), "USER").result().passed()).isTrue(); + for(String name : ExportFiles.NAMES) + { + String key = compliance.row().getValueString("object_prefix") + name; + var head = storage.head(key); + assertThat(head.getLockMode()).isEqualTo("COMPLIANCE"); + assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(head.getVersionId()))) + .isInstanceOf(S3Exception.class); + } + audit(3); + new WormAuditChainer(config).chainOnce(); config.withExportBucketAlias("plain"); var failed = new SealedExporter(config).run("SCHEDULE", Instant.now()); assertThat(failed.failed()).isTrue(); assertThat(failed.message()).contains("does not support Object Lock"); - assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1); + assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(2); assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(1); } finally diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/PostgresAcceptanceTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/PostgresAcceptanceTest.java index bc7b70b..a089bba 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/PostgresAcceptanceTest.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/PostgresAcceptanceTest.java @@ -113,7 +113,6 @@ class PostgresAcceptanceTest void database() throws Exception { config = ChainVerificationTest.signed().withSettleHorizonSeconds(0); - instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true); try(var c = owner()) { sql( @@ -124,6 +123,7 @@ class PostgresAcceptanceTest c, "DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='app') THEN" + " CREATE ROLE app LOGIN PASSWORD 'fixture'; END IF; END $$"); + instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true); RdbmsTestSupport.createTables(c, instance, true); String ddl = WormAuditDdl.postgres(config); sql(c, ddl); @@ -142,9 +142,17 @@ class PostgresAcceptanceTest { var storage = new com.kingsrook.qbits.wormaudit.demo.MemorySealedStore(); var clean = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()); - assertThat(clean.warnings()).containsExactly("Daily verification has never run"); + assertThat(clean.warnings()).contains("Daily verification has never run").noneMatch(w -> w.contains("guard") && w.contains("settings")); try(var connection = owner()) { + sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=7 WHERE id=1"); + config.invalidateSettings(); + assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings()) + .anyMatch(w -> w.contains("purge function's compiled interval (6 years)")); + sql(connection, WormAuditDdl.postgres(config)); + assertThat(count(connection, "SELECT audit_retention_years FROM worm_audit_settings WHERE id=1")).isEqualTo(7); + sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=6 WHERE id=1"); + config.invalidateSettings(); sql(connection, "DROP TRIGGER worm_audit_row_guard ON audit_detail"); sql(connection, "ALTER TABLE worm_audit_chain DISABLE TRIGGER worm_audit_truncate_guard"); sql(connection, "DROP FUNCTION public.worm_audit_purge(timestamptz)"); @@ -338,7 +346,7 @@ class PostgresAcceptanceTest void prefixCannotSplitAnAuditAndAdvisoryLockSkipsTick() throws Exception { // This retention fixture deliberately polls a detail for a six-year-old header. - config.withLateDetailLookbackHours(24 * 365 * 10); + setting("late_detail_lookback_hours", 24 * 365 * 10); audit(1); new WormAuditChainer(config).chainOnce(); try(var c = owner()) @@ -512,7 +520,7 @@ class PostgresAcceptanceTest void chainerUsesUtcCutoffInChicago() throws Exception { var original = java.util.TimeZone.getDefault(); - config.withSettleHorizonSeconds(60); + setting("settle_horizon_seconds", 60); try(var connection = owner()) { sql(connection, "ALTER TABLE audit ALTER COLUMN timestamp TYPE timestamp without time zone USING timestamp AT TIME ZONE 'UTC'"); @@ -543,7 +551,7 @@ class PostgresAcceptanceTest @Test void postgresLateDetailWindowAndCompleteness() throws Exception { - config.withSettleHorizonSeconds(60); + setting("settle_horizon_seconds", 60); var now = java.time.Instant.now().truncatedTo(java.time.temporal.ChronoUnit.SECONDS); var cutoff = now.minusSeconds(60); try(var connection = owner(); var statement = connection.prepareStatement( @@ -574,4 +582,16 @@ class PostgresAcceptanceTest assertThat(new WormAuditVerifier(config).verify().getBreaks()).anyMatch(b -> b.reason().contains("audit_detail 1 is not covered")); } + + /******************************************************************************* + ** The settings row (seeded by the DDL) overrides config: change it there. + *******************************************************************************/ + private void setting(String column, int value) throws Exception + { + try(var connection = owner()) + { + sql(connection, "UPDATE worm_audit_settings SET " + column + "=" + value + " WHERE id=1"); + } + config.invalidateSettings(); + } } diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/RecordIntegrationTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/RecordIntegrationTest.java index 8495da8..6019b4b 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/RecordIntegrationTest.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/RecordIntegrationTest.java @@ -55,7 +55,7 @@ class RecordIntegrationTest Map formats = Map.of( WormAuditTables.CHAIN, "Link #%s", WormAuditTables.CHECKPOINT, "Checkpoint #%s", WormAuditTables.EXPORT, "Export #%s (links %s–%s)", WormAuditTables.VERIFICATION, "Verification #%s (%s)", - WormAuditTables.PURGE, "Purge #%s (%s)"); + WormAuditTables.PURGE, "Purge #%s (%s)", WormAuditTables.SETTINGS, "WORM Audit Settings"); for(String name : WormAuditTables.ALL) { var table = instance.getTable(name); diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/SealedExportTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/SealedExportTest.java index 6f27c07..9f7bf6d 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/SealedExportTest.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/SealedExportTest.java @@ -84,7 +84,7 @@ class SealedExportTest boolean omitVersion; boolean racePut; boolean supported = true; - String echoMode = "COMPLIANCE"; + String echoMode; long shortenSeconds; int failPut; @@ -95,7 +95,7 @@ class SealedExportTest } @Override - public String put(String key, Path file, Instant until) throws Exception + public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception { if(failPut > 0 && objects.size() + 1 == failPut) { @@ -108,7 +108,7 @@ class SealedExportTest String version = omitVersion ? missingVersion : "v" + ++nextVersion; objects.put(key, Files.readAllBytes(file)); heads.put(key, new StoredObject().withKey(key).withVersionId(racePut ? "replacement" : version) - .withLockMode(echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds))); + .withLockMode(echoMode == null ? mode.name() : echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds))); return (version); } @@ -197,7 +197,7 @@ class SealedExportTest assertThat(row.getValues()).containsKeys("from_checkpoint_id", "to_checkpoint_id", "bucket_alias", "manifest_key", "manifest_sha256", "prev_manifest_sha256", "object_count", "byte_count", "lock_mode", "retain_until", "sealed_at"); assertThat(row.getValueLong("byte_count")).isEqualTo(storage.objects.values().stream().mapToLong(b -> b.length).sum()); - assertThat(row.getValueInstant("retain_until")).isEqualTo(Instant.parse("2032-10-14T02:00:00Z")); + assertThat(row.getValueInstant("retain_until")).isEqualTo(NOW.plusSeconds(30 * 86400L)); assertThat(verify(row).result().passed()).isTrue(); QRecord verification = STORE.last(WormAuditTables.VERIFICATION); assertThat(verification.getValueString("kind")).isEqualTo("SEALED_EXPORT"); @@ -224,7 +224,7 @@ class SealedExportTest { String linksKey = row.getValueString("object_prefix") + "links.jsonl"; Files.write(files.file("links.jsonl"), storage.objects.get(linksKey)); - storage.put(linksKey, files.file("links.jsonl"), row.getValueInstant("retain_until")); + storage.put(linksKey, files.file("links.jsonl"), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until")); var identical = verify(row); assertThat(identical.result().passed()).isFalse(); assertThat(identical.result().getLinksChecked()).isEqualTo(1); @@ -234,7 +234,7 @@ class SealedExportTest { String key = row.getValueString("object_prefix") + name; Files.writeString(files.file(name), "different bytes"); - String current = storage.put(key, files.file(name), row.getValueInstant("retain_until")); + String current = storage.put(key, files.file(name), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until")); var report = verify(row); assertThat(report.result().passed()).isFalse(); assertThat(report.result().getLinksChecked()).isEqualTo(1); @@ -330,9 +330,9 @@ class SealedExportTest assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).contains("does not support Object Lock"); assertThat(storage.objects).isEmpty(); storage.supported = true; - storage.echoMode = "GOVERNANCE"; - assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo COMPLIANCE"); storage.echoMode = "COMPLIANCE"; + assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo GOVERNANCE"); + storage.echoMode = null; storage.shortenSeconds = 1; assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue(); storage.shortenSeconds = 0; @@ -358,9 +358,9 @@ class SealedExportTest storage.objects.put(key, "tampered".getBytes(StandardCharsets.UTF_8)); assertThat(findings(first)).contains("object links.jsonl hash does not match the manifest"); storage.objects.put(key, original); - storage.heads.get(key).withLockMode("GOVERNANCE"); - assertThat(findings(first)).contains("lacks the recorded COMPLIANCE retention"); storage.heads.get(key).withLockMode("COMPLIANCE"); + assertThat(findings(first)).contains("lacks the recorded GOVERNANCE retention"); + storage.heads.get(key).withLockMode(first.getValueString("lock_mode")); audit(2); QRecord second = seal(); tamper(WormAuditTables.EXPORT, first.getValueLong("id"), "manifest_sha256", "f".repeat(64)); diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditHomeTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditHomeTest.java index 20c0572..4e00b77 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditHomeTest.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditHomeTest.java @@ -84,7 +84,7 @@ class WormAuditHomeTest assertThat(data.oldestAudit()).isNull(); assertThat(data.nextPurge()).isNull(); assertThat(data.exportAgeHours()).isNull(); - assertThat(data.warnings()).containsExactly("No signing key configured", "No trusted public keys configured", "Daily verification has never run"); + assertThat(data.warnings()).contains("No signing key configured", "No trusted public keys configured", "Daily verification has never run"); String html = WormAuditHomeView.render(data, "/app"); assertThat(html).contains("Chain status", "Export status", "Retention", "Recent purges", "Setup warnings", "No links chained yet", "No verification has run yet", "Unknown until verification runs", "No sealed exports yet", @@ -132,9 +132,9 @@ class WormAuditHomeTest VerificationRuns.save(new WormAuditVerifier(config).verify(null, now.plusSeconds(67)), "SCHEDULE", null, now.plusSeconds(68)); data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(69)); assertThat(data.openBreak()).isNull(); - assertThat(data.warnings()).isEmpty(); - assertThat(WormAuditHomeView.render(data, "/app")).doesNotContain("Setup warnings"); - assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("No open chain breaks reported"); + assertThat(data.warnings()).hasSize(2); + assertThat(WormAuditHomeView.render(data, "/app")).contains("Setup warnings", "GOVERNANCE", "30 days"); + assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Setup needs attention"); } @@ -164,8 +164,8 @@ class WormAuditHomeTest assertThat(data.exportGaps()).isEmpty(); assertThat(data.lockCapability()).isEqualTo("SUPPORTED"); assertThat(data.bucketAlias()).isEqualTo("worm_audit"); - assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("COMPLIANCE"); - assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "COMPLIANCE", "Retain until"); + assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("GOVERNANCE"); + assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "GOVERNANCE", "Retain until"); // Corruption fixture only: the real exporter never creates this hole. tamper(WormAuditTables.EXPORT, 1, "from_link_id", 3L); assertThat(WormAuditHomeData.load(config, storage, now).exportGaps()).containsExactly("1–2"); @@ -270,7 +270,7 @@ class WormAuditHomeTest assertThat(instance.getApp("evidence").getWidgets()).containsExactly(WormAuditApp.HOME_WIDGET); assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getName()).containsExactlyElementsOf(WormAuditTables.NAV); assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getLabel()) - .containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges"); + .containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges", "Settings"); assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getGridColumns()).isEqualTo(12); assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getIsCard()).isFalse(); assertThat(instance.getWidget(WormAuditApp.HEALTH_CARD_WIDGET).getIsCard()).isTrue(); @@ -300,7 +300,7 @@ class WormAuditHomeTest assertThat(data.links()).isEqualTo(1); assertThat(data.backlog()).isEqualTo(1); assertThat(data.settling()).isEqualTo(1); - assertThat(data.warnings()).containsExactly("Daily verification has never run"); + assertThat(data.warnings()).contains("Daily verification has never run"); } } @@ -325,7 +325,7 @@ class WormAuditHomeTest assertThat(data.exportLag()).isEqualTo(5); assertThat(data.backlog()).isEqualTo(2); assertThat(data.settling()).isEqualTo(1); - assertThat(data.warnings()).isEmpty(); + assertThat(data.warnings()).hasSize(2); assertThat(data.recentPurges()).extracting(r -> r.getValueString("outcome")).containsExactly("REFUSED", "PURGED"); assertThat(STORE.where(WormAuditTables.VERIFICATION, "outcome", "FAIL")).hasSize(1).allSatisfy(row -> { diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditSettingsTest.java b/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditSettingsTest.java new file mode 100644 index 0000000..8b5b511 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/wormaudit/WormAuditSettingsTest.java @@ -0,0 +1,307 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.wormaudit; + + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Set; +import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData; +import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView; +import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer; +import com.kingsrook.qbits.wormaudit.export.SealedExporter; +import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier; +import com.kingsrook.qbits.wormaudit.model.SettingsCustomizer; +import com.kingsrook.qbits.wormaudit.model.WormAuditTables; +import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses; +import com.kingsrook.qbits.wormaudit.processes.ExportProcesses; +import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.junit.jupiter.api.Test; +import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.*; +import static org.assertj.core.api.Assertions.*; + + +/******************************************************************************* + ** Editable settings, bounded cache, audited changes and historical lock modes. + *******************************************************************************/ +class WormAuditSettingsTest +{ + static final String TABLE = WormAuditTables.SETTINGS; + + /******************************************************************************* + ** Deterministic clock for the exact 30-second cache boundary. + *******************************************************************************/ + static class MutableClock extends Clock + { + Instant now = Instant.parse("2026-10-14T02:00:00Z"); + /******************************************************************************* + ** Verify get zone. + *******************************************************************************/ + public ZoneId getZone() + { + return (ZoneOffset.UTC); + } + /******************************************************************************* + ** Verify with zone. + *******************************************************************************/ + public Clock withZone(ZoneId zone) + { + return (this); + } + /******************************************************************************* + ** Verify instant. + *******************************************************************************/ + public Instant instant() + { + return (now); + } + } + + /******************************************************************************* + ** Verify editor. + *******************************************************************************/ + static void editor() + { + QContext.getQInstance().withEnforcePermissionsInCoreActions(true); + QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit", "worm_audit.settings")); + } + + /******************************************************************************* + ** Verify update. + *******************************************************************************/ + static void update(String field, java.io.Serializable value) throws Exception + { + var result = new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue(field, value)))); + assertThat(result.getRecords()).allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty()); + } + + /******************************************************************************* + ** Verify fallback overrides cache and insert only seed. + *******************************************************************************/ + @Test + void fallbackOverridesCacheAndInsertOnlySeed() throws Exception + { + var config = new WormAuditQBitConfig().withRetentionYears(8).withExportRetentionDays(45).withCheckpointEveryLinks(12); + var instance = init(config); + var clock = new MutableClock(); + var reader = new WormAuditSettings(config, clock); + assertThat(reader.read().audit_retention_years()).isEqualTo(8); + assertThat(reader.read().export_retention_days()).isEqualTo(45); + WormAuditSettings.seed(config); + editor(); + update("export_retention_days", 90); + assertThat(reader.read().export_retention_days()).isEqualTo(90); + update("export_retention_days", 120); + clock.now = clock.now.plusSeconds(29); + assertThat(reader.read().export_retention_days()).isEqualTo(90); + clock.now = clock.now.plusSeconds(1); + assertThat(reader.read().export_retention_days()).isEqualTo(120); + WormAuditSettings.seed(config); + assertThat(STORE.get(TABLE, 1).getValueInteger("export_retention_days")).isEqualTo(120); + assertThat(reader.read().checkpoint_every_links()).isEqualTo(12); + instance.getTables().remove(TABLE); + reader.invalidate(); + assertThat(reader.read().export_retention_days()).isEqualTo(45); + } + + /******************************************************************************* + ** Verify partial null row uses config and invalid persisted settings fail closed. + *******************************************************************************/ + @Test + void partialNullRowUsesConfigAndInvalidPersistedSettingsFailClosed() throws Exception + { + var config = new WormAuditQBitConfig().withExportRetentionDays(42); + init(config); + com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().insert( + new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 1).withValue("export_lock_mode", "COMPLIANCE")), false); + var reader = new WormAuditSettings(config); + assertThat(reader.read().export_retention_days()).isEqualTo(42); + assertThat(reader.read().export_lock_mode()).isEqualTo("COMPLIANCE"); + com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().update( + new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_retention_days", 0))), false); + reader.invalidate(); + assertThatThrownBy(reader::read).hasMessageContaining("Export retention days"); + } + + /******************************************************************************* + ** Verify missing physical table falls back. + *******************************************************************************/ + @Test + void missingPhysicalTableFallsBack() throws Exception + { + var config = new WormAuditQBitConfig().withExportRetentionDays(81); + var instance = RdbmsTestSupport.instance("jdbc:h2:mem:settings_missing;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", "", config, false); + QContext.init(instance, new QSession()); + assertThat(config.effectiveSettings().export_retention_days()).isEqualTo(81); + } + + /******************************************************************************* + ** Verify validation and capabilities. + *******************************************************************************/ + @Test + void validationAndCapabilities() throws Exception + { + var config = new WormAuditQBitConfig(); + var instance = init(config); + var table = instance.getTable(TABLE); + assertThat(table.getLabel()).isEqualTo("WORM Audit Settings"); + assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(AuditLevel.FIELD); + assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_UPDATE)).isTrue(); + assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_INSERT)).isFalse(); + assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_DELETE)).isFalse(); + WormAuditSettings.seed(config); + editor(); + assertThatThrownBy(() -> new DeleteAction().execute(new DeleteInput(TABLE).withPrimaryKeys(List.of(1)))).isInstanceOf(Exception.class); + assertThatThrownBy(() -> new InsertAction().execute(new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 2)))).isInstanceOf(Exception.class); + assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(new QRecord().withValue("id", 2)), false)).hasMessageContaining("one row only (ID 1)"); + assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(WormAuditSettings.defaults(config)), false)) + .hasMessageContaining("already exists"); + assertThatThrownBy(() -> new SettingsCustomizer().preDelete(new DeleteInput(TABLE), List.of(STORE.get(TABLE, 1)), false)) + .hasMessageContaining("cannot be deleted"); + for(String field : List.of("audit_retention_years", "export_retention_days", "checkpoint_every_links", "checkpoint_every_minutes", "chain_interval_seconds", "late_detail_lookback_hours")) + { + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue(field, 0))).hasMessageContaining("whole number"); + } + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("settle_horizon_seconds", -1))).hasMessageContaining("Settle horizon seconds"); + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("audit_retention_years", 101))).hasMessageContaining("100"); + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", 36501))).hasMessageContaining("36500"); + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", "1.5"))).hasMessageContaining("whole number"); + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", "NONE"))).hasMessageContaining("GOVERNANCE or COMPLIANCE"); + assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", null))).hasMessageContaining("GOVERNANCE or COMPLIANCE"); + for(String field : List.of("export_cron", "verify_cron")) + { + assertThatThrownBy(() -> new SettingsCustomizer().preUpdate(new UpdateInput(TABLE), List.of(new QRecord().withValue("id", 1).withValue(field, "bad")), false, java.util.Optional.empty())) + .hasMessageContaining("valid Quartz cron"); + } + assertThatThrownBy(() -> new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_cron", "bad"))))) + .hasMessageContaining("valid Quartz cron"); + assertThat(STORE.get(TABLE, 1).getValueString("export_cron")).isEqualTo(config.getExportCron()); + } + + /******************************************************************************* + ** Verify edit requires both permissions and produces field audit that chains. + *******************************************************************************/ + @Test + void editRequiresBothPermissionsAndProducesFieldAuditThatChains() throws Exception + { + var config = ChainVerificationTest.signed().withSettleHorizonSeconds(0); + init(config); + WormAuditSettings.seed(config); + QContext.getQInstance().withEnforcePermissionsInCoreActions(true); + QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit")); + assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit.settings"); + QContext.getQSession().setPermissions(new java.util.HashSet<>(Set.of("worm_audit.settings"))); + assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit_settings.edit"); + editor(); + update("export_lock_mode", "COMPLIANCE"); + assertThat(STORE.all("audit")).hasSize(1); + assertThat(STORE.where("auditDetail", "fieldName", "export_lock_mode")).hasSize(1); + assertThat(STORE.get(TABLE, 1).getValueInstant("modify_date")).isNotNull(); + assertThat(STORE.get(TABLE, 1).getValueString("modified_by")).isEqualTo("system"); + new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(1)); + assertThat(STORE.all(WormAuditTables.CHAIN)).hasSize(1); + assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(1)).passed()).isTrue(); + } + + /******************************************************************************* + ** Verify startup schedules and home use effective settings. + *******************************************************************************/ + @Test + void startupSchedulesAndHomeUseEffectiveSettings() throws Exception + { + var config = ChainVerificationTest.signed(); + var instance = init(config); + WormAuditSettings.seed(config); + editor(); + update("chain_interval_seconds", 45); + update("export_cron", "0 15 4 * * ?"); + update("verify_cron", "0 30 5 * * ?"); + update("audit_retention_years", 7); + assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(30); + var restarted = WormAuditTestSupport.core(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory") + .withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class)); + new WormAuditQBitProducer().withQBitConfig(config).produce(restarted); + assertThat(QContext.getQInstance()).isSameAs(instance); + instance = restarted; + QContext.init(instance, new QSession()); + assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(45); + assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 15 4 * * ?"); + assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 30 5 * * ?"); + assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC"); + audit(50); + var home = WormAuditHomeData.load(config, new SealedExportTest.MemorySealedStore(), Instant.now()); + assertThat(home.nextPurge()).isEqualTo(OLD.plusSeconds(50).atZone(ZoneOffset.UTC).plusYears(7).toInstant()); + assertThat(home.warnings()).anyMatch(w -> w.contains("differs from") && w.contains("purge interval")); + assertThat(WormAuditHomeView.render(home, "/app")).contains("GOVERNANCE", "s3:BypassGovernanceRetention", "Switch to COMPLIANCE", "30 days", "Export retention is shorter"); + } + + /******************************************************************************* + ** Verify switch affects new exports only and old checkpoint cadence remains valid. + *******************************************************************************/ + @Test + void switchAffectsNewExportsOnlyAndOldCheckpointCadenceRemainsValid() throws Exception + { + var config = ChainVerificationTest.signed().withCheckpointEveryLinks(2).withSettleHorizonSeconds(0); + init(config); + WormAuditSettings.seed(config); + var storage = new SealedExportTest.MemorySealedStore(); + audit(1); + audit(2); + Instant now = Instant.now().plusSeconds(1); + new WormAuditChainer(config).chainOnce(now); + var exporter = new SealedExporter(config, storage); + var first = exporter.run("USER", now).row(); + assertThat(first).isNotNull(); + assertThat(first.getValueString("lock_mode")).isEqualTo("GOVERNANCE"); + assertThat(first.getValueInstant("retain_until")).isEqualTo(now.truncatedTo(java.time.temporal.ChronoUnit.MILLIS).plusSeconds(30 * 86400L)); + assertThat(storage.heads.values()).allSatisfy(h -> assertThat(h.getLockMode()).isEqualTo("GOVERNANCE")); + var verifier = new SealedExportVerifier(config, storage); + assertThat(verifier.verify(first, "USER").result().passed()).isTrue(); + editor(); + update("export_lock_mode", "COMPLIANCE"); + update("checkpoint_every_links", 1); + new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(2)); + var second = exporter.run("USER", Instant.now().plusSeconds(2)).row(); + assertThat(second.getValueString("lock_mode")).isEqualTo("COMPLIANCE"); + assertThat(verifier.verify(first, "USER").result().passed()).isTrue(); + assertThat(verifier.verify(second, "USER").result().passed()).isTrue(); + assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(2)).passed()).isTrue(); + String key = first.getValueString("object_prefix") + "links.jsonl"; + storage.heads.get(key).withLockMode("COMPLIANCE"); + assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains("lacks the recorded GOVERNANCE retention")); + storage.heads.remove(key); + assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains(key) && b.reason().contains("GOVERNANCE bypass removal")); + } +} diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/demo/MemorySealedStore.java b/src/test/java/com/kingsrook/qbits/wormaudit/demo/MemorySealedStore.java index b576f28..079f48e 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/demo/MemorySealedStore.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/demo/MemorySealedStore.java @@ -57,7 +57,7 @@ public class MemorySealedStore implements SealedStore ** Refuse overwrites and retain the exact bytes. *******************************************************************************/ @Override - public String put(String key, Path file, Instant until) throws Exception + public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception { if(objects.containsKey(key)) { @@ -65,7 +65,7 @@ public class MemorySealedStore implements SealedStore } String version = "v" + (objects.size() + 1); objects.put(key, Files.readAllBytes(file)); - heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode("COMPLIANCE").withLockRetainUntil(until)); + heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode(mode.name()).withLockRetainUntil(until)); return (version); } diff --git a/src/test/java/com/kingsrook/qbits/wormaudit/demo/WormAuditDemoServer.java b/src/test/java/com/kingsrook/qbits/wormaudit/demo/WormAuditDemoServer.java index 33856b2..672a79f 100644 --- a/src/test/java/com/kingsrook/qbits/wormaudit/demo/WormAuditDemoServer.java +++ b/src/test/java/com/kingsrook/qbits/wormaudit/demo/WormAuditDemoServer.java @@ -176,6 +176,7 @@ public class WormAuditDemoServer extends AbstractQQQApplication public void seed(QInstance instance) throws Exception { QContext.init(instance, new QSession()); + com.kingsrook.qbits.wormaudit.WormAuditSettings.seed(config); AuditStore store = new AuditStore(null); Instant today = Instant.now().minus(Duration.ofHours(2)); try