Compare commits

...
Author SHA1 Message Date
james.maesandClaude Opus 5.5 f14a24fdbf feat: in-app WORM Audit Settings and GOVERNANCE lock mode for sealed exports
Owner decisions 2026-10-14:
- worm_audit_settings: one audited, editable row (changeset worm-audit:10,
  seeded insert-only) holding audit retention years, export lock mode
  (GOVERNANCE default, COMPLIANCE), export retention days (30), checkpoint
  cadence, chain interval, settle horizon, late-detail lookback and the
  export/verify schedules; edits are FIELD-audited and therefore chained;
  schedules apply after restart, the rest on the next run (30 s cache);
- exporter and verifier honour the recorded lock mode and retain-until;
  the home shows the mode and warns while GOVERNANCE is in use or the
  export retention is shorter than audit retention;
- README: Settings and Switching sealed exports to COMPLIANCE; DESIGN
  addendum; screenshots of the home and the Settings record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 12:23:52 -05:00
james.maes 0fb9e8912c Merge pull request #9 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:46:49 -05:00
james.maesandClaude Opus 5.5 626e153841 ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:08:11 -05:00
james.maes 85de1daf79 Merge pull request #8 from QRun-IO/feature/app-home
feat: WORM Audit app home, declared alerts and record screens
2026-10-14 07:09:35 -05:00
james.maes be26a370ac Merge pull request #7 from QRun-IO/feature/sealed-exports
feat: sealed exports to S3 Object Lock and the export verifier
2026-10-14 07:09:21 -05:00
james.maes 3c6e7d57a2 Merge pull request #6 from QRun-IO/feature/chain
feat: append-only audit tables, hash chain, checkpoints and verification
2026-10-14 07:09:09 -05:00
44 changed files with 1231 additions and 92 deletions

No files matched your search

+1 -1
View File
@@ -1,7 +1,7 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
+4
View File
@@ -6,6 +6,10 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
### Added
- RC.2 settings: editable, FIELD-audited singleton with insert-only defaults, validation, separate edit permission and a 30-second effective-settings cache. Schedules load at startup.
- GOVERNANCE exports with 30-day default retention; per-export retention verification, administrator-bypass diagnostics, home warnings and documented COMPLIANCE transition.
- New `worm-audit:10` settings migration; historical changesets remain unchanged.
- Phase (c): configurable WORM Audit app home, full-width summary widget and optional host health card; permission-aware navigation and content-free, responsive light/dark summaries.
- Chain backlog/settling counts, verification and export record links, sealed-range gaps/lag, retention eligibility and recent purge/refusal history.
- PostgreSQL guard/ownership/purge-function diagnostics, signing/trust warnings, tri-state Object Lock capability and stale daily-verification warnings.
+68 -11
View File
@@ -3,8 +3,8 @@
Phases (a)–(c) provide PostgreSQL append-only enforcement, an asynchronous audit hash chain,
Ed25519 checkpoints, live and sealed-export verification actions, and daily jobs with core QAlerts signals.
It hashes exactly the audit values core stored; it does not read application records or
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and COMPLIANCE
Object Lock. The WORM Audit app home summarizes chain health, exports, retention and setup.
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and configurable
Object Lock (GOVERNANCE for 30 days by default). The WORM Audit app home summarizes chain health, exports, retention and setup.
Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement;
memory/H2 support tamper-evidence tests without database enforcement.
@@ -12,8 +12,9 @@ memory/H2 support tamper-evidence tests without database enforcement.
## Tables and permissions
All sidecar tables use the configured host backend, snake_case fields, BIGINT identifiers,
record sections, and full table-name permissions. They do not audit themselves. The app lists
Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that order.
record sections, and full table-name permissions. Immutable sidecars do not audit themselves.
The editable Settings singleton uses an INTEGER ID and FIELD-level core audit. The app lists
Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges, and Settings in that order.
| Table | Purpose | Permission |
|---|---|---|
@@ -22,18 +23,19 @@ Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that orde
| `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` |
| `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` |
| `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` |
| `worm_audit_settings` | Editable singleton; every edit is audited and chained | `worm_audit_settings.read`; editing needs `worm_audit_settings.edit` **and** `worm_audit.settings` |
`worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action.
It also authorizes **Verify sealed export** on selected export records. `worm_audit.export`
authorizes **Export now** on the export table. Processes are table/record actions or schedules;
none is a navigation entry.
Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities
Insert/update/delete capabilities are absent on immutable sidecar tables; Settings allows updates only; update/delete capabilities
are removed from all four core audit tables. Internal appends use an explicit permission bypass.
Checkpoint, export and verification record screens have a bounded chain-link child list;
range endpoints and export/checkpoint references link to their records.
The table metadata customizer can configure host ESB publications. Quick Search registration
is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables.
is host-owned through `QuickSearchQBitConfig.searchableTables`; add the desired tables.
Verification and export inserts use QQQ's record-change listener SPI,
so an installed ESB publisher receives those events when the host configures its destinations
on these two tables. The qbit defines no destination and configures no chain-link publication.
@@ -70,6 +72,58 @@ to verification. It does not itself break the chain. Signing requires a matching
Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints.
Never store the private key in a database row or checked-in configuration.
## Settings
Open **WORM Audit → Settings**, record **1** (`worm_audit_settings`, labelled WORM Audit Settings).
This is a standard editable QQQ table. New records and deletion are disabled; a pre-insert
check and database constraint require ID 1. Every edit uses core FIELD-level auditing, so
the change enters the audit chain. `modify_date` and `modified_by` are stamped by the server.
Keep notes free of sensitive data; notes are audited too.
| Field | Default | Validation / effect |
|---|---|---|
| `audit_retention_years` | 6 | 1–100; home next eligible purge date |
| `export_lock_mode` | GOVERNANCE | GOVERNANCE or COMPLIANCE; new exports |
| `export_retention_days` | 30 | 1–36500; new exports |
| `checkpoint_every_links` | 10000 | Positive integer; new checkpoint ranges |
| `checkpoint_every_minutes` | 60 | Positive integer; checkpoint cadence |
| `chain_interval_seconds` | 30 | Positive integer; **takes effect after restart** |
| `settle_horizon_seconds` | 60 | Nonnegative integer; chaining and verification |
| `late_detail_lookback_hours` | 24 | Positive integer; late-detail polling |
| `export_cron` | `0 0 2 * * ?` | Quartz cron, UTC; **takes effect after restart** |
| `verify_cron` | `0 0 3 * * ?` | Quartz cron, UTC; **takes effect after restart** |
| `notes` | Empty | Administrative notes |
| `modify_date`, `modified_by` | Server supplied | Last edit time and user |
Config values supply defaults for missing rows or null fields; a missing settings table also
falls back to config. Other database failures fail closed. `exportLockMode` and
`exportRetentionDays` are the corresponding config defaults. Runtime reads cache the row
for at most 30 seconds per application instance; local QQQ edits invalidate that cache.
Each run reads effective settings. Startup schedules read the existing row, or config when
absent. Historical checkpoints remain valid after changing their creation cadence.
Changeset `worm-audit:10` creates and seeds the table with `INSERT ... SELECT ... WHERE NOT
EXISTS`; rerunning it never overwrites an administrator change. The renderer seeds from the
host config. Memory hosts and the demo can call `WormAuditSettings.seed(config)` for the same
insert-only behavior during single-threaded startup.
`audit_retention_years` does **not** rewrite the database purge function. The home warns if
it differs from the compiled interval; changing that interval requires a reviewed migration.
The home also warns when export retention is shorter than audit retention and while
GOVERNANCE permits administrator bypass deletion. The runtime role receives SELECT, INSERT
and UPDATE on Settings, with no DELETE or TRUNCATE grant and no append-only triggers.
## Switching sealed exports to COMPLIANCE
Edit `export_lock_mode` to **COMPLIANCE** in Settings. New exports use COMPLIANCE once the
settings cache refreshes (within 30 seconds); existing GOVERNANCE objects keep their mode
unless an administrator explicitly extends their retention and upgrades their mode. The
verifier continues to compare each object with its export row's recorded mode and date, so
an out-of-band mode change is reported as a mismatch. The bucket default retention should
match the chosen mode and retention days, although the exporter always sends explicit values.
The settings change is audited and chained. **COMPLIANCE cannot be undone**: protected
versions cannot be deleted or have retention shortened until expiry, including by administrators.
## App home
Open `/app/worm_audit` (or `/app/admin/worm_audit` with `parentAppName="admin"`).
@@ -82,7 +136,7 @@ it never renders audit messages or field values. Record links retain the target
- Chain status: live link count, latest link time, settled unchained backlog and settling
count at the configured horizon, latest verification and first open chain break.
A subsequent full scheduled CHAIN PASS clears a reported break; a selected USER PASS does not.
- Export status: latest sealed range, COMPLIANCE retention, link/time lag, coverage holes
- Export status: current lock mode and retention days, latest sealed range and its recorded retention, link/time lag, coverage holes
between the purge anchor and sealed head, and bucket Object Lock capability. The unsealed
trailing suffix is lag, not a gap. Provider failures show UNKNOWN with a value-free reason.
- Retention: configured years, purge anchor, oldest live audit and its next eligible purge date.
@@ -207,8 +261,11 @@ Install qbit-object-storage and configure `exportBucketAlias` (default `worm_aud
`BucketRole.BACKUP` bucket. **Object Lock must be enabled when the bucket is created**.
The exporter requires the bucket's `OBJECT_LOCK` capability to be `SUPPORTED`; run the
object-storage read-only capability probe during setup. Unknown, partial and unsupported
capabilities all refuse sealing. Every object receives explicit **COMPLIANCE** retention
until the sealing time plus `retentionYears` (six by default), checked by HEAD after PUT.
capabilities all refuse sealing. Every object receives the effective `export_lock_mode` and retention until sealing time plus
`export_retention_days` (GOVERNANCE and 30 days by default), checked by HEAD after PUT. The
export row records both values. Verification checks that historical mode and retain-until,
regardless of subsequent settings changes. Missing GOVERNANCE objects are reported as breaks
with a possible administrator bypass-removal explanation.
The runtime IAM policy needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
`s3:GetObjectRetention`, `s3:PutObjectRetention`, and
@@ -284,8 +341,8 @@ $env:CI = 'true'
./mvnw.cmd -o verify
```
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks COMPLIANCE
retention, denied version deletion, overwrite version preservation, export verification,
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks GOVERNANCE
with 30-day retention and a COMPLIANCE export, denied version deletion (without bypass for GOVERNANCE), overwrite version preservation, export verification,
exact live-tamper findings through the record action, and refusal of a bucket without lock:
```powershell
+34
View File
@@ -228,6 +228,10 @@ installed the alerts go to the log.
## 6. Sealed exports
### 6.1 Export job
**Default superseded by Owner decisions (2026-10-14) below:** lock mode and export retention
now come from Settings, defaulting to GOVERNANCE and 30 days. The original decision text is
retained for history.
Daily at 02:00 (configurable), and by the table action "Export now" on `worm_audit_export`:
1. Range: links after the last export's `to_link_id`, up to the latest checkpoint. Exports are
contiguous by construction. The app home flags a gap (a range not covered) and lag.
@@ -386,3 +390,33 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
These are owner-run (Praetor and cluster-gitops are applied by the owner). The qbit only
documents them.
## Owner decisions (2026-10-14)
The 2026-10-12 decisions above remain the historical design. These decisions supersede the
COMPLIANCE/six-year export default in §6.1 and the corresponding default assumptions in
§§2, 4.5, 9, 11, 12 and the §14 infrastructure proposal.
- Settings live in the application as the standard QQQ `worm_audit_settings` table, record
ID 1, labelled WORM Audit Settings and navigated as Settings after Purges. It allows edits,
has no append-only triggers, and uses FIELD-level core audit so every change is chained.
Reading needs `worm_audit_settings.read`; editing additionally needs
`worm_audit_settings.edit` and `worm_audit.settings`. New IDs and deletion are refused.
- Defaults are seeded insert-only in new changeset `worm-audit:10`; existing changesets are
unchanged. An administrator's edits survive subsequent seeding. Config supplies defaults
when a row/field/table is absent. Runtime reads cache at most 30 seconds, scoped to the
QInstance. Chain interval and UTC Quartz export/verify schedules take effect after restart.
- kof22 starts with **GOVERNANCE, 30 days** for sealed exports. An administrator with
`s3:BypassGovernanceRetention` can remove those copies. This is deliberately reversible;
it does not provide COMPLIANCE's irreversible retention. Runtime credentials still receive
no delete or bypass permission. Bucket defaults should match the selected settings.
- Changing Settings to COMPLIANCE affects new exports. Existing GOVERNANCE versions retain
their original mode unless an administrator extends/upgrades them. Verification checks
the catalog's recorded mode and date, so external mode changes are reported as mismatches.
COMPLIANCE cannot be undone. The settings change itself is audited and chained.
- Audit retention defaults to six years independently of export retention. The home warns
about a shorter export horizon and a mismatch with the purge function's compiled interval.
Updating the setting does not modify the purge function; that still requires a migration.
The README Settings table specifies all fields, defaults, bounds and timing. Signing keys,
public-key pins, bucket alias, prefix and backend remain host configuration, never settings.
+27
View File
@@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
-- changeset worm-audit:10
-- Editable singleton: deliberately excluded from append-only triggers.
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
id integer PRIMARY KEY CHECK (id = 1),
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
notes text,
modify_date timestamptz NOT NULL DEFAULT now(),
modified_by text NOT NULL DEFAULT current_user
);
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM "app";
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO "app";
INSERT INTO public.worm_audit_settings
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
export_cron, verify_cron)
SELECT 1, 6, 'GOVERNANCE', 30, 10000, 60, 30, 60, 24, '0 0 2 * * ?', '0 0 3 * * ?'
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
+9
View File
@@ -1,5 +1,14 @@
# WORM Audit screenshots
## RC.2 settings captures (2026-10-14)
Home (GOVERNANCE, 30-day exports) and the WORM Audit Settings record at 390 x 844 and 1440 x 1000,
light and dark, captured with Playwright against `WormAuditDemoServer` on port 8099:
`worm-audit-home-{390,1440}-{light,dark}.jpg` (replaced) and
`worm-audit-settings-{390,1440}-{light,dark}.jpg` (new). Overflow check on all eight: no page
horizontal overflow, no inner horizontal scrollers, no double scrollbars.
## Earlier captures
Refreshed after record-screen polish **2026-10-13 (America/Chicago)** from the rebuilt test-scope demo using
packaged Next UI 1.0.0-RC.14 and connected Edge through the browser screenshot
and viewport tools. All files are JPEG viewport captures, below 300 KB each.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

@@ -45,6 +45,9 @@ public class WormAuditQBitConfig implements QBitConfig
{
}
private String exportLockMode = "GOVERNANCE";
private int exportRetentionDays = 30;
private final WormAuditSettings settings = new WormAuditSettings(this);
private String appName = "worm_audit";
private String appLabel = "WORM Audit";
private String parentAppName;
@@ -70,6 +73,63 @@ public class WormAuditQBitConfig implements QBitConfig
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
/*******************************************************************************
** Read exportLockMode.
*******************************************************************************/
public String getExportLockMode()
{
return (exportLockMode);
}
/*******************************************************************************
** Configure exportLockMode.
*******************************************************************************/
public WormAuditQBitConfig withExportLockMode(String exportLockMode)
{
this.exportLockMode = exportLockMode;
return (this);
}
/*******************************************************************************
** Read exportRetentionDays.
*******************************************************************************/
public int getExportRetentionDays()
{
return (exportRetentionDays);
}
/*******************************************************************************
** Configure exportRetentionDays.
*******************************************************************************/
public WormAuditQBitConfig withExportRetentionDays(int exportRetentionDays)
{
this.exportRetentionDays = exportRetentionDays;
return (this);
}
/*******************************************************************************
** Read effective settings through the instance-scoped 30-second cache.
*******************************************************************************/
@JsonIgnore
public WormAuditSettings.Values effectiveSettings()
{
return (settings.read());
}
/*******************************************************************************
** Discard this instance's cached row after a successful local settings write.
*******************************************************************************/
public void invalidateSettings()
{
settings.invalidate();
}
/*******************************************************************************
** Read the app name.
*******************************************************************************/
@@ -527,6 +587,7 @@ public class WormAuditQBitConfig implements QBitConfig
{
throw (new IllegalArgumentException("Export bucket, prefix and schedule must be configured"));
}
WormAuditSettings.validate(WormAuditSettings.defaults(this));
if(backendName == null || backendName.isBlank() || retentionYears < 1 || chainIntervalSeconds < 1
|| lateDetailLookbackHours < 1 || settleHorizonSeconds < 0 || batchSize < 1 || checkpointEveryLinks < 1 || checkpointEveryMinutes < 1)
{
@@ -32,6 +32,7 @@ import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
import com.kingsrook.qqq.backend.core.alerts.AlertDefaultMonitor;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -182,21 +183,31 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList").withGridColumns(12)
.withCodeReference(new QCodeReference(LinkRangeWidget.class)).withDefaultValue("parentTable", name));
}
for(var process : WormAuditProcesses.processes(qBitConfig))
List<QTableMetaData> producedTables = output.getEach(QTableMetaData.class);
output.addSelfToInstance(qInstance);
output = new MetaDataProducerMultiOutput();
var previousInstance = QContext.getQInstance();
var previousSession = QContext.getQSession();
var previousTransaction = QContext.getQBackendTransaction();
try
{
process.setSourceQBitName(qBit.getName());
output.add(process);
// Production is still assembling the host metadata. Do not validate it mid-build.
QContext.setQInstance(qInstance);
QContext.setQSession(new com.kingsrook.qqq.backend.core.model.session.QSession());
QContext.setTransaction(null);
addProcesses(output, qBit);
}
for(var process : ExportProcesses.processes(qBitConfig))
finally
{
process.setSourceQBitName(qBit.getName());
output.add(process);
QContext.setQInstance(previousInstance);
QContext.setQSession(previousSession);
QContext.setTransaction(previousTransaction);
}
if(qInstance.getSchedulers() == null || !qInstance.getSchedulers().containsKey(qBitConfig.getSchedulerName()))
{
qInstance.addScheduler(new QuartzSchedulerMetaData().withName(qBitConfig.getSchedulerName()));
}
output.add(WormAuditApp.app(qBitConfig, output.getEach(QTableMetaData.class)));
output.add(WormAuditApp.app(qBitConfig, producedTables));
for(QWidgetMetaData widget : WormAuditApp.widgets())
{
output.add(widget);
@@ -224,6 +235,24 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
}
/*******************************************************************************
** Build schedules from settings available at producer startup.
*******************************************************************************/
private void addProcesses(MetaDataProducerMultiOutput output, QBitMetaData qBit)
{
for(var process : WormAuditProcesses.processes(qBitConfig))
{
process.setSourceQBitName(qBit.getName());
output.add(process);
}
for(var process : ExportProcesses.processes(qBitConfig))
{
process.setSourceQBitName(qBit.getName());
output.add(process);
}
}
/*******************************************************************************
** RC.5 compatibility: replace with a direct call when the qbit builds on RC.6 (qqq#1141, #1163).
*******************************************************************************/
@@ -0,0 +1,246 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.sql.SQLException;
import java.time.Clock;
import java.time.Instant;
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import org.quartz.CronExpression;
/*******************************************************************************
** Singleton settings merged over host defaults. Only absent rows/tables fall
** back; database outages and invalid persisted settings fail closed.
*******************************************************************************/
public class WormAuditSettings
{
private final WormAuditQBitConfig config;
private final Clock clock;
private QInstance instance;
private QRecord cached;
private Instant expires = Instant.MIN;
/*******************************************************************************
** An immutable snapshot for one run.
*******************************************************************************/
public record Values(int audit_retention_years,
String export_lock_mode,
int export_retention_days,
int checkpoint_every_links,
int checkpoint_every_minutes,
int chain_interval_seconds,
int settle_horizon_seconds,
int late_detail_lookback_hours,
String export_cron,
String verify_cron)
{
}
/*******************************************************************************
** Use the production clock.
*******************************************************************************/
public WormAuditSettings(WormAuditQBitConfig config)
{
this(config, Clock.systemUTC());
}
/*******************************************************************************
** Inject a clock for cache boundary tests.
*******************************************************************************/
public WormAuditSettings(WormAuditQBitConfig config, Clock clock)
{
this.config = config;
this.clock = clock;
}
/*******************************************************************************
** Invalidate after a local edit; other application instances refresh within 30 seconds.
*******************************************************************************/
public synchronized void invalidate()
{
expires = Instant.MIN;
}
/*******************************************************************************
** Seed the singleton only when absent; never overwrite an administrator edit.
*******************************************************************************/
public static void seed(WormAuditQBitConfig config) throws com.kingsrook.qqq.backend.core.exceptions.QException
{
var store = new AuditStore(null);
if(store.get(WormAuditTables.SETTINGS, 1) == null)
{
store.insert(WormAuditTables.SETTINGS, java.util.List.of(defaults(config)));
}
config.invalidateSettings();
}
/*******************************************************************************
** Read at most once per 30 seconds, isolated by QInstance identity.
*******************************************************************************/
public synchronized Values read()
{
QInstance current = QContext.getQInstance();
if(current != instance || !clock.instant().isBefore(expires))
{
QRecord row = null;
if(current != null && current.getTable(WormAuditTables.SETTINGS) != null)
{
try
{
row = new AuditStore(null).get(WormAuditTables.SETTINGS, 1);
}
catch(Exception e)
{
if(!missingTable(e))
{
throw (new IllegalStateException("Cannot read WORM Audit Settings", e));
}
}
}
cached = row;
instance = current;
expires = row == null ? Instant.MIN : clock.instant().plusSeconds(30);
}
QRecord row = defaults(config);
if(cached != null)
{
for(String field : row.getValues().keySet())
{
if(cached.getValue(field) != null)
{
row.setValue(field, cached.getValue(field));
}
}
}
validate(row);
return (new Values(row.getValueInteger("audit_retention_years"),
row.getValueString("export_lock_mode"),
row.getValueInteger("export_retention_days"),
row.getValueInteger("checkpoint_every_links"),
row.getValueInteger("checkpoint_every_minutes"),
row.getValueInteger("chain_interval_seconds"),
row.getValueInteger("settle_horizon_seconds"),
row.getValueInteger("late_detail_lookback_hours"),
row.getValueString("export_cron"),
row.getValueString("verify_cron")));
}
/*******************************************************************************
** Build insert-only seed values from the host defaults.
*******************************************************************************/
public static QRecord defaults(WormAuditQBitConfig config)
{
return (new QRecord().withValue("id", 1)
.withValue("audit_retention_years", config.getRetentionYears())
.withValue("export_lock_mode", config.getExportLockMode())
.withValue("export_retention_days", config.getExportRetentionDays())
.withValue("checkpoint_every_links", config.getCheckpointEveryLinks())
.withValue("checkpoint_every_minutes", config.getCheckpointEveryMinutes())
.withValue("chain_interval_seconds", config.getChainIntervalSeconds())
.withValue("settle_horizon_seconds", config.getSettleHorizonSeconds())
.withValue("late_detail_lookback_hours", config.getLateDetailLookbackHours())
.withValue("export_cron", config.getExportCron())
.withValue("verify_cron", config.getVerifyCron()));
}
/*******************************************************************************
** Validate supplied fields; partial updates leave omitted fields untouched.
*******************************************************************************/
public static void validate(QRecord row)
{
if(row.getValue("id") != null && !Integer.valueOf(1).equals(row.getValueInteger("id")))
{
throw (new IllegalArgumentException("WORM Audit Settings has one row only (ID 1)."));
}
range(row, "audit_retention_years", "Audit retention years", 1, 100);
range(row, "export_retention_days", "Export retention days", 1, 36500);
range(row, "checkpoint_every_links", "Checkpoint every links", 1, 2147483647);
range(row, "checkpoint_every_minutes", "Checkpoint every minutes", 1, 2147483647);
range(row, "chain_interval_seconds", "Chain interval seconds", 1, 2147483647);
range(row, "settle_horizon_seconds", "Settle horizon seconds", 0, 2147483647);
range(row, "late_detail_lookback_hours", "Late detail lookback hours", 1, 2147483647);
if(row.getValues().containsKey("export_lock_mode") && !java.util.Arrays.asList("GOVERNANCE", "COMPLIANCE").contains(row.getValueString("export_lock_mode")))
{
throw (new IllegalArgumentException("Export lock mode must be GOVERNANCE or COMPLIANCE."));
}
for(String field : java.util.List.of("export_cron", "verify_cron"))
{
if(row.getValues().containsKey(field) && (row.getValueString(field) == null || !CronExpression.isValidExpression(row.getValueString(field))))
{
throw (new IllegalArgumentException((field.equals("export_cron") ? "Export" : "Verify") + " schedule must be a valid Quartz cron expression (UTC)."));
}
}
}
/*******************************************************************************
** Reject null, fractional and out-of-range numbers with a plain field label.
*******************************************************************************/
private static void range(QRecord row, String field, String label, int minimum, int maximum)
{
if(!row.getValues().containsKey(field))
{
return;
}
try
{
long value = Long.parseLong(String.valueOf(row.getValue(field)));
if(value >= minimum && value <= maximum)
{
return;
}
}
catch(NumberFormatException ignored)
{
// Use the same clear message for invalid numbers and invalid ranges.
}
throw (new IllegalArgumentException(label + " must be a whole number from " + minimum + " to " + maximum + "."));
}
/*******************************************************************************
** Recognize PostgreSQL/H2 missing-table errors without hiding other failures.
*******************************************************************************/
private static boolean missingTable(Throwable error)
{
for(Throwable cause = error; cause != null; cause = cause.getCause())
{
if(cause instanceof SQLException sql && ("42P01".equals(sql.getSQLState()) || "42S02".equals(sql.getSQLState()) || "42S04".equals(sql.getSQLState())))
{
return (true);
}
}
return (false);
}
}
@@ -49,6 +49,11 @@ public final class PostgresSetupWarnings
public static List<String> load(QBackendTransaction transaction, WormAuditQBitConfig config)
{
List<String> warnings = new ArrayList<>();
int years = config.effectiveSettings().audit_retention_years();
if(years != config.getRetentionYears())
{
warnings.add("Audit retention setting differs from the configured database purge interval (" + config.getRetentionYears() + " years). A reviewed migration is required.");
}
if(!(transaction instanceof RDBMSTransaction rdbms))
{
return (warnings);
@@ -60,8 +65,10 @@ public final class PostgresSetupWarnings
{
return (warnings);
}
// The live function is authoritative on PostgreSQL, rather than the renderer defaults.
warnings.clear();
List<String> tables = new ArrayList<>(config.getCoreAuditTableNames().values());
tables.addAll(WormAuditTables.ALL);
tables.addAll(WormAuditTables.ALL.stream().filter(t -> !WormAuditTables.SETTINGS.equals(t)).toList());
for(String table : tables)
{
for(String trigger : List.of("worm_audit_row_guard", "worm_audit_truncate_guard"))
@@ -93,6 +100,17 @@ public final class PostgresSetupWarnings
}
}
}
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT pg_get_functiondef(to_regprocedure('public.worm_audit_purge(timestamp with time zone)'))"))
{
if(result.next() && result.getString(1) != null)
{
var interval = java.util.regex.Pattern.compile("(?i)interval\\s+'([0-9]+) years'").matcher(result.getString(1));
if(interval.find() && Integer.parseInt(interval.group(1)) != years)
{
warnings.add("Audit retention setting differs from the purge function's compiled interval (" + interval.group(1) + " years). A reviewed migration is required.");
}
}
}
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT to_regprocedure('public.worm_audit_purge(timestamp with time zone)') IS NOT NULL"))
{
result.next();
@@ -88,7 +88,8 @@ public final class WormAuditApp
for(String name : WormAuditTables.NAV)
{
QTableMetaData table = tables.stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table);
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(WormAuditTables.navigationLabel(name)),
WormAuditTables.SETTINGS.equals(name) ? table.clone().withLabel("Settings") : table);
}
return (app);
}
@@ -54,7 +54,7 @@ import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, long settling, int settleSeconds,
QRecord lastVerification, QRecord openBreak, QRecord lastExport, long exportLag, Long exportAgeHours,
List<String> exportGaps, String bucketAlias, String lockCapability, String lockReason,
int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
String exportLockMode, int exportRetentionDays, int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
List<QRecord> recentPurges, List<String> warnings)
{
/*******************************************************************************
@@ -71,7 +71,16 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
*******************************************************************************/
public static WormAuditHomeData load(WormAuditQBitConfig config, SealedStore storage, Instant now) throws QException
{
var settings = config.effectiveSettings();
List<String> warnings = new ArrayList<>();
if("GOVERNANCE".equals(settings.export_lock_mode()))
{
warnings.add("Sealed exports use GOVERNANCE mode: an administrator with s3:BypassGovernanceRetention can delete them. Switch to COMPLIANCE for write-once copies.");
}
if(now.plus(Duration.ofDays(settings.export_retention_days())).isBefore(now.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant()))
{
warnings.add("Export retention is shorter than audit retention.");
}
String capability = "UNKNOWN";
String reason = "Object Lock capability could not be confirmed";
try
@@ -113,7 +122,7 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
AuditStore store = new AuditStore(transaction);
QRecord lastLink = store.last(WormAuditTables.CHAIN);
long links = store.count(WormAuditTables.CHAIN, new QQueryFilter());
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
long backlog = store.unchainedCount(cutoff, true);
long settling = store.unchainedCount(cutoff, false);
QRecord verification = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(), "finished_at");
@@ -147,10 +156,10 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
warnings.addAll(PostgresSetupWarnings.load(transaction, config));
transaction.commit();
return (new WormAuditHomeData(links, lastLink == null ? null : lastLink.getValueInstant("chained_at"), backlog, settling,
config.getSettleHorizonSeconds(), verification, openBreak, export, lag,
settings.settle_horizon_seconds(), verification, openBreak, export, lag,
export == null || export.getValueInstant("sealed_at") == null ? null : Math.max(0, Duration.between(export.getValueInstant("sealed_at"), now).toHours()),
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, config.getRetentionYears(), anchorId == 0 ? null : anchor,
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, settings.export_lock_mode(), settings.export_retention_days(), settings.audit_retention_years(), anchorId == 0 ? null : anchor,
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
}
catch(Exception e)
{
@@ -82,6 +82,8 @@ public final class WormAuditHomeView
}
close(html);
open(html, "Export status");
row(html, "Export lock mode", data.exportLockMode());
row(html, "Export retention", data.exportRetentionDays() + " days");
QRecord export = data.lastExport();
if(export == null)
{
@@ -93,7 +95,7 @@ public final class WormAuditHomeView
row(html, "Link range", export.getValueLong("from_link_id") + "–" + export.getValueLong("to_link_id"));
row(html, "Sealed at", when(export.getValueInstant("sealed_at"), "Not recorded"));
row(html, "Retain until", when(export.getValueInstant("retain_until"), "Not recorded"));
row(html, "Lock mode", export.getValueString("lock_mode"));
row(html, "Last export lock mode", export.getValueString("lock_mode"));
}
row(html, "Export lag", data.exportLag() + " links; " + (data.exportAgeHours() == null ? "no previous export time" : data.exportAgeHours() + " h since last export"));
row(html, "Gaps in sealed ranges", data.exportGaps().isEmpty() ? (export == null ? "No sealed ranges to check" : "None") : String.join(", ", data.exportGaps()));
@@ -31,6 +31,7 @@ import java.util.TreeMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
@@ -99,7 +100,7 @@ public class WormAuditChainer
}
}
}
Counts counts = append(new AuditStore(transaction), now);
Counts counts = append(new AuditStore(transaction), now, config.effectiveSettings());
transaction.commit();
return (counts);
}
@@ -119,14 +120,14 @@ public class WormAuditChainer
/*******************************************************************************
** Append settled headers and uncovered detail chunks, then checkpoint.
*******************************************************************************/
private Counts append(AuditStore store, Instant now) throws Exception
private Counts append(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
{
QRecord tail = store.last(WormAuditTables.CHAIN);
QRecord anchor = store.anchor();
long nextId = tail == null ? anchor.getValueLong("anchor_link_id") + 1 : tail.getValueLong("id") + 1;
String previous = tail == null ? anchor.getValueString("anchor_hash") : tail.getValueString("link_hash");
List<QRecord> appended = new ArrayList<>();
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
List<QRecord> audits = store.unchained(cutoff, config.getBatchSize());
Map<Long, QRecord> details = AuditStore.index(store.in("auditDetail", "auditId", audits.stream().map(r -> r.getValueLong("id")).toList()));
Map<Long, List<QRecord>> grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId")));
@@ -149,7 +150,7 @@ public class WormAuditChainer
insertLinks(store, appended);
int linkCount = appended.size();
appended.clear();
details = AuditStore.index(store.lateDetails(cutoff, config.getLateDetailLookbackHours(), config.getBatchSize()));
details = AuditStore.index(store.lateDetails(cutoff, settings.late_detail_lookback_hours(), config.getBatchSize()));
grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId"), TreeMap::new, Collectors.toList()));
Map<Long, QRecord> headers = AuditStore.index(store.in("audit", "id", grouped.keySet()));
for(var entry : grouped.entrySet())
@@ -165,7 +166,7 @@ public class WormAuditChainer
}
}
insertLinks(store, appended);
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now), false));
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now, settings), false));
}
@@ -195,7 +196,7 @@ public class WormAuditChainer
/*******************************************************************************
** Checkpoint the uncheckpointed suffix by count or elapsed time.
*******************************************************************************/
private int checkpoint(AuditStore store, Instant now) throws Exception
private int checkpoint(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
{
if(config.getSigningKey() == null)
{
@@ -206,13 +207,13 @@ public class WormAuditChainer
int count = 0;
while(true)
{
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, config.getCheckpointEveryLinks());
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, settings.checkpoint_every_links());
if(pending.isEmpty())
{
return (count);
}
Instant since = latest == null ? pending.getFirst().getValueInstant("chained_at") : latest.getValueInstant("created_at");
if(pending.size() < config.getCheckpointEveryLinks() && now.isBefore(since.plusSeconds(config.getCheckpointEveryMinutes() * 60L)))
if(pending.size() < settings.checkpoint_every_links() && now.isBefore(since.plusSeconds(settings.checkpoint_every_minutes() * 60L)))
{
return (count);
}
@@ -102,6 +102,13 @@ public final class WormAuditDdl
{
sql = sql.replace("${" + entry.getValue() + "}", identifier(names.get(entry.getKey())));
}
var defaults = com.kingsrook.qbits.wormaudit.WormAuditSettings.defaults(config);
com.kingsrook.qbits.wormaudit.WormAuditSettings.validate(defaults);
String values = java.util.List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days", "checkpoint_every_links",
"checkpoint_every_minutes", "chain_interval_seconds", "settle_horizon_seconds", "late_detail_lookback_hours", "export_cron", "verify_cron")
.stream().map(field -> defaults.getValue(field) instanceof Number ? defaults.getValue(field).toString()
: "'" + defaults.getValueString(field).replace("'", "''") + "'").collect(java.util.stream.Collectors.joining(", "));
sql = sql.replace("${settings_values}", values);
return (sql.replace("${app_role}", identifier(config.getAppRole())).replace("${retention}", config.getRetentionYears() + " years"));
}
catch(IOException e)
@@ -152,9 +152,9 @@ public class ExportFiles implements AutoCloseable
/*******************************************************************************
** A lock echo is required even if a provider advertises the capability.
*******************************************************************************/
public static boolean locked(StoredObject object, java.time.Instant until)
public static boolean locked(StoredObject object, String mode, java.time.Instant until)
{
return (object != null && "COMPLIANCE".equals(object.getLockMode()) && object.getLockRetainUntil() != null
return (object != null && java.util.Objects.equals(mode, object.getLockMode()) && object.getLockRetainUntil() != null
&& !object.getLockRetainUntil().isBefore(until));
}
@@ -29,7 +29,6 @@ import java.time.Instant;
import com.kingsrook.qbits.objectstorage.api.Capability;
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
import com.kingsrook.qbits.objectstorage.api.ListRequest;
import com.kingsrook.qbits.objectstorage.api.LockMode;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
import com.kingsrook.qbits.objectstorage.api.Page;
@@ -84,12 +83,12 @@ public class ObjectStorageSealedStore implements SealedStore
** Stream a bounded upload and close the source even when the provider fails.
*******************************************************************************/
@Override
public String put(String key, Path file, Instant retainUntil) throws Exception
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception
{
try(InputStream input = Files.newInputStream(file))
{
return (bucket.put(PutRequest.of(key, input).withContentLength(Files.size(file))
.withObjectLock(LockMode.COMPLIANCE, retainUntil)).getVersionId());
.withObjectLock(mode, retainUntil)).getVersionId());
}
}
@@ -94,6 +94,7 @@ public class SealedExportVerifier
*******************************************************************************/
public Report verify(QRecord export, String trigger) throws QException
{
config.effectiveSettings();
VerificationResult failure = new VerificationResult();
failure.scope(0, export.getValueLong("from_link_id"), export.getValueLong("to_link_id"));
Report report;
@@ -121,22 +122,34 @@ public class SealedExportVerifier
throw (new IOException("Missing sealed object version"));
}
var head = storage.head(key);
if(head == null)
{
failure.add(null, null, "object " + key + " is missing" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
? "; possible GOVERNANCE bypass removal by an administrator" : ""));
}
if(head != null && !version.equals(head.getVersionId()))
{
failure.add(null, null, "object " + name + " was overwritten after sealing (current version " + head.getVersionId()
+ ", sealed version " + version + " is still retained under Object Lock)");
}
if(!ExportFiles.locked(head, export.getValueInstant("retain_until")))
if(!ExportFiles.locked(head, export.getValueString("lock_mode"), export.getValueInstant("retain_until")))
{
failure.add(null, null, "object " + name + " lacks the recorded COMPLIANCE retention");
failure.add(null, null, "object " + name + " lacks the recorded " + export.getValueString("lock_mode") + " retention");
}
}
for(String name : ExportFiles.NAMES)
{
try(InputStream input = storage.getVersion(export.getValueString("object_prefix") + name, (String) versions.get(name)))
String key = export.getValueString("object_prefix") + name;
try(InputStream input = storage.getVersion(key, (String) versions.get(name)))
{
Files.copy(input, files.file(name));
}
catch(Exception e)
{
failure.add(null, null, "object " + key + " sealed version cannot be read" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
? "; if removed, possible GOVERNANCE bypass removal by an administrator" : ""));
throw (new IOException("Cannot read sealed object version"));
}
}
report = inspect(files, export, new AuditStore(transaction));
for(var item : failure.getBreaks())
@@ -189,7 +202,7 @@ public class SealedExportVerifier
|| !Objects.equals(manifest.get("fromLinkId"), from) || !Objects.equals(manifest.get("toLinkId"), to)
|| !Objects.equals(manifest.get("linkCount"), to - from + 1)
|| !Objects.equals(manifest.get("createdAt"), export.getValueInstant("sealed_at").toString())
|| !"COMPLIANCE".equals(export.getValueString("lock_mode"))
|| !java.util.Set.of("GOVERNANCE", "COMPLIANCE").contains(export.getValueString("lock_mode"))
|| !Objects.equals(export.getValueString("manifest_key"), export.getValueString("object_prefix") + "manifest.json"))
{
result.add(null, null, "Manifest metadata does not match the export row");
@@ -256,7 +269,7 @@ public class SealedExportVerifier
long start = checkpoint.getValueLong("from_link_id");
long count = checkpoint.getValueLong("link_count");
if(start != cursor + 1 || end < start || end > to || count != end - start + 1
|| count > config.getCheckpointEveryLinks() || checkpointId <= lastCheckpoint)
|| count > Integer.MAX_VALUE || checkpointId <= lastCheckpoint)
{
result.add(null, null, label + "invalid or incomplete checkpoint range");
break;
@@ -243,7 +243,9 @@ public class SealedExporter
{
throw (new ExportFailure("Export refused: bucket does not support Object Lock"));
}
Instant until = now.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant();
var settings = config.effectiveSettings();
var mode = com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(settings.export_lock_mode());
Instant until = now.plus(settings.export_retention_days(), ChronoUnit.DAYS);
String prefix = config.getExportPrefix() + "/" + DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(now)
+ "/" + scope.from() + "-" + scope.to() + "/";
QRecord row;
@@ -270,7 +272,7 @@ public class SealedExporter
.withValue("bucket_alias", config.getExportBucketAlias()).withValue("object_prefix", prefix)
.withValue("manifest_key", prefix + "manifest.json").withValue("manifest_sha256", ExportFiles.sha256(files.file("manifest.json")))
.withValue("prev_manifest_sha256", previous).withValue("object_count", 3L)
.withValue("lock_mode", "COMPLIANCE").withValue("retain_until", until).withValue("sealed_at", now);
.withValue("lock_mode", mode.name()).withValue("retain_until", until).withValue("sealed_at", now);
if(!new SealedExportVerifier(config, storage).inspect(files, row, store).result().passed())
{
throw (new ExportFailure("Export refused: source does not match the signed chain"));
@@ -279,15 +281,15 @@ public class SealedExporter
Map<String, String> versions = new LinkedHashMap<>();
for(String name : ExportFiles.NAMES)
{
String version = storage.put(prefix + name, files.file(name), until);
String version = storage.put(prefix + name, files.file(name), mode, until);
if(!ExportFiles.validVersion(version))
{
throw (new ExportFailure("Export refused: object " + name + " returned no version id"));
}
var headObject = storage.head(prefix + name);
if(!ExportFiles.locked(headObject, until))
if(!ExportFiles.locked(headObject, mode.name(), until))
{
throw (new ExportFailure("Export refused: object " + name + " did not echo COMPLIANCE retention"));
throw (new ExportFailure("Export refused: object " + name + " did not echo " + mode.name() + " retention"));
}
if(!version.equals(headObject.getVersionId()))
{
@@ -50,9 +50,9 @@ public interface SealedStore
}
/*******************************************************************************
** Upload a file with COMPLIANCE retention.
** Upload a file with the selected Object Lock mode and retention.
*******************************************************************************/
String put(String key, Path file, Instant retainUntil) throws Exception;
String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception;
/*******************************************************************************
** Read current object metadata, or null for an absent object.
@@ -45,6 +45,13 @@ public class AppendOnlyGuard implements MetaDataCustomizerInterface<QTableMetaDa
@Override
public QTableMetaData customizeMetaData(QInstance instance, QTableMetaData table)
{
if(com.kingsrook.qbits.wormaudit.model.WormAuditTables.SETTINGS.equals(table.getName()))
{
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE);
table.withAuditRules(new com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules()
.withAuditLevel(com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel.FIELD));
return (table);
}
if(CORE_TABLES.contains(table.getName()) || table.getName().startsWith("worm_audit_"))
{
table.withoutCapabilities(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
@@ -0,0 +1,116 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.model;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Validate the editable singleton, including internal insert-only seeds.
*******************************************************************************/
public class SettingsCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Only the fixed singleton may be inserted by the internal seed path.
*******************************************************************************/
@Override
public List<QRecord> preInsert(InsertInput input, List<QRecord> records, boolean preview) throws QException
{
if(records.size() != 1 || !Integer.valueOf(1).equals(records.getFirst().getValueInteger("id")))
{
throw (new QUserFacingException("WORM Audit Settings has one row only (ID 1)."));
}
if(new com.kingsrook.qbits.wormaudit.chain.AuditStore(input.getTransaction()).get(WormAuditTables.SETTINGS, 1) != null)
{
throw (new QUserFacingException("WORM Audit Settings already exists. Edit record 1 instead of adding another record."));
}
return (validate(records));
}
/*******************************************************************************
** Validate partial edits without replacing the existing settings.
*******************************************************************************/
@Override
public List<QRecord> preUpdate(UpdateInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
{
PermissionsHelper.checkTablePermissionThrowing(input, TablePermissionSubType.EDIT);
return (validate(records));
}
/*******************************************************************************
** Reject deletion even for direct core calls where capability checks are off.
*******************************************************************************/
@Override
public List<QRecord> preDelete(DeleteInput input, List<QRecord> records, boolean preview) throws QException
{
throw (new QUserFacingException("WORM Audit Settings cannot be deleted."));
}
/*******************************************************************************
** Refresh local settings after persistence; remote JVMs retain the bounded cache.
*******************************************************************************/
@Override
public List<QRecord> postUpdate(UpdateInput input, List<QRecord> records, Optional<List<QRecord>> oldRecords) throws QException
{
com.kingsrook.qbits.wormaudit.processes.VerificationRuns.config().invalidateSettings();
return (records);
}
/*******************************************************************************
** Stamp the editor and time on every valid write; core writes FIELD audits.
*******************************************************************************/
private List<QRecord> validate(List<QRecord> records) throws QException
{
for(QRecord row : records)
{
try
{
WormAuditSettings.validate(row);
}
catch(IllegalArgumentException e)
{
throw (new QUserFacingException(e.getMessage()));
}
row.setValue("modify_date", Instant.now());
row.setValue("modified_by", QContext.getQSession() == null || QContext.getQSession().getUser() == null
? "system" : QContext.getQSession().getUser().getIdReference());
}
return (records);
}
}
@@ -0,0 +1,55 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit.model;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
/*******************************************************************************
** Reading uses the full table name; editing additionally needs settings access.
*******************************************************************************/
public class SettingsPermissions implements CustomPermissionChecker
{
/*******************************************************************************
** Enforce table permission and the separate settings permission.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
var session = QContext.getQSession();
boolean edit = input instanceof UpdateInput;
if(input instanceof InsertInput || input instanceof DeleteInput || session == null
|| !session.hasPermission(WormAuditTables.SETTINGS + (edit ? ".edit" : ".read"))
|| (edit && !session.hasPermission("worm_audit.settings")))
{
throw (new QPermissionDeniedException(edit ? "Editing WORM Audit Settings requires worm_audit_settings.edit and worm_audit.settings."
: "WORM Audit Settings requires read access; adding or deleting settings is not allowed."));
}
}
}
@@ -24,16 +24,21 @@ package com.kingsrook.qbits.wormaudit.model;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
@@ -51,8 +56,9 @@ public final class WormAuditTables
public static final String EXPORT = "worm_audit_export";
public static final String VERIFICATION = "worm_audit_verification";
public static final String PURGE = "worm_audit_purge";
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE);
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE);
public static final String SETTINGS = "worm_audit_settings";
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE, SETTINGS);
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE, SETTINGS);
/*******************************************************************************
** Utility class.
@@ -73,10 +79,72 @@ public final class WormAuditTables
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T object_versions:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
tables.add(table(config, VERIFICATION, "Verification", "id:L kind:S trigger:S started_at:D finished_at:D from_link_id:L to_link_id:L links_checked:L outcome:S break_count:L first_break_link_id:L first_break_audit_id:L first_break_reason:T export_id:L run_by:S"));
tables.add(table(config, PURGE, "Purge", "id:L requested_at:D requested_by:S cutoff:D retention_interval:S outcome:S reason:T audits_deleted:L details_deleted:L links_deleted:L anchor_link_id:L anchor_hash:S"));
tables.add(settings(config));
return (tables);
}
/*******************************************************************************
** Editable, FIELD-audited singleton with standard QQQ record sections.
*******************************************************************************/
public static QTableMetaData settings(WormAuditQBitConfig config)
{
QTableMetaData table = new QTableMetaData().withName(SETTINGS).withLabel("WORM Audit Settings")
.withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(SETTINGS))
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("WORM Audit Settings")
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS)
.withPermissionBaseName(SETTINGS).withCustomPermissionChecker(new QCodeReference(SettingsPermissions.class)))
.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE)
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withCustomizer(TableCustomizers.PRE_DELETE_RECORD, new QCodeReference(SettingsCustomizer.class))
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withLabel("ID").withBackendName("id").withIsEditable(false).withDefaultValue(1));
var defaults = WormAuditSettings.defaults(config);
table.withField(new QFieldMetaData("audit_retention_years", QFieldType.INTEGER).withBackendName("audit_retention_years").withLabel("Audit retention years")
.withIsRequired(true).withDefaultValue(defaults.getValue("audit_retention_years")));
table.withField(new QFieldMetaData("export_lock_mode", QFieldType.STRING).withBackendName("export_lock_mode").withLabel("Export lock mode")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_lock_mode")));
table.withField(new QFieldMetaData("export_retention_days", QFieldType.INTEGER).withBackendName("export_retention_days").withLabel("Export retention days")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_retention_days")));
table.withField(new QFieldMetaData("checkpoint_every_links", QFieldType.INTEGER).withBackendName("checkpoint_every_links").withLabel("Checkpoint every links")
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_links")));
table.withField(new QFieldMetaData("checkpoint_every_minutes", QFieldType.INTEGER).withBackendName("checkpoint_every_minutes").withLabel("Checkpoint every minutes")
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_minutes")));
table.withField(new QFieldMetaData("chain_interval_seconds", QFieldType.INTEGER).withBackendName("chain_interval_seconds").withLabel("Chain interval seconds")
.withIsRequired(true).withDefaultValue(defaults.getValue("chain_interval_seconds")));
table.withField(new QFieldMetaData("settle_horizon_seconds", QFieldType.INTEGER).withBackendName("settle_horizon_seconds").withLabel("Settle horizon seconds")
.withIsRequired(true).withDefaultValue(defaults.getValue("settle_horizon_seconds")));
table.withField(new QFieldMetaData("late_detail_lookback_hours", QFieldType.INTEGER).withBackendName("late_detail_lookback_hours").withLabel("Late detail lookback hours")
.withIsRequired(true).withDefaultValue(defaults.getValue("late_detail_lookback_hours")));
table.withField(new QFieldMetaData("export_cron", QFieldType.STRING).withBackendName("export_cron").withLabel("Export schedule (UTC)")
.withIsRequired(true).withDefaultValue(defaults.getValue("export_cron")));
table.withField(new QFieldMetaData("verify_cron", QFieldType.STRING).withBackendName("verify_cron").withLabel("Verify schedule (UTC)")
.withIsRequired(true).withDefaultValue(defaults.getValue("verify_cron")));
table.getField("export_lock_mode").withInlinePossibleValueSource(new QPossibleValueSource().withName(SETTINGS + "_lock_mode")
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(
new QPossibleValue<>("GOVERNANCE", "GOVERNANCE"), new QPossibleValue<>("COMPLIANCE", "COMPLIANCE"))));
for(String field : List.of("export_cron", "verify_cron", "chain_interval_seconds"))
{
table.getField(field).withHelpContent(new QHelpContent("Takes effect after restart. Cron schedules use UTC and Quartz syntax."));
}
table.getField("audit_retention_years").withHelpContent(new QHelpContent("Used for the next eligible purge date. The database purge interval changes only through a reviewed migration."));
table.withField(new QFieldMetaData("notes", QFieldType.TEXT).withLabel("Notes").withBackendName("notes"));
table.withField(new QFieldMetaData("modify_date", QFieldType.DATE_TIME).withLabel("Modified").withBackendName("modify_date").withIsEditable(false));
table.withField(new QFieldMetaData("modified_by", QFieldType.STRING).withLabel("Modified by").withBackendName("modified_by").withIsEditable(false));
table.withSection(new QFieldSection("retention", "Retention", new QIcon("lock"), Tier.T1,
List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days")).withGridColumns(12));
table.withSection(new QFieldSection("cadence", "Chain and checkpoints", new QIcon("link"), Tier.T2,
List.of("checkpoint_every_links", "checkpoint_every_minutes", "settle_horizon_seconds", "late_detail_lookback_hours")).withGridColumns(12));
table.withSection(new QFieldSection("schedules", "Schedules (after restart)", new QIcon("schedule"), Tier.T2,
List.of("chain_interval_seconds", "export_cron", "verify_cron")).withGridColumns(12));
table.withSection(new QFieldSection("changes", "Changes", new QIcon("edit_note"), Tier.T2,
List.of("notes", "modify_date", "modified_by")).withGridColumns(12));
return (table);
}
/*******************************************************************************
** Common metadata for a process-written table.
*******************************************************************************/
@@ -169,6 +237,7 @@ public final class WormAuditTables
{
return (switch(name)
{
case SETTINGS -> "Settings";
case CHAIN -> "Audit Chain";
case CHECKPOINT -> "Checkpoints";
case EXPORT -> "Sealed Exports";
@@ -84,6 +84,7 @@ public final class ExportProcesses
*******************************************************************************/
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
{
var settings = config.effectiveSettings();
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.EXPORT);
@@ -97,7 +98,7 @@ public final class ExportProcesses
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily sealed audit export")
.withStep(new QBackendStepMetaData().withName("export").withCode(new QCodeReference(Daily.class)))
.withPermissionRules(permission(PERMISSION)).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withCronExpression(config.getExportCron()).withCronTimeZoneId("UTC"));
.withCronExpression(settings.export_cron()).withCronTimeZoneId("UTC"));
return (List.of(export, verify, daily));
}
@@ -89,6 +89,7 @@ public final class WormAuditProcesses
*******************************************************************************/
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
{
var settings = config.effectiveSettings();
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.CHAIN);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.CHAIN);
@@ -100,11 +101,11 @@ public final class WormAuditProcesses
QProcessMetaData chain = new QProcessMetaData().withName(CHAIN).withLabel("Chain settled audit records")
.withStep(new QBackendStepMetaData().withName("chain").withCode(new QCodeReference(ChainTick.class)))
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withRepeatSeconds(config.getChainIntervalSeconds()).withInitialDelaySeconds(config.getChainIntervalSeconds()));
.withRepeatSeconds(settings.chain_interval_seconds()).withInitialDelaySeconds(settings.chain_interval_seconds()));
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily audit chain verification")
.withStep(new QBackendStepMetaData().withName("verify").withCode(new QCodeReference(Daily.class)))
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
.withCronExpression(config.getVerifyCron()).withCronTimeZoneId("UTC"));
.withCronExpression(settings.verify_cron()).withCronTimeZoneId("UTC"));
return (List.of(verify, chain, daily));
}
@@ -133,6 +133,7 @@ public class WormAuditVerifier
*******************************************************************************/
VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
{
var settings = config.effectiveSettings();
VerificationResult result = new VerificationResult(now);
if(config.getSigningKey() == null)
{
@@ -214,7 +215,7 @@ public class WormAuditVerifier
}
if(selection == null)
{
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
if(store.isRdbms())
{
store.completeness(cutoff, result);
@@ -500,7 +501,7 @@ public class WormAuditVerifier
result.add(null, null, prefix + "signature does not verify");
}
if(from < 1 || to < from || checkpoint.getValueLong("link_count") != to - from + 1
|| to - from + 1 > config.getCheckpointEveryLinks())
|| to - from + 1 > Integer.MAX_VALUE)
{
result.add(null, null, prefix + "invalid link range or count");
return;
@@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
-- changeset worm-audit:10
-- Editable singleton: deliberately excluded from append-only triggers.
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
id integer PRIMARY KEY CHECK (id = 1),
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
notes text,
modify_date timestamptz NOT NULL DEFAULT now(),
modified_by text NOT NULL DEFAULT current_user
);
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM ${app_role};
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO ${app_role};
INSERT INTO public.worm_audit_settings
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
export_cron, verify_cron)
SELECT ${settings_values}
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
@@ -78,7 +78,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
class MinioExportAcceptanceTest
{
/*******************************************************************************
** Production adapter, COMPLIANCE retention, version preservation and UI findings.
** Production adapter, GOVERNANCE and COMPLIANCE retention, version preservation and UI findings.
*******************************************************************************/
@Test
void lockedRoundTripVersionRetentionLiveTamperAndUnsupportedBucket() throws Exception
@@ -130,17 +130,17 @@ class MinioExportAcceptanceTest
{
String key = row.getValueString("object_prefix") + name;
var head = storage.head(key);
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
assertThat(Duration.between(now.atZone(java.time.ZoneOffset.UTC).plusYears(6).toInstant(), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
assertThat(head.getLockMode()).isEqualTo("GOVERNANCE");
assertThat(Duration.between(now.plus(Duration.ofDays(30)), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
String originalVersion = head.getVersionId();
assertThat(originalVersion).isNotBlank();
assertThat(ExportFiles.parse(row.getValueString("object_versions").getBytes(java.nio.charset.StandardCharsets.UTF_8)))
.containsEntry(name, originalVersion);
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)))
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion).bypassGovernanceRetention(false)))
.isInstanceOf(S3Exception.class);
byte[] original = s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray();
// S3 overwrites create new versions; they cannot replace the locked version.
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("COMPLIANCE")
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("GOVERNANCE")
.objectLockRetainUntilDate(row.getValueInstant("retain_until")), RequestBody.fromString("different replacement bytes"));
assertThat(replacement.versionId()).isNotEqualTo(originalVersion);
assertThat(s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray()).isEqualTo(original);
@@ -163,11 +163,26 @@ class MinioExportAcceptanceTest
tamper("auditDetail", 1, "newValue", "new");
audit(2);
new WormAuditChainer(config).chainOnce();
config.withExportLockMode("COMPLIANCE");
var compliance = new SealedExporter(config).run("USER", Instant.now());
assertThat(compliance.failed()).isFalse();
assertThat(compliance.row().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
assertThat(verifier.verify(compliance.row(), "USER").result().passed()).isTrue();
for(String name : ExportFiles.NAMES)
{
String key = compliance.row().getValueString("object_prefix") + name;
var head = storage.head(key);
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(head.getVersionId())))
.isInstanceOf(S3Exception.class);
}
audit(3);
new WormAuditChainer(config).chainOnce();
config.withExportBucketAlias("plain");
var failed = new SealedExporter(config).run("SCHEDULE", Instant.now());
assertThat(failed.failed()).isTrue();
assertThat(failed.message()).contains("does not support Object Lock");
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1);
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(2);
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(1);
}
finally
@@ -113,7 +113,6 @@ class PostgresAcceptanceTest
void database() throws Exception
{
config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
try(var c = owner())
{
sql(
@@ -124,6 +123,7 @@ class PostgresAcceptanceTest
c,
"DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='app') THEN"
+ " CREATE ROLE app LOGIN PASSWORD 'fixture'; END IF; END $$");
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
RdbmsTestSupport.createTables(c, instance, true);
String ddl = WormAuditDdl.postgres(config);
sql(c, ddl);
@@ -142,9 +142,17 @@ class PostgresAcceptanceTest
{
var storage = new com.kingsrook.qbits.wormaudit.demo.MemorySealedStore();
var clean = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
assertThat(clean.warnings()).containsExactly("Daily verification has never run");
assertThat(clean.warnings()).contains("Daily verification has never run").noneMatch(w -> w.contains("guard") && w.contains("settings"));
try(var connection = owner())
{
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=7 WHERE id=1");
config.invalidateSettings();
assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings())
.anyMatch(w -> w.contains("purge function's compiled interval (6 years)"));
sql(connection, WormAuditDdl.postgres(config));
assertThat(count(connection, "SELECT audit_retention_years FROM worm_audit_settings WHERE id=1")).isEqualTo(7);
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=6 WHERE id=1");
config.invalidateSettings();
sql(connection, "DROP TRIGGER worm_audit_row_guard ON audit_detail");
sql(connection, "ALTER TABLE worm_audit_chain DISABLE TRIGGER worm_audit_truncate_guard");
sql(connection, "DROP FUNCTION public.worm_audit_purge(timestamptz)");
@@ -338,7 +346,7 @@ class PostgresAcceptanceTest
void prefixCannotSplitAnAuditAndAdvisoryLockSkipsTick() throws Exception
{
// This retention fixture deliberately polls a detail for a six-year-old header.
config.withLateDetailLookbackHours(24 * 365 * 10);
setting("late_detail_lookback_hours", 24 * 365 * 10);
audit(1);
new WormAuditChainer(config).chainOnce();
try(var c = owner())
@@ -512,7 +520,7 @@ class PostgresAcceptanceTest
void chainerUsesUtcCutoffInChicago() throws Exception
{
var original = java.util.TimeZone.getDefault();
config.withSettleHorizonSeconds(60);
setting("settle_horizon_seconds", 60);
try(var connection = owner())
{
sql(connection, "ALTER TABLE audit ALTER COLUMN timestamp TYPE timestamp without time zone USING timestamp AT TIME ZONE 'UTC'");
@@ -543,7 +551,7 @@ class PostgresAcceptanceTest
@Test
void postgresLateDetailWindowAndCompleteness() throws Exception
{
config.withSettleHorizonSeconds(60);
setting("settle_horizon_seconds", 60);
var now = java.time.Instant.now().truncatedTo(java.time.temporal.ChronoUnit.SECONDS);
var cutoff = now.minusSeconds(60);
try(var connection = owner(); var statement = connection.prepareStatement(
@@ -574,4 +582,16 @@ class PostgresAcceptanceTest
assertThat(new WormAuditVerifier(config).verify().getBreaks()).anyMatch(b -> b.reason().contains("audit_detail 1 is not covered"));
}
/*******************************************************************************
** The settings row (seeded by the DDL) overrides config: change it there.
*******************************************************************************/
private void setting(String column, int value) throws Exception
{
try(var connection = owner())
{
sql(connection, "UPDATE worm_audit_settings SET " + column + "=" + value + " WHERE id=1");
}
config.invalidateSettings();
}
}
@@ -55,7 +55,7 @@ class RecordIntegrationTest
Map<String, String> formats = Map.of(
WormAuditTables.CHAIN, "Link #%s", WormAuditTables.CHECKPOINT, "Checkpoint #%s",
WormAuditTables.EXPORT, "Export #%s (links %s–%s)", WormAuditTables.VERIFICATION, "Verification #%s (%s)",
WormAuditTables.PURGE, "Purge #%s (%s)");
WormAuditTables.PURGE, "Purge #%s (%s)", WormAuditTables.SETTINGS, "WORM Audit Settings");
for(String name : WormAuditTables.ALL)
{
var table = instance.getTable(name);
@@ -84,7 +84,7 @@ class SealedExportTest
boolean omitVersion;
boolean racePut;
boolean supported = true;
String echoMode = "COMPLIANCE";
String echoMode;
long shortenSeconds;
int failPut;
@@ -95,7 +95,7 @@ class SealedExportTest
}
@Override
public String put(String key, Path file, Instant until) throws Exception
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
{
if(failPut > 0 && objects.size() + 1 == failPut)
{
@@ -108,7 +108,7 @@ class SealedExportTest
String version = omitVersion ? missingVersion : "v" + ++nextVersion;
objects.put(key, Files.readAllBytes(file));
heads.put(key, new StoredObject().withKey(key).withVersionId(racePut ? "replacement" : version)
.withLockMode(echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
.withLockMode(echoMode == null ? mode.name() : echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
return (version);
}
@@ -197,7 +197,7 @@ class SealedExportTest
assertThat(row.getValues()).containsKeys("from_checkpoint_id", "to_checkpoint_id", "bucket_alias", "manifest_key", "manifest_sha256",
"prev_manifest_sha256", "object_count", "byte_count", "lock_mode", "retain_until", "sealed_at");
assertThat(row.getValueLong("byte_count")).isEqualTo(storage.objects.values().stream().mapToLong(b -> b.length).sum());
assertThat(row.getValueInstant("retain_until")).isEqualTo(Instant.parse("2032-10-14T02:00:00Z"));
assertThat(row.getValueInstant("retain_until")).isEqualTo(NOW.plusSeconds(30 * 86400L));
assertThat(verify(row).result().passed()).isTrue();
QRecord verification = STORE.last(WormAuditTables.VERIFICATION);
assertThat(verification.getValueString("kind")).isEqualTo("SEALED_EXPORT");
@@ -224,7 +224,7 @@ class SealedExportTest
{
String linksKey = row.getValueString("object_prefix") + "links.jsonl";
Files.write(files.file("links.jsonl"), storage.objects.get(linksKey));
storage.put(linksKey, files.file("links.jsonl"), row.getValueInstant("retain_until"));
storage.put(linksKey, files.file("links.jsonl"), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
var identical = verify(row);
assertThat(identical.result().passed()).isFalse();
assertThat(identical.result().getLinksChecked()).isEqualTo(1);
@@ -234,7 +234,7 @@ class SealedExportTest
{
String key = row.getValueString("object_prefix") + name;
Files.writeString(files.file(name), "different bytes");
String current = storage.put(key, files.file(name), row.getValueInstant("retain_until"));
String current = storage.put(key, files.file(name), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
var report = verify(row);
assertThat(report.result().passed()).isFalse();
assertThat(report.result().getLinksChecked()).isEqualTo(1);
@@ -330,9 +330,9 @@ class SealedExportTest
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).contains("does not support Object Lock");
assertThat(storage.objects).isEmpty();
storage.supported = true;
storage.echoMode = "GOVERNANCE";
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo COMPLIANCE");
storage.echoMode = "COMPLIANCE";
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo GOVERNANCE");
storage.echoMode = null;
storage.shortenSeconds = 1;
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
storage.shortenSeconds = 0;
@@ -358,9 +358,9 @@ class SealedExportTest
storage.objects.put(key, "tampered".getBytes(StandardCharsets.UTF_8));
assertThat(findings(first)).contains("object links.jsonl hash does not match the manifest");
storage.objects.put(key, original);
storage.heads.get(key).withLockMode("GOVERNANCE");
assertThat(findings(first)).contains("lacks the recorded COMPLIANCE retention");
storage.heads.get(key).withLockMode("COMPLIANCE");
assertThat(findings(first)).contains("lacks the recorded GOVERNANCE retention");
storage.heads.get(key).withLockMode(first.getValueString("lock_mode"));
audit(2);
QRecord second = seal();
tamper(WormAuditTables.EXPORT, first.getValueLong("id"), "manifest_sha256", "f".repeat(64));
@@ -84,7 +84,7 @@ class WormAuditHomeTest
assertThat(data.oldestAudit()).isNull();
assertThat(data.nextPurge()).isNull();
assertThat(data.exportAgeHours()).isNull();
assertThat(data.warnings()).containsExactly("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
assertThat(data.warnings()).contains("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
String html = WormAuditHomeView.render(data, "/app");
assertThat(html).contains("Chain status", "Export status", "Retention", "Recent purges", "Setup warnings",
"No links chained yet", "No verification has run yet", "Unknown until verification runs", "No sealed exports yet",
@@ -132,9 +132,9 @@ class WormAuditHomeTest
VerificationRuns.save(new WormAuditVerifier(config).verify(null, now.plusSeconds(67)), "SCHEDULE", null, now.plusSeconds(68));
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(69));
assertThat(data.openBreak()).isNull();
assertThat(data.warnings()).isEmpty();
assertThat(WormAuditHomeView.render(data, "/app")).doesNotContain("Setup warnings");
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("No open chain breaks reported");
assertThat(data.warnings()).hasSize(2);
assertThat(WormAuditHomeView.render(data, "/app")).contains("Setup warnings", "GOVERNANCE", "30 days");
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Setup needs attention");
}
@@ -164,8 +164,8 @@ class WormAuditHomeTest
assertThat(data.exportGaps()).isEmpty();
assertThat(data.lockCapability()).isEqualTo("SUPPORTED");
assertThat(data.bucketAlias()).isEqualTo("worm_audit");
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "COMPLIANCE", "Retain until");
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("GOVERNANCE");
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "GOVERNANCE", "Retain until");
// Corruption fixture only: the real exporter never creates this hole.
tamper(WormAuditTables.EXPORT, 1, "from_link_id", 3L);
assertThat(WormAuditHomeData.load(config, storage, now).exportGaps()).containsExactly("1–2");
@@ -270,7 +270,7 @@ class WormAuditHomeTest
assertThat(instance.getApp("evidence").getWidgets()).containsExactly(WormAuditApp.HOME_WIDGET);
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getName()).containsExactlyElementsOf(WormAuditTables.NAV);
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getLabel())
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges");
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges", "Settings");
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getGridColumns()).isEqualTo(12);
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getIsCard()).isFalse();
assertThat(instance.getWidget(WormAuditApp.HEALTH_CARD_WIDGET).getIsCard()).isTrue();
@@ -300,7 +300,7 @@ class WormAuditHomeTest
assertThat(data.links()).isEqualTo(1);
assertThat(data.backlog()).isEqualTo(1);
assertThat(data.settling()).isEqualTo(1);
assertThat(data.warnings()).containsExactly("Daily verification has never run");
assertThat(data.warnings()).contains("Daily verification has never run");
}
}
@@ -325,7 +325,7 @@ class WormAuditHomeTest
assertThat(data.exportLag()).isEqualTo(5);
assertThat(data.backlog()).isEqualTo(2);
assertThat(data.settling()).isEqualTo(1);
assertThat(data.warnings()).isEmpty();
assertThat(data.warnings()).hasSize(2);
assertThat(data.recentPurges()).extracting(r -> r.getValueString("outcome")).containsExactly("REFUSED", "PURGED");
assertThat(STORE.where(WormAuditTables.VERIFICATION, "outcome", "FAIL")).hasSize(1).allSatisfy(row ->
{
@@ -0,0 +1,307 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.wormaudit;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData;
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView;
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
import com.kingsrook.qbits.wormaudit.model.SettingsCustomizer;
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.*;
import static org.assertj.core.api.Assertions.*;
/*******************************************************************************
** Editable settings, bounded cache, audited changes and historical lock modes.
*******************************************************************************/
class WormAuditSettingsTest
{
static final String TABLE = WormAuditTables.SETTINGS;
/*******************************************************************************
** Deterministic clock for the exact 30-second cache boundary.
*******************************************************************************/
static class MutableClock extends Clock
{
Instant now = Instant.parse("2026-10-14T02:00:00Z");
/*******************************************************************************
** Verify get zone.
*******************************************************************************/
public ZoneId getZone()
{
return (ZoneOffset.UTC);
}
/*******************************************************************************
** Verify with zone.
*******************************************************************************/
public Clock withZone(ZoneId zone)
{
return (this);
}
/*******************************************************************************
** Verify instant.
*******************************************************************************/
public Instant instant()
{
return (now);
}
}
/*******************************************************************************
** Verify editor.
*******************************************************************************/
static void editor()
{
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit", "worm_audit.settings"));
}
/*******************************************************************************
** Verify update.
*******************************************************************************/
static void update(String field, java.io.Serializable value) throws Exception
{
var result = new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue(field, value))));
assertThat(result.getRecords()).allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty());
}
/*******************************************************************************
** Verify fallback overrides cache and insert only seed.
*******************************************************************************/
@Test
void fallbackOverridesCacheAndInsertOnlySeed() throws Exception
{
var config = new WormAuditQBitConfig().withRetentionYears(8).withExportRetentionDays(45).withCheckpointEveryLinks(12);
var instance = init(config);
var clock = new MutableClock();
var reader = new WormAuditSettings(config, clock);
assertThat(reader.read().audit_retention_years()).isEqualTo(8);
assertThat(reader.read().export_retention_days()).isEqualTo(45);
WormAuditSettings.seed(config);
editor();
update("export_retention_days", 90);
assertThat(reader.read().export_retention_days()).isEqualTo(90);
update("export_retention_days", 120);
clock.now = clock.now.plusSeconds(29);
assertThat(reader.read().export_retention_days()).isEqualTo(90);
clock.now = clock.now.plusSeconds(1);
assertThat(reader.read().export_retention_days()).isEqualTo(120);
WormAuditSettings.seed(config);
assertThat(STORE.get(TABLE, 1).getValueInteger("export_retention_days")).isEqualTo(120);
assertThat(reader.read().checkpoint_every_links()).isEqualTo(12);
instance.getTables().remove(TABLE);
reader.invalidate();
assertThat(reader.read().export_retention_days()).isEqualTo(45);
}
/*******************************************************************************
** Verify partial null row uses config and invalid persisted settings fail closed.
*******************************************************************************/
@Test
void partialNullRowUsesConfigAndInvalidPersistedSettingsFailClosed() throws Exception
{
var config = new WormAuditQBitConfig().withExportRetentionDays(42);
init(config);
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().insert(
new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 1).withValue("export_lock_mode", "COMPLIANCE")), false);
var reader = new WormAuditSettings(config);
assertThat(reader.read().export_retention_days()).isEqualTo(42);
assertThat(reader.read().export_lock_mode()).isEqualTo("COMPLIANCE");
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().update(
new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_retention_days", 0))), false);
reader.invalidate();
assertThatThrownBy(reader::read).hasMessageContaining("Export retention days");
}
/*******************************************************************************
** Verify missing physical table falls back.
*******************************************************************************/
@Test
void missingPhysicalTableFallsBack() throws Exception
{
var config = new WormAuditQBitConfig().withExportRetentionDays(81);
var instance = RdbmsTestSupport.instance("jdbc:h2:mem:settings_missing;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", "", config, false);
QContext.init(instance, new QSession());
assertThat(config.effectiveSettings().export_retention_days()).isEqualTo(81);
}
/*******************************************************************************
** Verify validation and capabilities.
*******************************************************************************/
@Test
void validationAndCapabilities() throws Exception
{
var config = new WormAuditQBitConfig();
var instance = init(config);
var table = instance.getTable(TABLE);
assertThat(table.getLabel()).isEqualTo("WORM Audit Settings");
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(AuditLevel.FIELD);
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_UPDATE)).isTrue();
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_INSERT)).isFalse();
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_DELETE)).isFalse();
WormAuditSettings.seed(config);
editor();
assertThatThrownBy(() -> new DeleteAction().execute(new DeleteInput(TABLE).withPrimaryKeys(List.of(1)))).isInstanceOf(Exception.class);
assertThatThrownBy(() -> new InsertAction().execute(new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 2)))).isInstanceOf(Exception.class);
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(new QRecord().withValue("id", 2)), false)).hasMessageContaining("one row only (ID 1)");
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(WormAuditSettings.defaults(config)), false))
.hasMessageContaining("already exists");
assertThatThrownBy(() -> new SettingsCustomizer().preDelete(new DeleteInput(TABLE), List.of(STORE.get(TABLE, 1)), false))
.hasMessageContaining("cannot be deleted");
for(String field : List.of("audit_retention_years", "export_retention_days", "checkpoint_every_links", "checkpoint_every_minutes", "chain_interval_seconds", "late_detail_lookback_hours"))
{
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue(field, 0))).hasMessageContaining("whole number");
}
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("settle_horizon_seconds", -1))).hasMessageContaining("Settle horizon seconds");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("audit_retention_years", 101))).hasMessageContaining("100");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", 36501))).hasMessageContaining("36500");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", "1.5"))).hasMessageContaining("whole number");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", "NONE"))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", null))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
for(String field : List.of("export_cron", "verify_cron"))
{
assertThatThrownBy(() -> new SettingsCustomizer().preUpdate(new UpdateInput(TABLE), List.of(new QRecord().withValue("id", 1).withValue(field, "bad")), false, java.util.Optional.empty()))
.hasMessageContaining("valid Quartz cron");
}
assertThatThrownBy(() -> new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_cron", "bad")))))
.hasMessageContaining("valid Quartz cron");
assertThat(STORE.get(TABLE, 1).getValueString("export_cron")).isEqualTo(config.getExportCron());
}
/*******************************************************************************
** Verify edit requires both permissions and produces field audit that chains.
*******************************************************************************/
@Test
void editRequiresBothPermissionsAndProducesFieldAuditThatChains() throws Exception
{
var config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
init(config);
WormAuditSettings.seed(config);
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit"));
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit.settings");
QContext.getQSession().setPermissions(new java.util.HashSet<>(Set.of("worm_audit.settings")));
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit_settings.edit");
editor();
update("export_lock_mode", "COMPLIANCE");
assertThat(STORE.all("audit")).hasSize(1);
assertThat(STORE.where("auditDetail", "fieldName", "export_lock_mode")).hasSize(1);
assertThat(STORE.get(TABLE, 1).getValueInstant("modify_date")).isNotNull();
assertThat(STORE.get(TABLE, 1).getValueString("modified_by")).isEqualTo("system");
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(1));
assertThat(STORE.all(WormAuditTables.CHAIN)).hasSize(1);
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(1)).passed()).isTrue();
}
/*******************************************************************************
** Verify startup schedules and home use effective settings.
*******************************************************************************/
@Test
void startupSchedulesAndHomeUseEffectiveSettings() throws Exception
{
var config = ChainVerificationTest.signed();
var instance = init(config);
WormAuditSettings.seed(config);
editor();
update("chain_interval_seconds", 45);
update("export_cron", "0 15 4 * * ?");
update("verify_cron", "0 30 5 * * ?");
update("audit_retention_years", 7);
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(30);
var restarted = WormAuditTestSupport.core(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
new WormAuditQBitProducer().withQBitConfig(config).produce(restarted);
assertThat(QContext.getQInstance()).isSameAs(instance);
instance = restarted;
QContext.init(instance, new QSession());
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(45);
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 15 4 * * ?");
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 30 5 * * ?");
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC");
audit(50);
var home = WormAuditHomeData.load(config, new SealedExportTest.MemorySealedStore(), Instant.now());
assertThat(home.nextPurge()).isEqualTo(OLD.plusSeconds(50).atZone(ZoneOffset.UTC).plusYears(7).toInstant());
assertThat(home.warnings()).anyMatch(w -> w.contains("differs from") && w.contains("purge interval"));
assertThat(WormAuditHomeView.render(home, "/app")).contains("GOVERNANCE", "s3:BypassGovernanceRetention", "Switch to COMPLIANCE", "30 days", "Export retention is shorter");
}
/*******************************************************************************
** Verify switch affects new exports only and old checkpoint cadence remains valid.
*******************************************************************************/
@Test
void switchAffectsNewExportsOnlyAndOldCheckpointCadenceRemainsValid() throws Exception
{
var config = ChainVerificationTest.signed().withCheckpointEveryLinks(2).withSettleHorizonSeconds(0);
init(config);
WormAuditSettings.seed(config);
var storage = new SealedExportTest.MemorySealedStore();
audit(1);
audit(2);
Instant now = Instant.now().plusSeconds(1);
new WormAuditChainer(config).chainOnce(now);
var exporter = new SealedExporter(config, storage);
var first = exporter.run("USER", now).row();
assertThat(first).isNotNull();
assertThat(first.getValueString("lock_mode")).isEqualTo("GOVERNANCE");
assertThat(first.getValueInstant("retain_until")).isEqualTo(now.truncatedTo(java.time.temporal.ChronoUnit.MILLIS).plusSeconds(30 * 86400L));
assertThat(storage.heads.values()).allSatisfy(h -> assertThat(h.getLockMode()).isEqualTo("GOVERNANCE"));
var verifier = new SealedExportVerifier(config, storage);
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
editor();
update("export_lock_mode", "COMPLIANCE");
update("checkpoint_every_links", 1);
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(2));
var second = exporter.run("USER", Instant.now().plusSeconds(2)).row();
assertThat(second.getValueString("lock_mode")).isEqualTo("COMPLIANCE");
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
assertThat(verifier.verify(second, "USER").result().passed()).isTrue();
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(2)).passed()).isTrue();
String key = first.getValueString("object_prefix") + "links.jsonl";
storage.heads.get(key).withLockMode("COMPLIANCE");
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains("lacks the recorded GOVERNANCE retention"));
storage.heads.remove(key);
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains(key) && b.reason().contains("GOVERNANCE bypass removal"));
}
}
@@ -57,7 +57,7 @@ public class MemorySealedStore implements SealedStore
** Refuse overwrites and retain the exact bytes.
*******************************************************************************/
@Override
public String put(String key, Path file, Instant until) throws Exception
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
{
if(objects.containsKey(key))
{
@@ -65,7 +65,7 @@ public class MemorySealedStore implements SealedStore
}
String version = "v" + (objects.size() + 1);
objects.put(key, Files.readAllBytes(file));
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode("COMPLIANCE").withLockRetainUntil(until));
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode(mode.name()).withLockRetainUntil(until));
return (version);
}
@@ -176,6 +176,7 @@ public class WormAuditDemoServer extends AbstractQQQApplication
public void seed(QInstance instance) throws Exception
{
QContext.init(instance, new QSession());
com.kingsrook.qbits.wormaudit.WormAuditSettings.seed(config);
AuditStore store = new AuditStore(null);
Instant today = Instant.now().minus(Duration.ofHours(2));
try