Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f14a24fdbf
|
||
|
|
0fb9e8912c | ||
|
|
626e153841 | ||
|
|
85de1daf79 | ||
|
|
be26a370ac | ||
|
|
3c6e7d57a2 |
No files matched your search
@@ -1,7 +1,7 @@
|
||||
version: 2.1
|
||||
|
||||
orbs:
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.8
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.9
|
||||
|
||||
commands:
|
||||
configure-package-install:
|
||||
|
||||
@@ -6,6 +6,10 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
|
||||
|
||||
### Added
|
||||
|
||||
- RC.2 settings: editable, FIELD-audited singleton with insert-only defaults, validation, separate edit permission and a 30-second effective-settings cache. Schedules load at startup.
|
||||
- GOVERNANCE exports with 30-day default retention; per-export retention verification, administrator-bypass diagnostics, home warnings and documented COMPLIANCE transition.
|
||||
- New `worm-audit:10` settings migration; historical changesets remain unchanged.
|
||||
|
||||
- Phase (c): configurable WORM Audit app home, full-width summary widget and optional host health card; permission-aware navigation and content-free, responsive light/dark summaries.
|
||||
- Chain backlog/settling counts, verification and export record links, sealed-range gaps/lag, retention eligibility and recent purge/refusal history.
|
||||
- PostgreSQL guard/ownership/purge-function diagnostics, signing/trust warnings, tri-state Object Lock capability and stale daily-verification warnings.
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
Phases (a)–(c) provide PostgreSQL append-only enforcement, an asynchronous audit hash chain,
|
||||
Ed25519 checkpoints, live and sealed-export verification actions, and daily jobs with core QAlerts signals.
|
||||
It hashes exactly the audit values core stored; it does not read application records or
|
||||
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and COMPLIANCE
|
||||
Object Lock. The WORM Audit app home summarizes chain health, exports, retention and setup.
|
||||
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and configurable
|
||||
Object Lock (GOVERNANCE for 30 days by default). The WORM Audit app home summarizes chain health, exports, retention and setup.
|
||||
|
||||
Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement;
|
||||
memory/H2 support tamper-evidence tests without database enforcement.
|
||||
@@ -12,8 +12,9 @@ memory/H2 support tamper-evidence tests without database enforcement.
|
||||
## Tables and permissions
|
||||
|
||||
All sidecar tables use the configured host backend, snake_case fields, BIGINT identifiers,
|
||||
record sections, and full table-name permissions. They do not audit themselves. The app lists
|
||||
Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that order.
|
||||
record sections, and full table-name permissions. Immutable sidecars do not audit themselves.
|
||||
The editable Settings singleton uses an INTEGER ID and FIELD-level core audit. The app lists
|
||||
Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges, and Settings in that order.
|
||||
|
||||
| Table | Purpose | Permission |
|
||||
|---|---|---|
|
||||
@@ -22,18 +23,19 @@ Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that orde
|
||||
| `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` |
|
||||
| `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` |
|
||||
| `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` |
|
||||
| `worm_audit_settings` | Editable singleton; every edit is audited and chained | `worm_audit_settings.read`; editing needs `worm_audit_settings.edit` **and** `worm_audit.settings` |
|
||||
|
||||
`worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action.
|
||||
It also authorizes **Verify sealed export** on selected export records. `worm_audit.export`
|
||||
authorizes **Export now** on the export table. Processes are table/record actions or schedules;
|
||||
none is a navigation entry.
|
||||
Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities
|
||||
Insert/update/delete capabilities are absent on immutable sidecar tables; Settings allows updates only; update/delete capabilities
|
||||
are removed from all four core audit tables. Internal appends use an explicit permission bypass.
|
||||
Checkpoint, export and verification record screens have a bounded chain-link child list;
|
||||
range endpoints and export/checkpoint references link to their records.
|
||||
|
||||
The table metadata customizer can configure host ESB publications. Quick Search registration
|
||||
is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables.
|
||||
is host-owned through `QuickSearchQBitConfig.searchableTables`; add the desired tables.
|
||||
Verification and export inserts use QQQ's record-change listener SPI,
|
||||
so an installed ESB publisher receives those events when the host configures its destinations
|
||||
on these two tables. The qbit defines no destination and configures no chain-link publication.
|
||||
@@ -70,6 +72,58 @@ to verification. It does not itself break the chain. Signing requires a matching
|
||||
Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints.
|
||||
Never store the private key in a database row or checked-in configuration.
|
||||
|
||||
## Settings
|
||||
|
||||
Open **WORM Audit → Settings**, record **1** (`worm_audit_settings`, labelled WORM Audit Settings).
|
||||
This is a standard editable QQQ table. New records and deletion are disabled; a pre-insert
|
||||
check and database constraint require ID 1. Every edit uses core FIELD-level auditing, so
|
||||
the change enters the audit chain. `modify_date` and `modified_by` are stamped by the server.
|
||||
Keep notes free of sensitive data; notes are audited too.
|
||||
|
||||
| Field | Default | Validation / effect |
|
||||
|---|---|---|
|
||||
| `audit_retention_years` | 6 | 1–100; home next eligible purge date |
|
||||
| `export_lock_mode` | GOVERNANCE | GOVERNANCE or COMPLIANCE; new exports |
|
||||
| `export_retention_days` | 30 | 1–36500; new exports |
|
||||
| `checkpoint_every_links` | 10000 | Positive integer; new checkpoint ranges |
|
||||
| `checkpoint_every_minutes` | 60 | Positive integer; checkpoint cadence |
|
||||
| `chain_interval_seconds` | 30 | Positive integer; **takes effect after restart** |
|
||||
| `settle_horizon_seconds` | 60 | Nonnegative integer; chaining and verification |
|
||||
| `late_detail_lookback_hours` | 24 | Positive integer; late-detail polling |
|
||||
| `export_cron` | `0 0 2 * * ?` | Quartz cron, UTC; **takes effect after restart** |
|
||||
| `verify_cron` | `0 0 3 * * ?` | Quartz cron, UTC; **takes effect after restart** |
|
||||
| `notes` | Empty | Administrative notes |
|
||||
| `modify_date`, `modified_by` | Server supplied | Last edit time and user |
|
||||
|
||||
Config values supply defaults for missing rows or null fields; a missing settings table also
|
||||
falls back to config. Other database failures fail closed. `exportLockMode` and
|
||||
`exportRetentionDays` are the corresponding config defaults. Runtime reads cache the row
|
||||
for at most 30 seconds per application instance; local QQQ edits invalidate that cache.
|
||||
Each run reads effective settings. Startup schedules read the existing row, or config when
|
||||
absent. Historical checkpoints remain valid after changing their creation cadence.
|
||||
|
||||
Changeset `worm-audit:10` creates and seeds the table with `INSERT ... SELECT ... WHERE NOT
|
||||
EXISTS`; rerunning it never overwrites an administrator change. The renderer seeds from the
|
||||
host config. Memory hosts and the demo can call `WormAuditSettings.seed(config)` for the same
|
||||
insert-only behavior during single-threaded startup.
|
||||
|
||||
`audit_retention_years` does **not** rewrite the database purge function. The home warns if
|
||||
it differs from the compiled interval; changing that interval requires a reviewed migration.
|
||||
The home also warns when export retention is shorter than audit retention and while
|
||||
GOVERNANCE permits administrator bypass deletion. The runtime role receives SELECT, INSERT
|
||||
and UPDATE on Settings, with no DELETE or TRUNCATE grant and no append-only triggers.
|
||||
|
||||
## Switching sealed exports to COMPLIANCE
|
||||
|
||||
Edit `export_lock_mode` to **COMPLIANCE** in Settings. New exports use COMPLIANCE once the
|
||||
settings cache refreshes (within 30 seconds); existing GOVERNANCE objects keep their mode
|
||||
unless an administrator explicitly extends their retention and upgrades their mode. The
|
||||
verifier continues to compare each object with its export row's recorded mode and date, so
|
||||
an out-of-band mode change is reported as a mismatch. The bucket default retention should
|
||||
match the chosen mode and retention days, although the exporter always sends explicit values.
|
||||
The settings change is audited and chained. **COMPLIANCE cannot be undone**: protected
|
||||
versions cannot be deleted or have retention shortened until expiry, including by administrators.
|
||||
|
||||
## App home
|
||||
|
||||
Open `/app/worm_audit` (or `/app/admin/worm_audit` with `parentAppName="admin"`).
|
||||
@@ -82,7 +136,7 @@ it never renders audit messages or field values. Record links retain the target
|
||||
- Chain status: live link count, latest link time, settled unchained backlog and settling
|
||||
count at the configured horizon, latest verification and first open chain break.
|
||||
A subsequent full scheduled CHAIN PASS clears a reported break; a selected USER PASS does not.
|
||||
- Export status: latest sealed range, COMPLIANCE retention, link/time lag, coverage holes
|
||||
- Export status: current lock mode and retention days, latest sealed range and its recorded retention, link/time lag, coverage holes
|
||||
between the purge anchor and sealed head, and bucket Object Lock capability. The unsealed
|
||||
trailing suffix is lag, not a gap. Provider failures show UNKNOWN with a value-free reason.
|
||||
- Retention: configured years, purge anchor, oldest live audit and its next eligible purge date.
|
||||
@@ -207,8 +261,11 @@ Install qbit-object-storage and configure `exportBucketAlias` (default `worm_aud
|
||||
`BucketRole.BACKUP` bucket. **Object Lock must be enabled when the bucket is created**.
|
||||
The exporter requires the bucket's `OBJECT_LOCK` capability to be `SUPPORTED`; run the
|
||||
object-storage read-only capability probe during setup. Unknown, partial and unsupported
|
||||
capabilities all refuse sealing. Every object receives explicit **COMPLIANCE** retention
|
||||
until the sealing time plus `retentionYears` (six by default), checked by HEAD after PUT.
|
||||
capabilities all refuse sealing. Every object receives the effective `export_lock_mode` and retention until sealing time plus
|
||||
`export_retention_days` (GOVERNANCE and 30 days by default), checked by HEAD after PUT. The
|
||||
export row records both values. Verification checks that historical mode and retain-until,
|
||||
regardless of subsequent settings changes. Missing GOVERNANCE objects are reported as breaks
|
||||
with a possible administrator bypass-removal explanation.
|
||||
|
||||
The runtime IAM policy needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
|
||||
`s3:GetObjectRetention`, `s3:PutObjectRetention`, and
|
||||
@@ -284,8 +341,8 @@ $env:CI = 'true'
|
||||
./mvnw.cmd -o verify
|
||||
```
|
||||
|
||||
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks COMPLIANCE
|
||||
retention, denied version deletion, overwrite version preservation, export verification,
|
||||
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks GOVERNANCE
|
||||
with 30-day retention and a COMPLIANCE export, denied version deletion (without bypass for GOVERNANCE), overwrite version preservation, export verification,
|
||||
exact live-tamper findings through the record action, and refusal of a bucket without lock:
|
||||
|
||||
```powershell
|
||||
|
||||
@@ -228,6 +228,10 @@ installed the alerts go to the log.
|
||||
## 6. Sealed exports
|
||||
|
||||
### 6.1 Export job
|
||||
|
||||
**Default superseded by Owner decisions (2026-10-14) below:** lock mode and export retention
|
||||
now come from Settings, defaulting to GOVERNANCE and 30 days. The original decision text is
|
||||
retained for history.
|
||||
Daily at 02:00 (configurable), and by the table action "Export now" on `worm_audit_export`:
|
||||
1. Range: links after the last export's `to_link_id`, up to the latest checkpoint. Exports are
|
||||
contiguous by construction. The app home flags a gap (a range not covered) and lag.
|
||||
@@ -386,3 +390,33 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
|
||||
|
||||
These are owner-run (Praetor and cluster-gitops are applied by the owner). The qbit only
|
||||
documents them.
|
||||
|
||||
## Owner decisions (2026-10-14)
|
||||
|
||||
The 2026-10-12 decisions above remain the historical design. These decisions supersede the
|
||||
COMPLIANCE/six-year export default in §6.1 and the corresponding default assumptions in
|
||||
§§2, 4.5, 9, 11, 12 and the §14 infrastructure proposal.
|
||||
|
||||
- Settings live in the application as the standard QQQ `worm_audit_settings` table, record
|
||||
ID 1, labelled WORM Audit Settings and navigated as Settings after Purges. It allows edits,
|
||||
has no append-only triggers, and uses FIELD-level core audit so every change is chained.
|
||||
Reading needs `worm_audit_settings.read`; editing additionally needs
|
||||
`worm_audit_settings.edit` and `worm_audit.settings`. New IDs and deletion are refused.
|
||||
- Defaults are seeded insert-only in new changeset `worm-audit:10`; existing changesets are
|
||||
unchanged. An administrator's edits survive subsequent seeding. Config supplies defaults
|
||||
when a row/field/table is absent. Runtime reads cache at most 30 seconds, scoped to the
|
||||
QInstance. Chain interval and UTC Quartz export/verify schedules take effect after restart.
|
||||
- kof22 starts with **GOVERNANCE, 30 days** for sealed exports. An administrator with
|
||||
`s3:BypassGovernanceRetention` can remove those copies. This is deliberately reversible;
|
||||
it does not provide COMPLIANCE's irreversible retention. Runtime credentials still receive
|
||||
no delete or bypass permission. Bucket defaults should match the selected settings.
|
||||
- Changing Settings to COMPLIANCE affects new exports. Existing GOVERNANCE versions retain
|
||||
their original mode unless an administrator extends/upgrades them. Verification checks
|
||||
the catalog's recorded mode and date, so external mode changes are reported as mismatches.
|
||||
COMPLIANCE cannot be undone. The settings change itself is audited and chained.
|
||||
- Audit retention defaults to six years independently of export retention. The home warns
|
||||
about a shorter export horizon and a mismatch with the purge function's compiled interval.
|
||||
Updating the setting does not modify the purge function; that still requires a migration.
|
||||
|
||||
The README Settings table specifies all fields, defaults, bounds and timing. Signing keys,
|
||||
public-key pins, bucket alias, prefix and backend remain host configuration, never settings.
|
||||
@@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
|
||||
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
|
||||
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
|
||||
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
|
||||
|
||||
-- changeset worm-audit:10
|
||||
-- Editable singleton: deliberately excluded from append-only triggers.
|
||||
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
|
||||
id integer PRIMARY KEY CHECK (id = 1),
|
||||
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
|
||||
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
|
||||
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
|
||||
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
|
||||
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
|
||||
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
|
||||
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
|
||||
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
|
||||
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
|
||||
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
|
||||
notes text,
|
||||
modify_date timestamptz NOT NULL DEFAULT now(),
|
||||
modified_by text NOT NULL DEFAULT current_user
|
||||
);
|
||||
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM "app";
|
||||
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO "app";
|
||||
INSERT INTO public.worm_audit_settings
|
||||
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
|
||||
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
|
||||
export_cron, verify_cron)
|
||||
SELECT 1, 6, 'GOVERNANCE', 30, 10000, 60, 30, 60, 24, '0 0 2 * * ?', '0 0 3 * * ?'
|
||||
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
|
||||
@@ -1,5 +1,14 @@
|
||||
# WORM Audit screenshots
|
||||
|
||||
## RC.2 settings captures (2026-10-14)
|
||||
Home (GOVERNANCE, 30-day exports) and the WORM Audit Settings record at 390 x 844 and 1440 x 1000,
|
||||
light and dark, captured with Playwright against `WormAuditDemoServer` on port 8099:
|
||||
`worm-audit-home-{390,1440}-{light,dark}.jpg` (replaced) and
|
||||
`worm-audit-settings-{390,1440}-{light,dark}.jpg` (new). Overflow check on all eight: no page
|
||||
horizontal overflow, no inner horizontal scrollers, no double scrollbars.
|
||||
|
||||
## Earlier captures
|
||||
|
||||
Refreshed after record-screen polish **2026-10-13 (America/Chicago)** from the rebuilt test-scope demo using
|
||||
packaged Next UI 1.0.0-RC.14 and connected Edge through the browser screenshot
|
||||
and viewport tools. All files are JPEG viewport captures, below 300 KB each.
|
||||
|
||||
|
Before Width: | Height: | Size: 74 KiB After Width: | Height: | Size: 84 KiB |
|
Before Width: | Height: | Size: 72 KiB After Width: | Height: | Size: 82 KiB |
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 29 KiB After Width: | Height: | Size: 31 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 75 KiB |
|
After Width: | Height: | Size: 33 KiB |
|
After Width: | Height: | Size: 33 KiB |
@@ -45,6 +45,9 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
{
|
||||
}
|
||||
|
||||
private String exportLockMode = "GOVERNANCE";
|
||||
private int exportRetentionDays = 30;
|
||||
private final WormAuditSettings settings = new WormAuditSettings(this);
|
||||
private String appName = "worm_audit";
|
||||
private String appLabel = "WORM Audit";
|
||||
private String parentAppName;
|
||||
@@ -70,6 +73,63 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read exportLockMode.
|
||||
*******************************************************************************/
|
||||
public String getExportLockMode()
|
||||
{
|
||||
return (exportLockMode);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure exportLockMode.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withExportLockMode(String exportLockMode)
|
||||
{
|
||||
this.exportLockMode = exportLockMode;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read exportRetentionDays.
|
||||
*******************************************************************************/
|
||||
public int getExportRetentionDays()
|
||||
{
|
||||
return (exportRetentionDays);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure exportRetentionDays.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withExportRetentionDays(int exportRetentionDays)
|
||||
{
|
||||
this.exportRetentionDays = exportRetentionDays;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read effective settings through the instance-scoped 30-second cache.
|
||||
*******************************************************************************/
|
||||
@JsonIgnore
|
||||
public WormAuditSettings.Values effectiveSettings()
|
||||
{
|
||||
return (settings.read());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Discard this instance's cached row after a successful local settings write.
|
||||
*******************************************************************************/
|
||||
public void invalidateSettings()
|
||||
{
|
||||
settings.invalidate();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the app name.
|
||||
*******************************************************************************/
|
||||
@@ -527,6 +587,7 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
{
|
||||
throw (new IllegalArgumentException("Export bucket, prefix and schedule must be configured"));
|
||||
}
|
||||
WormAuditSettings.validate(WormAuditSettings.defaults(this));
|
||||
if(backendName == null || backendName.isBlank() || retentionYears < 1 || chainIntervalSeconds < 1
|
||||
|| lateDetailLookbackHours < 1 || settleHorizonSeconds < 0 || batchSize < 1 || checkpointEveryLinks < 1 || checkpointEveryMinutes < 1)
|
||||
{
|
||||
|
||||
@@ -32,6 +32,7 @@ import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertDefaultMonitor;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
@@ -182,21 +183,31 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList").withGridColumns(12)
|
||||
.withCodeReference(new QCodeReference(LinkRangeWidget.class)).withDefaultValue("parentTable", name));
|
||||
}
|
||||
for(var process : WormAuditProcesses.processes(qBitConfig))
|
||||
List<QTableMetaData> producedTables = output.getEach(QTableMetaData.class);
|
||||
output.addSelfToInstance(qInstance);
|
||||
output = new MetaDataProducerMultiOutput();
|
||||
var previousInstance = QContext.getQInstance();
|
||||
var previousSession = QContext.getQSession();
|
||||
var previousTransaction = QContext.getQBackendTransaction();
|
||||
try
|
||||
{
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
// Production is still assembling the host metadata. Do not validate it mid-build.
|
||||
QContext.setQInstance(qInstance);
|
||||
QContext.setQSession(new com.kingsrook.qqq.backend.core.model.session.QSession());
|
||||
QContext.setTransaction(null);
|
||||
addProcesses(output, qBit);
|
||||
}
|
||||
for(var process : ExportProcesses.processes(qBitConfig))
|
||||
finally
|
||||
{
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
QContext.setQInstance(previousInstance);
|
||||
QContext.setQSession(previousSession);
|
||||
QContext.setTransaction(previousTransaction);
|
||||
}
|
||||
if(qInstance.getSchedulers() == null || !qInstance.getSchedulers().containsKey(qBitConfig.getSchedulerName()))
|
||||
{
|
||||
qInstance.addScheduler(new QuartzSchedulerMetaData().withName(qBitConfig.getSchedulerName()));
|
||||
}
|
||||
output.add(WormAuditApp.app(qBitConfig, output.getEach(QTableMetaData.class)));
|
||||
output.add(WormAuditApp.app(qBitConfig, producedTables));
|
||||
for(QWidgetMetaData widget : WormAuditApp.widgets())
|
||||
{
|
||||
output.add(widget);
|
||||
@@ -224,6 +235,24 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build schedules from settings available at producer startup.
|
||||
*******************************************************************************/
|
||||
private void addProcesses(MetaDataProducerMultiOutput output, QBitMetaData qBit)
|
||||
{
|
||||
for(var process : WormAuditProcesses.processes(qBitConfig))
|
||||
{
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
}
|
||||
for(var process : ExportProcesses.processes(qBitConfig))
|
||||
{
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** RC.5 compatibility: replace with a direct call when the qbit builds on RC.6 (qqq#1141, #1163).
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.sql.SQLException;
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import org.quartz.CronExpression;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Singleton settings merged over host defaults. Only absent rows/tables fall
|
||||
** back; database outages and invalid persisted settings fail closed.
|
||||
*******************************************************************************/
|
||||
public class WormAuditSettings
|
||||
{
|
||||
private final WormAuditQBitConfig config;
|
||||
private final Clock clock;
|
||||
private QInstance instance;
|
||||
private QRecord cached;
|
||||
private Instant expires = Instant.MIN;
|
||||
|
||||
/*******************************************************************************
|
||||
** An immutable snapshot for one run.
|
||||
*******************************************************************************/
|
||||
public record Values(int audit_retention_years,
|
||||
String export_lock_mode,
|
||||
int export_retention_days,
|
||||
int checkpoint_every_links,
|
||||
int checkpoint_every_minutes,
|
||||
int chain_interval_seconds,
|
||||
int settle_horizon_seconds,
|
||||
int late_detail_lookback_hours,
|
||||
String export_cron,
|
||||
String verify_cron)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Use the production clock.
|
||||
*******************************************************************************/
|
||||
public WormAuditSettings(WormAuditQBitConfig config)
|
||||
{
|
||||
this(config, Clock.systemUTC());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Inject a clock for cache boundary tests.
|
||||
*******************************************************************************/
|
||||
public WormAuditSettings(WormAuditQBitConfig config, Clock clock)
|
||||
{
|
||||
this.config = config;
|
||||
this.clock = clock;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Invalidate after a local edit; other application instances refresh within 30 seconds.
|
||||
*******************************************************************************/
|
||||
public synchronized void invalidate()
|
||||
{
|
||||
expires = Instant.MIN;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Seed the singleton only when absent; never overwrite an administrator edit.
|
||||
*******************************************************************************/
|
||||
public static void seed(WormAuditQBitConfig config) throws com.kingsrook.qqq.backend.core.exceptions.QException
|
||||
{
|
||||
var store = new AuditStore(null);
|
||||
if(store.get(WormAuditTables.SETTINGS, 1) == null)
|
||||
{
|
||||
store.insert(WormAuditTables.SETTINGS, java.util.List.of(defaults(config)));
|
||||
}
|
||||
config.invalidateSettings();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read at most once per 30 seconds, isolated by QInstance identity.
|
||||
*******************************************************************************/
|
||||
public synchronized Values read()
|
||||
{
|
||||
QInstance current = QContext.getQInstance();
|
||||
if(current != instance || !clock.instant().isBefore(expires))
|
||||
{
|
||||
QRecord row = null;
|
||||
if(current != null && current.getTable(WormAuditTables.SETTINGS) != null)
|
||||
{
|
||||
try
|
||||
{
|
||||
row = new AuditStore(null).get(WormAuditTables.SETTINGS, 1);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
if(!missingTable(e))
|
||||
{
|
||||
throw (new IllegalStateException("Cannot read WORM Audit Settings", e));
|
||||
}
|
||||
}
|
||||
}
|
||||
cached = row;
|
||||
instance = current;
|
||||
expires = row == null ? Instant.MIN : clock.instant().plusSeconds(30);
|
||||
}
|
||||
QRecord row = defaults(config);
|
||||
if(cached != null)
|
||||
{
|
||||
for(String field : row.getValues().keySet())
|
||||
{
|
||||
if(cached.getValue(field) != null)
|
||||
{
|
||||
row.setValue(field, cached.getValue(field));
|
||||
}
|
||||
}
|
||||
}
|
||||
validate(row);
|
||||
return (new Values(row.getValueInteger("audit_retention_years"),
|
||||
row.getValueString("export_lock_mode"),
|
||||
row.getValueInteger("export_retention_days"),
|
||||
row.getValueInteger("checkpoint_every_links"),
|
||||
row.getValueInteger("checkpoint_every_minutes"),
|
||||
row.getValueInteger("chain_interval_seconds"),
|
||||
row.getValueInteger("settle_horizon_seconds"),
|
||||
row.getValueInteger("late_detail_lookback_hours"),
|
||||
row.getValueString("export_cron"),
|
||||
row.getValueString("verify_cron")));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build insert-only seed values from the host defaults.
|
||||
*******************************************************************************/
|
||||
public static QRecord defaults(WormAuditQBitConfig config)
|
||||
{
|
||||
return (new QRecord().withValue("id", 1)
|
||||
.withValue("audit_retention_years", config.getRetentionYears())
|
||||
.withValue("export_lock_mode", config.getExportLockMode())
|
||||
.withValue("export_retention_days", config.getExportRetentionDays())
|
||||
.withValue("checkpoint_every_links", config.getCheckpointEveryLinks())
|
||||
.withValue("checkpoint_every_minutes", config.getCheckpointEveryMinutes())
|
||||
.withValue("chain_interval_seconds", config.getChainIntervalSeconds())
|
||||
.withValue("settle_horizon_seconds", config.getSettleHorizonSeconds())
|
||||
.withValue("late_detail_lookback_hours", config.getLateDetailLookbackHours())
|
||||
.withValue("export_cron", config.getExportCron())
|
||||
.withValue("verify_cron", config.getVerifyCron()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validate supplied fields; partial updates leave omitted fields untouched.
|
||||
*******************************************************************************/
|
||||
public static void validate(QRecord row)
|
||||
{
|
||||
if(row.getValue("id") != null && !Integer.valueOf(1).equals(row.getValueInteger("id")))
|
||||
{
|
||||
throw (new IllegalArgumentException("WORM Audit Settings has one row only (ID 1)."));
|
||||
}
|
||||
range(row, "audit_retention_years", "Audit retention years", 1, 100);
|
||||
range(row, "export_retention_days", "Export retention days", 1, 36500);
|
||||
range(row, "checkpoint_every_links", "Checkpoint every links", 1, 2147483647);
|
||||
range(row, "checkpoint_every_minutes", "Checkpoint every minutes", 1, 2147483647);
|
||||
range(row, "chain_interval_seconds", "Chain interval seconds", 1, 2147483647);
|
||||
range(row, "settle_horizon_seconds", "Settle horizon seconds", 0, 2147483647);
|
||||
range(row, "late_detail_lookback_hours", "Late detail lookback hours", 1, 2147483647);
|
||||
if(row.getValues().containsKey("export_lock_mode") && !java.util.Arrays.asList("GOVERNANCE", "COMPLIANCE").contains(row.getValueString("export_lock_mode")))
|
||||
{
|
||||
throw (new IllegalArgumentException("Export lock mode must be GOVERNANCE or COMPLIANCE."));
|
||||
}
|
||||
for(String field : java.util.List.of("export_cron", "verify_cron"))
|
||||
{
|
||||
if(row.getValues().containsKey(field) && (row.getValueString(field) == null || !CronExpression.isValidExpression(row.getValueString(field))))
|
||||
{
|
||||
throw (new IllegalArgumentException((field.equals("export_cron") ? "Export" : "Verify") + " schedule must be a valid Quartz cron expression (UTC)."));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reject null, fractional and out-of-range numbers with a plain field label.
|
||||
*******************************************************************************/
|
||||
private static void range(QRecord row, String field, String label, int minimum, int maximum)
|
||||
{
|
||||
if(!row.getValues().containsKey(field))
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
long value = Long.parseLong(String.valueOf(row.getValue(field)));
|
||||
if(value >= minimum && value <= maximum)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
catch(NumberFormatException ignored)
|
||||
{
|
||||
// Use the same clear message for invalid numbers and invalid ranges.
|
||||
}
|
||||
throw (new IllegalArgumentException(label + " must be a whole number from " + minimum + " to " + maximum + "."));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Recognize PostgreSQL/H2 missing-table errors without hiding other failures.
|
||||
*******************************************************************************/
|
||||
private static boolean missingTable(Throwable error)
|
||||
{
|
||||
for(Throwable cause = error; cause != null; cause = cause.getCause())
|
||||
{
|
||||
if(cause instanceof SQLException sql && ("42P01".equals(sql.getSQLState()) || "42S02".equals(sql.getSQLState()) || "42S04".equals(sql.getSQLState())))
|
||||
{
|
||||
return (true);
|
||||
}
|
||||
}
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
@@ -49,6 +49,11 @@ public final class PostgresSetupWarnings
|
||||
public static List<String> load(QBackendTransaction transaction, WormAuditQBitConfig config)
|
||||
{
|
||||
List<String> warnings = new ArrayList<>();
|
||||
int years = config.effectiveSettings().audit_retention_years();
|
||||
if(years != config.getRetentionYears())
|
||||
{
|
||||
warnings.add("Audit retention setting differs from the configured database purge interval (" + config.getRetentionYears() + " years). A reviewed migration is required.");
|
||||
}
|
||||
if(!(transaction instanceof RDBMSTransaction rdbms))
|
||||
{
|
||||
return (warnings);
|
||||
@@ -60,8 +65,10 @@ public final class PostgresSetupWarnings
|
||||
{
|
||||
return (warnings);
|
||||
}
|
||||
// The live function is authoritative on PostgreSQL, rather than the renderer defaults.
|
||||
warnings.clear();
|
||||
List<String> tables = new ArrayList<>(config.getCoreAuditTableNames().values());
|
||||
tables.addAll(WormAuditTables.ALL);
|
||||
tables.addAll(WormAuditTables.ALL.stream().filter(t -> !WormAuditTables.SETTINGS.equals(t)).toList());
|
||||
for(String table : tables)
|
||||
{
|
||||
for(String trigger : List.of("worm_audit_row_guard", "worm_audit_truncate_guard"))
|
||||
@@ -93,6 +100,17 @@ public final class PostgresSetupWarnings
|
||||
}
|
||||
}
|
||||
}
|
||||
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT pg_get_functiondef(to_regprocedure('public.worm_audit_purge(timestamp with time zone)'))"))
|
||||
{
|
||||
if(result.next() && result.getString(1) != null)
|
||||
{
|
||||
var interval = java.util.regex.Pattern.compile("(?i)interval\\s+'([0-9]+) years'").matcher(result.getString(1));
|
||||
if(interval.find() && Integer.parseInt(interval.group(1)) != years)
|
||||
{
|
||||
warnings.add("Audit retention setting differs from the purge function's compiled interval (" + interval.group(1) + " years). A reviewed migration is required.");
|
||||
}
|
||||
}
|
||||
}
|
||||
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT to_regprocedure('public.worm_audit_purge(timestamp with time zone)') IS NOT NULL"))
|
||||
{
|
||||
result.next();
|
||||
|
||||
@@ -88,7 +88,8 @@ public final class WormAuditApp
|
||||
for(String name : WormAuditTables.NAV)
|
||||
{
|
||||
QTableMetaData table = tables.stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
|
||||
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table);
|
||||
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(WormAuditTables.navigationLabel(name)),
|
||||
WormAuditTables.SETTINGS.equals(name) ? table.clone().withLabel("Settings") : table);
|
||||
}
|
||||
return (app);
|
||||
}
|
||||
|
||||
@@ -54,7 +54,7 @@ import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
|
||||
public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, long settling, int settleSeconds,
|
||||
QRecord lastVerification, QRecord openBreak, QRecord lastExport, long exportLag, Long exportAgeHours,
|
||||
List<String> exportGaps, String bucketAlias, String lockCapability, String lockReason,
|
||||
int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
|
||||
String exportLockMode, int exportRetentionDays, int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
|
||||
List<QRecord> recentPurges, List<String> warnings)
|
||||
{
|
||||
/*******************************************************************************
|
||||
@@ -71,7 +71,16 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
|
||||
*******************************************************************************/
|
||||
public static WormAuditHomeData load(WormAuditQBitConfig config, SealedStore storage, Instant now) throws QException
|
||||
{
|
||||
var settings = config.effectiveSettings();
|
||||
List<String> warnings = new ArrayList<>();
|
||||
if("GOVERNANCE".equals(settings.export_lock_mode()))
|
||||
{
|
||||
warnings.add("Sealed exports use GOVERNANCE mode: an administrator with s3:BypassGovernanceRetention can delete them. Switch to COMPLIANCE for write-once copies.");
|
||||
}
|
||||
if(now.plus(Duration.ofDays(settings.export_retention_days())).isBefore(now.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant()))
|
||||
{
|
||||
warnings.add("Export retention is shorter than audit retention.");
|
||||
}
|
||||
String capability = "UNKNOWN";
|
||||
String reason = "Object Lock capability could not be confirmed";
|
||||
try
|
||||
@@ -113,7 +122,7 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
|
||||
AuditStore store = new AuditStore(transaction);
|
||||
QRecord lastLink = store.last(WormAuditTables.CHAIN);
|
||||
long links = store.count(WormAuditTables.CHAIN, new QQueryFilter());
|
||||
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
|
||||
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
|
||||
long backlog = store.unchainedCount(cutoff, true);
|
||||
long settling = store.unchainedCount(cutoff, false);
|
||||
QRecord verification = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(), "finished_at");
|
||||
@@ -147,10 +156,10 @@ public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, lo
|
||||
warnings.addAll(PostgresSetupWarnings.load(transaction, config));
|
||||
transaction.commit();
|
||||
return (new WormAuditHomeData(links, lastLink == null ? null : lastLink.getValueInstant("chained_at"), backlog, settling,
|
||||
config.getSettleHorizonSeconds(), verification, openBreak, export, lag,
|
||||
settings.settle_horizon_seconds(), verification, openBreak, export, lag,
|
||||
export == null || export.getValueInstant("sealed_at") == null ? null : Math.max(0, Duration.between(export.getValueInstant("sealed_at"), now).toHours()),
|
||||
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, config.getRetentionYears(), anchorId == 0 ? null : anchor,
|
||||
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
|
||||
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, settings.export_lock_mode(), settings.export_retention_days(), settings.audit_retention_years(), anchorId == 0 ? null : anchor,
|
||||
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(settings.audit_retention_years()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
|
||||
@@ -82,6 +82,8 @@ public final class WormAuditHomeView
|
||||
}
|
||||
close(html);
|
||||
open(html, "Export status");
|
||||
row(html, "Export lock mode", data.exportLockMode());
|
||||
row(html, "Export retention", data.exportRetentionDays() + " days");
|
||||
QRecord export = data.lastExport();
|
||||
if(export == null)
|
||||
{
|
||||
@@ -93,7 +95,7 @@ public final class WormAuditHomeView
|
||||
row(html, "Link range", export.getValueLong("from_link_id") + "–" + export.getValueLong("to_link_id"));
|
||||
row(html, "Sealed at", when(export.getValueInstant("sealed_at"), "Not recorded"));
|
||||
row(html, "Retain until", when(export.getValueInstant("retain_until"), "Not recorded"));
|
||||
row(html, "Lock mode", export.getValueString("lock_mode"));
|
||||
row(html, "Last export lock mode", export.getValueString("lock_mode"));
|
||||
}
|
||||
row(html, "Export lag", data.exportLag() + " links; " + (data.exportAgeHours() == null ? "no previous export time" : data.exportAgeHours() + " h since last export"));
|
||||
row(html, "Gaps in sealed ranges", data.exportGaps().isEmpty() ? (export == null ? "No sealed ranges to check" : "None") : String.join(", ", data.exportGaps()));
|
||||
|
||||
@@ -31,6 +31,7 @@ import java.util.TreeMap;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
import java.util.stream.Collectors;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
|
||||
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
|
||||
import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
@@ -99,7 +100,7 @@ public class WormAuditChainer
|
||||
}
|
||||
}
|
||||
}
|
||||
Counts counts = append(new AuditStore(transaction), now);
|
||||
Counts counts = append(new AuditStore(transaction), now, config.effectiveSettings());
|
||||
transaction.commit();
|
||||
return (counts);
|
||||
}
|
||||
@@ -119,14 +120,14 @@ public class WormAuditChainer
|
||||
/*******************************************************************************
|
||||
** Append settled headers and uncovered detail chunks, then checkpoint.
|
||||
*******************************************************************************/
|
||||
private Counts append(AuditStore store, Instant now) throws Exception
|
||||
private Counts append(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
|
||||
{
|
||||
QRecord tail = store.last(WormAuditTables.CHAIN);
|
||||
QRecord anchor = store.anchor();
|
||||
long nextId = tail == null ? anchor.getValueLong("anchor_link_id") + 1 : tail.getValueLong("id") + 1;
|
||||
String previous = tail == null ? anchor.getValueString("anchor_hash") : tail.getValueString("link_hash");
|
||||
List<QRecord> appended = new ArrayList<>();
|
||||
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
|
||||
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
|
||||
List<QRecord> audits = store.unchained(cutoff, config.getBatchSize());
|
||||
Map<Long, QRecord> details = AuditStore.index(store.in("auditDetail", "auditId", audits.stream().map(r -> r.getValueLong("id")).toList()));
|
||||
Map<Long, List<QRecord>> grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId")));
|
||||
@@ -149,7 +150,7 @@ public class WormAuditChainer
|
||||
insertLinks(store, appended);
|
||||
int linkCount = appended.size();
|
||||
appended.clear();
|
||||
details = AuditStore.index(store.lateDetails(cutoff, config.getLateDetailLookbackHours(), config.getBatchSize()));
|
||||
details = AuditStore.index(store.lateDetails(cutoff, settings.late_detail_lookback_hours(), config.getBatchSize()));
|
||||
grouped = details.values().stream().collect(Collectors.groupingBy(r -> r.getValueLong("auditId"), TreeMap::new, Collectors.toList()));
|
||||
Map<Long, QRecord> headers = AuditStore.index(store.in("audit", "id", grouped.keySet()));
|
||||
for(var entry : grouped.entrySet())
|
||||
@@ -165,7 +166,7 @@ public class WormAuditChainer
|
||||
}
|
||||
}
|
||||
insertLinks(store, appended);
|
||||
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now), false));
|
||||
return (new Counts(audits.size(), lateDetails, linkCount + appended.size(), checkpoint(store, now, settings), false));
|
||||
}
|
||||
|
||||
|
||||
@@ -195,7 +196,7 @@ public class WormAuditChainer
|
||||
/*******************************************************************************
|
||||
** Checkpoint the uncheckpointed suffix by count or elapsed time.
|
||||
*******************************************************************************/
|
||||
private int checkpoint(AuditStore store, Instant now) throws Exception
|
||||
private int checkpoint(AuditStore store, Instant now, WormAuditSettings.Values settings) throws Exception
|
||||
{
|
||||
if(config.getSigningKey() == null)
|
||||
{
|
||||
@@ -206,13 +207,13 @@ public class WormAuditChainer
|
||||
int count = 0;
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, config.getCheckpointEveryLinks());
|
||||
List<QRecord> pending = store.page(WormAuditTables.CHAIN, lastId, settings.checkpoint_every_links());
|
||||
if(pending.isEmpty())
|
||||
{
|
||||
return (count);
|
||||
}
|
||||
Instant since = latest == null ? pending.getFirst().getValueInstant("chained_at") : latest.getValueInstant("created_at");
|
||||
if(pending.size() < config.getCheckpointEveryLinks() && now.isBefore(since.plusSeconds(config.getCheckpointEveryMinutes() * 60L)))
|
||||
if(pending.size() < settings.checkpoint_every_links() && now.isBefore(since.plusSeconds(settings.checkpoint_every_minutes() * 60L)))
|
||||
{
|
||||
return (count);
|
||||
}
|
||||
|
||||
@@ -102,6 +102,13 @@ public final class WormAuditDdl
|
||||
{
|
||||
sql = sql.replace("${" + entry.getValue() + "}", identifier(names.get(entry.getKey())));
|
||||
}
|
||||
var defaults = com.kingsrook.qbits.wormaudit.WormAuditSettings.defaults(config);
|
||||
com.kingsrook.qbits.wormaudit.WormAuditSettings.validate(defaults);
|
||||
String values = java.util.List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days", "checkpoint_every_links",
|
||||
"checkpoint_every_minutes", "chain_interval_seconds", "settle_horizon_seconds", "late_detail_lookback_hours", "export_cron", "verify_cron")
|
||||
.stream().map(field -> defaults.getValue(field) instanceof Number ? defaults.getValue(field).toString()
|
||||
: "'" + defaults.getValueString(field).replace("'", "''") + "'").collect(java.util.stream.Collectors.joining(", "));
|
||||
sql = sql.replace("${settings_values}", values);
|
||||
return (sql.replace("${app_role}", identifier(config.getAppRole())).replace("${retention}", config.getRetentionYears() + " years"));
|
||||
}
|
||||
catch(IOException e)
|
||||
|
||||
@@ -152,9 +152,9 @@ public class ExportFiles implements AutoCloseable
|
||||
/*******************************************************************************
|
||||
** A lock echo is required even if a provider advertises the capability.
|
||||
*******************************************************************************/
|
||||
public static boolean locked(StoredObject object, java.time.Instant until)
|
||||
public static boolean locked(StoredObject object, String mode, java.time.Instant until)
|
||||
{
|
||||
return (object != null && "COMPLIANCE".equals(object.getLockMode()) && object.getLockRetainUntil() != null
|
||||
return (object != null && java.util.Objects.equals(mode, object.getLockMode()) && object.getLockRetainUntil() != null
|
||||
&& !object.getLockRetainUntil().isBefore(until));
|
||||
}
|
||||
|
||||
|
||||
@@ -29,7 +29,6 @@ import java.time.Instant;
|
||||
import com.kingsrook.qbits.objectstorage.api.Capability;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.LockMode;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
|
||||
import com.kingsrook.qbits.objectstorage.api.Page;
|
||||
@@ -84,12 +83,12 @@ public class ObjectStorageSealedStore implements SealedStore
|
||||
** Stream a bounded upload and close the source even when the provider fails.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String put(String key, Path file, Instant retainUntil) throws Exception
|
||||
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception
|
||||
{
|
||||
try(InputStream input = Files.newInputStream(file))
|
||||
{
|
||||
return (bucket.put(PutRequest.of(key, input).withContentLength(Files.size(file))
|
||||
.withObjectLock(LockMode.COMPLIANCE, retainUntil)).getVersionId());
|
||||
.withObjectLock(mode, retainUntil)).getVersionId());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -94,6 +94,7 @@ public class SealedExportVerifier
|
||||
*******************************************************************************/
|
||||
public Report verify(QRecord export, String trigger) throws QException
|
||||
{
|
||||
config.effectiveSettings();
|
||||
VerificationResult failure = new VerificationResult();
|
||||
failure.scope(0, export.getValueLong("from_link_id"), export.getValueLong("to_link_id"));
|
||||
Report report;
|
||||
@@ -121,22 +122,34 @@ public class SealedExportVerifier
|
||||
throw (new IOException("Missing sealed object version"));
|
||||
}
|
||||
var head = storage.head(key);
|
||||
if(head == null)
|
||||
{
|
||||
failure.add(null, null, "object " + key + " is missing" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
|
||||
? "; possible GOVERNANCE bypass removal by an administrator" : ""));
|
||||
}
|
||||
if(head != null && !version.equals(head.getVersionId()))
|
||||
{
|
||||
failure.add(null, null, "object " + name + " was overwritten after sealing (current version " + head.getVersionId()
|
||||
+ ", sealed version " + version + " is still retained under Object Lock)");
|
||||
}
|
||||
if(!ExportFiles.locked(head, export.getValueInstant("retain_until")))
|
||||
if(!ExportFiles.locked(head, export.getValueString("lock_mode"), export.getValueInstant("retain_until")))
|
||||
{
|
||||
failure.add(null, null, "object " + name + " lacks the recorded COMPLIANCE retention");
|
||||
failure.add(null, null, "object " + name + " lacks the recorded " + export.getValueString("lock_mode") + " retention");
|
||||
}
|
||||
}
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
try(InputStream input = storage.getVersion(export.getValueString("object_prefix") + name, (String) versions.get(name)))
|
||||
String key = export.getValueString("object_prefix") + name;
|
||||
try(InputStream input = storage.getVersion(key, (String) versions.get(name)))
|
||||
{
|
||||
Files.copy(input, files.file(name));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
failure.add(null, null, "object " + key + " sealed version cannot be read" + ("GOVERNANCE".equals(export.getValueString("lock_mode"))
|
||||
? "; if removed, possible GOVERNANCE bypass removal by an administrator" : ""));
|
||||
throw (new IOException("Cannot read sealed object version"));
|
||||
}
|
||||
}
|
||||
report = inspect(files, export, new AuditStore(transaction));
|
||||
for(var item : failure.getBreaks())
|
||||
@@ -189,7 +202,7 @@ public class SealedExportVerifier
|
||||
|| !Objects.equals(manifest.get("fromLinkId"), from) || !Objects.equals(manifest.get("toLinkId"), to)
|
||||
|| !Objects.equals(manifest.get("linkCount"), to - from + 1)
|
||||
|| !Objects.equals(manifest.get("createdAt"), export.getValueInstant("sealed_at").toString())
|
||||
|| !"COMPLIANCE".equals(export.getValueString("lock_mode"))
|
||||
|| !java.util.Set.of("GOVERNANCE", "COMPLIANCE").contains(export.getValueString("lock_mode"))
|
||||
|| !Objects.equals(export.getValueString("manifest_key"), export.getValueString("object_prefix") + "manifest.json"))
|
||||
{
|
||||
result.add(null, null, "Manifest metadata does not match the export row");
|
||||
@@ -256,7 +269,7 @@ public class SealedExportVerifier
|
||||
long start = checkpoint.getValueLong("from_link_id");
|
||||
long count = checkpoint.getValueLong("link_count");
|
||||
if(start != cursor + 1 || end < start || end > to || count != end - start + 1
|
||||
|| count > config.getCheckpointEveryLinks() || checkpointId <= lastCheckpoint)
|
||||
|| count > Integer.MAX_VALUE || checkpointId <= lastCheckpoint)
|
||||
{
|
||||
result.add(null, null, label + "invalid or incomplete checkpoint range");
|
||||
break;
|
||||
|
||||
@@ -243,7 +243,9 @@ public class SealedExporter
|
||||
{
|
||||
throw (new ExportFailure("Export refused: bucket does not support Object Lock"));
|
||||
}
|
||||
Instant until = now.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant();
|
||||
var settings = config.effectiveSettings();
|
||||
var mode = com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(settings.export_lock_mode());
|
||||
Instant until = now.plus(settings.export_retention_days(), ChronoUnit.DAYS);
|
||||
String prefix = config.getExportPrefix() + "/" + DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(now)
|
||||
+ "/" + scope.from() + "-" + scope.to() + "/";
|
||||
QRecord row;
|
||||
@@ -270,7 +272,7 @@ public class SealedExporter
|
||||
.withValue("bucket_alias", config.getExportBucketAlias()).withValue("object_prefix", prefix)
|
||||
.withValue("manifest_key", prefix + "manifest.json").withValue("manifest_sha256", ExportFiles.sha256(files.file("manifest.json")))
|
||||
.withValue("prev_manifest_sha256", previous).withValue("object_count", 3L)
|
||||
.withValue("lock_mode", "COMPLIANCE").withValue("retain_until", until).withValue("sealed_at", now);
|
||||
.withValue("lock_mode", mode.name()).withValue("retain_until", until).withValue("sealed_at", now);
|
||||
if(!new SealedExportVerifier(config, storage).inspect(files, row, store).result().passed())
|
||||
{
|
||||
throw (new ExportFailure("Export refused: source does not match the signed chain"));
|
||||
@@ -279,15 +281,15 @@ public class SealedExporter
|
||||
Map<String, String> versions = new LinkedHashMap<>();
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String version = storage.put(prefix + name, files.file(name), until);
|
||||
String version = storage.put(prefix + name, files.file(name), mode, until);
|
||||
if(!ExportFiles.validVersion(version))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: object " + name + " returned no version id"));
|
||||
}
|
||||
var headObject = storage.head(prefix + name);
|
||||
if(!ExportFiles.locked(headObject, until))
|
||||
if(!ExportFiles.locked(headObject, mode.name(), until))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: object " + name + " did not echo COMPLIANCE retention"));
|
||||
throw (new ExportFailure("Export refused: object " + name + " did not echo " + mode.name() + " retention"));
|
||||
}
|
||||
if(!version.equals(headObject.getVersionId()))
|
||||
{
|
||||
|
||||
@@ -50,9 +50,9 @@ public interface SealedStore
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Upload a file with COMPLIANCE retention.
|
||||
** Upload a file with the selected Object Lock mode and retention.
|
||||
*******************************************************************************/
|
||||
String put(String key, Path file, Instant retainUntil) throws Exception;
|
||||
String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant retainUntil) throws Exception;
|
||||
|
||||
/*******************************************************************************
|
||||
** Read current object metadata, or null for an absent object.
|
||||
|
||||
@@ -45,6 +45,13 @@ public class AppendOnlyGuard implements MetaDataCustomizerInterface<QTableMetaDa
|
||||
@Override
|
||||
public QTableMetaData customizeMetaData(QInstance instance, QTableMetaData table)
|
||||
{
|
||||
if(com.kingsrook.qbits.wormaudit.model.WormAuditTables.SETTINGS.equals(table.getName()))
|
||||
{
|
||||
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE);
|
||||
table.withAuditRules(new com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules()
|
||||
.withAuditLevel(com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel.FIELD));
|
||||
return (table);
|
||||
}
|
||||
if(CORE_TABLES.contains(table.getName()) || table.getName().startsWith("worm_audit_"))
|
||||
{
|
||||
table.withoutCapabilities(Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.model;
|
||||
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validate the editable singleton, including internal insert-only seeds.
|
||||
*******************************************************************************/
|
||||
public class SettingsCustomizer implements TableCustomizerInterface
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Only the fixed singleton may be inserted by the internal seed path.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> preInsert(InsertInput input, List<QRecord> records, boolean preview) throws QException
|
||||
{
|
||||
if(records.size() != 1 || !Integer.valueOf(1).equals(records.getFirst().getValueInteger("id")))
|
||||
{
|
||||
throw (new QUserFacingException("WORM Audit Settings has one row only (ID 1)."));
|
||||
}
|
||||
if(new com.kingsrook.qbits.wormaudit.chain.AuditStore(input.getTransaction()).get(WormAuditTables.SETTINGS, 1) != null)
|
||||
{
|
||||
throw (new QUserFacingException("WORM Audit Settings already exists. Edit record 1 instead of adding another record."));
|
||||
}
|
||||
return (validate(records));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validate partial edits without replacing the existing settings.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> preUpdate(UpdateInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords) throws QException
|
||||
{
|
||||
PermissionsHelper.checkTablePermissionThrowing(input, TablePermissionSubType.EDIT);
|
||||
return (validate(records));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reject deletion even for direct core calls where capability checks are off.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> preDelete(DeleteInput input, List<QRecord> records, boolean preview) throws QException
|
||||
{
|
||||
throw (new QUserFacingException("WORM Audit Settings cannot be deleted."));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Refresh local settings after persistence; remote JVMs retain the bounded cache.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> postUpdate(UpdateInput input, List<QRecord> records, Optional<List<QRecord>> oldRecords) throws QException
|
||||
{
|
||||
com.kingsrook.qbits.wormaudit.processes.VerificationRuns.config().invalidateSettings();
|
||||
return (records);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stamp the editor and time on every valid write; core writes FIELD audits.
|
||||
*******************************************************************************/
|
||||
private List<QRecord> validate(List<QRecord> records) throws QException
|
||||
{
|
||||
for(QRecord row : records)
|
||||
{
|
||||
try
|
||||
{
|
||||
WormAuditSettings.validate(row);
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
throw (new QUserFacingException(e.getMessage()));
|
||||
}
|
||||
row.setValue("modify_date", Instant.now());
|
||||
row.setValue("modified_by", QContext.getQSession() == null || QContext.getQSession().getUser() == null
|
||||
? "system" : QContext.getQSession().getUser().getIdReference());
|
||||
}
|
||||
return (records);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.model;
|
||||
|
||||
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reading uses the full table name; editing additionally needs settings access.
|
||||
*******************************************************************************/
|
||||
public class SettingsPermissions implements CustomPermissionChecker
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Enforce table permission and the separate settings permission.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
|
||||
{
|
||||
var session = QContext.getQSession();
|
||||
boolean edit = input instanceof UpdateInput;
|
||||
if(input instanceof InsertInput || input instanceof DeleteInput || session == null
|
||||
|| !session.hasPermission(WormAuditTables.SETTINGS + (edit ? ".edit" : ".read"))
|
||||
|| (edit && !session.hasPermission("worm_audit.settings")))
|
||||
{
|
||||
throw (new QPermissionDeniedException(edit ? "Editing WORM Audit Settings requires worm_audit_settings.edit and worm_audit.settings."
|
||||
: "WORM Audit Settings requires read access; adding or deleting settings is not allowed."));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -24,16 +24,21 @@ package com.kingsrook.qbits.wormaudit.model;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditSettings;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
|
||||
@@ -51,8 +56,9 @@ public final class WormAuditTables
|
||||
public static final String EXPORT = "worm_audit_export";
|
||||
public static final String VERIFICATION = "worm_audit_verification";
|
||||
public static final String PURGE = "worm_audit_purge";
|
||||
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE);
|
||||
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE);
|
||||
public static final String SETTINGS = "worm_audit_settings";
|
||||
public static final List<String> ALL = List.of(CHAIN, CHECKPOINT, EXPORT, VERIFICATION, PURGE, SETTINGS);
|
||||
public static final List<String> NAV = List.of(VERIFICATION, EXPORT, CHECKPOINT, CHAIN, PURGE, SETTINGS);
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
@@ -73,10 +79,72 @@ public final class WormAuditTables
|
||||
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T object_versions:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
|
||||
tables.add(table(config, VERIFICATION, "Verification", "id:L kind:S trigger:S started_at:D finished_at:D from_link_id:L to_link_id:L links_checked:L outcome:S break_count:L first_break_link_id:L first_break_audit_id:L first_break_reason:T export_id:L run_by:S"));
|
||||
tables.add(table(config, PURGE, "Purge", "id:L requested_at:D requested_by:S cutoff:D retention_interval:S outcome:S reason:T audits_deleted:L details_deleted:L links_deleted:L anchor_link_id:L anchor_hash:S"));
|
||||
tables.add(settings(config));
|
||||
return (tables);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Editable, FIELD-audited singleton with standard QQQ record sections.
|
||||
*******************************************************************************/
|
||||
public static QTableMetaData settings(WormAuditQBitConfig config)
|
||||
{
|
||||
QTableMetaData table = new QTableMetaData().withName(SETTINGS).withLabel("WORM Audit Settings")
|
||||
.withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(SETTINGS))
|
||||
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("WORM Audit Settings")
|
||||
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
|
||||
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS)
|
||||
.withPermissionBaseName(SETTINGS).withCustomPermissionChecker(new QCodeReference(SettingsPermissions.class)))
|
||||
.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_DELETE)
|
||||
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SettingsCustomizer.class))
|
||||
.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
|
||||
.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, new QCodeReference(SettingsCustomizer.class))
|
||||
.withCustomizer(TableCustomizers.PRE_DELETE_RECORD, new QCodeReference(SettingsCustomizer.class))
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withLabel("ID").withBackendName("id").withIsEditable(false).withDefaultValue(1));
|
||||
var defaults = WormAuditSettings.defaults(config);
|
||||
table.withField(new QFieldMetaData("audit_retention_years", QFieldType.INTEGER).withBackendName("audit_retention_years").withLabel("Audit retention years")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("audit_retention_years")));
|
||||
table.withField(new QFieldMetaData("export_lock_mode", QFieldType.STRING).withBackendName("export_lock_mode").withLabel("Export lock mode")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("export_lock_mode")));
|
||||
table.withField(new QFieldMetaData("export_retention_days", QFieldType.INTEGER).withBackendName("export_retention_days").withLabel("Export retention days")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("export_retention_days")));
|
||||
table.withField(new QFieldMetaData("checkpoint_every_links", QFieldType.INTEGER).withBackendName("checkpoint_every_links").withLabel("Checkpoint every links")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_links")));
|
||||
table.withField(new QFieldMetaData("checkpoint_every_minutes", QFieldType.INTEGER).withBackendName("checkpoint_every_minutes").withLabel("Checkpoint every minutes")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("checkpoint_every_minutes")));
|
||||
table.withField(new QFieldMetaData("chain_interval_seconds", QFieldType.INTEGER).withBackendName("chain_interval_seconds").withLabel("Chain interval seconds")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("chain_interval_seconds")));
|
||||
table.withField(new QFieldMetaData("settle_horizon_seconds", QFieldType.INTEGER).withBackendName("settle_horizon_seconds").withLabel("Settle horizon seconds")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("settle_horizon_seconds")));
|
||||
table.withField(new QFieldMetaData("late_detail_lookback_hours", QFieldType.INTEGER).withBackendName("late_detail_lookback_hours").withLabel("Late detail lookback hours")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("late_detail_lookback_hours")));
|
||||
table.withField(new QFieldMetaData("export_cron", QFieldType.STRING).withBackendName("export_cron").withLabel("Export schedule (UTC)")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("export_cron")));
|
||||
table.withField(new QFieldMetaData("verify_cron", QFieldType.STRING).withBackendName("verify_cron").withLabel("Verify schedule (UTC)")
|
||||
.withIsRequired(true).withDefaultValue(defaults.getValue("verify_cron")));
|
||||
table.getField("export_lock_mode").withInlinePossibleValueSource(new QPossibleValueSource().withName(SETTINGS + "_lock_mode")
|
||||
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(
|
||||
new QPossibleValue<>("GOVERNANCE", "GOVERNANCE"), new QPossibleValue<>("COMPLIANCE", "COMPLIANCE"))));
|
||||
for(String field : List.of("export_cron", "verify_cron", "chain_interval_seconds"))
|
||||
{
|
||||
table.getField(field).withHelpContent(new QHelpContent("Takes effect after restart. Cron schedules use UTC and Quartz syntax."));
|
||||
}
|
||||
table.getField("audit_retention_years").withHelpContent(new QHelpContent("Used for the next eligible purge date. The database purge interval changes only through a reviewed migration."));
|
||||
table.withField(new QFieldMetaData("notes", QFieldType.TEXT).withLabel("Notes").withBackendName("notes"));
|
||||
table.withField(new QFieldMetaData("modify_date", QFieldType.DATE_TIME).withLabel("Modified").withBackendName("modify_date").withIsEditable(false));
|
||||
table.withField(new QFieldMetaData("modified_by", QFieldType.STRING).withLabel("Modified by").withBackendName("modified_by").withIsEditable(false));
|
||||
table.withSection(new QFieldSection("retention", "Retention", new QIcon("lock"), Tier.T1,
|
||||
List.of("id", "audit_retention_years", "export_lock_mode", "export_retention_days")).withGridColumns(12));
|
||||
table.withSection(new QFieldSection("cadence", "Chain and checkpoints", new QIcon("link"), Tier.T2,
|
||||
List.of("checkpoint_every_links", "checkpoint_every_minutes", "settle_horizon_seconds", "late_detail_lookback_hours")).withGridColumns(12));
|
||||
table.withSection(new QFieldSection("schedules", "Schedules (after restart)", new QIcon("schedule"), Tier.T2,
|
||||
List.of("chain_interval_seconds", "export_cron", "verify_cron")).withGridColumns(12));
|
||||
table.withSection(new QFieldSection("changes", "Changes", new QIcon("edit_note"), Tier.T2,
|
||||
List.of("notes", "modify_date", "modified_by")).withGridColumns(12));
|
||||
return (table);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Common metadata for a process-written table.
|
||||
*******************************************************************************/
|
||||
@@ -169,6 +237,7 @@ public final class WormAuditTables
|
||||
{
|
||||
return (switch(name)
|
||||
{
|
||||
case SETTINGS -> "Settings";
|
||||
case CHAIN -> "Audit Chain";
|
||||
case CHECKPOINT -> "Checkpoints";
|
||||
case EXPORT -> "Sealed Exports";
|
||||
|
||||
@@ -84,6 +84,7 @@ public final class ExportProcesses
|
||||
*******************************************************************************/
|
||||
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
|
||||
{
|
||||
var settings = config.effectiveSettings();
|
||||
Map<String, Serializable> values = new HashMap<>();
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.EXPORT);
|
||||
@@ -97,7 +98,7 @@ public final class ExportProcesses
|
||||
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily sealed audit export")
|
||||
.withStep(new QBackendStepMetaData().withName("export").withCode(new QCodeReference(Daily.class)))
|
||||
.withPermissionRules(permission(PERMISSION)).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
|
||||
.withCronExpression(config.getExportCron()).withCronTimeZoneId("UTC"));
|
||||
.withCronExpression(settings.export_cron()).withCronTimeZoneId("UTC"));
|
||||
return (List.of(export, verify, daily));
|
||||
}
|
||||
|
||||
|
||||
@@ -89,6 +89,7 @@ public final class WormAuditProcesses
|
||||
*******************************************************************************/
|
||||
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
|
||||
{
|
||||
var settings = config.effectiveSettings();
|
||||
Map<String, Serializable> values = new HashMap<>();
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.CHAIN);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.CHAIN);
|
||||
@@ -100,11 +101,11 @@ public final class WormAuditProcesses
|
||||
QProcessMetaData chain = new QProcessMetaData().withName(CHAIN).withLabel("Chain settled audit records")
|
||||
.withStep(new QBackendStepMetaData().withName("chain").withCode(new QCodeReference(ChainTick.class)))
|
||||
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
|
||||
.withRepeatSeconds(config.getChainIntervalSeconds()).withInitialDelaySeconds(config.getChainIntervalSeconds()));
|
||||
.withRepeatSeconds(settings.chain_interval_seconds()).withInitialDelaySeconds(settings.chain_interval_seconds()));
|
||||
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily audit chain verification")
|
||||
.withStep(new QBackendStepMetaData().withName("verify").withCode(new QCodeReference(Daily.class)))
|
||||
.withPermissionRules(permission()).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
|
||||
.withCronExpression(config.getVerifyCron()).withCronTimeZoneId("UTC"));
|
||||
.withCronExpression(settings.verify_cron()).withCronTimeZoneId("UTC"));
|
||||
return (List.of(verify, chain, daily));
|
||||
}
|
||||
|
||||
|
||||
@@ -133,6 +133,7 @@ public class WormAuditVerifier
|
||||
*******************************************************************************/
|
||||
VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
|
||||
{
|
||||
var settings = config.effectiveSettings();
|
||||
VerificationResult result = new VerificationResult(now);
|
||||
if(config.getSigningKey() == null)
|
||||
{
|
||||
@@ -214,7 +215,7 @@ public class WormAuditVerifier
|
||||
}
|
||||
if(selection == null)
|
||||
{
|
||||
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
|
||||
Instant cutoff = now.minusSeconds(settings.settle_horizon_seconds());
|
||||
if(store.isRdbms())
|
||||
{
|
||||
store.completeness(cutoff, result);
|
||||
@@ -500,7 +501,7 @@ public class WormAuditVerifier
|
||||
result.add(null, null, prefix + "signature does not verify");
|
||||
}
|
||||
if(from < 1 || to < from || checkpoint.getValueLong("link_count") != to - from + 1
|
||||
|| to - from + 1 > config.getCheckpointEveryLinks())
|
||||
|| to - from + 1 > Integer.MAX_VALUE)
|
||||
{
|
||||
result.add(null, null, prefix + "invalid link range or count");
|
||||
return;
|
||||
|
||||
@@ -324,3 +324,30 @@ CREATE INDEX IF NOT EXISTS worm_audit_chain_detail_coverage ON public.worm_audit
|
||||
DROP TRIGGER IF EXISTS worm_audit_detail_coverage_guard ON public.worm_audit_chain;
|
||||
CREATE TRIGGER worm_audit_detail_coverage_guard BEFORE INSERT ON public.worm_audit_chain
|
||||
FOR EACH ROW EXECUTE FUNCTION public.worm_audit_detail_coverage();
|
||||
|
||||
-- changeset worm-audit:10
|
||||
-- Editable singleton: deliberately excluded from append-only triggers.
|
||||
CREATE TABLE IF NOT EXISTS public.worm_audit_settings (
|
||||
id integer PRIMARY KEY CHECK (id = 1),
|
||||
audit_retention_years integer NOT NULL DEFAULT 6 CHECK (audit_retention_years BETWEEN 1 AND 100),
|
||||
export_lock_mode text NOT NULL DEFAULT 'GOVERNANCE' CHECK (export_lock_mode IN ('GOVERNANCE', 'COMPLIANCE')),
|
||||
export_retention_days integer NOT NULL DEFAULT 30 CHECK (export_retention_days BETWEEN 1 AND 36500),
|
||||
checkpoint_every_links integer NOT NULL DEFAULT 10000 CHECK (checkpoint_every_links > 0),
|
||||
checkpoint_every_minutes integer NOT NULL DEFAULT 60 CHECK (checkpoint_every_minutes > 0),
|
||||
chain_interval_seconds integer NOT NULL DEFAULT 30 CHECK (chain_interval_seconds > 0),
|
||||
settle_horizon_seconds integer NOT NULL DEFAULT 60 CHECK (settle_horizon_seconds >= 0),
|
||||
late_detail_lookback_hours integer NOT NULL DEFAULT 24 CHECK (late_detail_lookback_hours > 0),
|
||||
export_cron text NOT NULL DEFAULT '0 0 2 * * ?',
|
||||
verify_cron text NOT NULL DEFAULT '0 0 3 * * ?',
|
||||
notes text,
|
||||
modify_date timestamptz NOT NULL DEFAULT now(),
|
||||
modified_by text NOT NULL DEFAULT current_user
|
||||
);
|
||||
REVOKE DELETE, TRUNCATE ON public.worm_audit_settings FROM ${app_role};
|
||||
GRANT SELECT, INSERT, UPDATE ON public.worm_audit_settings TO ${app_role};
|
||||
INSERT INTO public.worm_audit_settings
|
||||
(id, audit_retention_years, export_lock_mode, export_retention_days, checkpoint_every_links,
|
||||
checkpoint_every_minutes, chain_interval_seconds, settle_horizon_seconds, late_detail_lookback_hours,
|
||||
export_cron, verify_cron)
|
||||
SELECT ${settings_values}
|
||||
WHERE NOT EXISTS (SELECT 1 FROM public.worm_audit_settings WHERE id = 1);
|
||||
@@ -78,7 +78,7 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
class MinioExportAcceptanceTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Production adapter, COMPLIANCE retention, version preservation and UI findings.
|
||||
** Production adapter, GOVERNANCE and COMPLIANCE retention, version preservation and UI findings.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void lockedRoundTripVersionRetentionLiveTamperAndUnsupportedBucket() throws Exception
|
||||
@@ -130,17 +130,17 @@ class MinioExportAcceptanceTest
|
||||
{
|
||||
String key = row.getValueString("object_prefix") + name;
|
||||
var head = storage.head(key);
|
||||
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
|
||||
assertThat(Duration.between(now.atZone(java.time.ZoneOffset.UTC).plusYears(6).toInstant(), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
|
||||
assertThat(head.getLockMode()).isEqualTo("GOVERNANCE");
|
||||
assertThat(Duration.between(now.plus(Duration.ofDays(30)), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
|
||||
String originalVersion = head.getVersionId();
|
||||
assertThat(originalVersion).isNotBlank();
|
||||
assertThat(ExportFiles.parse(row.getValueString("object_versions").getBytes(java.nio.charset.StandardCharsets.UTF_8)))
|
||||
.containsEntry(name, originalVersion);
|
||||
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)))
|
||||
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion).bypassGovernanceRetention(false)))
|
||||
.isInstanceOf(S3Exception.class);
|
||||
byte[] original = s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray();
|
||||
// S3 overwrites create new versions; they cannot replace the locked version.
|
||||
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("COMPLIANCE")
|
||||
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("GOVERNANCE")
|
||||
.objectLockRetainUntilDate(row.getValueInstant("retain_until")), RequestBody.fromString("different replacement bytes"));
|
||||
assertThat(replacement.versionId()).isNotEqualTo(originalVersion);
|
||||
assertThat(s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray()).isEqualTo(original);
|
||||
@@ -163,11 +163,26 @@ class MinioExportAcceptanceTest
|
||||
tamper("auditDetail", 1, "newValue", "new");
|
||||
audit(2);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
config.withExportLockMode("COMPLIANCE");
|
||||
var compliance = new SealedExporter(config).run("USER", Instant.now());
|
||||
assertThat(compliance.failed()).isFalse();
|
||||
assertThat(compliance.row().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
|
||||
assertThat(verifier.verify(compliance.row(), "USER").result().passed()).isTrue();
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String key = compliance.row().getValueString("object_prefix") + name;
|
||||
var head = storage.head(key);
|
||||
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
|
||||
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(head.getVersionId())))
|
||||
.isInstanceOf(S3Exception.class);
|
||||
}
|
||||
audit(3);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
config.withExportBucketAlias("plain");
|
||||
var failed = new SealedExporter(config).run("SCHEDULE", Instant.now());
|
||||
assertThat(failed.failed()).isTrue();
|
||||
assertThat(failed.message()).contains("does not support Object Lock");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1);
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(2);
|
||||
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(1);
|
||||
}
|
||||
finally
|
||||
|
||||
@@ -113,7 +113,6 @@ class PostgresAcceptanceTest
|
||||
void database() throws Exception
|
||||
{
|
||||
config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
|
||||
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
|
||||
try(var c = owner())
|
||||
{
|
||||
sql(
|
||||
@@ -124,6 +123,7 @@ class PostgresAcceptanceTest
|
||||
c,
|
||||
"DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname='app') THEN"
|
||||
+ " CREATE ROLE app LOGIN PASSWORD 'fixture'; END IF; END $$");
|
||||
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), "app", "fixture", config, true);
|
||||
RdbmsTestSupport.createTables(c, instance, true);
|
||||
String ddl = WormAuditDdl.postgres(config);
|
||||
sql(c, ddl);
|
||||
@@ -142,9 +142,17 @@ class PostgresAcceptanceTest
|
||||
{
|
||||
var storage = new com.kingsrook.qbits.wormaudit.demo.MemorySealedStore();
|
||||
var clean = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
|
||||
assertThat(clean.warnings()).containsExactly("Daily verification has never run");
|
||||
assertThat(clean.warnings()).contains("Daily verification has never run").noneMatch(w -> w.contains("guard") && w.contains("settings"));
|
||||
try(var connection = owner())
|
||||
{
|
||||
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=7 WHERE id=1");
|
||||
config.invalidateSettings();
|
||||
assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings())
|
||||
.anyMatch(w -> w.contains("purge function's compiled interval (6 years)"));
|
||||
sql(connection, WormAuditDdl.postgres(config));
|
||||
assertThat(count(connection, "SELECT audit_retention_years FROM worm_audit_settings WHERE id=1")).isEqualTo(7);
|
||||
sql(connection, "UPDATE worm_audit_settings SET audit_retention_years=6 WHERE id=1");
|
||||
config.invalidateSettings();
|
||||
sql(connection, "DROP TRIGGER worm_audit_row_guard ON audit_detail");
|
||||
sql(connection, "ALTER TABLE worm_audit_chain DISABLE TRIGGER worm_audit_truncate_guard");
|
||||
sql(connection, "DROP FUNCTION public.worm_audit_purge(timestamptz)");
|
||||
@@ -338,7 +346,7 @@ class PostgresAcceptanceTest
|
||||
void prefixCannotSplitAnAuditAndAdvisoryLockSkipsTick() throws Exception
|
||||
{
|
||||
// This retention fixture deliberately polls a detail for a six-year-old header.
|
||||
config.withLateDetailLookbackHours(24 * 365 * 10);
|
||||
setting("late_detail_lookback_hours", 24 * 365 * 10);
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
try(var c = owner())
|
||||
@@ -512,7 +520,7 @@ class PostgresAcceptanceTest
|
||||
void chainerUsesUtcCutoffInChicago() throws Exception
|
||||
{
|
||||
var original = java.util.TimeZone.getDefault();
|
||||
config.withSettleHorizonSeconds(60);
|
||||
setting("settle_horizon_seconds", 60);
|
||||
try(var connection = owner())
|
||||
{
|
||||
sql(connection, "ALTER TABLE audit ALTER COLUMN timestamp TYPE timestamp without time zone USING timestamp AT TIME ZONE 'UTC'");
|
||||
@@ -543,7 +551,7 @@ class PostgresAcceptanceTest
|
||||
@Test
|
||||
void postgresLateDetailWindowAndCompleteness() throws Exception
|
||||
{
|
||||
config.withSettleHorizonSeconds(60);
|
||||
setting("settle_horizon_seconds", 60);
|
||||
var now = java.time.Instant.now().truncatedTo(java.time.temporal.ChronoUnit.SECONDS);
|
||||
var cutoff = now.minusSeconds(60);
|
||||
try(var connection = owner(); var statement = connection.prepareStatement(
|
||||
@@ -574,4 +582,16 @@ class PostgresAcceptanceTest
|
||||
assertThat(new WormAuditVerifier(config).verify().getBreaks()).anyMatch(b -> b.reason().contains("audit_detail 1 is not covered"));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The settings row (seeded by the DDL) overrides config: change it there.
|
||||
*******************************************************************************/
|
||||
private void setting(String column, int value) throws Exception
|
||||
{
|
||||
try(var connection = owner())
|
||||
{
|
||||
sql(connection, "UPDATE worm_audit_settings SET " + column + "=" + value + " WHERE id=1");
|
||||
}
|
||||
config.invalidateSettings();
|
||||
}
|
||||
}
|
||||
@@ -55,7 +55,7 @@ class RecordIntegrationTest
|
||||
Map<String, String> formats = Map.of(
|
||||
WormAuditTables.CHAIN, "Link #%s", WormAuditTables.CHECKPOINT, "Checkpoint #%s",
|
||||
WormAuditTables.EXPORT, "Export #%s (links %s–%s)", WormAuditTables.VERIFICATION, "Verification #%s (%s)",
|
||||
WormAuditTables.PURGE, "Purge #%s (%s)");
|
||||
WormAuditTables.PURGE, "Purge #%s (%s)", WormAuditTables.SETTINGS, "WORM Audit Settings");
|
||||
for(String name : WormAuditTables.ALL)
|
||||
{
|
||||
var table = instance.getTable(name);
|
||||
|
||||
@@ -84,7 +84,7 @@ class SealedExportTest
|
||||
boolean omitVersion;
|
||||
boolean racePut;
|
||||
boolean supported = true;
|
||||
String echoMode = "COMPLIANCE";
|
||||
String echoMode;
|
||||
long shortenSeconds;
|
||||
int failPut;
|
||||
|
||||
@@ -95,7 +95,7 @@ class SealedExportTest
|
||||
}
|
||||
|
||||
@Override
|
||||
public String put(String key, Path file, Instant until) throws Exception
|
||||
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
|
||||
{
|
||||
if(failPut > 0 && objects.size() + 1 == failPut)
|
||||
{
|
||||
@@ -108,7 +108,7 @@ class SealedExportTest
|
||||
String version = omitVersion ? missingVersion : "v" + ++nextVersion;
|
||||
objects.put(key, Files.readAllBytes(file));
|
||||
heads.put(key, new StoredObject().withKey(key).withVersionId(racePut ? "replacement" : version)
|
||||
.withLockMode(echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
|
||||
.withLockMode(echoMode == null ? mode.name() : echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
|
||||
return (version);
|
||||
}
|
||||
|
||||
@@ -197,7 +197,7 @@ class SealedExportTest
|
||||
assertThat(row.getValues()).containsKeys("from_checkpoint_id", "to_checkpoint_id", "bucket_alias", "manifest_key", "manifest_sha256",
|
||||
"prev_manifest_sha256", "object_count", "byte_count", "lock_mode", "retain_until", "sealed_at");
|
||||
assertThat(row.getValueLong("byte_count")).isEqualTo(storage.objects.values().stream().mapToLong(b -> b.length).sum());
|
||||
assertThat(row.getValueInstant("retain_until")).isEqualTo(Instant.parse("2032-10-14T02:00:00Z"));
|
||||
assertThat(row.getValueInstant("retain_until")).isEqualTo(NOW.plusSeconds(30 * 86400L));
|
||||
assertThat(verify(row).result().passed()).isTrue();
|
||||
QRecord verification = STORE.last(WormAuditTables.VERIFICATION);
|
||||
assertThat(verification.getValueString("kind")).isEqualTo("SEALED_EXPORT");
|
||||
@@ -224,7 +224,7 @@ class SealedExportTest
|
||||
{
|
||||
String linksKey = row.getValueString("object_prefix") + "links.jsonl";
|
||||
Files.write(files.file("links.jsonl"), storage.objects.get(linksKey));
|
||||
storage.put(linksKey, files.file("links.jsonl"), row.getValueInstant("retain_until"));
|
||||
storage.put(linksKey, files.file("links.jsonl"), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
|
||||
var identical = verify(row);
|
||||
assertThat(identical.result().passed()).isFalse();
|
||||
assertThat(identical.result().getLinksChecked()).isEqualTo(1);
|
||||
@@ -234,7 +234,7 @@ class SealedExportTest
|
||||
{
|
||||
String key = row.getValueString("object_prefix") + name;
|
||||
Files.writeString(files.file(name), "different bytes");
|
||||
String current = storage.put(key, files.file(name), row.getValueInstant("retain_until"));
|
||||
String current = storage.put(key, files.file(name), com.kingsrook.qbits.objectstorage.api.LockMode.valueOf(row.getValueString("lock_mode")), row.getValueInstant("retain_until"));
|
||||
var report = verify(row);
|
||||
assertThat(report.result().passed()).isFalse();
|
||||
assertThat(report.result().getLinksChecked()).isEqualTo(1);
|
||||
@@ -330,9 +330,9 @@ class SealedExportTest
|
||||
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).contains("does not support Object Lock");
|
||||
assertThat(storage.objects).isEmpty();
|
||||
storage.supported = true;
|
||||
storage.echoMode = "GOVERNANCE";
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo COMPLIANCE");
|
||||
storage.echoMode = "COMPLIANCE";
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo GOVERNANCE");
|
||||
storage.echoMode = null;
|
||||
storage.shortenSeconds = 1;
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
|
||||
storage.shortenSeconds = 0;
|
||||
@@ -358,9 +358,9 @@ class SealedExportTest
|
||||
storage.objects.put(key, "tampered".getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(findings(first)).contains("object links.jsonl hash does not match the manifest");
|
||||
storage.objects.put(key, original);
|
||||
storage.heads.get(key).withLockMode("GOVERNANCE");
|
||||
assertThat(findings(first)).contains("lacks the recorded COMPLIANCE retention");
|
||||
storage.heads.get(key).withLockMode("COMPLIANCE");
|
||||
assertThat(findings(first)).contains("lacks the recorded GOVERNANCE retention");
|
||||
storage.heads.get(key).withLockMode(first.getValueString("lock_mode"));
|
||||
audit(2);
|
||||
QRecord second = seal();
|
||||
tamper(WormAuditTables.EXPORT, first.getValueLong("id"), "manifest_sha256", "f".repeat(64));
|
||||
|
||||
@@ -84,7 +84,7 @@ class WormAuditHomeTest
|
||||
assertThat(data.oldestAudit()).isNull();
|
||||
assertThat(data.nextPurge()).isNull();
|
||||
assertThat(data.exportAgeHours()).isNull();
|
||||
assertThat(data.warnings()).containsExactly("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
|
||||
assertThat(data.warnings()).contains("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
|
||||
String html = WormAuditHomeView.render(data, "/app");
|
||||
assertThat(html).contains("Chain status", "Export status", "Retention", "Recent purges", "Setup warnings",
|
||||
"No links chained yet", "No verification has run yet", "Unknown until verification runs", "No sealed exports yet",
|
||||
@@ -132,9 +132,9 @@ class WormAuditHomeTest
|
||||
VerificationRuns.save(new WormAuditVerifier(config).verify(null, now.plusSeconds(67)), "SCHEDULE", null, now.plusSeconds(68));
|
||||
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(69));
|
||||
assertThat(data.openBreak()).isNull();
|
||||
assertThat(data.warnings()).isEmpty();
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).doesNotContain("Setup warnings");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("No open chain breaks reported");
|
||||
assertThat(data.warnings()).hasSize(2);
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).contains("Setup warnings", "GOVERNANCE", "30 days");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Setup needs attention");
|
||||
}
|
||||
|
||||
|
||||
@@ -164,8 +164,8 @@ class WormAuditHomeTest
|
||||
assertThat(data.exportGaps()).isEmpty();
|
||||
assertThat(data.lockCapability()).isEqualTo("SUPPORTED");
|
||||
assertThat(data.bucketAlias()).isEqualTo("worm_audit");
|
||||
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "COMPLIANCE", "Retain until");
|
||||
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("GOVERNANCE");
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "GOVERNANCE", "Retain until");
|
||||
// Corruption fixture only: the real exporter never creates this hole.
|
||||
tamper(WormAuditTables.EXPORT, 1, "from_link_id", 3L);
|
||||
assertThat(WormAuditHomeData.load(config, storage, now).exportGaps()).containsExactly("1–2");
|
||||
@@ -270,7 +270,7 @@ class WormAuditHomeTest
|
||||
assertThat(instance.getApp("evidence").getWidgets()).containsExactly(WormAuditApp.HOME_WIDGET);
|
||||
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getName()).containsExactlyElementsOf(WormAuditTables.NAV);
|
||||
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getLabel())
|
||||
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges");
|
||||
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges", "Settings");
|
||||
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getGridColumns()).isEqualTo(12);
|
||||
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getIsCard()).isFalse();
|
||||
assertThat(instance.getWidget(WormAuditApp.HEALTH_CARD_WIDGET).getIsCard()).isTrue();
|
||||
@@ -300,7 +300,7 @@ class WormAuditHomeTest
|
||||
assertThat(data.links()).isEqualTo(1);
|
||||
assertThat(data.backlog()).isEqualTo(1);
|
||||
assertThat(data.settling()).isEqualTo(1);
|
||||
assertThat(data.warnings()).containsExactly("Daily verification has never run");
|
||||
assertThat(data.warnings()).contains("Daily verification has never run");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -325,7 +325,7 @@ class WormAuditHomeTest
|
||||
assertThat(data.exportLag()).isEqualTo(5);
|
||||
assertThat(data.backlog()).isEqualTo(2);
|
||||
assertThat(data.settling()).isEqualTo(1);
|
||||
assertThat(data.warnings()).isEmpty();
|
||||
assertThat(data.warnings()).hasSize(2);
|
||||
assertThat(data.recentPurges()).extracting(r -> r.getValueString("outcome")).containsExactly("REFUSED", "PURGED");
|
||||
assertThat(STORE.where(WormAuditTables.VERIFICATION, "outcome", "FAIL")).hasSize(1).allSatisfy(row ->
|
||||
{
|
||||
|
||||
@@ -0,0 +1,307 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.time.Clock;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneId;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
|
||||
import com.kingsrook.qbits.wormaudit.model.SettingsCustomizer;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
|
||||
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.*;
|
||||
import static org.assertj.core.api.Assertions.*;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Editable settings, bounded cache, audited changes and historical lock modes.
|
||||
*******************************************************************************/
|
||||
class WormAuditSettingsTest
|
||||
{
|
||||
static final String TABLE = WormAuditTables.SETTINGS;
|
||||
|
||||
/*******************************************************************************
|
||||
** Deterministic clock for the exact 30-second cache boundary.
|
||||
*******************************************************************************/
|
||||
static class MutableClock extends Clock
|
||||
{
|
||||
Instant now = Instant.parse("2026-10-14T02:00:00Z");
|
||||
/*******************************************************************************
|
||||
** Verify get zone.
|
||||
*******************************************************************************/
|
||||
public ZoneId getZone()
|
||||
{
|
||||
return (ZoneOffset.UTC);
|
||||
}
|
||||
/*******************************************************************************
|
||||
** Verify with zone.
|
||||
*******************************************************************************/
|
||||
public Clock withZone(ZoneId zone)
|
||||
{
|
||||
return (this);
|
||||
}
|
||||
/*******************************************************************************
|
||||
** Verify instant.
|
||||
*******************************************************************************/
|
||||
public Instant instant()
|
||||
{
|
||||
return (now);
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify editor.
|
||||
*******************************************************************************/
|
||||
static void editor()
|
||||
{
|
||||
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
|
||||
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit", "worm_audit.settings"));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify update.
|
||||
*******************************************************************************/
|
||||
static void update(String field, java.io.Serializable value) throws Exception
|
||||
{
|
||||
var result = new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue(field, value))));
|
||||
assertThat(result.getRecords()).allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify fallback overrides cache and insert only seed.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void fallbackOverridesCacheAndInsertOnlySeed() throws Exception
|
||||
{
|
||||
var config = new WormAuditQBitConfig().withRetentionYears(8).withExportRetentionDays(45).withCheckpointEveryLinks(12);
|
||||
var instance = init(config);
|
||||
var clock = new MutableClock();
|
||||
var reader = new WormAuditSettings(config, clock);
|
||||
assertThat(reader.read().audit_retention_years()).isEqualTo(8);
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(45);
|
||||
WormAuditSettings.seed(config);
|
||||
editor();
|
||||
update("export_retention_days", 90);
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(90);
|
||||
update("export_retention_days", 120);
|
||||
clock.now = clock.now.plusSeconds(29);
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(90);
|
||||
clock.now = clock.now.plusSeconds(1);
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(120);
|
||||
WormAuditSettings.seed(config);
|
||||
assertThat(STORE.get(TABLE, 1).getValueInteger("export_retention_days")).isEqualTo(120);
|
||||
assertThat(reader.read().checkpoint_every_links()).isEqualTo(12);
|
||||
instance.getTables().remove(TABLE);
|
||||
reader.invalidate();
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(45);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify partial null row uses config and invalid persisted settings fail closed.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void partialNullRowUsesConfigAndInvalidPersistedSettingsFailClosed() throws Exception
|
||||
{
|
||||
var config = new WormAuditQBitConfig().withExportRetentionDays(42);
|
||||
init(config);
|
||||
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().insert(
|
||||
new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 1).withValue("export_lock_mode", "COMPLIANCE")), false);
|
||||
var reader = new WormAuditSettings(config);
|
||||
assertThat(reader.read().export_retention_days()).isEqualTo(42);
|
||||
assertThat(reader.read().export_lock_mode()).isEqualTo("COMPLIANCE");
|
||||
com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore.getInstance().update(
|
||||
new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_retention_days", 0))), false);
|
||||
reader.invalidate();
|
||||
assertThatThrownBy(reader::read).hasMessageContaining("Export retention days");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify missing physical table falls back.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void missingPhysicalTableFallsBack() throws Exception
|
||||
{
|
||||
var config = new WormAuditQBitConfig().withExportRetentionDays(81);
|
||||
var instance = RdbmsTestSupport.instance("jdbc:h2:mem:settings_missing;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE", "sa", "", config, false);
|
||||
QContext.init(instance, new QSession());
|
||||
assertThat(config.effectiveSettings().export_retention_days()).isEqualTo(81);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify validation and capabilities.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void validationAndCapabilities() throws Exception
|
||||
{
|
||||
var config = new WormAuditQBitConfig();
|
||||
var instance = init(config);
|
||||
var table = instance.getTable(TABLE);
|
||||
assertThat(table.getLabel()).isEqualTo("WORM Audit Settings");
|
||||
assertThat(table.getAuditRules().getAuditLevel()).isEqualTo(AuditLevel.FIELD);
|
||||
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_UPDATE)).isTrue();
|
||||
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_INSERT)).isFalse();
|
||||
assertThat(table.isCapabilityEnabled(instance.getBackendForTable(TABLE), Capability.TABLE_DELETE)).isFalse();
|
||||
WormAuditSettings.seed(config);
|
||||
editor();
|
||||
assertThatThrownBy(() -> new DeleteAction().execute(new DeleteInput(TABLE).withPrimaryKeys(List.of(1)))).isInstanceOf(Exception.class);
|
||||
assertThatThrownBy(() -> new InsertAction().execute(new InsertInput(TABLE).withRecord(new QRecord().withValue("id", 2)))).isInstanceOf(Exception.class);
|
||||
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(new QRecord().withValue("id", 2)), false)).hasMessageContaining("one row only (ID 1)");
|
||||
assertThatThrownBy(() -> new SettingsCustomizer().preInsert(new InsertInput(TABLE), List.of(WormAuditSettings.defaults(config)), false))
|
||||
.hasMessageContaining("already exists");
|
||||
assertThatThrownBy(() -> new SettingsCustomizer().preDelete(new DeleteInput(TABLE), List.of(STORE.get(TABLE, 1)), false))
|
||||
.hasMessageContaining("cannot be deleted");
|
||||
for(String field : List.of("audit_retention_years", "export_retention_days", "checkpoint_every_links", "checkpoint_every_minutes", "chain_interval_seconds", "late_detail_lookback_hours"))
|
||||
{
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue(field, 0))).hasMessageContaining("whole number");
|
||||
}
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("settle_horizon_seconds", -1))).hasMessageContaining("Settle horizon seconds");
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("audit_retention_years", 101))).hasMessageContaining("100");
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", 36501))).hasMessageContaining("36500");
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_retention_days", "1.5"))).hasMessageContaining("whole number");
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", "NONE"))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
|
||||
assertThatThrownBy(() -> WormAuditSettings.validate(new QRecord().withValue("export_lock_mode", null))).hasMessageContaining("GOVERNANCE or COMPLIANCE");
|
||||
for(String field : List.of("export_cron", "verify_cron"))
|
||||
{
|
||||
assertThatThrownBy(() -> new SettingsCustomizer().preUpdate(new UpdateInput(TABLE), List.of(new QRecord().withValue("id", 1).withValue(field, "bad")), false, java.util.Optional.empty()))
|
||||
.hasMessageContaining("valid Quartz cron");
|
||||
}
|
||||
assertThatThrownBy(() -> new UpdateAction().execute(new UpdateInput(TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("export_cron", "bad")))))
|
||||
.hasMessageContaining("valid Quartz cron");
|
||||
assertThat(STORE.get(TABLE, 1).getValueString("export_cron")).isEqualTo(config.getExportCron());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify edit requires both permissions and produces field audit that chains.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void editRequiresBothPermissionsAndProducesFieldAuditThatChains() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed().withSettleHorizonSeconds(0);
|
||||
init(config);
|
||||
WormAuditSettings.seed(config);
|
||||
QContext.getQInstance().withEnforcePermissionsInCoreActions(true);
|
||||
QContext.getQSession().withPermissions(Set.of(TABLE + ".read", TABLE + ".edit"));
|
||||
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit.settings");
|
||||
QContext.getQSession().setPermissions(new java.util.HashSet<>(Set.of("worm_audit.settings")));
|
||||
assertThatThrownBy(() -> update("export_lock_mode", "COMPLIANCE")).hasMessageContaining("worm_audit_settings.edit");
|
||||
editor();
|
||||
update("export_lock_mode", "COMPLIANCE");
|
||||
assertThat(STORE.all("audit")).hasSize(1);
|
||||
assertThat(STORE.where("auditDetail", "fieldName", "export_lock_mode")).hasSize(1);
|
||||
assertThat(STORE.get(TABLE, 1).getValueInstant("modify_date")).isNotNull();
|
||||
assertThat(STORE.get(TABLE, 1).getValueString("modified_by")).isEqualTo("system");
|
||||
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(1));
|
||||
assertThat(STORE.all(WormAuditTables.CHAIN)).hasSize(1);
|
||||
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(1)).passed()).isTrue();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify startup schedules and home use effective settings.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void startupSchedulesAndHomeUseEffectiveSettings() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed();
|
||||
var instance = init(config);
|
||||
WormAuditSettings.seed(config);
|
||||
editor();
|
||||
update("chain_interval_seconds", 45);
|
||||
update("export_cron", "0 15 4 * * ?");
|
||||
update("verify_cron", "0 30 5 * * ?");
|
||||
update("audit_retention_years", 7);
|
||||
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(30);
|
||||
var restarted = WormAuditTestSupport.core(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData().withName("memory")
|
||||
.withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class));
|
||||
new WormAuditQBitProducer().withQBitConfig(config).produce(restarted);
|
||||
assertThat(QContext.getQInstance()).isSameAs(instance);
|
||||
instance = restarted;
|
||||
QContext.init(instance, new QSession());
|
||||
assertThat(instance.getProcess(WormAuditProcesses.CHAIN).getSchedule().getRepeatSeconds()).isEqualTo(45);
|
||||
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 15 4 * * ?");
|
||||
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 30 5 * * ?");
|
||||
assertThat(instance.getProcess(WormAuditProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC");
|
||||
audit(50);
|
||||
var home = WormAuditHomeData.load(config, new SealedExportTest.MemorySealedStore(), Instant.now());
|
||||
assertThat(home.nextPurge()).isEqualTo(OLD.plusSeconds(50).atZone(ZoneOffset.UTC).plusYears(7).toInstant());
|
||||
assertThat(home.warnings()).anyMatch(w -> w.contains("differs from") && w.contains("purge interval"));
|
||||
assertThat(WormAuditHomeView.render(home, "/app")).contains("GOVERNANCE", "s3:BypassGovernanceRetention", "Switch to COMPLIANCE", "30 days", "Export retention is shorter");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify switch affects new exports only and old checkpoint cadence remains valid.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void switchAffectsNewExportsOnlyAndOldCheckpointCadenceRemainsValid() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed().withCheckpointEveryLinks(2).withSettleHorizonSeconds(0);
|
||||
init(config);
|
||||
WormAuditSettings.seed(config);
|
||||
var storage = new SealedExportTest.MemorySealedStore();
|
||||
audit(1);
|
||||
audit(2);
|
||||
Instant now = Instant.now().plusSeconds(1);
|
||||
new WormAuditChainer(config).chainOnce(now);
|
||||
var exporter = new SealedExporter(config, storage);
|
||||
var first = exporter.run("USER", now).row();
|
||||
assertThat(first).isNotNull();
|
||||
assertThat(first.getValueString("lock_mode")).isEqualTo("GOVERNANCE");
|
||||
assertThat(first.getValueInstant("retain_until")).isEqualTo(now.truncatedTo(java.time.temporal.ChronoUnit.MILLIS).plusSeconds(30 * 86400L));
|
||||
assertThat(storage.heads.values()).allSatisfy(h -> assertThat(h.getLockMode()).isEqualTo("GOVERNANCE"));
|
||||
var verifier = new SealedExportVerifier(config, storage);
|
||||
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
|
||||
editor();
|
||||
update("export_lock_mode", "COMPLIANCE");
|
||||
update("checkpoint_every_links", 1);
|
||||
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(2));
|
||||
var second = exporter.run("USER", Instant.now().plusSeconds(2)).row();
|
||||
assertThat(second.getValueString("lock_mode")).isEqualTo("COMPLIANCE");
|
||||
assertThat(verifier.verify(first, "USER").result().passed()).isTrue();
|
||||
assertThat(verifier.verify(second, "USER").result().passed()).isTrue();
|
||||
assertThat(new WormAuditVerifier(config).verify(null, Instant.now().plusSeconds(2)).passed()).isTrue();
|
||||
String key = first.getValueString("object_prefix") + "links.jsonl";
|
||||
storage.heads.get(key).withLockMode("COMPLIANCE");
|
||||
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains("lacks the recorded GOVERNANCE retention"));
|
||||
storage.heads.remove(key);
|
||||
assertThat(verifier.verify(first, "USER").result().getBreaks()).anyMatch(b -> b.reason().contains(key) && b.reason().contains("GOVERNANCE bypass removal"));
|
||||
}
|
||||
}
|
||||
@@ -57,7 +57,7 @@ public class MemorySealedStore implements SealedStore
|
||||
** Refuse overwrites and retain the exact bytes.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String put(String key, Path file, Instant until) throws Exception
|
||||
public String put(String key, Path file, com.kingsrook.qbits.objectstorage.api.LockMode mode, Instant until) throws Exception
|
||||
{
|
||||
if(objects.containsKey(key))
|
||||
{
|
||||
@@ -65,7 +65,7 @@ public class MemorySealedStore implements SealedStore
|
||||
}
|
||||
String version = "v" + (objects.size() + 1);
|
||||
objects.put(key, Files.readAllBytes(file));
|
||||
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode("COMPLIANCE").withLockRetainUntil(until));
|
||||
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode(mode.name()).withLockRetainUntil(until));
|
||||
return (version);
|
||||
}
|
||||
|
||||
|
||||
@@ -176,6 +176,7 @@ public class WormAuditDemoServer extends AbstractQQQApplication
|
||||
public void seed(QInstance instance) throws Exception
|
||||
{
|
||||
QContext.init(instance, new QSession());
|
||||
com.kingsrook.qbits.wormaudit.WormAuditSettings.seed(config);
|
||||
AuditStore store = new AuditStore(null);
|
||||
Instant today = Instant.now().minus(Duration.ofHours(2));
|
||||
try
|
||||
|
||||