Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4b6717ebdd
|
||
|
|
9c2766e324
|
No files matched your search
@@ -29,8 +29,8 @@ Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer im
|
||||
|
||||
PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM.
|
||||
Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set.
|
||||
Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata;
|
||||
signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration.
|
||||
Core RC.5 has no assistant deny API (reflective registration). Alert types use the
|
||||
`wormaudit.*` namespace and are declared through core alert-type metadata.
|
||||
|
||||
## Invariants and naming
|
||||
|
||||
|
||||
@@ -6,9 +6,19 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
|
||||
|
||||
### Added
|
||||
|
||||
- Phase (c): configurable WORM Audit app home, full-width summary widget and optional host health card; permission-aware navigation and content-free, responsive light/dark summaries.
|
||||
- Chain backlog/settling counts, verification and export record links, sealed-range gaps/lag, retention eligibility and recent purge/refusal history.
|
||||
- PostgreSQL guard/ownership/purge-function diagnostics, signing/trust warnings, tri-state Object Lock capability and stale daily-verification warnings.
|
||||
- Test-scope Next UI demo with synthetic tokenized patient notes, genuine verification failure/recovery, signed checkpoints, sealed memory exports and documented simulated purges.
|
||||
- Home privacy/permission/empty-state tests, memory/H2 summary tests and Docker-gated PostgreSQL setup-warning acceptance.
|
||||
|
||||
- Phase (a): PostgreSQL append-only tables/triggers, safe DDL renderer, retention purge with immutable refusal/anchor records, and runtime grants.
|
||||
- Canonical audit v1 documents, contiguous hash chain with late-detail coverage, pinned Ed25519 checkpoints and binary Merkle roots.
|
||||
- Snapshot verification, streamed ETL Review/Result action, scheduled chaining/daily verification, and core QAlerts raise/resolve/heartbeat integration.
|
||||
- Read-only metadata, WORM Audit table navigation, record range references and child lists; reflective RC.6 assistant deny registration.
|
||||
- Offline memory/H2 acceptance, real pre-insert PHI canary hook, and PostgreSQL Testcontainers acceptance with CI-strict Docker availability.
|
||||
- Adoption and operations documentation. Export execution and app-home widgets remain later phases.
|
||||
- Adoption, app-home, demo and operations documentation.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Keep runtime signing/trust objects and storage suppliers out of serialized metadata so signed-chain record child lists render successfully.
|
||||
@@ -29,8 +29,8 @@ Package: `com.kingsrook.qbits.wormaudit`. Entry point: `WormAuditQBitProducer im
|
||||
|
||||
PostgreSQL uses a transaction advisory lock; other backends require a single writer JVM.
|
||||
Use `./mvnw.cmd -o verify` offline. PostgreSQL tests skip without Docker locally, but fail whenever `CI` is set.
|
||||
Core RC.5 has no assistant deny API (reflective registration) and disallows underscores in declared alert-type metadata;
|
||||
signals retain the required `worm_audit.chain.break` name and dispatch through QAlerts without that declaration.
|
||||
Core RC.5 has no assistant deny API (reflective registration). Alert types use the
|
||||
`wormaudit.*` namespace and are declared through core alert-type metadata.
|
||||
|
||||
## Invariants and naming
|
||||
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# QBit WORM Audit
|
||||
|
||||
Phase (a) provides PostgreSQL append-only enforcement, an asynchronous audit hash chain,
|
||||
Ed25519 checkpoints, a verification action, and daily verification with core QAlerts signals.
|
||||
Phases (a)–(c) provide PostgreSQL append-only enforcement, an asynchronous audit hash chain,
|
||||
Ed25519 checkpoints, live and sealed-export verification actions, and daily jobs with core QAlerts signals.
|
||||
It hashes exactly the audit values core stored; it does not read application records or
|
||||
secured-data keys. Sealed exports and the app home are later phases. The export table and
|
||||
range references are present as integration hooks; no export job or home widgets are installed.
|
||||
secured-data keys. Sealed JSONL exports use qbit-object-storage 0.1.0-RC.2 and COMPLIANCE
|
||||
Object Lock. The WORM Audit app home summarizes chain health, exports, retention and setup.
|
||||
|
||||
Requires Java 21, Maven wrapper 3.9.11 and QQQ 4.1.0-RC.5. PostgreSQL provides enforcement;
|
||||
memory/H2 support tamper-evidence tests without database enforcement.
|
||||
@@ -18,13 +18,15 @@ Verifications, Sealed Exports, Checkpoints, Audit Chain, and Purges in that orde
|
||||
| Table | Purpose | Permission |
|
||||
|---|---|---|
|
||||
| `worm_audit_verification` | Immutable USER/SCHEDULE results | `worm_audit_verification.read` |
|
||||
| `worm_audit_export` | Reserved sealed-export catalog | `worm_audit_export.read` |
|
||||
| `worm_audit_export` | Successfully sealed export catalog | `worm_audit_export.read` |
|
||||
| `worm_audit_checkpoint` | Signed ranges and Merkle roots | `worm_audit_checkpoint.read` |
|
||||
| `worm_audit_chain` | Contiguous links and explicit detail coverage | `worm_audit_chain.read` |
|
||||
| `worm_audit_purge` | Refusals, removal counts and retention anchors | `worm_audit_purge.read` |
|
||||
|
||||
`worm_audit.verify` authorizes the standard streamed ETL **Verify audit chain** action.
|
||||
Processes are attached to the chain table or scheduled; none is a navigation entry.
|
||||
It also authorizes **Verify sealed export** on selected export records. `worm_audit.export`
|
||||
authorizes **Export now** on the export table. Processes are table/record actions or schedules;
|
||||
none is a navigation entry.
|
||||
Insert/update/delete capabilities are absent on sidecar tables; update/delete capabilities
|
||||
are removed from all four core audit tables. Internal appends use an explicit permission bypass.
|
||||
Checkpoint, export and verification record screens have a bounded chain-link child list;
|
||||
@@ -32,7 +34,7 @@ range endpoints and export/checkpoint references link to their records.
|
||||
|
||||
The table metadata customizer can configure host ESB publications. Quick Search registration
|
||||
is host-owned through `QuickSearchQBitConfig.searchableTables`; add all five tables.
|
||||
Verification inserts use QQQ's record-change listener SPI (as will phase-(b) export inserts),
|
||||
Verification and export inserts use QQQ's record-change listener SPI,
|
||||
so an installed ESB publisher receives those events when the host configures its destinations
|
||||
on these two tables. The qbit defines no destination and configures no chain-link publication.
|
||||
|
||||
@@ -41,11 +43,18 @@ on these two tables. The qbit defines no destination and configures no chain-lin
|
||||
| Field | Default / contract |
|
||||
|---|---|
|
||||
| `backendName` | Required; core and sidecar audit tables must use this backend |
|
||||
| `appName` / `appLabel` | `worm_audit` / `WORM Audit` |
|
||||
| `parentAppName` | Optional host app, for example `admin` |
|
||||
| `tablePathPrefix` | `/app`; standard QQQ table/record route prefix |
|
||||
| `sealedStoreSupplier` | Optional host/test storage boundary; defaults to qbit-object-storage |
|
||||
| `retentionYears` | 6; compiled into the migration, not changed by a runtime flag |
|
||||
| `signingKey` | Optional `Supplier<SigningKey>`; `SigningKey(keyId, PrivateKey)` |
|
||||
| `trustedPublicKeys` | `Map<String, PublicKey>`; must contain the signing key's public key |
|
||||
| `chainIntervalSeconds` | 30 |
|
||||
| `settleHorizonSeconds` | 60 (zero allowed) |
|
||||
| `exportBucketAlias` | `worm_audit`; qbit-object-storage BACKUP bucket alias |
|
||||
| `exportPrefix` | `worm-audit`; slash-separated letters, digits, underscores or hyphens |
|
||||
| `exportCron` | `0 0 2 * * ?`; 02:00 UTC on the configured scheduler |
|
||||
| `batchSize` | 1,000 headers per tick / links per insert batch |
|
||||
| `lateDetailLookbackHours` | 24; positive hours, older omissions checked by full verification |
|
||||
| `checkpointEveryLinks` | 10,000; maximum links in each checkpoint |
|
||||
@@ -61,6 +70,50 @@ to verification. It does not itself break the chain. Signing requires a matching
|
||||
Rotate by adding a new pin and changing the supplier; retain old pins for old checkpoints.
|
||||
Never store the private key in a database row or checked-in configuration.
|
||||
|
||||
## App home
|
||||
|
||||
Open `/app/worm_audit` (or `/app/admin/worm_audit` with `parentAppName="admin"`).
|
||||
The full-width `wormAuditHome` HTML widget uses the standard `/qqq/v1/widget` API.
|
||||
Hosts can add `wormAuditHealthCard` to their own dashboard. Both widgets and the app
|
||||
require read permission on at least one WORM table, using the host's table permission rules.
|
||||
The aggregate home shows only identifiers, counts, times, shortened hashes and reasons;
|
||||
it never renders audit messages or field values. Record links retain the target table's permissions.
|
||||
|
||||
- Chain status: live link count, latest link time, settled unchained backlog and settling
|
||||
count at the configured horizon, latest verification and first open chain break.
|
||||
A subsequent full scheduled CHAIN PASS clears a reported break; a selected USER PASS does not.
|
||||
- Export status: latest sealed range, COMPLIANCE retention, link/time lag, coverage holes
|
||||
between the purge anchor and sealed head, and bucket Object Lock capability. The unsealed
|
||||
trailing suffix is lag, not a gap. Provider failures show UNKNOWN with a value-free reason.
|
||||
- Retention: configured years, purge anchor, oldest live audit and its next eligible purge date.
|
||||
The date indicates retention eligibility; the database purge function still checks the sealed prefix.
|
||||
- Recent purges: the last ten requests, including refusals.
|
||||
- Setup warnings: missing signing/trusted keys, unsupported or unknown Object Lock,
|
||||
daily CHAIN verification never run or older than 26 hours. PostgreSQL also checks every
|
||||
protected table's row/truncate guards, runtime ownership and the purge function.
|
||||
|
||||
Sections and definition-list rows wrap at narrow widths and inherit the Next UI's light/dark colors.
|
||||
Empty states explicitly distinguish missing history from a passing check.
|
||||
|
||||
### Local demo
|
||||
|
||||
The test-scope server serves the API and bundled Next UI with a memory backend:
|
||||
|
||||
```powershell
|
||||
./mvnw.cmd -o test-compile exec:java "-Dexec.classpathScope=test" "-Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer" "-Dexec.args=8099"
|
||||
```
|
||||
|
||||
On macOS/Linux use `./mvnw` with the same arguments. Open <http://localhost:8099/app/worm_audit>.
|
||||
No Docker or credentials are required. This anonymous demo uses synthetic `patient_note`
|
||||
records audited at FIELD level through the `PhiCanaryTest` HMAC stand-in. It seeds two weeks
|
||||
of actual chainer ticks, signed checkpoints, daily verifier results and immutable memory exports.
|
||||
One temporary detail tamper produces a genuine FAIL before restoration and later PASSes.
|
||||
The PURGED/REFUSED receipts and memory-only prefix removal are explicitly marked demo data;
|
||||
PostgreSQL enforcement and real provider durability are not simulated production guarantees.
|
||||
|
||||
The demo includes a small unsealed suffix and unchained backlog, with no invented export gap.
|
||||
The final settling count naturally ages into the backlog while the demo is open.
|
||||
|
||||
## Adopting in another app
|
||||
|
||||
Add `com.kingsrook.qbits:qbit-worm-audit:0.1.0-SNAPSHOT`. Define the host's core audit tables
|
||||
@@ -95,10 +148,11 @@ String migration = WormAuditDdl.postgres(config);
|
||||
`sealed_at`, `finished_at`, and `requested_at` respectively (these tables have no `modifyDate`).
|
||||
|
||||
The RC.6 assistant deny API is registered reflectively when present; absence in RC.5 is a no-op.
|
||||
RC.5's alert-type metadata validator disallows underscores, so the qbit sends the specified
|
||||
`worm_audit.chain.break` signal without declaring incompatible alert-type metadata. `QAlerts`
|
||||
still dispatches it to the sink (or its logging fallback). Every scheduled verification sends
|
||||
`worm_audit.verify.daily`; FAIL raises with the first break and verification record reference;
|
||||
The qbit declares `wormaudit.chain.break` (CRITICAL), `wormaudit.export.failed` (ERROR),
|
||||
and the daily verification/export heartbeat types through core alert-type metadata.
|
||||
Daily heartbeat monitors expect a 24-hour interval with 2 hours of grace (overdue at 26 hours).
|
||||
`QAlerts` dispatches to the installed sink or its logging fallback. Every scheduled verification sends
|
||||
`wormaudit.verify.daily`; FAIL raises with the first break and verification record reference;
|
||||
PASS resolves dedup key `chain`.
|
||||
|
||||
## Chain and verification format
|
||||
@@ -119,7 +173,7 @@ PASS resolves dedup key `chain`.
|
||||
an odd node is promoted unchanged. Ed25519 signs JCS of the seven specified range/trust fields.
|
||||
- SQL verification uses a repeatable-read snapshot. Full runs check settled header/detail
|
||||
completeness; selected or filtered runs check their links, predecessor boundaries and
|
||||
intersecting checkpoints. Full verification currently materializes link metadata in memory.
|
||||
intersecting checkpoints.
|
||||
|
||||
## Retention operations
|
||||
|
||||
@@ -143,10 +197,70 @@ rights. The qbit maps those errors to a value-free user-facing message.
|
||||
|
||||
Verification starts from the latest successful purge anchor. Checkpoint signatures still verify
|
||||
after purge, but Merkle recomputation of a purged/partly purged checkpoint requires its sealed
|
||||
export and is reported as a warning. Sealed exports are phase (b); preserve external evidence
|
||||
before exercising retention removal in production. Superusers/owners can disable triggers;
|
||||
export and is reported as a warning. Seal and verify exports before exercising retention
|
||||
removal in production. Superusers/owners can disable triggers;
|
||||
protect signing keys and keep public-key pins outside the database.
|
||||
|
||||
## Sealed exports
|
||||
|
||||
Install qbit-object-storage and configure `exportBucketAlias` (default `worm_audit`) as a
|
||||
`BucketRole.BACKUP` bucket. **Object Lock must be enabled when the bucket is created**.
|
||||
The exporter requires the bucket's `OBJECT_LOCK` capability to be `SUPPORTED`; run the
|
||||
object-storage read-only capability probe during setup. Unknown, partial and unsupported
|
||||
capabilities all refuse sealing. Every object receives explicit **COMPLIANCE** retention
|
||||
until the sealing time plus `retentionYears` (six by default), checked by HEAD after PUT.
|
||||
|
||||
The runtime IAM policy needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
|
||||
`s3:GetObjectRetention`, `s3:PutObjectRetention`, and
|
||||
`s3:GetBucketObjectLockConfiguration`. Grant no delete or governance-bypass access.
|
||||
The storage bucket's upload size limit must accommodate the largest daily JSONL file.
|
||||
Retention protects object versions; protect the storage volumes and maintain off-site copies too.
|
||||
|
||||
At 02:00 UTC, or through **Export now**, the exporter takes the chainer's writer lock and
|
||||
seals links after the previous export through the latest checkpoint. The first export starts
|
||||
after the purge anchor, or at genesis. No new checkpoint produces an explicit "Nothing to
|
||||
seal" Result. If retention has removed an unexported gap, or a range cannot be proven by
|
||||
complete signed checkpoints, sealing refuses rather than recording incomplete evidence.
|
||||
Other database backends require a single writer JVM, as in phase (a).
|
||||
|
||||
Keys use `<exportPrefix>/<yyyy>/<MM>/<dd>/<from>-<to>/` with a UTC date:
|
||||
|
||||
- `links.jsonl`: JCS lines containing `link` identifiers/hashes and the embedded canonical
|
||||
`content` document exactly as hashed.
|
||||
- `checkpoints.jsonl`: JCS checkpoint envelopes including signatures and public keys.
|
||||
- `manifest.json`: JCS `worm-audit-export/1` with `fromLinkId`, `toLinkId`, `linkCount`,
|
||||
`checkpointCount`, `files` (filename to SHA-256), `headHash`, `prevManifestSha256`,
|
||||
`qbitVersion`, and `createdAt`.
|
||||
|
||||
Files are streamed through temporary disk storage with 1,000-link pages and batched content
|
||||
fetches, then removed. Only ids, hashes and object metadata enter the export catalog.
|
||||
The `object_versions` text column pins each upload's returned version id in a JCS JSON
|
||||
object keyed by `checkpoints.jsonl`, `links.jsonl`, and `manifest.json`. Sealing refuses
|
||||
missing version ids or a different version in the retention HEAD response, leaving no
|
||||
catalog row and raising the export failure alert.
|
||||
Partial upload failures leave no catalog row; retained orphan objects may remain. Retrying
|
||||
can create new locked versions under the same keys. Failures raise `wormaudit.export.failed`
|
||||
with dedup key `export`; success resolves it. Every scheduled run emits
|
||||
`wormaudit.export.daily`.
|
||||
|
||||
**Verify sealed export** accepts one or several selected records. It downloads and checks
|
||||
the object hashes, retention, manifest chain, link hashes, pinned checkpoint signatures and
|
||||
Merkle roots, then compares the sealed content and link metadata against a live database
|
||||
snapshot. Differences name exact link/audit ids without printing audit values. Each export
|
||||
gets its own `SEALED_EXPORT` verification row. Purged prefixes verify from the sealed copy
|
||||
alone and receive an INFO line. Verification does not write restored audit rows.
|
||||
|
||||
Verification compares every current HEAD version to the catalog and reports an ERROR for
|
||||
any overwrite, even when the replacement bytes are identical. It then reads the recorded
|
||||
sealed versions, so retained evidence remains verifiable after an overwrite; the overall
|
||||
verification still fails because the overwrite is a break. Missing catalog versions fail
|
||||
closed. RC.2's `StorageBucket.get` has no version parameter, so the adapter uses the public
|
||||
`ObjectStorageRuntime.connection(...).client()` accessor for an explicit S3 version GET,
|
||||
reusing its connection, key normalization and rate limiter and checking the response version.
|
||||
No object-storage change is required; this adapter depends on RC.2's public internal runtime
|
||||
accessors. The storage credentials need `s3:GetObjectVersion` in addition to the existing
|
||||
upload, HEAD and listing permissions.
|
||||
|
||||
## Build and acceptance
|
||||
|
||||
```powershell
|
||||
@@ -170,7 +284,21 @@ $env:CI = 'true'
|
||||
./mvnw.cmd -o verify
|
||||
```
|
||||
|
||||
POSIX equivalents: `CI=true ./mvnw -o -Dtest=PostgresAcceptanceTest test` and
|
||||
MinIO export acceptance uses the same `RequiresDockerCondition`. It checks COMPLIANCE
|
||||
retention, denied version deletion, overwrite version preservation, export verification,
|
||||
exact live-tamper findings through the record action, and refusal of a bucket without lock:
|
||||
|
||||
```powershell
|
||||
$env:CI = 'true'
|
||||
./mvnw.cmd -o '-Dtest=MinioExportAcceptanceTest' test
|
||||
```
|
||||
|
||||
The default MinIO image is pinned to the same Chainguard digest as object-storage's tests.
|
||||
Override it with `-Dqqq.test.minioImage=<image>` when necessary. Images must already be
|
||||
available for a completely network-free Docker run.
|
||||
|
||||
POSIX equivalents: `CI=true ./mvnw -o -Dtest=MinioExportAcceptanceTest test`,
|
||||
`CI=true ./mvnw -o -Dtest=PostgresAcceptanceTest test` and
|
||||
`CI=true ./mvnw -o verify`. No commits, publishing or deployment are part of these commands.
|
||||
|
||||
## License
|
||||
@@ -194,6 +322,9 @@ that window. Initial header details are always chained, including overflow
|
||||
chunks. Checkpoints contain at most `checkpointEveryLinks` links; multiple full
|
||||
ranges can be signed per tick. Verification uses 1,000-link keyset pages.
|
||||
|
||||
Full verification holds at most one 1,000-link page and one checkpoint range of hashes.
|
||||
SQL completeness uses bounded offender queries; other persistent backends use repeated bounded
|
||||
scans. Only the memory backend uses a full coverage index.
|
||||
Full verification retains at most 1,000 exact break diagnostics and 100 warnings,
|
||||
while persisting the total break count. SQL completeness counts all offenders
|
||||
and reports the first 100 header ids and first 100 detail ids. This keeps a badly
|
||||
|
||||
@@ -104,7 +104,9 @@ runtime role owns the audit tables (checked through `pg_tables.tableowner = curr
|
||||
The qbit ships `src/main/resources/com/kingsrook/qbits/wormaudit/ddl/postgres/worm-audit.sql`,
|
||||
a Liquibase formatted-SQL template with placeholders for the four core audit table names, the
|
||||
app role and the retention interval, plus a renderer (`WormAuditDdl.postgres(config)`) and the
|
||||
rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Hosts copy the rendered
|
||||
rendered default (`docs/ddl/worm-audit-postgres.sql`, snake_case names). Core audit physical
|
||||
columns must use snake_case (for example `audit_table_id`, `audit_id`, `new_value`); hosts map
|
||||
QQQ field names to those columns. Hosts copy the rendered
|
||||
file into their changelog (kof22: `changesets/features/YYYY-MM-DD-worm-audit.sql`). It creates the
|
||||
qbit tables, the functions, the triggers, the purger role (`CREATE ROLE ... NOLOGIN` when it is
|
||||
missing) and the grants. It is idempotent (`CREATE OR REPLACE`, `DROP TRIGGER IF EXISTS`).
|
||||
@@ -217,9 +219,9 @@ standard streamed ETL:
|
||||
does not verify".
|
||||
|
||||
**Daily verification**: a scheduled job runs the same verifier at 03:00. A failure raises
|
||||
`QAlerts.raise(new AlertSignal("worm_audit.chain.break").withDedupKey("chain")...)` with the first
|
||||
`QAlerts.raise(new AlertSignal("wormaudit.chain.break").withDedupKey("chain")...)` with the first
|
||||
break in the title and the verification record as the record ref; a later pass resolves it. Every
|
||||
run sends `QAlerts.heartbeat("worm_audit.verify.daily")`. `QAlerts` is the core producer SPI
|
||||
run sends `QAlerts.heartbeat("wormaudit.verify.daily")`. `QAlerts` is the core producer SPI
|
||||
(QQQ 4.1.0-RC.5), so the qbit needs no dependency on qbit-alerts. With no alerts implementation
|
||||
installed the alerts go to the log.
|
||||
|
||||
@@ -236,14 +238,17 @@ Daily at 02:00 (configurable), and by the table action "Export now" on `worm_aud
|
||||
the previous export's manifest SHA-256 (the exports form their own chain), qbit version.
|
||||
3. Each object is put through qbit-object-storage (`SystemStorage`, bucket alias from config,
|
||||
default `worm_audit`) with `LockMode.COMPLIANCE` and `retainUntil = now + retentionYears`. The
|
||||
export refuses to run, and alerts `worm_audit.export.failed`, when the bucket does not report
|
||||
export refuses to run, and alerts `wormaudit.export.failed`, when the bucket does not report
|
||||
the `OBJECT_LOCK` capability or the put does not echo the lock.
|
||||
4. On success it inserts one `worm_audit_export` row (insert-only; a failed export leaves no row,
|
||||
and its alert carries the reason).
|
||||
and its alert carries the reason). Every scheduled run sends `wormaudit.export.daily`.
|
||||
|
||||
### 6.2 Restore and verify a sealed export
|
||||
"Verify sealed export", a record action on `worm_audit_export` (standard ETL), downloads the
|
||||
objects. It checks the manifest hashes, the lock mode and retain-until of each object, the link
|
||||
exact object versions pinned in the catalog's `object_versions` map. A new version under an
|
||||
existing key is an overwrite even when bytes and retention match: verification checks the latest
|
||||
version against the pin and rejects replacements, then reads and checks the pinned version.
|
||||
It checks the manifest hashes, the lock mode and retain-until of each object, the link
|
||||
hashes recomputed from the exported content, the checkpoint signatures, and the export's link to
|
||||
the previous manifest. Then it compares each exported link and content document with the live
|
||||
chain and the live audit rows. Result lines name each difference ("link 88,412: live content
|
||||
@@ -265,7 +270,7 @@ events integrations care about); chain links are high-volume and publish none.
|
||||
- **`worm_audit_checkpoint`** [Checkpoint]: `id`, `from_link_id`, `to_link_id`, `link_count`,
|
||||
`head_hash`, `merkle_root`, `algorithm`, `key_id`, `public_key`, `signature`, `created_at`.
|
||||
- **`worm_audit_export`** [Sealed Export]: `id`, `from_link_id`, `to_link_id`,
|
||||
`from_checkpoint_id`, `to_checkpoint_id`, `bucket_alias`, `object_prefix`, `manifest_key`,
|
||||
`from_checkpoint_id`, `to_checkpoint_id`, `bucket_alias`, `object_prefix`, `object_versions`, `manifest_key`,
|
||||
`manifest_sha256`, `prev_manifest_sha256`, `object_count`, `byte_count`, `lock_mode`,
|
||||
`retain_until`, `sealed_at`.
|
||||
- **`worm_audit_verification`** [Verification]: `id`, `kind` (CHAIN, SEALED_EXPORT), `trigger`
|
||||
@@ -341,7 +346,7 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
|
||||
qbit's tests until the core hook ships), the audit header, the details (old, new, message),
|
||||
the canonical content and the export JSONL contain neither the canary nor any substring of it
|
||||
longer than 4 characters.
|
||||
6. **Alerts**: a break raises `worm_audit.chain.break` through `QAlerts` (captured by a test
|
||||
6. **Alerts**: a break raises `wormaudit.chain.break` through `QAlerts` (captured by a test
|
||||
sink); a later pass resolves it.
|
||||
7. Coverage gates: 70% instructions, 90% classes.
|
||||
|
||||
@@ -367,7 +372,7 @@ registers as a `MiddlewareSpecExtension` under `/qqq/v1/qbit/worm-audit/...`
|
||||
- **Bucket:** `kof22-worm-audit` on the kof22 MinIO, created by `minio-bucket-init` with
|
||||
`mc mb --with-lock --ignore-existing` and `mc retention set --default COMPLIANCE 6y`. Object
|
||||
Lock can only be enabled at bucket creation.
|
||||
- **Policy:** `kof22-app` gets `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`,
|
||||
- **Policy:** `kof22-app` gets `s3:PutObject`, `s3:GetObject`, `s3:GetObjectVersion`, `s3:ListBucket`,
|
||||
`s3:GetObjectRetention`, `s3:PutObjectRetention` (needed to set the lock on each put),
|
||||
`s3:GetBucketObjectLockConfiguration` on that bucket. No delete, and no
|
||||
`s3:BypassGovernanceRetention` (irrelevant under COMPLIANCE anyway).
|
||||
|
||||
@@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS public.worm_audit_export (
|
||||
to_checkpoint_id bigint,
|
||||
bucket_alias text,
|
||||
object_prefix text,
|
||||
object_versions text,
|
||||
manifest_key text,
|
||||
manifest_sha256 text,
|
||||
prev_manifest_sha256 text,
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# WORM Audit screenshots
|
||||
|
||||
Refreshed after record-screen polish **2026-10-13 (America/Chicago)** from the rebuilt test-scope demo using
|
||||
packaged Next UI 1.0.0-RC.14 and connected Edge through the browser screenshot
|
||||
and viewport tools. All files are JPEG viewport captures, below 300 KB each.
|
||||
|
||||
```powershell
|
||||
./mvnw.cmd -o test-compile exec:java "-Dexec.classpathScope=test" "-Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer" "-Dexec.args=8105"
|
||||
```
|
||||
|
||||
Open <http://localhost:8105/app/worm_audit/>. Port 8105 isolates this rebuilt instance from earlier demos. The seed creates genuine verification
|
||||
results and exports over synthetic data. Records captured: FAIL verification #8,
|
||||
PASS verification #15, sealed export #15, and chain link #329.
|
||||
|
||||
Required matrix: home at **390, 1440, 1920 px**, and all four records at **390,
|
||||
1440 px**, each in **light and dark**. Heights are 844, 1000 and 1080 px respectively.
|
||||
The existing supplementary 390 px light purge-history capture was refreshed too.
|
||||
Theme changes used Preferences and were restored to light; the temporary viewport override was reset afterward.
|
||||
|
||||
Every image was visually inspected. At each capture, the live DOM was checked after responsive layout settled for
|
||||
page-level horizontal overflow (`document.documentElement.scrollWidth > innerWidth`)
|
||||
and all rendered inner horizontal scrollers (`scrollWidth > clientWidth + 1` with
|
||||
computed `overflow-x` of `auto` or `scroll`). The scan included content below the
|
||||
viewport, including loaded child lists. Mobile verification accordions were also explicitly expanded and checked after their child rows loaded. There were **zero** page overflow failures
|
||||
and **zero** inner horizontal scrollers. Each home/verification/export view had
|
||||
one vertical scroll container (main); chain records had none. No double scrollbars
|
||||
were found. Long hashes, object versions and failure diagnostics wrap on mobile.
|
||||
No CSS or DOM overrides were injected for the captures.
|
||||
|
||||
All five tables have explicit field labels and meaningful record-label formats.
|
||||
Verification uses full-width Result, Scope, Run, and Chain links sections. Every
|
||||
child-list widget and section declares 12 grid columns. Kind and outcome use
|
||||
readable possible values; child queries translate those values too. Hashes stay
|
||||
complete because RC.5 has no separate list-only field display format. Numeric
|
||||
chain-reference labels keep the export title concise.
|
||||
|
||||
Core RC.5 shares the table label with its sidebar tree node. Existing plural
|
||||
navigation labels are retained, with singular record titles and record-section
|
||||
labels. Separate singular table labels and plural sidebar labels require core
|
||||
support; no application-wide metadata customizer was replaced to work around it.
|
||||
|
||||
`./mvnw.cmd -o verify` passed: 79 tests, no failures or errors, and 9
|
||||
Docker-dependent skips. Checkstyle and JaCoCo coverage gates passed. Metadata
|
||||
tests cover every field label, record-label formats and fields, verification
|
||||
section coverage, and 12-column sections and child widgets.
|
||||
|
||||
| Screenshot | Viewport | Size | Page horizontal overflow | Inner horizontal scrollers | Double scrollbars |
|
||||
|---|---|---|---|---|---|
|
||||
| [worm-audit-chain-link-1440-dark](worm-audit-chain-link-1440-dark.jpg) | 1440 x 1000 | 41.1 KB | None | None | None |
|
||||
| [worm-audit-chain-link-1440-light](worm-audit-chain-link-1440-light.jpg) | 1440 x 1000 | 41.4 KB | None | None | None |
|
||||
| [worm-audit-chain-link-390-dark](worm-audit-chain-link-390-dark.jpg) | 390 x 844 | 21.8 KB | None | None | None |
|
||||
| [worm-audit-chain-link-390-light](worm-audit-chain-link-390-light.jpg) | 390 x 844 | 21.8 KB | None | None | None |
|
||||
| [worm-audit-home-1440-dark](worm-audit-home-1440-dark.jpg) | 1440 x 1000 | 73.6 KB | None | None | None |
|
||||
| [worm-audit-home-1440-light](worm-audit-home-1440-light.jpg) | 1440 x 1000 | 72.4 KB | None | None | None |
|
||||
| [worm-audit-home-1920-dark](worm-audit-home-1920-dark.jpg) | 1920 x 1080 | 84.7 KB | None | None | None |
|
||||
| [worm-audit-home-1920-light](worm-audit-home-1920-light.jpg) | 1920 x 1080 | 83.6 KB | None | None | None |
|
||||
| [worm-audit-home-390-dark](worm-audit-home-390-dark.jpg) | 390 x 844 | 30.0 KB | None | None | None |
|
||||
| [worm-audit-home-390-light](worm-audit-home-390-light.jpg) | 390 x 844 | 28.8 KB | None | None | None |
|
||||
| [worm-audit-home-390-purges-light](worm-audit-home-390-purges-light.jpg) | 390 x 844 | 29.5 KB | None | None | None |
|
||||
| [worm-audit-sealed-export-1440-dark](worm-audit-sealed-export-1440-dark.jpg) | 1440 x 1000 | 117.6 KB | None | None | None |
|
||||
| [worm-audit-sealed-export-1440-light](worm-audit-sealed-export-1440-light.jpg) | 1440 x 1000 | 118.4 KB | None | None | None |
|
||||
| [worm-audit-sealed-export-390-dark](worm-audit-sealed-export-390-dark.jpg) | 390 x 844 | 38.4 KB | None | None | None |
|
||||
| [worm-audit-sealed-export-390-light](worm-audit-sealed-export-390-light.jpg) | 390 x 844 | 37.3 KB | None | None | None |
|
||||
| [worm-audit-verification-fail-1440-dark](worm-audit-verification-fail-1440-dark.jpg) | 1440 x 1000 | 53.6 KB | None | None | None |
|
||||
| [worm-audit-verification-fail-1440-light](worm-audit-verification-fail-1440-light.jpg) | 1440 x 1000 | 54.1 KB | None | None | None |
|
||||
| [worm-audit-verification-fail-390-dark](worm-audit-verification-fail-390-dark.jpg) | 390 x 844 | 28.1 KB | None | None | None |
|
||||
| [worm-audit-verification-fail-390-light](worm-audit-verification-fail-390-light.jpg) | 390 x 844 | 27.8 KB | None | None | None |
|
||||
| [worm-audit-verification-pass-1440-dark](worm-audit-verification-pass-1440-dark.jpg) | 1440 x 1000 | 49.0 KB | None | None | None |
|
||||
| [worm-audit-verification-pass-1440-light](worm-audit-verification-pass-1440-light.jpg) | 1440 x 1000 | 49.7 KB | None | None | None |
|
||||
| [worm-audit-verification-pass-390-dark](worm-audit-verification-pass-390-dark.jpg) | 390 x 844 | 23.5 KB | None | None | None |
|
||||
| [worm-audit-verification-pass-390-light](worm-audit-verification-pass-390-light.jpg) | 390 x 844 | 23.4 KB | None | None | None |
|
||||
|
After Width: | Height: | Size: 41 KiB |
|
After Width: | Height: | Size: 41 KiB |
|
After Width: | Height: | Size: 22 KiB |
|
After Width: | Height: | Size: 22 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 72 KiB |
|
After Width: | Height: | Size: 85 KiB |
|
After Width: | Height: | Size: 84 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 29 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 118 KiB |
|
After Width: | Height: | Size: 118 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 37 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 54 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 24 KiB |
|
After Width: | Height: | Size: 23 KiB |
@@ -83,6 +83,11 @@
|
||||
<groupId>com.kingsrook.qqq</groupId>
|
||||
<artifactId>qqq-backend-module-rdbms</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
<artifactId>qbit-object-storage</artifactId>
|
||||
<version>0.1.0-RC.2</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.kingsrook.qqq</groupId>
|
||||
<artifactId>qqq-middleware-javalin</artifactId>
|
||||
|
||||
@@ -25,6 +25,7 @@ import java.security.PrivateKey;
|
||||
import java.security.PublicKey;
|
||||
import java.util.Map;
|
||||
import java.util.function.Supplier;
|
||||
import com.fasterxml.jackson.annotation.JsonIgnore;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
@@ -44,6 +45,11 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
{
|
||||
}
|
||||
|
||||
private String appName = "worm_audit";
|
||||
private String appLabel = "WORM Audit";
|
||||
private String parentAppName;
|
||||
private String tablePathPrefix = "/app";
|
||||
private Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> sealedStoreSupplier;
|
||||
private String backendName = null;
|
||||
private int retentionYears = 6;
|
||||
private Supplier<SigningKey> signingKey = null;
|
||||
@@ -55,12 +61,111 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
private int checkpointEveryLinks = 10000;
|
||||
private int checkpointEveryMinutes = 60;
|
||||
private String verifyCron = "0 0 3 * * ?";
|
||||
private String exportBucketAlias = "worm_audit";
|
||||
private String exportPrefix = "worm-audit";
|
||||
private String exportCron = "0 0 2 * * ?";
|
||||
private String schedulerName = "worm_audit";
|
||||
private Map<String, String> coreAuditTableNames = Map.of();
|
||||
private String appRole = "app";
|
||||
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the app name.
|
||||
*******************************************************************************/
|
||||
public String getAppName()
|
||||
{
|
||||
return (appName);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the app name.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withAppName(String appName)
|
||||
{
|
||||
this.appName = appName;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the app label.
|
||||
*******************************************************************************/
|
||||
public String getAppLabel()
|
||||
{
|
||||
return (appLabel);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the app label.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withAppLabel(String appLabel)
|
||||
{
|
||||
this.appLabel = appLabel;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the optional parent app.
|
||||
*******************************************************************************/
|
||||
public String getParentAppName()
|
||||
{
|
||||
return (parentAppName);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the optional parent app.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withParentAppName(String parentAppName)
|
||||
{
|
||||
this.parentAppName = parentAppName;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the table route prefix.
|
||||
*******************************************************************************/
|
||||
public String getTablePathPrefix()
|
||||
{
|
||||
return (tablePathPrefix);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the table route prefix.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withTablePathPrefix(String tablePathPrefix)
|
||||
{
|
||||
this.tablePathPrefix = tablePathPrefix;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the optional storage boundary (defaults to qbit-object-storage).
|
||||
*******************************************************************************/
|
||||
@JsonIgnore
|
||||
public Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> getSealedStoreSupplier()
|
||||
{
|
||||
return (sealedStoreSupplier);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the optional storage boundary (defaults to qbit-object-storage).
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withSealedStoreSupplier(Supplier<com.kingsrook.qbits.wormaudit.export.SealedStore> sealedStoreSupplier)
|
||||
{
|
||||
this.sealedStoreSupplier = sealedStoreSupplier;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read backendName.
|
||||
*******************************************************************************/
|
||||
@@ -102,6 +207,7 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
/*******************************************************************************
|
||||
** Read signingKey.
|
||||
*******************************************************************************/
|
||||
@JsonIgnore
|
||||
public Supplier<SigningKey> getSigningKey()
|
||||
{
|
||||
return (signingKey);
|
||||
@@ -121,6 +227,7 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
/*******************************************************************************
|
||||
** Read trustedPublicKeys.
|
||||
*******************************************************************************/
|
||||
@JsonIgnore
|
||||
public Map<String, PublicKey> getTrustedPublicKeys()
|
||||
{
|
||||
return (trustedPublicKeys);
|
||||
@@ -260,6 +367,63 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the Object Lock bucket alias.
|
||||
*******************************************************************************/
|
||||
public String getExportBucketAlias()
|
||||
{
|
||||
return (exportBucketAlias);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the Object Lock bucket alias.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withExportBucketAlias(String exportBucketAlias)
|
||||
{
|
||||
this.exportBucketAlias = exportBucketAlias;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the export key prefix.
|
||||
*******************************************************************************/
|
||||
public String getExportPrefix()
|
||||
{
|
||||
return (exportPrefix);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the export key prefix.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withExportPrefix(String exportPrefix)
|
||||
{
|
||||
this.exportPrefix = exportPrefix;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the UTC export schedule.
|
||||
*******************************************************************************/
|
||||
public String getExportCron()
|
||||
{
|
||||
return (exportCron);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure the UTC export schedule.
|
||||
*******************************************************************************/
|
||||
public WormAuditQBitConfig withExportCron(String exportCron)
|
||||
{
|
||||
this.exportCron = exportCron;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Configure verifyCron.
|
||||
*******************************************************************************/
|
||||
@@ -352,6 +516,17 @@ public class WormAuditQBitConfig implements QBitConfig
|
||||
*******************************************************************************/
|
||||
public void validate()
|
||||
{
|
||||
if(appName == null || !appName.matches("[A-Za-z0-9_-]+") || appLabel == null || appLabel.isBlank()
|
||||
|| (parentAppName != null && !parentAppName.matches("[A-Za-z0-9_-]+"))
|
||||
|| tablePathPrefix == null || !tablePathPrefix.matches("/[A-Za-z0-9_/-]*") || tablePathPrefix.endsWith("/"))
|
||||
{
|
||||
throw (new IllegalArgumentException("App names and a local table path prefix are required"));
|
||||
}
|
||||
if(exportBucketAlias == null || exportBucketAlias.isBlank() || exportCron == null || exportCron.isBlank()
|
||||
|| exportPrefix == null || !exportPrefix.matches("[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*"))
|
||||
{
|
||||
throw (new IllegalArgumentException("Export bucket, prefix and schedule must be configured"));
|
||||
}
|
||||
if(backendName == null || backendName.isBlank() || retentionYears < 1 || chainIntervalSeconds < 1
|
||||
|| lateDetailLookbackHours < 1 || settleHorizonSeconds < 0 || batchSize < 1 || checkpointEveryLinks < 1 || checkpointEveryMinutes < 1)
|
||||
{
|
||||
|
||||
@@ -21,19 +21,22 @@
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditApp;
|
||||
import com.kingsrook.qbits.wormaudit.ddl.WormAuditDdl;
|
||||
import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
|
||||
import com.kingsrook.qbits.wormaudit.model.LinkRangeWidget;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
|
||||
import com.kingsrook.qbits.wormaudit.processes.WormAuditProcesses;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertDefaultMonitor;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
|
||||
@@ -122,7 +125,7 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Register phase (a) tables, processes and schedules on the host backend.
|
||||
** Register audit tables, verification/export actions and schedules.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException
|
||||
@@ -149,6 +152,14 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
{
|
||||
table = qBitConfig.getTableMetaDataCustomizer().customizeMetaData(qInstance, table);
|
||||
}
|
||||
for(var field : table.getFields().values())
|
||||
{
|
||||
if(field.getInlinePossibleValueSource() != null)
|
||||
{
|
||||
output.add(field.getInlinePossibleValueSource());
|
||||
field.withPossibleValueSourceName(field.getInlinePossibleValueSource().getName()).withInlinePossibleValueSource(null);
|
||||
}
|
||||
}
|
||||
guard.customizeMetaData(qInstance, table);
|
||||
table.setSourceQBitName(qBit.getName());
|
||||
output.add(table);
|
||||
@@ -159,11 +170,16 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
}
|
||||
for(String name : WormAuditTables.ALL)
|
||||
{
|
||||
output.add(QPossibleValueSource.newForTable(name));
|
||||
QPossibleValueSource source = QPossibleValueSource.newForTable(name);
|
||||
if(WormAuditTables.CHAIN.equals(name))
|
||||
{
|
||||
source.withValueFormat("%s").withValueFields(java.util.List.of("id"));
|
||||
}
|
||||
output.add(source);
|
||||
}
|
||||
for(String name : java.util.List.of(WormAuditTables.CHECKPOINT, WormAuditTables.EXPORT, WormAuditTables.VERIFICATION))
|
||||
{
|
||||
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList")
|
||||
output.add(new QWidgetMetaData().withName(name + "_links").withLabel("Chain links").withType("childRecordList").withGridColumns(12)
|
||||
.withCodeReference(new QCodeReference(LinkRangeWidget.class)).withDefaultValue("parentTable", name));
|
||||
}
|
||||
for(var process : WormAuditProcesses.processes(qBitConfig))
|
||||
@@ -171,19 +187,37 @@ public class WormAuditQBitProducer implements QBitMetaDataProducer<WormAuditQBit
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
}
|
||||
for(var process : ExportProcesses.processes(qBitConfig))
|
||||
{
|
||||
process.setSourceQBitName(qBit.getName());
|
||||
output.add(process);
|
||||
}
|
||||
if(qInstance.getSchedulers() == null || !qInstance.getSchedulers().containsKey(qBitConfig.getSchedulerName()))
|
||||
{
|
||||
qInstance.addScheduler(new QuartzSchedulerMetaData().withName(qBitConfig.getSchedulerName()));
|
||||
}
|
||||
QAppMetaData app = new QAppMetaData().withName("worm_audit").withLabel("WORM Audit").withIcon(new QIcon("policy"));
|
||||
for(String name : WormAuditTables.NAV)
|
||||
output.add(WormAuditApp.app(qBitConfig, output.getEach(QTableMetaData.class)));
|
||||
for(QWidgetMetaData widget : WormAuditApp.widgets())
|
||||
{
|
||||
QTableMetaData table = output.getEach(QTableMetaData.class).stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
|
||||
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table);
|
||||
output.add(widget);
|
||||
}
|
||||
output.add(new QAlertTypeMetaData().withName("wormaudit.chain.break").withLabel("Audit chain break")
|
||||
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.CRITICAL)
|
||||
.withDescription("Audit chain verification found a break in content, continuity, coverage or checkpoint trust.")
|
||||
.withSubjectTableName(WormAuditTables.VERIFICATION).withVisibilityPermission("worm_audit_verification.read"));
|
||||
output.add(new QAlertTypeMetaData().withName("wormaudit.export.failed").withLabel("Sealed audit export failed")
|
||||
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.ERROR)
|
||||
.withDescription("The audit export could not be sealed with verified retention and object versions.")
|
||||
.withSubjectTableName(WormAuditTables.EXPORT).withVisibilityPermission("worm_audit_export.read"));
|
||||
for(String job : List.of("verify", "export"))
|
||||
{
|
||||
String name = "wormaudit." + job + ".daily";
|
||||
output.add(new QAlertTypeMetaData().withName(name).withLabel("Daily audit " + job + " overdue")
|
||||
.withSourceQBitName(qBit.getName()).withDefaultSeverity(AlertSeverity.ERROR)
|
||||
.withDescription("The daily audit " + job + " job has not sent a heartbeat within 26 hours.")
|
||||
.withVisibilityPermission(job.equals("verify") ? "worm_audit_verification.read" : "worm_audit_export.read")
|
||||
.withDefaultMonitor(AlertDefaultMonitor.heartbeat(name, name, 24 * 3600, 2 * 3600)));
|
||||
}
|
||||
output.add(app);
|
||||
// RC.5 rejects underscores in declared alert type names. QAlerts still
|
||||
// dispatches the specified worm_audit.chain.break signal to installed sinks.
|
||||
output.addSelfToInstance(qInstance);
|
||||
denyAssistant(qInstance);
|
||||
return (new MetaDataProducerMultiOutput());
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.app;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.ddl.WormAuditDdl;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read-only PostgreSQL catalog probes; errors become value-free diagnostics.
|
||||
*******************************************************************************/
|
||||
public final class PostgresSetupWarnings
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private PostgresSetupWarnings()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Probe the runtime connection, never the configured migration role.
|
||||
*******************************************************************************/
|
||||
public static List<String> load(QBackendTransaction transaction, WormAuditQBitConfig config)
|
||||
{
|
||||
List<String> warnings = new ArrayList<>();
|
||||
if(!(transaction instanceof RDBMSTransaction rdbms))
|
||||
{
|
||||
return (warnings);
|
||||
}
|
||||
try
|
||||
{
|
||||
var connection = rdbms.getConnection();
|
||||
if(!"PostgreSQL".equals(connection.getMetaData().getDatabaseProductName()))
|
||||
{
|
||||
return (warnings);
|
||||
}
|
||||
List<String> tables = new ArrayList<>(config.getCoreAuditTableNames().values());
|
||||
tables.addAll(WormAuditTables.ALL);
|
||||
for(String table : tables)
|
||||
{
|
||||
for(String trigger : List.of("worm_audit_row_guard", "worm_audit_truncate_guard"))
|
||||
{
|
||||
try(var statement = connection.prepareStatement("SELECT EXISTS (SELECT 1 FROM pg_trigger WHERE tgrelid=to_regclass(?) AND tgname=? AND NOT tgisinternal AND tgenabled IN ('O','A'))"))
|
||||
{
|
||||
statement.setString(1, WormAuditDdl.identifier(table));
|
||||
statement.setString(2, trigger);
|
||||
try(var result = statement.executeQuery())
|
||||
{
|
||||
result.next();
|
||||
if(!result.getBoolean(1))
|
||||
{
|
||||
warnings.add("Missing or disabled " + trigger + " on " + table);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
try(var statement = connection.prepareStatement("SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tableowner=current_user AND to_regclass(quote_ident(schemaname)||'.'||quote_ident(tablename))=to_regclass(?))"))
|
||||
{
|
||||
statement.setString(1, WormAuditDdl.identifier(table));
|
||||
try(var result = statement.executeQuery())
|
||||
{
|
||||
result.next();
|
||||
if(result.getBoolean(1))
|
||||
{
|
||||
warnings.add("Runtime role owns audit table " + table);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
try(var statement = connection.createStatement(); var result = statement.executeQuery("SELECT to_regprocedure('public.worm_audit_purge(timestamp with time zone)') IS NOT NULL"))
|
||||
{
|
||||
result.next();
|
||||
if(!result.getBoolean(1))
|
||||
{
|
||||
warnings.add("Retention purge function is missing");
|
||||
}
|
||||
}
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
warnings.add("PostgreSQL setup checks unavailable");
|
||||
}
|
||||
return (warnings);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.app;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
|
||||
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
|
||||
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** WORM Audit navigation and standard HTML widgets, scoped to this QInstance.
|
||||
*******************************************************************************/
|
||||
public final class WormAuditApp
|
||||
{
|
||||
public static final String HOME_WIDGET = "wormAuditHome";
|
||||
public static final String HEALTH_CARD_WIDGET = "wormAuditHealthCard";
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private WormAuditApp()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The home and optional host-dashboard summary share the app's permissions.
|
||||
*******************************************************************************/
|
||||
public static List<QWidgetMetaData> widgets()
|
||||
{
|
||||
return (List.of(
|
||||
new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType())
|
||||
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false).withPermissionRules(permissions()),
|
||||
new QWidgetMetaData().withName(HEALTH_CARD_WIDGET).withLabel("WORM Audit").withType(WidgetType.HTML.getType())
|
||||
.withCodeReference(new QCodeReference(HealthCardRenderer.class)).withGridColumns(12).withIsCard(true).withPermissionRules(permissions())));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Home, Verifications, Sealed Exports, Checkpoints, Audit Chain, Purges.
|
||||
*******************************************************************************/
|
||||
public static QAppMetaData app(WormAuditQBitConfig config, List<QTableMetaData> tables)
|
||||
{
|
||||
QAppMetaData app = new QAppMetaData().withName(config.getAppName()).withLabel(config.getAppLabel())
|
||||
.withIcon(new QIcon("verified_user")).withPermissionRules(permissions()).withWidgets(List.of(HOME_WIDGET));
|
||||
app.setParentAppName(config.getParentAppName());
|
||||
for(String name : WormAuditTables.NAV)
|
||||
{
|
||||
QTableMetaData table = tables.stream().filter(t -> t.getName().equals(name)).findFirst().orElseThrow();
|
||||
app.withSectionOfChildren(new QAppSection().withName(name).withLabel(table.getLabel()), table);
|
||||
}
|
||||
return (app);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read on any WORM table grants the value-free aggregate overview.
|
||||
*******************************************************************************/
|
||||
public static QPermissionRules permissions()
|
||||
{
|
||||
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
|
||||
.withCustomPermissionChecker(new QCodeReference(HomePermissionChecker.class)));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Honor the host's actual table permission rules, including custom checkers.
|
||||
*******************************************************************************/
|
||||
public static class HomePermissionChecker implements CustomPermissionChecker
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Reject direct widget requests unless at least one table is readable.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
|
||||
{
|
||||
for(String table : WormAuditTables.ALL)
|
||||
{
|
||||
try
|
||||
{
|
||||
PermissionsHelper.checkTablePermissionThrowing(new QueryInput(table), TablePermissionSubType.READ);
|
||||
return;
|
||||
}
|
||||
catch(QPermissionDeniedException expected)
|
||||
{
|
||||
// Another WORM table may grant access to the aggregate overview.
|
||||
}
|
||||
}
|
||||
throw (new QPermissionDeniedException("Read access to a WORM audit table is required"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Full-width home renderer.
|
||||
*******************************************************************************/
|
||||
public static class HomeRenderer extends AbstractWidgetRenderer
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Render only metadata, counts and value-free reasons.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
|
||||
{
|
||||
new HomePermissionChecker().checkPermissionsThrowing(input, null);
|
||||
return (new RenderWidgetOutput(new RawHTML(null, WormAuditHomeView.render(WormAuditHomeData.load(), VerificationRuns.config().getTablePathPrefix()))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional compact host dashboard renderer.
|
||||
*******************************************************************************/
|
||||
public static class HealthCardRenderer extends AbstractWidgetRenderer
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Link the status summary to the owning app home.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
|
||||
{
|
||||
new HomePermissionChecker().checkPermissionsThrowing(input, null);
|
||||
WormAuditQBitConfig config = VerificationRuns.config();
|
||||
String path = config.getTablePathPrefix() + "/" + (config.getParentAppName() == null ? "" : config.getParentAppName() + "/") + config.getAppName();
|
||||
return (new RenderWidgetOutput(new RawHTML(config.getAppLabel(), WormAuditHomeView.renderHealthCard(WormAuditHomeData.load(), path))));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.app;
|
||||
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read-only home snapshot. No audit messages, detail values or canonical content
|
||||
** enter this model. Unexported suffixes are lag; holes before the sealed head
|
||||
** are gaps. Only a later full scheduled chain PASS clears an open chain break.
|
||||
*******************************************************************************/
|
||||
public record WormAuditHomeData(long links, Instant lastLinkAt, long backlog, long settling, int settleSeconds,
|
||||
QRecord lastVerification, QRecord openBreak, QRecord lastExport, long exportLag, Long exportAgeHours,
|
||||
List<String> exportGaps, String bucketAlias, String lockCapability, String lockReason,
|
||||
int retentionYears, QRecord anchor, Instant oldestAudit, Instant nextPurge,
|
||||
List<QRecord> recentPurges, List<String> warnings)
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Load using the current instance's configuration and production clock.
|
||||
*******************************************************************************/
|
||||
public static WormAuditHomeData load() throws QException
|
||||
{
|
||||
return (load(VerificationRuns.config(), null, Instant.now()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The storage and time seams also serve the memory demo and deterministic tests.
|
||||
*******************************************************************************/
|
||||
public static WormAuditHomeData load(WormAuditQBitConfig config, SealedStore storage, Instant now) throws QException
|
||||
{
|
||||
List<String> warnings = new ArrayList<>();
|
||||
String capability = "UNKNOWN";
|
||||
String reason = "Object Lock capability could not be confirmed";
|
||||
try
|
||||
{
|
||||
SealedStore resolved = storage != null ? storage : config.getSealedStoreSupplier() == null
|
||||
? new ObjectStorageSealedStore(config.getExportBucketAlias(), "home") : config.getSealedStoreSupplier().get();
|
||||
CapabilityStatus status = resolved.lockCapability();
|
||||
if(status == CapabilityStatus.SUPPORTED || status == CapabilityStatus.UNSUPPORTED)
|
||||
{
|
||||
capability = status.name();
|
||||
reason = status == CapabilityStatus.SUPPORTED ? null : "Export bucket does not support Object Lock";
|
||||
}
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
// Provider exception messages can contain credentials or object content.
|
||||
reason = "Object Lock capability check failed";
|
||||
}
|
||||
if(reason != null)
|
||||
{
|
||||
warnings.add(reason);
|
||||
}
|
||||
if(config.getSigningKey() == null)
|
||||
{
|
||||
warnings.add("No signing key configured");
|
||||
}
|
||||
if(config.getTrustedPublicKeys() == null || config.getTrustedPublicKeys().isEmpty())
|
||||
{
|
||||
warnings.add("No trusted public keys configured");
|
||||
}
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
|
||||
{
|
||||
try
|
||||
{
|
||||
if(transaction instanceof RDBMSTransaction rdbms)
|
||||
{
|
||||
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
|
||||
}
|
||||
AuditStore store = new AuditStore(transaction);
|
||||
QRecord lastLink = store.last(WormAuditTables.CHAIN);
|
||||
long links = store.count(WormAuditTables.CHAIN, new QQueryFilter());
|
||||
Instant cutoff = now.minusSeconds(config.getSettleHorizonSeconds());
|
||||
long backlog = store.unchainedCount(cutoff, true);
|
||||
long settling = store.unchainedCount(cutoff, false);
|
||||
QRecord verification = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(), "finished_at");
|
||||
QQueryFilter dailyFilter = new QQueryFilter(new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"),
|
||||
new QFilterCriteria("trigger", QCriteriaOperator.EQUALS, "SCHEDULE"));
|
||||
QRecord daily = latest(store, WormAuditTables.VERIFICATION, dailyFilter, "finished_at");
|
||||
if(daily == null || daily.getValueInstant("finished_at") == null)
|
||||
{
|
||||
warnings.add("Daily verification has never run");
|
||||
}
|
||||
else if(daily.getValueInstant("finished_at").isBefore(now.minus(Duration.ofHours(26))))
|
||||
{
|
||||
warnings.add("Daily verification last ran more than 26 hours ago");
|
||||
}
|
||||
QRecord failure = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(
|
||||
new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"), new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "FAIL")), "id");
|
||||
QRecord fullPass = latest(store, WormAuditTables.VERIFICATION, new QQueryFilter(
|
||||
new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "CHAIN"), new QFilterCriteria("trigger", QCriteriaOperator.EQUALS, "SCHEDULE"),
|
||||
new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "PASS")), "id");
|
||||
QRecord openBreak = failure != null && (fullPass == null || failure.getValueLong("id") > fullPass.getValueLong("id")) ? failure : null;
|
||||
QRecord export = store.last(WormAuditTables.EXPORT);
|
||||
QRecord anchor = store.anchor();
|
||||
long anchorId = anchor.getValueLong("anchor_link_id");
|
||||
long exported = export == null ? anchorId : export.getValueLong("to_link_id");
|
||||
long lag = store.count(WormAuditTables.CHAIN, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, exported)));
|
||||
List<String> gaps = gaps(store, anchorId);
|
||||
List<QRecord> oldest = store.summary("audit", new QQueryFilter().withOrderBy(new QFilterOrderBy("timestamp", true)).withLimit(1), Set.of("timestamp"));
|
||||
Instant oldestAt = oldest.isEmpty() ? null : oldest.getFirst().getValueInstant("timestamp");
|
||||
List<QRecord> purges = store.query(WormAuditTables.PURGE, new QQueryFilter()
|
||||
.withOrderBy(new QFilterOrderBy("requested_at", false)).withOrderBy(new QFilterOrderBy("id", false)).withLimit(10));
|
||||
warnings.addAll(PostgresSetupWarnings.load(transaction, config));
|
||||
transaction.commit();
|
||||
return (new WormAuditHomeData(links, lastLink == null ? null : lastLink.getValueInstant("chained_at"), backlog, settling,
|
||||
config.getSettleHorizonSeconds(), verification, openBreak, export, lag,
|
||||
export == null || export.getValueInstant("sealed_at") == null ? null : Math.max(0, Duration.between(export.getValueInstant("sealed_at"), now).toHours()),
|
||||
List.copyOf(gaps), config.getExportBucketAlias(), capability, reason, config.getRetentionYears(), anchorId == 0 ? null : anchor,
|
||||
oldestAt, oldestAt == null ? null : oldestAt.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant(), List.copyOf(purges), List.copyOf(warnings)));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
transaction.rollback();
|
||||
throw (new QException("Cannot load WORM audit home summary"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read one result, using id to break equal-time ties deterministically.
|
||||
*******************************************************************************/
|
||||
private static QRecord latest(AuditStore store, String table, QQueryFilter filter, String time) throws QException
|
||||
{
|
||||
List<QRecord> rows = store.query(table, filter.withOrderBy(new QFilterOrderBy(time, false))
|
||||
.withOrderBy(new QFilterOrderBy("id", false)).withLimit(1));
|
||||
return (rows.isEmpty() ? null : rows.getFirst());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Sweep export intervals in bounded pages; never enumerate links or objects.
|
||||
** The writer appends ranges in id order. Overlapping ranges extend coverage.
|
||||
*******************************************************************************/
|
||||
private static List<String> gaps(AuditStore store, long anchor) throws QException
|
||||
{
|
||||
List<String> gaps = new ArrayList<>();
|
||||
long next = anchor + 1;
|
||||
long cursor = 0;
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> page = store.page(WormAuditTables.EXPORT, cursor, 1000);
|
||||
if(page.isEmpty())
|
||||
{
|
||||
return (gaps);
|
||||
}
|
||||
for(QRecord export : page)
|
||||
{
|
||||
long from = export.getValueLong("from_link_id");
|
||||
long to = export.getValueLong("to_link_id");
|
||||
if(from > next)
|
||||
{
|
||||
gaps.add(next + "–" + (from - 1));
|
||||
}
|
||||
next = Math.max(next, to + 1);
|
||||
}
|
||||
cursor = page.getLast().getValueLong("id");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.app;
|
||||
|
||||
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stacked, wrapping definition lists. Colors follow BackupsHomeView exactly:
|
||||
** inherited text/link colors, translucent neutral borders, muted opacity. Only
|
||||
** allowlisted summary fields are rendered; no audit content is ever inspected.
|
||||
*******************************************************************************/
|
||||
public final class WormAuditHomeView
|
||||
{
|
||||
private static final DateTimeFormatter WHEN = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC);
|
||||
private static final String CARD = "border:1px solid rgba(127,127,127,.3);border-radius:8px;padding:1rem;min-width:0;overflow-wrap:anywhere;";
|
||||
private static final String LINK = "color:inherit;text-decoration:underline;text-underline-offset:2px;";
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private WormAuditHomeView()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Full home with explicit empty states and no fixed-width content.
|
||||
*******************************************************************************/
|
||||
public static String render(WormAuditHomeData data, String prefix)
|
||||
{
|
||||
StringBuilder html = new StringBuilder("<div class=\"worm-audit-home\" style=\"display:grid;gap:1rem;min-width:0;max-width:100%;\">");
|
||||
open(html, "Chain status");
|
||||
row(html, "Links chained", data.links());
|
||||
row(html, "Last link", when(data.lastLinkAt(), "No links chained yet"));
|
||||
row(html, "Unchained backlog", data.backlog() + " settled; " + data.settling() + " settling (" + data.settleSeconds() + " s horizon)");
|
||||
QRecord verification = data.lastVerification();
|
||||
if(verification == null)
|
||||
{
|
||||
row(html, "Last verification", "No verification has run yet");
|
||||
}
|
||||
else
|
||||
{
|
||||
linkedRow(html, "Last verification", recordLink(prefix, WormAuditTables.VERIFICATION, verification.getValueLong("id"), "Verification #" + verification.getValueLong("id")));
|
||||
row(html, "Verified at", when(verification.getValueInstant("finished_at"), "Not recorded"));
|
||||
row(html, "Result", verification.getValueString("outcome") + "; " + verification.getValueLong("links_checked") + " links checked");
|
||||
}
|
||||
QRecord broken = data.openBreak();
|
||||
if(broken == null)
|
||||
{
|
||||
row(html, "Open chain breaks", verification == null ? "Unknown until verification runs" : "None reported");
|
||||
}
|
||||
else
|
||||
{
|
||||
linkedRow(html, "Open chain breaks", recordLink(prefix, WormAuditTables.VERIFICATION, broken.getValueLong("id"), broken.getValueLong("break_count") + " reported"));
|
||||
linkedRow(html, "First break link", recordLink(prefix, WormAuditTables.CHAIN, broken.getValueLong("first_break_link_id"), "Link #" + broken.getValueLong("first_break_link_id")));
|
||||
row(html, "First break audit", broken.getValueLong("first_break_audit_id"));
|
||||
row(html, "Reason", broken.getValueString("first_break_reason"));
|
||||
}
|
||||
close(html);
|
||||
open(html, "Export status");
|
||||
QRecord export = data.lastExport();
|
||||
if(export == null)
|
||||
{
|
||||
row(html, "Last sealed export", "No sealed exports yet");
|
||||
}
|
||||
else
|
||||
{
|
||||
linkedRow(html, "Last sealed export", recordLink(prefix, WormAuditTables.EXPORT, export.getValueLong("id"), "Export #" + export.getValueLong("id")));
|
||||
row(html, "Link range", export.getValueLong("from_link_id") + "–" + export.getValueLong("to_link_id"));
|
||||
row(html, "Sealed at", when(export.getValueInstant("sealed_at"), "Not recorded"));
|
||||
row(html, "Retain until", when(export.getValueInstant("retain_until"), "Not recorded"));
|
||||
row(html, "Lock mode", export.getValueString("lock_mode"));
|
||||
}
|
||||
row(html, "Export lag", data.exportLag() + " links; " + (data.exportAgeHours() == null ? "no previous export time" : data.exportAgeHours() + " h since last export"));
|
||||
row(html, "Gaps in sealed ranges", data.exportGaps().isEmpty() ? (export == null ? "No sealed ranges to check" : "None") : String.join(", ", data.exportGaps()));
|
||||
row(html, "Bucket alias", data.bucketAlias());
|
||||
row(html, "Object Lock capability", data.lockCapability() + (data.lockReason() == null ? "" : ": " + data.lockReason()));
|
||||
close(html);
|
||||
open(html, "Retention");
|
||||
row(html, "Configured retention", data.retentionYears() + " years");
|
||||
QRecord anchor = data.anchor();
|
||||
row(html, "Purge anchor", anchor == null ? "None" : "Link #" + anchor.getValueLong("anchor_link_id") + "; hash " + shortHash(anchor.getValueString("anchor_hash")));
|
||||
row(html, "Oldest live audit", when(data.oldestAudit(), "No live audits"));
|
||||
row(html, "Next eligible purge", when(data.nextPurge(), "No live audits to purge"));
|
||||
close(html);
|
||||
open(html, "Recent purges");
|
||||
if(data.recentPurges().isEmpty())
|
||||
{
|
||||
row(html, "History", "No purge requests yet");
|
||||
}
|
||||
for(QRecord purge : data.recentPurges())
|
||||
{
|
||||
linkedRow(html, "Purge", recordLink(prefix, WormAuditTables.PURGE, purge.getValueLong("id"), "Purge #" + purge.getValueLong("id")));
|
||||
row(html, "Requested at", when(purge.getValueInstant("requested_at"), "Not recorded"));
|
||||
row(html, "Requested by", purge.getValueString("requested_by"));
|
||||
row(html, "Outcome", purge.getValueString("outcome"));
|
||||
row(html, "Cutoff", when(purge.getValueInstant("cutoff"), "Not recorded"));
|
||||
row(html, "Deleted", purge.getValueLong("audits_deleted") + " audits; " + purge.getValueLong("details_deleted") + " details; " + purge.getValueLong("links_deleted") + " links");
|
||||
row(html, "Reason", purge.getValueString("reason"));
|
||||
}
|
||||
close(html);
|
||||
if(!data.warnings().isEmpty())
|
||||
{
|
||||
open(html, "Setup warnings");
|
||||
for(String warning : data.warnings())
|
||||
{
|
||||
row(html, "Warning", warning);
|
||||
}
|
||||
close(html);
|
||||
}
|
||||
return (html.append("</div>").toString());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Honest compact status, owned by and linked to the full home.
|
||||
*******************************************************************************/
|
||||
public static String renderHealthCard(WormAuditHomeData data, String homePath)
|
||||
{
|
||||
String status = data.openBreak() != null ? "Chain breaks reported" : data.lastVerification() == null ? "Not verified yet"
|
||||
: data.warnings().isEmpty() ? "No open chain breaks reported" : "Setup needs attention";
|
||||
return ("<div style=\"" + CARD + "\"><strong>" + esc(status) + "</strong><p>" + data.links() + " links; " + data.backlog()
|
||||
+ " settled audits waiting; " + data.exportLag() + " links awaiting export.</p><a style=\"" + LINK + "\" href=\"" + esc(homePath) + "\">Open WORM Audit</a></div>");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Section and definition list, with fluid wrapping label/value rows.
|
||||
*******************************************************************************/
|
||||
private static void open(StringBuilder html, String title)
|
||||
{
|
||||
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0 0 .75rem;font-size:1rem;font-weight:600;\">")
|
||||
.append(esc(title)).append("</h3><dl style=\"margin:0;font-size:.9rem;\">");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Close a section.
|
||||
*******************************************************************************/
|
||||
private static void close(StringBuilder html)
|
||||
{
|
||||
html.append("</dl></section>");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Escape every value, including reasons and runtime identities.
|
||||
*******************************************************************************/
|
||||
private static void row(StringBuilder html, String label, Object value)
|
||||
{
|
||||
linkedRow(html, label, esc(value == null ? "Not recorded" : value.toString()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Only internally built and escaped links may supply HTML here.
|
||||
*******************************************************************************/
|
||||
private static void linkedRow(StringBuilder html, String label, String value)
|
||||
{
|
||||
html.append("<div style=\"display:flex;flex-wrap:wrap;gap:.25rem .75rem;margin:.4rem 0;min-width:0;\"><dt style=\"opacity:.75;flex:0 1 12rem;\">")
|
||||
.append(esc(label)).append("</dt><dd style=\"margin:0;flex:1 1 14rem;min-width:0;overflow-wrap:anywhere;\">").append(value).append("</dd></div>");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Standard QQQ record route; identifiers are numeric, never content.
|
||||
*******************************************************************************/
|
||||
private static String recordLink(String prefix, String table, Long id, String text)
|
||||
{
|
||||
return (id == null ? "Not recorded" : "<a style=\"" + LINK + "\" href=\"" + esc(prefix + "/" + table + "/" + id) + "\">" + esc(text) + "</a>");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** UTC time with an explicit empty state.
|
||||
*******************************************************************************/
|
||||
private static String when(Instant time, String empty)
|
||||
{
|
||||
return (time == null ? empty : WHEN.format(time));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Keep hashes compact even on a narrow screen.
|
||||
*******************************************************************************/
|
||||
private static String shortHash(String hash)
|
||||
{
|
||||
return (hash == null ? "Not recorded" : hash.substring(0, Math.min(12, hash.length())) + "…");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** HTML text and quoted attribute escaping.
|
||||
*******************************************************************************/
|
||||
private static String esc(String text)
|
||||
{
|
||||
return (text == null ? "" : text.replace("&", "&").replace("<", "<").replace(">", ">").replace("\"", """).replace("'", "'"));
|
||||
}
|
||||
}
|
||||
@@ -44,10 +44,12 @@ import com.kingsrook.qbits.wormaudit.guard.AppendOnlyGuard;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
@@ -91,6 +93,57 @@ public class AuditStore
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Project only summary fields; home reads never fetch audit content.
|
||||
*******************************************************************************/
|
||||
public List<QRecord> summary(String table, QQueryFilter filter, Set<String> fields) throws QException
|
||||
{
|
||||
QueryInput input = new QueryInput(table).withFilter(filter).withTransaction(transaction)
|
||||
.withFieldNamesToInclude(fields).withShouldFetchHeavyFields(true);
|
||||
input.withPermissionBypass("WORM audit home summary after widget permission check");
|
||||
return (new QueryAction().execute(input).getRecords());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Aggregate counts avoid loading the live chain for the home.
|
||||
*******************************************************************************/
|
||||
public long count(String table, QQueryFilter filter) throws QException
|
||||
{
|
||||
CountInput input = new CountInput(table).withFilter(filter).withTransaction(transaction);
|
||||
input.withPermissionBypass("WORM audit home count after widget permission check");
|
||||
return (new CountAction().execute(input).getCount());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Count the same AUDIT anti-join as the writer, split at the settle horizon.
|
||||
*******************************************************************************/
|
||||
public long unchainedCount(Instant cutoff, boolean settled) throws QException
|
||||
{
|
||||
if(isRdbms())
|
||||
{
|
||||
String sql = "SELECT count(*) FROM " + physical("audit") + " a WHERE a." + column("audit", "timestamp")
|
||||
+ (settled ? " < ?" : " >= ?") + unchainedPredicate();
|
||||
return (sqlIds(sql, cutoff).getFirst());
|
||||
}
|
||||
Set<Long> covered = summary(WormAuditTables.CHAIN, new QQueryFilter(new QFilterCriteria("kind", QCriteriaOperator.EQUALS, "AUDIT")), Set.of("audit_id"))
|
||||
.stream().map(r -> r.getValueLong("audit_id")).collect(Collectors.toSet());
|
||||
return (summary("audit", new QQueryFilter(), Set.of("id", "timestamp")).stream()
|
||||
.filter(r -> r.getValueInstant("timestamp") != null && r.getValueInstant("timestamp").isBefore(cutoff) == settled
|
||||
&& !covered.contains(r.getValueLong("id"))).count());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Shared SQL coverage predicate for both writer selection and home counts.
|
||||
*******************************************************************************/
|
||||
private String unchainedPredicate()
|
||||
{
|
||||
return (" AND NOT EXISTS (SELECT 1 FROM worm_audit_chain c WHERE c.audit_id=a." + column("audit", "id") + " AND c.kind='AUDIT')");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** All records ordered by id.
|
||||
*******************************************************************************/
|
||||
@@ -219,8 +272,7 @@ public class AuditStore
|
||||
if(isRdbms())
|
||||
{
|
||||
String sql = "SELECT a." + column("audit", "id") + " FROM " + physical("audit") + " a WHERE a." + column("audit", "timestamp")
|
||||
+ " < ? AND NOT EXISTS (SELECT 1 FROM worm_audit_chain c WHERE c.audit_id=a." + column("audit", "id")
|
||||
+ " AND c.kind='AUDIT') ORDER BY a." + column("audit", "timestamp") + ", a." + column("audit", "id") + " LIMIT " + limit;
|
||||
+ " < ?" + unchainedPredicate() + " ORDER BY a." + column("audit", "timestamp") + ", a." + column("audit", "id") + " LIMIT " + limit;
|
||||
Map<Long, QRecord> rows = index(in("audit", "id", sqlIds(sql, cutoff)));
|
||||
return (rows.values().stream().sorted(Comparator.comparing((QRecord r) -> r.getValueInstant("timestamp"))
|
||||
.thenComparing(r -> r.getValueLong("id"))).toList());
|
||||
@@ -251,6 +303,15 @@ public class AuditStore
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Only the in-memory backend may use an unbounded completeness index.
|
||||
*******************************************************************************/
|
||||
public boolean isMemory()
|
||||
{
|
||||
return ("memory".equals(QContext.getQInstance().getBackendForTable(WormAuditTables.CHAIN).getBackendType()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fetch one keyset page, never using an offset or a whole-chain read.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.export;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.Serializable;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.fasterxml.jackson.core.JsonParser;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.fasterxml.jackson.databind.DeserializationFeature;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.kingsrook.qbits.objectstorage.api.StoredObject;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Version-one export encoding and temporary files. JSON errors never echo input.
|
||||
*******************************************************************************/
|
||||
public class ExportFiles implements AutoCloseable
|
||||
{
|
||||
public static final List<String> NAMES = List.of("links.jsonl", "checkpoints.jsonl", "manifest.json");
|
||||
private static final ObjectMapper JSON = new ObjectMapper().enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION)
|
||||
.enable(DeserializationFeature.USE_LONG_FOR_INTS).enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
|
||||
private final Path directory;
|
||||
|
||||
/*******************************************************************************
|
||||
** Create a private temporary directory, removed on all normal failure paths.
|
||||
*******************************************************************************/
|
||||
public ExportFiles() throws IOException
|
||||
{
|
||||
directory = Files.createTempDirectory("worm-audit-");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve only one of the fixed filenames, never a manifest-supplied path.
|
||||
*******************************************************************************/
|
||||
public Path file(String name)
|
||||
{
|
||||
if(!NAMES.contains(name))
|
||||
{
|
||||
throw (new IllegalArgumentException("Unknown export file"));
|
||||
}
|
||||
return (directory.resolve(name));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Parse the JCS schema with strict duplicate and trailing-token rejection.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> parse(byte[] bytes) throws IOException
|
||||
{
|
||||
try
|
||||
{
|
||||
return (JSON.readValue(bytes, new TypeReference<Map<String, Object>>() {}));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IOException("Invalid sealed JSON"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Wire representation of a chain row, excluding database-only timestamps.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> link(QRecord row)
|
||||
{
|
||||
return (Map.of("id", row.getValueLong("id"), "kind", row.getValueString("kind"),
|
||||
"auditId", row.getValueLong("audit_id"), "detailIds", AuditStore.detailIds(row),
|
||||
"prevHash", row.getValueString("prev_hash"), "linkHash", row.getValueString("link_hash")));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Preserve every checkpoint field, representing the timestamp as UTC text.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> checkpoint(QRecord row)
|
||||
{
|
||||
Map<String, Object> fields = new LinkedHashMap<>();
|
||||
for(String key : List.of("id", "from_link_id", "to_link_id", "link_count"))
|
||||
{
|
||||
fields.put(key, row.getValueLong(key));
|
||||
}
|
||||
for(String key : List.of("head_hash", "merkle_root", "algorithm", "key_id", "public_key", "signature"))
|
||||
{
|
||||
fields.put(key, row.getValueString(key));
|
||||
}
|
||||
fields.put("created_at", row.getValueInstant("created_at").toString());
|
||||
return (fields);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Convert a checkpoint envelope back to the phase (a) signature input.
|
||||
*******************************************************************************/
|
||||
public static QRecord record(Map<String, Object> fields)
|
||||
{
|
||||
QRecord row = new QRecord();
|
||||
fields.forEach((key, value) -> row.setValue(key, (Serializable) value));
|
||||
return (row);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Compute a digest without materializing a file.
|
||||
*******************************************************************************/
|
||||
public static String sha256(Path path) throws Exception
|
||||
{
|
||||
MessageDigest digest = MessageDigest.getInstance("SHA-256");
|
||||
try(InputStream input = Files.newInputStream(path))
|
||||
{
|
||||
byte[] buffer = new byte[65536];
|
||||
int read;
|
||||
while((read = input.read(buffer)) != -1)
|
||||
{
|
||||
digest.update(buffer, 0, read);
|
||||
}
|
||||
}
|
||||
return (HexFormat.of().formatHex(digest.digest()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A lock echo is required even if a provider advertises the capability.
|
||||
*******************************************************************************/
|
||||
public static boolean locked(StoredObject object, java.time.Instant until)
|
||||
{
|
||||
return (object != null && "COMPLIANCE".equals(object.getLockMode()) && object.getLockRetainUntil() != null
|
||||
&& !object.getLockRetainUntil().isBefore(until));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A null version belongs to unversioned storage and cannot be sealed.
|
||||
*******************************************************************************/
|
||||
public static boolean validVersion(String versionId)
|
||||
{
|
||||
return (versionId != null && !versionId.isBlank() && !"null".equals(versionId));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fetch covered detail rows in bounded IN batches.
|
||||
*******************************************************************************/
|
||||
public static Map<Long, QRecord> details(AuditStore store, List<QRecord> links) throws Exception
|
||||
{
|
||||
List<Long> ids = new ArrayList<>();
|
||||
for(QRecord link : links)
|
||||
{
|
||||
List<Long> covered = AuditStore.detailIds(link);
|
||||
if(covered.size() > 250)
|
||||
{
|
||||
throw (new IOException("Invalid detail coverage"));
|
||||
}
|
||||
ids.addAll(covered);
|
||||
}
|
||||
Map<Long, QRecord> rows = new LinkedHashMap<>();
|
||||
for(int start = 0; start < ids.size(); start += 1000)
|
||||
{
|
||||
rows.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + 1000, ids.size())))));
|
||||
}
|
||||
return (rows);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove only the three owned files and their directory.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void close() throws IOException
|
||||
{
|
||||
IOException failure = null;
|
||||
for(String name : NAMES)
|
||||
{
|
||||
try
|
||||
{
|
||||
Files.deleteIfExists(file(name));
|
||||
}
|
||||
catch(IOException e)
|
||||
{
|
||||
failure = new IOException("Cannot remove temporary export files");
|
||||
}
|
||||
}
|
||||
Files.deleteIfExists(directory);
|
||||
if(failure != null)
|
||||
{
|
||||
throw (failure);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.export;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import com.kingsrook.qbits.objectstorage.api.Capability;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.LockMode;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
|
||||
import com.kingsrook.qbits.objectstorage.api.Page;
|
||||
import com.kingsrook.qbits.objectstorage.api.PutRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.StorageBucket;
|
||||
import com.kingsrook.qbits.objectstorage.api.StorageKeys;
|
||||
import com.kingsrook.qbits.objectstorage.api.StoredObject;
|
||||
import com.kingsrook.qbits.objectstorage.config.BucketRole;
|
||||
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
|
||||
import com.kingsrook.qbits.objectstorage.internal.SecretRefs;
|
||||
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Thin system-caller adapter to qbit-object-storage.
|
||||
*******************************************************************************/
|
||||
public class ObjectStorageSealedStore implements SealedStore
|
||||
{
|
||||
private final StorageBucket bucket;
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind the recorded alias, so old exports survive configuration changes.
|
||||
*******************************************************************************/
|
||||
public ObjectStorageSealedStore(String alias, String runId)
|
||||
{
|
||||
bucket = ObjectStorage.asSystem("worm_audit", runId).bucket(alias);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Refuse unknown, partial, or unsupported lock capabilities.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean supportsLock()
|
||||
{
|
||||
return (bucket.config().getRole() == BucketRole.BACKUP
|
||||
&& bucket.capability(Capability.OBJECT_LOCK) == CapabilityStatus.SUPPORTED);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Report the provider capability without collapsing UNKNOWN into UNSUPPORTED.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public CapabilityStatus lockCapability()
|
||||
{
|
||||
return (bucket.capability(Capability.OBJECT_LOCK));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stream a bounded upload and close the source even when the provider fails.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String put(String key, Path file, Instant retainUntil) throws Exception
|
||||
{
|
||||
try(InputStream input = Files.newInputStream(file))
|
||||
{
|
||||
return (bucket.put(PutRequest.of(key, input).withContentLength(Files.size(file))
|
||||
.withObjectLock(LockMode.COMPLIANCE, retainUntil)).getVersionId());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read retention metadata.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public StoredObject head(String key)
|
||||
{
|
||||
return (bucket.head(key).orElse(null));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** RC.2 has no version GET facade. Reuse its configured system connection,
|
||||
** key normalization and limiter; closing the stream closes the S3 response.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public InputStream getVersion(String key, String versionId) throws IOException
|
||||
{
|
||||
if(!ExportFiles.validVersion(versionId))
|
||||
{
|
||||
throw (new IOException("Missing sealed object version"));
|
||||
}
|
||||
var connection = ObjectStorageRuntime.connection(bucket.config().getConnection());
|
||||
try(var ignored = connection.getRateLimiter().acquire())
|
||||
{
|
||||
var input = connection.client().getObject(GetObjectRequest.builder()
|
||||
.bucket(SecretRefs.resolveSetting(bucket.config().getBucketName())).key(StorageKeys.normalize(key)).versionId(versionId).build());
|
||||
if(!versionId.equals(input.response().versionId()))
|
||||
{
|
||||
input.close();
|
||||
throw (new IOException("Storage returned a different object version"));
|
||||
}
|
||||
return (input);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Forward bounded listing.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public Page<ObjectSummary> list(ListRequest request)
|
||||
{
|
||||
return (bucket.list(request));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.export;
|
||||
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.sql.Connection;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.chain.Hashes;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
|
||||
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify the sealed evidence before comparing it to a repeatable live snapshot.
|
||||
** No exported content or provider/parser exception text enters diagnostics.
|
||||
*******************************************************************************/
|
||||
public class SealedExportVerifier
|
||||
{
|
||||
private final WormAuditQBitConfig config;
|
||||
private final SealedStore suppliedStorage;
|
||||
|
||||
/*******************************************************************************
|
||||
** Findings plus the number verified without live rows after a retention purge.
|
||||
*******************************************************************************/
|
||||
public record Report(VerificationResult result, long purged)
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Production entry point.
|
||||
*******************************************************************************/
|
||||
public SealedExportVerifier(WormAuditQBitConfig config)
|
||||
{
|
||||
this(config, null);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Testable storage boundary.
|
||||
*******************************************************************************/
|
||||
public SealedExportVerifier(WormAuditQBitConfig config, SealedStore storage)
|
||||
{
|
||||
this.config = config;
|
||||
this.suppliedStorage = storage;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Persist exactly one verification row for this selected export, even on failure.
|
||||
*******************************************************************************/
|
||||
public Report verify(QRecord export, String trigger) throws QException
|
||||
{
|
||||
VerificationResult failure = new VerificationResult();
|
||||
failure.scope(0, export.getValueLong("from_link_id"), export.getValueLong("to_link_id"));
|
||||
Report report;
|
||||
WormAuditChainer.JVM_LOCK.lock();
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN));
|
||||
ExportFiles files = new ExportFiles())
|
||||
{
|
||||
try
|
||||
{
|
||||
if(transaction instanceof RDBMSTransaction rdbms)
|
||||
{
|
||||
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
|
||||
}
|
||||
SealedStore storage = suppliedStorage == null
|
||||
? config.getSealedStoreSupplier() == null || !config.getExportBucketAlias().equals(export.getValueString("bucket_alias"))
|
||||
? new ObjectStorageSealedStore(export.getValueString("bucket_alias"), "verify-" + export.getValueLong("id"))
|
||||
: config.getSealedStoreSupplier().get() : suppliedStorage;
|
||||
Map<String, Object> versions = ExportFiles.parse(export.getValueString("object_versions").getBytes(StandardCharsets.UTF_8));
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String key = export.getValueString("object_prefix") + name;
|
||||
if(!(versions.get(name) instanceof String version) || !ExportFiles.validVersion(version))
|
||||
{
|
||||
failure.add(null, null, "object " + name + " has no recorded sealed version id");
|
||||
throw (new IOException("Missing sealed object version"));
|
||||
}
|
||||
var head = storage.head(key);
|
||||
if(head != null && !version.equals(head.getVersionId()))
|
||||
{
|
||||
failure.add(null, null, "object " + name + " was overwritten after sealing (current version " + head.getVersionId()
|
||||
+ ", sealed version " + version + " is still retained under Object Lock)");
|
||||
}
|
||||
if(!ExportFiles.locked(head, export.getValueInstant("retain_until")))
|
||||
{
|
||||
failure.add(null, null, "object " + name + " lacks the recorded COMPLIANCE retention");
|
||||
}
|
||||
}
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
try(InputStream input = storage.getVersion(export.getValueString("object_prefix") + name, (String) versions.get(name)))
|
||||
{
|
||||
Files.copy(input, files.file(name));
|
||||
}
|
||||
}
|
||||
report = inspect(files, export, new AuditStore(transaction));
|
||||
for(var item : failure.getBreaks())
|
||||
{
|
||||
report.result().add(item.linkId(), item.auditId(), item.reason());
|
||||
}
|
||||
transaction.commit();
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
transaction.rollback();
|
||||
throw (e);
|
||||
}
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
failure.add(null, null, "Sealed export cannot be read or has invalid structure");
|
||||
report = new Report(failure, 0);
|
||||
}
|
||||
finally
|
||||
{
|
||||
WormAuditChainer.JVM_LOCK.unlock();
|
||||
}
|
||||
VerificationRuns.save(report.result(), trigger, export.getValueLong("id"));
|
||||
return (report);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Inspect local files, also used by the exporter before committing any objects.
|
||||
*******************************************************************************/
|
||||
public Report inspect(ExportFiles files, QRecord export, AuditStore store) throws Exception
|
||||
{
|
||||
VerificationResult result = new VerificationResult();
|
||||
long from = export.getValueLong("from_link_id");
|
||||
long to = export.getValueLong("to_link_id");
|
||||
result.scope(0, from, to);
|
||||
if(!Objects.equals(ExportFiles.sha256(files.file("manifest.json")), export.getValueString("manifest_sha256")))
|
||||
{
|
||||
result.add(null, null, "object manifest.json hash does not match the export row");
|
||||
return (new Report(result, 0));
|
||||
}
|
||||
if(Files.size(files.file("manifest.json")) > 65536)
|
||||
{
|
||||
result.add(null, null, "Manifest exceeds the format size limit");
|
||||
return (new Report(result, 0));
|
||||
}
|
||||
Map<String, Object> manifest = ExportFiles.parse(Files.readAllBytes(files.file("manifest.json")));
|
||||
if(!"worm-audit-export/1".equals(manifest.get("format")) || from < 1 || to < from
|
||||
|| !Objects.equals(manifest.get("fromLinkId"), from) || !Objects.equals(manifest.get("toLinkId"), to)
|
||||
|| !Objects.equals(manifest.get("linkCount"), to - from + 1)
|
||||
|| !Objects.equals(manifest.get("createdAt"), export.getValueInstant("sealed_at").toString())
|
||||
|| !"COMPLIANCE".equals(export.getValueString("lock_mode"))
|
||||
|| !Objects.equals(export.getValueString("manifest_key"), export.getValueString("object_prefix") + "manifest.json"))
|
||||
{
|
||||
result.add(null, null, "Manifest metadata does not match the export row");
|
||||
}
|
||||
QQueryFilter previousFilter = new QQueryFilter().withOrderBy(new QFilterOrderBy("id", false)).withLimit(1);
|
||||
if(export.getValueLong("id") != null)
|
||||
{
|
||||
previousFilter.withCriteria(new QFilterCriteria("id", QCriteriaOperator.LESS_THAN, export.getValueLong("id")));
|
||||
}
|
||||
List<QRecord> previousRows = store.query(WormAuditTables.EXPORT, previousFilter);
|
||||
QRecord previous = previousRows.isEmpty() ? null : previousRows.getFirst();
|
||||
String previousHash = previous == null ? null : previous.getValueString("manifest_sha256");
|
||||
if(!Objects.equals(manifest.get("prevManifestSha256"), previousHash)
|
||||
|| !Objects.equals(export.getValueString("prev_manifest_sha256"), previousHash)
|
||||
|| (previous != null && previous.getValueLong("to_link_id") != from - 1))
|
||||
{
|
||||
result.add(null, null, "prevManifestSha256 does not match the previous export row");
|
||||
}
|
||||
Map<?, ?> hashes = (Map<?, ?>) manifest.get("files");
|
||||
for(String name : List.of("links.jsonl", "checkpoints.jsonl"))
|
||||
{
|
||||
if(!Objects.equals(ExportFiles.sha256(files.file(name)), hashes.get(name)))
|
||||
{
|
||||
result.add(null, null, "object " + name + " hash does not match the manifest");
|
||||
}
|
||||
}
|
||||
if(!result.passed())
|
||||
{
|
||||
return (new Report(result, 0));
|
||||
}
|
||||
return (checkLines(files, export, store, manifest, result));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Consume links alongside their checkpoint envelopes. Only one checkpoint's
|
||||
** hashes and one content page are retained, never the whole export.
|
||||
*******************************************************************************/
|
||||
private Report checkLines(ExportFiles files, QRecord export, AuditStore store, Map<String, Object> manifest, VerificationResult result) throws Exception
|
||||
{
|
||||
long from = export.getValueLong("from_link_id");
|
||||
long to = export.getValueLong("to_link_id");
|
||||
long cursor = from - 1;
|
||||
long checkpointCount = 0;
|
||||
long checked = 0;
|
||||
long purged = 0;
|
||||
long anchor = store.anchor().getValueLong("anchor_link_id");
|
||||
String previousHash = from == 1 ? Hashes.GENESIS : null;
|
||||
if(cursor == anchor)
|
||||
{
|
||||
previousHash = store.anchor().getValueString("anchor_hash");
|
||||
}
|
||||
long lastCheckpoint = 0;
|
||||
try(BufferedReader links = Files.newBufferedReader(files.file("links.jsonl"), StandardCharsets.UTF_8);
|
||||
BufferedReader checkpoints = Files.newBufferedReader(files.file("checkpoints.jsonl"), StandardCharsets.UTF_8))
|
||||
{
|
||||
String line;
|
||||
while((line = checkpoints.readLine()) != null)
|
||||
{
|
||||
QRecord checkpoint = ExportFiles.record(ExportFiles.parse(line.getBytes(StandardCharsets.UTF_8)));
|
||||
long checkpointId = checkpoint.getValueLong("id");
|
||||
String label = "checkpoint " + WormAuditVerifier.number(checkpointId) + ": ";
|
||||
long end = checkpoint.getValueLong("to_link_id");
|
||||
long start = checkpoint.getValueLong("from_link_id");
|
||||
long count = checkpoint.getValueLong("link_count");
|
||||
if(start != cursor + 1 || end < start || end > to || count != end - start + 1
|
||||
|| count > config.getCheckpointEveryLinks() || checkpointId <= lastCheckpoint)
|
||||
{
|
||||
result.add(null, null, label + "invalid or incomplete checkpoint range");
|
||||
break;
|
||||
}
|
||||
if(!Checkpoints.verify(checkpoint, config.getTrustedPublicKeys().get(checkpoint.getValueString("key_id"))))
|
||||
{
|
||||
result.add(null, null, label + "signature does not verify");
|
||||
}
|
||||
if(checkpointCount == 0 && !Objects.equals(export.getValueLong("from_checkpoint_id"), checkpointId))
|
||||
{
|
||||
result.add(null, null, label + "first checkpoint does not match the export row");
|
||||
}
|
||||
List<String> hashes = new ArrayList<>();
|
||||
while(cursor < end)
|
||||
{
|
||||
List<Map<String, Object>> page = new ArrayList<>();
|
||||
for(int i = 0, size = (int) Math.min(1000, end - cursor); i < size; i++)
|
||||
{
|
||||
String linkLine = links.readLine();
|
||||
if(linkLine == null)
|
||||
{
|
||||
result.add(cursor + 1, null, "Sealed links are missing from the checkpoint range");
|
||||
result.scope(checked, from, to);
|
||||
return (new Report(result, purged));
|
||||
}
|
||||
page.add(ExportFiles.parse(linkLine.getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
compareLive(page, store, export, anchor, result);
|
||||
for(Map<String, Object> document : page)
|
||||
{
|
||||
Map<?, ?> link = (Map<?, ?>) document.get("link");
|
||||
long id = (Long) link.get("id");
|
||||
long auditId = (Long) link.get("auditId");
|
||||
String prefix = "link " + WormAuditVerifier.number(id) + " (audit " + WormAuditVerifier.number(auditId) + "): ";
|
||||
String prev = (String) link.get("prevHash");
|
||||
String hash = (String) link.get("linkHash");
|
||||
if(id != cursor + 1 || (previousHash != null && !Objects.equals(prev, previousHash)))
|
||||
{
|
||||
result.add(id, auditId, prefix + "sealed chain continuity differs");
|
||||
}
|
||||
Map<?, ?> content = (Map<?, ?>) document.get("content");
|
||||
Map<?, ?> audit = (Map<?, ?>) content.get("audit");
|
||||
List<?> detailIds = (List<?>) link.get("detailIds");
|
||||
List<?> details = (List<?>) content.get("details");
|
||||
List<?> contentIds = details.stream().map(d -> ((Map<?, ?>) d).get("id")).toList();
|
||||
if(!Objects.equals(audit.get("id"), auditId) || !Objects.equals(content.get("v"), 1L)
|
||||
|| !List.of("AUDIT", "LATE_DETAIL").contains(link.get("kind"))
|
||||
|| detailIds.size() > 250 || !detailIds.equals(contentIds)
|
||||
|| detailIds.stream().distinct().count() != detailIds.size()
|
||||
|| ("LATE_DETAIL".equals(link.get("kind")) && detailIds.isEmpty()))
|
||||
{
|
||||
result.add(id, auditId, prefix + "sealed content identifiers or coverage differ");
|
||||
}
|
||||
if(!Hashes.link(prev, Jcs.bytes(content)).equals(hash))
|
||||
{
|
||||
result.add(id, auditId, prefix + "sealed content differs from the chained hash");
|
||||
}
|
||||
hashes.add(hash);
|
||||
previousHash = hash;
|
||||
cursor++;
|
||||
checked++;
|
||||
if(id <= anchor)
|
||||
{
|
||||
purged++;
|
||||
}
|
||||
}
|
||||
}
|
||||
if(!Objects.equals(Hashes.merkle(hashes), checkpoint.getValueString("merkle_root"))
|
||||
|| !Objects.equals(previousHash, checkpoint.getValueString("head_hash")))
|
||||
{
|
||||
result.add(null, null, label + "Merkle root or head hash differs");
|
||||
}
|
||||
lastCheckpoint = checkpointId;
|
||||
checkpointCount++;
|
||||
}
|
||||
if(cursor != to || links.readLine() != null || !Objects.equals(manifest.get("checkpointCount"), checkpointCount)
|
||||
|| !Objects.equals(export.getValueLong("to_checkpoint_id"), lastCheckpoint)
|
||||
|| !Objects.equals(manifest.get("headHash"), previousHash))
|
||||
{
|
||||
result.add(null, null, "Sealed range, counts or head hash differ from the manifest");
|
||||
}
|
||||
}
|
||||
result.scope(checked, from, to);
|
||||
return (new Report(result, purged));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Compare a bounded page by exported ids, so missing live rows cannot vanish
|
||||
** from the verification scope. Purged ids are checked from signed copies only.
|
||||
*******************************************************************************/
|
||||
private void compareLive(List<Map<String, Object>> page, AuditStore store, QRecord export, long anchor, VerificationResult result) throws Exception
|
||||
{
|
||||
List<QRecord> sealed = page.stream().map(document ->
|
||||
{
|
||||
Map<?, ?> link = (Map<?, ?>) document.get("link");
|
||||
List<?> ids = (List<?>) link.get("detailIds");
|
||||
return (new QRecord().withValue("id", (Long) link.get("id")).withValue("audit_id", (Long) link.get("auditId"))
|
||||
.withValue("kind", (String) link.get("kind")).withValue("prev_hash", (String) link.get("prevHash"))
|
||||
.withValue("link_hash", (String) link.get("linkHash"))
|
||||
.withValue("detail_ids", ids.stream().map(Object::toString).collect(java.util.stream.Collectors.joining(","))));
|
||||
}).toList();
|
||||
List<QRecord> liveScope = sealed.stream().filter(r -> r.getValueLong("id") > anchor).toList();
|
||||
Map<Long, QRecord> live = AuditStore.index(store.in(WormAuditTables.CHAIN, "id", liveScope.stream().map(r -> r.getValueLong("id")).toList()));
|
||||
Map<Long, QRecord> audits = AuditStore.index(store.in("audit", "id", liveScope.stream().map(r -> r.getValueLong("audit_id")).toList()));
|
||||
Map<Long, QRecord> details = ExportFiles.details(store, liveScope);
|
||||
for(int i = 0; i < sealed.size(); i++)
|
||||
{
|
||||
QRecord link = sealed.get(i);
|
||||
long id = link.getValueLong("id");
|
||||
long auditId = link.getValueLong("audit_id");
|
||||
if(id <= anchor)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
String prefix = "link " + WormAuditVerifier.number(id) + " (audit " + WormAuditVerifier.number(auditId) + "): ";
|
||||
QRecord current = live.get(id);
|
||||
if(current == null)
|
||||
{
|
||||
result.add(id, auditId, "link " + WormAuditVerifier.number(id) + " is missing from the live chain");
|
||||
}
|
||||
else
|
||||
{
|
||||
for(String field : List.of("kind", "audit_id", "detail_ids", "prev_hash", "link_hash"))
|
||||
{
|
||||
if(!Objects.equals(link.getValue(field), current.getValue(field)))
|
||||
{
|
||||
result.add(id, auditId, prefix + "live chain " + field + " differs from the sealed copy");
|
||||
}
|
||||
}
|
||||
}
|
||||
try
|
||||
{
|
||||
QRecord audit = audits.get(auditId);
|
||||
if(audit == null)
|
||||
{
|
||||
result.add(id, auditId, prefix + "live audit header is missing");
|
||||
}
|
||||
else if(!Arrays.equals(store.canonical(audit, AuditStore.detailIds(link), details), Jcs.bytes(page.get(i).get("content"))))
|
||||
{
|
||||
result.add(id, auditId, prefix + "live content differs from the sealed copy (sealed "
|
||||
+ export.getValueInstant("sealed_at").atZone(ZoneOffset.UTC).toLocalDate() + ")");
|
||||
}
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
result.add(id, auditId, prefix + "live content is missing or invalid");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,390 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.export;
|
||||
|
||||
|
||||
import java.io.BufferedWriter;
|
||||
import java.io.IOException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.sql.ResultSet;
|
||||
import java.sql.Statement;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitProducer;
|
||||
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.chain.Hashes;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Seal the checkpointed suffix under the chainer/purger lock. Only successful
|
||||
** uploads with verified lock echoes produce an export row.
|
||||
*******************************************************************************/
|
||||
public class SealedExporter
|
||||
{
|
||||
private final WormAuditQBitConfig config;
|
||||
private final SealedStore suppliedStorage;
|
||||
|
||||
/*******************************************************************************
|
||||
** A review scope; execution always recomputes it under the writer lock.
|
||||
*******************************************************************************/
|
||||
public record Scope(long from, long to, long checkpoints, QRecord previous)
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Number of links, including the empty checkpoint suffix.
|
||||
*******************************************************************************/
|
||||
public long count()
|
||||
{
|
||||
return (Math.max(0, to - from + 1));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Value-free Review text.
|
||||
*******************************************************************************/
|
||||
public String review()
|
||||
{
|
||||
return (count() == 0 ? "Nothing to seal: no new checkpoint" : WormAuditVerifier.number(count())
|
||||
+ " chain links will be sealed (links " + WormAuditVerifier.number(from) + "–" + WormAuditVerifier.number(to)
|
||||
+ ", " + WormAuditVerifier.number(checkpoints) + " checkpoints)");
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A completed, empty, or failed run suitable for a Result line.
|
||||
*******************************************************************************/
|
||||
public record Result(QRecord row, String message, boolean failed)
|
||||
{
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Production storage is resolved lazily, so an empty run needs no bucket.
|
||||
*******************************************************************************/
|
||||
public SealedExporter(WormAuditQBitConfig config)
|
||||
{
|
||||
this(config, null);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Inject the tiny storage boundary for deterministic tests.
|
||||
*******************************************************************************/
|
||||
public SealedExporter(WormAuditQBitConfig config, SealedStore storage)
|
||||
{
|
||||
this.config = config;
|
||||
this.suppliedStorage = storage;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Select the contiguous suffix, never silently skipping an unexported purge.
|
||||
*******************************************************************************/
|
||||
public static Scope scope(AuditStore store) throws QException
|
||||
{
|
||||
QRecord previous = store.last(WormAuditTables.EXPORT);
|
||||
long anchor = store.anchor().getValueLong("anchor_link_id");
|
||||
long after = previous == null ? anchor : previous.getValueLong("to_link_id");
|
||||
if(after < anchor)
|
||||
{
|
||||
throw (new QException("Unexported links were purged"));
|
||||
}
|
||||
QRecord checkpoint = store.last(WormAuditTables.CHECKPOINT);
|
||||
long to = checkpoint == null ? after : Math.max(after, checkpoint.getValueLong("to_link_id"));
|
||||
long count = 0;
|
||||
long cursor = 0;
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> page = checkpoints(store, after + 1, to, cursor);
|
||||
if(page.isEmpty())
|
||||
{
|
||||
break;
|
||||
}
|
||||
count += page.size();
|
||||
cursor = page.getLast().getValueLong("id");
|
||||
}
|
||||
return (new Scope(after + 1, to, count, previous));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Select checkpoint envelopes wholly contained in the export range.
|
||||
*******************************************************************************/
|
||||
private static List<QRecord> checkpoints(AuditStore store, long from, long to, long after) throws QException
|
||||
{
|
||||
return (store.query(WormAuditTables.CHECKPOINT, new QQueryFilter(
|
||||
new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, after),
|
||||
new QFilterCriteria("from_link_id", QCriteriaOperator.GREATER_THAN_OR_EQUALS, from),
|
||||
new QFilterCriteria("to_link_id", QCriteriaOperator.LESS_THAN_OR_EQUALS, to))
|
||||
.withOrderBy(new QFilterOrderBy("id", true)).withLimit(1000)));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Execute with alerts on user and scheduled failures; schedules always beat.
|
||||
*******************************************************************************/
|
||||
public Result run(String trigger, Instant now)
|
||||
{
|
||||
try
|
||||
{
|
||||
config.validate();
|
||||
Result result = lockedRun(now);
|
||||
QAlerts.resolve("wormaudit.export.failed", "export", "Sealed audit export completed");
|
||||
return (result);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
// All exposed failures originate here or use a fixed, value-free code.
|
||||
String reason = e instanceof ExportFailure ? e.getMessage() : "Sealed export failed: storage or database operation failed";
|
||||
QAlerts.raise(new AlertSignal("wormaudit.export.failed").withDedupKey("export").withTitle(reason));
|
||||
return (new Result(null, reason, true));
|
||||
}
|
||||
finally
|
||||
{
|
||||
if("SCHEDULE".equals(trigger))
|
||||
{
|
||||
QAlerts.heartbeat("wormaudit.export.daily");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Serialize with writers in this JVM and PostgreSQL writers on other hosts.
|
||||
*******************************************************************************/
|
||||
private Result lockedRun(Instant now) throws Exception
|
||||
{
|
||||
WormAuditChainer.JVM_LOCK.lock();
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
|
||||
{
|
||||
try
|
||||
{
|
||||
if(transaction instanceof RDBMSTransaction rdbms && "PostgreSQL".equals(rdbms.getConnection().getMetaData().getDatabaseProductName()))
|
||||
{
|
||||
try(Statement statement = rdbms.getConnection().createStatement();
|
||||
ResultSet result = statement.executeQuery("SELECT pg_advisory_xact_lock(" + WormAuditChainer.ADVISORY_LOCK + ")"))
|
||||
{
|
||||
result.next();
|
||||
}
|
||||
}
|
||||
AuditStore store = new AuditStore(transaction);
|
||||
Scope scope = scope(store);
|
||||
Result result;
|
||||
if(scope.count() == 0)
|
||||
{
|
||||
result = new Result(null, scope.review(), false);
|
||||
}
|
||||
else
|
||||
{
|
||||
SealedStore storage = suppliedStorage == null
|
||||
? config.getSealedStoreSupplier() == null ? new ObjectStorageSealedStore(config.getExportBucketAlias(), java.util.UUID.randomUUID().toString())
|
||||
: config.getSealedStoreSupplier().get() : suppliedStorage;
|
||||
result = seal(store, storage, scope, now.truncatedTo(ChronoUnit.MILLIS));
|
||||
}
|
||||
transaction.commit();
|
||||
return (result);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
transaction.rollback();
|
||||
throw (e);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
WormAuditChainer.JVM_LOCK.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build locally before uploading. Partial remote uploads have no database row.
|
||||
*******************************************************************************/
|
||||
private Result seal(AuditStore store, SealedStore storage, Scope scope, Instant now) throws Exception
|
||||
{
|
||||
if(!storage.supportsLock())
|
||||
{
|
||||
throw (new ExportFailure("Export refused: bucket does not support Object Lock"));
|
||||
}
|
||||
Instant until = now.atZone(ZoneOffset.UTC).plusYears(config.getRetentionYears()).toInstant();
|
||||
String prefix = config.getExportPrefix() + "/" + DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(now)
|
||||
+ "/" + scope.from() + "-" + scope.to() + "/";
|
||||
QRecord row;
|
||||
try(ExportFiles files = new ExportFiles())
|
||||
{
|
||||
String head = writeLinks(files, store, scope);
|
||||
long[] checkpointIds = writeCheckpoints(files, store, scope);
|
||||
Map<String, Object> manifest = new LinkedHashMap<>();
|
||||
manifest.put("format", "worm-audit-export/1");
|
||||
manifest.put("fromLinkId", scope.from());
|
||||
manifest.put("toLinkId", scope.to());
|
||||
manifest.put("linkCount", scope.count());
|
||||
manifest.put("checkpointCount", scope.checkpoints());
|
||||
manifest.put("files", Map.of("links.jsonl", ExportFiles.sha256(files.file("links.jsonl")),
|
||||
"checkpoints.jsonl", ExportFiles.sha256(files.file("checkpoints.jsonl"))));
|
||||
manifest.put("headHash", head);
|
||||
String previous = scope.previous() == null ? null : scope.previous().getValueString("manifest_sha256");
|
||||
manifest.put("prevManifestSha256", previous);
|
||||
manifest.put("qbitVersion", WormAuditQBitProducer.VERSION);
|
||||
manifest.put("createdAt", now.toString());
|
||||
Files.write(files.file("manifest.json"), Jcs.bytes(manifest));
|
||||
row = new QRecord().withValue("from_link_id", scope.from()).withValue("to_link_id", scope.to())
|
||||
.withValue("from_checkpoint_id", checkpointIds[0]).withValue("to_checkpoint_id", checkpointIds[1])
|
||||
.withValue("bucket_alias", config.getExportBucketAlias()).withValue("object_prefix", prefix)
|
||||
.withValue("manifest_key", prefix + "manifest.json").withValue("manifest_sha256", ExportFiles.sha256(files.file("manifest.json")))
|
||||
.withValue("prev_manifest_sha256", previous).withValue("object_count", 3L)
|
||||
.withValue("lock_mode", "COMPLIANCE").withValue("retain_until", until).withValue("sealed_at", now);
|
||||
if(!new SealedExportVerifier(config, storage).inspect(files, row, store).result().passed())
|
||||
{
|
||||
throw (new ExportFailure("Export refused: source does not match the signed chain"));
|
||||
}
|
||||
long bytes = 0;
|
||||
Map<String, String> versions = new LinkedHashMap<>();
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String version = storage.put(prefix + name, files.file(name), until);
|
||||
if(!ExportFiles.validVersion(version))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: object " + name + " returned no version id"));
|
||||
}
|
||||
var headObject = storage.head(prefix + name);
|
||||
if(!ExportFiles.locked(headObject, until))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: object " + name + " did not echo COMPLIANCE retention"));
|
||||
}
|
||||
if(!version.equals(headObject.getVersionId()))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: object " + name + " version changed after upload"));
|
||||
}
|
||||
versions.put(name, version);
|
||||
bytes += Files.size(files.file(name));
|
||||
}
|
||||
row.withValue("byte_count", bytes).withValue("object_versions", Jcs.string(versions));
|
||||
}
|
||||
row = store.insert(WormAuditTables.EXPORT, List.of(row)).getFirst();
|
||||
return (new Result(row, WormAuditVerifier.number(scope.count()) + " links sealed in export " + row.getValueLong("id")
|
||||
+ " (retained until " + until + ")", false));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stream canonical lines in pages of 1000 with batched content lookups.
|
||||
*******************************************************************************/
|
||||
private String writeLinks(ExportFiles files, AuditStore store, Scope scope) throws Exception
|
||||
{
|
||||
long cursor = scope.from() - 1;
|
||||
String head = null;
|
||||
try(BufferedWriter writer = Files.newBufferedWriter(files.file("links.jsonl"), StandardCharsets.UTF_8))
|
||||
{
|
||||
while(cursor < scope.to())
|
||||
{
|
||||
List<QRecord> links = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(1000, scope.to() - cursor));
|
||||
if(links.isEmpty())
|
||||
{
|
||||
throw (new ExportFailure("Export refused: chain range is incomplete"));
|
||||
}
|
||||
Map<Long, QRecord> audits = AuditStore.index(store.in("audit", "id", links.stream().map(r -> r.getValueLong("audit_id")).toList()));
|
||||
Map<Long, QRecord> details = ExportFiles.details(store, links);
|
||||
for(QRecord link : links)
|
||||
{
|
||||
if(link.getValueLong("id") != cursor + 1 || link.getValueLong("id") > scope.to())
|
||||
{
|
||||
throw (new ExportFailure("Export refused: chain range is incomplete"));
|
||||
}
|
||||
byte[] content = store.canonical(audits.get(link.getValueLong("audit_id")), AuditStore.detailIds(link), details);
|
||||
if(!Hashes.link(link.getValueString("prev_hash"), content).equals(link.getValueString("link_hash")))
|
||||
{
|
||||
throw (new ExportFailure("Export refused: content differs at link " + link.getValueLong("id")));
|
||||
}
|
||||
writer.write(Jcs.string(Map.of("link", ExportFiles.link(link), "content", ExportFiles.parse(content))));
|
||||
writer.write('\n');
|
||||
head = link.getValueString("link_hash");
|
||||
cursor = link.getValueLong("id");
|
||||
}
|
||||
}
|
||||
}
|
||||
return (head);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stream complete signed checkpoint envelopes in keyset pages.
|
||||
*******************************************************************************/
|
||||
private long[] writeCheckpoints(ExportFiles files, AuditStore store, Scope scope) throws Exception
|
||||
{
|
||||
long first = 0;
|
||||
long cursor = 0;
|
||||
try(BufferedWriter writer = Files.newBufferedWriter(files.file("checkpoints.jsonl"), StandardCharsets.UTF_8))
|
||||
{
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> page = checkpoints(store, scope.from(), scope.to(), cursor);
|
||||
if(page.isEmpty())
|
||||
{
|
||||
break;
|
||||
}
|
||||
for(QRecord row : page)
|
||||
{
|
||||
first = first == 0 ? row.getValueLong("id") : first;
|
||||
cursor = row.getValueLong("id");
|
||||
writer.write(Jcs.string(ExportFiles.checkpoint(row)));
|
||||
writer.write('\n');
|
||||
}
|
||||
}
|
||||
}
|
||||
return (new long[] { first, cursor });
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Only fixed messages constructed here may reach an alert or Result screen.
|
||||
*******************************************************************************/
|
||||
private static class ExportFailure extends IOException
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Carry a value-free exporter reason without a provider exception cause.
|
||||
*******************************************************************************/
|
||||
ExportFailure(String reason)
|
||||
{
|
||||
super(reason);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.export;
|
||||
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
|
||||
import com.kingsrook.qbits.objectstorage.api.Page;
|
||||
import com.kingsrook.qbits.objectstorage.api.StoredObject;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Minimal storage boundary. The application never deletes sealed objects.
|
||||
*******************************************************************************/
|
||||
public interface SealedStore
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Only positively confirmed Object Lock support is acceptable.
|
||||
*******************************************************************************/
|
||||
boolean supportsLock();
|
||||
|
||||
/*******************************************************************************
|
||||
** Preserve unknown provider capabilities for read-only setup diagnostics.
|
||||
*******************************************************************************/
|
||||
default CapabilityStatus lockCapability()
|
||||
{
|
||||
return (supportsLock() ? CapabilityStatus.SUPPORTED : CapabilityStatus.UNSUPPORTED);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Upload a file with COMPLIANCE retention.
|
||||
*******************************************************************************/
|
||||
String put(String key, Path file, Instant retainUntil) throws Exception;
|
||||
|
||||
/*******************************************************************************
|
||||
** Read current object metadata, or null for an absent object.
|
||||
*******************************************************************************/
|
||||
StoredObject head(String key) throws Exception;
|
||||
|
||||
/*******************************************************************************
|
||||
** Open the exact sealed version; the caller owns and closes the stream.
|
||||
*******************************************************************************/
|
||||
InputStream getVersion(String key, String versionId) throws Exception;
|
||||
|
||||
/*******************************************************************************
|
||||
** List one bounded page.
|
||||
*******************************************************************************/
|
||||
Page<ObjectSummary> list(ListRequest request) throws Exception;
|
||||
}
|
||||
|
||||
@@ -81,6 +81,7 @@ public class LinkRangeWidget extends AbstractWidgetRenderer
|
||||
String path = QContext.getQInstance().getTablePath(WormAuditTables.CHAIN);
|
||||
String viewAll = path + "?filter=" + URLEncoder.encode(JsonUtils.toJson(filter), StandardCharsets.UTF_8);
|
||||
QueryInput query = new QueryInput(WormAuditTables.CHAIN).withFilter(filter.clone().withLimit(100)).withInputSource(QInputSource.USER);
|
||||
query.withShouldTranslatePossibleValues(true);
|
||||
query.withPermissionEnforcement(PermissionEnforcement.ENFORCE);
|
||||
return (new RenderWidgetOutput(new ChildRecordListData("Chain links", new QueryAction().execute(query),
|
||||
QContext.getQInstance().getTable(WormAuditTables.CHAIN), path, viewAll, total)));
|
||||
|
||||
@@ -31,6 +31,9 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
|
||||
@@ -39,7 +42,7 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDe
|
||||
|
||||
/*******************************************************************************
|
||||
** Append-only sidecar tables. Snake-case fields map directly to DDL columns.
|
||||
** Export metadata is the integration hook for phase (b); no export runs here.
|
||||
** Labels also supply the standard QQQ app navigation wording.
|
||||
*******************************************************************************/
|
||||
public final class WormAuditTables
|
||||
{
|
||||
@@ -67,7 +70,7 @@ public final class WormAuditTables
|
||||
List<QTableMetaData> tables = new ArrayList<>();
|
||||
tables.add(table(config, CHAIN, "Audit Chain Link", "id:L kind:S audit_id:L detail_ids:T prev_hash:S link_hash:S chained_at:D"));
|
||||
tables.add(table(config, CHECKPOINT, "Checkpoint", "id:L from_link_id:L to_link_id:L link_count:L head_hash:S merkle_root:S algorithm:S key_id:S public_key:T signature:T created_at:D"));
|
||||
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
|
||||
tables.add(table(config, EXPORT, "Sealed Export", "id:L from_link_id:L to_link_id:L from_checkpoint_id:L to_checkpoint_id:L bucket_alias:S object_prefix:T object_versions:T manifest_key:T manifest_sha256:S prev_manifest_sha256:S object_count:L byte_count:L lock_mode:S retain_until:D sealed_at:D"));
|
||||
tables.add(table(config, VERIFICATION, "Verification", "id:L kind:S trigger:S started_at:D finished_at:D from_link_id:L to_link_id:L links_checked:L outcome:S break_count:L first_break_link_id:L first_break_audit_id:L first_break_reason:T export_id:L run_by:S"));
|
||||
tables.add(table(config, PURGE, "Purge", "id:L requested_at:D requested_by:S cutoff:D retention_interval:S outcome:S reason:T audits_deleted:L details_deleted:L links_deleted:L anchor_link_id:L anchor_hash:S"));
|
||||
return (tables);
|
||||
@@ -79,9 +82,9 @@ public final class WormAuditTables
|
||||
*******************************************************************************/
|
||||
private static QTableMetaData table(WormAuditQBitConfig config, String name, String label, String fields)
|
||||
{
|
||||
QTableMetaData table = new QTableMetaData().withName(name).withLabel(label)
|
||||
QTableMetaData table = new QTableMetaData().withName(name).withLabel(navigationLabel(name))
|
||||
.withBackendName(config.getBackendName()).withBackendDetails(new RDBMSTableBackendDetails().withTableName(name))
|
||||
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("%s")
|
||||
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat(label + " #%s")
|
||||
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.NONE))
|
||||
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS).withPermissionBaseName(name));
|
||||
for(String spec : fields.split(" "))
|
||||
@@ -94,7 +97,7 @@ public final class WormAuditTables
|
||||
case "T" -> QFieldType.TEXT;
|
||||
default -> QFieldType.STRING;
|
||||
};
|
||||
table.withField(new QFieldMetaData(parts[0], type).withBackendName(parts[0]));
|
||||
table.withField(new QFieldMetaData(parts[0], type).withBackendName(parts[0]).withLabel(fieldLabel(parts[0])));
|
||||
}
|
||||
for(String reference : List.of("from_link_id", "to_link_id", "first_break_link_id", "anchor_link_id"))
|
||||
{
|
||||
@@ -114,11 +117,126 @@ public final class WormAuditTables
|
||||
{
|
||||
table.getField("export_id").withPossibleValueSourceName(EXPORT);
|
||||
}
|
||||
table.withSection(new QFieldSection("record", label, new QIcon("fact_check"), Tier.T1, new ArrayList<>(table.getFields().keySet())));
|
||||
switch(name)
|
||||
{
|
||||
case CHAIN -> table.withRecordLabelFormat("Link #%s");
|
||||
case EXPORT -> table.withRecordLabelFormatAndFields("Export #%s (links %s–%s)", "id", "from_link_id", "to_link_id");
|
||||
case VERIFICATION, PURGE -> table.withRecordLabelFormatAndFields(label + " #%s (%s)", "id", "outcome");
|
||||
default ->
|
||||
{
|
||||
}
|
||||
}
|
||||
if(table.getFields().containsKey("kind"))
|
||||
{
|
||||
table.getField("kind").withInlinePossibleValueSource(new QPossibleValueSource().withName(name + "_kind")
|
||||
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(new QPossibleValue<>("AUDIT", "Audit"),
|
||||
new QPossibleValue<>("LATE_DETAIL", "Late detail"), new QPossibleValue<>("CHAIN", "Chain"),
|
||||
new QPossibleValue<>("SEALED_EXPORT", "Sealed export"))));
|
||||
}
|
||||
if(table.getFields().containsKey("outcome"))
|
||||
{
|
||||
table.getField("outcome").withInlinePossibleValueSource(new QPossibleValueSource().withName(name + "_outcome")
|
||||
.withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING).withEnumValues(List.of(new QPossibleValue<>("PASS", "Pass"),
|
||||
new QPossibleValue<>("FAIL", "Fail"), new QPossibleValue<>("PURGED", "Purged"), new QPossibleValue<>("REFUSED", "Refused"))));
|
||||
}
|
||||
if(VERIFICATION.equals(name))
|
||||
{
|
||||
table.withSection(new QFieldSection("result", "Result", new QIcon("fact_check"), Tier.T2,
|
||||
List.of("id", "outcome", "break_count", "first_break_link_id", "first_break_audit_id", "first_break_reason")).withGridColumns(12));
|
||||
table.withSection(new QFieldSection("scope", "Scope", new QIcon("link"), Tier.T2,
|
||||
List.of("kind", "trigger", "from_link_id", "to_link_id", "links_checked", "export_id")).withGridColumns(12));
|
||||
table.withSection(new QFieldSection("run", "Run", new QIcon("schedule"), Tier.T2,
|
||||
List.of("started_at", "finished_at", "run_by")).withGridColumns(12));
|
||||
}
|
||||
else
|
||||
{
|
||||
table.withSection(new QFieldSection("record", label, new QIcon("fact_check"), Tier.T1,
|
||||
new ArrayList<>(table.getFields().keySet())).withGridColumns(12));
|
||||
}
|
||||
if(List.of(CHECKPOINT, EXPORT, VERIFICATION).contains(name))
|
||||
{
|
||||
table.withSection(new QFieldSection("links", new QIcon("link"), Tier.T2).withLabel("Chain links").withWidgetName(name + "_links"));
|
||||
table.withSection(new QFieldSection("links", new QIcon("link"), Tier.T2).withLabel("Chain links")
|
||||
.withWidgetName(name + "_links").withGridColumns(12));
|
||||
}
|
||||
return (table);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Core RC.5 shares the table label with its navigation tree node.
|
||||
*******************************************************************************/
|
||||
public static String navigationLabel(String name)
|
||||
{
|
||||
return (switch(name)
|
||||
{
|
||||
case CHAIN -> "Audit Chain";
|
||||
case CHECKPOINT -> "Checkpoints";
|
||||
case EXPORT -> "Sealed Exports";
|
||||
case VERIFICATION -> "Verifications";
|
||||
default -> "Purges";
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Plain record labels independent of physical column names.
|
||||
*******************************************************************************/
|
||||
private static String fieldLabel(String name)
|
||||
{
|
||||
return (switch(name)
|
||||
{
|
||||
case "id" -> "ID";
|
||||
case "kind" -> "Kind";
|
||||
case "audit_id" -> "Audit";
|
||||
case "detail_ids" -> "Detail ids";
|
||||
case "prev_hash" -> "Previous hash";
|
||||
case "link_hash" -> "Link hash";
|
||||
case "chained_at" -> "Chained";
|
||||
case "from_link_id" -> "From link";
|
||||
case "to_link_id" -> "To link";
|
||||
case "link_count" -> "Links";
|
||||
case "head_hash" -> "Head hash";
|
||||
case "merkle_root" -> "Merkle root";
|
||||
case "algorithm" -> "Algorithm";
|
||||
case "key_id" -> "Key ID";
|
||||
case "public_key" -> "Public key";
|
||||
case "signature" -> "Signature";
|
||||
case "created_at" -> "Created";
|
||||
case "from_checkpoint_id" -> "From checkpoint";
|
||||
case "to_checkpoint_id" -> "To checkpoint";
|
||||
case "bucket_alias" -> "Bucket alias";
|
||||
case "object_prefix" -> "Object prefix";
|
||||
case "object_versions" -> "Object versions";
|
||||
case "manifest_key" -> "Manifest key";
|
||||
case "manifest_sha256" -> "Manifest SHA-256";
|
||||
case "prev_manifest_sha256" -> "Previous manifest SHA-256";
|
||||
case "object_count" -> "Objects";
|
||||
case "byte_count" -> "Bytes";
|
||||
case "lock_mode" -> "Lock mode";
|
||||
case "retain_until" -> "Retain until";
|
||||
case "sealed_at" -> "Sealed";
|
||||
case "trigger" -> "Trigger";
|
||||
case "started_at" -> "Started";
|
||||
case "finished_at" -> "Finished";
|
||||
case "links_checked" -> "Links checked";
|
||||
case "outcome" -> "Outcome";
|
||||
case "break_count" -> "Breaks";
|
||||
case "first_break_link_id" -> "First break link";
|
||||
case "first_break_audit_id" -> "First break audit";
|
||||
case "first_break_reason" -> "First break";
|
||||
case "export_id" -> "Sealed export";
|
||||
case "run_by" -> "Run by";
|
||||
case "requested_at" -> "Requested";
|
||||
case "requested_by" -> "Requested by";
|
||||
case "cutoff" -> "Cutoff";
|
||||
case "retention_interval" -> "Retention interval";
|
||||
case "reason" -> "Reason";
|
||||
case "audits_deleted" -> "Audits deleted";
|
||||
case "details_deleted" -> "Details deleted";
|
||||
case "links_deleted" -> "Links deleted";
|
||||
case "anchor_link_id" -> "Anchor link";
|
||||
case "anchor_hash" -> "Anchor hash";
|
||||
default -> throw new IllegalArgumentException("Missing field label: " + name);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,376 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.processes;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.actions.permissions.UseOtherPermissionNameCustomPermissionChecker;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunInput;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunOutput;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Standard Review/Result actions for sealing and verifying exports.
|
||||
*******************************************************************************/
|
||||
public final class ExportProcesses
|
||||
{
|
||||
public static final String EXPORT = "worm_audit_export_now";
|
||||
public static final String VERIFY = "worm_audit_verify_export";
|
||||
public static final String DAILY = "worm_audit_export_daily";
|
||||
public static final String PERMISSION = "worm_audit.export";
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private ExportProcesses()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Export is a table action; verification requires one or more selected exports.
|
||||
*******************************************************************************/
|
||||
public static List<QProcessMetaData> processes(WormAuditQBitConfig config)
|
||||
{
|
||||
Map<String, Serializable> values = new HashMap<>();
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, WormAuditTables.EXPORT);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
|
||||
QProcessMetaData export = StreamedETLWithFrontendProcess.defineProcessMetaData(ExportExtract.class, ExportReview.class, ExportLoad.class, new HashMap<>(values))
|
||||
.withName(EXPORT).withLabel("Export now").withTableName(WormAuditTables.EXPORT).withPermissionRules(permission(PERMISSION));
|
||||
QProcessMetaData verify = StreamedETLWithFrontendProcess.defineProcessMetaData(VerifyExtract.class, VerifyReview.class, VerifyLoad.class, new HashMap<>(values))
|
||||
.withName(VERIFY).withLabel("Verify sealed export").withTableName(WormAuditTables.EXPORT).withMinInputRecords(1)
|
||||
.withPermissionRules(permission(WormAuditProcesses.PERMISSION));
|
||||
QProcessMetaData daily = new QProcessMetaData().withName(DAILY).withLabel("Daily sealed audit export")
|
||||
.withStep(new QBackendStepMetaData().withName("export").withCode(new QCodeReference(Daily.class)))
|
||||
.withPermissionRules(permission(PERMISSION)).withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName())
|
||||
.withCronExpression(config.getExportCron()).withCronTimeZoneId("UTC"));
|
||||
return (List.of(export, verify, daily));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reuse the phase (a) explicit permission-name checker.
|
||||
*******************************************************************************/
|
||||
private static QPermissionRules permission(String name)
|
||||
{
|
||||
return (new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
|
||||
.withCustomPermissionChecker(UseOtherPermissionNameCustomPermissionChecker.build(name)));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Plain text summaries must not acquire the framework's default zero count.
|
||||
*******************************************************************************/
|
||||
private static ProcessSummaryLine text(Status status, String message)
|
||||
{
|
||||
ProcessSummaryLine line = new ProcessSummaryLine(status, message);
|
||||
line.setCount(null);
|
||||
return (line);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One synthetic work item allows the first export on an empty export table.
|
||||
*******************************************************************************/
|
||||
public static class ExportExtract extends AbstractExtractStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Count the single export operation, not previous export rows.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public Integer doCount(RunBackendStepInput input)
|
||||
{
|
||||
return (1);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Feed one work item through the standard stream.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
getRecordPipe().addRecord(new QRecord().withValue("id", 0L));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Review is read-only; the writer reselects the suffix after confirmation.
|
||||
*******************************************************************************/
|
||||
public static class ExportReview extends AbstractTransformStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private String message;
|
||||
private boolean failed;
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the planned range once without uploading or inserting.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
try
|
||||
{
|
||||
message = SealedExporter.scope(new AuditStore(null)).review();
|
||||
failed = false;
|
||||
}
|
||||
catch(QException e)
|
||||
{
|
||||
message = "Cannot select a contiguous export range";
|
||||
failed = true;
|
||||
}
|
||||
output.getRecords().addAll(input.getRecords());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Show the range, including the explicit no-checkpoint case.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
return (new ArrayList<>(List.of(text(failed ? Status.ERROR : Status.INFO, message == null ? "Nothing to seal: no new checkpoint" : message))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Seal once after the stream is complete.
|
||||
*******************************************************************************/
|
||||
public static class ExportLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private final SealedStore storage;
|
||||
private SealedExporter.Result result;
|
||||
|
||||
/*******************************************************************************
|
||||
** Framework constructor.
|
||||
*******************************************************************************/
|
||||
public ExportLoad()
|
||||
{
|
||||
this(null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Storage-injected action for unit tests.
|
||||
*******************************************************************************/
|
||||
public ExportLoad(SealedStore storage)
|
||||
{
|
||||
this.storage = storage;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Forward the work item without retaining content.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output)
|
||||
{
|
||||
output.getRecords().addAll(input.getRecords());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The same execution path as the daily schedule.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void postRun(BackendStepPostRunInput input, BackendStepPostRunOutput output)
|
||||
{
|
||||
result = new SealedExporter(VerificationRuns.config(), storage).run("USER", Instant.now());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Result text is value-free even when the provider fails.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
return (result == null ? new ArrayList<>() : new ArrayList<>(List.of(text(result.failed() ? Status.ERROR : Status.OK, result.message()))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Respect core's selected-record/filter interpretation; no implicit all-export run.
|
||||
*******************************************************************************/
|
||||
public static class VerifyExtract extends ExtractViaQueryStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Deterministic selection with full text fields from the export table.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public QQueryFilter getQueryFilter(RunBackendStepInput input) throws QException
|
||||
{
|
||||
input.addValue(FIELD_SOURCE_TABLE, WormAuditTables.EXPORT);
|
||||
QQueryFilter filter = super.getQueryFilter(input).clone();
|
||||
filter.setOrderBys(List.of(new QFilterOrderBy("id", true)));
|
||||
return (filter);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Review selected exports without reading their content.
|
||||
*******************************************************************************/
|
||||
public static class VerifyReview extends AbstractTransformStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private long count;
|
||||
|
||||
/*******************************************************************************
|
||||
** Count and forward the selected export records.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output)
|
||||
{
|
||||
count += input.getRecords().size();
|
||||
output.getRecords().addAll(input.getRecords());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One Review summary.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
return (new ArrayList<>(List.of(text(Status.INFO, WormAuditVerifier.number(count) + " sealed exports will be verified"))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify each selected export and persist its independent result.
|
||||
*******************************************************************************/
|
||||
public static class VerifyLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private final SealedStore storage;
|
||||
private final ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
|
||||
|
||||
/*******************************************************************************
|
||||
** Framework constructor.
|
||||
*******************************************************************************/
|
||||
public VerifyLoad()
|
||||
{
|
||||
this(null);
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Storage-injected action for unit tests.
|
||||
*******************************************************************************/
|
||||
public VerifyLoad(SealedStore storage)
|
||||
{
|
||||
this.storage = storage;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Process the current page without retaining exported content.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
for(QRecord selected : input.getRecords())
|
||||
{
|
||||
QRecord row = new AuditStore(null).get(WormAuditTables.EXPORT, selected.getValueLong("id"));
|
||||
if(row == null)
|
||||
{
|
||||
lines.add(text(Status.ERROR, "Selected export " + selected.getValueLong("id") + " is missing"));
|
||||
continue;
|
||||
}
|
||||
var report = new SealedExportVerifier(VerificationRuns.config(), storage).verify(row, "USER");
|
||||
if(report.result().passed())
|
||||
{
|
||||
lines.add(text(Status.OK, WormAuditVerifier.number(report.result().getLinksChecked()) + " links verified in export " + row.getValueLong("id")));
|
||||
}
|
||||
report.result().getBreaks().forEach(b -> lines.add(text(Status.ERROR, b.reason())));
|
||||
if(report.purged() > 0 && report.result().passed())
|
||||
{
|
||||
lines.add(text(Status.INFO, WormAuditVerifier.number(report.purged()) + " links verified from the sealed copy only (purged from the live database)"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exact per-link differences and purged-copy information.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
return (lines);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Daily export sends heartbeat even when sealing fails.
|
||||
*******************************************************************************/
|
||||
public static class Daily implements BackendStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Run the exporter with the installed host configuration.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output)
|
||||
{
|
||||
var result = new SealedExporter(VerificationRuns.config()).run("SCHEDULE", Instant.now());
|
||||
output.addValue("outcome", result.failed() ? "FAIL" : "PASS");
|
||||
output.addValue("result", result.message());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,11 +75,11 @@ public final class VerificationRuns
|
||||
{
|
||||
if(result.passed())
|
||||
{
|
||||
QAlerts.resolve("worm_audit.chain.break", "chain", "Audit chain verification passed");
|
||||
QAlerts.resolve("wormaudit.chain.break", "chain", "Audit chain verification passed");
|
||||
}
|
||||
else
|
||||
{
|
||||
QAlerts.raise(new AlertSignal("worm_audit.chain.break").withDedupKey("chain")
|
||||
QAlerts.raise(new AlertSignal("wormaudit.chain.break").withDedupKey("chain")
|
||||
.withTitle(result.getBreaks().getFirst().reason()).withRecordRef(new AlertRecordRef(WormAuditTables.VERIFICATION, row.getValue("id"))));
|
||||
}
|
||||
}
|
||||
@@ -89,7 +89,7 @@ public final class VerificationRuns
|
||||
{
|
||||
if(scheduled)
|
||||
{
|
||||
QAlerts.heartbeat("worm_audit.verify.daily");
|
||||
QAlerts.heartbeat("wormaudit.verify.daily");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -99,10 +99,28 @@ public final class VerificationRuns
|
||||
** Persist identifiers and a value-free first reason, never audit content.
|
||||
*******************************************************************************/
|
||||
public static QRecord save(VerificationResult result, String trigger) throws QException
|
||||
{
|
||||
return (save(result, trigger, null));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Append a chain or sealed-export result using the same immutable schema.
|
||||
*******************************************************************************/
|
||||
public static QRecord save(VerificationResult result, String trigger, Long exportId) throws QException
|
||||
{
|
||||
return (save(result, trigger, exportId, Instant.now()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Clock-injected persistence for genuine historical verification runs.
|
||||
*******************************************************************************/
|
||||
public static QRecord save(VerificationResult result, String trigger, Long exportId, Instant finishedAt) throws QException
|
||||
{
|
||||
String user = QContext.getQSession() == null || QContext.getQSession().getUser() == null ? "system" : QContext.getQSession().getUser().getIdReference();
|
||||
QRecord row = new QRecord().withValue("kind", "CHAIN").withValue("trigger", trigger)
|
||||
.withValue("started_at", result.getStartedAt()).withValue("finished_at", Instant.now())
|
||||
QRecord row = new QRecord().withValue("kind", exportId == null ? "CHAIN" : "SEALED_EXPORT").withValue("export_id", exportId).withValue("trigger", trigger)
|
||||
.withValue("started_at", result.getStartedAt()).withValue("finished_at", finishedAt)
|
||||
.withValue("from_link_id", result.getFromLinkId()).withValue("to_link_id", result.getToLinkId())
|
||||
.withValue("links_checked", result.getLinksChecked()).withValue("outcome", result.passed() ? "PASS" : "FAIL")
|
||||
.withValue("break_count", result.getBreakCount()).withValue("run_by", user);
|
||||
|
||||
@@ -41,12 +41,30 @@ public class VerificationResult
|
||||
|
||||
private final List<Break> breaks = new ArrayList<>();
|
||||
private final List<String> warnings = new ArrayList<>();
|
||||
private final Instant startedAt = Instant.now();
|
||||
private final Instant startedAt;
|
||||
private long breakCount;
|
||||
private long linksChecked;
|
||||
private Long fromLinkId;
|
||||
private Long toLinkId;
|
||||
|
||||
/*******************************************************************************
|
||||
** Use the production clock by default.
|
||||
*******************************************************************************/
|
||||
public VerificationResult()
|
||||
{
|
||||
this(Instant.now());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind the verification start to the caller's clock.
|
||||
*******************************************************************************/
|
||||
public VerificationResult(Instant startedAt)
|
||||
{
|
||||
this.startedAt = startedAt;
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Add a finding in deterministic verification order.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -54,13 +54,28 @@ public class WormAuditVerifier
|
||||
{
|
||||
public static final int PAGE_SIZE = 1000;
|
||||
private final WormAuditQBitConfig config;
|
||||
private final int pageSize;
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind trust pins and settle horizon.
|
||||
*******************************************************************************/
|
||||
public WormAuditVerifier(WormAuditQBitConfig config)
|
||||
{
|
||||
this(config, PAGE_SIZE);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Smaller pages support constrained hosts and boundary regression tests.
|
||||
*******************************************************************************/
|
||||
public WormAuditVerifier(WormAuditQBitConfig config, int pageSize)
|
||||
{
|
||||
if(pageSize < 1 || pageSize > PAGE_SIZE)
|
||||
{
|
||||
throw (new IllegalArgumentException("Verification page size must be between 1 and 1000"));
|
||||
}
|
||||
this.config = config;
|
||||
this.pageSize = pageSize;
|
||||
}
|
||||
|
||||
|
||||
@@ -77,6 +92,15 @@ public class WormAuditVerifier
|
||||
** Null means full verification; an empty set is an empty selection.
|
||||
*******************************************************************************/
|
||||
public VerificationResult verify(Set<Long> selection) throws QException
|
||||
{
|
||||
return (verify(selection, Instant.now()));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Clock-injected verification uses the same horizon and snapshot as production.
|
||||
*******************************************************************************/
|
||||
public VerificationResult verify(Set<Long> selection, Instant now) throws QException
|
||||
{
|
||||
WormAuditChainer.JVM_LOCK.lock();
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(WormAuditTables.CHAIN)))
|
||||
@@ -87,7 +111,7 @@ public class WormAuditVerifier
|
||||
{
|
||||
rdbms.getConnection().setTransactionIsolation(Connection.TRANSACTION_REPEATABLE_READ);
|
||||
}
|
||||
VerificationResult result = verify(new AuditStore(transaction), selection, Instant.now());
|
||||
VerificationResult result = verify(new AuditStore(transaction), selection, now);
|
||||
transaction.commit();
|
||||
return (result);
|
||||
}
|
||||
@@ -107,9 +131,9 @@ public class WormAuditVerifier
|
||||
/*******************************************************************************
|
||||
** Verify one consistent storage snapshot.
|
||||
*******************************************************************************/
|
||||
private VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
|
||||
VerificationResult verify(AuditStore store, Set<Long> selection, Instant now) throws QException
|
||||
{
|
||||
VerificationResult result = new VerificationResult();
|
||||
VerificationResult result = new VerificationResult(now);
|
||||
if(config.getSigningKey() == null)
|
||||
{
|
||||
result.warn("no signing key configured");
|
||||
@@ -125,7 +149,7 @@ public class WormAuditVerifier
|
||||
CheckpointCursor checkpoints = new CheckpointCursor(store, selection, anchor, result);
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> page = store.page(WormAuditTables.CHAIN, lastId, PAGE_SIZE);
|
||||
List<QRecord> page = store.page(WormAuditTables.CHAIN, lastId, pageSize);
|
||||
if(page.isEmpty())
|
||||
{
|
||||
break;
|
||||
@@ -150,9 +174,9 @@ public class WormAuditVerifier
|
||||
}
|
||||
Map<Long, QRecord> details = new HashMap<>();
|
||||
List<Long> ids = new ArrayList<>(detailIds);
|
||||
for(int start = 0; start < ids.size(); start += PAGE_SIZE)
|
||||
for(int start = 0; start < ids.size(); start += pageSize)
|
||||
{
|
||||
details.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + PAGE_SIZE, ids.size())))));
|
||||
details.putAll(AuditStore.index(store.in("auditDetail", "id", ids.subList(start, Math.min(start + pageSize, ids.size())))));
|
||||
}
|
||||
for(QRecord link : page)
|
||||
{
|
||||
@@ -180,8 +204,8 @@ public class WormAuditVerifier
|
||||
previous = link.getValueString("link_hash");
|
||||
lastId = id;
|
||||
}
|
||||
checkpoints.through(lastId);
|
||||
}
|
||||
// The main scan's page is out of scope before checkpoint pages are read.
|
||||
checkpoints.through(Long.MAX_VALUE);
|
||||
result.scope(checked, first, last);
|
||||
if(missing != null)
|
||||
@@ -195,10 +219,14 @@ public class WormAuditVerifier
|
||||
{
|
||||
store.completeness(cutoff, result);
|
||||
}
|
||||
else
|
||||
else if(store.isMemory())
|
||||
{
|
||||
completeness(store, store.all(WormAuditTables.CHAIN), cutoff, result);
|
||||
}
|
||||
else
|
||||
{
|
||||
pagedCompleteness(store, cutoff, result);
|
||||
}
|
||||
}
|
||||
return (result);
|
||||
}
|
||||
@@ -263,6 +291,73 @@ public class WormAuditVerifier
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Portable completeness for non-SQL backends. Trade repeated scans for bounded
|
||||
** memory; only the memory backend uses the whole-chain coverage index.
|
||||
*******************************************************************************/
|
||||
private void pagedCompleteness(AuditStore store, Instant cutoff, VerificationResult result) throws QException
|
||||
{
|
||||
for(String table : List.of("audit", "auditDetail"))
|
||||
{
|
||||
long after = 0;
|
||||
while(true)
|
||||
{
|
||||
List<QRecord> rows = store.page(table, after, pageSize);
|
||||
if(rows.isEmpty())
|
||||
{
|
||||
break;
|
||||
}
|
||||
Map<Long, Integer> counts = new HashMap<>();
|
||||
for(QRecord row : rows)
|
||||
{
|
||||
QRecord audit = table.equals("audit") ? row : store.get("audit", row.getValueLong("auditId"));
|
||||
if(audit == null || audit.getValueInstant("timestamp") == null || audit.getValueInstant("timestamp").isBefore(cutoff))
|
||||
{
|
||||
counts.put(row.getValueLong("id"), 0);
|
||||
if(table.equals("audit") && audit.getValueInstant("timestamp") == null)
|
||||
{
|
||||
result.add(null, row.getValueLong("id"), "audit " + number(row.getValueLong("id")) + " has no timestamp");
|
||||
}
|
||||
}
|
||||
}
|
||||
long linkAfter = 0;
|
||||
while(!counts.isEmpty())
|
||||
{
|
||||
List<QRecord> links = store.page(WormAuditTables.CHAIN, linkAfter, pageSize);
|
||||
if(links.isEmpty())
|
||||
{
|
||||
break;
|
||||
}
|
||||
for(QRecord link : links)
|
||||
{
|
||||
if(table.equals("audit"))
|
||||
{
|
||||
if("AUDIT".equals(link.getValueString("kind")))
|
||||
{
|
||||
counts.computeIfPresent(link.getValueLong("audit_id"), (id, count) -> count + 1);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
try
|
||||
{
|
||||
AuditStore.detailIds(link).forEach(id -> counts.computeIfPresent(id, (key, count) -> count + 1));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
// Malformed coverage was reported by content verification.
|
||||
}
|
||||
}
|
||||
}
|
||||
linkAfter = links.getLast().getValueLong("id");
|
||||
}
|
||||
counts.forEach((id, count) -> coverage(table.equals("audit") ? "audit" : "audit_detail", id, count, result));
|
||||
after = rows.getLast().getValueLong("id");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every settled header and detail must occur once; orphan details also fail.
|
||||
*******************************************************************************/
|
||||
@@ -429,7 +524,7 @@ public class WormAuditVerifier
|
||||
String head = null;
|
||||
while(cursor < to)
|
||||
{
|
||||
List<QRecord> page = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(PAGE_SIZE, to - cursor));
|
||||
List<QRecord> page = store.page(WormAuditTables.CHAIN, cursor, (int) Math.min(pageSize, to - cursor));
|
||||
if(page.isEmpty())
|
||||
{
|
||||
break;
|
||||
|
||||
@@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS public.worm_audit_export (
|
||||
to_checkpoint_id bigint,
|
||||
bucket_alias text,
|
||||
object_prefix text,
|
||||
object_versions text,
|
||||
manifest_key text,
|
||||
manifest_sha256 text,
|
||||
prev_manifest_sha256 text,
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.net.URI;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
|
||||
import com.kingsrook.qbits.objectstorage.api.Capability;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
|
||||
import com.kingsrook.qbits.objectstorage.config.AddressingStyle;
|
||||
import com.kingsrook.qbits.objectstorage.config.BucketRole;
|
||||
import com.kingsrook.qbits.objectstorage.config.ChecksumMode;
|
||||
import com.kingsrook.qbits.objectstorage.config.ProviderType;
|
||||
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
|
||||
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
|
||||
import com.kingsrook.qbits.objectstorage.internal.CapabilityProbe;
|
||||
import com.kingsrook.qbits.objectstorage.internal.CapabilityStore;
|
||||
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.export.ExportFiles;
|
||||
import com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.testcontainers.containers.GenericContainer;
|
||||
import org.testcontainers.containers.wait.strategy.Wait;
|
||||
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
|
||||
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
|
||||
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
|
||||
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
|
||||
import software.amazon.awssdk.core.sync.RequestBody;
|
||||
import software.amazon.awssdk.regions.Region;
|
||||
import software.amazon.awssdk.services.s3.S3Client;
|
||||
import software.amazon.awssdk.services.s3.model.S3Exception;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Real Object Lock acceptance. Missing Docker skips locally and fails in CI.
|
||||
*******************************************************************************/
|
||||
@ExtendWith(RequiresDockerCondition.class)
|
||||
class MinioExportAcceptanceTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Production adapter, COMPLIANCE retention, version preservation and UI findings.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void lockedRoundTripVersionRetentionLiveTamperAndUnsupportedBucket() throws Exception
|
||||
{
|
||||
String image = System.getProperty("qqq.test.minioImage",
|
||||
"cgr.dev/chainguard/minio@sha256:dc455cdd4db6e390b1b26514f15ff37681a777bf8dab41aac445a0259f751e56");
|
||||
try(GenericContainer<?> minio = new GenericContainer<>(image)
|
||||
.withEnv("MINIO_ROOT_USER", "worm-test-access").withEnv("MINIO_ROOT_PASSWORD", "worm-test-secret")
|
||||
.withCommand("server", "/tmp/minio-data").withExposedPorts(9000)
|
||||
.waitingFor(Wait.forHttp("/minio/health/live").forPort(9000).withStartupTimeout(Duration.ofMinutes(2))))
|
||||
{
|
||||
minio.start();
|
||||
String endpoint = "http://" + minio.getHost() + ":" + minio.getMappedPort(9000);
|
||||
try(S3Client s3 = S3Client.builder().endpointOverride(URI.create(endpoint)).region(Region.US_EAST_1).forcePathStyle(true)
|
||||
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("worm-test-access", "worm-test-secret")))
|
||||
.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
|
||||
.responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED).build())
|
||||
{
|
||||
s3.createBucket(b -> b.bucket("worm-locked").objectLockEnabledForBucket(true));
|
||||
s3.createBucket(b -> b.bucket("worm-plain"));
|
||||
var config = ChainVerificationTest.signed();
|
||||
var instance = init(config);
|
||||
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(ProcessesAndAlertsTest.Sink.class));
|
||||
ProcessesAndAlertsTest.Sink.raised.clear();
|
||||
System.setProperty("worm.test.access", "worm-test-access");
|
||||
System.setProperty("worm.test.secret", "worm-test-secret");
|
||||
ObjectStorage.configureStandalone(new ObjectStorageQBitConfig().withBackendName("memory")
|
||||
.withAllowedSecretRefPatterns(List.of("prop.worm.test.*"))
|
||||
.withConnection(new StorageConnectionConfig("minio").withProviderType(ProviderType.MINIO).withEndpoint(endpoint)
|
||||
.withRegion("us-east-1").withAddressingStyle(AddressingStyle.PATH).withChecksumMode(ChecksumMode.WHEN_REQUIRED)
|
||||
.withAccessKeyRef("${prop.worm.test.access}").withSecretKeyRef("${prop.worm.test.secret}"))
|
||||
.withBucket(new StorageBucketConfig("worm_audit").withConnection("minio").withBucketName("worm-locked").withRole(BucketRole.BACKUP).withIndexed(false))
|
||||
.withBucket(new StorageBucketConfig("plain").withConnection("minio").withBucketName("worm-plain").withRole(BucketRole.BACKUP).withIndexed(false)));
|
||||
CapabilityProbe.probeConnection("minio", false);
|
||||
assertThat(ObjectStorage.asSystem("test", "1").bucket("worm_audit").capability(Capability.OBJECT_LOCK)).isEqualTo(CapabilityStatus.SUPPORTED);
|
||||
assertThat(ObjectStorage.asSystem("test", "1").bucket("plain").capability(Capability.OBJECT_LOCK)).isEqualTo(CapabilityStatus.UNSUPPORTED);
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
Instant now = Instant.now();
|
||||
var exported = new SealedExporter(config).run("USER", now);
|
||||
assertThat(exported.failed()).as(exported.message()).isFalse();
|
||||
var row = exported.row();
|
||||
var storage = new ObjectStorageSealedStore("worm_audit", "acceptance");
|
||||
assertThat(storage.list(ListRequest.prefix(row.getValueString("object_prefix"))).getItems()).hasSize(3);
|
||||
var verifier = new SealedExportVerifier(config);
|
||||
assertThat(verifier.verify(row, "USER").result().passed()).isTrue();
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String key = row.getValueString("object_prefix") + name;
|
||||
var head = storage.head(key);
|
||||
assertThat(head.getLockMode()).isEqualTo("COMPLIANCE");
|
||||
assertThat(Duration.between(now.atZone(java.time.ZoneOffset.UTC).plusYears(6).toInstant(), head.getLockRetainUntil()).abs()).isLessThan(Duration.ofSeconds(2));
|
||||
String originalVersion = head.getVersionId();
|
||||
assertThat(originalVersion).isNotBlank();
|
||||
assertThat(ExportFiles.parse(row.getValueString("object_versions").getBytes(java.nio.charset.StandardCharsets.UTF_8)))
|
||||
.containsEntry(name, originalVersion);
|
||||
assertThatThrownBy(() -> s3.deleteObject(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)))
|
||||
.isInstanceOf(S3Exception.class);
|
||||
byte[] original = s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray();
|
||||
// S3 overwrites create new versions; they cannot replace the locked version.
|
||||
var replacement = s3.putObject(b -> b.bucket("worm-locked").key(key).objectLockMode("COMPLIANCE")
|
||||
.objectLockRetainUntilDate(row.getValueInstant("retain_until")), RequestBody.fromString("different replacement bytes"));
|
||||
assertThat(replacement.versionId()).isNotEqualTo(originalVersion);
|
||||
assertThat(s3.getObjectAsBytes(b -> b.bucket("worm-locked").key(key).versionId(originalVersion)).asByteArray()).isEqualTo(original);
|
||||
try(var sealed = storage.getVersion(key, originalVersion))
|
||||
{
|
||||
assertThat(sealed.readAllBytes()).isEqualTo(original);
|
||||
}
|
||||
var report = verifier.verify(row, "USER");
|
||||
assertThat(report.result().passed()).isFalse();
|
||||
assertThat(report.result().getLinksChecked()).isEqualTo(1);
|
||||
assertThat(report.result().getBreaks()).allSatisfy(b -> assertThat(b.reason()).contains("was overwritten after sealing"));
|
||||
assertThat(report.result().getBreaks().toString()).contains("object " + name + " was overwritten after sealing (current version "
|
||||
+ replacement.versionId() + ", sealed version " + originalVersion + " is still retained under Object Lock)");
|
||||
}
|
||||
tamper("auditDetail", 1, "newValue", "tampered live value");
|
||||
var load = new ExportProcesses.VerifyLoad();
|
||||
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
|
||||
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString())
|
||||
.contains("ERROR", "link 1 (audit 1): live content differs from the sealed copy");
|
||||
tamper("auditDetail", 1, "newValue", "new");
|
||||
audit(2);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
config.withExportBucketAlias("plain");
|
||||
var failed = new SealedExporter(config).run("SCHEDULE", Instant.now());
|
||||
assertThat(failed.failed()).isTrue();
|
||||
assertThat(failed.message()).contains("does not support Object Lock");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1);
|
||||
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ObjectStorageRuntime.reset();
|
||||
CapabilityStore.clearCache();
|
||||
System.clearProperty("worm.test.access");
|
||||
System.clearProperty("worm.test.secret");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
/*******************************************************************************
|
||||
** Tests and fixtures for phi canary test.
|
||||
*******************************************************************************/
|
||||
class PhiCanaryTest
|
||||
public class PhiCanaryTest
|
||||
{
|
||||
// RC.5 has AiFieldMetaData but not DataClassification; use a test-local marker.
|
||||
/*******************************************************************************
|
||||
@@ -93,6 +93,10 @@ class PhiCanaryTest
|
||||
{
|
||||
for(String column : List.of("oldValue", "newValue"))
|
||||
{
|
||||
if(detail.getValueString(column) == null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
try
|
||||
{
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
|
||||
@@ -133,6 +133,31 @@ class PostgresAcceptanceTest
|
||||
lookups();
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Home setup probes see dropped/disabled guards, missing purge and runtime ownership.
|
||||
** This test uses the same Docker condition as the existing PostgreSQL suite.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void homeSetupWarnings() throws Exception
|
||||
{
|
||||
var storage = new com.kingsrook.qbits.wormaudit.demo.MemorySealedStore();
|
||||
var clean = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
|
||||
assertThat(clean.warnings()).containsExactly("Daily verification has never run");
|
||||
try(var connection = owner())
|
||||
{
|
||||
sql(connection, "DROP TRIGGER worm_audit_row_guard ON audit_detail");
|
||||
sql(connection, "ALTER TABLE worm_audit_chain DISABLE TRIGGER worm_audit_truncate_guard");
|
||||
sql(connection, "DROP FUNCTION public.worm_audit_purge(timestamptz)");
|
||||
}
|
||||
var missing = com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now());
|
||||
assertThat(missing.warnings()).contains("Missing or disabled worm_audit_row_guard on audit_detail",
|
||||
"Missing or disabled worm_audit_truncate_guard on worm_audit_chain", "Retention purge function is missing");
|
||||
instance = RdbmsTestSupport.instance(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword(), config, true);
|
||||
QContext.init(instance, new QSession());
|
||||
assertThat(com.kingsrook.qbits.wormaudit.app.WormAuditHomeData.load(config, storage, java.time.Instant.now()).warnings())
|
||||
.contains("Runtime role owns audit table audit", "Runtime role owns audit table worm_audit_purge");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify grants triggers lookup rules and spoofed flag.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -84,6 +84,38 @@ class ProcessesAndAlertsTest
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** RC.5 validates declared severities, names and daily heartbeat expectations.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void alertTypesAndDailyMonitors() throws Exception
|
||||
{
|
||||
var instance = init(new WormAuditQBitConfig());
|
||||
new com.kingsrook.qqq.backend.core.instances.QInstanceValidator().validate(instance);
|
||||
assertThat(new com.kingsrook.qbits.wormaudit.chain.AuditStore(null).isMemory()).isTrue();
|
||||
var types = AlertsInstanceMetaData.of(instance).getAlertTypes();
|
||||
assertThat(types).containsOnlyKeys("wormaudit.chain.break", "wormaudit.export.failed", "wormaudit.verify.daily", "wormaudit.export.daily");
|
||||
assertThat(types.get("wormaudit.chain.break").getDefaultSeverity())
|
||||
.isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertSeverity.CRITICAL);
|
||||
assertThat(types.get("wormaudit.export.failed").getDefaultSeverity())
|
||||
.isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertSeverity.ERROR);
|
||||
types.values().forEach(type ->
|
||||
{
|
||||
assertThat(type.getLabel()).isNotBlank();
|
||||
assertThat(type.getDescription()).isNotBlank();
|
||||
assertThat(AlertsInstanceMetaData.ALERT_TYPE_NAME_PATTERN.matcher(type.getName()).matches()).isTrue();
|
||||
});
|
||||
for(String name : List.of("wormaudit.verify.daily", "wormaudit.export.daily"))
|
||||
{
|
||||
var monitors = types.get(name).getDefaultMonitors();
|
||||
assertThat(monitors).hasSize(1);
|
||||
assertThat(monitors.getFirst().getKind()).isEqualTo(com.kingsrook.qqq.backend.core.alerts.AlertMonitorKind.HEARTBEAT);
|
||||
assertThat(monitors.getFirst().getDefinition()).containsEntry("heartbeatKey", name)
|
||||
.containsEntry("expectedEverySeconds", 86400).containsEntry("graceSeconds", 7200);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify daily raise resolve heartbeat and user result.
|
||||
*******************************************************************************/
|
||||
@@ -106,7 +138,7 @@ class ProcessesAndAlertsTest
|
||||
daily.run(new RunBackendStepInput(), output);
|
||||
assertThat(output.getValueString("outcome")).isEqualTo("FAIL");
|
||||
assertThat(Sink.raised).hasSize(1);
|
||||
assertThat(Sink.raised.getFirst().getAlertTypeName()).isEqualTo("worm_audit.chain.break");
|
||||
assertThat(Sink.raised.getFirst().getAlertTypeName()).isEqualTo("wormaudit.chain.break");
|
||||
assertThat(Sink.raised.getFirst().getDedupKey()).isEqualTo("chain");
|
||||
assertThat(Sink.raised.getFirst().getRecordRefs().getFirst().getTableName())
|
||||
.isEqualTo(WormAuditTables.VERIFICATION);
|
||||
@@ -119,7 +151,7 @@ class ProcessesAndAlertsTest
|
||||
daily.run(new RunBackendStepInput(), new RunBackendStepOutput());
|
||||
assertThat(Sink.resolved).hasSize(1);
|
||||
assertThat(Sink.beats)
|
||||
.containsExactly("worm_audit.verify.daily", "worm_audit.verify.daily");
|
||||
.containsExactly("wormaudit.verify.daily", "wormaudit.verify.daily");
|
||||
assertThat(STORE.all(WormAuditTables.VERIFICATION)).hasSize(2);
|
||||
assertThat(VerificationRuns.run(null, "USER").passed()).isTrue();
|
||||
assertThat(Sink.beats).hasSize(2);
|
||||
|
||||
@@ -45,6 +45,45 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
*******************************************************************************/
|
||||
class RecordIntegrationTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** All record screens have readable labels and full-width sections and lists.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void recordScreensHaveReadableLabelsAndFullWidthLists() throws Exception
|
||||
{
|
||||
var instance = init(new WormAuditQBitConfig());
|
||||
Map<String, String> formats = Map.of(
|
||||
WormAuditTables.CHAIN, "Link #%s", WormAuditTables.CHECKPOINT, "Checkpoint #%s",
|
||||
WormAuditTables.EXPORT, "Export #%s (links %s–%s)", WormAuditTables.VERIFICATION, "Verification #%s (%s)",
|
||||
WormAuditTables.PURGE, "Purge #%s (%s)");
|
||||
for(String name : WormAuditTables.ALL)
|
||||
{
|
||||
var table = instance.getTable(name);
|
||||
assertThat(table.getRecordLabelFormat()).isEqualTo(formats.get(name));
|
||||
assertThat(table.getRecordLabelFields()).allMatch(table.getFields()::containsKey);
|
||||
table.getFields().forEach((fieldName, field) ->
|
||||
assertThat(field.getLabel()).as(name + "." + fieldName).isNotBlank().isNotEqualTo(fieldName).doesNotContain("_"));
|
||||
table.getSections().forEach(section ->
|
||||
{
|
||||
assertThat(section.getGridColumns()).isEqualTo(12);
|
||||
if(section.getWidgetName() != null)
|
||||
{
|
||||
assertThat(instance.getWidget(section.getWidgetName()).getGridColumns()).isEqualTo(12);
|
||||
}
|
||||
});
|
||||
}
|
||||
assertThat(instance.getTable(WormAuditTables.EXPORT).getRecordLabelFields()).containsExactly("id", "from_link_id", "to_link_id");
|
||||
for(String name : List.of(WormAuditTables.VERIFICATION, WormAuditTables.PURGE))
|
||||
{
|
||||
assertThat(instance.getTable(name).getRecordLabelFields()).containsExactly("id", "outcome");
|
||||
}
|
||||
var verification = instance.getTable(WormAuditTables.VERIFICATION);
|
||||
assertThat(verification.getSections()).extracting(section -> section.getLabel()).containsExactly("Result", "Scope", "Run", "Chain links");
|
||||
assertThat(verification.getSections().stream().filter(section -> section.getWidgetName() == null)
|
||||
.flatMap(section -> section.getFieldNames().stream()).toList()).containsExactlyInAnyOrderElementsOf(verification.getFields().keySet());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Tests and fixtures for event sink.
|
||||
*******************************************************************************/
|
||||
@@ -93,6 +132,8 @@ class RecordIntegrationTest
|
||||
assertThat(data.getTotalRows()).isEqualTo(2);
|
||||
assertThat(data.getQueryOutput().getRecords()).hasSize(2);
|
||||
assertThat(data.getViewAllLink()).contains("filter=");
|
||||
String json = com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(data);
|
||||
assertThat(json).contains("Chain links").doesNotContain("signingKey", "trustedPublicKeys", "sealedStoreSupplier", "privateKey");
|
||||
input.setQueryParams(Map.of("id", "999"));
|
||||
assertThatThrownBy(() -> new LinkRangeWidget().render(input))
|
||||
.hasMessageContaining("was not found");
|
||||
|
||||
@@ -0,0 +1,710 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.InputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
|
||||
import com.kingsrook.qbits.objectstorage.api.Page;
|
||||
import com.kingsrook.qbits.objectstorage.api.StoredObject;
|
||||
import com.kingsrook.qbits.wormaudit.canonical.Jcs;
|
||||
import com.kingsrook.qbits.wormaudit.chain.Hashes;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.export.ExportFiles;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.ExportProcesses;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunInput;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.BackendStepPostRunOutput;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.remove;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Offline sealed-export contract, corruption, retention, batching and UI tests.
|
||||
*******************************************************************************/
|
||||
class SealedExportTest
|
||||
{
|
||||
static final Instant NOW = Instant.parse("2026-10-14T02:00:00Z");
|
||||
WormAuditQBitConfig config;
|
||||
MemorySealedStore storage;
|
||||
|
||||
/*******************************************************************************
|
||||
** Tiny object boundary; bytes exist only in tests.
|
||||
*******************************************************************************/
|
||||
static class MemorySealedStore implements SealedStore
|
||||
{
|
||||
final Map<String, byte[]> objects = new LinkedHashMap<>();
|
||||
final Map<String, StoredObject> heads = new LinkedHashMap<>();
|
||||
final Map<String, Map<String, byte[]>> history = new LinkedHashMap<>();
|
||||
int nextVersion;
|
||||
String missingVersion;
|
||||
boolean omitVersion;
|
||||
boolean racePut;
|
||||
boolean supported = true;
|
||||
String echoMode = "COMPLIANCE";
|
||||
long shortenSeconds;
|
||||
int failPut;
|
||||
|
||||
@Override
|
||||
public boolean supportsLock()
|
||||
{
|
||||
return (supported);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String put(String key, Path file, Instant until) throws Exception
|
||||
{
|
||||
if(failPut > 0 && objects.size() + 1 == failPut)
|
||||
{
|
||||
throw (new IllegalStateException("CANARY-PRIVATE-PROVIDER-TEXT"));
|
||||
}
|
||||
if(objects.containsKey(key))
|
||||
{
|
||||
history.computeIfAbsent(key, k -> new LinkedHashMap<>()).put(heads.get(key).getVersionId(), objects.get(key));
|
||||
}
|
||||
String version = omitVersion ? missingVersion : "v" + ++nextVersion;
|
||||
objects.put(key, Files.readAllBytes(file));
|
||||
heads.put(key, new StoredObject().withKey(key).withVersionId(racePut ? "replacement" : version)
|
||||
.withLockMode(echoMode).withLockRetainUntil(until.minusSeconds(shortenSeconds)));
|
||||
return (version);
|
||||
}
|
||||
|
||||
@Override
|
||||
public StoredObject head(String key)
|
||||
{
|
||||
return (heads.get(key));
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream getVersion(String key, String versionId)
|
||||
{
|
||||
byte[] bytes = versionId.equals(heads.get(key).getVersionId()) ? objects.get(key) : history.get(key).get(versionId);
|
||||
return (new ByteArrayInputStream(bytes));
|
||||
}
|
||||
|
||||
@Override
|
||||
public Page<ObjectSummary> list(ListRequest request)
|
||||
{
|
||||
return (new Page<ObjectSummary>().withItems(objects.keySet().stream().filter(k -> k.startsWith(request.getPrefix()))
|
||||
.map(k -> new ObjectSummary().withKey(k)).toList()));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise setUp.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
config = ChainVerificationTest.signed();
|
||||
var instance = init(config);
|
||||
storage = new MemorySealedStore();
|
||||
ProcessesAndAlertsTest.Sink.raised.clear();
|
||||
ProcessesAndAlertsTest.Sink.resolved.clear();
|
||||
ProcessesAndAlertsTest.Sink.beats.clear();
|
||||
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(ProcessesAndAlertsTest.Sink.class));
|
||||
}
|
||||
|
||||
|
||||
QRecord seal() throws Exception
|
||||
{
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
var result = new SealedExporter(config, storage).run("USER", NOW);
|
||||
assertThat(result.failed()).as(result.message()).isFalse();
|
||||
assertThat(result.row()).isNotNull();
|
||||
return (result.row());
|
||||
}
|
||||
|
||||
|
||||
SealedExportVerifier.Report verify(QRecord row) throws Exception
|
||||
{
|
||||
return (new SealedExportVerifier(config, storage).verify(row, "USER"));
|
||||
}
|
||||
|
||||
|
||||
String findings(QRecord row) throws Exception
|
||||
{
|
||||
return (verify(row).result().getBreaks().toString());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise canonicalGoldenFilesAndRoundTrip.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void canonicalGoldenFilesAndRoundTrip() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
QRecord row = seal();
|
||||
String prefix = "worm-audit/2026/10/14/1-1/";
|
||||
assertThat(row.getValueString("object_prefix")).isEqualTo(prefix);
|
||||
String content = "{\"audit\":{\"clientChannel\":null,\"id\":1,\"message\":\"Record was Edited\",\"recordId\":\"17\",\"table\":\"cms_page\",\"timestamp\":\"2020-01-02T03:04:06.123456Z\",\"user\":\"james\"},\"details\":[{\"fieldName\":\"title\",\"id\":1,\"message\":\"Changed Title\",\"newValue\":\"new\",\"oldValue\":\"old\"}],\"v\":1}";
|
||||
String hash = Hashes.link(Hashes.GENESIS, content.getBytes(StandardCharsets.UTF_8));
|
||||
String golden = "{\"content\":" + content + ",\"link\":{\"auditId\":1,\"detailIds\":[1],\"id\":1,\"kind\":\"AUDIT\",\"linkHash\":\"" + hash
|
||||
+ "\",\"prevHash\":\"" + Hashes.GENESIS + "\"}}\n";
|
||||
assertThat(new String(storage.objects.get(prefix + "links.jsonl"), StandardCharsets.UTF_8)).isEqualTo(golden);
|
||||
var manifest = ExportFiles.parse(storage.objects.get(prefix + "manifest.json"));
|
||||
assertThat(manifest.keySet()).containsExactly("checkpointCount", "createdAt", "files", "format", "fromLinkId", "headHash", "linkCount",
|
||||
"prevManifestSha256", "qbitVersion", "toLinkId");
|
||||
assertThat(manifest).containsEntry("format", "worm-audit-export/1").containsEntry("linkCount", 1L).containsEntry("checkpointCount", 1L)
|
||||
.containsEntry("prevManifestSha256", null).containsEntry("headHash", hash).containsEntry("createdAt", NOW.toString());
|
||||
assertThat(storage.objects.get(prefix + "manifest.json")).isEqualTo(Jcs.bytes(manifest));
|
||||
assertThat(row.getValues()).containsKeys("from_checkpoint_id", "to_checkpoint_id", "bucket_alias", "manifest_key", "manifest_sha256",
|
||||
"prev_manifest_sha256", "object_count", "byte_count", "lock_mode", "retain_until", "sealed_at");
|
||||
assertThat(row.getValueLong("byte_count")).isEqualTo(storage.objects.values().stream().mapToLong(b -> b.length).sum());
|
||||
assertThat(row.getValueInstant("retain_until")).isEqualTo(Instant.parse("2032-10-14T02:00:00Z"));
|
||||
assertThat(verify(row).result().passed()).isTrue();
|
||||
QRecord verification = STORE.last(WormAuditTables.VERIFICATION);
|
||||
assertThat(verification.getValueString("kind")).isEqualTo("SEALED_EXPORT");
|
||||
assertThat(verification.getValueLong("export_id")).isEqualTo(row.getValueLong("id"));
|
||||
assertThat(verification.getValueString("outcome")).isEqualTo("PASS");
|
||||
assertThat(ProcessesAndAlertsTest.Sink.resolved).hasSize(1);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every overwrite is a break while verification still uses the sealed bytes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void detectsOverwritesAndReadsSealedVersions() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
QRecord row = seal();
|
||||
Map<String, Object> versions = ExportFiles.parse(row.getValueString("object_versions").getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(versions.keySet()).containsExactly("checkpoints.jsonl", "links.jsonl", "manifest.json");
|
||||
assertThat(row.getValueString("object_versions")).isEqualTo(Jcs.string(versions));
|
||||
assertThat(STORE.get(WormAuditTables.EXPORT, row.getValueLong("id")).getValueString("object_versions"))
|
||||
.isEqualTo(row.getValueString("object_versions"));
|
||||
try(var files = new ExportFiles())
|
||||
{
|
||||
String linksKey = row.getValueString("object_prefix") + "links.jsonl";
|
||||
Files.write(files.file("links.jsonl"), storage.objects.get(linksKey));
|
||||
storage.put(linksKey, files.file("links.jsonl"), row.getValueInstant("retain_until"));
|
||||
var identical = verify(row);
|
||||
assertThat(identical.result().passed()).isFalse();
|
||||
assertThat(identical.result().getLinksChecked()).isEqualTo(1);
|
||||
assertThat(identical.result().getBreaks()).singleElement().satisfies(b ->
|
||||
assertThat(b.reason()).contains("object links.jsonl was overwritten after sealing"));
|
||||
for(String name : ExportFiles.NAMES)
|
||||
{
|
||||
String key = row.getValueString("object_prefix") + name;
|
||||
Files.writeString(files.file(name), "different bytes");
|
||||
String current = storage.put(key, files.file(name), row.getValueInstant("retain_until"));
|
||||
var report = verify(row);
|
||||
assertThat(report.result().passed()).isFalse();
|
||||
assertThat(report.result().getLinksChecked()).isEqualTo(1);
|
||||
assertThat(report.result().getBreaks()).allSatisfy(b -> assertThat(b.reason()).contains("was overwritten after sealing"));
|
||||
assertThat(report.result().getBreaks().toString()).contains("object " + name + " was overwritten after sealing (current version "
|
||||
+ current + ", sealed version " + versions.get(name) + " is still retained under Object Lock)");
|
||||
}
|
||||
}
|
||||
var load = new ExportProcesses.VerifyLoad(storage);
|
||||
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
|
||||
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString()).contains("ERROR", "was overwritten after sealing");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Unversioned uploads and upload/HEAD races cannot create a sealed catalog row.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesMissingVersionsAndUploadRaces() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
storage.omitVersion = true;
|
||||
for(String version : java.util.Arrays.asList(null, "", " ", "null"))
|
||||
{
|
||||
storage.missingVersion = version;
|
||||
var result = new SealedExporter(config, storage).run("USER", NOW);
|
||||
assertThat(result.failed()).isTrue();
|
||||
assertThat(result.message()).contains("returned no version id");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
|
||||
}
|
||||
storage.omitVersion = false;
|
||||
storage.racePut = true;
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("version changed after upload");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
|
||||
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(5);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Missing catalog versions must never silently fall back to current content.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesMissingRecordedVersion() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
QRecord row = seal();
|
||||
row.setValue("object_versions", "{}");
|
||||
assertThat(findings(row)).contains("object links.jsonl has no recorded sealed version id");
|
||||
row.setValue("object_versions", null);
|
||||
assertThat(verify(row).result().passed()).isFalse();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise rangeSelectionNoCheckpointContiguousAndPurgeAnchor.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void rangeSelectionNoCheckpointContiguousAndPurgeAnchor() throws Exception
|
||||
{
|
||||
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).isEqualTo("Nothing to seal: no new checkpoint");
|
||||
assertThat(ProcessesAndAlertsTest.Sink.beats).containsExactly("wormaudit.export.daily");
|
||||
audit(1);
|
||||
QRecord first = seal();
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).row()).isNull();
|
||||
audit(2);
|
||||
QRecord second = seal();
|
||||
assertThat(second.getValueLong("from_link_id")).isEqualTo(2L);
|
||||
assertThat(second.getValueString("prev_manifest_sha256")).isEqualTo(first.getValueString("manifest_sha256"));
|
||||
assertThat(verify(second).result().passed()).isTrue();
|
||||
|
||||
init(config);
|
||||
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 10L).withValue("anchor_hash", Hashes.GENESIS)));
|
||||
audit(11);
|
||||
QRecord anchored = seal();
|
||||
assertThat(anchored.getValueLong("from_link_id")).isEqualTo(11L);
|
||||
assertThat(anchored.getValueString("prev_manifest_sha256")).isNull();
|
||||
assertThat(verify(anchored).result().passed()).isTrue();
|
||||
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 20L).withValue("anchor_hash", Hashes.GENESIS)));
|
||||
assertThatThrownBy(() -> SealedExporter.scope(STORE)).hasMessage("Unexported links were purged");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise refusesUnsupportedAndIncorrectLockEchoAndPartialUpload.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesUnsupportedAndIncorrectLockEchoAndPartialUpload() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
storage.supported = false;
|
||||
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).contains("does not support Object Lock");
|
||||
assertThat(storage.objects).isEmpty();
|
||||
storage.supported = true;
|
||||
storage.echoMode = "GOVERNANCE";
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("did not echo COMPLIANCE");
|
||||
storage.echoMode = "COMPLIANCE";
|
||||
storage.shortenSeconds = 1;
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
|
||||
storage.shortenSeconds = 0;
|
||||
storage.objects.clear();
|
||||
storage.failPut = 2;
|
||||
assertThat(new SealedExporter(config, storage).run("SCHEDULE", NOW).message()).doesNotContain("CANARY").contains("storage or database");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
|
||||
assertThat(ProcessesAndAlertsTest.Sink.raised).hasSize(4);
|
||||
assertThat(ProcessesAndAlertsTest.Sink.beats).hasSize(2);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise tamperedObjectsBrokenManifestChainAndLockRemoval.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void tamperedObjectsBrokenManifestChainAndLockRemoval() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
QRecord first = seal();
|
||||
String key = first.getValueString("object_prefix") + "links.jsonl";
|
||||
byte[] original = storage.objects.get(key);
|
||||
storage.objects.put(key, "tampered".getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(findings(first)).contains("object links.jsonl hash does not match the manifest");
|
||||
storage.objects.put(key, original);
|
||||
storage.heads.get(key).withLockMode("GOVERNANCE");
|
||||
assertThat(findings(first)).contains("lacks the recorded COMPLIANCE retention");
|
||||
storage.heads.get(key).withLockMode("COMPLIANCE");
|
||||
audit(2);
|
||||
QRecord second = seal();
|
||||
tamper(WormAuditTables.EXPORT, first.getValueLong("id"), "manifest_sha256", "f".repeat(64));
|
||||
assertThat(findings(second)).contains("prevManifestSha256 does not match");
|
||||
assertThat(findings(STORE.get(WormAuditTables.EXPORT, first.getValueLong("id")))).contains("manifest.json hash does not match");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise exactLiveDifferencesMissingRowsAndPurgedCopies.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void exactLiveDifferencesMissingRowsAndPurgedCopies() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
audit(2);
|
||||
QRecord row = seal();
|
||||
tamper("auditDetail", 1, "newValue", "CANARY-PRIVATE");
|
||||
assertThat(findings(row)).contains("link 1 (audit 1): live content differs from the sealed copy (sealed 2026-10-14)").doesNotContain("CANARY");
|
||||
remove(WormAuditTables.CHAIN, 2);
|
||||
assertThat(findings(row)).contains("link 2 is missing from the live chain");
|
||||
tamper(WormAuditTables.CHAIN, 1, "kind", "LATE_DETAIL");
|
||||
assertThat(findings(row)).contains("live chain kind differs");
|
||||
remove("audit", 1);
|
||||
assertThat(findings(row)).contains("live audit header is missing");
|
||||
String head = (String) ExportFiles.parse(storage.objects.get(row.getValueString("manifest_key"))).get("headHash");
|
||||
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("outcome", "PURGED").withValue("anchor_link_id", 2L).withValue("anchor_hash", head)));
|
||||
remove(WormAuditTables.CHAIN, 1);
|
||||
remove("audit", 2);
|
||||
var report = verify(row);
|
||||
assertThat(report.result().passed()).isTrue();
|
||||
assertThat(report.purged()).isEqualTo(2);
|
||||
var load = new ExportProcesses.VerifyLoad(storage);
|
||||
load.runOnePage(new RunBackendStepInput().withRecords(List.of(row)), new RunBackendStepOutput());
|
||||
assertThat(load.getProcessSummary(new RunBackendStepOutput(), true).toString()).contains("INFO", "2 links verified from the sealed copy only");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise refusesBadSourceAndDetectsMissingLiveDetails.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesBadSourceAndDetectsMissingLiveDetails() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
tamper("auditDetail", 1, "newValue", "changed");
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).message()).contains("content differs at link 1");
|
||||
assertThat(storage.objects).isEmpty();
|
||||
tamper("auditDetail", 1, "newValue", "new");
|
||||
QRecord row = seal();
|
||||
remove("auditDetail", 1);
|
||||
assertThat(findings(row)).contains("link 1 (audit 1): live content is missing or invalid");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise spansPagesLateDetailsAndCheckpointBoundaries.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void spansPagesLateDetailsAndCheckpointBoundaries() throws Exception
|
||||
{
|
||||
config.withBatchSize(1100).withCheckpointEveryLinks(500);
|
||||
for(long id = 1; id <= 1100; id++)
|
||||
{
|
||||
audit(id);
|
||||
}
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
// The final 100 links need the hourly checkpoint.
|
||||
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(3601));
|
||||
QRecord row = seal();
|
||||
assertThat(row.getValueLong("to_link_id")).isEqualTo(1100);
|
||||
assertThat(verify(row).result().getLinksChecked()).isEqualTo(1100);
|
||||
detail(1, 1);
|
||||
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(7202));
|
||||
new WormAuditChainer(config).chainOnce(NOW.plusSeconds(10803));
|
||||
QRecord late = seal();
|
||||
assertThat(late.getValueLong("from_link_id")).isEqualTo(1101);
|
||||
assertThat(verify(late).result().passed()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise reviewLoadSelectionAndSchedulesUseStandardEtl.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void reviewLoadSelectionAndSchedulesUseStandardEtl() throws Exception
|
||||
{
|
||||
var instance = com.kingsrook.qqq.backend.core.context.QContext.getQInstance();
|
||||
assertThat(config.getExportBucketAlias()).isEqualTo("worm_audit");
|
||||
assertThat(config.getExportPrefix()).isEqualTo("worm-audit");
|
||||
assertThat(instance.getProcess(ExportProcesses.EXPORT).getTableName()).isEqualTo(WormAuditTables.EXPORT);
|
||||
assertThat(instance.getProcess(ExportProcesses.VERIFY).getMinInputRecords()).isEqualTo(1);
|
||||
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronTimeZoneId()).isEqualTo("UTC");
|
||||
assertThat(instance.getProcess(ExportProcesses.DAILY).getSchedule().getCronExpression()).isEqualTo("0 0 2 * * ?");
|
||||
var output = new RunBackendStepOutput();
|
||||
var input = new RunBackendStepInput().withRecords(List.of(new QRecord().withValue("id", 0L)));
|
||||
var extract = new ExportProcesses.ExportExtract();
|
||||
assertThat(extract.doCount(input)).isEqualTo(1);
|
||||
extract.setRecordPipe(new com.kingsrook.qqq.backend.core.actions.reporting.RecordPipe());
|
||||
extract.run(input, output);
|
||||
var review = new ExportProcesses.ExportReview();
|
||||
review.runOnePage(input, output);
|
||||
assertThat(review.getProcessSummary(output, false).toString()).contains("Nothing to seal");
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
review.runOnePage(input, output);
|
||||
assertThat(review.getProcessSummary(output, false).toString()).contains("1 chain links will be sealed", "1 checkpoints");
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).isEmpty();
|
||||
var load = new ExportProcesses.ExportLoad(storage);
|
||||
assertThat(load.getProcessSummary(output, true)).isEmpty();
|
||||
load.runOnePage(input, output);
|
||||
load.postRun(new BackendStepPostRunInput(input), new BackendStepPostRunOutput(output));
|
||||
assertThat(load.getProcessSummary(output, true).toString()).contains("1 links sealed in export");
|
||||
var rows = STORE.all(WormAuditTables.EXPORT);
|
||||
var verifyInput = new RunBackendStepInput().withRecords(rows).addValue("recordIds", rows.getFirst().getValueString("id"));
|
||||
assertThat(new ExportProcesses.VerifyExtract().getQueryFilter(verifyInput).getCriteria()).isNotEmpty();
|
||||
var verifyReview = new ExportProcesses.VerifyReview();
|
||||
verifyReview.runOnePage(verifyInput, output);
|
||||
assertThat(verifyReview.getProcessSummary(output, false).toString()).contains("1 sealed exports");
|
||||
var verifyLoad = new ExportProcesses.VerifyLoad(storage);
|
||||
verifyLoad.runOnePage(verifyInput, output);
|
||||
assertThat(verifyLoad.getProcessSummary(output, true).toString()).contains("1 links verified");
|
||||
new ExportProcesses.Daily().run(input, output);
|
||||
assertThat(output.getValueString("outcome")).isEqualTo("PASS");
|
||||
audit(2);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
new ExportProcesses.Daily().run(input, output);
|
||||
assertThat(output.getValueString("outcome")).isEqualTo("FAIL");
|
||||
assertThat(ProcessesAndAlertsTest.Sink.beats).contains("wormaudit.export.daily");
|
||||
new ExportProcesses.ExportLoad();
|
||||
new ExportProcesses.VerifyLoad();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise malformedJsonIsValueFreeAndConfigRejectsUnsafePrefixes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void malformedJsonIsValueFreeAndConfigRejectsUnsafePrefixes() throws Exception
|
||||
{
|
||||
assertThatThrownBy(() -> ExportFiles.parse("{\"x\":\"CANARY\",\"x\":2}".getBytes(StandardCharsets.UTF_8)))
|
||||
.hasMessage("Invalid sealed JSON").hasNoCause();
|
||||
assertThatThrownBy(() -> config.withExportPrefix("../escape").validate()).isInstanceOf(IllegalArgumentException.class);
|
||||
config.withExportPrefix("evidence/audit").withExportBucketAlias("backup").withExportCron("0 5 2 * * ?").validate();
|
||||
assertThat(config.getExportBucketAlias()).isEqualTo("backup");
|
||||
try(var files = new ExportFiles())
|
||||
{
|
||||
assertThatThrownBy(() -> files.file("../escape")).isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Re-hash a tampered object and catalog to reach the independent signature checks.
|
||||
*******************************************************************************/
|
||||
QRecord rewrite(QRecord row, String name, byte[] bytes) throws Exception
|
||||
{
|
||||
String prefix = row.getValueString("object_prefix");
|
||||
storage.objects.put(prefix + name, bytes);
|
||||
Map<String, Object> manifest = ExportFiles.parse(storage.objects.get(prefix + "manifest.json"));
|
||||
Map<String, Object> hashes = new LinkedHashMap<>((Map<String, Object>) manifest.get("files"));
|
||||
hashes.put(name, java.util.HexFormat.of().formatHex(Hashes.digest(bytes)));
|
||||
manifest.put("files", hashes);
|
||||
byte[] encoded = Jcs.bytes(manifest);
|
||||
storage.objects.put(prefix + "manifest.json", encoded);
|
||||
tamper(WormAuditTables.EXPORT, row.getValueLong("id"), "manifest_sha256", java.util.HexFormat.of().formatHex(Hashes.digest(encoded)));
|
||||
return (STORE.get(WormAuditTables.EXPORT, row.getValueLong("id")));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise independentSignaturesMerkleAndSealedContentChecks.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void independentSignaturesMerkleAndSealedContentChecks() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
QRecord row = seal();
|
||||
String prefix = row.getValueString("object_prefix");
|
||||
byte[] originalCheckpoint = storage.objects.get(prefix + "checkpoints.jsonl");
|
||||
Map<String, Object> checkpoint = ExportFiles.parse(originalCheckpoint);
|
||||
checkpoint.put("signature", java.util.Base64.getEncoder().encodeToString(new byte[64]));
|
||||
row = rewrite(row, "checkpoints.jsonl", (Jcs.string(checkpoint) + "\n").getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(findings(row)).contains("checkpoint 1: signature does not verify");
|
||||
checkpoint = ExportFiles.parse(originalCheckpoint);
|
||||
checkpoint.put("merkle_root", "a".repeat(64));
|
||||
row = rewrite(row, "checkpoints.jsonl", (Jcs.string(checkpoint) + "\n").getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(findings(row)).contains("Merkle root or head hash differs");
|
||||
row = rewrite(row, "checkpoints.jsonl", originalCheckpoint);
|
||||
Map<String, Object> document = ExportFiles.parse(storage.objects.get(prefix + "links.jsonl"));
|
||||
Map<String, Object> content = (Map<String, Object>) document.get("content");
|
||||
((Map<String, Object>) content.get("audit")).put("message", "CANARY-SEALED");
|
||||
row = rewrite(row, "links.jsonl", (Jcs.string(document) + "\n").getBytes(StandardCharsets.UTF_8));
|
||||
assertThat(findings(row)).contains("sealed content differs from the chained hash").doesNotContain("CANARY");
|
||||
storage.objects.remove(prefix + "links.jsonl");
|
||||
assertThat(findings(row)).contains("cannot be read or has invalid structure").doesNotContain("CANARY");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise concurrentExportersDoNotOverlapAndMultipleSelectionsPersistSeparately.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void concurrentExportersDoNotOverlapAndMultipleSelectionsPersistSeparately() throws Exception
|
||||
{
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
var instance = com.kingsrook.qqq.backend.core.context.QContext.getQInstance();
|
||||
var workers = java.util.concurrent.Executors.newFixedThreadPool(2);
|
||||
try
|
||||
{
|
||||
var jobs = new java.util.ArrayList<java.util.concurrent.Future<SealedExporter.Result>>();
|
||||
for(int i = 0; i < 2; i++)
|
||||
{
|
||||
jobs.add(workers.submit(() ->
|
||||
{
|
||||
com.kingsrook.qqq.backend.core.context.QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSession());
|
||||
try
|
||||
{
|
||||
return (new SealedExporter(config, storage).run("USER", NOW));
|
||||
}
|
||||
finally
|
||||
{
|
||||
com.kingsrook.qqq.backend.core.context.QContext.clear();
|
||||
}
|
||||
}));
|
||||
}
|
||||
int sealed = 0;
|
||||
for(var job : jobs)
|
||||
{
|
||||
var result = job.get(10, java.util.concurrent.TimeUnit.SECONDS);
|
||||
assertThat(result.failed()).as(result.message()).isFalse();
|
||||
sealed += result.row() == null ? 0 : 1;
|
||||
}
|
||||
assertThat(sealed).isEqualTo(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
workers.shutdownNow();
|
||||
}
|
||||
audit(2);
|
||||
seal();
|
||||
var load = new ExportProcesses.VerifyLoad(storage);
|
||||
load.runOnePage(new RunBackendStepInput().withRecords(STORE.all(WormAuditTables.EXPORT)), new RunBackendStepOutput());
|
||||
assertThat(STORE.all(WormAuditTables.VERIFICATION)).hasSize(2)
|
||||
.allMatch(r -> "PASS".equals(r.getValueString("outcome")));
|
||||
assertThat(STORE.all(WormAuditTables.VERIFICATION)).extracting(r -> r.getValueLong("export_id")).doesNotHaveDuplicates();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise h2TransactionRoundTripAndFailedPutHasNoCatalogRow.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void h2TransactionRoundTripAndFailedPutHasNoCatalogRow() throws Exception
|
||||
{
|
||||
String url = "jdbc:h2:mem:exports_" + java.util.UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE";
|
||||
var instance = RdbmsTestSupport.instance(url, "sa", "", config, false);
|
||||
try(var connection = java.sql.DriverManager.getConnection(url, "sa", ""))
|
||||
{
|
||||
RdbmsTestSupport.createTables(connection, instance, false);
|
||||
com.kingsrook.qqq.backend.core.context.QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSession());
|
||||
WormAuditTestSupport.lookups();
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
QRecord row = seal();
|
||||
assertThat(verify(row).result().passed()).isTrue();
|
||||
assertThat(verify(STORE.get(WormAuditTables.EXPORT, row.getValueLong("id"))).result().passed()).isTrue();
|
||||
connection.createStatement().execute("UPDATE audit_detail SET new_value='changed' WHERE id=1");
|
||||
assertThat(findings(row)).contains("link 1 (audit 1): live content differs");
|
||||
audit(2);
|
||||
new WormAuditChainer(config).chainOnce(NOW);
|
||||
storage.failPut = storage.objects.size() + 1;
|
||||
assertThat(new SealedExporter(config, storage).run("USER", NOW).failed()).isTrue();
|
||||
assertThat(STORE.all(WormAuditTables.EXPORT)).hasSize(1);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise realAdapterRequiresBackupRoleAndPositiveCapability.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void realAdapterRequiresBackupRoleAndPositiveCapability() throws Exception
|
||||
{
|
||||
var bucket = new com.kingsrook.qbits.objectstorage.config.StorageBucketConfig("worm_audit")
|
||||
.withConnection("test").withBucketName("test-bucket").withIndexed(false)
|
||||
.withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.BACKUP);
|
||||
System.setProperty("worm.unit.access", "unit-access");
|
||||
System.setProperty("worm.unit.secret", "unit-secret");
|
||||
var server = com.sun.net.httpserver.HttpServer.create(new java.net.InetSocketAddress("127.0.0.1", 0), 0);
|
||||
var responseVersion = new java.util.concurrent.atomic.AtomicReference<>("sealed-version");
|
||||
var requested = new java.util.concurrent.CopyOnWriteArrayList<String>();
|
||||
server.createContext("/test-bucket/links.jsonl", exchange ->
|
||||
{
|
||||
requested.add(exchange.getRequestURI().getRawQuery());
|
||||
exchange.getResponseHeaders().add("x-amz-version-id", responseVersion.get());
|
||||
byte[] bytes = "sealed fixture".getBytes(StandardCharsets.UTF_8);
|
||||
exchange.sendResponseHeaders(200, bytes.length);
|
||||
try(var output = exchange.getResponseBody())
|
||||
{
|
||||
output.write(bytes);
|
||||
}
|
||||
});
|
||||
server.start();
|
||||
try
|
||||
{
|
||||
com.kingsrook.qbits.objectstorage.api.ObjectStorage.configureStandalone(
|
||||
new com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig().withBackendName("memory")
|
||||
.withAllowedSecretRefPatterns(List.of("prop.worm.unit.*"))
|
||||
.withConnection(new com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig("test")
|
||||
.withEndpoint("http://127.0.0.1:" + server.getAddress().getPort()).withRegion("us-east-1")
|
||||
.withAddressingStyle(com.kingsrook.qbits.objectstorage.config.AddressingStyle.PATH)
|
||||
.withProviderType(com.kingsrook.qbits.objectstorage.config.ProviderType.MINIO)
|
||||
.withAccessKeyRef("${prop.worm.unit.access}").withSecretKeyRef("${prop.worm.unit.secret}"))
|
||||
.withBucket(bucket));
|
||||
var adapter = new com.kingsrook.qbits.wormaudit.export.ObjectStorageSealedStore("worm_audit", "unit");
|
||||
assertThat(adapter.supportsLock()).isFalse();
|
||||
com.kingsrook.qbits.objectstorage.internal.CapabilityStore.record(bucket,
|
||||
com.kingsrook.qbits.objectstorage.api.Capability.OBJECT_LOCK,
|
||||
com.kingsrook.qbits.objectstorage.api.CapabilityStatus.SUPPORTED, "unit fixture");
|
||||
assertThat(adapter.supportsLock()).isTrue();
|
||||
try(var input = adapter.getVersion("links.jsonl", "sealed-version"))
|
||||
{
|
||||
assertThat(new String(input.readAllBytes(), StandardCharsets.UTF_8)).isEqualTo("sealed fixture");
|
||||
}
|
||||
responseVersion.set("different-version");
|
||||
assertThatThrownBy(() -> adapter.getVersion("links.jsonl", "sealed-version"))
|
||||
.hasMessage("Storage returned a different object version");
|
||||
assertThatThrownBy(() -> adapter.getVersion("links.jsonl", null)).hasMessage("Missing sealed object version");
|
||||
assertThat(requested).containsExactly("versionId=sealed-version", "versionId=sealed-version");
|
||||
bucket.withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.PRIVATE);
|
||||
assertThat(adapter.supportsLock()).isFalse();
|
||||
}
|
||||
finally
|
||||
{
|
||||
server.stop(0);
|
||||
com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime.reset();
|
||||
System.clearProperty("worm.unit.access");
|
||||
System.clearProperty("worm.unit.secret");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit;
|
||||
|
||||
|
||||
import java.sql.DriverManager;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.UUID;
|
||||
import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditApp;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeData;
|
||||
import com.kingsrook.qbits.wormaudit.app.WormAuditHomeView;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.demo.MemorySealedStore;
|
||||
import com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExportVerifier;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
|
||||
import com.kingsrook.qbits.wormaudit.verify.VerificationResult;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
|
||||
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
|
||||
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.OLD;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.STORE;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.audit;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.detail;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.init;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.lookups;
|
||||
import static com.kingsrook.qbits.wormaudit.WormAuditTestSupport.tamper;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Home acceptance: truthful summaries, permissions, privacy and real demo data.
|
||||
*******************************************************************************/
|
||||
class WormAuditHomeTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Empty tables do not imply successful verification or export coverage.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void emptyStatesAndWarnings() throws Exception
|
||||
{
|
||||
var config = new WormAuditQBitConfig();
|
||||
init(config);
|
||||
var data = WormAuditHomeData.load(config, new MemorySealedStore(), OLD);
|
||||
assertThat(data.links()).isZero();
|
||||
assertThat(data.backlog()).isZero();
|
||||
assertThat(data.settling()).isZero();
|
||||
assertThat(data.anchor()).isNull();
|
||||
assertThat(data.oldestAudit()).isNull();
|
||||
assertThat(data.nextPurge()).isNull();
|
||||
assertThat(data.exportAgeHours()).isNull();
|
||||
assertThat(data.warnings()).containsExactly("No signing key configured", "No trusted public keys configured", "Daily verification has never run");
|
||||
String html = WormAuditHomeView.render(data, "/app");
|
||||
assertThat(html).contains("Chain status", "Export status", "Retention", "Recent purges", "Setup warnings",
|
||||
"No links chained yet", "No verification has run yet", "Unknown until verification runs", "No sealed exports yet",
|
||||
"No sealed ranges to check", "No live audits", "No purge requests yet");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Not verified yet");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Strict settle boundary and AUDIT anti-join, including late-detail links.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void chainBacklogRetentionAndRealFailureRecovery() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed();
|
||||
init(config);
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
Instant now = OLD.plusSeconds(120);
|
||||
new WormAuditChainer(config).chainOnce(now);
|
||||
audit(2);
|
||||
audit(3);
|
||||
tamper("audit", 3, "timestamp", now.minusSeconds(60));
|
||||
var data = WormAuditHomeData.load(config, new MemorySealedStore(), now);
|
||||
assertThat(data.links()).isEqualTo(1);
|
||||
assertThat(data.lastLinkAt()).isEqualTo(now);
|
||||
assertThat(data.backlog()).isEqualTo(1);
|
||||
assertThat(data.settling()).isEqualTo(1);
|
||||
assertThat(data.oldestAudit()).isEqualTo(OLD.plusSeconds(1));
|
||||
assertThat(data.nextPurge()).isEqualTo(OLD.plusSeconds(1).atZone(ZoneOffset.UTC).plusYears(6).toInstant());
|
||||
new WormAuditChainer(config).chainOnce(now.plusSeconds(61));
|
||||
tamper("auditDetail", 1, "newValue", "forced mismatch");
|
||||
var failure = new WormAuditVerifier(config).verify(null, now.plusSeconds(61));
|
||||
QRecord failed = VerificationRuns.save(failure, "SCHEDULE", null, now.plusSeconds(62));
|
||||
assertThat(failure.passed()).isFalse();
|
||||
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(63));
|
||||
assertThat(data.openBreak().getValueLong("id")).isEqualTo(failed.getValueLong("id"));
|
||||
String html = WormAuditHomeView.render(data, "/app/admin/worm");
|
||||
assertThat(html).contains("/app/admin/worm/worm_audit_verification/1", "/app/admin/worm/worm_audit_chain/1", "content differs from the chained hash");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("Chain breaks reported");
|
||||
// A passing selected run must not clear a full-chain failure.
|
||||
VerificationRuns.save(new WormAuditVerifier(config).verify(Set.of(2L), now.plusSeconds(64)), "USER", null, now.plusSeconds(65));
|
||||
assertThat(WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(66)).openBreak()).isNotNull();
|
||||
tamper("auditDetail", 1, "newValue", "new");
|
||||
VerificationRuns.save(new WormAuditVerifier(config).verify(null, now.plusSeconds(67)), "SCHEDULE", null, now.plusSeconds(68));
|
||||
data = WormAuditHomeData.load(config, new MemorySealedStore(), now.plusSeconds(69));
|
||||
assertThat(data.openBreak()).isNull();
|
||||
assertThat(data.warnings()).isEmpty();
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).doesNotContain("Setup warnings");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(data, "/app/worm_audit")).contains("No open chain breaks reported");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Genuine exports, trailing lag and a corruption fixture for a coverage hole.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sealedRangesLagGapsAndPurges() throws Exception
|
||||
{
|
||||
var storage = new MemorySealedStore();
|
||||
var config = ChainVerificationTest.signed().withSealedStoreSupplier(() -> storage);
|
||||
init(config);
|
||||
for(long id = 1; id <= 4; id++)
|
||||
{
|
||||
audit(id);
|
||||
}
|
||||
Instant now = OLD.plusSeconds(180);
|
||||
new WormAuditChainer(config).chainOnce(now);
|
||||
var export = new SealedExporter(config).run("USER", now);
|
||||
assertThat(export.failed()).isFalse();
|
||||
assertThat(new SealedExportVerifier(config).verify(export.row(), "USER").result().passed()).isTrue();
|
||||
audit(5);
|
||||
new WormAuditChainer(config).chainOnce(now.plusSeconds(120));
|
||||
var data = WormAuditHomeData.load(config, null, now.plus(Duration.ofHours(3)));
|
||||
assertThat(data.exportLag()).isEqualTo(1);
|
||||
assertThat(data.exportAgeHours()).isEqualTo(3);
|
||||
assertThat(data.exportGaps()).isEmpty();
|
||||
assertThat(data.lockCapability()).isEqualTo("SUPPORTED");
|
||||
assertThat(data.bucketAlias()).isEqualTo("worm_audit");
|
||||
assertThat(data.lastExport().getValueString("lock_mode")).isEqualTo("COMPLIANCE");
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).contains("/app/worm_audit_export/1", "COMPLIANCE", "Retain until");
|
||||
// Corruption fixture only: the real exporter never creates this hole.
|
||||
tamper(WormAuditTables.EXPORT, 1, "from_link_id", 3L);
|
||||
assertThat(WormAuditHomeData.load(config, storage, now).exportGaps()).containsExactly("1–2");
|
||||
for(long id = 1; id <= 12; id++)
|
||||
{
|
||||
STORE.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("id", id).withValue("requested_at", now.plusSeconds(id))
|
||||
.withValue("requested_by", "<operator>").withValue("cutoff", OLD).withValue("outcome", id == 1 ? "PURGED" : "REFUSED")
|
||||
.withValue("reason", "Retention not elapsed").withValue("anchor_link_id", 2L).withValue("anchor_hash", "abcd".repeat(16))
|
||||
.withValue("audits_deleted", 0L).withValue("details_deleted", 0L).withValue("links_deleted", 0L)));
|
||||
}
|
||||
data = WormAuditHomeData.load(config, storage, now.plusSeconds(20));
|
||||
assertThat(data.anchor().getValueLong("anchor_link_id")).isEqualTo(2);
|
||||
assertThat(data.recentPurges()).hasSize(10).extracting(r -> r.getValueLong("id")).containsExactly(12L, 11L, 10L, 9L, 8L, 7L, 6L, 5L, 4L, 3L);
|
||||
assertThat(data.exportGaps()).isEmpty();
|
||||
assertThat(WormAuditHomeView.render(data, "/app")).contains("abcdabcdabcd…", "<operator>", "REFUSED").doesNotContain("abcd".repeat(16));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Unknown/unsupported capability and stale schedule diagnostics stay value-free.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void capabilityAndDailyWarnings() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed();
|
||||
init(config);
|
||||
var unsupported = new MemorySealedStore()
|
||||
{
|
||||
@Override
|
||||
public boolean supportsLock()
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
};
|
||||
var unknown = new MemorySealedStore()
|
||||
{
|
||||
@Override
|
||||
public CapabilityStatus lockCapability()
|
||||
{
|
||||
return (CapabilityStatus.UNKNOWN);
|
||||
}
|
||||
};
|
||||
var error = new MemorySealedStore()
|
||||
{
|
||||
@Override
|
||||
public boolean supportsLock()
|
||||
{
|
||||
throw (new IllegalStateException("CANARY-provider-secret"));
|
||||
}
|
||||
};
|
||||
assertThat(WormAuditHomeData.load(config, unsupported, OLD).lockCapability()).isEqualTo("UNSUPPORTED");
|
||||
assertThat(WormAuditHomeData.load(config, unknown, OLD).lockCapability()).isEqualTo("UNKNOWN");
|
||||
String html = WormAuditHomeView.render(WormAuditHomeData.load(config, error, OLD), "/app");
|
||||
assertThat(html).contains("UNKNOWN", "Object Lock capability check failed").doesNotContain("CANARY", "provider-secret");
|
||||
VerificationRuns.save(new VerificationResult(OLD), "SCHEDULE", null, OLD);
|
||||
assertThat(WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofHours(26))).warnings()).doesNotContain("Daily verification last ran more than 26 hours ago");
|
||||
var stale = WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofHours(26)).plusSeconds(1));
|
||||
assertThat(stale.warnings()).contains("Daily verification last ran more than 26 hours ago");
|
||||
assertThat(WormAuditHomeView.renderHealthCard(stale, "/app/worm_audit")).contains("Setup needs attention");
|
||||
config.withTrustedPublicKeys(Map.of());
|
||||
assertThat(WormAuditHomeData.load(config, unsupported, OLD).warnings()).contains("No trusted public keys configured");
|
||||
// USER and SEALED_EXPORT runs cannot refresh the daily CHAIN heartbeat.
|
||||
VerificationRuns.save(new VerificationResult(OLD.plus(Duration.ofDays(2))), "USER", null, OLD.plus(Duration.ofDays(2)));
|
||||
VerificationRuns.save(new VerificationResult(OLD.plus(Duration.ofDays(2))), "SCHEDULE", 1L, OLD.plus(Duration.ofDays(2)));
|
||||
assertThat(WormAuditHomeData.load(config, unsupported, OLD.plus(Duration.ofDays(2))).warnings()).contains("Daily verification last ran more than 26 hours ago");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** No portion of raw audit content belongs in either HTML widget.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void canaryNeverAppearsAndWidgetsRequireReadPermission() throws Exception
|
||||
{
|
||||
var config = ChainVerificationTest.signed().withAppName("evidence").withAppLabel("Evidence").withTablePathPrefix("/app");
|
||||
var instance = init(config);
|
||||
instance.addApp(new QAppMetaData().withName("admin").withLabel("Admin"));
|
||||
config.withParentAppName("admin");
|
||||
instance.getApp("evidence").setParentAppName("admin");
|
||||
String canary = "CANARY-PHI-" + UUID.randomUUID();
|
||||
audit(1);
|
||||
detail(1, 1);
|
||||
tamper("audit", 1, "message", canary);
|
||||
tamper("auditDetail", 1, "message", canary);
|
||||
tamper("auditDetail", 1, "oldValue", canary);
|
||||
tamper("auditDetail", 1, "newValue", canary);
|
||||
new WormAuditChainer(config).chainOnce();
|
||||
var input = new RenderWidgetInput().withWidgetMetaData(instance.getWidget(WormAuditApp.HOME_WIDGET));
|
||||
assertThatThrownBy(() -> new WormAuditApp.HomeRenderer().render(input)).isInstanceOf(QPermissionDeniedException.class);
|
||||
assertThatThrownBy(() -> new WormAuditApp.HealthCardRenderer().render(input)).isInstanceOf(QPermissionDeniedException.class);
|
||||
QContext.getQSession().withPermissions(WormAuditTables.PURGE + ".read");
|
||||
String home = ((RawHTML) new WormAuditApp.HomeRenderer().render(input).getWidgetData()).getHtml();
|
||||
String card = ((RawHTML) new WormAuditApp.HealthCardRenderer().render(input).getWidgetData()).getHtml();
|
||||
for(int start = 0; start <= canary.length() - 5; start++)
|
||||
{
|
||||
assertThat(home + card).doesNotContain(canary.substring(start, start + 5));
|
||||
}
|
||||
assertThat(home + card).contains("overflow-wrap:anywhere", "color:inherit").doesNotContain("overflow-x", "<table", "#fff");
|
||||
assertThat(card).contains("href=\"/app/admin/evidence\"");
|
||||
assertThat(instance.getApp("evidence").getLabel()).isEqualTo("Evidence");
|
||||
assertThat(instance.getApp("evidence").getParentAppName()).isEqualTo("admin");
|
||||
assertThat(instance.getApp("evidence").getWidgets()).containsExactly(WormAuditApp.HOME_WIDGET);
|
||||
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getName()).containsExactlyElementsOf(WormAuditTables.NAV);
|
||||
assertThat(instance.getApp("evidence").getChildren()).extracting(c -> c.getLabel())
|
||||
.containsExactly("Verifications", "Sealed Exports", "Checkpoints", "Audit Chain", "Purges");
|
||||
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getGridColumns()).isEqualTo(12);
|
||||
assertThat(instance.getWidget(WormAuditApp.HOME_WIDGET).getIsCard()).isFalse();
|
||||
assertThat(instance.getWidget(WormAuditApp.HEALTH_CARD_WIDGET).getIsCard()).isTrue();
|
||||
new QInstanceValidator().validate(instance);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise the real SQL anti-join without Docker, including its exact boundary.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void sqlSummaryUsesTheWriterAntiJoin() throws Exception
|
||||
{
|
||||
String url = "jdbc:h2:mem:home_" + UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=TRUE;INIT=SET TIME ZONE 'UTC'";
|
||||
var config = ChainVerificationTest.signed();
|
||||
var instance = RdbmsTestSupport.instance(url, "sa", "", config, false);
|
||||
try(var connection = DriverManager.getConnection(url, "sa", ""))
|
||||
{
|
||||
RdbmsTestSupport.createTables(connection, instance, false);
|
||||
QContext.init(instance, new QSession());
|
||||
lookups();
|
||||
audit(1);
|
||||
new WormAuditChainer(config).chainOnce(OLD.plusSeconds(100));
|
||||
audit(2);
|
||||
audit(3);
|
||||
var data = WormAuditHomeData.load(config, new MemorySealedStore(), OLD.plusSeconds(63));
|
||||
assertThat(data.links()).isEqualTo(1);
|
||||
assertThat(data.backlog()).isEqualTo(1);
|
||||
assertThat(data.settling()).isEqualTo(1);
|
||||
assertThat(data.warnings()).containsExactly("Daily verification has never run");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The demo's history comes from real engines, including FAIL, recovery and PHI tokens.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void demoSeedsRealHistory() throws Exception
|
||||
{
|
||||
MemoryRecordStore.fullReset();
|
||||
var application = new WormAuditDemoServer();
|
||||
var instance = application.defineQInstance();
|
||||
new QInstanceValidator().validate(instance);
|
||||
application.seed(instance);
|
||||
QContext.init(instance, new QSession());
|
||||
var data = WormAuditHomeData.load();
|
||||
assertThat(data.links()).isGreaterThanOrEqualTo(350);
|
||||
assertThat(data.lastVerification().getValueString("outcome")).isEqualTo("PASS");
|
||||
assertThat(data.openBreak()).isNull();
|
||||
assertThat(data.exportGaps()).isEmpty();
|
||||
assertThat(data.exportLag()).isEqualTo(5);
|
||||
assertThat(data.backlog()).isEqualTo(2);
|
||||
assertThat(data.settling()).isEqualTo(1);
|
||||
assertThat(data.warnings()).isEmpty();
|
||||
assertThat(data.recentPurges()).extracting(r -> r.getValueString("outcome")).containsExactly("REFUSED", "PURGED");
|
||||
assertThat(STORE.where(WormAuditTables.VERIFICATION, "outcome", "FAIL")).hasSize(1).allSatisfy(row ->
|
||||
{
|
||||
assertThat(row.getValueLong("first_break_link_id")).isPositive();
|
||||
assertThat(row.getValueLong("first_break_audit_id")).isPositive();
|
||||
});
|
||||
assertThat(STORE.all("auditDetail")).filteredOn(r -> "diagnosis".equals(r.getValueString("fieldName")))
|
||||
.isNotEmpty().allSatisfy(row -> assertThat(row.getValueString("newValue")).startsWith("hmac:v1:test:"));
|
||||
var config = VerificationRuns.config();
|
||||
// Settle and chain the final backlog before checking the entire live suffix.
|
||||
new WormAuditChainer(config).chainOnce(Instant.now().plusSeconds(120));
|
||||
assertThat(new WormAuditVerifier(config).verify().passed()).isTrue();
|
||||
assertThat(new SealedExportVerifier(config).verify(STORE.last(WormAuditTables.EXPORT), "USER").result().passed()).isTrue();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.demo;
|
||||
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Instant;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.objectstorage.api.ListRequest;
|
||||
import com.kingsrook.qbits.objectstorage.api.ObjectSummary;
|
||||
import com.kingsrook.qbits.objectstorage.api.Page;
|
||||
import com.kingsrook.qbits.objectstorage.api.StoredObject;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedStore;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Tiny test-scope immutable store with version and COMPLIANCE retention echoes.
|
||||
*******************************************************************************/
|
||||
public class MemorySealedStore implements SealedStore
|
||||
{
|
||||
private final Map<String, byte[]> objects = new LinkedHashMap<>();
|
||||
private final Map<String, StoredObject> heads = new LinkedHashMap<>();
|
||||
|
||||
/*******************************************************************************
|
||||
** Demo capability.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean supportsLock()
|
||||
{
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Refuse overwrites and retain the exact bytes.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String put(String key, Path file, Instant until) throws Exception
|
||||
{
|
||||
if(objects.containsKey(key))
|
||||
{
|
||||
throw (new IllegalStateException("Demo sealed object already exists"));
|
||||
}
|
||||
String version = "v" + (objects.size() + 1);
|
||||
objects.put(key, Files.readAllBytes(file));
|
||||
heads.put(key, new StoredObject().withKey(key).withVersionId(version).withLockMode("COMPLIANCE").withLockRetainUntil(until));
|
||||
return (version);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exact retention metadata.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public StoredObject head(String key)
|
||||
{
|
||||
return (heads.get(key));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exact version read.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public InputStream getVersion(String key, String versionId)
|
||||
{
|
||||
if(!heads.containsKey(key) || !heads.get(key).getVersionId().equals(versionId))
|
||||
{
|
||||
throw (new IllegalArgumentException("Demo sealed version is missing"));
|
||||
}
|
||||
return (new ByteArrayInputStream(objects.get(key)));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The small demo fits one page.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public Page<ObjectSummary> list(ListRequest request)
|
||||
{
|
||||
return (new Page<ObjectSummary>().withItems(objects.keySet().stream().filter(k -> k.startsWith(request.getPrefix()))
|
||||
.map(k -> new ObjectSummary().withKey(k)).toList()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,286 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.demo;
|
||||
|
||||
|
||||
import java.security.KeyPairGenerator;
|
||||
import java.time.Clock;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.time.ZoneOffset;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.wormaudit.PhiCanaryTest;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitProducer;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.chain.WormAuditChainer;
|
||||
import com.kingsrook.qbits.wormaudit.export.SealedExporter;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qbits.wormaudit.processes.VerificationRuns;
|
||||
import com.kingsrook.qbits.wormaudit.verify.WormAuditVerifier;
|
||||
import com.kingsrook.qqq.backend.core.actions.audits.AuditActionCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
|
||||
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.audits.AuditSingleInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
|
||||
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Local demo: synthetic notes, test HMAC tokens, real chain, checkpoints,
|
||||
** verifications and sealed exports over a moving clock. Memory storage and its
|
||||
** simulated purge are not production enforcement.
|
||||
**
|
||||
** <pre>
|
||||
** ./mvnw -o test-compile exec:java -Dexec.classpathScope=test -Dexec.mainClass=com.kingsrook.qbits.wormaudit.demo.WormAuditDemoServer -Dexec.args=8099
|
||||
** </pre>
|
||||
*******************************************************************************/
|
||||
public class WormAuditDemoServer extends AbstractQQQApplication
|
||||
{
|
||||
private static Clock clock = Clock.systemUTC();
|
||||
private final MemorySealedStore storage = new MemorySealedStore();
|
||||
private WormAuditQBitConfig config;
|
||||
|
||||
/*******************************************************************************
|
||||
** The PhiCanaryTest stand-in, with a moving timestamp before audit insertion.
|
||||
*******************************************************************************/
|
||||
public static class DemoTokenHook extends PhiCanaryTest.TokenHook
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Tokenize through the same core customization point as the canary test.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void customizeInput(AuditSingleInput input)
|
||||
{
|
||||
input.withTimestamp(clock.instant());
|
||||
super.customizeInput(input);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fully anonymous demo with a FIELD-audited synthetic note table.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public QInstance defineQInstance() throws QException
|
||||
{
|
||||
try
|
||||
{
|
||||
var keys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair();
|
||||
config = new WormAuditQBitConfig().withBackendName("memory").withCheckpointEveryLinks(20).withCheckpointEveryMinutes(1)
|
||||
.withSigningKey(() -> new WormAuditQBitConfig.SigningKey("demo", keys.getPrivate())).withTrustedPublicKeys(Map.of("demo", keys.getPublic()))
|
||||
.withSealedStoreSupplier(() -> storage)
|
||||
// Synthetic local demo only: anonymous readers can view its tables.
|
||||
.withTableMetaDataCustomizer((instance, table) -> table.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED)));
|
||||
QInstance instance = new QInstance().withInstanceDefaultAuthentication(new QAuthenticationMetaData()
|
||||
.withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
|
||||
instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
|
||||
new AuditsMetaDataProvider().defineAll(instance, "memory", null);
|
||||
instance.addTable(new QTableMetaData().withName("patient_note").withLabel("Synthetic patient note").withBackendName("memory")
|
||||
.withPrimaryKeyField("id").withRecordLabelFields("id").withRecordLabelFormat("Note #%s")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
|
||||
.withField(new QFieldMetaData("diagnosis", QFieldType.STRING).withLabel("Diagnosis").withSupplementalMetaData(new PhiCanaryTest.Phi()))
|
||||
.withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)));
|
||||
instance.addSupplementalCustomizer(AuditActionCustomizerInterface.CUSTOMIZER_TYPE, new QCodeReference(DemoTokenHook.class));
|
||||
new WormAuditQBitProducer().withQBitConfig(config).produce(instance);
|
||||
instance.setHomeAppName(config.getAppName());
|
||||
return (instance);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new QException("Cannot define WORM Audit demo", e));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Write through core DML and the test token hook at the current demo time.
|
||||
*******************************************************************************/
|
||||
private void notes(int count) throws QException
|
||||
{
|
||||
for(int i = 0; i < count; i++)
|
||||
{
|
||||
new InsertAction().execute(new InsertInput("patient_note").withRecord(new QRecord().withValue("diagnosis", "SYNTHETIC-DEMO-NOTE-" + i)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Persist a real verifier result with the historical run time.
|
||||
*******************************************************************************/
|
||||
private void verify() throws QException
|
||||
{
|
||||
var result = new WormAuditVerifier(config).verify(null, clock.instant());
|
||||
VerificationRuns.save(result, "SCHEDULE", null, clock.instant());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Use the production exporter and configured storage boundary.
|
||||
*******************************************************************************/
|
||||
private void seal() throws QException
|
||||
{
|
||||
var result = new SealedExporter(config).run("SCHEDULE", clock.instant());
|
||||
if(result.failed() || result.row() == null)
|
||||
{
|
||||
throw (new QException("Demo seeding failed to seal its checkpointed range"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A fortnight of activity and a genuinely expired, sealed prefix.
|
||||
*******************************************************************************/
|
||||
public void seed(QInstance instance) throws Exception
|
||||
{
|
||||
QContext.init(instance, new QSession());
|
||||
AuditStore store = new AuditStore(null);
|
||||
Instant today = Instant.now().minus(Duration.ofHours(2));
|
||||
try
|
||||
{
|
||||
Instant expired = today.atZone(ZoneOffset.UTC).minusYears(7).toInstant();
|
||||
clock = Clock.fixed(expired, ZoneOffset.UTC);
|
||||
notes(3);
|
||||
clock = Clock.fixed(expired.plusSeconds(120), ZoneOffset.UTC);
|
||||
new WormAuditChainer(config).chainOnce(clock.instant());
|
||||
clock = Clock.fixed(expired.plusSeconds(240), ZoneOffset.UTC);
|
||||
new WormAuditChainer(config).chainOnce(clock.instant());
|
||||
verify();
|
||||
seal();
|
||||
QRecord anchor = store.last(WormAuditTables.CHAIN);
|
||||
long auditCount = store.count("audit", new QQueryFilter());
|
||||
long detailCount = store.count("auditDetail", new QQueryFilter());
|
||||
// Demo data: memory has no purge function. Remove the expired, sealed
|
||||
// prefix and append the receipt the PostgreSQL function would write.
|
||||
for(String table : List.of("auditDetail", "audit", WormAuditTables.CHAIN))
|
||||
{
|
||||
for(QRecord row : store.all(table))
|
||||
{
|
||||
MemoryRecordStore.getInstance().delete(new DeleteInput(table).withPrimaryKeys(List.of(row.getValue("id"))));
|
||||
}
|
||||
}
|
||||
store.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("requested_at", today.minus(Duration.ofDays(15)))
|
||||
.withValue("requested_by", "demo-purger").withValue("cutoff", expired.plusSeconds(1)).withValue("retention_interval", "6 years")
|
||||
.withValue("outcome", "PURGED").withValue("reason", "Expired sealed prefix removed (demo memory purge)")
|
||||
.withValue("audits_deleted", auditCount).withValue("details_deleted", detailCount).withValue("links_deleted", anchor.getValueLong("id"))
|
||||
.withValue("anchor_link_id", anchor.getValueLong("id")).withValue("anchor_hash", anchor.getValueString("link_hash"))));
|
||||
for(int day = 13; day >= 0; day--)
|
||||
{
|
||||
Instant time = today.minus(Duration.ofDays(day));
|
||||
clock = Clock.fixed(time, ZoneOffset.UTC);
|
||||
notes(25);
|
||||
clock = Clock.fixed(time.plusSeconds(120), ZoneOffset.UTC);
|
||||
new WormAuditChainer(config).chainOnce(clock.instant());
|
||||
clock = Clock.fixed(time.plusSeconds(240), ZoneOffset.UTC);
|
||||
new WormAuditChainer(config).chainOnce(clock.instant());
|
||||
if(day == 7)
|
||||
{
|
||||
// Demo seeding only: temporarily tamper a memory detail, persist a
|
||||
// genuine FAIL naming that link, then restore before later PASSes.
|
||||
QRecord detail = store.all("auditDetail").getFirst();
|
||||
var original = detail.getValue("newValue");
|
||||
memoryValue(detail, "hmac:v1:demo:tampered");
|
||||
verify();
|
||||
memoryValue(detail, original);
|
||||
}
|
||||
else
|
||||
{
|
||||
verify();
|
||||
}
|
||||
seal();
|
||||
}
|
||||
// Demo REFUSED receipt: too-recent cutoff; no rows removed.
|
||||
store.insert(WormAuditTables.PURGE, List.of(new QRecord().withValue("requested_at", today.plusSeconds(300))
|
||||
.withValue("requested_by", "demo-purger").withValue("cutoff", today).withValue("retention_interval", "6 years")
|
||||
.withValue("outcome", "REFUSED").withValue("reason", "Cutoff is inside the retention interval")
|
||||
.withValue("audits_deleted", 0L).withValue("details_deleted", 0L).withValue("links_deleted", 0L)));
|
||||
// Real unsealed suffix and unchained backlog; no invented export gap.
|
||||
clock = Clock.fixed(today.plusSeconds(600), ZoneOffset.UTC);
|
||||
notes(5);
|
||||
new WormAuditChainer(config).chainOnce(today.plusSeconds(720));
|
||||
clock = Clock.fixed(Instant.now().minusSeconds(120), ZoneOffset.UTC);
|
||||
notes(2);
|
||||
clock = Clock.systemUTC();
|
||||
notes(1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
clock = Clock.systemUTC();
|
||||
QContext.clear();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Deliberate memory-only tamper used solely while seeding the FAIL receipt.
|
||||
*******************************************************************************/
|
||||
private static void memoryValue(QRecord detail, java.io.Serializable value) throws QException
|
||||
{
|
||||
MemoryRecordStore.getInstance().update(new UpdateInput("auditDetail").withRecords(List.of(new QRecord()
|
||||
.withValue("id", detail.getValue("id")).withValue("newValue", value))), false);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Serve the standard widget API and packaged Next UI, reusing the seeded keys.
|
||||
*******************************************************************************/
|
||||
public static void main(String[] args) throws Exception
|
||||
{
|
||||
int port = args.length == 0 ? 8099 : Integer.parseInt(args[0]);
|
||||
WormAuditDemoServer application = new WormAuditDemoServer();
|
||||
QInstance instance = application.defineQInstance();
|
||||
new QInstanceValidator().validate(instance);
|
||||
application.seed(instance);
|
||||
new QApplicationJavalinServer(new AbstractQQQApplication()
|
||||
{
|
||||
@Override
|
||||
public QInstance defineQInstance()
|
||||
{
|
||||
return (instance);
|
||||
}
|
||||
}).withPort(port).withServeFrontendNext(true).start();
|
||||
System.out.println("WORM Audit demo on http://localhost:" + port + "/app/worm_audit");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,272 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.wormaudit.verify;
|
||||
|
||||
|
||||
import java.security.KeyPairGenerator;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.wormaudit.WormAuditQBitConfig;
|
||||
import com.kingsrook.qbits.wormaudit.canonical.AuditCanonical;
|
||||
import com.kingsrook.qbits.wormaudit.chain.AuditStore;
|
||||
import com.kingsrook.qbits.wormaudit.chain.Hashes;
|
||||
import com.kingsrook.qbits.wormaudit.checkpoint.Checkpoints;
|
||||
import com.kingsrook.qbits.wormaudit.model.WormAuditTables;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.params.ParameterizedTest;
|
||||
import org.junit.jupiter.params.provider.ValueSource;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Instrument the storage boundary independently of the memory backend: no
|
||||
** unbounded reads are allowed, including portable completeness checks.
|
||||
*******************************************************************************/
|
||||
class PagedVerificationTest
|
||||
{
|
||||
private static final int LINKS = 25000;
|
||||
private static final int PAGE = 137;
|
||||
private static final int RANGE = 997;
|
||||
private static final Instant NOW = Instant.parse("2026-10-13T00:00:00Z");
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cross many non-aligned page/checkpoint boundaries with genuine signatures.
|
||||
*******************************************************************************/
|
||||
@ParameterizedTest
|
||||
@ValueSource(booleans = { true, false })
|
||||
void fullRunBoundsPagesAndCheckpointRanges(boolean sql) throws Exception
|
||||
{
|
||||
var keys = KeyPairGenerator.getInstance("Ed25519").generateKeyPair();
|
||||
var config = new WormAuditQBitConfig().withCheckpointEveryLinks(RANGE)
|
||||
.withSigningKey(() -> new WormAuditQBitConfig.SigningKey("test", keys.getPrivate()))
|
||||
.withTrustedPublicKeys(Map.of("test", keys.getPublic()));
|
||||
var store = new InstrumentedStore(config, sql);
|
||||
var result = new WormAuditVerifier(config, PAGE).verify(store, null, NOW);
|
||||
assertThat(result.getBreaks()).isEmpty();
|
||||
assertThat(result.getLinksChecked()).isEqualTo(LINKS);
|
||||
assertThat(result.getFromLinkId()).isEqualTo(1);
|
||||
assertThat(result.getToLinkId()).isEqualTo(LINKS);
|
||||
assertThat(store.maxPage).isEqualTo(PAGE);
|
||||
assertThat(store.maxRange).isEqualTo(RANGE);
|
||||
assertThat(store.checkpointsRead).isEqualTo(26);
|
||||
assertThat(store.completenessVisited).isTrue();
|
||||
if(!sql)
|
||||
{
|
||||
store.links.get(0).withValue("detail_ids", "1,1,2");
|
||||
store.links.get(1).withValue("detail_ids", "2");
|
||||
store.links.get(2).withValue("detail_ids", "malformed");
|
||||
store.links.get(3).withValue("audit_id", 1L);
|
||||
store.audits.get(2).withValue("timestamp", null);
|
||||
store.audits.get(4).withValue("timestamp", NOW);
|
||||
store.details.addAll(List.of(new QRecord().withValue("id", 1L).withValue("auditId", 1L),
|
||||
new QRecord().withValue("id", 2L).withValue("auditId", 2L),
|
||||
new QRecord().withValue("id", 3L).withValue("auditId", 99999L),
|
||||
new QRecord().withValue("id", 4L).withValue("auditId", 3L)));
|
||||
var broken = new WormAuditVerifier(config, PAGE).verify(store, null, NOW);
|
||||
assertThat(broken.getBreaks()).extracting(finding -> finding.reason()).contains(
|
||||
"audit 1 is covered twice or more by the chain", "audit 4 is not covered by the chain",
|
||||
"audit 3 has no timestamp", "audit_detail 1 is covered twice or more by the chain",
|
||||
"audit_detail 2 is covered twice or more by the chain", "audit_detail 3 is not covered by the chain",
|
||||
"audit_detail 4 is not covered by the chain");
|
||||
}
|
||||
assertThatThrownBy(() -> new WormAuditVerifier(config, 0)).isInstanceOf(IllegalArgumentException.class);
|
||||
assertThatThrownBy(() -> new WormAuditVerifier(config, 1001)).isInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixture storage may retain its data; the verifier can only fetch bounded
|
||||
** pages. Track the total hashes read between successive checkpoint requests.
|
||||
*******************************************************************************/
|
||||
private static class InstrumentedStore extends AuditStore
|
||||
{
|
||||
private final List<QRecord> audits = new ArrayList<>();
|
||||
private final List<QRecord> details = new ArrayList<>();
|
||||
private final List<QRecord> links = new ArrayList<>();
|
||||
private final List<QRecord> checkpoints = new ArrayList<>();
|
||||
private final boolean sql;
|
||||
private int maxPage;
|
||||
private int maxRange;
|
||||
private int rangeRead;
|
||||
private int checkpointsRead;
|
||||
private boolean inCheckpoints;
|
||||
private boolean completenessVisited;
|
||||
private boolean mainScanFinished;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Generate signed, valid content without involving the backend under test.
|
||||
*******************************************************************************/
|
||||
InstrumentedStore(WormAuditQBitConfig config, boolean sql) throws Exception
|
||||
{
|
||||
super(null);
|
||||
this.sql = sql;
|
||||
String previous = Hashes.GENESIS;
|
||||
for(long id = 1; id <= LINKS; id++)
|
||||
{
|
||||
QRecord audit = new QRecord().withValue("id", id).withValue("timestamp", NOW.minusSeconds(120))
|
||||
.withValue("auditTableId", 1).withValue("auditUserId", 1).withValue("recordId", id);
|
||||
audits.add(audit);
|
||||
String hash = Hashes.link(previous, AuditCanonical.bytes(audit, "fixture", "fixture", List.of()));
|
||||
links.add(new QRecord().withValue("id", id).withValue("audit_id", id).withValue("kind", "AUDIT")
|
||||
.withValue("detail_ids", "").withValue("prev_hash", previous).withValue("link_hash", hash).withValue("chained_at", NOW));
|
||||
previous = hash;
|
||||
}
|
||||
for(int start = 0; start < LINKS; start += RANGE)
|
||||
{
|
||||
checkpoints.add(Checkpoints.sign(links.subList(start, Math.min(start + RANGE, LINKS)), config, NOW)
|
||||
.withValue("id", (long) checkpoints.size() + 1));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** No full-table read is ever permitted through this boundary.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> all(String table)
|
||||
{
|
||||
throw (new AssertionError("Unbounded verifier read: " + table));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A fresh copied page prevents subList from retaining the fixture chain.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> page(String table, long after, int limit)
|
||||
{
|
||||
assertThat(limit).isBetween(1, PAGE);
|
||||
maxPage = Math.max(maxPage, limit);
|
||||
List<QRecord> source;
|
||||
if(table.equals(WormAuditTables.CHECKPOINT))
|
||||
{
|
||||
assertThat(mainScanFinished).isTrue();
|
||||
assertThat(limit).isEqualTo(1);
|
||||
maxRange = Math.max(maxRange, rangeRead);
|
||||
assertThat(rangeRead).isLessThanOrEqualTo(RANGE);
|
||||
rangeRead = 0;
|
||||
inCheckpoints = after < checkpoints.size();
|
||||
if(inCheckpoints)
|
||||
{
|
||||
checkpointsRead++;
|
||||
}
|
||||
source = checkpoints;
|
||||
}
|
||||
else if(table.equals(WormAuditTables.CHAIN))
|
||||
{
|
||||
source = links;
|
||||
if(after == LINKS && !inCheckpoints)
|
||||
{
|
||||
mainScanFinished = true;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
completenessVisited = true;
|
||||
source = table.equals("audit") ? audits : details;
|
||||
}
|
||||
int start = (int) Math.min(after, source.size());
|
||||
var page = new ArrayList<>(source.subList(start, Math.min(start + limit, source.size())));
|
||||
if(inCheckpoints && table.equals(WormAuditTables.CHAIN))
|
||||
{
|
||||
rangeRead += page.size();
|
||||
}
|
||||
return (page);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Batch only the audit headers belonging to the current chain page.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> in(String table, String field, Collection<Long> ids)
|
||||
{
|
||||
assertThat(ids.size()).isLessThanOrEqualTo(PAGE);
|
||||
return (ids.stream().distinct().map(id -> (table.equals("audit") ? audits : details).get(id.intValue() - 1)).toList());
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve detail parents without collecting audit history.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public QRecord get(String table, long id)
|
||||
{
|
||||
return (id > audits.size() ? null : audits.get((int) id - 1));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable lookup values match the fixture canonical documents.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public String name(String table, Long id)
|
||||
{
|
||||
return ("fixture");
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Start at genesis.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public QRecord anchor()
|
||||
{
|
||||
return (new QRecord().withValue("anchor_link_id", 0L).withValue("anchor_hash", Hashes.GENESIS));
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise both SQL dispatch and portable non-memory completeness.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean isRdbms()
|
||||
{
|
||||
return (sql);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Never permit the memory fallback in this test.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean isMemory()
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** SQL completeness has separate acceptance tests; verify dispatch here.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void completeness(Instant cutoff, VerificationResult result)
|
||||
{
|
||||
completenessVisited = true;
|
||||
}
|
||||
}
|
||||
}
|
||||