feat(verify): backup verification, SLA checker, alerts, health, events #5

2026-10-12T21:35:53Z

KofTwentyTwo commented

2026-10-12 21:35:53 +00:00

(Migrated from github.com)

Spec sections 10.1, 10.2, 10.4 and 4.5.

Base: this branch contains the foundation commit (PR #4, feature/backups-foundation, not merged yet); the verify work is the single commit d2fcf74 on top, so it rebases cleanly. Merge #4 first, then retarget/rebase this onto develop.

What it does

Verify backup (backupVerify, record action on backups_run, streamed ETL with Review/Result lines). STORAGE (head every object incl. every file-store object in the file index; sizes, provider SHA-256 where present) then CONTENT (decrypt hook via ArtifactTransform.unwrap, gunzip, parse every row with the codec incl. exact round trip, recount, recompute sha256Stored/sha256Raw/rowsHash per table, re-hash every referenced file-store object against its SHA-256 key, bounded by fileReverifyDays using backups_file_object.lastVerifiedDate). TEST_RESTORE through a ScratchTarget interface; MEMORY implementation restores every table into cloned in-memory table metadata, compares counts and rowsHash, restores files to qqq-verify/<runKey>/, re-hashes, drops. RDBMS_SCHEMA is a stub that fails with a clear ERROR (needs the restore writer).
Writes backups_verification rows, updates backups_run.verificationStatus and backups_plan.lastVerified* (CONTENT or deeper).
backupCheckSla (scheduled every slaCheckIntervalSeconds when a scheduler is configured): OK, LATE, FAILING, MISSING_IN_STORAGE (head on manifestKey, positive cached 1h), UNVERIFIED, ABANDONED (RUNNING heartbeat older than 10 min, run marked ABANDONED), OWNER_INVALID (planOwnerValidator hook), UNKNOWN for a brand-new plan. Daily bucket listing for orphans and debris.
Alerts: every alert of the spec table with stable dedupe keys, raised and resolved through BackupAlertSink; backups.plan.missed is raised by the SLA checker itself, independent of any heartbeat.
AlertsQBitAlertSink: reflective, optional adapter for the Alerts producer SPI (QRun-IO/qqq#1108, not in RC.4); no-op without the Alerts qbit; declares the 9 alert types.
BackupsHealthIndicator (optional dep qqq-middleware-health), ESB events (optional dep qqq-esb, fire and forget), BackupsMetrics Prometheus text (cached 60s).

Stand-ins

Alerts SPI bound by reflection (tested against a same-shape test double) until RC.5; replace with direct QAlerts calls.
ScratchTarget.RDBMS_SCHEMA stub (TODO: restore agent's writer, #1111).
/metrics/backups route is a documented stub: needs qqq-middleware-javalin; the app calls BackupsMetrics.renderCached().
Decrypt hook is BackupsQBitConfig.withArtifactTransformResolver(keyRef -> ArtifactTransform); replace with the core key provider (#1107) when it ships.

Small edits outside this area

BackupsQBitConfig (new fields), BackupsProcesses.processes() (one line), BackupsQBitProducer (alerts bind), BackupsTables (plan field verifyFileSamplePercent), BackupEngine.signal (one line: ESB event), pom (two optional deps).

Tests

./mvnw verify: 69 tests, jacoco gates met (80% instruction). Failure injection: deleted manifest, corrupted/truncated/replaced part, deleted file-store object, wrong file checksum, provider checksum mismatch, wrong key, no decrypt hook. SLA states on a synthetic clock; alerts raised and resolved through a recording sink.

🤖 Generated with Claude Code

Spec sections 10.1, 10.2, 10.4 and 4.5. **Base:** this branch contains the foundation commit (PR #4, `feature/backups-foundation`, not merged yet); the verify work is the single commit `d2fcf74` on top, so it rebases cleanly. Merge #4 first, then retarget/rebase this onto `develop`. ## What it does - **Verify backup** (`backupVerify`, record action on `backups_run`, streamed ETL with Review/Result lines). STORAGE (head every object incl. every file-store object in the file index; sizes, provider SHA-256 where present) then CONTENT (decrypt hook via `ArtifactTransform.unwrap`, gunzip, parse every row with the codec incl. exact round trip, recount, recompute sha256Stored/sha256Raw/rowsHash per table, re-hash every referenced file-store object against its SHA-256 key, bounded by `fileReverifyDays` using `backups_file_object.lastVerifiedDate`). TEST_RESTORE through a `ScratchTarget` interface; MEMORY implementation restores every table into cloned in-memory table metadata, compares counts and rowsHash, restores files to `qqq-verify/<runKey>/`, re-hashes, drops. `RDBMS_SCHEMA` is a stub that fails with a clear ERROR (needs the restore writer). - Writes `backups_verification` rows, updates `backups_run.verificationStatus` and `backups_plan.lastVerified*` (CONTENT or deeper). - **backupCheckSla** (scheduled every `slaCheckIntervalSeconds` when a scheduler is configured): OK, LATE, FAILING, MISSING_IN_STORAGE (head on manifestKey, positive cached 1h), UNVERIFIED, ABANDONED (RUNNING heartbeat older than 10 min, run marked ABANDONED), OWNER_INVALID (`planOwnerValidator` hook), UNKNOWN for a brand-new plan. Daily bucket listing for orphans and debris. - **Alerts**: every alert of the spec table with stable dedupe keys, raised and resolved through `BackupAlertSink`; `backups.plan.missed` is raised by the SLA checker itself, independent of any heartbeat. - **AlertsQBitAlertSink**: reflective, optional adapter for the Alerts producer SPI (QRun-IO/qqq#1108, not in RC.4); no-op without the Alerts qbit; declares the 9 alert types. - `BackupsHealthIndicator` (optional dep qqq-middleware-health), ESB events (optional dep qqq-esb, fire and forget), `BackupsMetrics` Prometheus text (cached 60s). ## Stand-ins - Alerts SPI bound by reflection (tested against a same-shape test double) until RC.5; replace with direct `QAlerts` calls. - `ScratchTarget.RDBMS_SCHEMA` stub (TODO: restore agent's writer, #1111). - `/metrics/backups` route is a documented stub: needs qqq-middleware-javalin; the app calls `BackupsMetrics.renderCached()`. - Decrypt hook is `BackupsQBitConfig.withArtifactTransformResolver(keyRef -> ArtifactTransform)`; replace with the core key provider (#1107) when it ships. ## Small edits outside this area `BackupsQBitConfig` (new fields), `BackupsProcesses.processes()` (one line), `BackupsQBitProducer` (alerts bind), `BackupsTables` (plan field `verifyFileSamplePercent`), `BackupEngine.signal` (one line: ESB event), pom (two optional deps). ## Tests `./mvnw verify`: 69 tests, jacoco gates met (80% instruction). Failure injection: deleted manifest, corrupted/truncated/replaced part, deleted file-store object, wrong file checksum, provider checksum mismatch, wrong key, no decrypt hook. SLA states on a synthetic clock; alerts raised and resolved through a recording sink. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-backups#5