feat: run lifecycle - schedule, lease, cancel, resume, GFS prune, object lock, plan validation #7

2026-10-12T21:45:05Z

KofTwentyTwo commented

2026-10-12 21:45:05 +00:00

(Migrated from github.com)

Summary

Run lifecycle for qbit-backups (spec 9.1, 10.3, 6.4/6.5, 11.1, and the Back up these records, Cancel, Resume, Test destination and Prune processes).

Note: foundation PR #4 is not merged yet, so this branch contains its commit (eea9f63). The lifecycle work is the single commit on top (ef3a55a) and rebases cleanly once #4 lands.

Schedule: BackupPlanSyncToScheduledJobProcess (foreignKeyType backups_plan) plus a verify variant (backups_plan_verify), wired on the plan table when BackupsQBitConfig.schedulerName is set. No cron means manual-only (no job); inactive plans have their jobs switched off; deleting a plan deletes them.
Lease and queueing: BackupRunCoordinator. One run per plan (live-heartbeat check in BackupCatalog, QUEUED to RUNNING, a second claim check for cross-pod races); SKIPPED_CONCURRENT leaves a visible CANCELLED run row and a backups.run.skipped_concurrent warning; maxConcurrentRuns queues, drainQueue() starts queued runs after every managed run. Manual and scheduled runs both go through it.
Scheduled entry point: backupRunScheduledPlan (the process the job runs) validates the plan owner, then runs the engine as the scheduler system session (system:scheduler, securityScope FULL). A bad owner raises backups.plan.owner_invalid, sets slaStatus OWNER_INVALID and does not run.
Plan save validation (BackupPlanTableCustomizer / BackupPlanValidator): saving user has backups.admin and read on every resolved table; objectLockMode checked against BackupStorage.capabilities(); INCLUDE_ENCRYPTED_ONLY needs encryption; PARQUET, ZSTD, incremental, native dump and (until a cipher exists) client encryption refused with clear messages; cron validity; planKey is a slug and immutable after the first run; the saving user becomes owner when none is given.
Prune: PruneCalculator (GFS, pure, tested on a synthetic clock) and BackupPruner. Classifies runs into retentionClass, deletes through deletePermanent on the prune alias, reports kept: locked until, deletes debris older than 7 days, never deletes a run a restore used in the last 30 days, always keeps the newest verified run, sets prunedDate instead of deleting rows, and does mark-and-sweep of the content-addressed file store (recomputes referenceCount from the file indexes of retained runs, deletes unreferenced unlocked objects, sets deletedDate; deferred while a run is live or an index cannot be read). Review shows runs to delete, kept by class, locked, debris, CAS objects to sweep and MiB freed.
Object lock at write time: the engine applies WriteOptions.withObjectLock (plan mode and days) to every part, the index, checksums, manifest and file-store objects (a server-side copy cannot lock, so with a lock the bytes are streamed); refuses a lock the bucket does not support.
Processes: <table>.backupExportTable (one per backup-able table), backupCancelRun / backupCancelRestoreRun, backupResumeRun, backupTestDestination, backupPrune (plans) and backupPruneRuns (runs; prune also runs daily when a scheduler is configured).
Resume: parts are now catalogued the moment they are complete. A resume continues the same run: finished tables are kept, the interrupted JSONL table continues after the last key of its last complete part (rows hash rebuilt from the parts), the file index is rebuilt. CSV, a second format and encrypted tables restart that table. Runs that held a snapshot refuse and say so.

Engine edits (minimal)

BackupEngine: existing run id (QUEUED row claimed by the coordinator), resume state, per-part artifact rows, lock options, consistencyAchieved persisted at start, startAfterKey passed to the reader. TableArtifactWriter: part listener, lock, resume seeding. FileCopier: lock. BackupCatalog: live/queued run queries, cancel flag, delete by id. BackupsTables: plan customizers; run and restore status enums gain CANCELLING (cooperative cancel sets it, the engine checks the row each page). BackupsProcesses.processes() / BackupsQBitProducer: one line each.

Stand-ins

None added. Relies on the foundation's TableReader stand-in (core KeysetPager) and on BackupStorage (qbit-object-storage adapter written separately). The ad-hoc export holds the selected records in memory until it runs.

Tests (`./mvnw -o verify` green, 58 tests, jacoco gate met)

PruneCalculatorTest (GFS fixtures, time zones, ISO week across new year, guards), BackupPrunerTest (real runs on a synthetic clock: prune and sweep, retained counts, locked run kept until the lock passes, locked CAS object, debris, restore and verified guards, sweep deferral, prune alias, review writes nothing), BackupRunCoordinatorTest (skip, stale heartbeat, queueing and drain, raised limit, cancel at next page, cancel process, resume after injected failure with identical rows hash/parts/file index, refusals, CSV restart), BackupPlanValidationTest, BackupPlanScheduleTest (jobs follow the plan, scheduled entry point, owner validation), LifecycleProcessesTest (export, test destination incl. head-only on a locked bucket, prune review/result).

Owner steps / interfaces for other work

The verify schedule job runs process backupVerifyPlan (table backups_plan, takes the plan record); the verify work must register it.
BackupPlanValidator.setEncryptionAvailable(true) once client encryption ships (plans asking for it are refused until then; the coordinator also refuses to write plain files for an encrypted plan).
BackupScheduledRunProcess.setOwnerHasAdminCheck(predicate) to plug in the user directory (default accepts any non-blank owner).
Set BackupsQBitConfig.schedulerName to enable plan jobs and the daily prune.

🤖 Generated with Claude Code

## Summary Run lifecycle for qbit-backups (spec 9.1, 10.3, 6.4/6.5, 11.1, and the Back up these records, Cancel, Resume, Test destination and Prune processes). **Note: foundation PR #4 is not merged yet, so this branch contains its commit (eea9f63). The lifecycle work is the single commit on top (ef3a55a) and rebases cleanly once #4 lands.** - **Schedule:** `BackupPlanSyncToScheduledJobProcess` (foreignKeyType `backups_plan`) plus a verify variant (`backups_plan_verify`), wired on the plan table when `BackupsQBitConfig.schedulerName` is set. No cron means manual-only (no job); inactive plans have their jobs switched off; deleting a plan deletes them. - **Lease and queueing:** `BackupRunCoordinator`. One run per plan (live-heartbeat check in `BackupCatalog`, `QUEUED` to `RUNNING`, a second claim check for cross-pod races); `SKIPPED_CONCURRENT` leaves a visible CANCELLED run row and a `backups.run.skipped_concurrent` warning; `maxConcurrentRuns` queues, `drainQueue()` starts queued runs after every managed run. Manual and scheduled runs both go through it. - **Scheduled entry point:** `backupRunScheduledPlan` (the process the job runs) validates the plan owner, then runs the engine as the scheduler system session (`system:scheduler`, securityScope FULL). A bad owner raises `backups.plan.owner_invalid`, sets slaStatus OWNER_INVALID and does not run. - **Plan save validation** (`BackupPlanTableCustomizer` / `BackupPlanValidator`): saving user has backups.admin and read on every resolved table; objectLockMode checked against `BackupStorage.capabilities()`; INCLUDE_ENCRYPTED_ONLY needs encryption; PARQUET, ZSTD, incremental, native dump and (until a cipher exists) client encryption refused with clear messages; cron validity; planKey is a slug and immutable after the first run; the saving user becomes owner when none is given. - **Prune:** `PruneCalculator` (GFS, pure, tested on a synthetic clock) and `BackupPruner`. Classifies runs into `retentionClass`, deletes through `deletePermanent` on the prune alias, reports `kept: locked until`, deletes debris older than 7 days, never deletes a run a restore used in the last 30 days, always keeps the newest verified run, sets `prunedDate` instead of deleting rows, and does mark-and-sweep of the content-addressed file store (recomputes `referenceCount` from the file indexes of retained runs, deletes unreferenced unlocked objects, sets `deletedDate`; deferred while a run is live or an index cannot be read). Review shows runs to delete, kept by class, locked, debris, CAS objects to sweep and MiB freed. - **Object lock at write time:** the engine applies `WriteOptions.withObjectLock` (plan mode and days) to every part, the index, checksums, manifest and file-store objects (a server-side copy cannot lock, so with a lock the bytes are streamed); refuses a lock the bucket does not support. - **Processes:** `<table>.backupExportTable` (one per backup-able table), `backupCancelRun` / `backupCancelRestoreRun`, `backupResumeRun`, `backupTestDestination`, `backupPrune` (plans) and `backupPruneRuns` (runs; prune also runs daily when a scheduler is configured). - **Resume:** parts are now catalogued the moment they are complete. A resume continues the same run: finished tables are kept, the interrupted JSONL table continues after the last key of its last complete part (rows hash rebuilt from the parts), the file index is rebuilt. CSV, a second format and encrypted tables restart that table. Runs that held a snapshot refuse and say so. ## Engine edits (minimal) `BackupEngine`: existing run id (QUEUED row claimed by the coordinator), resume state, per-part artifact rows, lock options, `consistencyAchieved` persisted at start, `startAfterKey` passed to the reader. `TableArtifactWriter`: part listener, lock, resume seeding. `FileCopier`: lock. `BackupCatalog`: live/queued run queries, cancel flag, delete by id. `BackupsTables`: plan customizers; run and restore status enums gain `CANCELLING` (cooperative cancel sets it, the engine checks the row each page). `BackupsProcesses.processes()` / `BackupsQBitProducer`: one line each. ## Stand-ins None added. Relies on the foundation's `TableReader` stand-in (core KeysetPager) and on `BackupStorage` (qbit-object-storage adapter written separately). The ad-hoc export holds the selected records in memory until it runs. ## Tests (`./mvnw -o verify` green, 58 tests, jacoco gate met) PruneCalculatorTest (GFS fixtures, time zones, ISO week across new year, guards), BackupPrunerTest (real runs on a synthetic clock: prune and sweep, retained counts, locked run kept until the lock passes, locked CAS object, debris, restore and verified guards, sweep deferral, prune alias, review writes nothing), BackupRunCoordinatorTest (skip, stale heartbeat, queueing and drain, raised limit, cancel at next page, cancel process, resume after injected failure with identical rows hash/parts/file index, refusals, CSV restart), BackupPlanValidationTest, BackupPlanScheduleTest (jobs follow the plan, scheduled entry point, owner validation), LifecycleProcessesTest (export, test destination incl. head-only on a locked bucket, prune review/result). ## Owner steps / interfaces for other work - The verify schedule job runs process `backupVerifyPlan` (table backups_plan, takes the plan record); the verify work must register it. - `BackupPlanValidator.setEncryptionAvailable(true)` once client encryption ships (plans asking for it are refused until then; the coordinator also refuses to write plain files for an encrypted plan). - `BackupScheduledRunProcess.setOwnerHasAdminCheck(predicate)` to plug in the user directory (default accepts any non-blank owner). - Set `BackupsQBitConfig.schedulerName` to enable plan jobs and the daily prune. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-backups#7