feat(progress): backup phases, counts, bytes and log lines (durable job progress) #26

2026-10-14T22:33:01Z

KofTwentyTwo commented

2026-10-14 22:33:01 +00:00

(Migrated from github.com)

Owner ask (2026-10-14): "Run backup now" showed only "Executing Process 1 of 1". This PR makes backups report real progress through the QQQ job progress model (spec: QRun-IO/qqq#1210, core: QRun-IO/qqq#1212).

backupRunPlan, restore and verify are declared always durable (DurableProcessMetaData, chunk size 1). They never run inline and survive a pod restart. Idempotency is NONE: a reclaimed run goes to review, and a retry resumes through the existing run and resume path. External S3 writes cannot commit with the chunk, so this does not claim natural idempotency.
Phases: prepare (2), tables (75: N of M tables, bytes stored), media (18: N of M files, copied vs already stored), upload (5: catalog, checksums, manifest). Verify processes have their own single verify phase.
Log lines: one per table ("kof22_blog_post: 1,204 rows, 2.1 MB in 3.2 s"), artifact keys relative to the run prefix, media totals, resume ("12 tables already done"), every engine warning, and the failure. Logs carry table names and counts only: no row values, credentials or endpoints, and core redaction is the backstop.
Restore and verify check for pause and cancel between tables, pages and parts.
Tests: phase order, N of M counts, monotonic bytes, per-table lines, no secrets, the failure ERROR line, and the durability declaration. 173 tests pass, and JaCoCo passes.
pom.xml: qqq.snapshot.version moves from RC.6 to 4.1.0-RC.8. CI stays red until QQQ RC.8 (with #1212) is published. Locally it was verified with -Dqqq.snapshot.version=4.1.0-SNAPSHOT against #1212.

🤖 Generated with Claude Code

Owner ask (2026-10-14): "Run backup now" showed only "Executing Process 1 of 1". This PR makes backups report real progress through the QQQ job progress model (spec: QRun-IO/qqq#1210, core: QRun-IO/qqq#1212). - `backupRunPlan`, restore and verify are declared **always durable** (`DurableProcessMetaData`, chunk size 1). They never run inline and survive a pod restart. Idempotency is `NONE`: a reclaimed run goes to review, and a retry resumes through the existing run and resume path. External S3 writes cannot commit with the chunk, so this does not claim natural idempotency. - **Phases:** prepare (2), tables (75: N of M tables, bytes stored), media (18: N of M files, copied vs already stored), upload (5: catalog, checksums, manifest). Verify processes have their own single verify phase. - **Log lines:** one per table ("kof22_blog_post: 1,204 rows, 2.1 MB in 3.2 s"), artifact keys relative to the run prefix, media totals, resume ("12 tables already done"), every engine warning, and the failure. Logs carry table names and counts only: no row values, credentials or endpoints, and core redaction is the backstop. - Restore and verify check for pause and cancel between tables, pages and parts. - Tests: phase order, N of M counts, monotonic bytes, per-table lines, no secrets, the failure ERROR line, and the durability declaration. 173 tests pass, and JaCoCo passes. - `pom.xml`: `qqq.snapshot.version` moves from RC.6 to **4.1.0-RC.8**. **CI stays red until QQQ RC.8 (with #1212) is published.** Locally it was verified with `-Dqqq.snapshot.version=4.1.0-SNAPSHOT` against #1212. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-backups#26