feat(restore): restore, catalog import and download (spec 9.3-9.7) #6

2026-10-12T21:43:23Z

KofTwentyTwo commented

2026-10-12 21:43:23 +00:00

(Migrated from github.com)

What

Spec sections 9.3 to 9.7 and tables 4.6, 4.7 and 4.9, as standard streamed-ETL processes (record or table actions, never in the nav).

backupRestore (Restore... on backups_run) and backupRestoreTable (Restore this table on backups_artifact). First screen takes tables, per-table filters (JSON), target table, mode (INSERT_NEW, UPSERT, REPLACE), match key (primary key or a unique key), id strategy (PRESERVE, REMAP_NEW_IDS) and the options (suppressAutomations, publishEvents, runCustomizers, perRecordAudit, restoreFiles with optional other bucket alias and prefix, preRestoreBackup, typed confirmation).
Review is a real dry run of the whole restore (same algorithm as Execute, so the numbers are what happens). Per table: inserted, updated, unchanged, skipped, deleted (REPLACE), unique-key conflicts, orphaned references, schema status (MATCH, COMPATIBLE, MISMATCH via TableFingerprints plus a field diff), unmapped fields and references, sequences to resync, files to restore, present with the same checksum, conflicting, missing from the backup, and a sample diff of up to 20 changed records with sensitive values masked.
Execute: pre-restore backup (BackupEngine, trigger PRE_RESTORE), tables in manifest restore order, writes behind a RestoreWriter in page transactions (REPLACE deletes and upserts share one transaction per table), one extra pass for rows that failed to insert, REMAP with backups_id_map and reference rewriting in restore order (map cleared at the end), sequence resync, file restore from the content-addressed store through a temp key with a SHA-256 check against the file index, one summary audit per table, backups_restore_run and backups_restore_run_table rows. Every backup part is checked against its stored SHA-256 as it is read.
Safety rails in the engine (not only the screens): needs backups.restore; REPLACE also needs backups.admin and the typed confirmation (table name, or REPLACE n TABLES); REPLACE cannot use new ids; SECRET and BACKUPS tables are refused; EPHEMERAL tables only when named; refused while a backup covering the same tables is RUNNING (fresh heartbeat).
backupImportCatalog (table action on backups_run, backups.admin): lists manifests under a bucket alias and prefix, validates them (readable, known version, run key matches folder, checksums.sha256 agrees with the manifest, every part present with the recorded size), creates backups_run and backups_artifact rows (trigger IMPORTED) and re-registers the content-addressed file store. Idempotent.
backupDownload (on runs) and backupDownloadArtifact (on artifacts): presigned links (15 minute TTL), ciphertext only, each issue audited and logged without the URL.
BackupsProcesses.processes() gets two lines (RestoreProcesses, CatalogProcesses).

Stand-ins (replace when QQQ 4.1.0-RC.5 ships)

RestoreWriteFlag (QRun-IO/qqq#1111): selected reflectively in StandardRestoreWriter when the class exists. Without it the standard Insert, Update and Delete actions are used with the DML audit omitted (one summary audit per table is written instead). Until the flag ships, created and modified dates are set by the normal defaults, so restored rows get a new createDate; Review says so in a note. The tests leave those dates out of the content hash until usesCoreRestoreFlag() is true.
resyncKeySequences (#1112): called reflectively on the backend module when present; otherwise a no-op, and Review warns that sequences need a manual resync.
Decryption: BackupSource.setTransformResolver is the seam for the crypto package (#1107). Restoring an encrypted run without a resolver fails with a clear message.
Storage only through BackupStorage (memory backend in tests); no object-storage dependency.

Not in this PR

The PREVIEWED status row (nothing is written at Review), the table multi-select limited to the manifest (the tables field is a comma separated text), Restore these records from a filtered list (use the filters field), incremental chains, cycle breaking by nulling references (only the retry pass), the backup-side refusal while a restore runs (it belongs in BackupEngine; RestoreEngine.plan shows the check to mirror).

Tests (spec 13)

RestoreModesTest, RestoreSafetyTest, RestoreProcessTest, CatalogImportTest: back up 30 posts, then delete 10, edit 10, add 10; Review of each mode asserts the summary lines; restore asserts counts, hashes and files (checksum); REMAP (all new, and matched on a unique key); REPLACE confirmation and admin; conflicts, orphans, schema mismatch, masked sample diff, filters, pluggable writer and page transactions, sequence resync call; the full process flow (input, Review, Execute, Result); import into a fresh instance (empty catalog, tables and media) and restore from it; a damaged part is refused; downloads; a user without backups.restore cannot run it (process permission and engine). ./mvnw verify: 37 tests, jacoco gate green. Also ran the restore tests against the core worktrees with the #1111 and #1112 classes on the classpath (the flag is created and accepted; the worktree class output was stale so its date behavior was not exercised).

Base: this branch contains the foundation commit (PR #4, not merged yet); my commit is separate and rebases cleanly.

🤖 Generated with Claude Code

## What Spec sections 9.3 to 9.7 and tables 4.6, 4.7 and 4.9, as standard streamed-ETL processes (record or table actions, never in the nav). - `backupRestore` (Restore... on `backups_run`) and `backupRestoreTable` (Restore this table on `backups_artifact`). First screen takes tables, per-table filters (JSON), target table, mode (INSERT_NEW, UPSERT, REPLACE), match key (primary key or a unique key), id strategy (PRESERVE, REMAP_NEW_IDS) and the options (suppressAutomations, publishEvents, runCustomizers, perRecordAudit, restoreFiles with optional other bucket alias and prefix, preRestoreBackup, typed confirmation). - **Review is a real dry run** of the whole restore (same algorithm as Execute, so the numbers are what happens). Per table: inserted, updated, unchanged, skipped, deleted (REPLACE), unique-key conflicts, orphaned references, schema status (MATCH, COMPATIBLE, MISMATCH via `TableFingerprints` plus a field diff), unmapped fields and references, sequences to resync, files to restore, present with the same checksum, conflicting, missing from the backup, and a sample diff of up to 20 changed records with sensitive values masked. - **Execute**: pre-restore backup (`BackupEngine`, trigger PRE_RESTORE), tables in manifest restore order, writes behind a `RestoreWriter` in page transactions (REPLACE deletes and upserts share one transaction per table), one extra pass for rows that failed to insert, REMAP with `backups_id_map` and reference rewriting in restore order (map cleared at the end), sequence resync, file restore from the content-addressed store through a temp key with a SHA-256 check against the file index, one summary audit per table, `backups_restore_run` and `backups_restore_run_table` rows. Every backup part is checked against its stored SHA-256 as it is read. - **Safety rails in the engine** (not only the screens): needs `backups.restore`; REPLACE also needs `backups.admin` and the typed confirmation (table name, or `REPLACE n TABLES`); REPLACE cannot use new ids; SECRET and BACKUPS tables are refused; EPHEMERAL tables only when named; refused while a backup covering the same tables is RUNNING (fresh heartbeat). - `backupImportCatalog` (table action on `backups_run`, `backups.admin`): lists manifests under a bucket alias and prefix, validates them (readable, known version, run key matches folder, `checksums.sha256` agrees with the manifest, every part present with the recorded size), creates `backups_run` and `backups_artifact` rows (trigger IMPORTED) and re-registers the content-addressed file store. Idempotent. - `backupDownload` (on runs) and `backupDownloadArtifact` (on artifacts): presigned links (15 minute TTL), ciphertext only, each issue audited and logged without the URL. - `BackupsProcesses.processes()` gets two lines (`RestoreProcesses`, `CatalogProcesses`). ## Stand-ins (replace when QQQ 4.1.0-RC.5 ships) - `RestoreWriteFlag` (QRun-IO/qqq#1111): selected reflectively in `StandardRestoreWriter` when the class exists. Without it the standard Insert, Update and Delete actions are used with the DML audit omitted (one summary audit per table is written instead). **Until the flag ships, created and modified dates are set by the normal defaults**, so restored rows get a new `createDate`; Review says so in a note. The tests leave those dates out of the content hash until `usesCoreRestoreFlag()` is true. - `resyncKeySequences` (#1112): called reflectively on the backend module when present; otherwise a no-op, and Review warns that sequences need a manual resync. - Decryption: `BackupSource.setTransformResolver` is the seam for the crypto package (#1107). Restoring an encrypted run without a resolver fails with a clear message. - Storage only through `BackupStorage` (memory backend in tests); no object-storage dependency. ## Not in this PR - The PREVIEWED status row (nothing is written at Review), the table multi-select limited to the manifest (the tables field is a comma separated text), Restore these records from a filtered list (use the filters field), incremental chains, cycle breaking by nulling references (only the retry pass), the backup-side refusal while a restore runs (it belongs in `BackupEngine`; `RestoreEngine.plan` shows the check to mirror). ## Tests (spec 13) `RestoreModesTest`, `RestoreSafetyTest`, `RestoreProcessTest`, `CatalogImportTest`: back up 30 posts, then delete 10, edit 10, add 10; Review of each mode asserts the summary lines; restore asserts counts, hashes and files (checksum); REMAP (all new, and matched on a unique key); REPLACE confirmation and admin; conflicts, orphans, schema mismatch, masked sample diff, filters, pluggable writer and page transactions, sequence resync call; the full process flow (input, Review, Execute, Result); import into a fresh instance (empty catalog, tables and media) and restore from it; a damaged part is refused; downloads; a user without `backups.restore` cannot run it (process permission and engine). `./mvnw verify`: 37 tests, jacoco gate green. Also ran the restore tests against the core worktrees with the #1111 and #1112 classes on the classpath (the flag is created and accepted; the worktree class output was stale so its date behavior was not exercised). **Base:** this branch contains the foundation commit (PR #4, not merged yet); my commit is separate and rebases cleanly. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-backups#6