fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT

On release branches the bundle's version and the producer's differed, so the
bundle test that asserts they match failed, and released bundles recorded
SNAPSHOT as their provenance. help/*.yaml is now filtered:
version: ${project.version}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -5,12 +5,19 @@ All notable changes to this project are documented here. The format follows Keep
 ## [0.1.0] - Unreleased
 
 ### Added
+- Always-durable backup, resume, scheduled backup, restore and verify processes with conservative `NONE` idempotency and stable backup identities on explicit job retry.
+- Weighted backup phases, table/file counts, stored bytes, safe per-table/artifact logs, warnings, failures and resume notices through QQQ `JobProgress`.
+- Cooperative job cancellation during backup pages, media copying, restore pages and verification scans; local media-reference spooling keeps phase separation from growing heap usage across tables.
+- Complete host-neutral in-app help bundle with live references and a help coverage gate. Hosts can override entries with a HOST bundle.
 - Guided empty states: help bundle `help/qbit-backups.yaml` (app and all nine tables, with "How rows get created") applied by `BackupsHelp`, and a Setup card on the Backups home (plan, scope rules, storage, schedule, verification, prune bucket).
 - "Create default plan" table action on Backup plans: adds "Nightly full" for a storage bucket alias; insert-only.
 - `backupVerifyPlan`: the scheduled verification process. A plan verify cron created a scheduled job for a process that did not exist, so scheduled verification never ran.
 - Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.
 
 ### Changed
+- Target QQQ `4.1.0-RC.8` (job progress).
+- Persist controlled backup failure messages instead of untrusted provider exception text.
+- Pin QQQ to released 4.1.0-RC.7 and replace the temporary help loader with core HelpBundles registration.
 - Saving a plan with Use read replica on is refused (it was never read); the not-yet-available settings (read replica, native dump, incremental, PARQUET, ZSTD) say so in their field help; scheduledJobId is documented as unused.
 - Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
 - Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.

README.md

@@ -10,7 +10,7 @@
 ## Requirements
 
 - Java 21.
-- QQQ 4.1 (targets `4.1.0-RC.6`, not yet released).
+- QQQ `4.1.0-RC.8` or later.
 
 ## Install
 
@@ -40,17 +40,43 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
 ./mvnw verify
 ```
 
-Before RC.6 is released, verify against the locally installed core build only:
+See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for fingerprint compatibility,
+paging guarantees and restore permissions.
 
-```bash
-./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
-```
+## In-app help
 
-The POM stays on RC.6. See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for
-fingerprint compatibility, paging guarantees and restore permissions.
+The producer registers [help/qbit-backups.yaml](src/main/resources/help/qbit-backups.yaml)
+as a QBIT bundle. Hosts can register a `layer: HOST` bundle with matching entry keys
+and roles to override its text. Core's help startup sync applies the bundle when the
+host installs the help content table; registration also works without that table.
+
+`BackupPlanScheduleTest` checks coverage with `HelpBundleAssertions`, including
+live references and optional scheduler processes. `BackupsTablesGuardrailTest`
+checks empty-state, widget and possible-value-source coverage. Run it with
+`./mvnw verify`. Keep the bundle's literal version aligned with the POM version.
 
 Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
 
+## Durable jobs and progress
+
+Run backup now, Resume, scheduled backups, both Restore actions, and Verify backup
+are always durable jobs. Hosts must configure QQQ durable jobs and its worker.
+Review remains a preview; execution runs in the background and reports through
+the job page and jobs tray.
+
+Backups report Resolve scope, Back up tables, Copy media files, and Write catalog
+and manifest. Separate verification jobs report a single Verify phase.
+The job includes table/file counts, stored bytes, per-table
+rows and duration, artifact names, warnings and failures. Logs exclude row values,
+source file keys, storage credentials and endpoints. Provider errors use controlled
+failure messages; detailed exceptions stay in server diagnostics.
+
+These processes use `NONE` idempotency: a worker interruption requires review before
+retry because object writes and catalog updates are not atomic. Explicitly retrying
+a backup job reuses its run and resumes completed parts; it does not create another
+backup. Snapshot runs that cannot resume must be started as a new backup. See
+[the progress and retry contract](docs/DESIGN.md#job-progress-implementation) for limits and cancellation.
+
 ## License
 
 Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).

docs/DESIGN.md

@@ -1,3 +1,18 @@
 # QBit Backups: design
 
 The design for this qbit is tracked in [QRun-IO/qqq#1091](https://github.com/QRun-IO/qqq/pull/1091). This repository implements it; the implementation is in progress.
+
+
+## Job progress implementation
+
+Backup, resume, scheduled backup, restore, and verify execution use always-durable
+jobs with `NONE` idempotency against the QQQ job-progress API targeted for RC.8.
+Backup phases are `prepare`, `tables`, `media`, and `upload`, weighted 2, 75, 18,
+and 5. Verification runs separately with a single `verify` phase.
+
+Progress reports table/file counts, stored bytes, and safe artifact names without
+row data, source keys, credentials, or endpoints. Cancellation is cooperative at
+page and media-copy boundaries. Media references spool locally between phases.
+Object writes and catalog updates are not atomic, so worker interruptions require
+review before retry. Explicit backup retries reuse the run and resume completed
+parts; snapshot runs that cannot resume require a new backup.

pom.xml

@@ -131,6 +131,23 @@
    </dependencies>
 
    <build>
+      <resources>
+         <!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>true</filtering>
+            <includes>
+               <include>help/*.yaml</include>
+            </includes>
+         </resource>
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>false</filtering>
+            <excludes>
+               <exclude>help/*.yaml</exclude>
+            </excludes>
+         </resource>
+      </resources>
       <plugins>
          <plugin>
             <groupId>org.apache.maven.plugins</groupId>
@@ -194,14 +211,14 @@
    <profiles>
       <!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent  -->
       <!-- imports the released 4.1 BOM, this profile is active by default and     -->
-      <!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM.         -->
+      <!-- imports qqq-bom-pom 4.1.0-RC.8 ahead of the parent's 4.0.0 BOM.         -->
       <profile>
          <id>qqq-snapshot</id>
          <activation>
             <activeByDefault>true</activeByDefault>
          </activation>
          <properties>
-            <qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
+            <qqq.snapshot.version>4.1.0-RC.8</qqq.snapshot.version>
          </properties>
          <dependencyManagement>
             <dependencies>

src/main/java/com/kingsrook/qbits/backups/BackupsQBitProducer.java

@@ -17,6 +17,7 @@
 package com.kingsrook.qbits.backups;
 
 
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
 
@@ -160,7 +161,8 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
       com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData app = com.kingsrook.qbits.backups.app.BackupsApp.app(qBitConfig, tables);
       output.add(app);
       output.addSelfToInstance(qInstance);
-      com.kingsrook.qbits.backups.model.BackupsHelp.apply(qInstance, app.getName());
+      // Registration is metadata-only; core guards startup sync when helpContent is absent.
+      HelpBundles.register(qInstance, "help/qbit-backups.yaml");
 
       // A parent app only lists the apps that are its children, so attach ourselves (for example Admin, then Backups).
       com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData parent = qBitConfig.getParentAppName() == null ? null : qInstance.getApp(qBitConfig.getParentAppName());

src/main/java/com/kingsrook/qbits/backups/model/BackupsHelp.java

@@ -1,126 +0,0 @@
-/*
- * Copyright 2026 Kingsrook, LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package com.kingsrook.qbits.backups.model;
-
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.util.LinkedHashMap;
-import java.util.Map;
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
-import com.kingsrook.qqq.backend.core.exceptions.QException;
-import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
-import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
-import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
-
-
-/*******************************************************************************
- ** Applies help/qbit-backups.yaml (the QQQ help-bundle format) to the Backups app
- ** and its backups_* tables: what each table is for and how its rows get created,
- ** shown on the table screen and when it is empty.
- ** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with HelpBundles.register once the pinned QQQ
- ** release includes it; the bundle file stays the same.
- *******************************************************************************/
-public final class BackupsHelp
-{
-   public static final String RESOURCE = "/help/qbit-backups.yaml";
-   public static final String SLOT     = "help";
-   public static final String APP_KEY  = "app:backups;slot:help";
-
-
-
-   /*******************************************************************************
-    ** Utility class.
-    *******************************************************************************/
-   private BackupsHelp()
-   {
-   }
-
-
-
-   /*******************************************************************************
-    ** The bundle's entries by key, validated (fails startup if malformed).
-    *******************************************************************************/
-   public static Map<String, JsonNode> entries() throws QException
-   {
-      try(InputStream stream = BackupsHelp.class.getResourceAsStream(RESOURCE))
-      {
-         if(stream == null)
-         {
-            throw (new IOException("missing resource"));
-         }
-         JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
-         if(!"qbit-backups".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
-         {
-            throw (new IOException("invalid bundle header"));
-         }
-         Map<String, JsonNode> entries = new LinkedHashMap<>();
-         for(JsonNode entry : bundle.path("entries"))
-         {
-            String key     = entry.path("key").asText();
-            String summary = entry.path("summary").asText();
-            if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
-            {
-               throw (new IOException("invalid or duplicate entry: " + key));
-            }
-         }
-         return (entries);
-      }
-      catch(IOException | RuntimeException e)
-      {
-         throw (new QException("Could not load " + RESOURCE, e));
-      }
-   }
-
-
-
-   /*******************************************************************************
-    ** Apply the help to the app and every backups_* table in the instance. A host's
-    ** own help in the slot is kept (the bundle only fills it).
-    *******************************************************************************/
-   public static void apply(QInstance qInstance, String appName) throws QException
-   {
-      Map<String, JsonNode> entries = entries();
-      for(QTableMetaData table : qInstance.getTables().values())
-      {
-         JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
-         if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
-         {
-            table.withHelpContent(SLOT, content(entry));
-         }
-      }
-      QAppMetaData app      = qInstance.getApp(appName);
-      JsonNode     appEntry = entries.get(APP_KEY);
-      if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
-      {
-         app.withHelpContent(SLOT, content(appEntry));
-      }
-   }
-
-
-
-   /*******************************************************************************
-    ** Summary as the first paragraph, then the content, as markdown.
-    *******************************************************************************/
-   static QHelpContent content(JsonNode entry)
-   {
-      return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
-   }
-}

src/main/java/com/kingsrook/qbits/backups/processes/BackupsProcesses.java

@@ -30,6 +30,11 @@ import com.kingsrook.qbits.backups.run.BackupRunFactory;
 import com.kingsrook.qbits.backups.run.BackupRunResult;
 import com.kingsrook.qbits.backups.scope.ScopeResolver;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.DurableProcessMetaData;
+import com.kingsrook.qqq.backend.core.jobs.model.JobIdempotency;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
@@ -71,7 +76,7 @@ public final class BackupsProcesses
    {
       List<QProcessMetaData> processes = new ArrayList<>();
       processes.add(permission(etl(RUN_PLAN, "Run backup now", BackupsTables.PLAN, RunPlanTransform.class, RunPlanLoad.class,
-         "This is a preview of what the backup will include. Nothing is written until you continue."), BackupsPermissions.RUN));
+         "This is a preview of what the backup will include. Nothing is written until you continue.").withSupplementalMetaData(durability()), BackupsPermissions.RUN));
       processes.add(DefaultPlanProcess.process());
       processes.addAll(VerifyProcesses.processes());
       processes.addAll(RestoreProcesses.processes());
@@ -82,6 +87,40 @@ public final class BackupsProcesses
 
 
 
+   /*******************************************************************************
+    ** External writes cannot commit with the job chunk. Require review before retry
+    ** instead of claiming that replay is naturally idempotent.
+    *******************************************************************************/
+   public static DurableProcessMetaData durability()
+   {
+      return (new DurableProcessMetaData().withEligible(true).withAlwaysDurable(true)
+         .withChunkSize(1).withIdempotency(JobIdempotency.NONE));
+   }
+
+
+
+   /*******************************************************************************
+    ** A durable worker must see failures/cancellation instead of a successful ETL
+    ** summary. Inline callers retain the existing result-screen behavior.
+    *******************************************************************************/
+   public static void requireSuccessfulJob(BackupRunResult result) throws QException
+   {
+      if(JobChunkScope.current().isEmpty())
+      {
+         return;
+      }
+      if("CANCELLED".equals(result.getStatus()))
+      {
+         throw (new JobInterruptedException(JobInterruptedException.Reason.CANCEL));
+      }
+      if(!result.isUsable())
+      {
+         throw (new QException(result.getErrorMessage() == null ? "Backup did not complete" : result.getErrorMessage()));
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Record the permission a process needs.
     *******************************************************************************/
@@ -229,8 +268,9 @@ public final class BackupsProcesses
       {
          for(QRecord plan : input.getRecords())
          {
-            BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "MANUAL");
+            BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "MANUAL", JobProgress.of(input), null);
             results.add(result);
+            requireSuccessfulJob(result);
             switch(result.getStatus())
             {
                case "SUCCEEDED" -> succeeded.incrementCountAndAddPrimaryKey(result.getRunId());

src/main/java/com/kingsrook/qbits/backups/processes/RestoreProcesses.java

@@ -33,6 +33,8 @@ import com.kingsrook.qbits.backups.restore.RestoreRequest;
 import com.kingsrook.qbits.backups.restore.RestoreSummaries;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
@@ -107,7 +109,7 @@ public final class RestoreProcesses
       process.setLabel(label);
       process.setTableName(table);
       process.setPermissionRules(BackupsTables.permissionRules());
-      return (process);
+      return (process.withSupplementalMetaData(BackupsProcesses.durability()));
    }
 
 
@@ -241,13 +243,38 @@ public final class RestoreProcesses
       @Override
       public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
       {
+         JobProgress progress = JobProgress.of(input);
+         progress.phases(JobPhase.of("restore", "Restore backups", 100)).phase("restore");
+         Map<Integer, Set<String>> sources = sources(input.getRecords());
+         progress.total(sources.size(), "backups").done(0);
+         int done = 0;
          RestoreEngine engine = new RestoreEngine();
-         for(Map.Entry<Integer, Set<String>> source : sources(input.getRecords()).entrySet())
+         try
          {
-            RestoreRequest request = RestoreRequest.fromInput(input, source.getKey(), new ArrayList<>(source.getValue()));
-            outcomes.add(engine.execute(request));
+            for(Map.Entry<Integer, Set<String>> source : sources.entrySet())
+            {
+               progress.message("Restoring backup " + source.getKey());
+               RestoreRequest request = RestoreRequest.fromInput(input, source.getKey(), new ArrayList<>(source.getValue()));
+               outcomes.add(engine.execute(request));
+               progress.info("Restored backup " + source.getKey());
+               progress.done(++done);
+            }
+            progress.getSnapshot().getPhases().getFirst().setStatus(JobPhase.Status.DONE);
+            output.getRecords().addAll(input.getRecords());
+         }
+         catch(QException | RuntimeException e)
+         {
+            progress.getSnapshot().getPhases().getFirst().setStatus(JobPhase.Status.FAILED);
+            if(com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException.find(e) == null)
+            {
+               progress.error("Restore failed (" + e.getClass().getSimpleName() + ")");
+            }
+            throw (e);
+         }
+         finally
+         {
+            progress.flush();
          }
-         output.getRecords().addAll(input.getRecords());
       }
 
 

src/main/java/com/kingsrook/qbits/backups/processes/ResumeRunProcess.java

@@ -70,7 +70,7 @@ public final class ResumeRunProcess
    public static QProcessMetaData process()
    {
       return (BackupsProcesses.permission(BackupsProcesses.etl(NAME, "Resume", BackupsTables.RUN, Transform.class, Load.class,
-         "This is a preview of where each selected run will carry on. Nothing is written until you continue."), BackupsPermissions.RUN));
+         "This is a preview of where each selected run will carry on. Nothing is written until you continue.").withSupplementalMetaData(BackupsProcesses.durability()), BackupsPermissions.RUN));
    }
 
 
@@ -167,7 +167,8 @@ public final class ResumeRunProcess
       {
          for(QRecord run : input.getRecords())
          {
-            BackupRunResult result = new BackupRunCoordinator().resume(run);
+            BackupRunResult result = new BackupRunCoordinator().resume(run, com.kingsrook.qqq.backend.core.jobs.progress.JobProgress.of(input));
+            BackupsProcesses.requireSuccessfulJob(result);
             switch(result.getStatus())
             {
                case "SUCCEEDED" -> succeeded.incrementCountAndAddPrimaryKey(run.getValue("id"));

src/main/java/com/kingsrook/qbits/backups/processes/VerifyProcesses.java

@@ -30,6 +30,9 @@ import com.kingsrook.qbits.backups.verify.VerificationOutcome;
 import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
 import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
@@ -85,7 +88,7 @@ public final class VerifyProcesses
    {
       List<QProcessMetaData> processes = new ArrayList<>();
       processes.add(BackupsProcesses.permission(BackupsProcesses.etl(VERIFY, "Verify backup", BackupsTables.RUN, VerifyTransform.class, VerifyLoad.class,
-         "This checks that the backup's objects are all in storage, then reads every byte back, decrypts and decodes it, and compares counts and hashes with the manifest. Nothing is changed in the backup."),
+         "This checks that the backup's objects are all in storage, then reads every byte back, decrypts and decodes it, and compares counts and hashes with the manifest. Nothing is changed in the backup.").withSupplementalMetaData(BackupsProcesses.durability()),
          BackupsPermissions.RUN));
       processes.add(checkSlaProcess(BackupsRuntime.getConfig()));
       return (processes);
@@ -220,12 +223,31 @@ public final class VerifyProcesses
       @Override
       public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
       {
+         JobProgress progress = JobProgress.of(input);
+         progress.phases(JobPhase.of("verify", "Verify", 100)).phase("verify").total(input.getRecords().size(), "backups").done(0);
+         int done = 0;
          String level = input.getValueString(FIELD_VERIFY_LEVEL) == null ? BackupVerifier.CONTENT : input.getValueString(FIELD_VERIFY_LEVEL);
          String user  = QContext.getQSession() == null || QContext.getQSession().getUser() == null ? "system" : QContext.getQSession().getUser().getIdReference();
          for(QRecord run : input.getRecords())
          {
+            JobChunkScope.checkBetweenPages();
+            progress.message("Verifying backup " + run.getValueInteger("id"));
             List<VerificationOutcome> outcomes = new BackupVerifier().verifyThrough(run.getValueInteger("id"), level, "MANUAL", user);
             VerificationOutcome       last     = outcomes.get(outcomes.size() - 1);
+            for(VerificationOutcome outcome : outcomes)
+            {
+               progress.info("Verification " + outcome.level() + ": " + outcome.status());
+            }
+            if(!last.passed())
+            {
+               progress.getSnapshot().getPhases().getFirst().setStatus(JobPhase.Status.FAILED);
+               progress.error("Backup verification " + last.status());
+               if(JobChunkScope.current().isPresent())
+               {
+                  throw (new QException("Backup verification " + last.status()));
+               }
+            }
+            progress.done(++done);
             switch(last.status())
             {
                case "PASSED" -> passed.incrementCountAndAddPrimaryKey(run.getValue("id"));
@@ -248,6 +270,8 @@ public final class VerifyProcesses
             }
             output.getRecords().add(run);
          }
+         progress.getSnapshot().getPhases().getFirst().setStatus(failed.getCount() > 0 || errored.getCount() > 0 ? JobPhase.Status.FAILED : JobPhase.Status.DONE);
+         progress.flush();
       }
 
 

src/main/java/com/kingsrook/qbits/backups/restore/RestoreEngine.java

@@ -49,6 +49,8 @@ import com.kingsrook.qqq.backend.core.actions.audits.AuditAction;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
@@ -373,6 +375,15 @@ public class RestoreEngine
       }
       catch(QException | RuntimeException e)
       {
+         JobInterruptedException interrupted = JobInterruptedException.find(e);
+         if(interrupted != null)
+         {
+            if(interrupted.getReason() == JobInterruptedException.Reason.CANCEL)
+            {
+               catalog.update(BackupsTables.RESTORE_RUN, restoreRunId, Map.of("status", "CANCELLED", "endDate", BackupsRuntime.getClock().instant()));
+            }
+            throw (interrupted);
+         }
          LOG.warn("Restore failed", e, logPair("restoreRunId", restoreRunId));
          String message = e.getClass().getSimpleName() + ": " + e.getMessage();
          try
@@ -447,6 +458,7 @@ public class RestoreEngine
       }
 
       BackupRunRequest backup = new BackupRunRequest()
+         .withProgress(new com.kingsrook.qqq.backend.core.jobs.progress.JobProgress((com.kingsrook.qqq.backend.core.actions.async.AsyncJobCallback) null))
          .withSettings(settings)
          .withScope(new ScopeResolver(QContext.getQInstance()).resolve(settings, rules))
          .withTrigger("PRE_RESTORE")
@@ -507,10 +519,11 @@ public class RestoreEngine
     ** index.  A conflicting file (different content at the target) is only replaced
     ** by a REPLACE restore.
     *******************************************************************************/
-   private void restoreFiles(RestoreRequest request, Plan plan, RestorePass pass, RestoreOutcome outcome, Integer restoreRunId)
+   private void restoreFiles(RestoreRequest request, Plan plan, RestorePass pass, RestoreOutcome outcome, Integer restoreRunId) throws QException
    {
       for(RestorePass.FileWork work : pass.getFileWork())
       {
+         JobChunkScope.checkBetweenPages();
          if(work.conflict() && !RestoreRequest.MODE_REPLACE.equals(request.getMode()))
          {
             continue;

src/main/java/com/kingsrook/qbits/backups/restore/RestorePass.java

@@ -46,6 +46,7 @@ import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
@@ -200,6 +201,7 @@ public class RestorePass
 
       for(String name : order)
       {
+         JobChunkScope.checkBetweenPages();
          restoreTable(name);
       }
       retryPass();
@@ -566,6 +568,7 @@ public class RestorePass
     *******************************************************************************/
    private void processPage(TableContext c, List<QRecord> page) throws QException
    {
+      JobChunkScope.checkBetweenPages();
       RestoreOutcome.TableOutcome o = c.outcome;
       o.addRows(page.size());
 

src/main/java/com/kingsrook/qbits/backups/run/BackupEngine.java

@@ -17,16 +17,21 @@
 package com.kingsrook.qbits.backups.run;
 
 
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
 import java.io.IOException;
 import java.io.OutputStream;
 import java.io.Serializable;
 import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
 import java.time.Instant;
 import java.time.ZoneOffset;
 import java.time.format.DateTimeFormatter;
 import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
+import java.util.Locale;
 import java.util.Map;
 import java.util.UUID;
 import java.util.function.Function;
@@ -47,6 +52,11 @@ import com.kingsrook.qbits.backups.scope.ScopeResolver;
 import com.kingsrook.qbits.backups.storage.BackupStorage;
 import com.kingsrook.qbits.backups.storage.WriteOptions;
 import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -85,13 +95,19 @@ public class BackupEngine
 
    /*******************************************************************************
     ** Run a backup.  Never throws for a backup failure: the result carries the
-    ** status and message (and the run row is FAILED).
+    ** status and message (and the run row is FAILED). Cooperative job interrupts
+    ** propagate as QException so the durable worker handles cancellation/lease loss.
     *******************************************************************************/
-   public BackupRunResult execute(BackupRunRequest request)
+   public BackupRunResult execute(BackupRunRequest request) throws QException
    {
+      JobProgress progress = request.getProgress();
+      if(!"prepare".equals(progress.getSnapshot().getCurrentPhase()))
+      {
+         prepare(progress);
+      }
       BackupRunResult result  = new BackupRunResult();
       Instant         started = request.getClock().instant();
-      String          runKey  = request.getResume() != null ? request.getResume().runKey() : RUN_KEY_FORMAT.format(started) + "-" + UUID.randomUUID().toString().substring(0, 4);
+      String          runKey  = request.getResume() != null ? request.getResume().runKey() : request.getRunKey() != null ? request.getRunKey() : RUN_KEY_FORMAT.format(started) + "-" + UUID.randomUUID().toString().substring(0, 4);
       String          planPrefix = request.getSettings().string("prefix", "qqq-backups");
       String          instancePrefix = trimSlashes(planPrefix) + "/" + request.getInstanceName() + "/";
       String          runPrefix  = request.getResume() != null ? request.getResume().runPrefix() : instancePrefix + request.getSettings().planKey() + "/" + DATE_PATH.format(started) + "/" + runKey + "/";
@@ -139,27 +155,52 @@ public class BackupEngine
       catch(Exception e)
       {
          result.setStatus("FAILED");
-         result.setErrorMessage("Could not create the run record: " + e.getMessage());
+         result.setErrorMessage("Could not create the run record (" + e.getClass().getSimpleName() + ")");
+         failProgress(progress, "Could not create the run record (" + e.getClass().getSimpleName() + ")");
          return (result);
       }
 
       try
       {
+         checkJob();
+         if(request.getResume() != null)
+         {
+            long complete = request.getResume().tables().values().stream().filter(ResumeState.TableState::complete).count();
+            progress.info("Resuming run " + result.getRunId() + ": " + complete + " tables already done");
+         }
          doRun(request, result, runPrefix, instancePrefix, started);
+         completePhase(progress, JobPhase.Status.DONE);
       }
       catch(CancelledException e)
       {
          finish(request, result, "CANCELLED", "Cancelled", started);
+         progress.info("Backup cancelled");
       }
       catch(TimedOutException e)
       {
          finish(request, result, "TIMED_OUT", e.getMessage(), started);
+         failProgress(progress, e.getMessage());
       }
       catch(Exception e)
       {
+         JobInterruptedException interrupted = JobInterruptedException.find(e);
+         if(interrupted != null)
+         {
+            if(interrupted.getReason() != JobInterruptedException.Reason.LEASE_LOST)
+            {
+               finish(request, result, interrupted.getReason() == JobInterruptedException.Reason.CANCEL ? "CANCELLED" : "ABANDONED", "Job interrupted", started);
+            }
+            throw (interrupted);
+         }
          LOG.warn("Backup run failed", e, logPair("runKey", runKey));
+         // Provider and reader messages can contain credentials or row values.
+         failProgress(progress, scrub(e));
          finish(request, result, "FAILED", scrub(e), started);
       }
+      finally
+      {
+         progress.flush();
+      }
       return (result);
    }
 
@@ -170,6 +211,7 @@ public class BackupEngine
     *******************************************************************************/
    private void doRun(BackupRunRequest request, BackupRunResult result, String runPrefix, String instancePrefix, Instant started) throws Exception
    {
+      JobProgress      progress = request.getProgress();
       QInstance        qInstance = QContext.getQInstance();
       BackupStorage    storage   = request.getStorage();
       var              settings  = request.getSettings();
@@ -188,6 +230,7 @@ public class BackupEngine
          throw (new IllegalStateException("The sensitive field policy INCLUDE_ENCRYPTED_ONLY requires client-side encryption."));
       }
       String consistency = consistencyFor(settings.string("consistency", "SNAPSHOT_PREFERRED"), result);
+      result.getWarnings().forEach(progress::warn);
       catalog.update(BackupsTables.RUN, result.getRunId(), Map.of("consistencyAchieved", result.getConsistencyAchieved()));
 
       //////////////////////////////////////////////////////////////////////////////////////
@@ -230,10 +273,15 @@ public class BackupEngine
       long                            fileBytes      = 0;
       long                            fileBytesRef   = 0;
 
-      try
+      progress.phase("tables").total(request.getScope().tables().size(), "tables").done(0).bytesDone(0);
+      try(PendingFiles pendingFiles = new PendingFiles())
       {
          for(ScopeResolver.ResolvedTable resolved : request.getScope().tables())
          {
+            checkJob();
+            long tableStarted = System.nanoTime();
+            long tableBytesBefore = bytesStored;
+            progress.message("Backing up " + resolved.name() + " (" + (done + 1) + " of " + request.getScope().tables().size() + ")");
             QTableMetaData table = qInstance.getTable(resolved.name());
             String         backendType = qInstance.getBackend(table.getBackendName()) == null ? null : qInstance.getBackend(table.getBackendName()).getBackendType();
 
@@ -242,6 +290,7 @@ public class BackupEngine
 
             ResumeState.TableState prior = request.getResume() == null ? null : request.getResume().table(table.getName());
 
+            long[] tableWritten = { 0 };
             List<TableArtifactWriter> writers = new ArrayList<>();
             String tablePrefix = runPrefix + "tables/" + table.getName() + "/";
             Map<TableArtifactWriter, Integer> lastArtifactIds = new LinkedHashMap<>();
@@ -277,6 +326,9 @@ public class BackupEngine
                   {
                      try
                      {
+                        tableWritten[0] += part.bytesStored();
+                        progress.bytesDone(tableBytesBefore + tableWritten[0]);
+                        progress.info("Wrote " + part.key().substring(runPrefix.length()));
                         lastArtifactIds.put(writer, catalog.insertArtifact(result.getRunId(), "TABLE_DATA", table.getName(), part.partNumber(), part.key().substring(runPrefix.length()),
                            part.format(), part.compression(), part.encrypted(), part.rows(), part.bytesRaw(), part.bytesStored(), part.sha256Stored(), part.sha256Raw(), null,
                            part.firstKey(), part.lastKey(), TableFingerprints.of(table, backendType)));
@@ -296,6 +348,8 @@ public class BackupEngine
                   if(canContinue)
                   {
                      TableArtifactWriter writer = writers.get(0);
+                     tableWritten[0] = prior.parts().stream().mapToLong(TableArtifactWriter.PartResult::bytesStored).sum();
+                     progress.bytesDone(tableBytesBefore + tableWritten[0]);
                      writer.resumeFrom(prior.parts());
                      seedRowsHash(storage, runPrefix, prior.parts(), writer);
                      lastArtifactIds.put(writer, prior.artifactIds().get(prior.artifactIds().size() - 1));
@@ -368,8 +422,31 @@ public class BackupEngine
             }
             rows += counter[0];
 
-            for(FileReference ref : references)
+            pendingFiles.write(references);
+
+            Map<String, String> keyRange = new LinkedHashMap<>();
+            keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
+            keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
+            manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
+               fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
+            done++;
+            catalogProgress(request, result, done, rows, bytesRaw, bytesStored);
+            progress.done(done).bytesDone(bytesStored);
+            progress.info(String.format(Locale.US, "%s: %,d rows, %.1f MB in %.1f s", table.getName(), counter[0],
+               (bytesStored - tableBytesBefore) / 1_000_000.0, (System.nanoTime() - tableStarted) / 1_000_000_000.0));
+            if(request.getCancelRequested().getAsBoolean())
             {
+               throw (new CancelledException());
+            }
+         }
+         progress.phase("media").total(pendingFiles.count, "files").done(0).bytesDone(bytesStored);
+         try(BufferedReader pending = pendingFiles.openReader())
+         {
+            String line;
+            while((line = pending.readLine()) != null)
+            {
+               FileReference ref = MAPPER.readValue(line, FileReference.class);
+               checkJob();
                if(indexWriter == null)
                {
                   indexWriter = new FileIndexWriter(storage, runPrefix + "files/index-00001.jsonl.gz", result.getRunKey(), fileLockMode, fileRetainUntil);
@@ -388,24 +465,16 @@ public class BackupEngine
                   case FileCopier.MISSING -> missing++;
                   default -> fileBytesRef += Math.max(0, entry.size());
                }
+               progress.done(referenced).bytesDone(bytesStored + fileBytes)
+                  .message("Media: " + copied + " copied, " + deduplicated + " already stored (" + referenced + " of " + pendingFiles.count + ")");
                if(request.getCancelRequested().getAsBoolean())
                {
                   throw (new CancelledException());
                }
             }
-
-            Map<String, String> keyRange = new LinkedHashMap<>();
-            keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
-            keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
-            manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
-               fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
-            done++;
-            catalogProgress(result, done, rows, bytesRaw, bytesStored);
-            if(request.getCancelRequested().getAsBoolean())
-            {
-               throw (new CancelledException());
-            }
          }
+
+         progress.info("Media: " + referenced + " files, " + copied + " copied, " + deduplicated + " already stored, " + missing + " missing");
       }
       finally
       {
@@ -415,10 +484,13 @@ public class BackupEngine
          }
       }
 
+      checkJob();
+      progress.phase("upload").message("Writing catalog and manifest");
       List<String> indexKeys = new ArrayList<>();
       if(indexWriter != null)
       {
          indexKeys.add("files/index-00001.jsonl.gz");
+         progress.info("Wrote files/index-00001.jsonl.gz");
          checksums.add(indexWriter.sha256Stored() + "  files/index-00001.jsonl.gz");
          catalog.insertArtifact(result.getRunId(), "FILE_OBJECTS_INDEX", null, 1, "files/index-00001.jsonl.gz", "JSONL", "GZIP", request.getTransform().encrypts(),
             indexWriter.rows(), indexWriter.bytesRaw(), indexWriter.bytesStored(), indexWriter.sha256Stored(), indexWriter.sha256Raw(), null, null, null, null);
@@ -431,7 +503,9 @@ public class BackupEngine
          {
             throw (new IllegalStateException(missing + " referenced files are missing at the source"));
          }
-         result.getWarnings().add(missing + " referenced files were missing at the source");
+         String warning = missing + " referenced files were missing at the source";
+         result.getWarnings().add(warning);
+         progress.warn(warning);
       }
 
       /////////////////////////////////////////////////////////
@@ -439,6 +513,7 @@ public class BackupEngine
       /////////////////////////////////////////////////////////
       byte[] checksumBytes = (String.join("\n", checksums) + "\n").getBytes(StandardCharsets.UTF_8);
       putBytes(storage, runPrefix + "checksums.sha256", checksumBytes, result.getRunKey(), lockMode, retainUntil);
+      progress.info("Wrote checksums.sha256");
       catalog.insertArtifact(result.getRunId(), "CHECKSUMS", null, 1, "checksums.sha256", "TEXT", "NONE", false, checksums.size(), checksumBytes.length,
          checksumBytes.length, HashingOutputStream.sha256Hex(checksumBytes), HashingOutputStream.sha256Hex(checksumBytes), null, null, null, null);
 
@@ -463,6 +538,7 @@ public class BackupEngine
       {
          throw (new IllegalStateException("The manifest was written but is not visible in the bucket"));
       }
+      progress.info("Wrote manifest.json");
       catalog.insertArtifact(result.getRunId(), "MANIFEST", null, 1, "manifest.json", "JSON", "NONE", false, manifestTables.size(), manifestBytes.length,
          manifestBytes.length, HashingOutputStream.sha256Hex(manifestBytes), HashingOutputStream.sha256Hex(manifestBytes), null, null, null, null);
 
@@ -501,6 +577,58 @@ public class BackupEngine
 
 
 
+   /*******************************************************************************
+    ** Declare the work before scope resolution. No byte estimate requires a scan.
+    *******************************************************************************/
+   public static void prepare(JobProgress progress)
+   {
+      progress.phases(JobPhase.of("prepare", "Resolve scope", 2), JobPhase.of("tables", "Back up tables", 75),
+         JobPhase.of("media", "Copy media files", 18), JobPhase.of("upload", "Write catalog and manifest", 5));
+      progress.phase("prepare").message("Resolving scope");
+   }
+
+
+
+   /*******************************************************************************
+    ** Complete or fail the current phase; the core has no phase-finish method.
+    *******************************************************************************/
+   private static void completePhase(JobProgress progress, JobPhase.Status status)
+   {
+      progress.getSnapshot().getPhases().stream().filter(p -> p.getStatus() == JobPhase.Status.RUNNING).forEach(p -> p.setStatus(status));
+      progress.flush();
+   }
+
+
+
+   /*******************************************************************************
+    ** Log only a controlled failure description, never provider or row text.
+    *******************************************************************************/
+   private static void failProgress(JobProgress progress, String message)
+   {
+      completePhase(progress, JobPhase.Status.FAILED);
+      progress.error(message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Bridge checked job interrupts through the row consumer, preserving the cause
+    ** so execute can propagate it back to the durable worker.
+    *******************************************************************************/
+   private static void checkJob()
+   {
+      try
+      {
+         JobChunkScope.checkBetweenPages();
+      }
+      catch(QException e)
+      {
+         throw (new IllegalStateException(e));
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Work out the consistency achieved.  Until the core consistent-read hook ships
     ** (QQQ core PR feature/consistent-read-transactions, next 4.1.x RC), no backend
@@ -563,6 +691,7 @@ public class BackupEngine
     *******************************************************************************/
    private void heartbeat(BackupRunRequest request, BackupRunResult result, Instant deadline, String tableName, long rowsSoFar)
    {
+      checkJob();
       Instant now = request.getClock().instant();
       if(request.getCancelRequested().getAsBoolean())
       {
@@ -579,6 +708,7 @@ public class BackupEngine
       catch(Exception e)
       {
          LOG.warn("Could not record a heartbeat", e);
+         request.getProgress().warn("Could not record a heartbeat");
       }
    }
 
@@ -587,7 +717,7 @@ public class BackupEngine
    /*******************************************************************************
     **
     *******************************************************************************/
-   private void catalogProgress(BackupRunResult result, int done, long rows, long bytesRaw, long bytesStored)
+   private void catalogProgress(BackupRunRequest request, BackupRunResult result, int done, long rows, long bytesRaw, long bytesStored)
    {
       try
       {
@@ -601,6 +731,7 @@ public class BackupEngine
       catch(Exception e)
       {
          LOG.warn("Could not record progress", e);
+         request.getProgress().warn("Could not record progress");
       }
    }
 
@@ -618,6 +749,7 @@ public class BackupEngine
       catch(Exception e)
       {
          LOG.warn("Could not finish the run record", e);
+         request.getProgress().warn("Could not finish the run record");
          result.setStatus(status);
          result.setErrorMessage(message);
       }
@@ -694,13 +826,12 @@ public class BackupEngine
 
 
    /*******************************************************************************
-    ** A message safe to store: the exception type and message, never row values (the
-    ** engine never puts values in messages), truncated.
+    ** Provider exceptions may contain credentials or row values. Persist only the
+    ** failure type; detailed diagnostics stay in the server log.
     *******************************************************************************/
    private static String scrub(Exception e)
    {
-      String message = e.getClass().getSimpleName() + ": " + e.getMessage();
-      return (message.length() > 2000 ? message.substring(0, 2000) : message);
+      return ("Backup failed (" + e.getClass().getSimpleName() + "); see server logs for details.");
    }
 
 
@@ -769,6 +900,11 @@ public class BackupEngine
             {
                throw (new StopReadingException());
             }
+            checkJob();
+            if(request.getCancelRequested().getAsBoolean())
+            {
+               throw (new CancelledException());
+            }
             for(FileReferenceProvider provider : request.getFileProviders())
             {
                references.addAll(provider.referencesIn(table, record));
@@ -854,6 +990,75 @@ public class BackupEngine
 
 
 
+   /***************************************************************************
+    ** Spool references locally between table export and media copy. Keeping the
+    ** whole run's references in heap would make memory grow with every table.
+    ** The temporary file is removed on success, failure and cancellation.
+    ***************************************************************************/
+   private static class PendingFiles implements AutoCloseable
+   {
+      private final Path path;
+      private final BufferedWriter writer;
+      private long count;
+
+
+
+      /*******************************************************************************
+       ** Create a private temporary spool, outside the stored backup artifacts.
+       *******************************************************************************/
+      PendingFiles() throws IOException
+      {
+         path = Files.createTempFile("qqq-backup-references-", ".jsonl");
+         writer = Files.newBufferedWriter(path, StandardCharsets.UTF_8);
+      }
+
+
+
+      /*******************************************************************************
+       ** Append this table's references without retaining earlier tables in memory.
+       *******************************************************************************/
+      void write(List<FileReference> references) throws IOException
+      {
+         for(FileReference reference : references)
+         {
+            writer.write(MAPPER.writeValueAsString(reference));
+            writer.newLine();
+            count++;
+         }
+      }
+
+
+
+      /*******************************************************************************
+       ** Finish the spool and stream it back during the media phase.
+       *******************************************************************************/
+      BufferedReader openReader() throws IOException
+      {
+         writer.close();
+         return (Files.newBufferedReader(path, StandardCharsets.UTF_8));
+      }
+
+
+
+      /*******************************************************************************
+       ** Always release the local references, including on exceptions.
+       *******************************************************************************/
+      @Override
+      public void close() throws IOException
+      {
+         try
+         {
+            writer.close();
+         }
+         finally
+         {
+            Files.deleteIfExists(path);
+         }
+      }
+   }
+
+
+
    /***************************************************************************
     ** The per-run file index, written as it goes: one JSON line per reference.
     ***************************************************************************/

src/main/java/com/kingsrook/qbits/backups/run/BackupRunCoordinator.java

@@ -30,6 +30,8 @@ import com.kingsrook.qbits.backups.BackupsRuntime;
 import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -85,7 +87,28 @@ public class BackupRunCoordinator
     *******************************************************************************/
    public BackupRunResult runPlan(QRecord plan, String trigger, java.util.function.Consumer<BackupRunRequest> customizer) throws QException
    {
-      BackupRunRequest request = BackupRunFactory.forPlan(plan, trigger);
+      return (runPlan(plan, trigger, JobProgress.current(), customizer));
+   }
+
+
+
+   /*******************************************************************************
+    ** Share the step reporter and reuse a durable job's run on an explicit retry.
+    ** NONE idempotency prevents automatic replay of non-atomic external writes.
+    *******************************************************************************/
+   public BackupRunResult runPlan(QRecord plan, String trigger, JobProgress progress, java.util.function.Consumer<BackupRunRequest> customizer) throws QException
+   {
+      BackupEngine.prepare(progress);
+      BackupRunRequest request;
+      try
+      {
+         request = BackupRunFactory.forPlan(plan, trigger).withProgress(progress);
+      }
+      catch(QException | RuntimeException e)
+      {
+         progress.error("Could not resolve backup scope (" + e.getClass().getSimpleName() + ")");
+         throw (new QException("Could not resolve backup scope", e));
+      }
       if(customizer != null)
       {
          customizer.accept(request);
@@ -97,21 +120,72 @@ public class BackupRunCoordinator
          BackupRunResult result = new BackupRunResult();
          result.setStatus("FAILED");
          result.setErrorMessage(refusal);
+         progress.error(refusal);
          return (result);
       }
 
-      Claim claim = claim(plan.getValueInteger("id"), null, trigger, request.getRequestedBy(), true);
+      JobChunkScope.ChunkIdentity identity = JobChunkScope.current().map(JobChunkScope::getIdentity).orElse(null);
+      Integer existingRunId = null;
+      if(identity != null)
+      {
+         String key = "job-" + identity.jobId() + "-plan-" + plan.getValueInteger("id");
+         request.withRunKey(key);
+         List<QRecord> previous = catalog.where(BackupsTables.RUN, "runKey", key);
+         if(!previous.isEmpty())
+         {
+            QRecord run = previous.get(0);
+            existingRunId = run.getValueInteger("id");
+            if(run.getValueString("manifestKey") != null)
+            {
+               progress.info("Run " + existingRunId + " already has a manifest; nothing was rewritten. Use Verify backup to recheck it.");
+               BackupRunResult result = new BackupRunResult();
+               result.setRunId(existingRunId);
+               result.setRunKey(key);
+               result.setRunPrefix(run.getValueString("runPrefix"));
+               result.setManifestKey(run.getValueString("manifestKey"));
+               result.setStatus(run.getValueString("status"));
+               return (result);
+            }
+            if(run.getValueString("runPrefix") != null)
+            {
+               // Only the same durable job can recover its interrupted RUNNING row.
+               if("RUNNING".equals(run.getValueString("status")))
+               {
+                  catalog.update(BackupsTables.RUN, existingRunId, Map.of("status", "ABANDONED"));
+                  run.setValue("status", "ABANDONED");
+               }
+               String why = BackupResumePlanner.whyNotResumable(run, plan);
+               if(why != null)
+               {
+                  progress.error(why);
+                  throw (new QException(why));
+               }
+               request.withResume(BackupResumePlanner.plan(run));
+            }
+         }
+      }
+      Claim claim = claim(plan.getValueInteger("id"), existingRunId, trigger, request.getRequestedBy(), identity == null, request.getRunKey());
       try
       {
          if(claim.outcome != Outcome.STARTED)
          {
+            progress.warn(claim.message);
+            if(identity != null)
+            {
+               if(claim.outcome == Outcome.QUEUED)
+               {
+                  catalog.update(BackupsTables.RUN, claim.runId, Map.of("status", "FAILED", "endDate", now(), "errorMessage", claim.message));
+               }
+               progress.error("Backup could not start: " + claim.message);
+               throw (new QException("Backup could not start: " + claim.message));
+            }
             return (claim.toResult());
          }
          return (execute(request, claim.runId));
       }
       finally
       {
-         if(claim.outcome == Outcome.STARTED)
+         if(claim.outcome == Outcome.STARTED && JobChunkScope.current().isEmpty())
          {
             drainQueue();
          }
@@ -133,7 +207,10 @@ public class BackupRunCoordinator
       }
       finally
       {
-         drainQueue();
+         if(JobChunkScope.current().isEmpty())
+         {
+            drainQueue();
+         }
       }
    }
 
@@ -144,6 +221,16 @@ public class BackupRunCoordinator
     ** message) when it cannot be resumed, or when a run of the plan is live.
     *******************************************************************************/
    public BackupRunResult resume(QRecord run) throws QException
+   {
+      return (resume(run, JobProgress.current()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Resume using the caller's reporter (including an inline async callback).
+    *******************************************************************************/
+   public BackupRunResult resume(QRecord run, JobProgress progress) throws QException
    {
       QRecord plan = run.getValueInteger("backupPlanId") == null ? null : catalog.get(BackupsTables.PLAN, run.getValueInteger("backupPlanId")).orElse(null);
       String  why  = BackupResumePlanner.whyNotResumable(run, plan);
@@ -156,7 +243,7 @@ public class BackupRunCoordinator
          return (refused);
       }
 
-      BackupRunRequest request = BackupRunFactory.forPlan(plan, "MANUAL");
+      BackupRunRequest request = BackupRunFactory.forPlan(plan, "MANUAL").withProgress(progress);
       Claim claim = claim(plan.getValueInteger("id"), run.getValueInteger("id"), run.getValueString("trigger"), request.getRequestedBy(), false);
       if(claim.outcome != Outcome.STARTED)
       {
@@ -173,7 +260,10 @@ public class BackupRunCoordinator
       }
       finally
       {
-         drainQueue();
+         if(JobChunkScope.current().isEmpty())
+         {
+            drainQueue();
+         }
       }
    }
 
@@ -238,7 +328,7 @@ public class BackupRunCoordinator
    /*******************************************************************************
     ** Run the engine on a claimed run row, then return its result.
     *******************************************************************************/
-   private BackupRunResult execute(BackupRunRequest request, Integer runId)
+   private BackupRunResult execute(BackupRunRequest request, Integer runId) throws QException
    {
       request.withExistingRunId(runId).withCancelRequested(() -> catalog.isCancelRequested(runId));
       return (new BackupEngine().execute(request));
@@ -250,6 +340,16 @@ public class BackupRunCoordinator
     ** The lease: insert (or reuse) the run row and decide.
     *******************************************************************************/
    private Claim claim(Integer planId, Integer existingRunId, String trigger, String requestedBy, boolean queueIfFull) throws QException
+   {
+      return (claim(planId, existingRunId, trigger, requestedBy, queueIfFull, null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Persist the job identity in the initial claim, before any external writes.
+    *******************************************************************************/
+   private Claim claim(Integer planId, Integer existingRunId, String trigger, String requestedBy, boolean queueIfFull, String runKey) throws QException
    {
       synchronized(LEASE_LOCK)
       {
@@ -261,7 +361,7 @@ public class BackupRunCoordinator
          {
             Map<String, Serializable> values = new LinkedHashMap<>();
             values.put("backupPlanId", planId);
-            values.put("runKey", "queued-" + UUID.randomUUID().toString().substring(0, 8));
+            values.put("runKey", runKey != null ? runKey : "queued-" + UUID.randomUUID().toString().substring(0, 8));
             values.put("trigger", trigger);
             values.put("requestedByUserId", requestedBy);
             values.put("status", QUEUED);

src/main/java/com/kingsrook/qbits/backups/run/BackupRunRequest.java

@@ -26,6 +26,7 @@ import com.kingsrook.qbits.backups.files.FileReferenceProvider;
 import com.kingsrook.qbits.backups.scope.PlanSettings;
 import com.kingsrook.qbits.backups.scope.ScopeResolver;
 import com.kingsrook.qbits.backups.storage.BackupStorage;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 
 
 /*******************************************************************************
@@ -54,6 +55,50 @@ public class BackupRunRequest
    private java.util.function.Function<String, BackupStorage> sourceResolver;
    private Integer                    existingRunId;
    private ResumeState                resume;
+   private JobProgress                progress = JobProgress.current();
+   private String                     runKey;
+
+
+
+   /*******************************************************************************
+    ** The reporter shared by the process, coordinator and engine.
+    *******************************************************************************/
+   public JobProgress getProgress()
+   {
+      return (progress);
+   }
+
+
+
+   /*******************************************************************************
+    ** Supply the process reporter, including its inline async callback.
+    *******************************************************************************/
+   public BackupRunRequest withProgress(JobProgress progress)
+   {
+      this.progress = progress;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** A stable identity for an explicitly retried durable job.
+    *******************************************************************************/
+   public String getRunKey()
+   {
+      return (runKey);
+   }
+
+
+
+   /*******************************************************************************
+    ** Set the durable job's backup identity before creating any artifacts.
+    *******************************************************************************/
+   public BackupRunRequest withRunKey(String runKey)
+   {
+      this.runKey = runKey;
+      return (this);
+   }
 
 
 

src/main/java/com/kingsrook/qbits/backups/schedule/BackupScheduledRunProcess.java

@@ -80,7 +80,7 @@ public class BackupScheduledRunProcess implements BackendStep
             .withInputData(new QFunctionInputMetaData().withRecordListMetaData(new QRecordListMetaData().withTableName(BackupsTables.PLAN)))
             .withCode(new QCodeReference(BackupScheduledRunProcess.class)));
       process.setPermissionRules(BackupsTables.permissionRules());
-      return (BackupsProcesses.permission(process, BackupsPermissions.RUN));
+      return (BackupsProcesses.permission(process.withSupplementalMetaData(BackupsProcesses.durability()), BackupsPermissions.RUN));
    }
 
 
@@ -147,11 +147,12 @@ public class BackupScheduledRunProcess implements BackendStep
       }
       BackupsRuntime.getAlertSink().resolve("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"));
 
-      BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
+      BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED", com.kingsrook.qqq.backend.core.jobs.progress.JobProgress.of(input), null);
       output.addValue("status", result.getStatus());
       output.addValue("runId", result.getRunId());
       if(!"SUCCEEDED".equals(result.getStatus()) && !"SUCCEEDED_WITH_WARNINGS".equals(result.getStatus()))
       {
+         BackupsProcesses.requireSuccessfulJob(result);
          LOG.info("A scheduled backup did not succeed", logPair("plan", plan.getValueString("planKey")), logPair("status", result.getStatus()), logPair("message", result.getErrorMessage()));
       }
    }

src/main/java/com/kingsrook/qbits/backups/verify/BackupVerifier.java

@@ -47,6 +47,8 @@ import com.kingsrook.qbits.backups.storage.BackupStorageException;
 import com.kingsrook.qbits.backups.storage.StoredObjectInfo;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -153,6 +155,10 @@ public class BackupVerifier
          }
          status = findings.status();
       }
+      catch(JobInterruptedException e)
+      {
+         throw (e);
+      }
       catch(VerifyException | BackupStorageException | QException e)
       {
          status = "ERROR";
@@ -188,7 +194,7 @@ public class BackupVerifier
    /*******************************************************************************
     ** STORAGE level: head every object, including every file-store object.
     *******************************************************************************/
-   void storageLevel(RunBundle bundle, VerifyFindings findings)
+   void storageLevel(RunBundle bundle, VerifyFindings findings) throws QException
    {
       BackupManifest manifest = bundle.manifest();
       BackupStorage  storage  = bundle.storage();
@@ -199,6 +205,7 @@ public class BackupVerifier
 
       for(BackupManifest.TableEntry table : manifest.tables())
       {
+         JobChunkScope.checkBetweenPages();
          for(BackupManifest.Part part : table.parts() == null ? List.<BackupManifest.Part>of() : table.parts())
          {
             findings.objectsChecked++;
@@ -231,6 +238,7 @@ public class BackupVerifier
       int missingFiles = 0;
       for(RunBundle.FileIndexEntry entry : bundle.distinctStoredFiles(findings))
       {
+         JobChunkScope.checkBetweenPages();
          findings.objectsChecked++;
          findings.filesChecked++;
          Optional<StoredObjectInfo> head = storage.head(entry.casKey());
@@ -262,6 +270,7 @@ public class BackupVerifier
 
       for(BackupManifest.TableEntry entry : manifest.tables())
       {
+         JobChunkScope.checkBetweenPages();
          QTableMetaData table = qInstance.getTable(entry.name());
          JsonlRowCodec  codec = null;
          boolean        roundTrip = false;
@@ -291,7 +300,7 @@ public class BackupVerifier
    /*******************************************************************************
     ** All parts of one table.
     *******************************************************************************/
-   private void verifyTable(RunBundle bundle, BackupManifest.TableEntry entry, JsonlRowCodec codec, boolean roundTrip, VerifyFindings findings)
+   private void verifyTable(RunBundle bundle, BackupManifest.TableEntry entry, JsonlRowCodec codec, boolean roundTrip, VerifyFindings findings) throws QException
    {
       MessageDigest rowsDigest = HashingOutputStream.newSha256();
       long          rows       = 0;
@@ -339,6 +348,11 @@ public class BackupVerifier
          }
          catch(QException e)
          {
+            JobInterruptedException interrupted = JobInterruptedException.find(e);
+            if(interrupted != null)
+            {
+               throw (interrupted);
+            }
             findings.mismatch("The part " + part.key() + " could not be read: " + e.getMessage());
          }
       }
@@ -451,6 +465,7 @@ public class BackupVerifier
 
       for(RunBundle.FileIndexEntry entry : bundle.distinctStoredFiles(findings))
       {
+         JobChunkScope.checkBetweenPages();
          QRecord fileObject = catalog.where(BackupsTables.FILE_OBJECT, "sha256", entry.sha256()).stream().filter(r -> r.getValue("deletedDate") == null).findFirst().orElse(null);
          if(fileObject != null && "PASSED".equals(fileObject.getValueString("lastVerifyStatus"))
             && fileObject.getValueInstant("lastVerifiedDate") != null && fileObject.getValueInstant("lastVerifiedDate").isAfter(cutoff))

src/main/java/com/kingsrook/qbits/backups/verify/PartScanner.java

@@ -29,6 +29,7 @@ import java.util.zip.GZIPInputStream;
 import com.kingsrook.qbits.backups.codec.HashingOutputStream;
 import com.kingsrook.qbits.backups.manifest.BackupManifest;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
 
 
 /*******************************************************************************
@@ -80,6 +81,7 @@ final class PartScanner
     *******************************************************************************/
    static Scan scan(RunBundle bundle, BackupManifest.Part part, LineConsumer consumer) throws IOException, QException
    {
+      JobChunkScope.checkBetweenPages();
       MessageDigest storedDigest = HashingOutputStream.newSha256();
       MessageDigest rawDigest    = HashingOutputStream.newSha256();
       long          lines        = 0;
@@ -102,6 +104,10 @@ final class PartScanner
                String         line;
                while((line = reader.readLine()) != null)
                {
+                  if(lines % 1000 == 0)
+                  {
+                     JobChunkScope.checkBetweenPages();
+                  }
                   lines++;
                   consumer.accept(line);
                }

src/main/resources/help/qbit-backups.yaml

@@ -1,58 +1,782 @@
 bundle: qbit-backups
-version: 1
+version: ${project.version}
 layer: QBIT
-# Help for the Backups app and every backups_* table, in the QQQ help-bundle format
-# (qqq#1198). Each table entry says what the table is for and how its rows get
-# created, so an empty table explains itself. BackupsHelp applies it until the core
-# HelpBundles registry ships in a QQQ release (STAND-IN(core=#1198)).
-# Summaries stay under 120 characters (BackupsTablesGuardrailTest).
 entries:
-  - key: app:backups;slot:help
-    summary: "Back up, verify and restore the application's data to S3-compatible storage."
-    content: |
-      The Setup card on the home lists what a working backup needs (a plan, a storage bucket, a schedule, verification) with the next step for each. A new installation has no plan: run Create default plan on Backup plans, which asks only for the storage bucket alias, or add a plan with New. Everything else (runs, artifacts, verifications, restores) is written by the backups themselves.
+- key: table:backups_plan;field:name
+  summary: 'The display name of the backup plan.'
+- key: table:backups_plan;field:description
+  summary: 'Your notes about the purpose and coverage of this plan.'
+- key: table:backups_plan;field:planKey
+  summary: 'A stable short name used in the storage paths of this plan’s backups.'
+  content: |
+    Choose a stable lowercase name using letters, digits, dashes or underscores, within {{maxLength:backups_plan.planKey}} characters. This name becomes part of stored object paths. It cannot change after the plan has run. If you need a different storage identity, create another plan and keep the existing plan and its run history available for recovery.
+- key: table:backups_plan;field:isActive
+  summary: 'Whether this plan is active for scheduled execution.'
+- key: table:backups_plan;field:scopeType
+  summary: 'How this plan selects the data to back up.'
+  content: |
+    Scope and table roles control which data a plan selects. Use **Backup Plan Scope** rules to include or exclude particular targets and narrow records when needed. Review the resolved selection with **Run backup now** before continuing, especially after application metadata changes. Excluded data will not be available from this backup even if the run itself completes successfully.
+- key: table:backups_plan;field:includeRoles
+  summary: 'The comma-separated table roles included by the plan.'
+  content: |
+    Scope and table roles control which data a plan selects. Use **Backup Plan Scope** rules to include or exclude particular targets and narrow records when needed. Review the resolved selection with **Run backup now** before continuing, especially after application metadata changes. Excluded data will not be available from this backup even if the run itself completes successfully.
+- key: table:backups_plan;field:pruneBucketAlias
+  summary: 'The storage connection alias used when removing expired backups.'
+  content: |
+    This alias selects the configured storage access used for pruning. It can separate deletion access from the destination used to write backups. Enter a storage alias rather than connection credentials, and check it before requesting retention cleanup. **Prune per retention** previews proposed deletions; locked objects and retained backups remain subject to their protection rules.
+- key: table:backups_plan;field:prefix
+  summary: 'The storage path prefix under which this plan writes backups.'
+- key: table:backups_plan;field:critical
+  summary: 'Whether this plan is marked as critical for backup monitoring.'
+  content: |
+    Monitoring compares successful backup and verification history with the plan’s expected ages. Use the health result to decide which **Backup Run** or **Verification** to inspect. A recent run can still need verification, and a failed run can leave an older successful backup in place. Adjust expectations to your recovery needs rather than treating a changed threshold as a repair.
+- key: table:backups_plan;field:format
+  summary: 'The file format used for exported table records.'
+  content: |
+    The default is {{default:backups_plan.format}} and the metadata choices are {{values:backups_format}}. JSONL supports structured record content, while CSV provides a tabular representation; both can be requested together. Parquet is listed but is not implemented in this version and is rejected by validation. Review the backup’s artifacts afterward to see the files actually produced.
+- key: table:backups_plan;field:compression
+  summary: 'The compression applied to exported backup data.'
+  content: |
+    The default is {{default:backups_plan.compression}} and the metadata choices are {{values:backups_compression}}. Use GZIP to compress exported data or NONE to leave it uncompressed. ZSTD is listed but is not implemented in this version. Compare raw and stored byte counts on **Backup Run** or **Artifacts** when assessing the resulting storage size.
+- key: table:backups_plan;field:encryptionMode
+  summary: 'The encryption mode requested for backup content.'
+  content: |
+    Review these settings together before including sensitive data. Encryption support depends on the configured backup implementation; unsupported client encryption is rejected. A key reference names a key and must never contain its value. Encrypted-only inclusion requires client encryption, while including sensitive fields in the clear requires backup administrator permission. Check the preview and validation messages before running the plan.
+- key: table:backups_plan;field:encryptionKeyRef
+  summary: 'The name of the encryption key to resolve, never the key value.'
+  content: |
+    Review these settings together before including sensitive data. Encryption support depends on the configured backup implementation; unsupported client encryption is rejected. A key reference names a key and must never contain its value. Encrypted-only inclusion requires client encryption, while including sensitive fields in the clear requires backup administrator permission. Check the preview and validation messages before running the plan.
+- key: table:backups_plan;field:sensitiveFieldPolicy
+  summary: 'How fields marked sensitive are handled in this backup.'
+  content: |
+    Review these settings together before including sensitive data. Encryption support depends on the configured backup implementation; unsupported client encryption is rejected. A key reference names a key and must never contain its value. Encrypted-only inclusion requires client encryption, while including sensitive fields in the clear requires backup administrator permission. Check the preview and validation messages before running the plan.
+- key: table:backups_plan;field:heavyFieldPolicy
+  summary: 'How large field values are included or omitted in this backup.'
+  content: |
+    Choose how large field values should be represented in the backup. The default is {{default:backups_plan.heavyFieldPolicy}} and the configured choices are {{values:backups_heavy_policy}}. Including values inline, placing them in separate objects and excluding them have different recovery implications. Review the resulting artifacts and verify a backup before relying on it to recover records with large content.
+- key: table:backups_plan;field:fileObjectPolicy
+  summary: 'Whether referenced files are copied or only referenced in the backup.'
+  content: |
+    File handling is separate from exporting database records. Copying referenced files keeps their content with the backup; reference-only handling does not make an independent copy. The missing-file policy determines whether an unavailable source file produces a warning or fails the run. Check copied and missing file counts on **Backup Run**, and verify media recovery as well as record recovery.
+- key: table:backups_plan;field:fileMissingPolicy
+  summary: 'Whether a missing referenced file warns or fails the backup.'
+  content: |
+    File handling is separate from exporting database records. Copying referenced files keeps their content with the backup; reference-only handling does not make an independent copy. The missing-file policy determines whether an unavailable source file produces a warning or fails the run. Check copied and missing file counts on **Backup Run**, and verify media recovery as well as record recovery.
+- key: table:backups_plan;field:consistency
+  summary: 'The requested consistency of reads across the backed-up data.'
+  content: |
+    The default is {{default:backups_plan.consistency}} and the available requests are {{values:backups_consistency}}. A shared snapshot helps keep related records consistent while the application changes. Support depends on the source backend. After a run, inspect **{{label:backups_run.consistencyAchieved}}** rather than assuming the requested level was achieved. Review warnings before relying on the backup for recovery.
+- key: table:backups_plan;field:useReadReplica
+  summary: 'Read replica requests are unavailable and are rejected when you save the plan.'
+- key: table:backups_plan;field:incrementalMode
+  summary: 'Whether the plan requests a full backup or changed records only.'
+  content: |
+    The default is {{default:backups_plan.incrementalMode}}. This version supports full backups; requesting only records changed since the previous backup is rejected by validation. Keep the plan on the supported full mode and use scope rules when you need to limit selected data. Review **Backup Run** and its artifacts to confirm the resulting coverage.
+- key: table:backups_plan;field:nativeDump
+  summary: 'The native database dump mode requested in addition to logical data.'
+  content: |
+    The default is {{default:backups_plan.nativeDump}}. Native database dumps are not implemented in this version, so selecting a native dump mode is rejected. Use the supported logical backup formats to save table records and referenced files. Those backups have their own manifests, verification and restore workflow; do not assume they contain a database-engine dump.
+- key: table:backups_plan;field:cronTimeZoneId
+  summary: 'The time zone used to interpret this plan’s schedule.'
+- key: table:backups_plan;field:scheduledJobId
+  summary: 'An unused legacy scheduler reference; you do not need to set it.'
+- key: table:backups_plan;field:slaHours
+  summary: 'The maximum expected age of a successful backup, in hours.'
+  content: |
+    Monitoring compares successful backup and verification history with the plan’s expected ages. Use the health result to decide which **Backup Run** or **Verification** to inspect. A recent run can still need verification, and a failed run can leave an older successful backup in place. Adjust expectations to your recovery needs rather than treating a changed threshold as a repair.
+- key: table:backups_plan;field:verifySlaDays
+  summary: 'The maximum expected age of a verified backup, in days.'
+  content: |
+    Monitoring compares successful backup and verification history with the plan’s expected ages. Use the health result to decide which **Backup Run** or **Verification** to inspect. A recent run can still need verification, and a failed run can leave an older successful backup in place. Adjust expectations to your recovery needs rather than treating a changed threshold as a repair.
+- key: table:backups_plan;field:verifyFileSamplePercent
+  summary: 'The percentage of referenced files selected for verification.'
+  content: |
+    Verification has its own depth, file selection and schedule. Checking stored objects is different from comparing decoded content or testing a restore. Review **Verification** to see which level actually ran and what it checked. Scheduling depends on the host integration; a saved expression alone is not evidence of a completed check. Use **Verify backup** for a direct content check.
+- key: table:backups_plan;field:verifyCronExpression
+  summary: 'The schedule expression for automatic verification.'
+  content: |
+    Verification has its own depth, file selection and schedule. Checking stored objects is different from comparing decoded content or testing a restore. Review **Verification** to see which level actually ran and what it checked. Scheduling depends on the host integration; a saved expression alone is not evidence of a completed check. Use **Verify backup** for a direct content check.
+- key: table:backups_plan;field:verifyLevel
+  summary: 'The depth of checking requested for automatic verification.'
+  content: |
+    Verification has its own depth, file selection and schedule. Checking stored objects is different from comparing decoded content or testing a restore. Review **Verification** to see which level actually ran and what it checked. Scheduling depends on the host integration; a saved expression alone is not evidence of a completed check. Use **Verify backup** for a direct content check.
+- key: table:backups_plan;field:retainDaily
+  summary: 'The number of daily backup periods retained by pruning.'
+  content: |
+    Retention selects backups to keep across daily, weekly, monthly and yearly periods, with a minimum number protected from pruning. Changing these settings does not itself prove which objects will be deleted. Use **Prune per retention** and review its proposed deletions before continuing. Storage locks can prevent deletion even when a backup falls outside the requested retention periods.
+- key: table:backups_plan;field:retainWeekly
+  summary: 'The number of weekly backup periods retained by pruning.'
+  content: |
+    Retention selects backups to keep across daily, weekly, monthly and yearly periods, with a minimum number protected from pruning. Changing these settings does not itself prove which objects will be deleted. Use **Prune per retention** and review its proposed deletions before continuing. Storage locks can prevent deletion even when a backup falls outside the requested retention periods.
+- key: table:backups_plan;field:retainMonthly
+  summary: 'The number of monthly backup periods retained by pruning.'
+  content: |
+    Retention selects backups to keep across daily, weekly, monthly and yearly periods, with a minimum number protected from pruning. Changing these settings does not itself prove which objects will be deleted. Use **Prune per retention** and review its proposed deletions before continuing. Storage locks can prevent deletion even when a backup falls outside the requested retention periods.
+- key: table:backups_plan;field:retainYearly
+  summary: 'The number of yearly backup periods retained by pruning.'
+  content: |
+    Retention selects backups to keep across daily, weekly, monthly and yearly periods, with a minimum number protected from pruning. Changing these settings does not itself prove which objects will be deleted. Use **Prune per retention** and review its proposed deletions before continuing. Storage locks can prevent deletion even when a backup falls outside the requested retention periods.
+- key: table:backups_plan;field:retainMinimum
+  summary: 'The minimum number of successful backups protected from pruning.'
+  content: |
+    Retention selects backups to keep across daily, weekly, monthly and yearly periods, with a minimum number protected from pruning. Changing these settings does not itself prove which objects will be deleted. Use **Prune per retention** and review its proposed deletions before continuing. Storage locks can prevent deletion even when a backup falls outside the requested retention periods.
+- key: table:backups_plan;field:objectLockMode
+  summary: 'The storage retention-lock mode requested for backup objects.'
+  content: |
+    Object locking asks the storage service to protect backup objects from deletion for the specified period. The selected destination must support locking, and a non-disabled mode requires a duration. These protections can prevent pruning from removing objects. Use **Test destination** to inspect capability information and review the retention requirements before choosing a locked destination.
+- key: table:backups_plan;field:objectLockDays
+  summary: 'How many days backup objects should remain locked in storage.'
+  content: |
+    Object locking asks the storage service to protect backup objects from deletion for the specified period. The selected destination must support locking, and a non-disabled mode requires a duration. These protections can prevent pruning from removing objects. Use **Test destination** to inspect capability information and review the retention requirements before choosing a locked destination.
+- key: table:backups_plan;field:maxDurationMinutes
+  summary: 'The configured backup duration limit in minutes.'
+- key: table:backups_plan;field:pageSize
+  summary: 'The number of records requested per backup read page.'
+- key: table:backups_plan;field:targetPartBytes
+  summary: 'The target size in bytes for each exported data part.'
+- key: table:backups_plan;field:ownerUserId
+  summary: 'The user whose access is checked for this plan’s scheduled backups.'
+  content: |
+    Scheduled backups check that the plan owner can read the selected tables. An active plan can run with broad data access, so saving an active plan requires backup administrator permission. Keep the owner and scope aligned as access changes. If scheduling reports an invalid owner, correct the owner or permissions and inspect **Backup Run** for subsequent results.
+- key: table:backups_plan;field:lastRunId
+  summary: 'The latest backup run associated with this plan.'
+- key: table:backups_plan;field:lastSuccessRunId
+  summary: 'The latest successful backup run for this plan.'
+- key: table:backups_plan;field:lastSuccessDate
+  summary: 'When this plan last completed a successful backup.'
+- key: table:backups_plan;field:lastVerifiedRunId
+  summary: 'The most recently verified backup run for this plan.'
+- key: table:backups_plan;field:lastVerifiedDate
+  summary: 'When a backup for this plan was last verified.'
+- key: table:backups_plan;field:slaStatus
+  summary: 'The current assessment of this plan’s backup health and timeliness.'
+  content: |
+    Monitoring compares successful backup and verification history with the plan’s expected ages. Use the health result to decide which **Backup Run** or **Verification** to inspect. A recent run can still need verification, and a failed run can leave an older successful backup in place. Adjust expectations to your recovery needs rather than treating a changed threshold as a repair.
+- key: table:backups_plan_scope;field:backupPlanId
+  summary: 'The backup plan this scope rule belongs to.'
+- key: table:backups_plan_scope;field:action
+  summary: 'Whether matching data is included in or excluded from the plan.'
+- key: table:backups_plan_scope;field:selectorType
+  summary: 'Whether the selector identifies a table, application, component or table role.'
+  content: |
+    Use these settings to describe the data matched by this scope rule. The selector identifies a target; a structured filter can narrow its records, child inclusion can expand related records, and exclusions omit fields. Review the resolved result through **Run backup now** before continuing. A successful backup cannot recover records or fields that its scope left out.
+- key: table:backups_plan_scope;field:selectorValue
+  summary: 'The name matched by the chosen selector type.'
+  content: |
+    Use these settings to describe the data matched by this scope rule. The selector identifies a target; a structured filter can narrow its records, child inclusion can expand related records, and exclusions omit fields. Review the resolved result through **Run backup now** before continuing. A successful backup cannot recover records or fields that its scope left out.
+- key: table:backups_plan_scope;field:filterJson
+  summary: 'A structured record filter that narrows the selected data.'
+  content: |
+    Use these settings to describe the data matched by this scope rule. The selector identifies a target; a structured filter can narrow its records, child inclusion can expand related records, and exclusions omit fields. Review the resolved result through **Run backup now** before continuing. A successful backup cannot recover records or fields that its scope left out.
+- key: table:backups_plan_scope;field:includeAssociatedChildren
+  summary: 'Whether the selection also follows associated child records.'
+  content: |
+    Use these settings to describe the data matched by this scope rule. The selector identifies a target; a structured filter can narrow its records, child inclusion can expand related records, and exclusions omit fields. Review the resolved result through **Run backup now** before continuing. A successful backup cannot recover records or fields that its scope left out.
+- key: table:backups_plan_scope;field:fieldExclusions
+  summary: 'The fields to omit from data selected by this scope rule.'
+  content: |
+    Use these settings to describe the data matched by this scope rule. The selector identifies a target; a structured filter can narrow its records, child inclusion can expand related records, and exclusions omit fields. Review the resolved result through **Run backup now** before continuing. A successful backup cannot recover records or fields that its scope left out.
+- key: table:backups_run;field:backupPlanId
+  summary: 'The plan that produced this backup.'
+- key: table:backups_run;field:runKey
+  summary: 'The unique storage reference for this backup execution.'
+- key: table:backups_run;field:trigger
+  summary: 'What caused this backup to run.'
+- key: table:backups_run;field:requestedByUserId
+  summary: 'The user who requested this operation.'
+- key: table:backups_run;field:status
+  summary: 'The recorded progress or final state of this operation.'
+  content: |
+    The run states are {{values:backups_run_status}}. Check the state alongside completed table counts, warnings and errors. A run that succeeded with warnings needs review, especially if referenced files were missing. Completion and verification are separate: inspect **Verification** before relying on the saved content. A cancellation request can take time to reach a final state.
+- key: table:backups_run;field:startDate
+  summary: 'When this operation started.'
+- key: table:backups_run;field:endDate
+  summary: 'When this operation ended.'
+- key: table:backups_run;field:durationMs
+  summary: 'The elapsed backup time in milliseconds.'
+- key: table:backups_run;field:heartbeatDate
+  summary: 'When the running backup last reported activity.'
+- key: table:backups_run;field:tablesPlanned
+  summary: 'The number of tables selected for this backup.'
+- key: table:backups_run;field:tablesDone
+  summary: 'The number of tables completed by this backup.'
+- key: table:backups_run;field:rowsTotal
+  summary: 'The total number of records exported.'
+- key: table:backups_run;field:bytesRaw
+  summary: 'The data size in bytes before storage encoding.'
+- key: table:backups_run;field:bytesStored
+  summary: 'The data size in bytes as stored.'
+- key: table:backups_run;field:filesReferenced
+  summary: 'The number of referenced files encountered during the backup.'
+- key: table:backups_run;field:filesCopied
+  summary: 'The number of files copied into backup storage.'
+- key: table:backups_run;field:filesDeduplicated
+  summary: 'The number of files whose existing backup copies were reused.'
+- key: table:backups_run;field:filesMissing
+  summary: 'The number of referenced files that could not be found.'
+- key: table:backups_run;field:fileBytesCopied
+  summary: 'The total bytes of file content copied into backup storage.'
+- key: table:backups_run;field:fileBytesReferenced
+  summary: 'The total bytes of file content referenced by this backup.'
+- key: table:backups_run;field:consistencyAchieved
+  summary: 'The read consistency actually achieved by this run.'
+  content: |
+    These values describe the saved backup rather than the current plan. The manifest records content and integrity information, while consistency and security scope describe how records were read. Use them when comparing backup coverage or investigating **Verification** failures. A user-selected export can contain fewer records than a broader scheduled plan because the user’s record access still applies.
+- key: table:backups_run;field:snapshotId
+  summary: 'The database snapshot identifier recorded for this backup.'
+- key: table:backups_run;field:storageBucketAlias
+  summary: 'The configured storage alias holding this backup content.'
+- key: table:backups_run;field:runPrefix
+  summary: 'The storage path prefix for this run’s objects.'
+- key: table:backups_run;field:manifestKey
+  summary: 'The storage path of the manifest describing this backup.'
+  content: |
+    These values describe the saved backup rather than the current plan. The manifest records content and integrity information, while consistency and security scope describe how records were read. Use them when comparing backup coverage or investigating **Verification** failures. A user-selected export can contain fewer records than a broader scheduled plan because the user’s record access still applies.
+- key: table:backups_run;field:manifestSha256
+  summary: 'The checksum used to identify and check the backup manifest.'
+  content: |
+    These values describe the saved backup rather than the current plan. The manifest records content and integrity information, while consistency and security scope describe how records were read. Use them when comparing backup coverage or investigating **Verification** failures. A user-selected export can contain fewer records than a broader scheduled plan because the user’s record access still applies.
+- key: table:backups_run;field:qqqVersion
+  summary: 'The framework version recorded when this backup was made.'
+- key: table:backups_run;field:appVersion
+  summary: 'The application version recorded when this backup was made.'
+- key: table:backups_run;field:instanceName
+  summary: 'The application instance from which the backup was made.'
+- key: table:backups_run;field:schemaFingerprint
+  summary: 'A fingerprint of the data structure recorded in the backup.'
+  content: |
+    These values describe the saved backup rather than the current plan. The manifest records content and integrity information, while consistency and security scope describe how records were read. Use them when comparing backup coverage or investigating **Verification** failures. A user-selected export can contain fewer records than a broader scheduled plan because the user’s record access still applies.
+- key: table:backups_run;field:securityScope
+  summary: 'Whether the backup reflects user-limited access or a broader plan execution.'
+  content: |
+    These values describe the saved backup rather than the current plan. The manifest records content and integrity information, while consistency and security scope describe how records were read. Use them when comparing backup coverage or investigating **Verification** failures. A user-selected export can contain fewer records than a broader scheduled plan because the user’s record access still applies.
+- key: table:backups_run;field:encryptionKeyRef
+  summary: 'The key reference recorded for the backup, without the key value.'
+- key: table:backups_run;field:encryptionKeyFingerprint
+  summary: 'A fingerprint identifying the encryption key without exposing it.'
+- key: table:backups_run;field:retentionClass
+  summary: 'The retention period category assigned to this backup.'
+- key: table:backups_run;field:lockedUntil
+  summary: 'The time until which storage retention protects this content from deletion.'
+- key: table:backups_run;field:prunedDate
+  summary: 'When this backup was pruned from storage.'
+- key: table:backups_run;field:prunedByUserId
+  summary: 'The user recorded as pruning this backup.'
+- key: table:backups_run;field:verificationStatus
+  summary: 'The latest recorded level and outcome of verification for this backup.'
+  content: |
+    The recorded states are {{values:backups_verification_status}}. Storage checks, content checks and test restores provide different evidence; a storage-only result does not establish that every record can be restored. Open the related **Verification** for its level, counts and diagnostics. This status describes checking after backup creation and is separate from the backup run’s own completion status.
+- key: table:backups_run;field:warningCount
+  summary: 'The number of warnings recorded during the backup.'
+- key: table:backups_run;field:errorMessage
+  summary: 'The error message recorded for this operation.'
+- key: table:backups_run;field:errorDetail
+  summary: 'Additional error information recorded for the failed backup.'
+- key: table:backups_artifact;field:backupRunId
+  summary: 'The backup run this record belongs to.'
+- key: table:backups_artifact;field:artifactType
+  summary: 'The kind of backup content stored in this artifact.'
+- key: table:backups_artifact;field:tableName
+  summary: 'The source table described by this record.'
+- key: table:backups_artifact;field:partNumber
+  summary: 'The sequence number of this part within the exported table data.'
+- key: table:backups_artifact;field:objectKey
+  summary: 'The path identifying the object within its storage bucket.'
+- key: table:backups_artifact;field:format
+  summary: 'The format recorded for this artifact’s content.'
+- key: table:backups_artifact;field:compression
+  summary: 'The compression recorded for this artifact’s content.'
+- key: table:backups_artifact;field:encrypted
+  summary: 'Whether this stored object is recorded as encrypted.'
+- key: table:backups_artifact;field:rowCount
+  summary: 'The number of records written to this artifact.'
+- key: table:backups_artifact;field:bytesRaw
+  summary: 'The data size in bytes before storage encoding.'
+- key: table:backups_artifact;field:bytesStored
+  summary: 'The data size in bytes as stored.'
+- key: table:backups_artifact;field:sha256Stored
+  summary: 'The checksum of the bytes as stored in the bucket.'
+- key: table:backups_artifact;field:sha256Raw
+  summary: 'The checksum of the content before storage encoding.'
+- key: table:backups_artifact;field:rowsHash
+  summary: 'The checksum representing the exported records.'
+- key: table:backups_artifact;field:firstKey
+  summary: 'The first record key recorded for this data part.'
+- key: table:backups_artifact;field:lastKey
+  summary: 'The last record key recorded for this data part.'
+- key: table:backups_artifact;field:tableFingerprint
+  summary: 'A fingerprint of the table structure used by this artifact.'
+- key: table:backups_artifact;field:status
+  summary: 'The recorded progress or final state of this operation.'
+- key: table:backups_artifact;field:storageVersionId
+  summary: 'The storage service’s version identifier for the saved object.'
+- key: table:backups_verification;field:backupRunId
+  summary: 'The backup run this record belongs to.'
+- key: table:backups_verification;field:level
+  summary: 'The depth of verification performed on this backup.'
+  content: |
+    The levels are {{values:backups_verify_level}}. Storage verification checks saved objects, content verification reads and compares backup data, and a test restore exercises recovery in a temporary destination. Review the result and detail to see what actually passed. A deeper check can provide more recovery evidence, but the level alone does not mean the check succeeded.
+- key: table:backups_verification;field:trigger
+  summary: 'What caused this verification to run.'
+- key: table:backups_verification;field:requestedByUserId
+  summary: 'The user who requested this operation.'
+- key: table:backups_verification;field:startDate
+  summary: 'When this operation started.'
+- key: table:backups_verification;field:endDate
+  summary: 'When this operation ended.'
+- key: table:backups_verification;field:status
+  summary: 'The recorded progress or final state of this operation.'
+  content: |
+    The possible results are {{values:backups_verify_result}}. Review the level, mismatch count and details together with the result. A check can pass with warnings or fail before completing all planned work. Use the linked **Backup Run** and its artifacts to investigate; changing a plan’s settings does not repair the stored content of an earlier backup.
+- key: table:backups_verification;field:tablesChecked
+  summary: 'The number of tables examined during verification.'
+- key: table:backups_verification;field:rowsChecked
+  summary: 'The number of records examined during verification.'
+- key: table:backups_verification;field:mismatches
+  summary: 'The number of discrepancies found during verification.'
+- key: table:backups_verification;field:scratchTarget
+  summary: 'The temporary destination recorded for a test restore.'
+- key: table:backups_verification;field:detailJson
+  summary: 'The structured results and diagnostics from verification.'
+- key: table:backups_restore_run;field:sourceBackupRunId
+  summary: 'The backup run used as the source for recovery.'
+- key: table:backups_restore_run;field:requestedByUserId
+  summary: 'The user who requested this operation.'
+- key: table:backups_restore_run;field:startDate
+  summary: 'When this operation started.'
+- key: table:backups_restore_run;field:endDate
+  summary: 'When this operation ended.'
+- key: table:backups_restore_run;field:status
+  summary: 'The recorded progress or final state of this operation.'
+- key: table:backups_restore_run;field:mode
+  summary: 'How the restore handles new and existing destination records.'
+  content: |
+    Restore mode controls how destination records are handled, while matching and identifier strategy control how saved records connect to them. Review these choices before execution: replacement can delete existing records and requires explicit confirmation. This row records the choices made for that run. Inspect the table results and any pre-restore backup before responding to an unexpected recovery outcome.
+- key: table:backups_restore_run;field:idStrategy
+  summary: 'Whether original record identifiers are preserved or replaced during recovery.'
+  content: |
+    Restore mode controls how destination records are handled, while matching and identifier strategy control how saved records connect to them. Review these choices before execution: replacement can delete existing records and requires explicit confirmation. This row records the choices made for that run. Inspect the table results and any pre-restore backup before responding to an unexpected recovery outcome.
+- key: table:backups_restore_run;field:matchOn
+  summary: 'The key used to match saved records to destination records.'
+  content: |
+    Restore mode controls how destination records are handled, while matching and identifier strategy control how saved records connect to them. Review these choices before execution: replacement can delete existing records and requires explicit confirmation. This row records the choices made for that run. Inspect the table results and any pre-restore backup before responding to an unexpected recovery outcome.
+- key: table:backups_restore_run;field:targetInstanceName
+  summary: 'The application instance receiving the restored data.'
+- key: table:backups_restore_run;field:scopeSummary
+  summary: 'A description of the tables and records selected for recovery.'
+- key: table:backups_restore_run;field:preRestoreBackupRunId
+  summary: 'The backup taken of the destination before this restore.'
+- key: table:backups_restore_run;field:rowsInserted
+  summary: 'The number of records inserted by the restore.'
+- key: table:backups_restore_run;field:rowsUpdated
+  summary: 'The number of existing records updated by the restore.'
+- key: table:backups_restore_run;field:rowsUnchanged
+  summary: 'The number of matched records that needed no changes.'
+- key: table:backups_restore_run;field:rowsDeleted
+  summary: 'The number of records deleted by the restore.'
+- key: table:backups_restore_run;field:rowsSkipped
+  summary: 'The number of records skipped by the restore.'
+- key: table:backups_restore_run;field:rowsErrored
+  summary: 'The number of records that encountered restore errors.'
+- key: table:backups_restore_run;field:filesRestored
+  summary: 'The number of referenced files restored from backup storage.'
+- key: table:backups_restore_run;field:filesAlreadyPresent
+  summary: 'The number of files already present at the destination.'
+- key: table:backups_restore_run;field:filesConflicting
+  summary: 'The number of files that conflict with existing destination content.'
+- key: table:backups_restore_run;field:filesMissingFromBackup
+  summary: 'The number of required files absent from the backup.'
+- key: table:backups_restore_run;field:fileBytesRestored
+  summary: 'The total bytes of file content restored.'
+- key: table:backups_restore_run;field:sequencesResynced
+  summary: 'The number of automatic record-number sequences adjusted after recovery.'
+- key: table:backups_restore_run;field:automationsSuppressed
+  summary: 'Whether record automations were suppressed during this restore.'
+  content: |
+    These recorded flags explain whether restoring records was allowed to trigger automations or publish change events. Restore options normally suppress automations and do not publish events, but the selected options determine the actual run. Use these values when investigating missing or unexpected downstream activity, and inspect the **Restore Run** results before manually repeating any follow-up work.
+- key: table:backups_restore_run;field:esbSuppressed
+  summary: 'Whether change-event publication was suppressed during this restore.'
+  content: |
+    These recorded flags explain whether restoring records was allowed to trigger automations or publish change events. Restore options normally suppress automations and do not publish events, but the selected options determine the actual run. Use these values when investigating missing or unexpected downstream activity, and inspect the **Restore Run** results before manually repeating any follow-up work.
+- key: table:backups_restore_run;field:confirmationText
+  summary: 'The confirmation entered for a replacement restore.'
+  content: |
+    Restore mode controls how destination records are handled, while matching and identifier strategy control how saved records connect to them. Review these choices before execution: replacement can delete existing records and requires explicit confirmation. This row records the choices made for that run. Inspect the table results and any pre-restore backup before responding to an unexpected recovery outcome.
+- key: table:backups_restore_run;field:errorMessage
+  summary: 'The error message recorded for this operation.'
+- key: table:backups_restore_run_table;field:backupRestoreRunId
+  summary: 'The restore run this table result belongs to.'
+- key: table:backups_restore_run_table;field:tableName
+  summary: 'The source table described by this record.'
+- key: table:backups_restore_run_table;field:targetTableName
+  summary: 'The destination table that received the restored records.'
+- key: table:backups_restore_run_table;field:rowsInserted
+  summary: 'The number of records inserted by the restore.'
+- key: table:backups_restore_run_table;field:rowsUpdated
+  summary: 'The number of existing records updated by the restore.'
+- key: table:backups_restore_run_table;field:rowsUnchanged
+  summary: 'The number of matched records that needed no changes.'
+- key: table:backups_restore_run_table;field:rowsDeleted
+  summary: 'The number of records deleted by the restore.'
+- key: table:backups_restore_run_table;field:rowsSkipped
+  summary: 'The number of records skipped by the restore.'
+- key: table:backups_restore_run_table;field:rowsErrored
+  summary: 'The number of records that encountered restore errors.'
+- key: table:backups_restore_run_table;field:schemaStatus
+  summary: 'How the saved table structure compares with the destination structure.'
+  content: |
+    The comparison states are {{values:backups_schema_status}}. Differences can arise when the application’s fields change after a backup was taken. Review **{{label:backups_restore_run_table.unmappedFields}}** and the restore’s record counts before assuming all saved data was recovered. The recorded comparison concerns this table; inspect the parent **Restore Run** for the overall outcome and any errors.
+- key: table:backups_restore_run_table;field:unmappedFields
+  summary: 'The saved fields that could not be mapped to destination fields.'
+- key: table:backups_restore_run_table;field:sequenceResynced
+  summary: 'Whether automatic record numbering was adjusted for this table.'
+- key: table:backups_restore_run_table;field:status
+  summary: 'The recorded progress or final state of this operation.'
+- key: table:backups_file_object;field:sha256
+  summary: 'The checksum identifying this saved file’s content.'
+- key: table:backups_file_object;field:sizeBytes
+  summary: 'The file’s recorded size in bytes.'
+- key: table:backups_file_object;field:contentType
+  summary: 'The media type recorded for this file.'
+- key: table:backups_file_object;field:storageBucketAlias
+  summary: 'The configured storage alias holding this backup content.'
+- key: table:backups_file_object;field:objectKey
+  summary: 'The path identifying the object within its storage bucket.'
+- key: table:backups_file_object;field:encrypted
+  summary: 'Whether this stored object is recorded as encrypted.'
+- key: table:backups_file_object;field:encryptionKeyFingerprint
+  summary: 'A fingerprint identifying the encryption key without exposing it.'
+- key: table:backups_file_object;field:firstRunId
+  summary: 'The first backup run recorded as using this file.'
+- key: table:backups_file_object;field:lastReferencedRunId
+  summary: 'The latest backup run recorded as using this file.'
+- key: table:backups_file_object;field:referenceCount
+  summary: 'The number of backup references recorded for this file.'
+- key: table:backups_file_object;field:lastVerifiedDate
+  summary: 'When this file was last verified.'
+- key: table:backups_file_object;field:lastVerifyStatus
+  summary: 'The latest recorded verification outcome for this file.'
+- key: table:backups_file_object;field:lockedUntil
+  summary: 'The time until which storage retention protects this content from deletion.'
+- key: table:backups_file_object;field:deletedDate
+  summary: 'When this file’s deletion was recorded.'
+- key: table:backups_id_map;field:restoreRunId
+  summary: 'The restore run that assigned the new identifier.'
+- key: table:backups_id_map;field:tableName
+  summary: 'The table containing the remapped record.'
+- key: table:backups_id_map;field:oldId
+  summary: 'The record identifier in the source backup.'
+- key: table:backups_id_map;field:newId
+  summary: 'The record identifier assigned at the destination.'
+- key: process:backupRunPlan;slot:help
+  summary: 'Preview a backup plan’s scope and run it when you continue.'
+  content: |
+    Use **Run backup now** on a **Backup Plan** when you need a fresh copy before a change or want to check the plan. Review the resolved scope and any warnings first; nothing is written until you continue. The run writes its output to the configured destination and records progress in **Backup Run**. Inspect its result and verify the saved content before relying on it for recovery.
+- key: process:backupVerify;slot:help
+  summary: 'Check a backup’s stored content against its manifest.'
+  content: |
+    Use **Verify backup** on a **Backup Run** to check that its objects are present and their contents match the recorded backup. The content check reads stored bytes, decodes them and compares counts and checksums with the manifest. It records a **Verification** result without changing the backup. Review mismatches and warnings before using that run as a recovery source.
+- key: process:backupCheckSla;slot:help
+  summary: 'Check whether backup plans meet their freshness and verification expectations.'
+  content: |
+    This monitoring action checks backup health against each plan’s expected backup and verification ages. Use **Backup Plan** and the Backups home page to inspect its assessment, then open the latest **Backup Run** or **Verification** for the underlying evidence. A health problem may require a new run, a storage repair or investigation of a failed check rather than simply changing the monitoring threshold.
+- key: process:backupRestore;slot:help
+  summary: 'Preview and restore data from a saved backup run.'
+  content: |
+    Use **Restore...** on a **Backup Run** to recover its data. Choose the tables, matching rule, record handling and file options, then review the proposed changes. Nothing is written until you continue. Replacement mode can delete destination records and requires confirmation. Review the option to back up the target first. After execution, inspect **Restore Run** and its table results for skipped records, warnings or failures.
+- key: process:backupRestoreTable;slot:help
+  summary: 'Preview and restore table data from a selected backup artifact.'
+  content: |
+    Use the table restore action when you need data from a particular backup artifact rather than the entire run. Check the source table, destination, matching rule and restore mode in the options and review screen. Nothing is written until you continue. Replacement can remove destination data. Inspect **Restore Run** and its table results afterward to confirm which records and files were recovered.
+- key: process:backupImportCatalog;slot:help
+  summary: 'Find saved backups in storage and add their metadata to the local catalog.'
+  content: |
+    Use **Import backups from storage** when backup objects exist in storage but their run records are absent from this application. Enter the backup bucket alias and folder prefix, then review the discovered backups before continuing. This imports catalog information; it does not restore application records. Once imported, inspect **Backup Run**, verify the saved content and choose a restore action when needed.
+- key: process:backupDownload;slot:help
+  summary: 'Create download links for the stored artifacts of selected backup runs.'
+  content: |
+    Use **Download** when you need the files belonging to a backup run outside the application. Review the files offered before continuing to obtain their links. The action does not restore records or change the saved backup. Treat the downloaded content according to the data it contains, and use **Artifacts** or **Verification** when you need to inspect completeness and integrity.
+- key: process:backupDownloadArtifact;slot:help
+  summary: 'Create a download link for a selected backup artifact.'
+  content: |
+    Use the artifact download action when you need a particular stored part of a backup. Review the selected artifact and its related **Backup Run** so you know which table or supporting file you are retrieving. The action returns a download link and does not change application records. A single part may not contain everything needed for a complete restore.
+- key: process:backupRunScheduledPlan;slot:help
+  summary: 'Execute a backup plan on behalf of its configured schedule.'
+  content: |
+    This background action runs a plan when invoked by scheduling. It checks the plan owner before starting the backup and records the resulting **Backup Run**. Use the plan’s latest run and error information to investigate a missed or failed execution. For an interactive preview and manual start, use **Run backup now** on the plan instead.
+- key: process:backupCancelRun;slot:help
+  summary: 'Request cancellation of a backup run that is still in progress.'
+  content: |
+    Use **Cancel** on a running backup when it should stop. Review the selected runs before continuing. Cancellation is a request to the running process, so check **Backup Run** afterward for its final state. It does not turn partial output into a complete backup or undo a previous successful run. Use a successfully completed and verified backup for recovery.
+- key: process:backupCancelRestoreRun;slot:help
+  summary: 'Request cancellation of a restore that is still in progress.'
+  content: |
+    Use **Cancel** on a restore when further recovery work should stop. Review the selection and then check **Restore Run** for its final status and table results. Cancellation does not imply that records already changed have been rolled back. Inspect the counts and any target backup before deciding how to recover from a partially completed restore.
+- key: process:backupResumeRun;slot:help
+  summary: 'Preview where an interrupted backup can continue and resume eligible work.'
+  content: |
+    Use **Resume** on an interrupted backup when you want to continue eligible work from its recorded progress. Review the proposed continuation and any reasons the run cannot resume before proceeding. Nothing is written during the preview. After execution, inspect **Backup Run** and its artifacts, then verify the completed backup before treating it as a recovery source.
+- key: process:backupTestDestination;slot:help
+  summary: 'Check whether a plan’s configured backup storage can be used.'
+  content: |
+    Use **Test destination** after configuring a **Backup Plan** destination. After review, the test writes a small probe object, reads it back, lists it and deletes it. For a plan using object locking, it only reads and writes no probe. Review each check and the capability report. This tests destination access, not the integrity of an existing backup; use **Verify backup** for that.
+- key: process:backupPrune;slot:help
+  summary: 'Preview and delete backups that a plan’s retention rules no longer keep.'
+  content: |
+    Use **Prune per retention** on a **Backup Plan** to remove expired backup content. Review exactly what will be deleted before continuing; the preview deletes nothing. Retention periods, minimum retained backups and storage locks affect eligibility. Pruning removes recovery material, so review the retained backups and their verification results before proceeding. Inspect the run catalog afterward for the recorded pruning result.
+- key: process:backupPruneRuns;slot:help
+  summary: 'Apply retention pruning to the plans associated with selected backup runs.'
+  content: |
+    Use **Prune per retention** on selected runs to evaluate the retention rules of their plans. The action is not simply a command to delete each selected row. Review the proposed deletions before continuing. Minimum retention and storage locks can protect backups from deletion. Pruning removes saved recovery content, so inspect the backups that will remain and their verification results first.
+- key: widget:backupsHome;slot:label
+  summary: 'Backup plan health, recent runs and verification information.'
+- key: widget:backupsHealthCard;slot:label
+  summary: 'A summary of backup health and plans needing attention.'
+- key: widget:backupsPlanScopeJoinPlan;slot:label
+  summary: 'Inclusion and exclusion rules for this backup plan.'
+- key: widget:backupsRunJoinPlan;slot:label
+  summary: 'Backup runs produced by this plan.'
+- key: widget:backupsArtifactJoinRun;slot:label
+  summary: 'Stored artifacts belonging to this backup run.'
+- key: widget:backupsVerificationJoinRun;slot:label
+  summary: 'Verification checks performed on this backup run.'
+- key: widget:backupsRestoreRunJoinRun;slot:label
+  summary: 'Restore runs using this backup as their source.'
+- key: widget:backupsRestoreRunTableJoinRestoreRun;slot:label
+  summary: 'Per-table results for this restore run.'
 
-  - key: table:backups_plan;slot:help
-    summary: "A backup plan: what is backed up, where it goes, when it runs, how long it is kept and how it is verified."
-    content: |
-      **How rows get created:** an administrator adds one with New, or runs Create default plan, which adds "Nightly full" (the whole instance, data and audit tables, JSONL and gzip, every night at 02:00 UTC, verified on Sundays) and asks only for the storage bucket alias. The alias names a bucket your application provides to Backups; no key or endpoint is ever stored here. A plan with a cron expression runs on its own once it is active; Run backup now runs it by hand. A few settings (PARQUET, ZSTD, incremental, native dump, read replica) are not available yet and are refused when the plan is saved.
+- key: app:backups;slot:help
+  summary: "Back up, verify and restore the application's data to S3-compatible storage."
+  content: |
+    Use the **Setup** card on the home to review what a working backup needs (a plan, a storage bucket, a schedule, verification) with the next step for each. A new installation has no plan: run Create default plan on Backup plans, which asks only for the storage bucket alias, or add a plan with New. Everything else (runs, artifacts, verifications, restores) is written by the backups themselves.
 
-  - key: table:backups_plan_scope;slot:help
-    summary: "One include or exclude rule of a plan: an app, a qbit, a table or a table role."
-    content: |
-      **How rows get created:** an administrator adds them as Scope rules on a plan record. A plan whose scope is Instance backs up everything allowed by its roles and needs no rules; a plan limited to apps, qbits or tables is made of these rules, and an Exclude rule always wins over an Include.
+- key: table:backups_plan;slot:help
+  summary: "A backup plan: what is backed up, where it goes, when it runs, how long it is kept and how it is verified."
+  content: |
+    **How rows get created:** add a plan with **New**, or run **Create default plan**, which adds "Nightly full" (the whole instance, data and audit tables, JSONL and gzip, every night at 02:00 UTC, verified on Sundays) and asks only for the storage bucket alias. The alias names a bucket your application provides to Backups; no key or endpoint is ever stored here. With the scheduler integration installed, an active plan with a cron expression runs on schedule; Run backup now runs it by hand. A few settings (PARQUET, ZSTD, incremental, native dump, read replica) are not available yet and are refused when the plan is saved.
 
-  - key: table:backups_run;slot:help
-    summary: "One execution of a plan, or an ad-hoc export: its status, size, rows, files and storage location."
-    content: |
-      **How rows get created:** only the backups write it. Run backup now, a plan's schedule, a pre-restore safety backup, Resume and Import backups from storage each add a row. Nothing is added by hand; Verify backup, Download, Restore and Prune are actions on a run.
+- key: table:backups_plan_scope;slot:help
+  summary: "One include or exclude rule of a plan: an app, a qbit, a table or a table role."
+  content: |
+    **How rows get created:** add **Scope rules** on a plan record. A plan whose scope is Instance backs up everything allowed by its roles and needs no rules; a plan limited to apps, qbits or tables is made of these rules, and an Exclude rule always wins over an Include.
 
-  - key: table:backups_artifact;slot:help
-    summary: "One stored object of a run: a table part, its manifest, checksums or a file index, with its hash and size."
-    content: |
-      **How rows get created:** only the backups write it, one row per object a run stores. They appear as Artifacts on a run record, and Restore this table and Download work from here.
+- key: table:backups_run;slot:help
+  summary: "One execution of a plan, or an ad-hoc export: its status, size, rows, files and storage location."
+  content: |
+    **How rows get created:** only the backups write it. Run backup now, a plan's schedule, a pre-restore safety backup, Resume and Import backups from storage each add a row. Nothing is added by hand; Verify backup, Download, Restore and Prune are actions on a run.
 
-  - key: table:backups_verification;slot:help
-    summary: "The result of checking a backup: that it is in storage, that it decodes, or that it restores into scratch."
-    content: |
-      **How rows get created:** only verification writes it. Run Verify backup on a run (storage, content or test restore), or give a plan a verify cron expression so it verifies its latest backup on a schedule. The newest result also shows on the plan and the home.
+- key: table:backups_artifact;slot:help
+  summary: "One stored object of a run: a table part, its manifest, checksums or a file index, with its hash and size."
+  content: |
+    **How rows get created:** only the backups write it, one row per object a run stores. Open **Artifacts** on a run record to use **Restore this table** or **Download**.
 
-  - key: table:backups_restore_run;slot:help
-    summary: "One restore of a backup into the application: mode, id strategy, row and file counts and its safety backup."
-    content: |
-      **How rows get created:** only Restore writes it. Run Restore on a backup run, or Restore this table on an artifact: Review previews what would change before anything is written. A restore that replaces data takes a safety backup first and links it here.
+- key: table:backups_verification;slot:help
+  summary: "The result of checking a backup: that it is in storage, that it decodes, or that it restores into scratch."
+  content: |
+    **How rows get created:** only verification writes it. Run Verify backup on a run (storage, content or test restore), or give a plan a verify cron expression so it verifies its latest backup on a schedule. Inspect the newest result on the plan and the home.
 
-  - key: table:backups_restore_run_table;slot:help
-    summary: "Per-table results of a restore: rows inserted, updated and skipped and how the schema matched."
-    content: |
-      **How rows get created:** only Restore writes it, one row per table restored. They appear as Tables restored on a restore record.
+- key: table:backups_restore_run;slot:help
+  summary: "One restore of a backup into the application: mode, id strategy, row and file counts and its safety backup."
+  content: |
+    **How rows get created:** only Restore writes it. Run Restore on a backup run, or Restore this table on an artifact: Review previews what would change before anything is written. A restore that replaces data takes a safety backup first and links it here.
 
-  - key: table:backups_file_object;slot:help
-    summary: "The content-addressed store of copied files, shared by runs, with reference counts and verification."
-    content: |
-      **How rows get created:** only backups write it. When a plan copies the files its rows reference, each distinct file (by SHA-256) is stored once and every run that uses it adds a reference; Prune lowers the counts and removes a file when no run needs it.
+- key: table:backups_restore_run_table;slot:help
+  summary: "Per-table results of a restore: rows inserted, updated and skipped and how the schema matched."
+  content: |
+    **How rows get created:** only Restore writes it, one row per table restored. Open **Tables restored** on a restore record to inspect these results.
 
-  - key: table:backups_id_map;slot:help
-    summary: "Old id to new id pairs recorded while restoring with new ids, so related rows can be rewritten."
-    content: |
-      **How rows get created:** only Restore writes it, and only when a restore uses the Remap new ids strategy. The rows are working data for that restore and are cleared when it finishes, so this table is usually empty.
+- key: table:backups_file_object;slot:help
+  summary: "The content-addressed store of copied files, shared by runs, with reference counts and verification."
+  content: |
+    **How rows get created:** only backups write it. When a plan copies the files its rows reference, each distinct file (by SHA-256) is stored once and every run that uses it adds a reference; Prune lowers the counts and removes a file when no run needs it.
+
+- key: table:backups_id_map;slot:help
+  summary: "Old id to new id pairs recorded while restoring with new ids, so related rows can be rewritten."
+  content: |
+    **How rows get created:** only Restore writes it, and only when a restore uses the Remap new ids strategy. The rows are working data for that restore and are cleared when it finishes, so you will usually find this table empty.
+
+- key: table:backups_plan;field:storageBucketAlias
+  summary: The configured storage alias where you want this plan to write backups.
+  content: |
+    Enter the alias of a bucket your application provides to Backups, never credentials or an endpoint. Use **Test destination** to check access before running the plan. You can select a separate **Prune bucket alias** for deletion access.
+- key: table:backups_plan;field:cronExpression
+  summary: The schedule expression for running this plan automatically.
+  content: |
+    Set the schedule and **{{label:backups_plan.cronTimeZoneId}}** together. Your application must install the scheduler integration, and the plan must be active. Leave the expression blank for manual runs. Use **Run backup now** to preview and start a backup yourself.
+- key: process:backupsPlanCreateDefault;slot:help
+  summary: Create a starter backup plan for a storage bucket you choose.
+  content: |
+    Use **Create default plan** on **Backup plans** and enter a **Storage bucket alias** provided by your application. Review the proposed plan before continuing. The action adds **Nightly full** for data and audit tables with JSONL, gzip and copied files, scheduled at 02:00 UTC with content verification on Sundays at 04:00 UTC.
+
+    You can edit the settings afterward. The action creates a plan only when its plan key does not already exist; it does not change an existing plan. Scheduled execution requires your application's scheduler integration.
+- key: process:backupVerifyPlan;slot:help
+  summary: Verify the latest successful backup of a scheduled plan.
+  content: |
+    Use the plan's verification schedule and level to control automatic checks. This background action selects its latest successful backup and records the check in **Verification**. If no successful backup exists, there is nothing to verify. Inspect the plan and its latest verification result when investigating missed or failed checks.
+- key: process:*.backupExportTable;slot:help
+  summary: Preview and back up selected table records using your current access.
+  content: |
+    Use **Back up these records** to export records from a table. Review the selected scope before continuing; the preview writes nothing. Your record access applies to this export. The destination comes from the supplied storage alias or an active plan. Inspect the resulting **Backup Run** and verify it before relying on it for recovery.
+
+- key: possibleValueSource:backups_scope_type;slot:help
+  summary: The available scope type choices for backups.
+  content: |
+    You can see these choices: {{values:backups_scope_type}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_table_role;slot:help
+  summary: The available table role choices for backups.
+  content: |
+    You can see these choices: {{values:backups_table_role}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_scope_action;slot:help
+  summary: The available scope action choices for backups.
+  content: |
+    You can see these choices: {{values:backups_scope_action}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_selector_type;slot:help
+  summary: The available selector type choices for backups.
+  content: |
+    You can see these choices: {{values:backups_selector_type}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_format;slot:help
+  summary: The available format choices for backups.
+  content: |
+    You can see these choices: {{values:backups_format}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_compression;slot:help
+  summary: The available compression choices for backups.
+  content: |
+    You can see these choices: {{values:backups_compression}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_encryption_mode;slot:help
+  summary: The available encryption mode choices for backups.
+  content: |
+    You can see these choices: {{values:backups_encryption_mode}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_sensitive_policy;slot:help
+  summary: The available sensitive policy choices for backups.
+  content: |
+    You can see these choices: {{values:backups_sensitive_policy}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_heavy_policy;slot:help
+  summary: The available heavy policy choices for backups.
+  content: |
+    You can see these choices: {{values:backups_heavy_policy}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_file_policy;slot:help
+  summary: The available file policy choices for backups.
+  content: |
+    You can see these choices: {{values:backups_file_policy}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_file_missing_policy;slot:help
+  summary: The available file missing policy choices for backups.
+  content: |
+    You can see these choices: {{values:backups_file_missing_policy}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_consistency;slot:help
+  summary: The available consistency choices for backups.
+  content: |
+    You can see these choices: {{values:backups_consistency}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_consistency_achieved;slot:help
+  summary: The available consistency achieved choices for backups.
+  content: |
+    You can see these choices: {{values:backups_consistency_achieved}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_incremental_mode;slot:help
+  summary: The available incremental mode choices for backups.
+  content: |
+    You can see these choices: {{values:backups_incremental_mode}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_native_dump;slot:help
+  summary: The available native dump choices for backups.
+  content: |
+    You can see these choices: {{values:backups_native_dump}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_lock_mode;slot:help
+  summary: The available lock mode choices for backups.
+  content: |
+    You can see these choices: {{values:backups_lock_mode}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_sla_status;slot:help
+  summary: The available sla status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_sla_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_run_trigger;slot:help
+  summary: The available run trigger choices for backups.
+  content: |
+    You can see these choices: {{values:backups_run_trigger}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_run_status;slot:help
+  summary: The available run status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_run_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_retention_class;slot:help
+  summary: The available retention class choices for backups.
+  content: |
+    You can see these choices: {{values:backups_retention_class}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_verification_status;slot:help
+  summary: The available verification status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_verification_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_artifact_type;slot:help
+  summary: The available artifact type choices for backups.
+  content: |
+    You can see these choices: {{values:backups_artifact_type}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_artifact_status;slot:help
+  summary: The available artifact status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_artifact_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_verify_level;slot:help
+  summary: The available verify level choices for backups.
+  content: |
+    You can see these choices: {{values:backups_verify_level}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_verify_result;slot:help
+  summary: The available verify result choices for backups.
+  content: |
+    You can see these choices: {{values:backups_verify_result}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_restore_status;slot:help
+  summary: The available restore status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_restore_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_restore_mode;slot:help
+  summary: The available restore mode choices for backups.
+  content: |
+    You can see these choices: {{values:backups_restore_mode}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_id_strategy;slot:help
+  summary: The available id strategy choices for backups.
+  content: |
+    You can see these choices: {{values:backups_id_strategy}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_schema_status;slot:help
+  summary: The available schema status choices for backups.
+  content: |
+    You can see these choices: {{values:backups_schema_status}}. Check the related field help for supported settings and how the choice affects your backup or restore.
+
+- key: possibleValueSource:backups_plan;slot:help
+  summary: The backup plans available for linking related records.
+
+- key: possibleValueSource:backups_run;slot:help
+  summary: The backup runs available for linking related records.
+
+- key: possibleValueSource:backups_restore_run;slot:help
+  summary: The restore runs available for linking related records.
+
+- key: process:BackupPlanSyncToScheduledJobProcess;slot:help
+  summary: Keep scheduled backup jobs aligned with your plan settings.
+  content: |
+    Set the plan's schedule, time zone and active status to control its scheduled backup job. This background integration synchronizes those settings when your application configures a scheduler. It does not start a backup immediately. Use **Run backup now** for a manual backup and inspect the plan's latest run to confirm execution.
+- key: process:BackupPlanVerifySyncToScheduledJobProcess;slot:help
+  summary: Keep scheduled verification jobs aligned with your plan settings.
+  content: |
+    Set the plan's verification schedule to request automatic checks when your application configures a scheduler. This background integration keeps the verification job aligned with the plan. Inspect **Verification** for completed checks; synchronizing a schedule does not verify a backup. Use **Verify backup** on a saved run for a manual check.

src/test/java/com/kingsrook/qbits/backups/BackupsQBitProducerTest.java

@@ -79,4 +79,26 @@ class BackupsQBitProducerTest
       assertThat(BackupsQBitConfig.PERMISSION_BASE_NAME).isEqualTo("backups");
    }
 
+
+
+   /*******************************************************************************
+    ** Long operations always enqueue, with no unsafe automatic external replay.
+    *******************************************************************************/
+   @Test
+   void testDurableProcesses() throws Exception
+   {
+      BackupsTestUtils.init();
+      for(String name : java.util.List.of("backupRunPlan", "backupRunScheduledPlan", "backupResumeRun", "backupRestore", "backupRestoreTable", "backupVerify"))
+      {
+         var process = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getProcess(name);
+         assertThat(process).as(name).isNotNull();
+         var durability = com.kingsrook.qqq.backend.core.jobs.DurableProcessMetaData.of(process);
+         assertThat(durability).as(name).isNotNull();
+         assertThat(durability.getAlwaysDurable()).isTrue();
+         assertThat(durability.getEligible()).isTrue();
+         assertThat(durability.getIdempotency()).isEqualTo(com.kingsrook.qqq.backend.core.jobs.model.JobIdempotency.NONE);
+         assertThat(durability.getChunkSize()).isEqualTo(1);
+      }
+   }
+
 }

src/test/java/com/kingsrook/qbits/backups/restore/RestoreSafetyTest.java

@@ -205,4 +205,24 @@ class RestoreSafetyTest
          List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", 99)), request, null)).hasMessageContaining("backups.restore");
    }
 
+
+
+   /*******************************************************************************
+    ** Job-page cancellation reaches the restore worker and its catalog status.
+    *******************************************************************************/
+   @Test
+   void testDurableCancellation() throws Exception
+   {
+      BackupRunResult run = RestoreTestSupport.backup(Map.of());
+      RestoreRequest request = RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT).withPreRestoreBackup(false);
+      try(var scope = com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.open(null, () ->
+      {
+         throw (new com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException(com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException.Reason.CANCEL));
+      }))
+      {
+         assertThatThrownBy(() -> new RestoreEngine().execute(request)).isInstanceOf(com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException.class);
+      }
+      assertThat(new BackupCatalog().where(BackupsTables.RESTORE_RUN, "status", "CANCELLED")).hasSize(1);
+   }
+
 }

src/test/java/com/kingsrook/qbits/backups/run/BackupEngineTest.java

@@ -20,6 +20,7 @@ package com.kingsrook.qbits.backups.run;
 import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.zip.GZIPInputStream;
@@ -28,11 +29,18 @@ import com.kingsrook.qbits.backups.manifest.BackupManifest;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.scope.PlanSettings;
 import com.kingsrook.qbits.backups.scope.ScopeResolver;
+import com.kingsrook.qqq.backend.core.actions.async.AsyncJobCallback;
 import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope;
+import com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobLogLevel;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 
 /*******************************************************************************
@@ -187,7 +195,9 @@ class BackupEngineTest
       assertThat(gunzip(BackupsTestUtils.BACKUPS.getBytes(refOnly.getRunPrefix() + "files/index-00001.jsonl.gz"))).contains("REFERENCE_ONLY");
 
       BackupsTestUtils.MEDIA.removeIgnoringLock("media/photo-1.jpg");
-      BackupRunResult missing = run(new BackupCatalog().where(BackupsTables.PLAN, "planKey", "nightly").get(0));
+      RecordingProgress progress = new RecordingProgress();
+      BackupRunResult missing = new BackupEngine().execute(BackupRunFactory.forPlan(new BackupCatalog().where(BackupsTables.PLAN, "planKey", "nightly").get(0), "MANUAL").withProgress(progress));
+      assertThat(progress.logs).anyMatch(l -> l.startsWith("WARN:") && l.contains("missing"));
       assertThat(missing.getFilesMissing()).isGreaterThan(0);
       assertThat(missing.getWarnings()).anyMatch(w -> w.contains("missing"));
       assertThat(missing.getStatus()).isEqualTo("SUCCEEDED_WITH_WARNINGS");
@@ -208,7 +218,7 @@ class BackupEngineTest
          BackupRunResult result = run(plan(Map.of()));
          assertThat(result.getStatus()).isEqualTo("FAILED");
          assertThat(result.isUsable()).isFalse();
-         assertThat(result.getErrorMessage()).contains("injected write failure");
+         assertThat(result.getErrorMessage()).contains("Backup failed");
          assertThat(BackupsTestUtils.BACKUPS.list(result.getRunPrefix() + "manifest.json")).isEmpty();
          assertThat(new BackupCatalog().get(BackupsTables.RUN, result.getRunId()).orElseThrow().getValueString("status")).isEqualTo("FAILED");
       }
@@ -281,4 +291,215 @@ class BackupEngineTest
       assertThat(scope.tables()).isEmpty();
       assertThat(scope.excluded()).extracting(ScopeResolver.ExcludedTable::reason).containsExactly("BACKUPS");
    }
+
+
+   /*******************************************************************************
+    ** A real plan reports work in order, relative artifact names and no row data.
+    *******************************************************************************/
+   @Test
+   void testJobProgress() throws Exception
+   {
+      RecordingProgress progress = new RecordingProgress();
+      BackupRunResult result = new BackupRunCoordinator().runPlan(plan(Map.of("prefix", "private-prefix")), "MANUAL", progress, null);
+      assertThat(result.isUsable()).isTrue();
+      assertThat(progress.phases).containsExactly("prepare", "tables", "media", "upload");
+      assertThat(progress.getSnapshot().getPhases()).extracting(JobPhase::getWeight).containsExactly(2.0, 75.0, 18.0, 5.0);
+      assertThat(progress.getSnapshot().getPhases()).extracting(JobPhase::getStatus)
+         .containsExactly(JobPhase.Status.DONE, JobPhase.Status.DONE, JobPhase.Status.DONE, JobPhase.Status.DONE);
+      assertThat(progress.counts).contains("tables:0/2 tables", "tables:1/2 tables", "tables:2/2 tables", "media:0/4 files", "media:4/4 files");
+      assertThat(progress.bytes).isSorted();
+      assertThat(progress.bytes.getLast()).isGreaterThan(0);
+      assertThat(progress.getSnapshot().getBytesTotal()).isNull();
+      assertThat(progress.messages).contains("Backing up author (1 of 2)", "Backing up blogPost (2 of 2)");
+      assertThat(progress.logs).anyMatch(l -> l.matches("INFO:author: 3 rows, .* MB in .* s"));
+      assertThat(progress.logs).anyMatch(l -> l.matches("INFO:blogPost: 7 rows, .* MB in .* s"));
+      assertThat(progress.logs).contains("INFO:Media: 4 files, 3 copied, 1 already stored, 0 missing", "INFO:Wrote checksums.sha256", "INFO:Wrote manifest.json", "INFO:Wrote files/index-00001.jsonl.gz");
+      assertThat(progress.logs).anyMatch(l -> l.startsWith("INFO:Wrote tables/blogPost/"));
+      for(String warning : result.getWarnings())
+      {
+         assertThat(progress.logs).contains("WARN:" + warning);
+      }
+      assertThat(String.join("\n", progress.logs)).doesNotContain("private-prefix", "do-not-leak", "Body of", "media/photo-");
+   }
+
+
+
+   /*******************************************************************************
+    ** Repeated backups report deduplicated media and complete all four phases.
+    *******************************************************************************/
+   @Test
+   void testProgressWithDeduplication() throws Exception
+   {
+      QRecord plan = plan(Map.of());
+      run(plan);
+      RecordingProgress progress = new RecordingProgress();
+      BackupRunResult result = new BackupEngine().execute(BackupRunFactory.forPlan(plan, "MANUAL").withProgress(progress));
+      assertThat(result.isUsable()).isTrue();
+      assertThat(progress.phases).containsExactly("prepare", "tables", "media", "upload");
+      assertThat(progress.getSnapshot().getPhases()).allMatch(p -> p.getStatus() == JobPhase.Status.DONE);
+      assertThat(progress.logs).contains("INFO:Media: 4 files, 0 copied, 4 already stored, 0 missing");
+   }
+
+
+
+   /*******************************************************************************
+    ** Arbitrary provider failures must not put credentials or row values in jobs.
+    *******************************************************************************/
+   @Test
+   void testProgressFailureDoesNotLeakProviderText() throws Exception
+   {
+      RecordingProgress progress = new RecordingProgress();
+      BackupRunRequest request = BackupRunFactory.forPlan(plan(Map.of()), "MANUAL").withProgress(progress)
+         .withReader((table, filter, start, size, transaction, sink) ->
+         {
+            throw (new com.kingsrook.qqq.backend.core.exceptions.QException("https://storage-user:storage-password@host/?secret=storage-secret accessKey=storage-access row=private-row"));
+         });
+      BackupRunResult result = new BackupEngine().execute(request);
+      assertThat(result.getStatus()).isEqualTo("FAILED");
+      assertThat(progress.logs).contains("ERROR:" + result.getErrorMessage());
+      assertThat(progress.getSnapshot().getPhases()).anyMatch(p -> p.getStatus() == JobPhase.Status.FAILED);
+      assertThat(String.join("\n", progress.logs) + result.getErrorMessage()).doesNotContain("storage-user", "storage-password", "storage-secret", "storage-access", "private-row", "https://");
+   }
+
+
+
+   /*******************************************************************************
+    ** The job page's cancel check must escape the engine to the durable worker.
+    *******************************************************************************/
+   @Test
+   void testJobCancellationBetweenPages() throws Exception
+   {
+      BackupRunRequest request = BackupRunFactory.forPlan(plan(Map.of("pageSize", 1)), "MANUAL");
+      int[] checks = { 0 };
+      try(JobChunkScope ignored = JobChunkScope.open(null, () ->
+      {
+         if(++checks[0] == 3)
+         {
+            throw (new JobInterruptedException(JobInterruptedException.Reason.CANCEL));
+         }
+      }))
+      {
+         assertThatThrownBy(() -> new BackupEngine().execute(request)).isInstanceOf(JobInterruptedException.class);
+      }
+      assertThat(new BackupCatalog().where(BackupsTables.RUN, "status", "CANCELLED")).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Storage providers can include endpoint user-info and raw keys in exceptions.
+    *******************************************************************************/
+   @Test
+   void testStorageFailureDoesNotLeakCredentials() throws Exception
+   {
+      RecordingProgress progress = new RecordingProgress();
+      var storage = new com.kingsrook.qbits.backups.storage.InMemoryBackupStorage()
+      {
+         /*******************************************************************************
+          ** Deliberately unsafe provider diagnostics must stay out of job output.
+          *******************************************************************************/
+         @Override
+         public java.io.OutputStream openWrite(String key, com.kingsrook.qbits.backups.storage.WriteOptions options)
+         {
+            throw (new com.kingsrook.qbits.backups.storage.BackupStorageException(com.kingsrook.qbits.backups.storage.BackupStorageException.Reason.ACCESS_DENIED,
+               "https://endpoint-user:endpoint-password@storage accessKey=access-secret secretKey=secret-secret"));
+         }
+      };
+      BackupRunResult result = new BackupEngine().execute(BackupRunFactory.forPlan(plan(Map.of()), "MANUAL").withProgress(progress).withStorage(storage));
+      assertThat(result.getStatus()).isEqualTo("FAILED");
+      assertThat(progress.logs).contains("ERROR:" + result.getErrorMessage());
+      assertThat(String.join("\n", progress.logs) + result.getErrorMessage()).doesNotContain("endpoint-user", "endpoint-password", "access-secret", "secret-secret", "https://");
+   }
+
+
+
+   /*******************************************************************************
+    ** Failed backups must fail the durable chunk, not just add a summary line.
+    *******************************************************************************/
+   @Test
+   void testDurableLoadPropagatesFailure() throws Exception
+   {
+      var input = new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput();
+      input.setRecords(List.of(plan(Map.of())));
+      BackupsTestUtils.BACKUPS.withFailWritesFor(key -> key.contains("/tables/"));
+      try(JobChunkScope scope = JobChunkScope.open(null).withProgress(new RecordingProgress()))
+      {
+         assertThatThrownBy(() -> new com.kingsrook.qbits.backups.processes.BackupsProcesses.RunPlanLoad().runOnePage(input,
+            new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput()))
+            .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class).hasMessageContaining("Backup failed");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Record values at call time: JobProgress's live snapshot is mutable.
+    *******************************************************************************/
+   static class RecordingProgress extends JobProgress
+   {
+      final List<String> phases = new ArrayList<>();
+      final List<String> counts = new ArrayList<>();
+      final List<Long> bytes = new ArrayList<>();
+      final List<String> logs = new ArrayList<>();
+      final List<String> messages = new ArrayList<>();
+
+      /*******************************************************************************
+       ** Use the core's safe standalone destination.
+       *******************************************************************************/
+      RecordingProgress()
+      {
+         super((AsyncJobCallback) null);
+      }
+
+      /*******************************************************************************
+       ** Record transitions.
+       *******************************************************************************/
+      @Override
+      public JobProgress phase(String name)
+      {
+         phases.add(name);
+         return (super.phase(name));
+      }
+
+      /*******************************************************************************
+       ** Record counts and their units.
+       *******************************************************************************/
+      @Override
+      public JobProgress done(long done)
+      {
+         counts.add(getSnapshot().getCurrentPhase() + ":" + done + "/" + getSnapshot().getTotal() + " " + getSnapshot().getUnitLabel());
+         return (super.done(done));
+      }
+
+      /*******************************************************************************
+       ** Record byte counts.
+       *******************************************************************************/
+      @Override
+      public JobProgress bytesDone(long done)
+      {
+         bytes.add(done);
+         return (super.bytesDone(done));
+      }
+
+      /*******************************************************************************
+       ** Capture text before core redaction as well as exercising core publication.
+       *******************************************************************************/
+      @Override
+      public JobProgress log(JobLogLevel level, String message)
+      {
+         logs.add(level + ":" + message);
+         return (super.log(level, message));
+      }
+
+      /*******************************************************************************
+       ** Record live messages.
+       *******************************************************************************/
+      @Override
+      public JobProgress message(String message)
+      {
+         messages.add(message);
+         return (super.message(message));
+      }
+   }
+
 }

src/test/java/com/kingsrook/qbits/backups/run/BackupRunCoordinatorTest.java

@@ -24,10 +24,10 @@ import java.util.List;
 import java.util.Map;
 import com.kingsrook.qbits.backups.BackupsRuntime;
 import com.kingsrook.qbits.backups.BackupsTestUtils;
-import com.kingsrook.qbits.backups.LifecycleTestSupport;
 import com.kingsrook.qbits.backups.LifecycleTestSupport.EtlSummaries;
 import com.kingsrook.qbits.backups.LifecycleTestSupport.MutableClock;
 import com.kingsrook.qbits.backups.LifecycleTestSupport.RecordingAlertSink;
+import com.kingsrook.qbits.backups.LifecycleTestSupport;
 import com.kingsrook.qbits.backups.manifest.BackupManifest;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.processes.CancelRunProcess;
@@ -296,7 +296,7 @@ class BackupRunCoordinatorTest
       QRecord         plan   = LifecycleTestSupport.plan(perTableTinyParts);
       BackupRunResult failed = new BackupRunCoordinator().runPlan(plan, "MANUAL", request -> request.withReader(failingAfter(request.getReader(), "blogPost", 4)));
       assertThat(failed.getStatus()).isEqualTo("FAILED");
-      assertThat(failed.getErrorMessage()).contains("injected failure in blogPost");
+      assertThat(failed.getErrorMessage()).contains("Backup failed");
       assertThat(BackupsTestUtils.BACKUPS.list(failed.getRunPrefix() + "manifest.json")).isEmpty();
 
       QRecord failedRow = catalog.get(BackupsTables.RUN, failed.getRunId()).orElseThrow();
@@ -384,4 +384,59 @@ class BackupRunCoordinatorTest
       assertThat(manifest.table("blogPost").parts()).hasSize(7);
       assertThat(catalog.where(BackupsTables.ARTIFACT, "backupRunId", failed.getRunId()).stream().filter(a -> "blogPost".equals(a.getValueString("tableName")))).hasSize(7);
    }
+
+
+   /*******************************************************************************
+    ** Explicit retry of a durable chunk must reuse its backup and completed parts.
+    *******************************************************************************/
+   @Test
+   void testDurableRetryResumesSameRun() throws Exception
+   {
+      QRecord plan = LifecycleTestSupport.plan(Map.of());
+      BackupEngineTest.RecordingProgress progress = new BackupEngineTest.RecordingProgress();
+      var identity = new com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.ChunkIdentity(42, 8, 0, "backupRunPlan");
+      try(var scope = com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.open(null, null, identity).withProgress(progress))
+      {
+         BackupsTestUtils.BACKUPS.withFailWritesFor(key -> key.contains("/tables/blogPost/"));
+         BackupRunResult failed = new BackupRunCoordinator().runPlan(plan, "MANUAL");
+         assertThat(failed.getStatus()).isEqualTo("FAILED");
+         BackupsTestUtils.BACKUPS.withFailWritesFor(key -> false);
+         // A worker death can leave RUNNING rather than FAILED; identity, not age,
+         // ties this recovery to the same durable job.
+         catalog.update(BackupsTables.RUN, failed.getRunId(), Map.of("status", "RUNNING"));
+         BackupRunResult resumed = new BackupRunCoordinator().runPlan(plan, "MANUAL");
+         assertThat(resumed.isUsable()).isTrue();
+         assertThat(resumed.getRunId()).isEqualTo(failed.getRunId());
+         assertThat(resumed.getRunPrefix()).isEqualTo(failed.getRunPrefix());
+         assertThat(progress.logs).contains("INFO:Resuming run " + failed.getRunId() + ": 1 tables already done");
+         assertThat(catalog.where(BackupsTables.RUN, "backupPlanId", plan.getValueInteger("id"))).hasSize(1);
+         int artifacts = catalog.where(BackupsTables.ARTIFACT, "backupRunId", failed.getRunId()).size();
+         BackupRunResult again = new BackupRunCoordinator().runPlan(plan, "MANUAL");
+         assertThat(again.getRunId()).isEqualTo(failed.getRunId());
+         assertThat(catalog.where(BackupsTables.ARTIFACT, "backupRunId", failed.getRunId())).hasSize(artifacts);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A durable job refused for capacity must not escape into the inline queue.
+    *******************************************************************************/
+   @Test
+   void testDurableCapacityRefusalDoesNotQueue() throws Exception
+   {
+      QRecord occupied = LifecycleTestSupport.plan(Map.of());
+      liveRun(occupied.getValueInteger("id"), "RUNNING", clock.instant());
+      QRecord requested = LifecycleTestSupport.plan(Map.of("planKey", "second", "name", "Second"));
+      var identity = new com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.ChunkIdentity(43, 9, 0, "backupRunPlan");
+      try(var scope = com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.open(null, null, identity))
+      {
+         org.assertj.core.api.Assertions.assertThatThrownBy(() -> new BackupRunCoordinator().runPlan(requested, "MANUAL"))
+            .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class).hasMessageContaining("could not start");
+      }
+      assertThat(catalog.queuedRuns()).isEmpty();
+      assertThat(catalog.where(BackupsTables.RUN, "runKey", "job-43-plan-" + requested.getValueInteger("id")))
+         .singleElement().satisfies(run -> assertThat(run.getValueString("status")).isEqualTo("FAILED"));
+   }
+
 }

src/test/java/com/kingsrook/qbits/backups/schedule/BackupPlanScheduleTest.java

@@ -61,6 +61,7 @@ import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
 import com.kingsrook.qqq.backend.core.scheduler.QScheduleManager;
+import com.kingsrook.qqq.backend.core.testing.HelpBundleAssertions;
 import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -320,4 +321,22 @@ class BackupPlanScheduleTest
       assertThat(inactive.getValues().get("status")).isEqualTo("INACTIVE");
       assertThat(catalog.where(BackupsTables.RUN, "backupPlanId", planId)).isEmpty();
    }
+
+
+   /*******************************************************************************
+    ** Cover every qbit target, including optional scheduler integration processes.
+    *******************************************************************************/
+   @Test
+   void helpBundleIsComplete()
+   {
+      QInstance qInstance = QContext.getQInstance();
+      HelpBundleAssertions.forInstance(qInstance)
+         .bundles("help/qbit-backups.yaml")
+         // Core owns the generated bulk CRUD actions; this qbit owns its export actions on host tables.
+         .owning(target -> target.owner() instanceof com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData process
+            ? qInstance.getQBits().containsKey(process.getSourceQBitName())
+            : target.name().startsWith("backups"))
+         .assertComplete();
+   }
+
 }

src/test/java/com/kingsrook/qbits/backups/tables/BackupsTablesGuardrailTest.java

@@ -27,11 +27,11 @@ import java.util.Map;
 import java.util.regex.Pattern;
 import java.util.stream.Stream;
 import com.kingsrook.qbits.backups.BackupsTestUtils;
-import com.kingsrook.qbits.backups.model.BackupsHelp;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -128,18 +128,24 @@ class BackupsTablesGuardrailTest
    @Test
    void everyTableHasEmptyStateHelp() throws Exception
    {
-      assertThat(BackupsHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
+      var bundle = HelpBundles.of(qInstance).getBundles().get(0);
       for(String table : backupsTables())
       {
-         List<QHelpContent> help = qInstance.getTable(table).getHelpContent() == null ? null : qInstance.getTable(table).getHelpContent().get(BackupsHelp.SLOT);
-         assertThat(help).as(table + " table help").isNotEmpty();
-         assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created");
+         assertThat(bundle.getEntries()).filteredOn(entry -> entry.getKey().equals("table:" + table + ";slot:help"))
+            .singleElement().satisfies(entry -> assertThat(entry.getContent()).contains("How rows get created"));
+      }
+      assertThat(bundle.getEntries()).filteredOn(entry -> entry.getKey().equals("app:backups;slot:help")).hasSize(1);
+      assertThat(bundle.getEntries()).filteredOn(entry -> entry.getKey().equals("table:backups_plan;slot:help"))
+         .singleElement().satisfies(entry -> assertThat(entry.getContent()).contains("Create default plan"));
+      // RC.7 validates supplied widget/value-source entries but does not require their coverage.
+      for(String name : qInstance.getWidgets().keySet().stream().filter(name -> name.startsWith("backups")).toList())
+      {
+         assertThat(bundle.getEntries()).anyMatch(entry -> entry.getKey().startsWith("widget:" + name + ";slot:"));
+      }
+      for(String name : qInstance.getPossibleValueSources().keySet().stream().filter(name -> name.startsWith("backups_")).toList())
+      {
+         assertThat(bundle.getEntries()).anyMatch(entry -> entry.getKey().startsWith("possibleValueSource:" + name + ";slot:"));
       }
-      assertThat(qInstance.getApp("backups").getHelpContent()).containsKey(BackupsHelp.SLOT);
-      assertThat(BackupsHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || qInstance.getTable(key.substring(6, key.indexOf(';'))) != null,
-         "help names only real tables");
-
-      assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("Create default plan");
       assertThat(qInstance.getProcess(DefaultPlanProcess.NAME)).as("the action the plan help names").isNotNull();
       assertThat(qInstance.getProcess(DefaultPlanProcess.NAME).getLabel()).isEqualTo("Create default plan");
    }
@@ -153,12 +159,14 @@ class BackupsTablesGuardrailTest
    void hostHelpIsKept() throws Exception
    {
       QTableMetaData table = qInstance.getTable(BackupsTables.RUN);
-      table.withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for runs."));
-      qInstance.getApp("backups").withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for the app."));
-      BackupsHelp.apply(qInstance, "backups");
-      assertThat(table.getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for runs."));
-      assertThat(qInstance.getApp("backups").getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for the app."));
-      assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("How rows get created");
+      table.withHelpContent("help", new QHelpContent("Host help for runs."));
+      qInstance.getApp("backups").withHelpContent("help", new QHelpContent("Host help for the app."));
+      HelpBundles.register(qInstance, "help/qbit-backups.yaml");
+      assertThat(qInstance.getTable("helpContent")).isNull();
+      HelpBundles.syncOnStartup(qInstance);
+      assertThat(table.getHelpContent().get("help")).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for runs."));
+      assertThat(qInstance.getApp("backups").getHelpContent().get("help")).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for the app."));
+      assertThat(HelpBundles.of(qInstance).getBundles()).isNotEmpty();
    }
 
 

src/test/java/com/kingsrook/qbits/backups/verify/BackupVerifierTest.java

@@ -591,4 +591,23 @@ class BackupVerifierTest
       }
       return (bytes.toByteArray());
    }
+
+
+   /*******************************************************************************
+    ** A verification must propagate a job interrupt, not report it as corruption.
+    *******************************************************************************/
+   @Test
+   void testDurableCancellation() throws Exception
+   {
+      BackupRunResult run = backup(plan(Map.of()));
+      try(var scope = com.kingsrook.qqq.backend.core.jobs.plans.JobChunkScope.open(null, () ->
+      {
+         throw (new com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException(com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException.Reason.CANCEL));
+      }))
+      {
+         org.assertj.core.api.Assertions.assertThatThrownBy(() -> new BackupVerifier().verify(run.getRunId(), BackupVerifier.CONTENT, "MANUAL", "test"))
+            .isInstanceOf(com.kingsrook.qqq.backend.core.jobs.plans.JobInterruptedException.class);
+      }
+   }
+
 }