Compare commits

...
Author SHA1 Message Date
CircleCI 3df24cda5a Bump version to 0.1.0-RC.5 [skip ci] 2026-10-14 22:22:38 +00:00
james.maesandClaude Opus 5.5 987c20899b Merge develop into release/0.1 for 0.1.0-RC.5 (#25 setup card, default plan, scheduled verify)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:19:54 -05:00
james.maes 01db11b123 Merge pull request #25 from QRun-IO/feature/tables-populated-defaults
feat: populated defaults, guided empty states, scheduled verify
2026-10-14 17:13:31 -05:00
james.maesandClaude Opus 5.5 6dea78881d feat: host-scheduler config so a host that ticks plans itself is not shown a false needs-setup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:10:21 -05:00
james.maesandClaude Opus 5.5 89f218937a fix: setup card reflects the scheduler, review checks the plan name, host help test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:07:54 -05:00
james.maesandClaude Opus 5.5 85f4ca0a6f feat: guided empty states, setup card, Create default plan, scheduled verify process
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:49:20 -05:00
james.maes 48e7320228 Merge pull request #22 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:56:39 -05:00
james.maesandClaude Opus 5.5 a5297e709d ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:06:12 -05:00
CircleCI 14bb6fa1b4 Bump version to 0.1.0-RC.4 [skip ci] 2026-10-14 12:26:09 +00:00
james.maesandClaude Opus 5.5 b1ff70d7d3 Merge develop into release/0.1 for 0.1.0-RC.4 (#21 schema fingerprint, keyset pager, guardrails)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:21:32 -05:00
james.maes f2af36ec5f Merge pull request #21 from QRun-IO/feature/arch-rc6-keyset
refactor(backups): core TableSchemaFingerprint + KeysetPager on RC.6 (#19 part 2)
2026-10-14 07:19:52 -05:00
james.maesandClaude Opus 5.5 054f6d90d7 build: qbit-build-parent 2.1.0 with architecture guardrails enforced
Enforce the qbit-bom#10 guardrails (G1, G3, G4, G7) and run tests in
strict raw-route mode (G2). All pass on the released QQQ 4.1.0-RC.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:15:24 -05:00
james.maesandClaude Opus 5.5 5b7106b981 refactor(backups): core TableSchemaFingerprint and KeysetPager (QQQ 4.1.0-RC.6)
Replace the RC.6 stand-ins in TableFingerprints, TableReader and
StandardRestoreWriter with core TableSchemaFingerprint and KeysetPager.

Core fingerprints differ from the local ones: descriptor names, unique
keys as lists, association joinName, and empty-string backend types. New
manifests therefore record fingerprintAlgorithm "qqq-table-schema-v1".
A manifest without the field uses the frozen LegacyTableFingerprint, so
old backups still verify and restore.

Paging keeps ascending primary-key order, exclusive resume cursors,
bounded pages and the caller's transaction. Restore writes now require
the explicit backups.restore permission, checked during Review.

Closes #19 (architecture review QRun-IO/qqq#1150, finding F3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:36:44 -05:00
CircleCI cda334f2fe Bump version to 0.1.0-RC.3 [skip ci] 2026-10-14 00:53:05 +00:00
james.maesandClaude Opus 5.5 590b132fb6 Merge develop into release/0.1 for 0.1.0-RC.3 (core alerts, renamed alert types, #20)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:50:34 -05:00
james.maes 32034e5762 Merge pull request #20 from QRun-IO/feature/arch-alerts-core
refactor(alerts): direct core QAlerts binding on QQQ RC.5 (#19 part 1)
2026-10-13 19:37:01 -05:00
james.maesandClaude Opus 5.5 bc97d59b96 refactor(alerts): bind core QAlerts directly (QQQ 4.1.0-RC.5)
Core QAlerts shipped in 4.1.0-RC.5. Replace the reflective
AlertsQBitAlertSink stand-in with a compiled binding:
- raise maps to QAlerts.raise, resolve to QAlerts.resolve and heartbeat to
  QAlerts.heartbeat;
- the backup alert types register as QAlertTypeMetaData.
It stays quiet when no alerts implementation is installed, and an alerting
failure never fails a backup. The fake SPI test classes are deleted.

RC.5 rejects underscores in alert type names, so two types are renamed:
- backups.run.missing_in_storage becomes backups.run.missing-in-storage;
- backups.plan.owner_invalid becomes backups.plan.owner-invalid.

TableFingerprints, TableReader and StandardRestoreWriter are tagged
STAND-IN(..., removeBy=4.1.0-RC.6) for TableSchemaFingerprint and
KeysetPager.

Refs #19 (architecture review QRun-IO/qqq#1150, finding F3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:33:55 -05:00
james.maes 506741bd90 Merge pull request #18 from QRun-IO/feature/release-rc-automation
ci: automate RC releases; hotfix/* pushes no longer publish
2026-10-13 14:42:59 -05:00
james.maesandClaude Opus 5.5 95b971a5fb ci: automate RC releases; hotfix/* pushes no longer publish
Removes publish_hotfix_release: a hotfix/* branch push published a GA
release to Maven Central with no tag or review (QRun-IO/qbit-cdn#12,
QRun-IO/qqq#1121). hotfix/* branches now run test_only; a hotfix
release publishes from a reviewed vX.Y.Z tag through publish_release.
Adds .github/workflows/release-rc.yml (reusable QRun-IO/.github
qbit-release-rc): a manual dispatch, or merging a develop PR labelled
release:rc, merges develop into the release line and the release/* push
runs the existing CircleCI RC publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:29:57 -05:00
CircleCI f0a8c5000e Bump version to 0.1.0-RC.2 [skip ci] 2026-10-13 16:11:19 +00:00
james.maesandClaude Opus 5.5 8f3f9ec60a Merge develop into release/0.1 for 0.1.0-RC.2
qbit-object-storage 0.1.0-RC.2 (AWS SDK v2 2.41.34, GHSA-443w-3rq3-5m5h; #17).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:08:28 -05:00
james.maes e56cc1b021 Merge pull request #17 from QRun-IO/feature/object-storage-rc2
build(deps): qbit-object-storage 0.1.0-RC.2 (AWS SDK security fix)
2026-10-13 11:07:46 -05:00
CircleCI cdaf218201 Bump version to 0.1.0-RC.1 [skip ci] 2026-10-13 00:26:41 +00:00
54 changed files with 2909 additions and 943 deletions

No files matched your search

+2 -13
View File
@@ -1,7 +1,7 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
@@ -22,7 +22,7 @@ workflows:
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|release\/.*|hotfix\/.*)/
ignore: /(develop|release\/.*)/
publish_snapshot:
jobs:
@@ -58,14 +58,3 @@ workflows:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
publish_hotfix_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: hotfix
filters:
branches:
only: [/hotfix\/.*/]
+25
View File
@@ -0,0 +1,25 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
type: boolean
default: false
pull_request:
types: [closed]
branches: [develop]
permissions:
contents: write
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+1 -1
View File
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
## Dependencies
- Java 21.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Tests: JUnit 5, AssertJ.
## Release
+13
View File
@@ -5,4 +5,17 @@ All notable changes to this project are documented here. The format follows Keep
## [0.1.0] - Unreleased
### Added
- Guided empty states: help bundle `help/qbit-backups.yaml` (app and all nine tables, with "How rows get created") applied by `BackupsHelp`, and a Setup card on the Backups home (plan, scope rules, storage, schedule, verification, prune bucket).
- "Create default plan" table action on Backup plans: adds "Nightly full" for a storage bucket alias; insert-only.
- `backupVerifyPlan`: the scheduled verification process. A plan verify cron created a scheduled job for a process that did not exist, so scheduled verification never ran.
- Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.
### Changed
- Saving a plan with Use read replica on is refused (it was never read); the not-yet-available settings (read replica, native dump, incremental, PARQUET, ZSTD) say so in their field help; scheduledJobId is documented as unused.
- Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
- Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.
- Use RC.5-valid alert names `backups.run.missing-in-storage` and `backups.plan.owner-invalid` (and matching deduplication keys). Hosts using the old underscore names must update routing rules.
- Target QQQ 4.1.0-RC.6 and replace the fingerprint, keyset pager and reflective restore bridges with typed core APIs.
- New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
- Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
- Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
+29 -1
View File
@@ -10,7 +10,7 @@
## Requirements
- Java 21.
- QQQ 4.1 (built against `4.1.0-RC.4`).
- QQQ 4.1 (targets `4.1.0-RC.6`, not yet released).
## Install
@@ -22,6 +22,10 @@ new BackupsQBitProducer()
.produce(qInstance);
```
## Setup
This qbit ships no DDL and seeds nothing from a changelog: its one setup value, the storage bucket alias, belongs to your application. On a fresh install open Backups, read the Setup card, and run **Create default plan** on Backup plans (it asks only for the alias of a bucket your storage resolver provides). It creates the insert-only "Nightly full" plan: whole instance, data and audit tables, JSONL and gzip, 02:00 UTC, verified Sundays 04:00 UTC. Edit any setting afterwards. Help for every table, including how its rows get created, ships in `help/qbit-backups.yaml`.
## Naming
Tables are `backups_<name>` (lowercase snake_case, backend name equal to the table name). Permissions are `backups.<action>`.
@@ -36,8 +40,32 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
./mvnw verify
```
Before RC.6 is released, verify against the locally installed core build only:
```bash
./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
```
The POM stays on RC.6. See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for
fingerprint compatibility, paging guarantees and restore permissions.
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
## License
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
### Alerts integration
Backups binds directly to core QAlerts in QQQ 4.1.0-RC.5 and declares its nine
alert types on each instance. Hosts install an `AlertSinkInterface` through
`AlertsInstanceMetaData.withSink(QCodeReference)`. Without a configured sink,
backup alert calls remain no-ops (they do not use core's logging fallback).
Custom `BackupAlertSink` bindings are preserved. Core handles dispatch failures;
adapter failures are caught and logged once, so alerting cannot fail a backup.
Raise and resolve follow core's transaction-aware dispatch; heartbeats run immediately.
The backup SLA checker still owns expected intervals and grace periods.
RC.5 requires hyphenated alert names: `backups.run.missing-in-storage` and
`backups.plan.owner-invalid` replace the former underscore spellings, including
their deduplication keys. Update host routing rules that used those spellings.
+7
View File
@@ -0,0 +1,7 @@
# Upgrading
## Unreleased
- No schema change and no new include line. Hosts get the help text and the Setup card automatically.
- Plans with `use_read_replica` true are now refused on the next save (the setting was never used). Turn it off.
- Hosts that set a plan `verifyCronExpression` and a scheduler now get working scheduled verification: the `backupVerifyPlan` process exists. It verifies the plan's latest successful run through the plan's `verifyLevel` (CONTENT when unset). Expect verification rows and, on failure, alerts.
+100
View File
@@ -0,0 +1,100 @@
# RC.6 core migration (issue #19, F3)
The POM targets `4.1.0-RC.6`. Until that release is available, local verification
uses `./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` against the
installed build from `qqq-wt-archrc6`. No core source changes are needed.
## Fingerprint compatibility
The algorithms are **not byte-compatible**, even for identical table metadata:
| Descriptor property | Pre-RC.6 backup algorithm | Core algorithm |
| --- | --- | --- |
| Table and key | `name`, `primaryKey` | `table`, `primaryKeyField` |
| Fields | Map keyed by name | Sorted list with explicit `name` |
| Field flags | `required`, `pvs`, `heavy`, `sensitive` | `isRequired`, `possibleValueSource`, `isHeavy`, `isSensitive` |
| Unique keys | Sorted `+`-joined strings | Sorted lists of field names |
| Associations | Sorted `name->table` strings | Objects with name, associatedTable and joinName |
| Null backend type | JSON null | Empty string |
| Escaping | Historical hand-written escaping | Jackson JSON escaping |
New table entries contain `fingerprintAlgorithm: "qqq-table-schema-v1"`, the core
`schema` descriptor, and `fingerprint: "sha256:" + core.getHash()`. The outer
manifest remains version 1; row parts, checksums, compression and storage paths
are unchanged. The instance/catalog hash still combines the ordered table
fingerprint strings, so its value changes too. New manifests require this RC.6
reader; older readers do not understand the new descriptor shape.
An absent algorithm identifies the old `qbit-backups-table-v1` algorithm. The
reader also accepts that explicit identifier. `LegacyTableFingerprint` freezes
the original descriptor and hash implementation solely for comparison against
existing backups; no new backup uses it. Restore and content verification select
the declared algorithm. Unknown identifiers never match a known fingerprint.
Restore normalizes either field shape for its existing compatibility analysis
when a table has changed. Fingerprint equality is not required for a compatible
schema migration, as before.
`src/test/resources/manifests/legacy-schema.json` freezes descriptors and hashes
produced by the original stand-in for the blog fixture. Tests place these into a
real backup manifest without the algorithm field, verify its content, mutate the
live data, then restore and compare the complete content hash including dates.
They also cover optional additions and incompatible type changes in old schemas.
## Paging and consistency
`TableReader.Keyset` delegates to core `KeysetPager`. Every query uses
`(scope filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize`, with no offset.
The resume key is exclusive. Page size is a row bound, **not a JDBC fetch-size
setting**. Core's normal bounded QueryAction path retains backend driver defaults;
this adapter does not opt into the large-result streaming query hint or change
JDBC fetch size. Heavy/hidden/password fields and untranslated stored values keep
the same query settings and existing backup permission boundary.
Every page receives the caller's transaction. The pager does not open, commit,
close, or replace that transaction. A test opens a real core memory snapshot,
changes a later page's data, and proves every page still sees the original state.
The existing BackupEngine continues to report BEST_EFFORT and reject
SNAPSHOT_REQUIRED: adopting the consistent-read opening API at engine level is
separate work. This migration does not claim snapshot guarantees for ordinary
backup runs.
Core can stream non-orderable keys without deterministic key order; backups
explicitly reject those keys instead. Core rejects scope filters carrying skip
or limit, and replaces user ordering with ascending primary-key ordering. Tests
cover OR filters, resume, full and partial pages, empty scans and invalid inputs.
## Restore writes
`StandardRestoreWriter` now constructs `RestoreWriteFlag` directly and calls
`QBackendModuleInterface.resyncKeySequences`, consuming typed `SequenceResyncResult`
values. Only RESYNCED results are reported as sequence changes; NOT_NEEDED and
NOT_APPLICABLE remain normal outcomes. There is no reflection or unavailable-core
fallback. The core flag preserves supplied dates and carries the requested
customizer, automation, audit and event settings.
Core always requires an actual session permission `backups.restore` for its
restore flag, including hosts with general core enforcement disabled. The qbit's
anonymous-mode convenience setting does not grant that permission. Review now
reports this requirement before executing a restore; test sessions explicitly
hold it. Custom writers that do not use the core flag keep their existing qbit
permission checks.
## HTTP operations and wire changes
No HTTP operations are added, removed or modified: method, path and operationId
are unchanged (no operation list entries for this migration). The RC.6 UI API
extension documentation was reviewed; this task introduces no Javalin route or
middleware extension. Wire changes are confined to the manifest fields and
fingerprint values described above, plus an earlier restore permission error in
Review for anonymous sessions lacking `backups.restore`.
## Local verification (2026-10-13)
- `./mvnw -B clean verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`:
BUILD SUCCESS; 158 tests, zero failures/errors/skips; both JaCoCo gates passed.
Existing backup, restore and MinIO integration tests ran.
- Separate Checkstyle 14.3.0 audit of all 16 changed Java files (main and test):
zero violations, using the shared Kingsrook-derived rules with this repository's
Apache file header. This repository has no Checkstyle execution configured in
its POM; audit configuration and results are local under `.cache/style/`.
- `git diff --check`: passed. No commit, push or core source modification.
+6 -4
View File
@@ -18,7 +18,9 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<revision>0.1.0-SNAPSHOT</revision>
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
<revision>0.1.0-RC.5</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<jacoco.version>0.8.15</jacoco.version>
@@ -27,7 +29,7 @@
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
<version>2.1.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -192,14 +194,14 @@
<profiles>
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
<!-- imports the released 4.1 BOM, this profile is active by default and -->
<!-- imports qqq-bom-pom 4.1.0-RC.4 ahead of the parent's 4.0.0 BOM. -->
<!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -47,6 +47,8 @@ public class BackupsQBitConfig implements QBitConfig
private String parentAppName;
private String tablePathPrefix;
private String schedulerName;
private String hostSchedulerDescription;
private Boolean hostSchedulerRunsVerification = false;
private String instanceName;
private Integer maxConcurrentRuns = 1;
@@ -202,6 +204,51 @@ public class BackupsQBitConfig implements QBitConfig
/*******************************************************************************
** Getter for hostSchedulerDescription: when the application runs plan crons with
** its own tick instead of a QQQ scheduler (schedulerName), a short name for it.
** The Setup card then treats plan crons as firing.
*******************************************************************************/
public String getHostSchedulerDescription()
{
return (hostSchedulerDescription);
}
/*******************************************************************************
** Fluent setter for hostSchedulerDescription.
*******************************************************************************/
public BackupsQBitConfig withHostSchedulerDescription(String hostSchedulerDescription)
{
this.hostSchedulerDescription = hostSchedulerDescription;
return (this);
}
/*******************************************************************************
** Getter for hostSchedulerRunsVerification: whether the host's own tick also runs
** the verify cron (by calling the backupVerifyPlan process). False by default.
*******************************************************************************/
public Boolean getHostSchedulerRunsVerification()
{
return (hostSchedulerRunsVerification);
}
/*******************************************************************************
** Fluent setter for hostSchedulerRunsVerification.
*******************************************************************************/
public BackupsQBitConfig withHostSchedulerRunsVerification(Boolean hostSchedulerRunsVerification)
{
this.hostSchedulerRunsVerification = hostSchedulerRunsVerification;
return (this);
}
/*******************************************************************************
** Getter for schedulerName (the QQQ scheduler used for plan jobs and the SLA checker)
*******************************************************************************/
@@ -119,13 +119,8 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
qInstance.addQBit(qBit);
BackupsRuntime.install(qBitConfig);
///////////////////////////////////////////////////////////////////////////////
// bind the Alerts qbit producer SPI when it is on the class path (optional) //
///////////////////////////////////////////////////////////////////////////////
if(com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfPresent())
{
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
}
com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfAbsent();
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput output = new com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput();
for(var possibleValueSource : com.kingsrook.qbits.backups.model.BackupsTables.possibleValueSources())
@@ -165,6 +160,7 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData app = com.kingsrook.qbits.backups.app.BackupsApp.app(qBitConfig, tables);
output.add(app);
output.addSelfToInstance(qInstance);
com.kingsrook.qbits.backups.model.BackupsHelp.apply(qInstance, app.getName());
// A parent app only lists the apps that are its children, so attach ourselves (for example Admin, then Backups).
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData parent = qBitConfig.getParentAppName() == null ? null : qInstance.getApp(qBitConfig.getParentAppName());
@@ -17,105 +17,39 @@
package com.kingsrook.qbits.backups.alerts;
import java.lang.reflect.Method;
import java.util.Map;
import java.util.stream.Collectors;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** The BackupAlertSink adapter for the Alerts qbit producer SPI (QRun-IO/qqq
** PR #1108, com.kingsrook.qqq.backend.core.alerts: QAlerts, AlertSignal,
** AlertResolution, AlertSeverity, QAlertTypeMetaData; design spec
** docs/specs/2026-10-12-qbit-alerts-design.md).
**
** STAND-IN: the SPI is an open core PR and is not in QQQ 4.1.0-RC.4, so this
** adapter is bound REFLECTIVELY and optionally. It compiles against nothing in
** the SPI; it looks the classes up by name, and every call is guarded:
** <ul>
** <li>no SPI classes on the class path: {@link #bindIfPresent()} binds nothing
** and the sink stays the no-op;</li>
** <li>the SPI is present but the instance has no alerts meta-data (the Alerts
** qbit is not installed): every call is a no-op;</li>
** <li>any reflective or SPI failure is logged once and swallowed, because an
** alerting problem must never fail a backup.</li>
** </ul>
** Replace with a direct binding (QAlerts.raise and friends) when RC.5 ships.
**
** Mapping: raise to QAlerts.raise(AlertSignal), resolve to QAlerts.resolve(type,
** dedupKey, note), heartbeat to QAlerts.heartbeat(key). The SPI has no per-plan
** heartbeat registration (monitors are declared on the alert type), so the
** expected interval and grace are not passed on; the SLA checker raises
** backups.plan.missed itself, independent of any heartbeat (spec 10.4).
** Direct adapter to core QAlerts. Calls are no-ops without a configured alerts
** sink, including when only the backup alert types have been registered.
** QAlerts catches and logs dispatch failures; adapter failures are logged once.
** Heartbeat intervals remain the responsibility of the backup SLA checker.
*******************************************************************************/
public class AlertsQBitAlertSink implements BackupAlertSink
{
private static final QLogger LOG = QLogger.getLogger(AlertsQBitAlertSink.class);
/** The package of the Alerts producer SPI in qqq-backend-core. */
public static final String DEFAULT_API_PACKAGE = "com.kingsrook.qqq.backend.core.alerts";
private final String apiPackage;
private boolean warned = false;
private Class<?> qAlerts;
private Class<?> alertSignal;
private Class<?> alertSeverity;
private Class<?> instanceMetaData;
/*******************************************************************************
** Constructor for the real SPI package.
** Bind the core adapter unless the host already supplied a backup sink.
*******************************************************************************/
public AlertsQBitAlertSink()
public static boolean bindIfAbsent()
{
this(DEFAULT_API_PACKAGE);
}
/*******************************************************************************
** Constructor for another package (a test double of the SPI shape).
*******************************************************************************/
public AlertsQBitAlertSink(String apiPackage)
{
this.apiPackage = apiPackage;
}
/*******************************************************************************
** Whether the SPI classes are on the class path.
*******************************************************************************/
public static boolean isPresent(String apiPackage)
{
try
{
Class.forName(apiPackage + ".QAlerts");
Class.forName(apiPackage + ".AlertSignal");
return (true);
}
catch(ClassNotFoundException | LinkageError e)
{
return (false);
}
}
/*******************************************************************************
** Bind this adapter as the runtime's alert sink when the SPI is present and no
** other sink was bound already. Returns whether it bound.
*******************************************************************************/
public static boolean bindIfPresent()
{
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink) || !isPresent(DEFAULT_API_PACKAGE))
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink))
{
return (false);
}
@@ -126,38 +60,32 @@ public class AlertsQBitAlertSink implements BackupAlertSink
/*******************************************************************************
** Declare the backup alert types with the Alerts qbit (QAlertTypeMetaData), when
** the SPI is present. Returns how many were added.
** Declare missing backup alert types on each instance, independently of sink
** installation order. Returns how many were added.
*******************************************************************************/
public int registerAlertTypes(QInstance qInstance)
{
int added = 0;
try
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
{
Class<?> typeClass = Class.forName(apiPackage + ".QAlertTypeMetaData");
load();
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
try
{
try
if(QAlertTypeMetaData.of(qInstance, alertType.name()) == null)
{
Object type = typeClass.getConstructor().newInstance();
typeClass.getMethod("withName", String.class).invoke(type, alertType.name());
typeClass.getMethod("withLabel", String.class).invoke(type, alertType.label());
typeClass.getMethod("withGroupName", String.class).invoke(type, "Backups");
typeClass.getMethod("withHowToFix", String.class).invoke(type, alertType.howToFix());
typeClass.getMethod("withDefaultSeverity", alertSeverity).invoke(type, severity(alertType.defaultSeverity()));
typeClass.getMethod("addSelfToInstance", QInstance.class).invoke(type, qInstance);
new QAlertTypeMetaData()
.withName(alertType.name())
.withLabel(alertType.label())
.withGroupName("Backups")
.withHowToFix(alertType.howToFix())
.withDefaultSeverity(AlertSeverity.valueOf(alertType.defaultSeverity().name()))
.addSelfToInstance(qInstance);
added++;
}
catch(ReflectiveOperationException | RuntimeException e)
{
warnOnce("Could not declare the alert type " + alertType.name(), e);
}
}
}
catch(ReflectiveOperationException | LinkageError e)
{
warnOnce("The alerts SPI is not available", e);
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not declare the alert type " + alertType.name(), e);
}
}
return (added);
}
@@ -170,28 +98,24 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void raise(String type, String dedupeKey, Severity severity, String title, Map<String, String> detail)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
Object signal = alertSignal.getConstructor(String.class).newInstance(type);
alertSignal.getMethod("setDedupKey", String.class).invoke(signal, dedupeKey);
alertSignal.getMethod("setSeverity", alertSeverity).invoke(signal, severity(severity));
alertSignal.getMethod("setTitle", String.class).invoke(signal, title);
if(!isAlertsQBitInstalled())
{
return;
}
AlertSignal signal = new AlertSignal(type)
.withDedupKey(dedupeKey)
.withSeverity(AlertSeverity.valueOf(severity.name()))
.withTitle(title);
if(detail != null && !detail.isEmpty())
{
alertSignal.getMethod("setSummary", String.class).invoke(signal, detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
Method withAttribute = alertSignal.getMethod("withAttribute", String.class, String.class);
for(Map.Entry<String, String> entry : detail.entrySet())
{
withAttribute.invoke(signal, entry.getKey(), entry.getValue());
}
signal.withSummary(detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
detail.forEach(signal::withAttribute);
}
qAlerts.getMethod("raise", alertSignal).invoke(null, signal);
QAlerts.raise(signal);
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not raise an alert", e);
}
@@ -205,15 +129,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void resolve(String type, String dedupeKey)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
qAlerts.getMethod("resolve", String.class, String.class, String.class).invoke(null, type, dedupeKey, "Cleared by the Backups qbit");
if(isAlertsQBitInstalled())
{
QAlerts.resolve(type, dedupeKey, "Cleared by the Backups qbit");
}
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not resolve an alert", e);
}
@@ -227,15 +150,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void heartbeat(String monitorKey, long expectedEverySeconds, long graceSeconds)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
qAlerts.getMethod("heartbeat", String.class).invoke(null, monitorKey);
if(isAlertsQBitInstalled())
{
QAlerts.heartbeat(monitorKey);
}
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not check in a heartbeat", e);
}
@@ -244,67 +166,26 @@ public class AlertsQBitAlertSink implements BackupAlertSink
/*******************************************************************************
** The SPI classes load, and the instance has alerts meta-data (the Alerts qbit
** is installed). Checked on every call so the order qbits are produced in
** does not matter.
** Check on every call so installing the alerts sink after this qbit works.
** Type declarations alone do not mean an alerts implementation is installed.
*******************************************************************************/
boolean isAlertsQBitInstalled()
{
try
{
load();
QInstance qInstance = QContext.getQInstance();
if(qInstance == null)
{
return (false);
}
return (instanceMetaData.getMethod("of", QInstance.class).invoke(null, qInstance) != null);
}
catch(ReflectiveOperationException | RuntimeException | LinkageError e)
{
warnOnce("The alerts SPI is not usable", e);
return (false);
}
AlertsInstanceMetaData metaData = AlertsInstanceMetaData.of(QContext.getQInstance());
return (metaData != null && metaData.getSink() != null);
}
/*******************************************************************************
**
** Log adapter failures once; dispatch failures are already logged by core.
*******************************************************************************/
private synchronized void load() throws ClassNotFoundException
{
if(qAlerts == null)
{
qAlerts = Class.forName(apiPackage + ".QAlerts");
alertSignal = Class.forName(apiPackage + ".AlertSignal");
alertSeverity = Class.forName(apiPackage + ".AlertSeverity");
instanceMetaData = Class.forName(apiPackage + ".AlertsInstanceMetaData");
}
}
/*******************************************************************************
** Map our severity to the SPI's enum constant of the same name.
*******************************************************************************/
@SuppressWarnings({ "unchecked", "rawtypes" })
private Object severity(Severity severity)
{
return (Enum.valueOf((Class<Enum>) alertSeverity, severity.name()));
}
/*******************************************************************************
**
*******************************************************************************/
private void warnOnce(String message, Throwable t)
private synchronized void warnOnce(String message, Throwable t)
{
if(!warned)
{
warned = true;
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t, logPair("apiPackage", apiPackage));
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t);
}
}
}
@@ -36,11 +36,11 @@ public final class BackupAlerts
{
public static final String PLAN_MISSED = "backups.plan.missed";
public static final String RUN_FAILED = "backups.run.failed";
public static final String RUN_MISSING = "backups.run.missing_in_storage";
public static final String RUN_MISSING = "backups.run.missing-in-storage";
public static final String FILES_MISSING = "backups.files.missing";
public static final String VERIFY_FAILED = "backups.verify.failed";
public static final String VERIFY_OVERDUE = "backups.verify.overdue";
public static final String PLAN_OWNER_INVALID = "backups.plan.owner_invalid";
public static final String PLAN_OWNER_INVALID = "backups.plan.owner-invalid";
public static final String STORAGE_DEBRIS = "backups.storage.debris";
public static final String RESTORE_COMPLETED = "backups.restore.completed";
@@ -70,6 +70,7 @@ public final class BackupsHomeView
{
StringBuilder html = new StringBuilder("<div class=\"backups-home\" style=\"" + ROOT + "\">");
planStatus(html, data, pathPrefix);
setup(html, data, pathPrefix);
html.append("<div style=\"").append(GRID).append("\">");
needsAttention(html, data, pathPrefix);
latestTestRestore(html, data, pathPrefix);
@@ -139,6 +140,50 @@ public final class BackupsHomeView
/*******************************************************************************
** The Setup card: what a working backup needs, whether each part is in place,
** and the next step, linked to its table.
*******************************************************************************/
private static void setup(StringBuilder html, BackupsHomeData data, String prefix)
{
List<BackupsSetupStatus.Item> items = BackupsSetupStatus.load(data.plans());
open(html, "Setup");
long todo = items.stream().filter(i -> i.state() == BackupsSetupStatus.State.TODO).count();
html.append("<p style=\"margin:0 0 .6rem;font-size:.9rem;\">")
.append(todo == 0 ? "Everything backups need is set up." : esc(todo + (todo == 1 ? " item needs" : " items need") + " attention."))
.append("</p><ul style=\"list-style:none;margin:0;padding:0;\">");
for(BackupsSetupStatus.Item item : items)
{
String color = switch(item.state())
{
case OK -> "#16a34a";
case TODO -> "#dc2626";
case DEFAULT -> "#d97706";
case OPTIONAL -> "#9ca3af";
};
String label = switch(item.state())
{
case OK -> "Set up";
case TODO -> "Needs setup";
case DEFAULT -> "Using defaults";
case OPTIONAL -> "Optional";
};
html.append("<li style=\"").append(ROW).append("\"><div style=\"display:flex;flex-wrap:wrap;align-items:center;gap:.5rem .75rem;\">")
.append("<strong>").append("<a style=\"").append(LINK).append("\" href=\"").append(esc(prefix + "/" + item.tableName())).append("\">").append(esc(item.label())).append("</a></strong>")
.append("<span style=\"").append(CHIP).append("\"><span aria-hidden=\"true\" style=\"width:.55rem;height:.55rem;border-radius:50%;background:").append(color).append(";display:inline-block;\"></span>").append(esc(label)).append("</span>")
.append("</div><div style=\"font-size:.9rem;margin-top:.35rem;\">").append(esc(item.detail())).append("</div>");
if(item.nextStep() != null)
{
html.append("<div style=\"").append(MUTED).append("font-size:.9rem;margin-top:.2rem;\">Next: ").append(esc(item.nextStep())).append("</div>");
}
html.append("</li>");
}
html.append("</ul>");
close(html);
}
/*******************************************************************************
** Needs attention.
*******************************************************************************/
@@ -0,0 +1,207 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.app;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** The Setup card on the Backups home: what a working backup needs (a plan, a
** storage bucket that resolves, a schedule, verification, a separate prune
** bucket), whether each is in place, and the next step. Computed from the plans
** the home already loaded and the runtime's storage resolver; it reads nothing
** else.
*******************************************************************************/
public final class BackupsSetupStatus
{
/*******************************************************************************
** OK: in place. TODO: backups need it. DEFAULT: working, with a default worth a
** look. OPTIONAL: a feature that is off until set up.
*******************************************************************************/
public enum State
{
OK, TODO, DEFAULT, OPTIONAL
}
/*******************************************************************************
** One line of the card.
*******************************************************************************/
public record Item(String label, String tableName, State state, String detail, String nextStep)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BackupsSetupStatus()
{
}
/*******************************************************************************
** Build the items from the plans.
*******************************************************************************/
public static List<Item> load(List<QRecord> plans)
{
BackupsQBitConfig config = BackupsRuntime.isInstalled() ? BackupsRuntime.getConfig() : null;
return (load(plans, config != null && hasContent(config.getSchedulerName()), config == null ? null : config.getHostSchedulerDescription(),
config != null && Boolean.TRUE.equals(config.getHostSchedulerRunsVerification())));
}
/*******************************************************************************
** Build the items from the plans, given whether a scheduler is configured (a plan
** cron only fires through the scheduler the qbit config names).
*******************************************************************************/
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured)
{
return (load(plans, schedulerConfigured, null, false));
}
/*******************************************************************************
** Build the items from the plans. A host that runs plan crons with its own tick
** (hostScheduler names it) has crons that fire; hostRunsVerification says whether
** that tick also runs the verify cron.
*******************************************************************************/
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured, String hostScheduler, boolean hostRunsVerification)
{
boolean host = hasContent(hostScheduler);
List<Item> items = new ArrayList<>();
List<QRecord> active = plans.stream().filter(p -> Boolean.TRUE.equals(p.getValueBoolean("isActive"))).toList();
///////////
// plans //
///////////
if(plans.isEmpty())
{
items.add(new Item("Backup plans", BackupsTables.PLAN, State.TODO, "No plan: nothing is backed up.",
"Run Create default plan on Backup plans (it asks only for the storage bucket alias), or add a plan with New."));
items.add(new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "None needed yet.",
"A plan that backs up everything (scope Instance) needs no rules. Add them on a plan that is limited to apps, qbits or tables."));
return (items);
}
items.add(active.isEmpty()
? new Item("Backup plans", BackupsTables.PLAN, State.TODO, plans.size() + (plans.size() == 1 ? " plan, but it is" : " plans, but none is") + " active: nothing runs on its own.", "Turn a plan on (Active).")
: new Item("Backup plans", BackupsTables.PLAN, State.OK, active.size() + " active plan" + (active.size() == 1 ? "" : "s") + " of " + plans.size() + ".", null));
/////////////////////////////////////////
// scope rules: needed unless INSTANCE //
/////////////////////////////////////////
List<String> limited = plans.stream().filter(p -> p.getValueString("scopeType") != null && !"INSTANCE".equals(p.getValueString("scopeType"))).map(p -> p.getValueString("name")).toList();
items.add(limited.isEmpty()
? new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "Every plan backs up the whole instance, so no rules are needed.",
"Add rules to a plan only to include or leave out apps, qbits, tables or roles.")
: new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OK, limited.size() + (limited.size() == 1 ? " plan is" : " plans are") + " limited by scope rules: " + String.join(", ", limited) + ".",
"Check each plan's Scope rules to see what it covers."));
///////////////////////////////////////////////
// storage: every alias a plan names resolves //
///////////////////////////////////////////////
Set<String> aliases = new LinkedHashSet<>();
for(QRecord plan : plans)
{
for(String field : List.of("storageBucketAlias", "pruneBucketAlias"))
{
if(plan.getValueString(field) != null)
{
aliases.add(plan.getValueString(field));
}
}
}
List<String> broken = new ArrayList<>();
for(String alias : aliases)
{
try
{
BackupsRuntime.storageFor(alias);
}
catch(RuntimeException e)
{
broken.add(alias);
}
}
items.add(broken.isEmpty()
? new Item("Storage", BackupsTables.PLAN, State.OK, aliases.size() + " storage bucket alias" + (aliases.size() == 1 ? "" : "es") + " in use, and each resolves.", "Run Test destination on a plan to prove it can write.")
: new Item("Storage", BackupsTables.PLAN, State.TODO, "These aliases do not resolve to a bucket: " + String.join(", ", broken) + ".",
"Fix the alias on the plan, or add the bucket to the application's storage resolver."));
/////////////////////////////////////////
// schedule: an active plan has a cron //
/////////////////////////////////////////
long scheduled = active.stream().filter(p -> hasContent(p.getValueString("cronExpression"))).count();
String noScheduler = "No scheduler is configured (schedulerName on the qbit config), so plan crons are saved but never fire";
items.add(!schedulerConfigured && !host
? new Item("Schedule", BackupsTables.PLAN, State.TODO, noScheduler + ". Backups run only when someone clicks Run backup now.",
"Set schedulerName on the qbit config (or run your own tick if the application does), then give each plan a cron expression.")
: scheduled == 0
? new Item("Schedule", BackupsTables.PLAN, State.TODO, "No active plan has a schedule: backups run only when someone clicks Run backup now.", "Set a Cron expression on a plan (for example " + DefaultPlanProcess.CRON + ").")
: new Item("Schedule", BackupsTables.PLAN, State.OK, scheduled + " active plan" + (scheduled == 1 ? " runs" : "s run") + " on a schedule"
+ (host && !schedulerConfigured ? ", run by " + hostScheduler + " (the application scheduler)." : "."), null));
//////////////////
// verification //
//////////////////
long verified = active.stream().filter(p -> hasContent(p.getValueString("verifyCronExpression"))).count();
items.add(host && !schedulerConfigured && !hostRunsVerification
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "Not run by " + hostScheduler + ": verify cron expressions are saved but nothing fires them.",
"Run Verify backup on a run by hand, or have the application scheduler run the backupVerifyPlan process for each plan.")
: !schedulerConfigured && !host
? new Item("Verification", BackupsTables.VERIFICATION, State.TODO, noScheduler + ", so no verification is scheduled.",
"Set schedulerName on the qbit config, then a verify cron expression on a plan; or run Verify backup on a run by hand.")
: verified == 0
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "No plan verifies its backups on a schedule.",
"Set a Verify cron expression on a plan, or run Verify backup on a run. A backup nobody has restored is a guess.")
: new Item("Verification", BackupsTables.VERIFICATION, State.OK, verified + " active plan" + (verified == 1 ? " verifies" : "s verify") + " its latest backup on a schedule.", null));
////////////////////////////////////////////
// prune bucket: separate delete authority //
////////////////////////////////////////////
long noPrune = plans.stream().filter(p -> !hasContent(p.getValueString("pruneBucketAlias"))).count();
items.add(noPrune == 0
? new Item("Prune bucket", BackupsTables.PLAN, State.OK, "Every plan prunes through its own delete-capable bucket alias.", null)
: new Item("Prune bucket", BackupsTables.PLAN, State.DEFAULT, noPrune + (noPrune == 1 ? " plan has" : " plans have") + " no prune bucket: Prune uses the backup bucket, so one key can both write and delete backups.",
"Give the plan a separate delete-capable Prune bucket alias."));
return (items);
}
/*******************************************************************************
**
*******************************************************************************/
private static boolean hasContent(String value)
{
return (value != null && !value.isBlank());
}
}
@@ -137,6 +137,7 @@ public record BackupManifest(
String backend,
String primaryKey,
String fingerprint,
String fingerprintAlgorithm,
Map<String, Object> schema,
String filter,
List<ExcludedField> excludedFields,
@@ -0,0 +1,170 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
/*******************************************************************************
** Frozen pre-RC.6 fingerprint algorithm, used only to read existing manifests.
** Keep its descriptor and escaping byte-compatible with the original writer.
*******************************************************************************/
final class LegacyTableFingerprint
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private LegacyTableFingerprint()
{
}
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -18,28 +18,23 @@ package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint;
/*******************************************************************************
** A stable schema descriptor and SHA-256 fingerprint of a table (spec 5.5).
**
** This is an internal stand-in for the QQQ core TableSchemaFingerprint (core PR
** feature/schema-fingerprint). When that ships in the next 4.1.x RC, this class
** delegates to it and the descriptor below is kept only for reading old manifests.
** Labels, adornments and help text are excluded, so cosmetic changes do not count.
** Core schema fingerprints with read compatibility for pre-RC.6 manifests.
*******************************************************************************/
public final class TableFingerprints
{
public static final String CORE_ALGORITHM = "qqq-table-schema-v1";
public static final String LEGACY_ALGORITHM = "qbit-backups-table-v1";
/*******************************************************************************
** Utility class.
@@ -51,134 +46,83 @@ public final class TableFingerprints
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
** The core descriptor stored in new manifests.
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
return (TableSchemaFingerprint.of(table, backendType).getDescriptor());
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
** Retain the manifest's sha256 prefix around the core hash.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
return ("sha256:" + TableSchemaFingerprint.of(table, backendType).getHash());
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
** An absent algorithm identifies manifests written before RC.6. Unknown
** algorithms must not silently fall back to a different fingerprint.
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
public static boolean matches(BackupManifest.TableEntry entry, QTableMetaData table, String backendType)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
if(entry.fingerprint() == null)
{
return (false);
}
String algorithm = entry.fingerprintAlgorithm();
if(algorithm == null || LEGACY_ALGORITHM.equals(algorithm))
{
return (entry.fingerprint().equals(LegacyTableFingerprint.of(table, backendType)));
}
return (CORE_ALGORITHM.equals(algorithm) && entry.fingerprint().equals(of(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a whole instance's tables: a hash of the table fingerprints.
** Normalize both descriptor versions to field-name maps for restore's
** compatibility analysis. Core fields use a list and explicit flag names.
*******************************************************************************/
@SuppressWarnings("unchecked")
public static Map<String, Object> fields(Map<String, Object> descriptor)
{
Object fields = descriptor == null ? null : descriptor.get("fields");
if(fields instanceof Map<?, ?> map)
{
return ((Map<String, Object>) map);
}
Map<String, Object> result = new LinkedHashMap<>();
if(fields instanceof List<?> list)
{
for(Object item : list)
{
Map<String, Object> field = (Map<String, Object>) item;
Map<String, Object> normalized = new LinkedHashMap<>();
normalized.put("type", field.get("type"));
normalized.put("required", field.get("isRequired"));
normalized.put("maxLength", field.get("maxLength"));
normalized.put("pvs", field.get("possibleValueSource"));
normalized.put("heavy", field.get("isHeavy"));
normalized.put("sensitive", field.get("isSensitive"));
result.put((String) field.get("name"), normalized);
}
}
return (result);
}
/*******************************************************************************
** The instance fingerprint remains a hash of the ordered table fingerprints.
*******************************************************************************/
public static String combine(List<String> tableFingerprints)
{
return ("sha256:" + HashingOutputStream.sha256Hex(String.join("\n", tableFingerprints).getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -0,0 +1,126 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.model;
import java.io.IOException;
import java.io.InputStream;
import java.util.LinkedHashMap;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Applies help/qbit-backups.yaml (the QQQ help-bundle format) to the Backups app
** and its backups_* tables: what each table is for and how its rows get created,
** shown on the table screen and when it is empty.
** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with HelpBundles.register once the pinned QQQ
** release includes it; the bundle file stays the same.
*******************************************************************************/
public final class BackupsHelp
{
public static final String RESOURCE = "/help/qbit-backups.yaml";
public static final String SLOT = "help";
public static final String APP_KEY = "app:backups;slot:help";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BackupsHelp()
{
}
/*******************************************************************************
** The bundle's entries by key, validated (fails startup if malformed).
*******************************************************************************/
public static Map<String, JsonNode> entries() throws QException
{
try(InputStream stream = BackupsHelp.class.getResourceAsStream(RESOURCE))
{
if(stream == null)
{
throw (new IOException("missing resource"));
}
JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
if(!"qbit-backups".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
{
throw (new IOException("invalid bundle header"));
}
Map<String, JsonNode> entries = new LinkedHashMap<>();
for(JsonNode entry : bundle.path("entries"))
{
String key = entry.path("key").asText();
String summary = entry.path("summary").asText();
if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
{
throw (new IOException("invalid or duplicate entry: " + key));
}
}
return (entries);
}
catch(IOException | RuntimeException e)
{
throw (new QException("Could not load " + RESOURCE, e));
}
}
/*******************************************************************************
** Apply the help to the app and every backups_* table in the instance. A host's
** own help in the slot is kept (the bundle only fills it).
*******************************************************************************/
public static void apply(QInstance qInstance, String appName) throws QException
{
Map<String, JsonNode> entries = entries();
for(QTableMetaData table : qInstance.getTables().values())
{
JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
{
table.withHelpContent(SLOT, content(entry));
}
}
QAppMetaData app = qInstance.getApp(appName);
JsonNode appEntry = entries.get(APP_KEY);
if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
{
app.withHelpContent(SLOT, content(appEntry));
}
}
/*******************************************************************************
** Summary as the first paragraph, then the content, as markdown.
*******************************************************************************/
static QHelpContent content(JsonNode entry)
{
return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
}
}
@@ -38,6 +38,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
@@ -446,8 +447,8 @@ public final class BackupsTables
.withField(new QFieldMetaData("pruneBucketAlias", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("prefix", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("critical", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(enumField("format", "format", "JSONL"))
.withField(enumField("compression", "compression", "GZIP"))
.withField(enumField("format", "format", "JSONL").withHelpContent(new QHelpContent("JSONL, CSV and JSONL_AND_CSV are available. PARQUET is not available yet and is refused when the plan is saved.")))
.withField(enumField("compression", "compression", "GZIP").withHelpContent(new QHelpContent("GZIP and NONE are available. ZSTD is not available yet and is refused when the plan is saved.")))
.withField(enumField("encryptionMode", "encryption_mode", "NONE"))
.withField(new QFieldMetaData("encryptionKeyRef", QFieldType.STRING).withMaxLength(100))
.withField(enumField("sensitiveFieldPolicy", "sensitive_policy", "EXCLUDE"))
@@ -455,12 +456,15 @@ public final class BackupsTables
.withField(enumField("fileObjectPolicy", "file_policy", "COPY"))
.withField(enumField("fileMissingPolicy", "file_missing_policy", "WARN"))
.withField(enumField("consistency", "consistency", "SNAPSHOT_PREFERRED"))
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(enumField("incrementalMode", "incremental_mode", "FULL"))
.withField(enumField("nativeDump", "native_dump", "NONE"))
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false)
.withHelpContent(new QHelpContent("Not available yet: backups always read from the application's own backend. Leave this off; turning it on is refused when the plan is saved.")))
.withField(enumField("incrementalMode", "incremental_mode", "FULL")
.withHelpContent(new QHelpContent("Only FULL is available yet. CHANGED_SINCE_LAST is not available yet and is refused when the plan is saved.")))
.withField(enumField("nativeDump", "native_dump", "NONE")
.withHelpContent(new QHelpContent("Only NONE is available yet. Native database dumps (PG_DUMP_CUSTOM) are not available yet and are refused when the plan is saved.")))
.withField(new QFieldMetaData("cronExpression", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("cronTimeZoneId", QFieldType.STRING).withMaxLength(60))
.withField(roInt("scheduledJobId"))
.withField(roInt("scheduledJobId").withHelpContent(new QHelpContent("Not used: a plan's schedules are scheduled jobs found by their foreign key (backups_plan for the backup, backups_plan_verify for the verification), so look there.")))
.withField(new QFieldMetaData("slaHours", QFieldType.INTEGER).withDefaultValue(26))
.withField(new QFieldMetaData("verifySlaDays", QFieldType.INTEGER).withDefaultValue(8))
.withField(new QFieldMetaData("verifyFileSamplePercent", QFieldType.INTEGER).withDefaultValue(100))
@@ -48,7 +48,7 @@ import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
public class BackupPlanTableCustomizer implements TableCustomizerInterface
{
private static final List<String> CONFIGURATION_FIELDS = List.of("planKey", "isActive", "scopeType", "includeRoles", "storageBucketAlias", "pruneBucketAlias",
"format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "incrementalMode", "nativeDump", "cronExpression", "cronTimeZoneId",
"format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "incrementalMode", "nativeDump", "useReadReplica", "cronExpression", "cronTimeZoneId",
"verifyCronExpression", "retainDaily", "retainWeekly", "retainMonthly", "retainYearly", "retainMinimum", "objectLockMode", "objectLockDays",
"slaHours", "pageSize", "ownerUserId");
@@ -176,6 +176,10 @@ public final class BackupPlanValidator
{
errors.add("Native database dumps are not available yet. Use NONE.");
}
if(settings.bool("useReadReplica", false))
{
errors.add("Reading from a replica is not available yet: backups read from the application's own backend. Turn Use read replica off.");
}
}
@@ -72,6 +72,7 @@ public final class BackupsProcesses
List<QProcessMetaData> processes = new ArrayList<>();
processes.add(permission(etl(RUN_PLAN, "Run backup now", BackupsTables.PLAN, RunPlanTransform.class, RunPlanLoad.class,
"This is a preview of what the backup will include. Nothing is written until you continue."), BackupsPermissions.RUN));
processes.add(DefaultPlanProcess.process());
processes.addAll(VerifyProcesses.processes());
processes.addAll(RestoreProcesses.processes());
processes.addAll(CatalogProcesses.processes());
@@ -0,0 +1,314 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.processes;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
import com.kingsrook.qbits.backups.plan.BackupPlanValidator;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.scope.PlanSettings;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** "Create default plan": a table action on backups_plan that adds one working
** plan, "Nightly full", so a fresh install has something to run. The only thing
** it asks for is the storage bucket alias, because that is the one value the qbit
** cannot know (it names a bucket the host's storage resolver provides). Review
** shows the plan that will be created and checks it against the same rules a
** hand-made plan must pass; Result links the new plan.
**
** Insert-only: when a plan with the key nightly-full exists (edited or not), nothing
** is created or changed. The plan goes through the normal insert, so the owner is
** the user running the action and the schedule syncs like any other plan.
*******************************************************************************/
public final class DefaultPlanProcess
{
public static final String NAME = "backupsPlanCreateDefault";
public static final String FIELD_ALIAS = "storageBucketAlias";
public static final String PLAN_KEY = "nightly-full";
public static final String PLAN_NAME = "Nightly full";
public static final String CRON = "0 0 2 * * ?";
public static final String VERIFY_CRON = "0 0 4 ? * SUN";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private DefaultPlanProcess()
{
}
/*******************************************************************************
** The process: a table-level action with one question (the bucket alias).
*******************************************************************************/
public static QProcessMetaData process()
{
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, BackupsTables.PLAN);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, BackupsTables.PLAN);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, "This plan will be created. Nothing is written until you continue. You can edit every setting afterwards.");
QProcessMetaData process = StreamedETLWithFrontendProcess.defineProcessMetaData(Extract.class, Transform.class, Load.class, values);
process.setName(NAME);
process.setLabel("Create default plan");
process.setIcon(new QIcon("playlist_add"));
process.setTableName(BackupsTables.PLAN);
process.withStep(0, new QFrontendStepMetaData()
.withName("input")
.withLabel("Where should the backups go?")
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM))
.withFormField(new QFieldMetaData(FIELD_ALIAS, QFieldType.STRING).withLabel("Storage bucket alias").withIsRequired(true)
.withHelpContent(new QHelpContent("The alias of a bucket your application provides to Backups (never a key or an endpoint). It becomes the plan's Storage bucket alias."))));
process.setMinInputRecords(0);
process.setPermissionRules(BackupsTables.permissionRules());
return (BackupsProcesses.permission(process, BackupsPermissions.ADMIN));
}
/*******************************************************************************
** The values of the default plan for a bucket alias. Every setting is explicit
** so the plan does not depend on the backend applying field defaults.
*******************************************************************************/
public static Map<String, Serializable> planValues(String alias)
{
Map<String, Serializable> plan = new LinkedHashMap<>();
plan.put("name", PLAN_NAME);
plan.put("planKey", PLAN_KEY);
plan.put("description", "Every night: all data and audit tables of the application, and the files they reference, to the storage bucket. Edit any setting on this plan.");
plan.put("isActive", true);
plan.put("scopeType", "INSTANCE");
plan.put("includeRoles", "DATA,AUDIT");
plan.put("storageBucketAlias", alias);
plan.put("critical", true);
plan.put("format", "JSONL");
plan.put("compression", "GZIP");
plan.put("encryptionMode", "NONE");
plan.put("sensitiveFieldPolicy", "EXCLUDE");
plan.put("heavyFieldPolicy", "INLINE");
plan.put("fileObjectPolicy", "COPY");
plan.put("fileMissingPolicy", "WARN");
plan.put("consistency", "SNAPSHOT_PREFERRED");
plan.put("useReadReplica", false);
plan.put("incrementalMode", "FULL");
plan.put("nativeDump", "NONE");
plan.put("cronExpression", CRON);
plan.put("cronTimeZoneId", "UTC");
plan.put("slaHours", 26);
plan.put("verifySlaDays", 8);
plan.put("verifyFileSamplePercent", 100);
plan.put("verifyCronExpression", VERIFY_CRON);
plan.put("verifyLevel", "CONTENT");
plan.put("retainDaily", 7);
plan.put("retainWeekly", 4);
plan.put("retainMonthly", 12);
plan.put("retainYearly", 0);
plan.put("retainMinimum", 3);
plan.put("objectLockMode", "NONE");
plan.put("maxDurationMinutes", 240);
plan.put("pageSize", 5000);
plan.put("targetPartBytes", 268435456L);
return (plan);
}
/*******************************************************************************
** Whether the default plan already exists.
*******************************************************************************/
private static boolean exists() throws QException
{
return (!new BackupCatalog().where(BackupsTables.PLAN, "planKey", PLAN_KEY).isEmpty());
}
/*******************************************************************************
** Extract: the plan to create, or nothing when it exists already.
*******************************************************************************/
public static class Extract extends AbstractExtractStep
{
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
String alias = input.getValueString(FIELD_ALIAS);
if(alias == null || alias.isBlank())
{
throw (new QUserFacingException("Enter the storage bucket alias the backups should go to."));
}
if(!exists())
{
QRecord record = new QRecord().withTableName(BackupsTables.PLAN);
record.setValues(new LinkedHashMap<>(planValues(alias.strip())));
getRecordPipe().addRecord(record);
}
}
}
/*******************************************************************************
** Review: the plan, checked against the rules a plan must pass.
*******************************************************************************/
public static class Transform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine create = BackupsProcesses.line(Status.OK, "plan will be created", "plans will be created", "plan was created", "plans were created");
private final ProcessSummaryLine invalid = BackupsProcesses.line(Status.ERROR, "plan cannot be created", "plans cannot be created", "plan could not be created", "plans could not be created");
private final List<String> problems = new ArrayList<>();
private boolean sawRecord;
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord record : input.getRecords())
{
sawRecord = true;
BackupPlanValidator.Result result = BackupPlanValidator.validate(PlanSettings.of(record), null, false, List.of());
boolean nameTaken = !new BackupCatalog().where(BackupsTables.PLAN, "name", record.getValueString("name")).isEmpty();
if(nameTaken)
{
problems.add("A plan named " + record.getValueString("name") + " already exists. Rename it or edit that plan.");
}
if(result.ok() && !nameTaken)
{
create.incrementCountAndAddPrimaryKey(record.getValueString("planKey"));
output.getRecords().add(record);
}
else
{
invalid.incrementCount();
problems.addAll(result.errors());
}
}
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
create.addSelfToListIfAnyCount(lines);
invalid.setBulletsOfText(new ArrayList<>(problems));
invalid.addSelfToListIfAnyCount(lines);
if(!sawRecord)
{
lines.add(new ProcessSummaryLine(Status.INFO, null, "A plan with the key " + PLAN_KEY + " already exists, so nothing will be created. Edit that plan instead."));
}
return (lines);
}
}
/*******************************************************************************
** Load: insert through the normal insert (validation, owner, schedule).
*******************************************************************************/
public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine created = BackupsProcesses.line(Status.OK, "plan will be created", "plans will be created", "plan was created", "plans were created");
private final ProcessSummaryLine failed = BackupsProcesses.line(Status.ERROR, "plan could not be created", "plans could not be created", "plan could not be created", "plans could not be created");
private final List<String> problems = new ArrayList<>();
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
InsertInput insert = new InsertInput(BackupsTables.PLAN);
insert.setRecords(input.getRecords());
InsertOutput inserted = new InsertAction().execute(insert);
for(QRecord record : inserted.getRecords())
{
if(record.getErrors() == null || record.getErrors().isEmpty())
{
created.incrementCountAndAddPrimaryKey(record.getValue("id"));
output.getRecords().add(record);
}
else
{
failed.incrementCount();
record.getErrors().forEach(error -> problems.add(error.getMessage()));
}
}
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
created.addSelfToListIfAnyCount(lines);
failed.setBulletsOfText(new ArrayList<>(problems));
failed.addSelfToListIfAnyCount(lines);
return (lines);
}
}
}
@@ -53,6 +53,7 @@ public final class LifecycleProcesses
{
List<QProcessMetaData> processes = new ArrayList<>();
processes.add(BackupScheduledRunProcess.processMetaData());
processes.add(com.kingsrook.qbits.backups.schedule.BackupScheduledVerifyProcess.processMetaData());
processes.addAll(CancelRunProcess.processes());
processes.add(ResumeRunProcess.process());
processes.add(TestDestinationProcess.process());
@@ -105,7 +105,8 @@ public class RestoreEngine
Map<String, String> targets = new LinkedHashMap<>();
QInstance instance = QContext.getQInstance();
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE))
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE)
|| (request.getWriter().usesCoreRestoreFlag() && (QContext.getQSession() == null || !QContext.getQSession().hasPermission(BackupsPermissions.RESTORE))))
{
errors.add("Restoring needs the " + BackupsPermissions.RESTORE + " permission.");
}
@@ -341,14 +341,14 @@ public class RestorePass
String backendType = instance.getBackend(c.target.getBackendName()) == null ? null : instance.getBackend(c.target.getBackendName()).getBackendType();
RestoreOutcome.TableOutcome o = c.outcome;
if(c.entry.fingerprint() != null && c.entry.fingerprint().equals(TableFingerprints.of(c.target, backendType)))
if(TableFingerprints.matches(c.entry, c.target, backendType))
{
o.setSchemaStatus("MATCH");
return;
}
Map<String, Object> backupFields = c.entry.schema() == null ? Map.of() : (Map<String, Object>) c.entry.schema().getOrDefault("fields", Map.of());
Map<String, Object> targetFields = (Map<String, Object>) TableFingerprints.descriptor(c.target, backendType).get("fields");
Map<String, Object> backupFields = TableFingerprints.fields(c.entry.schema());
Map<String, Object> targetFields = TableFingerprints.fields(TableFingerprints.descriptor(c.target, backendType));
boolean incompatible = false;
boolean different = false;
@@ -28,9 +28,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** How a restore writes (spec 5.3 and 5.4). The default implementation uses the
** plain Insert, Update and Delete actions, and selects the core RestoreWriteFlag
** (QRun-IO/qqq#1111) and the core key-sequence resync (#1112) reflectively when the
** running QQQ has them. A backend specific writer can be supplied on the request.
** plain Insert, Update and Delete actions with core RestoreWriteFlag and typed
** key-sequence resync. A backend specific writer can be supplied on the request.
**
** Whatever the implementation, record security, field security, unique keys and
** required-field validation still apply to every write.
@@ -18,7 +18,6 @@ package com.kingsrook.qbits.backups.restore;
import java.io.Serializable;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashMap;
@@ -30,8 +29,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.ActionFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
@@ -41,48 +40,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
import com.kingsrook.qqq.backend.core.modules.backend.SequenceResyncResult;
/*******************************************************************************
** The default RestoreWriter: the standard Insert, Update and Delete actions.
**
** Stand-ins (replace with the core API when QQQ 4.1.0-RC.5 ships):
** <ul>
** <li>RestoreWriteFlag (QRun-IO/qqq#1111) is created reflectively when the class is
** present. Without it, the DML audit is omitted (a summary audit is written per
** table instead), but created and modified dates may be set by the normal defaults.</li>
** <li>resyncKeySequences (QRun-IO/qqq#1112) is called reflectively on the backend
** module when it exists; otherwise it is a no-op that the Review reports.</li>
** </ul>
** The default RestoreWriter using core restore flags and key sequence resync.
*******************************************************************************/
public class StandardRestoreWriter implements RestoreWriter
{
private static final QLogger LOG = QLogger.getLogger(StandardRestoreWriter.class);
private static final String FLAG_CLASS = "com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag";
private static final Class<?> FLAG = lookup(FLAG_CLASS);
/*******************************************************************************
**
*******************************************************************************/
private static Class<?> lookup(String name)
{
try
{
return (Class.forName(name));
}
catch(ClassNotFoundException | LinkageError e)
{
return (null);
}
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@@ -148,27 +113,15 @@ public class StandardRestoreWriter implements RestoreWriter
/*******************************************************************************
** Create the core flag (when the running QQQ has it) and hand it to the input.
** Apply the core restore behavior requested by the caller.
*******************************************************************************/
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink) throws QException
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink)
{
if(FLAG == null)
{
return;
}
try
{
Object flag = FLAG.getDeclaredConstructor().newInstance();
FLAG.getMethod("withSuppressAutomations", boolean.class).invoke(flag, options.getSuppressAutomations());
FLAG.getMethod("withRunCustomizers", boolean.class).invoke(flag, options.getRunCustomizers());
FLAG.getMethod("withPerRecordAudit", boolean.class).invoke(flag, options.getPerRecordAudit());
FLAG.getMethod("withPublishEvents", boolean.class).invoke(flag, options.getPublishEvents());
sink.accept((ActionFlag) flag);
}
catch(ReflectiveOperationException | ClassCastException e)
{
throw (new QException("Could not use the core restore flag: " + e.getMessage(), e));
}
sink.accept(new RestoreWriteFlag()
.withSuppressAutomations(options.getSuppressAutomations())
.withRunCustomizers(options.getRunCustomizers())
.withPerRecordAudit(options.getPerRecordAudit())
.withPublishEvents(options.getPublishEvents()));
}
@@ -179,7 +132,7 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public boolean usesCoreRestoreFlag()
{
return (FLAG != null);
return (true);
}
@@ -190,24 +143,7 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public boolean canResyncSequences()
{
return (resyncMethod() != null);
}
/*******************************************************************************
**
*******************************************************************************/
private static Method resyncMethod()
{
try
{
return (QBackendModuleInterface.class.getMethod("resyncKeySequences", QBackendMetaData.class, Collection.class));
}
catch(NoSuchMethodException | LinkageError e)
{
return (null);
}
return (true);
}
@@ -218,12 +154,6 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public List<String> resyncSequences(Collection<QTableMetaData> tables) throws QException
{
Method method = resyncMethod();
if(method == null)
{
return (List.of());
}
Map<String, List<QTableMetaData>> byBackend = new LinkedHashMap<>();
for(QTableMetaData table : tables)
{
@@ -234,23 +164,14 @@ public class StandardRestoreWriter implements RestoreWriter
for(Map.Entry<String, List<QTableMetaData>> entry : byBackend.entrySet())
{
QBackendMetaData backend = QContext.getQInstance().getBackend(entry.getKey());
try
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
for(SequenceResyncResult result : module.resyncKeySequences(backend, entry.getValue()))
{
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
for(Object result : (List<?>) method.invoke(module, backend, entry.getValue()))
if(result.getStatus() == SequenceResyncResult.Status.RESYNCED)
{
Object status = result.getClass().getMethod("getStatus").invoke(result);
if("RESYNCED".equals(String.valueOf(status)))
{
resynced.add(String.valueOf(result.getClass().getMethod("getTableName").invoke(result)));
}
resynced.add(result.getTableName());
}
}
catch(ReflectiveOperationException e)
{
LOG.warn("Could not resync key sequences", e, logPair("backend", entry.getKey()));
throw (new QException("Could not resync key sequences on backend " + entry.getKey() + ": " + e.getMessage(), e));
}
}
return (resynced);
}
@@ -35,8 +35,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.events.BackupEvents;
import com.kingsrook.qbits.backups.codec.TableArtifactWriter;
import com.kingsrook.qbits.backups.events.BackupEvents;
import com.kingsrook.qbits.backups.files.FileCopier;
import com.kingsrook.qbits.backups.files.FileReference;
import com.kingsrook.qbits.backups.files.FileReferenceProvider;
@@ -398,7 +398,7 @@ public class BackupEngine
keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
fingerprint, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
done++;
catalogProgress(result, done, rows, bytesRaw, bytesStored);
if(request.getCancelRequested().getAsBoolean())
@@ -18,30 +18,20 @@ package com.kingsrook.qbits.backups.run;
import java.io.Serializable;
import java.util.List;
import java.util.function.Consumer;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.KeysetPager;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** Reads the rows of one table in primary key order, in keyset pages (WHERE pk >
** lastKey ORDER BY pk LIMIT n), so the cost stays flat with depth and a read can
** resume from a key. A read raw: hidden fields, passwords and heavy fields are
** returned as stored, and possible values are not translated.
**
** This is an internal stand-in for the QQQ core KeysetPager and QueryInput
** withRawRead (core PR feature/keyset-pager-raw-read); the run engine only
** depends on this interface, so it is swapped when that ships in the next RC.
** Reads rows in primary key order through core KeysetPager. Each bounded page
** uses the supplied transaction and returns hidden, password and heavy fields
** as stored, without possible-value translation or display generation.
*******************************************************************************/
public interface TableReader
{
@@ -81,48 +71,25 @@ public interface TableReader
@Override
public void read(QTableMetaData table, String filterJson, Serializable startAfterKey, int pageSize, QBackendTransaction transaction, Consumer<QRecord> sink) throws QException
{
String pk = table.getPrimaryKeyField();
Serializable lastKey = startAfterKey;
QQueryFilter user = parseFilter(filterJson);
while(true)
KeysetPager pager = KeysetPager.forTable(table.getName())
.withFilter(parseFilter(filterJson))
.startingAfter(startAfterKey)
.withPageSize(pageSize)
.withTransaction(transaction)
.withQueryInputCustomizer(input ->
{
input.setShouldOmitHiddenFields(false);
input.setShouldMaskPasswords(false);
input.setShouldFetchHeavyFields(true);
input.setShouldTranslatePossibleValues(false);
input.setShouldGenerateDisplayValues(false);
input.withPermissionBypass("backups run");
});
if(!pager.isPrimaryKeyOrderable())
{
QQueryFilter filter = new QQueryFilter();
if(lastKey != null)
{
filter.addCriteria(new QFilterCriteria(pk, QCriteriaOperator.GREATER_THAN, List.of(lastKey)));
}
if(user != null)
{
filter.addSubFilter(user);
}
filter.withOrderBy(new QFilterOrderBy(pk, true));
filter.setLimit(pageSize);
QueryInput queryInput = new QueryInput(table.getName());
queryInput.setFilter(filter);
queryInput.setShouldOmitHiddenFields(false);
queryInput.setShouldMaskPasswords(false);
queryInput.setShouldFetchHeavyFields(true);
queryInput.setShouldTranslatePossibleValues(false);
queryInput.setShouldGenerateDisplayValues(false);
if(transaction != null)
{
queryInput.setTransaction(transaction);
}
queryInput.withPermissionBypass("backups run");
List<QRecord> page = new QueryAction().execute(queryInput).getRecords();
for(QRecord record : page)
{
sink.accept(record);
lastKey = record.getValue(pk);
}
if(page.size() < pageSize)
{
return;
}
throw (new QException("Backup requires an orderable primary key on table " + table.getName()));
}
pager.forEachPage(page -> page.forEach(sink));
}
}
}
@@ -50,7 +50,7 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
** run is a full-instance, system-session backup (securityScope FULL, requested by
** system:scheduler), which is why the plan owner is validated first: a plan whose
** owner has gone, or lost backups.admin, is not run silently. It raises
** backups.plan.owner_invalid, marks the plan OWNER_INVALID, and stops.
** backups.plan.owner-invalid, marks the plan OWNER_INVALID, and stops.
**
** Everything goes through the run coordinator, so a trigger that finds the plan
** already running is skipped visibly, and a full instance queues the run.
@@ -141,11 +141,11 @@ public class BackupScheduledRunProcess implements BackendStep
if(problem != null)
{
catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaStatus", "OWNER_INVALID"));
BackupsRuntime.getAlertSink().raise("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
BackupsRuntime.getAlertSink().raise("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
"A scheduled backup did not run: " + problem, Map.of("plan", plan.getValueString("planKey")));
throw (new QUserFacingException("The scheduled backup did not run: " + problem + "."));
}
BackupsRuntime.getAlertSink().resolve("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"));
BackupsRuntime.getAlertSink().resolve("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"));
BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
output.addValue("status", result.getStatus());
@@ -0,0 +1,136 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.schedule;
import java.util.List;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
import com.kingsrook.qbits.backups.processes.BackupsProcesses;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.verify.BackupVerifier;
import com.kingsrook.qbits.backups.verify.VerificationOutcome;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QRecordListMetaData;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** The scheduled verify entry point: the process the verify scheduled job of a
** plan (backups_plan.verifyCronExpression) runs. Until this class existed the
** verify job was created but pointed at a process nobody had registered, so a
** verify cron never verified anything.
**
** It verifies the plan's latest successful run, through the level the plan names
** (verifyLevel, CONTENT when unset), stopping at the first level that does not
** pass. Findings, alerts and the plan's last-verified fields are written by the
** verifier, exactly as for Verify backup on a run. A plan that has no
** successful run yet has nothing to verify, and says so.
*******************************************************************************/
public class BackupScheduledVerifyProcess implements BackendStep
{
private static final QLogger LOG = QLogger.getLogger(BackupScheduledVerifyProcess.class);
public static final String NAME = BackupPlanVerifySyncToScheduledJobProcess.VERIFY_PROCESS_NAME;
public static final String SCHEDULER_USER = "system:scheduler";
/*******************************************************************************
** The process metadata: not in the nav, a record process on backups_plan.
*******************************************************************************/
public static QProcessMetaData processMetaData()
{
QProcessMetaData process = new QProcessMetaData()
.withName(NAME)
.withLabel("Run scheduled verification")
.withTableName(BackupsTables.PLAN)
.withIcon(new QIcon().withName("verified"))
.withStep(new QBackendStepMetaData()
.withName("execute")
.withInputData(new QFunctionInputMetaData().withRecordListMetaData(new QRecordListMetaData().withTableName(BackupsTables.PLAN)))
.withCode(new QCodeReference(BackupScheduledVerifyProcess.class)));
process.setPermissionRules(BackupsTables.permissionRules());
return (BackupsProcesses.permission(process, BackupsPermissions.RUN));
}
/*******************************************************************************
** The level a plan verifies to: its verifyLevel, CONTENT when unset or unknown.
*******************************************************************************/
public static String levelOf(QRecord plan)
{
String level = plan.getValueString("verifyLevel");
return (level != null && BackupVerifier.LEVELS.contains(level) ? level : BackupVerifier.CONTENT);
}
/*******************************************************************************
** Verify the latest successful run of the plan.
*******************************************************************************/
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
List<QRecord> records = input.getRecords();
if(records == null || records.isEmpty())
{
throw (new QUserFacingException("No backup plan was selected or found."));
}
QRecord plan = new BackupCatalog().get(BackupsTables.PLAN, records.get(0).getValueInteger("id"))
.orElseThrow(() -> new QUserFacingException("The backup plan no longer exists."));
if(!Boolean.TRUE.equals(plan.getValueBoolean("isActive")))
{
LOG.info("A scheduled verification fired for a plan that is not active; nothing was verified", logPair("plan", plan.getValueString("planKey")));
output.addValue("status", "INACTIVE");
return;
}
Integer runId = plan.getValueInteger("lastSuccessRunId");
if(runId == null)
{
LOG.info("A scheduled verification fired for a plan with no successful backup yet; nothing was verified", logPair("plan", plan.getValueString("planKey")));
output.addValue("status", "NO_BACKUP");
return;
}
String level = levelOf(plan);
List<VerificationOutcome> outcomes = new BackupVerifier().verifyThrough(runId, level, "SCHEDULED", SCHEDULER_USER);
VerificationOutcome last = outcomes.get(outcomes.size() - 1);
output.addValue("status", last.status());
output.addValue("runId", runId);
output.addValue("level", level);
if(!last.passed())
{
LOG.info("A scheduled verification did not pass", logPair("plan", plan.getValueString("planKey")), logPair("runId", runId), logPair("status", last.status()));
}
}
}
@@ -33,8 +33,8 @@ import java.util.Optional;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
import com.kingsrook.qbits.backups.codec.ArtifactTransform;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.codec.JsonlRowCodec;
@@ -274,7 +274,7 @@ public class BackupVerifier
List<String> excluded = entry.excludedFields() == null ? List.of() : entry.excludedFields().stream().map(BackupManifest.ExcludedField::name).toList();
codec = new JsonlRowCodec(table, excluded);
String backendType = qInstance.getBackend(table.getBackendName()) == null ? null : qInstance.getBackend(table.getBackendName()).getBackendType();
roundTrip = TableFingerprints.of(table, backendType).equals(entry.fingerprint());
roundTrip = TableFingerprints.matches(entry, table, backendType);
if(!roundTrip)
{
findings.warn("Table " + entry.name() + " has changed since this backup; rows were parsed but not checked for an exact round trip");
+58
View File
@@ -0,0 +1,58 @@
bundle: qbit-backups
version: 1
layer: QBIT
# Help for the Backups app and every backups_* table, in the QQQ help-bundle format
# (qqq#1198). Each table entry says what the table is for and how its rows get
# created, so an empty table explains itself. BackupsHelp applies it until the core
# HelpBundles registry ships in a QQQ release (STAND-IN(core=#1198)).
# Summaries stay under 120 characters (BackupsTablesGuardrailTest).
entries:
- key: app:backups;slot:help
summary: "Back up, verify and restore the application's data to S3-compatible storage."
content: |
The Setup card on the home lists what a working backup needs (a plan, a storage bucket, a schedule, verification) with the next step for each. A new installation has no plan: run Create default plan on Backup plans, which asks only for the storage bucket alias, or add a plan with New. Everything else (runs, artifacts, verifications, restores) is written by the backups themselves.
- key: table:backups_plan;slot:help
summary: "A backup plan: what is backed up, where it goes, when it runs, how long it is kept and how it is verified."
content: |
**How rows get created:** an administrator adds one with New, or runs Create default plan, which adds "Nightly full" (the whole instance, data and audit tables, JSONL and gzip, every night at 02:00 UTC, verified on Sundays) and asks only for the storage bucket alias. The alias names a bucket your application provides to Backups; no key or endpoint is ever stored here. A plan with a cron expression runs on its own once it is active; Run backup now runs it by hand. A few settings (PARQUET, ZSTD, incremental, native dump, read replica) are not available yet and are refused when the plan is saved.
- key: table:backups_plan_scope;slot:help
summary: "One include or exclude rule of a plan: an app, a qbit, a table or a table role."
content: |
**How rows get created:** an administrator adds them as Scope rules on a plan record. A plan whose scope is Instance backs up everything allowed by its roles and needs no rules; a plan limited to apps, qbits or tables is made of these rules, and an Exclude rule always wins over an Include.
- key: table:backups_run;slot:help
summary: "One execution of a plan, or an ad-hoc export: its status, size, rows, files and storage location."
content: |
**How rows get created:** only the backups write it. Run backup now, a plan's schedule, a pre-restore safety backup, Resume and Import backups from storage each add a row. Nothing is added by hand; Verify backup, Download, Restore and Prune are actions on a run.
- key: table:backups_artifact;slot:help
summary: "One stored object of a run: a table part, its manifest, checksums or a file index, with its hash and size."
content: |
**How rows get created:** only the backups write it, one row per object a run stores. They appear as Artifacts on a run record, and Restore this table and Download work from here.
- key: table:backups_verification;slot:help
summary: "The result of checking a backup: that it is in storage, that it decodes, or that it restores into scratch."
content: |
**How rows get created:** only verification writes it. Run Verify backup on a run (storage, content or test restore), or give a plan a verify cron expression so it verifies its latest backup on a schedule. The newest result also shows on the plan and the home.
- key: table:backups_restore_run;slot:help
summary: "One restore of a backup into the application: mode, id strategy, row and file counts and its safety backup."
content: |
**How rows get created:** only Restore writes it. Run Restore on a backup run, or Restore this table on an artifact: Review previews what would change before anything is written. A restore that replaces data takes a safety backup first and links it here.
- key: table:backups_restore_run_table;slot:help
summary: "Per-table results of a restore: rows inserted, updated and skipped and how the schema matched."
content: |
**How rows get created:** only Restore writes it, one row per table restored. They appear as Tables restored on a restore record.
- key: table:backups_file_object;slot:help
summary: "The content-addressed store of copied files, shared by runs, with reference counts and verification."
content: |
**How rows get created:** only backups write it. When a plan copies the files its rows reference, each distinct file (by SHA-256) is stored once and every run that uses it adds a reference; Prune lowers the counts and removes a file when no run needs it.
- key: table:backups_id_map;slot:help
summary: "Old id to new id pairs recorded while restoring with new ids, so related rows can be rewritten."
content: |
**How rows get created:** only Restore writes it, and only when a restore uses the Remap new ids strategy. The rows are working data for that restore and are cleared when it finishes, so this table is usually empty.
@@ -106,7 +106,7 @@ public final class BackupsTestUtils
new BackupsQBitProducer().withQBitConfig(config).withNamespace("backups").produce(qInstance);
new QInstanceValidator().validate(qInstance);
QContext.init(qInstance, new QSession());
QContext.init(qInstance, new QSession().withPermissions("backups.restore"));
return (qInstance);
}
@@ -18,16 +18,24 @@ package com.kingsrook.qbits.backups.alerts;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.VerifyTestSupport;
import com.kingsrook.qbits.backups.alerts.fakeapi.AlertsInstanceMetaData;
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlertTypeMetaData;
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlerts;
import com.kingsrook.qbits.backups.sla.SlaChecker;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -35,14 +43,10 @@ import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The Alerts qbit adapter, bound reflectively. The SPI is not in QQQ 4.1.0-RC.4,
** so these tests run it against a test double with the same shape as the real
** com.kingsrook.qqq.backend.core.alerts package (QRun-IO/qqq PR #1108).
** The backup adapter against the released core alerts API and a registered sink.
*******************************************************************************/
class AlertsQBitAlertSinkTest
{
private static final String FAKE_API = "com.kingsrook.qbits.backups.alerts.fakeapi";
/*******************************************************************************
@@ -52,9 +56,11 @@ class AlertsQBitAlertSinkTest
void setUp() throws Exception
{
VerifyTestSupport.setUp(3);
QAlerts.reset();
QAlertTypeMetaData.ADDED.clear();
AlertsInstanceMetaData.installed = true;
RecordingSink.raised.clear();
RecordingSink.resolved.clear();
RecordingSink.heartbeats.clear();
RecordingSink.failWith = null;
AlertsInstanceMetaData.ofOrWithNew(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
}
@@ -65,8 +71,8 @@ class AlertsQBitAlertSinkTest
@AfterEach
void tearDown()
{
AlertsInstanceMetaData.installed = false;
QAlerts.reset();
BackupsRuntime.reset();
QContext.clear();
}
@@ -78,7 +84,7 @@ class AlertsQBitAlertSinkTest
@Test
void testRaiseResolveHeartbeat()
{
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
Map<String, String> detail = new LinkedHashMap<>();
detail.put("plan", "nightly");
detail.put("slaHours", "26");
@@ -87,60 +93,90 @@ class AlertsQBitAlertSinkTest
sink.resolve("backups.plan.missed", "backups.plan.missed:nightly");
sink.heartbeat("backups.plan:nightly", 93600, 0);
assertThat(QAlerts.RAISED).hasSize(1);
var signal = QAlerts.RAISED.get(0);
assertThat(signal.alertTypeName).isEqualTo("backups.plan.missed");
assertThat(signal.dedupKey).isEqualTo("backups.plan.missed:nightly");
assertThat(signal.severity.name()).isEqualTo("CRITICAL");
assertThat(signal.title).isEqualTo("A backup did not run in time");
assertThat(signal.summary).isEqualTo("plan=nightly, slaHours=26");
assertThat(signal.attributes).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
assertThat(QAlerts.RESOLVED).containsExactly("backups.plan.missed|backups.plan.missed:nightly|Cleared by the Backups qbit");
assertThat(QAlerts.HEARTBEATS).containsExactly("backups.plan:nightly");
assertThat(RecordingSink.raised).hasSize(1);
var signal = RecordingSink.raised.get(0);
assertThat(signal.getAlertTypeName()).isEqualTo("backups.plan.missed");
assertThat(signal.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
assertThat(signal.getSeverity().name()).isEqualTo("CRITICAL");
assertThat(signal.getTitle()).isEqualTo("A backup did not run in time");
assertThat(signal.getSummary()).isEqualTo("plan=nightly, slaHours=26");
assertThat(signal.getAttributes()).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
assertThat(RecordingSink.resolved).singleElement().satisfies(resolution ->
{
assertThat(resolution.getAlertTypeName()).isEqualTo("backups.plan.missed");
assertThat(resolution.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
assertThat(resolution.getNote()).isEqualTo("Cleared by the Backups qbit");
});
assertThat(RecordingSink.heartbeats).containsExactly("backups.plan:nightly");
sink.raise("backups.storage.debris", "k", BackupAlertSink.Severity.INFO, "no detail", null);
assertThat(QAlerts.RAISED.get(1).summary).isNull();
assertThat(RecordingSink.raised.get(1).getSummary()).isNull();
}
/*******************************************************************************
** Without the Alerts qbit installed (no alerts meta-data on the instance) every
** Without a configured sink, even with alert type metadata present, every
** call is a no-op.
*******************************************************************************/
@Test
void testNoOpWithoutTheAlertsQBit()
{
AlertsInstanceMetaData.installed = false;
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
sink.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
sink.resolve("t", "k");
sink.heartbeat("m", 1, 1);
assertThat(QAlerts.RAISED).isEmpty();
assertThat(QAlerts.RESOLVED).isEmpty();
assertThat(QAlerts.HEARTBEATS).isEmpty();
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(null);
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
long failures = QAlerts.getProducerFailureCount();
sink.raise(null, "k", BackupAlertSink.Severity.INFO, "x", Map.of());
sink.resolve(null, "k");
sink.heartbeat(null, 1, 1);
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures);
assertThat(RecordingSink.raised).isEmpty();
assertThat(RecordingSink.resolved).isEmpty();
assertThat(RecordingSink.heartbeats).isEmpty();
}
/*******************************************************************************
** A broken or missing SPI never throws into the backup.
** The same adapter follows the current instance and a sink installed later.
*******************************************************************************/
@Test
void testLateInstallationAndInstanceSwitch()
{
var original = QContext.getQInstance();
QContext.setQInstance(new QInstance());
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
assertThat(sink.isAlertsQBitInstalled()).isFalse();
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
assertThat(RecordingSink.raised).isEmpty();
sink.registerAlertTypes(QContext.getQInstance());
assertThat(sink.isAlertsQBitInstalled()).isFalse();
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
assertThat(RecordingSink.raised).hasSize(1);
assertThat(RecordingSink.raised.get(0).getSummary()).isNull();
QContext.setQInstance(original);
assertThat(sink.isAlertsQBitInstalled()).isTrue();
}
/*******************************************************************************
** Sink and adapter failures never throw into the backup.
*******************************************************************************/
@Test
void testFailuresAreSwallowed()
{
AlertsQBitAlertSink missing = new AlertsQBitAlertSink("no.such.package");
missing.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
missing.resolve("t", "k");
missing.heartbeat("m", 1, 1);
assertThat(missing.registerAlertTypes(QContext.getQInstance())).isZero();
QAlerts.failWith = new IllegalStateException("alerts down");
AlertsQBitAlertSink broken = new AlertsQBitAlertSink(FAKE_API);
long failures = QAlerts.getProducerFailureCount();
RecordingSink.failWith = new IllegalStateException("alerts down");
AlertsQBitAlertSink broken = new AlertsQBitAlertSink();
broken.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
broken.resolve("t", "k");
broken.heartbeat("m", 1, 1);
assertThat(QAlerts.RAISED).isEmpty();
assertThat(RecordingSink.raised).isEmpty();
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures + 3);
broken.raise("t", "k", null, "bad severity", Map.of());
broken.raise("t", "k", null, "bad severity again", Map.of());
assertThat(broken.registerAlertTypes(null)).isZero();
}
@@ -149,41 +185,41 @@ class AlertsQBitAlertSinkTest
** The backup alert types are declared with the Alerts qbit.
*******************************************************************************/
@Test
void testRegisterAlertTypes()
void testRegisterAlertTypes() throws Exception
{
int added = new AlertsQBitAlertSink(FAKE_API).registerAlertTypes(QContext.getQInstance());
QInstance instance = new QInstance();
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
int added = sink.registerAlertTypes(instance);
var types = new ArrayList<>(AlertsInstanceMetaData.of(instance).getAlertTypes().values());
assertThat(added).isEqualTo(BackupAlerts.alertTypes().size()).isEqualTo(9);
assertThat(QAlertTypeMetaData.ADDED).extracting(t -> t.name).containsExactly(
"backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing", "backups.verify.failed",
"backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
assertThat(QAlertTypeMetaData.ADDED).allSatisfy(t ->
assertThat(types).extracting(t -> t.getName()).containsExactly(
"backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing", "backups.verify.failed",
"backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
assertThat(types).allSatisfy(t ->
{
assertThat(t.groupName).isEqualTo("Backups");
assertThat(t.howToFix).isNotBlank();
assertThat(t.label).isNotBlank();
assertThat(t.getGroupName()).isEqualTo("Backups");
assertThat(t.getHowToFix()).isNotBlank();
assertThat(t.getLabel()).isNotBlank();
});
assertThat(QAlertTypeMetaData.ADDED.get(2).defaultSeverity.name()).isEqualTo("CRITICAL");
assertThat(types.get(2).getDefaultSeverity().name()).isEqualTo("CRITICAL");
assertThat(sink.registerAlertTypes(instance)).isZero();
new QInstanceValidator().validate(QContext.getQInstance());
}
/*******************************************************************************
** Detection: the SPI is present or not by class name; the real SPI is not in
** RC.4, so nothing is bound by default there.
** Default binding preserves a host-provided backup sink.
*******************************************************************************/
@Test
void testDetectionAndBinding()
{
assertThat(AlertsQBitAlertSink.isPresent(FAKE_API)).isTrue();
assertThat(AlertsQBitAlertSink.isPresent("no.such.package")).isFalse();
boolean realSpiPresent = AlertsQBitAlertSink.isPresent(AlertsQBitAlertSink.DEFAULT_API_PACKAGE);
assertThat(AlertsQBitAlertSink.bindIfPresent()).isEqualTo(realSpiPresent);
if(realSpiPresent)
{
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
assertThat(AlertsQBitAlertSink.bindIfPresent()).as("an already bound sink is not replaced").isFalse();
}
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
assertThat(BackupsRuntime.getAlertSink()).isSameAs(VerifyTestSupport.sink);
BackupsRuntime.setAlertSink(null);
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isTrue();
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
}
@@ -194,22 +230,80 @@ class AlertsQBitAlertSinkTest
@Test
void testSlaAlertsThroughTheAdapter() throws Exception
{
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink(FAKE_API));
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink());
QRecord plan = VerifyTestSupport.plan(Map.of());
VerifyTestSupport.backup(plan);
assertThat(QAlerts.HEARTBEATS).contains("backups.plan:nightly");
assertThat(RecordingSink.heartbeats).contains("backups.plan:nightly");
VerifyTestSupport.clock.advance(Duration.ofHours(30));
new SlaChecker().check(VerifyTestSupport.reload(plan));
assertThat(QAlerts.RAISED).anySatisfy(s ->
assertThat(RecordingSink.raised).anySatisfy(s ->
{
assertThat(s.alertTypeName).isEqualTo("backups.plan.missed");
assertThat(s.dedupKey).isEqualTo("backups.plan.missed:nightly");
assertThat(s.severity.name()).isEqualTo("ERROR");
assertThat(s.getAlertTypeName()).isEqualTo("backups.plan.missed");
assertThat(s.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
assertThat(s.getSeverity().name()).isEqualTo("ERROR");
});
VerifyTestSupport.backup(plan);
new SlaChecker().check(VerifyTestSupport.reload(plan));
assertThat(QAlerts.RESOLVED).anyMatch(r -> r.startsWith("backups.plan.missed|backups.plan.missed:nightly|"));
assertThat(RecordingSink.resolved).anyMatch(r -> r.getAlertTypeName().equals("backups.plan.missed") && r.getDedupKey().equals("backups.plan.missed:nightly"));
}
/*******************************************************************************
** Sink loaded by the real QCodeLoader through AlertsInstanceMetaData.
*******************************************************************************/
public static class RecordingSink implements AlertSinkInterface
{
static final List<AlertSignal> raised = new ArrayList<>();
static final List<AlertResolution> resolved = new ArrayList<>();
static final List<String> heartbeats = new ArrayList<>();
static RuntimeException failWith;
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void raise(AlertSignal signal)
{
if(failWith != null)
{
throw (failWith);
}
raised.add(signal);
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void resolve(AlertResolution resolution)
{
if(failWith != null)
{
throw (failWith);
}
resolved.add(resolution);
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public void heartbeat(String heartbeatKey, Instant at)
{
if(failWith != null)
{
throw (failWith);
}
heartbeats.add(heartbeatKey);
}
}
}
@@ -70,8 +70,8 @@ class BackupAlertsTest
void testTheSpecTable()
{
Set<String> names = BackupAlerts.alertTypes().stream().map(BackupAlerts.AlertType::name).collect(Collectors.toSet());
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing",
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing",
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
assertThat(BackupAlerts.alertTypes()).allSatisfy(t -> assertThat(t.howToFix()).isNotBlank());
assertThat(BackupAlerts.alertTypes().stream().filter(t -> t.name().equals(BackupAlerts.RUN_MISSING)).findFirst().orElseThrow().defaultSeverity())
.isEqualTo(BackupAlertSink.Severity.CRITICAL);
@@ -1,30 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.alerts.fakeapi;
/*******************************************************************************
** Test double of the Alerts producer SPI (QRun-IO/qqq PR #1108): the same shape as
** com.kingsrook.qqq.backend.core.alerts.AlertSeverity.
*******************************************************************************/
public enum AlertSeverity
{
INFO,
WARNING,
ERROR,
CRITICAL
}
@@ -1,96 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.alerts.fakeapi;
import java.util.LinkedHashMap;
import java.util.Map;
/*******************************************************************************
** Test double of the Alerts producer SPI AlertSignal (only what the adapter uses).
*******************************************************************************/
public class AlertSignal
{
public final String alertTypeName;
public String dedupKey;
public AlertSeverity severity;
public String title;
public String summary;
public final Map<String, String> attributes = new LinkedHashMap<>();
/*******************************************************************************
** Constructor.
*******************************************************************************/
public AlertSignal(String alertTypeName)
{
this.alertTypeName = alertTypeName;
}
/*******************************************************************************
** Setter.
*******************************************************************************/
public void setDedupKey(String dedupKey)
{
this.dedupKey = dedupKey;
}
/*******************************************************************************
** Setter.
*******************************************************************************/
public void setSeverity(AlertSeverity severity)
{
this.severity = severity;
}
/*******************************************************************************
** Setter.
*******************************************************************************/
public void setTitle(String title)
{
this.title = title;
}
/*******************************************************************************
** Setter.
*******************************************************************************/
public void setSummary(String summary)
{
this.summary = summary;
}
/*******************************************************************************
** Fluent attribute.
*******************************************************************************/
public AlertSignal withAttribute(String key, String value)
{
attributes.put(key, value);
return (this);
}
}
@@ -1,40 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.alerts.fakeapi;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
/*******************************************************************************
** Test double of the Alerts producer SPI AlertsInstanceMetaData: of(instance) is
** non-null only when a test says the Alerts qbit is installed.
*******************************************************************************/
public class AlertsInstanceMetaData
{
public static boolean installed = false;
/*******************************************************************************
** The alerts meta-data of an instance, or null.
*******************************************************************************/
public static AlertsInstanceMetaData of(QInstance qInstance)
{
return (installed ? new AlertsInstanceMetaData() : null);
}
}
@@ -1,103 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.alerts.fakeapi;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
/*******************************************************************************
** Test double of the Alerts producer SPI QAlertTypeMetaData: records the types
** added to an instance.
*******************************************************************************/
public class QAlertTypeMetaData
{
public static final List<QAlertTypeMetaData> ADDED = new ArrayList<>();
public String name;
public String label;
public String groupName;
public String howToFix;
public AlertSeverity defaultSeverity;
/*******************************************************************************
** Fluent.
*******************************************************************************/
public QAlertTypeMetaData withName(String name)
{
this.name = name;
return (this);
}
/*******************************************************************************
** Fluent.
*******************************************************************************/
public QAlertTypeMetaData withLabel(String label)
{
this.label = label;
return (this);
}
/*******************************************************************************
** Fluent.
*******************************************************************************/
public QAlertTypeMetaData withGroupName(String groupName)
{
this.groupName = groupName;
return (this);
}
/*******************************************************************************
** Fluent.
*******************************************************************************/
public QAlertTypeMetaData withHowToFix(String howToFix)
{
this.howToFix = howToFix;
return (this);
}
/*******************************************************************************
** Fluent.
*******************************************************************************/
public QAlertTypeMetaData withDefaultSeverity(AlertSeverity defaultSeverity)
{
this.defaultSeverity = defaultSeverity;
return (this);
}
/*******************************************************************************
** Add to the instance (recorded).
*******************************************************************************/
public void addSelfToInstance(QInstance qInstance)
{
ADDED.add(this);
}
}
@@ -1,104 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.alerts.fakeapi;
import java.util.ArrayList;
import java.util.List;
/*******************************************************************************
** Test double of the Alerts producer SPI QAlerts: static calls that record what
** they were given.
*******************************************************************************/
public final class QAlerts
{
public static final List<AlertSignal> RAISED = new ArrayList<>();
public static final List<String> RESOLVED = new ArrayList<>();
public static final List<String> HEARTBEATS = new ArrayList<>();
/** When set, every call throws it (the SPI is meant not to, but the adapter must survive). */
public static RuntimeException failWith;
/*******************************************************************************
** Utility class.
*******************************************************************************/
private QAlerts()
{
}
/*******************************************************************************
** Forget everything.
*******************************************************************************/
public static void reset()
{
RAISED.clear();
RESOLVED.clear();
HEARTBEATS.clear();
failWith = null;
}
/*******************************************************************************
** Raise.
*******************************************************************************/
public static void raise(AlertSignal signal)
{
maybeFail();
RAISED.add(signal);
}
/*******************************************************************************
** Resolve.
*******************************************************************************/
public static void resolve(String alertTypeName, String dedupKey, String note)
{
maybeFail();
RESOLVED.add(alertTypeName + "|" + dedupKey + "|" + note);
}
/*******************************************************************************
** Heartbeat.
*******************************************************************************/
public static void heartbeat(String heartbeatKey)
{
maybeFail();
HEARTBEATS.add(heartbeatKey);
}
/*******************************************************************************
**
*******************************************************************************/
private static void maybeFail()
{
if(failWith != null)
{
throw (failWith);
}
}
}
@@ -0,0 +1,172 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.app;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.app.BackupsSetupStatus.Item;
import com.kingsrook.qbits.backups.app.BackupsSetupStatus.State;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The Setup card of the Backups home: with no plan it tells the user to run Create
** default plan; with a plan it checks the storage alias, the schedule, the
** verification and the prune bucket, and links each to its table.
*******************************************************************************/
class BackupsSetupStatusTest
{
/*******************************************************************************
**
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
}
/*******************************************************************************
**
*******************************************************************************/
private static Item item(List<Item> items, String label)
{
return (items.stream().filter(i -> i.label().equals(label)).findFirst().orElseThrow());
}
/*******************************************************************************
**
*******************************************************************************/
private static QRecord plan(Map<String, Object> values)
{
QRecord plan = new QRecord().withValue("name", "Nightly").withValue("isActive", true).withValue("scopeType", "INSTANCE").withValue("storageBucketAlias", "backups");
values.forEach((k, v) -> plan.setValue(k, (java.io.Serializable) v));
return (plan);
}
/*******************************************************************************
** No plans: the first step is the Create default plan action.
*******************************************************************************/
@Test
void noPlansPointsAtTheDefaultAction()
{
List<Item> items = BackupsSetupStatus.load(List.of());
assertThat(item(items, "Backup plans").state()).isEqualTo(State.TODO);
assertThat(item(items, "Backup plans").nextStep()).contains("Create default plan");
assertThat(item(items, "Plan scope rules").state()).isEqualTo(State.OPTIONAL);
String html = BackupsHomeView.render(new BackupsHomeData(List.of(), List.of(), List.of(), List.of(), List.of(), Instant.now()), "/app");
assertThat(html).contains("Setup").contains("1 item needs attention").contains("Create default plan").contains("href=\"/app/backups_plan\"");
}
/*******************************************************************************
** A bare plan: storage resolves, but no schedule, no verification and no prune bucket.
*******************************************************************************/
@Test
void aBarePlanNeedsASchedule()
{
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of())), true);
assertThat(item(items, "Backup plans").state()).isEqualTo(State.OK);
assertThat(item(items, "Storage").state()).isEqualTo(State.OK);
assertThat(item(items, "Schedule").state()).isEqualTo(State.TODO);
assertThat(item(items, "Verification").state()).isEqualTo(State.OPTIONAL);
assertThat(item(items, "Prune bucket").state()).isEqualTo(State.DEFAULT);
}
/*******************************************************************************
** A complete plan: everything is in place.
*******************************************************************************/
@Test
void aCompletePlanIsSetUp()
{
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"))), true);
assertThat(items).extracting(Item::state).doesNotContain(State.TODO, State.DEFAULT);
assertThat(items).extracting(Item::tableName).contains(BackupsTables.PLAN, BackupsTables.PLAN_SCOPE, BackupsTables.VERIFICATION);
}
/*******************************************************************************
** Without a scheduler the crons never fire: Schedule and Verification are TODO, and a
** plan whose isActive is null is not counted active (as at runtime).
*******************************************************************************/
@Test
void noSchedulerMeansCronsNeverFire()
{
QRecord complete = plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"));
List<Item> items = BackupsSetupStatus.load(List.of(complete), false);
assertThat(item(items, "Schedule").state()).isEqualTo(State.TODO);
assertThat(item(items, "Schedule").detail()).contains("No scheduler is configured");
assertThat(item(items, "Verification").state()).isEqualTo(State.TODO);
assertThat(item(items, "Verification").nextStep()).contains("schedulerName");
QRecord unset = new QRecord().withValue("name", "N").withValue("storageBucketAlias", "backups");
assertThat(item(BackupsSetupStatus.load(List.of(unset), true), "Backup plans").state()).isEqualTo(State.TODO);
assertThat(item(BackupsSetupStatus.load(List.of(complete)), "Schedule").state()).as("the test instance has no scheduler").isEqualTo(State.TODO);
}
/*******************************************************************************
** A host that ticks plans itself is not told it needs a scheduler; a verify cron its
** tick does not run is called out, and one it does run is fine.
*******************************************************************************/
@Test
void hostSchedulerCountsButUnrunVerifyIsCalledOut()
{
QRecord complete = plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"));
List<Item> items = BackupsSetupStatus.load(List.of(complete), false, "the website ticker", false);
assertThat(item(items, "Schedule").state()).isEqualTo(State.OK);
assertThat(item(items, "Schedule").detail()).contains("the website ticker");
assertThat(item(items, "Verification").state()).isEqualTo(State.OPTIONAL);
assertThat(item(items, "Verification").detail()).contains("Not run by the website ticker");
assertThat(item(BackupsSetupStatus.load(List.of(complete), false, "the website ticker", true), "Verification").state()).isEqualTo(State.OK);
}
/*******************************************************************************
** An alias that does not resolve, an inactive plan and a scoped plan are called out.
*******************************************************************************/
@Test
void problemsAreCalledOut()
{
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of("storageBucketAlias", "nowhere", "isActive", false, "scopeType", "TABLES"))), true);
assertThat(item(items, "Backup plans").state()).isEqualTo(State.TODO);
assertThat(item(items, "Storage").state()).isEqualTo(State.TODO);
assertThat(item(items, "Storage").detail()).contains("nowhere");
assertThat(item(items, "Plan scope rules").detail()).contains("limited by scope rules");
}
}
@@ -0,0 +1,65 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.manifest;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qqq.backend.core.context.QContext;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Fingerprint algorithm dispatch is explicit; a marker cannot be ignored just
** because its hash happens to match a different supported algorithm.
*******************************************************************************/
class TableFingerprintsTest
{
/*******************************************************************************
** A frozen old hash matches only the legacy algorithm, and new hashes only
** match the declared core algorithm. Cosmetic changes preserve both.
*******************************************************************************/
@Test
void testAlgorithmDispatch() throws Exception
{
BackupsTestUtils.init();
var table = QContext.getQInstance().getTable("blogPost");
ObjectMapper mapper = new ObjectMapper();
ObjectNode entry;
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
{
entry = (ObjectNode) mapper.readTree(in).get("blogPost");
}
String oldHash = entry.get("fingerprint").asText();
String coreHash = TableFingerprints.of(table, "memory");
assertThat(coreHash).isNotEqualTo(oldHash);
table.setLabel("Cosmetic label");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", TableFingerprints.LEGACY_ALGORITHM);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", TableFingerprints.CORE_ALGORITHM);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
entry.put("fingerprint", coreHash);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", "unknown-v2");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
entry.remove("fingerprint");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
}
}
@@ -0,0 +1,123 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.restore;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.manifest.BackupManifest;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.run.BackupRunResult;
import com.kingsrook.qbits.backups.verify.BackupVerifier;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Exercise pre-RC.6 manifest fingerprints through verification and restore.
** The schema fixture was frozen from the original stand-in, not recomputed
** with the implementation being tested.
*******************************************************************************/
class LegacyManifestRestoreTest
{
/*******************************************************************************
** Load the blog and grant the permission required by the core restore flag.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupsTestUtils.loadBlog(30);
}
/*******************************************************************************
** Build an old-format manifest around real backup parts and re-seal its hash.
*******************************************************************************/
private BackupRunResult legacyBackup() throws Exception
{
BackupRunResult run = RestoreTestSupport.backup(Map.of());
ObjectMapper mapper = new ObjectMapper();
ObjectNode manifest = (ObjectNode) mapper.readTree(BackupsTestUtils.BACKUPS.getBytes(run.getManifestKey()));
JsonNode legacy;
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
{
legacy = mapper.readTree(in);
}
for(JsonNode table : manifest.get("tables"))
{
ObjectNode entry = (ObjectNode) table;
JsonNode old = legacy.get(entry.get("name").asText());
entry.remove("fingerprintAlgorithm");
entry.set("schema", old.get("schema"));
entry.set("fingerprint", old.get("fingerprint"));
}
byte[] bytes = mapper.writeValueAsBytes(manifest);
BackupsTestUtils.BACKUPS.putBytes(run.getManifestKey(), bytes);
new BackupCatalog().update(BackupsTables.RUN, run.getRunId(), Map.of("manifestSha256", HashingOutputStream.sha256Hex(bytes)));
assertThat(BackupManifest.read(bytes).table("blogPost").fingerprintAlgorithm()).isNull();
return (run);
}
/*******************************************************************************
** Old hashes still receive exact codec verification and restore all data,
** including original timestamps, after deletes, edits and additions.
*******************************************************************************/
@Test
void testLegacyManifestVerifiesAndRestores() throws Exception
{
String before = RestoreTestSupport.contentHash("blogPost");
BackupRunResult run = legacyBackup();
var verified = new BackupVerifier().verify(run.getRunId(), BackupVerifier.CONTENT, "MANUAL", "tester");
assertThat(verified.status()).isEqualTo("PASSED");
assertThat(verified.messages()).noneMatch(message -> message.contains("has changed since"));
RestoreTestSupport.mutate();
RestoreOutcome outcome = new RestoreEngine().execute(RestoreTestSupport.request(run, RestoreRequest.MODE_REPLACE, "blogPost").withConfirmationText("blogPost"));
assertThat(outcome.getErrors()).isEmpty();
assertThat(outcome.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MATCH");
assertThat(RestoreTestSupport.contentHash("blogPost")).isEqualTo(before);
}
/*******************************************************************************
** Legacy field maps and core field lists also compare correctly when hashes
** differ: optional additions are compatible and type changes are rejected.
*******************************************************************************/
@Test
void testLegacySchemaChangesAreAnalyzed() throws Exception
{
BackupRunResult run = legacyBackup();
QContext.getQInstance().getTable("author").withField(new QFieldMetaData("nickname", QFieldType.STRING));
QContext.getQInstance().getTable("blogPost").getField("title").setType(QFieldType.INTEGER);
RestoreOutcome review = new RestoreEngine().review(RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "author", "blogPost"));
assertThat(review.getTables().get("author").getSchemaStatus()).isEqualTo("COMPATIBLE");
assertThat(review.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MISMATCH");
}
}
@@ -187,4 +187,22 @@ class RestoreSafetyTest
QContext.setQSession(new QSession().withPermissions("backups.view", "backups.restore"));
assertThat(new RestoreEngine().plan(request).errors()).isEmpty();
}
/*******************************************************************************
** The core restore permission is required even in anonymous demo mode, and
** Review reports it before a restore can create a run or write any data.
*******************************************************************************/
@Test
void testCoreRestorePermissionIsRequiredWhenAnonymous() throws Exception
{
BackupRunResult run = RestoreTestSupport.backup(Map.of());
QContext.setQSession(new QSession());
RestoreRequest request = RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "blogPost");
assertThat(new RestoreEngine().plan(request).errors()).anyMatch(e -> e.contains("backups.restore"));
assertThatThrownBy(() -> new RestoreEngine().execute(request)).hasMessageContaining("backups.restore");
assertThatThrownBy(() -> new StandardRestoreWriter().insert(QContext.getQInstance().getTable("blogPost"),
List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", 99)), request, null)).hasMessageContaining("backups.restore");
}
}
@@ -109,7 +109,9 @@ class BackupEngineTest
BackupManifest.TableEntry posts = manifest.table("blogPost");
assertThat(posts.rowCount()).isEqualTo(7);
assertThat(posts.excludedFields()).extracting(BackupManifest.ExcludedField::name).containsExactly("secretNote");
assertThat(posts.fingerprint()).startsWith("sha256:");
assertThat(posts.fingerprintAlgorithm()).isEqualTo(com.kingsrook.qbits.backups.manifest.TableFingerprints.CORE_ALGORITHM);
assertThat(posts.fingerprint()).isEqualTo("sha256:" + com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getHash());
assertThat(posts.schema()).isEqualTo(com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getDescriptor());
BackupManifest.Part part = posts.parts().get(0);
byte[] stored = BackupsTestUtils.BACKUPS.getBytes(result.getRunPrefix() + part.key());
@@ -0,0 +1,163 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.run;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.consistentread.ConsistentReadInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Integration coverage for the core pager adapter's ordering, bounded fetches,
** raw fields and transaction lifetime.
*******************************************************************************/
class TableReaderTest
{
private QTableMetaData table;
/*******************************************************************************
** Seed more records than fit in a page.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupsTestUtils.loadBlog(7);
table = QContext.getQInstance().getTable("blogPost");
table.getField("body").setIsHeavy(true);
table.getField("secretNote").setIsHidden(true);
table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(CaptureQueries.class));
CaptureQueries.inputs.clear();
}
/*******************************************************************************
** An OR filter and a resume cursor are ANDed, with ascending keys even when
** the caller requests descending order. Every query has the requested bound.
*******************************************************************************/
@Test
void testFilterResumeOrderingAndRawFields() throws Exception
{
String filter = "{\"booleanOperator\":\"OR\",\"criteria\":[{\"fieldName\":\"id\",\"operator\":\"LESS_THAN_OR_EQUALS\",\"values\":[3]},{\"fieldName\":\"id\",\"operator\":\"GREATER_THAN\",\"values\":[5]}],\"orderBys\":[{\"fieldName\":\"id\",\"isAscending\":false}]}";
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, filter, 2, 2, null, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(3, 6, 7);
assertThat(rows.get(0).getValueString("body")).isEqualTo("Body of 3, with a comma");
assertThat(rows.get(0).getValueString("secretNote")).isEqualTo("do-not-leak-3");
assertThat(CaptureQueries.inputs).hasSize(2).allSatisfy(input ->
{
assertThat(input.getFilter().getLimit()).isEqualTo(2);
assertThat(input.getFilter().getSkip()).isNull();
assertThat(input.getShouldOmitHiddenFields()).isFalse();
assertThat(input.getShouldMaskPasswords()).isFalse();
assertThat(input.getShouldFetchHeavyFields()).isTrue();
assertThat(input.getShouldTranslatePossibleValues()).isFalse();
assertThat(input.getShouldGenerateDisplayValues()).isFalse();
});
}
/*******************************************************************************
** Changes made after a snapshot was opened are invisible on every page. The
** reader leaves the transaction open, with its commit callbacks unexecuted.
*******************************************************************************/
@Test
void testConsistentSnapshotIsRetainedAcrossPages() throws Exception
{
List<Boolean> commits = new ArrayList<>();
try(QBackendTransaction transaction = QBackendTransaction.openConsistentReadFor(new ConsistentReadInput().withBackendName(table.getBackendName())))
{
transaction.addAfterCommitCallback(() -> commits.add(true));
new UpdateAction().execute(new UpdateInput("blogPost").withRecords(List.of(new QRecord().withValue("id", 7).withValue("title", "Changed after snapshot"))));
CaptureQueries.inputs.clear();
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, null, null, 2, transaction, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(1, 2, 3, 4, 5, 6, 7);
assertThat(rows.get(6).getValueString("title")).isEqualTo("Post 7");
assertThat(CaptureQueries.inputs).hasSize(4).allSatisfy(input -> assertThat(input.getTransaction()).isSameAs(transaction));
assertThat(commits).isEmpty();
transaction.commit();
assertThat(commits).containsExactly(true);
}
}
/*******************************************************************************
** Exact page multiples, empty scans, invalid bounds and unsupported key types
** must terminate or fail explicitly rather than drop rows or stream unordered.
*******************************************************************************/
@Test
void testBoundariesAndInvalidPaging() throws Exception
{
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, null, 3, 2, null, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(4, 5, 6, 7);
rows.clear();
new TableReader.Keyset().read(table, null, 7, 2, null, rows::add);
assertThat(rows).isEmpty();
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 0, null, rows::add)).hasMessageContaining("Page size");
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{\"limit\":1}", null, 2, null, rows::add)).hasMessageContaining("skip or limit");
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{", null, 2, null, rows::add)).hasMessageContaining("scope filter");
table.getField("id").setType(QFieldType.BOOLEAN);
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 2, null, rows::add)).hasMessageContaining("orderable primary key");
}
/*******************************************************************************
** Observe actual QueryAction inputs without replacing the backend or pager.
*******************************************************************************/
public static class CaptureQueries implements TableCustomizerInterface
{
static final List<QueryInput> inputs = new ArrayList<>();
/*******************************************************************************
** Record each page and preserve the result unchanged.
*******************************************************************************/
@Override
public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records)
{
inputs.add((QueryInput) input);
return (records);
}
}
}
@@ -306,7 +306,7 @@ class BackupPlanScheduleTest
assertThat(catalog.where(BackupsTables.RUN, "backupPlanId", planId)).isEmpty();
assertThat(BackupsTestUtils.BACKUPS.list("")).isEmpty();
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValueString("slaStatus")).isEqualTo("OWNER_INVALID");
assertThat(alerts.raised).containsExactly("backups.plan.owner_invalid|backups.plan.owner_invalid:nightly");
assertThat(alerts.raised).containsExactly("backups.plan.owner-invalid|backups.plan.owner-invalid:nightly");
BackupScheduledRunProcess.setOwnerHasAdminCheck(null);
Integer noOwner = insertPlan(Map.of("name", "Ownerless", "planKey", "ownerless", "isActive", true));
@@ -180,12 +180,12 @@ class SlaCheckerTest
clock.advance(Duration.ofMinutes(55));
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
BackupAlertsAssert.assertOpen("backups.run.missing_in_storage:nightly", BackupAlertSink.Severity.CRITICAL);
BackupAlertsAssert.assertOpen("backups.run.missing-in-storage:nightly", BackupAlertSink.Severity.CRITICAL);
assertThat(check(plan).status()).as("a negative answer is never cached").isEqualTo(SlaChecker.MISSING_IN_STORAGE);
BackupsTestUtils.BACKUPS.putBytes(result.getManifestKey(), bytes);
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isFalse();
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isFalse();
}
@@ -260,11 +260,11 @@ class SlaCheckerTest
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> false);
assertThat(check(plan).status()).isEqualTo(SlaChecker.OWNER_INVALID);
BackupAlertsAssert.assertOpen("backups.plan.owner_invalid:nightly", BackupAlertSink.Severity.WARNING);
BackupAlertsAssert.assertOpen("backups.plan.owner-invalid:nightly", BackupAlertSink.Severity.WARNING);
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> "bob".equals(owner));
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
assertThat(sink.isOpen("backups.plan.owner_invalid:nightly")).isFalse();
assertThat(sink.isOpen("backups.plan.owner-invalid:nightly")).isFalse();
BackupsRuntime.getConfig().withPlanOwnerValidator(owner ->
{
@@ -287,7 +287,7 @@ class SlaCheckerTest
clock.advance(Duration.ofDays(2));
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
assertThat(sink.isOpen("backups.plan.missed:nightly")).as("the alerts are independent of the status shown").isTrue();
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isTrue();
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isTrue();
}
@@ -0,0 +1,192 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.tables;
import java.lang.reflect.Field;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
import java.util.stream.Stream;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.model.BackupsHelp;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Guardrail (owner requirement 2026-10-14: "all tables should be populated and
** the qbit should be fully using them"). For every backups_* table:
** - runtime code (not just its model, metadata or home screen) reads or writes it,
** so no table exists only on paper;
** - it is listed here with why it starts empty and, where a user sets it up, how
** (this qbit ships no DDL and its one setup value, the storage bucket alias, is
** the host's, so nothing is seeded by a changelog);
** - it has bundle help that says what it is for and how its rows get created
** (shown on the table screen and when it is empty).
** A new table fails this test until it is wired and explained.
*******************************************************************************/
class BackupsTablesGuardrailTest
{
private static final Path MAIN = Path.of("src", "main", "java", "com", "kingsrook", "qbits", "backups");
/*******************************************************************************
** Why each table legitimately starts empty, and where its rows come from.
*******************************************************************************/
static final Map<String, String> STARTS_EMPTY = Map.ofEntries(
Map.entry(BackupsTables.PLAN, "host-specific: its storage bucket alias names a bucket the host provides (the Create default plan action adds Nightly full for an alias)"),
Map.entry(BackupsTables.PLAN_SCOPE, "user content: include/exclude rules; an INSTANCE plan needs none"),
Map.entry(BackupsTables.RUN, "runtime: a plan runs, or Import backups from storage"),
Map.entry(BackupsTables.ARTIFACT, "runtime: every object a run stores"),
Map.entry(BackupsTables.VERIFICATION, "runtime: Verify backup, or a plan verify schedule"),
Map.entry(BackupsTables.RESTORE_RUN, "runtime: a restore"),
Map.entry(BackupsTables.RESTORE_RUN_TABLE, "runtime: each table of a restore"),
Map.entry(BackupsTables.FILE_OBJECT, "runtime: each distinct file a plan copies"),
Map.entry(BackupsTables.ID_MAP, "runtime: working data of a restore with new ids, cleared when it ends"));
private QInstance qInstance;
/*******************************************************************************
**
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
qInstance = BackupsTestUtils.init();
}
/*******************************************************************************
** Every table is used by runtime code.
*******************************************************************************/
@Test
void everyTableIsUsedAtRuntime() throws Exception
{
List<String> runtimeSource = new ArrayList<>();
try(Stream<Path> files = Files.walk(MAIN))
{
for(Path file : files.filter(p -> p.toString().endsWith(".java")).filter(p -> !p.startsWith(MAIN.resolve("model")) && !p.startsWith(MAIN.resolve("app"))).toList())
{
runtimeSource.add(Files.readString(file, StandardCharsets.UTF_8));
}
}
for(String table : backupsTables())
{
String constant = constantFor(table);
Pattern use = Pattern.compile("\\bBackupsTables\\." + constant + "\\b");
assertThat(runtimeSource.stream().anyMatch(source -> use.matcher(source).find()))
.as(table + " (BackupsTables." + constant + ") must be read or written by runtime code, not only declared").isTrue();
}
}
/*******************************************************************************
** Every table is explained: where its rows come from.
*******************************************************************************/
@Test
void everyTableIsExplained()
{
assertThat(backupsTables()).containsExactlyInAnyOrderElementsOf(STARTS_EMPTY.keySet());
assertThat(BackupsTables.ALL).containsExactlyInAnyOrderElementsOf(STARTS_EMPTY.keySet());
assertThat(STARTS_EMPTY.values()).allMatch(reason -> !reason.isBlank());
}
/*******************************************************************************
** Every table and the app have bundle help that says how rows get created, and
** the plan help names the action that sets it up.
*******************************************************************************/
@Test
void everyTableHasEmptyStateHelp() throws Exception
{
assertThat(BackupsHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
for(String table : backupsTables())
{
List<QHelpContent> help = qInstance.getTable(table).getHelpContent() == null ? null : qInstance.getTable(table).getHelpContent().get(BackupsHelp.SLOT);
assertThat(help).as(table + " table help").isNotEmpty();
assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created");
}
assertThat(qInstance.getApp("backups").getHelpContent()).containsKey(BackupsHelp.SLOT);
assertThat(BackupsHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || qInstance.getTable(key.substring(6, key.indexOf(';'))) != null,
"help names only real tables");
assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("Create default plan");
assertThat(qInstance.getProcess(DefaultPlanProcess.NAME)).as("the action the plan help names").isNotNull();
assertThat(qInstance.getProcess(DefaultPlanProcess.NAME).getLabel()).isEqualTo("Create default plan");
}
/*******************************************************************************
** A host's own help in the slot is never replaced by the bundle.
*******************************************************************************/
@Test
void hostHelpIsKept() throws Exception
{
QTableMetaData table = qInstance.getTable(BackupsTables.RUN);
table.withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for runs."));
qInstance.getApp("backups").withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for the app."));
BackupsHelp.apply(qInstance, "backups");
assertThat(table.getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for runs."));
assertThat(qInstance.getApp("backups").getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for the app."));
assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("How rows get created");
}
/*******************************************************************************
**
*******************************************************************************/
private List<String> backupsTables()
{
List<String> tables = qInstance.getTables().values().stream().map(QTableMetaData::getName).filter(n -> n.startsWith("backups_")).sorted().toList();
assertThat(tables).hasSize(9);
return (tables);
}
/*******************************************************************************
** The BackupsTables constant whose value is the table name.
*******************************************************************************/
private static String constantFor(String table) throws Exception
{
for(Field field : BackupsTables.class.getFields())
{
if(field.getType() == String.class && table.equals(field.get(null)) && !field.getName().equals("PVS_PREFIX"))
{
return (field.getName());
}
}
throw (new IllegalStateException("no constant for " + table));
}
}
@@ -0,0 +1,205 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.tables;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
import com.kingsrook.qbits.backups.plan.BackupValidatorReset;
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The "Create default plan" table action: it asks for a storage bucket alias,
** creates the Nightly full plan through the normal insert, refuses an alias that
** does not resolve, and is insert-only (a second run changes nothing, an edited
** plan survives).
*******************************************************************************/
class DefaultPlanProcessTest
{
private final BackupCatalog catalog = new BackupCatalog();
/*******************************************************************************
**
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupValidatorReset.reset();
QContext.getQSession().withPermissions(BackupsPermissions.ADMIN);
}
/*******************************************************************************
** Run the action through Review and then Result.
*******************************************************************************/
@SuppressWarnings("unchecked")
private List<ProcessSummaryLine> run(String alias, boolean execute) throws QException
{
RunProcessInput input = new RunProcessInput();
input.setProcessName(DefaultPlanProcess.NAME);
input.addValue(DefaultPlanProcess.FIELD_ALIAS, alias);
//////////////////////////////////////////////////////////////////////////
// the first call stops at the input form; the next runs Review (full //
// validation), the last executes and gives the Result //
//////////////////////////////////////////////////////////////////////////
RunProcessOutput output = new RunProcessAction().execute(input);
input.addValue("processUUID", output.getProcessUUID());
input.setStartAfterStep("input");
output = new RunProcessAction().execute(input);
if(!execute)
{
return ((List<ProcessSummaryLine>) output.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY));
}
input.setStartAfterStep("review");
output = new RunProcessAction().execute(input);
return ((List<ProcessSummaryLine>) output.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY));
}
/*******************************************************************************
** The action is a table action that needs no selected records and backups.admin.
*******************************************************************************/
@Test
void isATableActionForAdmins() throws Exception
{
var process = QContext.getQInstance().getProcess(DefaultPlanProcess.NAME);
assertThat(process.getTableName()).isEqualTo(BackupsTables.PLAN);
assertThat(process.getMinInputRecords()).isEqualTo(0);
assertThat(((com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData) process.getStep("input")).getFormFields().stream().map(f -> f.getName())).containsExactly(DefaultPlanProcess.FIELD_ALIAS);
}
/*******************************************************************************
** Review previews without writing; Result creates the plan, and the plan works.
*******************************************************************************/
@Test
void createsTheNightlyFullPlan() throws Exception
{
List<ProcessSummaryLine> review = run("backups", false);
assertThat(review).anySatisfy(line -> assertThat(line.getPluralFutureMessage()).contains("will be created"));
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).as("Review writes nothing").isEmpty();
List<ProcessSummaryLine> result = run("backups", true);
assertThat(result).singleElement().satisfies(line -> assertThat(line.getStatus().name()).isEqualTo("OK"));
QRecord plan = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0);
assertThat(plan.getValueString("name")).isEqualTo("Nightly full");
assertThat(plan.getValueString("storageBucketAlias")).isEqualTo("backups");
assertThat(plan.getValueBoolean("isActive")).isTrue();
assertThat(plan.getValueString("scopeType")).isEqualTo("INSTANCE");
assertThat(plan.getValueString("cronExpression")).isEqualTo(DefaultPlanProcess.CRON);
assertThat(plan.getValueString("verifyCronExpression")).isEqualTo(DefaultPlanProcess.VERIFY_CRON);
assertThat(plan.getValueInteger("slaHours")).isEqualTo(26);
assertThat(plan.getValueString("format")).isEqualTo("JSONL");
}
/*******************************************************************************
** A plan made by the action is a working plan: it runs and backs up the instance.
*******************************************************************************/
@Test
void theCreatedPlanRuns() throws Exception
{
BackupsTestUtils.loadBlog(3);
run("backups", true);
QRecord plan = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0);
var result = new com.kingsrook.qbits.backups.run.BackupRunCoordinator().runPlan(plan, "MANUAL");
assertThat(result.getStatus()).isIn("SUCCEEDED", "SUCCEEDED_WITH_WARNINGS");
}
/*******************************************************************************
** Insert-only: a second run creates nothing, and an edited plan keeps its edit.
*******************************************************************************/
@Test
void isInsertOnly() throws Exception
{
run("backups", true);
Integer id = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0).getValueInteger("id");
catalog.update(BackupsTables.PLAN, id, Map.of("slaHours", 48, "name", "Our nightly"));
List<ProcessSummaryLine> review = run("backups", false);
assertThat(review).singleElement().satisfies(line -> assertThat(line.getMessage()).contains("already exists"));
run("backups", true);
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).hasSize(1);
QRecord plan = catalog.get(BackupsTables.PLAN, id).orElseThrow();
assertThat(plan.getValueInteger("slaHours")).isEqualTo(48);
assertThat(plan.getValueString("name")).isEqualTo("Our nightly");
}
/*******************************************************************************
** An alias the host does not provide is refused in Review, and nothing is created.
*******************************************************************************/
@Test
void refusesAnUnknownAlias() throws Exception
{
List<ProcessSummaryLine> review = run("no-such-bucket", false);
assertThat(review).anySatisfy(line ->
{
assertThat(line.getStatus().name()).isEqualTo("ERROR");
assertThat(String.join(" ", line.getBulletsOfText())).contains("no-such-bucket");
});
run("no-such-bucket", true);
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).isEmpty();
}
/*******************************************************************************
** A plan that already has the name Nightly full (under another key) is refused in
** Review, never "will be created".
*******************************************************************************/
@Test
void reviewRefusesATakenName() throws Exception
{
catalog.insert(BackupsTables.PLAN, new java.util.LinkedHashMap<>(Map.of("name", DefaultPlanProcess.PLAN_NAME, "planKey", "other-key", "isActive", false, "scopeType", "INSTANCE", "storageBucketAlias", "backups")));
List<ProcessSummaryLine> review = run("backups", false);
assertThat(review).noneSatisfy(line -> assertThat(line.getPluralFutureMessage()).contains("will be created"));
assertThat(review).anySatisfy(line ->
{
assertThat(line.getStatus().name()).isEqualTo("ERROR");
assertThat(String.join(" ", line.getBulletsOfText())).contains("already exists");
});
}
}
@@ -0,0 +1,188 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.tables;
import java.io.Serializable;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.plan.BackupValidatorReset;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.run.BackupRunCoordinator;
import com.kingsrook.qbits.backups.schedule.BackupPlanVerifySyncToScheduledJobProcess;
import com.kingsrook.qbits.backups.schedule.BackupScheduledVerifyProcess;
import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Two half-wired settings of backups_plan, made honest or made to work:
** - the settings that are not available yet (read replica, native dump,
** incremental, PARQUET, ZSTD) are refused at save with a clear message and say so
** in their field help;
** - verifyCronExpression creates a scheduled job that runs the process
** backupVerifyPlan; that process now exists, and verifies the plan's latest
** successful backup through the plan's verifyLevel.
*******************************************************************************/
class UnavailableSettingsAndScheduledVerifyTest
{
private final BackupCatalog catalog = new BackupCatalog();
/*******************************************************************************
**
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupsTestUtils.loadBlog(3);
BackupValidatorReset.reset();
}
/*******************************************************************************
** Save a plan through the table (validation runs), returning the record.
*******************************************************************************/
private QRecord save(Map<String, Serializable> overrides) throws QException
{
Map<String, Serializable> values = new LinkedHashMap<>(Map.of("name", "P" + System.nanoTime(), "planKey", "p" + System.nanoTime(), "isActive", true, "scopeType", "INSTANCE", "storageBucketAlias", "backups"));
values.putAll(overrides);
QRecord record = new QRecord();
record.setValues(values);
InsertInput input = new InsertInput(BackupsTables.PLAN);
input.setRecords(List.of(record));
input.withPermissionBypass("test");
return (new InsertAction().execute(input).getRecords().get(0));
}
/*******************************************************************************
** Every setting that is not available yet is refused, and the field help says so.
*******************************************************************************/
@Test
void unavailableSettingsAreRefusedAndSayWhy() throws Exception
{
assertThat(save(Map.of()).getErrors()).as("the defaults are fine").isEmpty();
Map<String, Map<String, Serializable>> refused = Map.of(
"Reading from a replica is not available yet", Map.of("useReadReplica", true),
"Native database dumps are not available yet", Map.of("nativeDump", "PG_DUMP_CUSTOM"),
"Incremental backups are not available yet", Map.of("incrementalMode", "CHANGED_SINCE_LAST"),
"PARQUET format is not available yet", Map.of("format", "PARQUET"),
"ZSTD compression is not available yet", Map.of("compression", "ZSTD"));
for(Map.Entry<String, Map<String, Serializable>> entry : refused.entrySet())
{
QRecord record = save(entry.getValue());
assertThat(record.getErrors()).as(entry.getValue().toString()).isNotEmpty();
assertThat(record.getErrors().toString()).contains(entry.getKey());
}
var table = QContext.getQInstance().getTable(BackupsTables.PLAN);
for(String field : List.of("useReadReplica", "nativeDump", "incrementalMode", "format", "compression", "scheduledJobId"))
{
assertThat(table.getField(field).getHelpContents()).as(field + " help").isNotEmpty();
assertThat(table.getField(field).getHelpContents().get(0).getContent()).as(field + " help").matches("(?is).*(not available yet|not used).*");
}
}
/*******************************************************************************
** The process a verify cron schedules exists, and verifies the plan's latest
** successful run, as the scheduler, through the plan's verifyLevel.
*******************************************************************************/
@Test
void theVerifyCronRunsARealProcess() throws Exception
{
assertThat(QContext.getQInstance().getProcess(BackupPlanVerifySyncToScheduledJobProcess.VERIFY_PROCESS_NAME)).as("the process the scheduled verify job names").isNotNull();
Integer planId = catalog.insert(BackupsTables.PLAN, new LinkedHashMap<>(Map.of("name", "Nightly", "planKey", "nightly", "isActive", true, "scopeType", "INSTANCE",
"storageBucketAlias", "backups", "verifyLevel", "STORAGE")));
assertThat(run(planId).getValues().get("status")).as("nothing to verify before the first backup").isEqualTo("NO_BACKUP");
QRecord plan = catalog.get(BackupsTables.PLAN, planId).orElseThrow();
var result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
assertThat(result.getStatus()).isIn("SUCCEEDED", "SUCCEEDED_WITH_WARNINGS");
RunProcessOutput output = run(planId);
assertThat(output.getException()).isEmpty();
assertThat(output.getValues().get("status")).isEqualTo("PASSED");
assertThat(output.getValues().get("level")).isEqualTo("STORAGE");
List<QRecord> verifications = catalog.where(BackupsTables.VERIFICATION, "backupRunId", result.getRunId());
assertThat(verifications).singleElement().satisfies(v ->
{
assertThat(v.getValueString("level")).isEqualTo("STORAGE");
assertThat(v.getValueString("trigger")).isEqualTo("SCHEDULED");
assertThat(v.getValueString("requestedByUserId")).isEqualTo(BackupScheduledVerifyProcess.SCHEDULER_USER);
});
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValue("lastVerifiedDate")).as("a storage check alone does not count as verified").isNull();
/////////////////////////////////////////////////////////////////////
// the plan's level decides how far it goes; an inactive plan is skipped //
/////////////////////////////////////////////////////////////////////
catalog.update(BackupsTables.PLAN, planId, Map.of("verifyLevel", "CONTENT"));
assertThat(run(planId).getValues().get("level")).isEqualTo("CONTENT");
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValue("lastVerifiedDate")).isNotNull();
catalog.update(BackupsTables.PLAN, planId, Map.of("isActive", false));
assertThat(run(planId).getValues().get("status")).isEqualTo("INACTIVE");
}
/*******************************************************************************
** An unset or unknown level falls back to CONTENT.
*******************************************************************************/
@Test
void levelFallsBackToContent()
{
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord())).isEqualTo("CONTENT");
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord().withValue("verifyLevel", "BOGUS"))).isEqualTo("CONTENT");
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord().withValue("verifyLevel", "TEST_RESTORE"))).isEqualTo("TEST_RESTORE");
}
/*******************************************************************************
** Run the scheduled verify entry point for a plan, as the scheduler would.
*******************************************************************************/
private RunProcessOutput run(Integer planId) throws QException
{
RunProcessInput input = new RunProcessInput();
input.setProcessName(BackupScheduledVerifyProcess.NAME);
input.setCallback(QProcessCallbackFactory.forPrimaryKey("id", planId));
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
return (new RunProcessAction().execute(input));
}
}
@@ -0,0 +1,130 @@
{
"author" : {
"schema" : {
"name" : "author",
"primaryKey" : "id",
"backendType" : "memory",
"fields" : {
"bio" : {
"type" : "TEXT",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"id" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"name" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
}
},
"uniqueKeys" : [ ],
"associations" : [ ]
},
"fingerprint" : "sha256:c0fce31286898a15a98f9f46515b15c6eee4ffeb1f1eb39d6100833dfe3f760e"
},
"blogPost" : {
"schema" : {
"name" : "blogPost",
"primaryKey" : "id",
"backendType" : "memory",
"fields" : {
"authorId" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : "author",
"heavy" : false,
"sensitive" : false
},
"body" : {
"type" : "TEXT",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"createDate" : {
"type" : "DATE_TIME",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"id" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"mediaKey" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"price" : {
"type" : "DECIMAL",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"published" : {
"type" : "BOOLEAN",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"publishedOn" : {
"type" : "DATE",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"secretNote" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : true
},
"title" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
}
},
"uniqueKeys" : [ ],
"associations" : [ ]
},
"fingerprint" : "sha256:3a4de7479908cd4b4ce8eaa7a7af43e7cbaabbdbc2b9741bebeb24a6bd41b9ea"
}
}