Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3df24cda5a | ||
|
|
987c20899b
|
||
|
|
01db11b123 | ||
|
|
6dea78881d
|
||
|
|
89f218937a
|
||
|
|
85f4ca0a6f
|
||
|
|
48e7320228 | ||
|
|
a5297e709d | ||
|
|
14bb6fa1b4 | ||
|
|
b1ff70d7d3
|
||
|
|
f2af36ec5f | ||
|
|
054f6d90d7
|
||
|
|
5b7106b981
|
||
|
|
cda334f2fe | ||
|
|
590b132fb6
|
||
|
|
32034e5762 | ||
|
|
bc97d59b96
|
||
|
|
506741bd90 | ||
|
|
95b971a5fb
|
||
|
|
f0a8c5000e | ||
|
|
8f3f9ec60a
|
||
|
|
e56cc1b021 | ||
|
|
cdaf218201 |
No files matched your search
+2
-13
@@ -1,7 +1,7 @@
|
||||
version: 2.1
|
||||
|
||||
orbs:
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.8
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.9
|
||||
|
||||
commands:
|
||||
configure-package-install:
|
||||
@@ -22,7 +22,7 @@ workflows:
|
||||
context: [qqq-maven-registry-credentials]
|
||||
filters:
|
||||
branches:
|
||||
ignore: /(develop|release\/.*|hotfix\/.*)/
|
||||
ignore: /(develop|release\/.*)/
|
||||
|
||||
publish_snapshot:
|
||||
jobs:
|
||||
@@ -58,14 +58,3 @@ workflows:
|
||||
ignore: /.*/
|
||||
tags:
|
||||
only: [/^v\d+\.\d+\.\d+$/]
|
||||
|
||||
publish_hotfix_release:
|
||||
jobs:
|
||||
- qqq-orb/mvn_publish:
|
||||
pre-steps:
|
||||
- configure-package-install
|
||||
context: [qqq-maven-registry-credentials]
|
||||
branch_type: hotfix
|
||||
filters:
|
||||
branches:
|
||||
only: [/hotfix\/.*/]
|
||||
@@ -0,0 +1,25 @@
|
||||
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
|
||||
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
|
||||
# the release line and the release/* push runs CircleCI publish_release_candidate.
|
||||
name: Release candidate
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
description: Print the plan without merging or creating branches
|
||||
type: boolean
|
||||
default: false
|
||||
pull_request:
|
||||
types: [closed]
|
||||
branches: [develop]
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
release-rc:
|
||||
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
|
||||
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
|
||||
with:
|
||||
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
|
||||
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
|
||||
## Dependencies
|
||||
|
||||
- Java 21.
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
|
||||
- Tests: JUnit 5, AssertJ.
|
||||
|
||||
## Release
|
||||
|
||||
@@ -5,4 +5,17 @@ All notable changes to this project are documented here. The format follows Keep
|
||||
## [0.1.0] - Unreleased
|
||||
|
||||
### Added
|
||||
- Guided empty states: help bundle `help/qbit-backups.yaml` (app and all nine tables, with "How rows get created") applied by `BackupsHelp`, and a Setup card on the Backups home (plan, scope rules, storage, schedule, verification, prune bucket).
|
||||
- "Create default plan" table action on Backup plans: adds "Nightly full" for a storage bucket alias; insert-only.
|
||||
- `backupVerifyPlan`: the scheduled verification process. A plan verify cron created a scheduled job for a process that did not exist, so scheduled verification never ran.
|
||||
- Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.
|
||||
|
||||
### Changed
|
||||
- Saving a plan with Use read replica on is refused (it was never read); the not-yet-available settings (read replica, native dump, incremental, PARQUET, ZSTD) say so in their field help; scheduledJobId is documented as unused.
|
||||
- Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
|
||||
- Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.
|
||||
- Use RC.5-valid alert names `backups.run.missing-in-storage` and `backups.plan.owner-invalid` (and matching deduplication keys). Hosts using the old underscore names must update routing rules.
|
||||
- Target QQQ 4.1.0-RC.6 and replace the fingerprint, keyset pager and reflective restore bridges with typed core APIs.
|
||||
- New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
|
||||
- Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
|
||||
- Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
|
||||
@@ -10,7 +10,7 @@
|
||||
## Requirements
|
||||
|
||||
- Java 21.
|
||||
- QQQ 4.1 (built against `4.1.0-RC.4`).
|
||||
- QQQ 4.1 (targets `4.1.0-RC.6`, not yet released).
|
||||
|
||||
## Install
|
||||
|
||||
@@ -22,6 +22,10 @@ new BackupsQBitProducer()
|
||||
.produce(qInstance);
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
This qbit ships no DDL and seeds nothing from a changelog: its one setup value, the storage bucket alias, belongs to your application. On a fresh install open Backups, read the Setup card, and run **Create default plan** on Backup plans (it asks only for the alias of a bucket your storage resolver provides). It creates the insert-only "Nightly full" plan: whole instance, data and audit tables, JSONL and gzip, 02:00 UTC, verified Sundays 04:00 UTC. Edit any setting afterwards. Help for every table, including how its rows get created, ships in `help/qbit-backups.yaml`.
|
||||
|
||||
## Naming
|
||||
|
||||
Tables are `backups_<name>` (lowercase snake_case, backend name equal to the table name). Permissions are `backups.<action>`.
|
||||
@@ -36,8 +40,32 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
|
||||
./mvnw verify
|
||||
```
|
||||
|
||||
Before RC.6 is released, verify against the locally installed core build only:
|
||||
|
||||
```bash
|
||||
./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
|
||||
```
|
||||
|
||||
The POM stays on RC.6. See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for
|
||||
fingerprint compatibility, paging guarantees and restore permissions.
|
||||
|
||||
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
|
||||
|
||||
### Alerts integration
|
||||
|
||||
Backups binds directly to core QAlerts in QQQ 4.1.0-RC.5 and declares its nine
|
||||
alert types on each instance. Hosts install an `AlertSinkInterface` through
|
||||
`AlertsInstanceMetaData.withSink(QCodeReference)`. Without a configured sink,
|
||||
backup alert calls remain no-ops (they do not use core's logging fallback).
|
||||
Custom `BackupAlertSink` bindings are preserved. Core handles dispatch failures;
|
||||
adapter failures are caught and logged once, so alerting cannot fail a backup.
|
||||
Raise and resolve follow core's transaction-aware dispatch; heartbeats run immediately.
|
||||
The backup SLA checker still owns expected intervals and grace periods.
|
||||
|
||||
RC.5 requires hyphenated alert names: `backups.run.missing-in-storage` and
|
||||
`backups.plan.owner-invalid` replace the former underscore spellings, including
|
||||
their deduplication keys. Update host routing rules that used those spellings.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Upgrading
|
||||
|
||||
## Unreleased
|
||||
|
||||
- No schema change and no new include line. Hosts get the help text and the Setup card automatically.
|
||||
- Plans with `use_read_replica` true are now refused on the next save (the setting was never used). Turn it off.
|
||||
- Hosts that set a plan `verifyCronExpression` and a scheduler now get working scheduled verification: the `backupVerifyPlan` process exists. It verifies the plan's latest successful run through the plan's `verifyLevel` (CONTENT when unset). Expect verification rows and, on failure, alerts.
|
||||
@@ -0,0 +1,100 @@
|
||||
# RC.6 core migration (issue #19, F3)
|
||||
|
||||
The POM targets `4.1.0-RC.6`. Until that release is available, local verification
|
||||
uses `./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` against the
|
||||
installed build from `qqq-wt-archrc6`. No core source changes are needed.
|
||||
|
||||
## Fingerprint compatibility
|
||||
|
||||
The algorithms are **not byte-compatible**, even for identical table metadata:
|
||||
|
||||
| Descriptor property | Pre-RC.6 backup algorithm | Core algorithm |
|
||||
| --- | --- | --- |
|
||||
| Table and key | `name`, `primaryKey` | `table`, `primaryKeyField` |
|
||||
| Fields | Map keyed by name | Sorted list with explicit `name` |
|
||||
| Field flags | `required`, `pvs`, `heavy`, `sensitive` | `isRequired`, `possibleValueSource`, `isHeavy`, `isSensitive` |
|
||||
| Unique keys | Sorted `+`-joined strings | Sorted lists of field names |
|
||||
| Associations | Sorted `name->table` strings | Objects with name, associatedTable and joinName |
|
||||
| Null backend type | JSON null | Empty string |
|
||||
| Escaping | Historical hand-written escaping | Jackson JSON escaping |
|
||||
|
||||
New table entries contain `fingerprintAlgorithm: "qqq-table-schema-v1"`, the core
|
||||
`schema` descriptor, and `fingerprint: "sha256:" + core.getHash()`. The outer
|
||||
manifest remains version 1; row parts, checksums, compression and storage paths
|
||||
are unchanged. The instance/catalog hash still combines the ordered table
|
||||
fingerprint strings, so its value changes too. New manifests require this RC.6
|
||||
reader; older readers do not understand the new descriptor shape.
|
||||
|
||||
An absent algorithm identifies the old `qbit-backups-table-v1` algorithm. The
|
||||
reader also accepts that explicit identifier. `LegacyTableFingerprint` freezes
|
||||
the original descriptor and hash implementation solely for comparison against
|
||||
existing backups; no new backup uses it. Restore and content verification select
|
||||
the declared algorithm. Unknown identifiers never match a known fingerprint.
|
||||
Restore normalizes either field shape for its existing compatibility analysis
|
||||
when a table has changed. Fingerprint equality is not required for a compatible
|
||||
schema migration, as before.
|
||||
|
||||
`src/test/resources/manifests/legacy-schema.json` freezes descriptors and hashes
|
||||
produced by the original stand-in for the blog fixture. Tests place these into a
|
||||
real backup manifest without the algorithm field, verify its content, mutate the
|
||||
live data, then restore and compare the complete content hash including dates.
|
||||
They also cover optional additions and incompatible type changes in old schemas.
|
||||
|
||||
## Paging and consistency
|
||||
|
||||
`TableReader.Keyset` delegates to core `KeysetPager`. Every query uses
|
||||
`(scope filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize`, with no offset.
|
||||
The resume key is exclusive. Page size is a row bound, **not a JDBC fetch-size
|
||||
setting**. Core's normal bounded QueryAction path retains backend driver defaults;
|
||||
this adapter does not opt into the large-result streaming query hint or change
|
||||
JDBC fetch size. Heavy/hidden/password fields and untranslated stored values keep
|
||||
the same query settings and existing backup permission boundary.
|
||||
|
||||
Every page receives the caller's transaction. The pager does not open, commit,
|
||||
close, or replace that transaction. A test opens a real core memory snapshot,
|
||||
changes a later page's data, and proves every page still sees the original state.
|
||||
The existing BackupEngine continues to report BEST_EFFORT and reject
|
||||
SNAPSHOT_REQUIRED: adopting the consistent-read opening API at engine level is
|
||||
separate work. This migration does not claim snapshot guarantees for ordinary
|
||||
backup runs.
|
||||
|
||||
Core can stream non-orderable keys without deterministic key order; backups
|
||||
explicitly reject those keys instead. Core rejects scope filters carrying skip
|
||||
or limit, and replaces user ordering with ascending primary-key ordering. Tests
|
||||
cover OR filters, resume, full and partial pages, empty scans and invalid inputs.
|
||||
|
||||
## Restore writes
|
||||
|
||||
`StandardRestoreWriter` now constructs `RestoreWriteFlag` directly and calls
|
||||
`QBackendModuleInterface.resyncKeySequences`, consuming typed `SequenceResyncResult`
|
||||
values. Only RESYNCED results are reported as sequence changes; NOT_NEEDED and
|
||||
NOT_APPLICABLE remain normal outcomes. There is no reflection or unavailable-core
|
||||
fallback. The core flag preserves supplied dates and carries the requested
|
||||
customizer, automation, audit and event settings.
|
||||
|
||||
Core always requires an actual session permission `backups.restore` for its
|
||||
restore flag, including hosts with general core enforcement disabled. The qbit's
|
||||
anonymous-mode convenience setting does not grant that permission. Review now
|
||||
reports this requirement before executing a restore; test sessions explicitly
|
||||
hold it. Custom writers that do not use the core flag keep their existing qbit
|
||||
permission checks.
|
||||
|
||||
## HTTP operations and wire changes
|
||||
|
||||
No HTTP operations are added, removed or modified: method, path and operationId
|
||||
are unchanged (no operation list entries for this migration). The RC.6 UI API
|
||||
extension documentation was reviewed; this task introduces no Javalin route or
|
||||
middleware extension. Wire changes are confined to the manifest fields and
|
||||
fingerprint values described above, plus an earlier restore permission error in
|
||||
Review for anonymous sessions lacking `backups.restore`.
|
||||
|
||||
## Local verification (2026-10-13)
|
||||
|
||||
- `./mvnw -B clean verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`:
|
||||
BUILD SUCCESS; 158 tests, zero failures/errors/skips; both JaCoCo gates passed.
|
||||
Existing backup, restore and MinIO integration tests ran.
|
||||
- Separate Checkstyle 14.3.0 audit of all 16 changed Java files (main and test):
|
||||
zero violations, using the shared Kingsrook-derived rules with this repository's
|
||||
Apache file header. This repository has no Checkstyle execution configured in
|
||||
its POM; audit configuration and results are local under `.cache/style/`.
|
||||
- `git diff --check`: passed. No commit, push or core source modification.
|
||||
@@ -18,7 +18,9 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
|
||||
<properties>
|
||||
<revision>0.1.0-SNAPSHOT</revision>
|
||||
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
|
||||
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
|
||||
<revision>0.1.0-RC.5</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<jacoco.version>0.8.15</jacoco.version>
|
||||
@@ -27,7 +29,7 @@
|
||||
<parent>
|
||||
<groupId>com.kingsrook</groupId>
|
||||
<artifactId>qbit-build-parent</artifactId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
@@ -192,14 +194,14 @@
|
||||
<profiles>
|
||||
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
|
||||
<!-- imports the released 4.1 BOM, this profile is active by default and -->
|
||||
<!-- imports qqq-bom-pom 4.1.0-RC.4 ahead of the parent's 4.0.0 BOM. -->
|
||||
<!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM. -->
|
||||
<profile>
|
||||
<id>qqq-snapshot</id>
|
||||
<activation>
|
||||
<activeByDefault>true</activeByDefault>
|
||||
</activation>
|
||||
<properties>
|
||||
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
|
||||
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
|
||||
</properties>
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
|
||||
@@ -47,6 +47,8 @@ public class BackupsQBitConfig implements QBitConfig
|
||||
private String parentAppName;
|
||||
private String tablePathPrefix;
|
||||
private String schedulerName;
|
||||
private String hostSchedulerDescription;
|
||||
private Boolean hostSchedulerRunsVerification = false;
|
||||
private String instanceName;
|
||||
|
||||
private Integer maxConcurrentRuns = 1;
|
||||
@@ -202,6 +204,51 @@ public class BackupsQBitConfig implements QBitConfig
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for hostSchedulerDescription: when the application runs plan crons with
|
||||
** its own tick instead of a QQQ scheduler (schedulerName), a short name for it.
|
||||
** The Setup card then treats plan crons as firing.
|
||||
*******************************************************************************/
|
||||
public String getHostSchedulerDescription()
|
||||
{
|
||||
return (hostSchedulerDescription);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for hostSchedulerDescription.
|
||||
*******************************************************************************/
|
||||
public BackupsQBitConfig withHostSchedulerDescription(String hostSchedulerDescription)
|
||||
{
|
||||
this.hostSchedulerDescription = hostSchedulerDescription;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for hostSchedulerRunsVerification: whether the host's own tick also runs
|
||||
** the verify cron (by calling the backupVerifyPlan process). False by default.
|
||||
*******************************************************************************/
|
||||
public Boolean getHostSchedulerRunsVerification()
|
||||
{
|
||||
return (hostSchedulerRunsVerification);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for hostSchedulerRunsVerification.
|
||||
*******************************************************************************/
|
||||
public BackupsQBitConfig withHostSchedulerRunsVerification(Boolean hostSchedulerRunsVerification)
|
||||
{
|
||||
this.hostSchedulerRunsVerification = hostSchedulerRunsVerification;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for schedulerName (the QQQ scheduler used for plan jobs and the SLA checker)
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -119,13 +119,8 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
|
||||
qInstance.addQBit(qBit);
|
||||
BackupsRuntime.install(qBitConfig);
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
// bind the Alerts qbit producer SPI when it is on the class path (optional) //
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
if(com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfPresent())
|
||||
{
|
||||
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
|
||||
}
|
||||
com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfAbsent();
|
||||
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
|
||||
|
||||
com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput output = new com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput();
|
||||
for(var possibleValueSource : com.kingsrook.qbits.backups.model.BackupsTables.possibleValueSources())
|
||||
@@ -165,6 +160,7 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
|
||||
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData app = com.kingsrook.qbits.backups.app.BackupsApp.app(qBitConfig, tables);
|
||||
output.add(app);
|
||||
output.addSelfToInstance(qInstance);
|
||||
com.kingsrook.qbits.backups.model.BackupsHelp.apply(qInstance, app.getName());
|
||||
|
||||
// A parent app only lists the apps that are its children, so attach ourselves (for example Admin, then Backups).
|
||||
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData parent = qBitConfig.getParentAppName() == null ? null : qInstance.getApp(qBitConfig.getParentAppName());
|
||||
|
||||
@@ -17,105 +17,39 @@
|
||||
package com.kingsrook.qbits.backups.alerts;
|
||||
|
||||
|
||||
import java.lang.reflect.Method;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The BackupAlertSink adapter for the Alerts qbit producer SPI (QRun-IO/qqq
|
||||
** PR #1108, com.kingsrook.qqq.backend.core.alerts: QAlerts, AlertSignal,
|
||||
** AlertResolution, AlertSeverity, QAlertTypeMetaData; design spec
|
||||
** docs/specs/2026-10-12-qbit-alerts-design.md).
|
||||
**
|
||||
** STAND-IN: the SPI is an open core PR and is not in QQQ 4.1.0-RC.4, so this
|
||||
** adapter is bound REFLECTIVELY and optionally. It compiles against nothing in
|
||||
** the SPI; it looks the classes up by name, and every call is guarded:
|
||||
** <ul>
|
||||
** <li>no SPI classes on the class path: {@link #bindIfPresent()} binds nothing
|
||||
** and the sink stays the no-op;</li>
|
||||
** <li>the SPI is present but the instance has no alerts meta-data (the Alerts
|
||||
** qbit is not installed): every call is a no-op;</li>
|
||||
** <li>any reflective or SPI failure is logged once and swallowed, because an
|
||||
** alerting problem must never fail a backup.</li>
|
||||
** </ul>
|
||||
** Replace with a direct binding (QAlerts.raise and friends) when RC.5 ships.
|
||||
**
|
||||
** Mapping: raise to QAlerts.raise(AlertSignal), resolve to QAlerts.resolve(type,
|
||||
** dedupKey, note), heartbeat to QAlerts.heartbeat(key). The SPI has no per-plan
|
||||
** heartbeat registration (monitors are declared on the alert type), so the
|
||||
** expected interval and grace are not passed on; the SLA checker raises
|
||||
** backups.plan.missed itself, independent of any heartbeat (spec 10.4).
|
||||
** Direct adapter to core QAlerts. Calls are no-ops without a configured alerts
|
||||
** sink, including when only the backup alert types have been registered.
|
||||
** QAlerts catches and logs dispatch failures; adapter failures are logged once.
|
||||
** Heartbeat intervals remain the responsibility of the backup SLA checker.
|
||||
*******************************************************************************/
|
||||
public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(AlertsQBitAlertSink.class);
|
||||
|
||||
/** The package of the Alerts producer SPI in qqq-backend-core. */
|
||||
public static final String DEFAULT_API_PACKAGE = "com.kingsrook.qqq.backend.core.alerts";
|
||||
|
||||
private final String apiPackage;
|
||||
|
||||
private boolean warned = false;
|
||||
|
||||
private Class<?> qAlerts;
|
||||
private Class<?> alertSignal;
|
||||
private Class<?> alertSeverity;
|
||||
private Class<?> instanceMetaData;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor for the real SPI package.
|
||||
** Bind the core adapter unless the host already supplied a backup sink.
|
||||
*******************************************************************************/
|
||||
public AlertsQBitAlertSink()
|
||||
public static boolean bindIfAbsent()
|
||||
{
|
||||
this(DEFAULT_API_PACKAGE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor for another package (a test double of the SPI shape).
|
||||
*******************************************************************************/
|
||||
public AlertsQBitAlertSink(String apiPackage)
|
||||
{
|
||||
this.apiPackage = apiPackage;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether the SPI classes are on the class path.
|
||||
*******************************************************************************/
|
||||
public static boolean isPresent(String apiPackage)
|
||||
{
|
||||
try
|
||||
{
|
||||
Class.forName(apiPackage + ".QAlerts");
|
||||
Class.forName(apiPackage + ".AlertSignal");
|
||||
return (true);
|
||||
}
|
||||
catch(ClassNotFoundException | LinkageError e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind this adapter as the runtime's alert sink when the SPI is present and no
|
||||
** other sink was bound already. Returns whether it bound.
|
||||
*******************************************************************************/
|
||||
public static boolean bindIfPresent()
|
||||
{
|
||||
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink) || !isPresent(DEFAULT_API_PACKAGE))
|
||||
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
@@ -126,38 +60,32 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Declare the backup alert types with the Alerts qbit (QAlertTypeMetaData), when
|
||||
** the SPI is present. Returns how many were added.
|
||||
** Declare missing backup alert types on each instance, independently of sink
|
||||
** installation order. Returns how many were added.
|
||||
*******************************************************************************/
|
||||
public int registerAlertTypes(QInstance qInstance)
|
||||
{
|
||||
int added = 0;
|
||||
try
|
||||
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
|
||||
{
|
||||
Class<?> typeClass = Class.forName(apiPackage + ".QAlertTypeMetaData");
|
||||
load();
|
||||
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
|
||||
try
|
||||
{
|
||||
try
|
||||
if(QAlertTypeMetaData.of(qInstance, alertType.name()) == null)
|
||||
{
|
||||
Object type = typeClass.getConstructor().newInstance();
|
||||
typeClass.getMethod("withName", String.class).invoke(type, alertType.name());
|
||||
typeClass.getMethod("withLabel", String.class).invoke(type, alertType.label());
|
||||
typeClass.getMethod("withGroupName", String.class).invoke(type, "Backups");
|
||||
typeClass.getMethod("withHowToFix", String.class).invoke(type, alertType.howToFix());
|
||||
typeClass.getMethod("withDefaultSeverity", alertSeverity).invoke(type, severity(alertType.defaultSeverity()));
|
||||
typeClass.getMethod("addSelfToInstance", QInstance.class).invoke(type, qInstance);
|
||||
new QAlertTypeMetaData()
|
||||
.withName(alertType.name())
|
||||
.withLabel(alertType.label())
|
||||
.withGroupName("Backups")
|
||||
.withHowToFix(alertType.howToFix())
|
||||
.withDefaultSeverity(AlertSeverity.valueOf(alertType.defaultSeverity().name()))
|
||||
.addSelfToInstance(qInstance);
|
||||
added++;
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
{
|
||||
warnOnce("Could not declare the alert type " + alertType.name(), e);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | LinkageError e)
|
||||
{
|
||||
warnOnce("The alerts SPI is not available", e);
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not declare the alert type " + alertType.name(), e);
|
||||
}
|
||||
}
|
||||
return (added);
|
||||
}
|
||||
@@ -170,28 +98,24 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void raise(String type, String dedupeKey, Severity severity, String title, Map<String, String> detail)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
Object signal = alertSignal.getConstructor(String.class).newInstance(type);
|
||||
alertSignal.getMethod("setDedupKey", String.class).invoke(signal, dedupeKey);
|
||||
alertSignal.getMethod("setSeverity", alertSeverity).invoke(signal, severity(severity));
|
||||
alertSignal.getMethod("setTitle", String.class).invoke(signal, title);
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
AlertSignal signal = new AlertSignal(type)
|
||||
.withDedupKey(dedupeKey)
|
||||
.withSeverity(AlertSeverity.valueOf(severity.name()))
|
||||
.withTitle(title);
|
||||
if(detail != null && !detail.isEmpty())
|
||||
{
|
||||
alertSignal.getMethod("setSummary", String.class).invoke(signal, detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
|
||||
Method withAttribute = alertSignal.getMethod("withAttribute", String.class, String.class);
|
||||
for(Map.Entry<String, String> entry : detail.entrySet())
|
||||
{
|
||||
withAttribute.invoke(signal, entry.getKey(), entry.getValue());
|
||||
}
|
||||
signal.withSummary(detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
|
||||
detail.forEach(signal::withAttribute);
|
||||
}
|
||||
qAlerts.getMethod("raise", alertSignal).invoke(null, signal);
|
||||
QAlerts.raise(signal);
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not raise an alert", e);
|
||||
}
|
||||
@@ -205,15 +129,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void resolve(String type, String dedupeKey)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
qAlerts.getMethod("resolve", String.class, String.class, String.class).invoke(null, type, dedupeKey, "Cleared by the Backups qbit");
|
||||
if(isAlertsQBitInstalled())
|
||||
{
|
||||
QAlerts.resolve(type, dedupeKey, "Cleared by the Backups qbit");
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not resolve an alert", e);
|
||||
}
|
||||
@@ -227,15 +150,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void heartbeat(String monitorKey, long expectedEverySeconds, long graceSeconds)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
qAlerts.getMethod("heartbeat", String.class).invoke(null, monitorKey);
|
||||
if(isAlertsQBitInstalled())
|
||||
{
|
||||
QAlerts.heartbeat(monitorKey);
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not check in a heartbeat", e);
|
||||
}
|
||||
@@ -244,67 +166,26 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The SPI classes load, and the instance has alerts meta-data (the Alerts qbit
|
||||
** is installed). Checked on every call so the order qbits are produced in
|
||||
** does not matter.
|
||||
** Check on every call so installing the alerts sink after this qbit works.
|
||||
** Type declarations alone do not mean an alerts implementation is installed.
|
||||
*******************************************************************************/
|
||||
boolean isAlertsQBitInstalled()
|
||||
{
|
||||
try
|
||||
{
|
||||
load();
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
if(qInstance == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
return (instanceMetaData.getMethod("of", QInstance.class).invoke(null, qInstance) != null);
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("The alerts SPI is not usable", e);
|
||||
return (false);
|
||||
}
|
||||
AlertsInstanceMetaData metaData = AlertsInstanceMetaData.of(QContext.getQInstance());
|
||||
return (metaData != null && metaData.getSink() != null);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
** Log adapter failures once; dispatch failures are already logged by core.
|
||||
*******************************************************************************/
|
||||
private synchronized void load() throws ClassNotFoundException
|
||||
{
|
||||
if(qAlerts == null)
|
||||
{
|
||||
qAlerts = Class.forName(apiPackage + ".QAlerts");
|
||||
alertSignal = Class.forName(apiPackage + ".AlertSignal");
|
||||
alertSeverity = Class.forName(apiPackage + ".AlertSeverity");
|
||||
instanceMetaData = Class.forName(apiPackage + ".AlertsInstanceMetaData");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Map our severity to the SPI's enum constant of the same name.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings({ "unchecked", "rawtypes" })
|
||||
private Object severity(Severity severity)
|
||||
{
|
||||
return (Enum.valueOf((Class<Enum>) alertSeverity, severity.name()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private void warnOnce(String message, Throwable t)
|
||||
private synchronized void warnOnce(String message, Throwable t)
|
||||
{
|
||||
if(!warned)
|
||||
{
|
||||
warned = true;
|
||||
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t, logPair("apiPackage", apiPackage));
|
||||
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -36,11 +36,11 @@ public final class BackupAlerts
|
||||
{
|
||||
public static final String PLAN_MISSED = "backups.plan.missed";
|
||||
public static final String RUN_FAILED = "backups.run.failed";
|
||||
public static final String RUN_MISSING = "backups.run.missing_in_storage";
|
||||
public static final String RUN_MISSING = "backups.run.missing-in-storage";
|
||||
public static final String FILES_MISSING = "backups.files.missing";
|
||||
public static final String VERIFY_FAILED = "backups.verify.failed";
|
||||
public static final String VERIFY_OVERDUE = "backups.verify.overdue";
|
||||
public static final String PLAN_OWNER_INVALID = "backups.plan.owner_invalid";
|
||||
public static final String PLAN_OWNER_INVALID = "backups.plan.owner-invalid";
|
||||
public static final String STORAGE_DEBRIS = "backups.storage.debris";
|
||||
public static final String RESTORE_COMPLETED = "backups.restore.completed";
|
||||
|
||||
|
||||
@@ -70,6 +70,7 @@ public final class BackupsHomeView
|
||||
{
|
||||
StringBuilder html = new StringBuilder("<div class=\"backups-home\" style=\"" + ROOT + "\">");
|
||||
planStatus(html, data, pathPrefix);
|
||||
setup(html, data, pathPrefix);
|
||||
html.append("<div style=\"").append(GRID).append("\">");
|
||||
needsAttention(html, data, pathPrefix);
|
||||
latestTestRestore(html, data, pathPrefix);
|
||||
@@ -139,6 +140,50 @@ public final class BackupsHomeView
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Setup card: what a working backup needs, whether each part is in place,
|
||||
** and the next step, linked to its table.
|
||||
*******************************************************************************/
|
||||
private static void setup(StringBuilder html, BackupsHomeData data, String prefix)
|
||||
{
|
||||
List<BackupsSetupStatus.Item> items = BackupsSetupStatus.load(data.plans());
|
||||
open(html, "Setup");
|
||||
long todo = items.stream().filter(i -> i.state() == BackupsSetupStatus.State.TODO).count();
|
||||
html.append("<p style=\"margin:0 0 .6rem;font-size:.9rem;\">")
|
||||
.append(todo == 0 ? "Everything backups need is set up." : esc(todo + (todo == 1 ? " item needs" : " items need") + " attention."))
|
||||
.append("</p><ul style=\"list-style:none;margin:0;padding:0;\">");
|
||||
for(BackupsSetupStatus.Item item : items)
|
||||
{
|
||||
String color = switch(item.state())
|
||||
{
|
||||
case OK -> "#16a34a";
|
||||
case TODO -> "#dc2626";
|
||||
case DEFAULT -> "#d97706";
|
||||
case OPTIONAL -> "#9ca3af";
|
||||
};
|
||||
String label = switch(item.state())
|
||||
{
|
||||
case OK -> "Set up";
|
||||
case TODO -> "Needs setup";
|
||||
case DEFAULT -> "Using defaults";
|
||||
case OPTIONAL -> "Optional";
|
||||
};
|
||||
html.append("<li style=\"").append(ROW).append("\"><div style=\"display:flex;flex-wrap:wrap;align-items:center;gap:.5rem .75rem;\">")
|
||||
.append("<strong>").append("<a style=\"").append(LINK).append("\" href=\"").append(esc(prefix + "/" + item.tableName())).append("\">").append(esc(item.label())).append("</a></strong>")
|
||||
.append("<span style=\"").append(CHIP).append("\"><span aria-hidden=\"true\" style=\"width:.55rem;height:.55rem;border-radius:50%;background:").append(color).append(";display:inline-block;\"></span>").append(esc(label)).append("</span>")
|
||||
.append("</div><div style=\"font-size:.9rem;margin-top:.35rem;\">").append(esc(item.detail())).append("</div>");
|
||||
if(item.nextStep() != null)
|
||||
{
|
||||
html.append("<div style=\"").append(MUTED).append("font-size:.9rem;margin-top:.2rem;\">Next: ").append(esc(item.nextStep())).append("</div>");
|
||||
}
|
||||
html.append("</li>");
|
||||
}
|
||||
html.append("</ul>");
|
||||
close(html);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Needs attention.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.app;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
import com.kingsrook.qbits.backups.BackupsQBitConfig;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Setup card on the Backups home: what a working backup needs (a plan, a
|
||||
** storage bucket that resolves, a schedule, verification, a separate prune
|
||||
** bucket), whether each is in place, and the next step. Computed from the plans
|
||||
** the home already loaded and the runtime's storage resolver; it reads nothing
|
||||
** else.
|
||||
*******************************************************************************/
|
||||
public final class BackupsSetupStatus
|
||||
{
|
||||
/*******************************************************************************
|
||||
** OK: in place. TODO: backups need it. DEFAULT: working, with a default worth a
|
||||
** look. OPTIONAL: a feature that is off until set up.
|
||||
*******************************************************************************/
|
||||
public enum State
|
||||
{
|
||||
OK, TODO, DEFAULT, OPTIONAL
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One line of the card.
|
||||
*******************************************************************************/
|
||||
public record Item(String label, String tableName, State state, String detail, String nextStep)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private BackupsSetupStatus()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build the items from the plans.
|
||||
*******************************************************************************/
|
||||
public static List<Item> load(List<QRecord> plans)
|
||||
{
|
||||
BackupsQBitConfig config = BackupsRuntime.isInstalled() ? BackupsRuntime.getConfig() : null;
|
||||
return (load(plans, config != null && hasContent(config.getSchedulerName()), config == null ? null : config.getHostSchedulerDescription(),
|
||||
config != null && Boolean.TRUE.equals(config.getHostSchedulerRunsVerification())));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build the items from the plans, given whether a scheduler is configured (a plan
|
||||
** cron only fires through the scheduler the qbit config names).
|
||||
*******************************************************************************/
|
||||
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured)
|
||||
{
|
||||
return (load(plans, schedulerConfigured, null, false));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build the items from the plans. A host that runs plan crons with its own tick
|
||||
** (hostScheduler names it) has crons that fire; hostRunsVerification says whether
|
||||
** that tick also runs the verify cron.
|
||||
*******************************************************************************/
|
||||
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured, String hostScheduler, boolean hostRunsVerification)
|
||||
{
|
||||
boolean host = hasContent(hostScheduler);
|
||||
List<Item> items = new ArrayList<>();
|
||||
List<QRecord> active = plans.stream().filter(p -> Boolean.TRUE.equals(p.getValueBoolean("isActive"))).toList();
|
||||
|
||||
///////////
|
||||
// plans //
|
||||
///////////
|
||||
if(plans.isEmpty())
|
||||
{
|
||||
items.add(new Item("Backup plans", BackupsTables.PLAN, State.TODO, "No plan: nothing is backed up.",
|
||||
"Run Create default plan on Backup plans (it asks only for the storage bucket alias), or add a plan with New."));
|
||||
items.add(new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "None needed yet.",
|
||||
"A plan that backs up everything (scope Instance) needs no rules. Add them on a plan that is limited to apps, qbits or tables."));
|
||||
return (items);
|
||||
}
|
||||
items.add(active.isEmpty()
|
||||
? new Item("Backup plans", BackupsTables.PLAN, State.TODO, plans.size() + (plans.size() == 1 ? " plan, but it is" : " plans, but none is") + " active: nothing runs on its own.", "Turn a plan on (Active).")
|
||||
: new Item("Backup plans", BackupsTables.PLAN, State.OK, active.size() + " active plan" + (active.size() == 1 ? "" : "s") + " of " + plans.size() + ".", null));
|
||||
|
||||
/////////////////////////////////////////
|
||||
// scope rules: needed unless INSTANCE //
|
||||
/////////////////////////////////////////
|
||||
List<String> limited = plans.stream().filter(p -> p.getValueString("scopeType") != null && !"INSTANCE".equals(p.getValueString("scopeType"))).map(p -> p.getValueString("name")).toList();
|
||||
items.add(limited.isEmpty()
|
||||
? new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "Every plan backs up the whole instance, so no rules are needed.",
|
||||
"Add rules to a plan only to include or leave out apps, qbits, tables or roles.")
|
||||
: new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OK, limited.size() + (limited.size() == 1 ? " plan is" : " plans are") + " limited by scope rules: " + String.join(", ", limited) + ".",
|
||||
"Check each plan's Scope rules to see what it covers."));
|
||||
|
||||
///////////////////////////////////////////////
|
||||
// storage: every alias a plan names resolves //
|
||||
///////////////////////////////////////////////
|
||||
Set<String> aliases = new LinkedHashSet<>();
|
||||
for(QRecord plan : plans)
|
||||
{
|
||||
for(String field : List.of("storageBucketAlias", "pruneBucketAlias"))
|
||||
{
|
||||
if(plan.getValueString(field) != null)
|
||||
{
|
||||
aliases.add(plan.getValueString(field));
|
||||
}
|
||||
}
|
||||
}
|
||||
List<String> broken = new ArrayList<>();
|
||||
for(String alias : aliases)
|
||||
{
|
||||
try
|
||||
{
|
||||
BackupsRuntime.storageFor(alias);
|
||||
}
|
||||
catch(RuntimeException e)
|
||||
{
|
||||
broken.add(alias);
|
||||
}
|
||||
}
|
||||
items.add(broken.isEmpty()
|
||||
? new Item("Storage", BackupsTables.PLAN, State.OK, aliases.size() + " storage bucket alias" + (aliases.size() == 1 ? "" : "es") + " in use, and each resolves.", "Run Test destination on a plan to prove it can write.")
|
||||
: new Item("Storage", BackupsTables.PLAN, State.TODO, "These aliases do not resolve to a bucket: " + String.join(", ", broken) + ".",
|
||||
"Fix the alias on the plan, or add the bucket to the application's storage resolver."));
|
||||
|
||||
/////////////////////////////////////////
|
||||
// schedule: an active plan has a cron //
|
||||
/////////////////////////////////////////
|
||||
long scheduled = active.stream().filter(p -> hasContent(p.getValueString("cronExpression"))).count();
|
||||
String noScheduler = "No scheduler is configured (schedulerName on the qbit config), so plan crons are saved but never fire";
|
||||
items.add(!schedulerConfigured && !host
|
||||
? new Item("Schedule", BackupsTables.PLAN, State.TODO, noScheduler + ". Backups run only when someone clicks Run backup now.",
|
||||
"Set schedulerName on the qbit config (or run your own tick if the application does), then give each plan a cron expression.")
|
||||
: scheduled == 0
|
||||
? new Item("Schedule", BackupsTables.PLAN, State.TODO, "No active plan has a schedule: backups run only when someone clicks Run backup now.", "Set a Cron expression on a plan (for example " + DefaultPlanProcess.CRON + ").")
|
||||
: new Item("Schedule", BackupsTables.PLAN, State.OK, scheduled + " active plan" + (scheduled == 1 ? " runs" : "s run") + " on a schedule"
|
||||
+ (host && !schedulerConfigured ? ", run by " + hostScheduler + " (the application scheduler)." : "."), null));
|
||||
|
||||
//////////////////
|
||||
// verification //
|
||||
//////////////////
|
||||
long verified = active.stream().filter(p -> hasContent(p.getValueString("verifyCronExpression"))).count();
|
||||
items.add(host && !schedulerConfigured && !hostRunsVerification
|
||||
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "Not run by " + hostScheduler + ": verify cron expressions are saved but nothing fires them.",
|
||||
"Run Verify backup on a run by hand, or have the application scheduler run the backupVerifyPlan process for each plan.")
|
||||
: !schedulerConfigured && !host
|
||||
? new Item("Verification", BackupsTables.VERIFICATION, State.TODO, noScheduler + ", so no verification is scheduled.",
|
||||
"Set schedulerName on the qbit config, then a verify cron expression on a plan; or run Verify backup on a run by hand.")
|
||||
: verified == 0
|
||||
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "No plan verifies its backups on a schedule.",
|
||||
"Set a Verify cron expression on a plan, or run Verify backup on a run. A backup nobody has restored is a guess.")
|
||||
: new Item("Verification", BackupsTables.VERIFICATION, State.OK, verified + " active plan" + (verified == 1 ? " verifies" : "s verify") + " its latest backup on a schedule.", null));
|
||||
|
||||
////////////////////////////////////////////
|
||||
// prune bucket: separate delete authority //
|
||||
////////////////////////////////////////////
|
||||
long noPrune = plans.stream().filter(p -> !hasContent(p.getValueString("pruneBucketAlias"))).count();
|
||||
items.add(noPrune == 0
|
||||
? new Item("Prune bucket", BackupsTables.PLAN, State.OK, "Every plan prunes through its own delete-capable bucket alias.", null)
|
||||
: new Item("Prune bucket", BackupsTables.PLAN, State.DEFAULT, noPrune + (noPrune == 1 ? " plan has" : " plans have") + " no prune bucket: Prune uses the backup bucket, so one key can both write and delete backups.",
|
||||
"Give the plan a separate delete-capable Prune bucket alias."));
|
||||
return (items);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static boolean hasContent(String value)
|
||||
{
|
||||
return (value != null && !value.isBlank());
|
||||
}
|
||||
}
|
||||
@@ -137,6 +137,7 @@ public record BackupManifest(
|
||||
String backend,
|
||||
String primaryKey,
|
||||
String fingerprint,
|
||||
String fingerprintAlgorithm,
|
||||
Map<String, Object> schema,
|
||||
String filter,
|
||||
List<ExcludedField> excludedFields,
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Frozen pre-RC.6 fingerprint algorithm, used only to read existing manifests.
|
||||
** Keep its descriptor and escaping byte-compatible with the original writer.
|
||||
*******************************************************************************/
|
||||
final class LegacyTableFingerprint
|
||||
{
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private LegacyTableFingerprint()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
|
||||
** what goes in the manifest schema).
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
|
||||
{
|
||||
Map<String, Object> d = new LinkedHashMap<>();
|
||||
d.put("name", table.getName());
|
||||
d.put("primaryKey", table.getPrimaryKeyField());
|
||||
d.put("backendType", backendType);
|
||||
|
||||
Map<String, Object> fields = new TreeMap<>();
|
||||
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
|
||||
{
|
||||
Map<String, Object> f = new LinkedHashMap<>();
|
||||
f.put("type", field.getType().name());
|
||||
f.put("required", field.getIsRequired());
|
||||
f.put("maxLength", field.getMaxLength());
|
||||
f.put("pvs", field.getPossibleValueSourceName());
|
||||
f.put("heavy", field.getIsHeavy());
|
||||
f.put("sensitive", field.getIsSensitive());
|
||||
fields.put(field.getName(), f);
|
||||
}
|
||||
d.put("fields", fields);
|
||||
|
||||
List<String> uniqueKeys = new ArrayList<>();
|
||||
if(table.getUniqueKeys() != null)
|
||||
{
|
||||
for(UniqueKey key : table.getUniqueKeys())
|
||||
{
|
||||
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
|
||||
}
|
||||
}
|
||||
uniqueKeys.sort(String::compareTo);
|
||||
d.put("uniqueKeys", uniqueKeys);
|
||||
|
||||
List<String> associations = new ArrayList<>();
|
||||
if(table.getAssociations() != null)
|
||||
{
|
||||
for(Association association : table.getAssociations())
|
||||
{
|
||||
associations.add(association.getName() + "->" + association.getAssociatedTableName());
|
||||
}
|
||||
}
|
||||
associations.sort(String::compareTo);
|
||||
d.put("associations", associations);
|
||||
return (d);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a table: sha256:hex of the canonical descriptor.
|
||||
*******************************************************************************/
|
||||
public static String of(QTableMetaData table, String backendType)
|
||||
{
|
||||
return (fingerprintOf(descriptor(table, backendType)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
|
||||
*******************************************************************************/
|
||||
public static String fingerprintOf(Map<String, Object> descriptor)
|
||||
{
|
||||
StringBuilder sb = new StringBuilder();
|
||||
canonical(descriptor, sb);
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private static void canonical(Object o, StringBuilder sb)
|
||||
{
|
||||
if(o == null)
|
||||
{
|
||||
sb.append("null");
|
||||
}
|
||||
else if(o instanceof Map<?, ?> map)
|
||||
{
|
||||
sb.append('{');
|
||||
boolean first = true;
|
||||
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
|
||||
{
|
||||
if(!first)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
first = false;
|
||||
sb.append('"').append(e.getKey()).append("\":");
|
||||
canonical(e.getValue(), sb);
|
||||
}
|
||||
sb.append('}');
|
||||
}
|
||||
else if(o instanceof List<?> list)
|
||||
{
|
||||
sb.append('[');
|
||||
for(int i = 0; i < list.size(); i++)
|
||||
{
|
||||
if(i > 0)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
canonical(list.get(i), sb);
|
||||
}
|
||||
sb.append(']');
|
||||
}
|
||||
else if(o instanceof Number || o instanceof Boolean)
|
||||
{
|
||||
sb.append(o);
|
||||
}
|
||||
else
|
||||
{
|
||||
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -18,28 +18,23 @@ package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A stable schema descriptor and SHA-256 fingerprint of a table (spec 5.5).
|
||||
**
|
||||
** This is an internal stand-in for the QQQ core TableSchemaFingerprint (core PR
|
||||
** feature/schema-fingerprint). When that ships in the next 4.1.x RC, this class
|
||||
** delegates to it and the descriptor below is kept only for reading old manifests.
|
||||
** Labels, adornments and help text are excluded, so cosmetic changes do not count.
|
||||
** Core schema fingerprints with read compatibility for pre-RC.6 manifests.
|
||||
*******************************************************************************/
|
||||
public final class TableFingerprints
|
||||
{
|
||||
public static final String CORE_ALGORITHM = "qqq-table-schema-v1";
|
||||
public static final String LEGACY_ALGORITHM = "qbit-backups-table-v1";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
@@ -51,134 +46,83 @@ public final class TableFingerprints
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
|
||||
** what goes in the manifest schema).
|
||||
** The core descriptor stored in new manifests.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
|
||||
{
|
||||
Map<String, Object> d = new LinkedHashMap<>();
|
||||
d.put("name", table.getName());
|
||||
d.put("primaryKey", table.getPrimaryKeyField());
|
||||
d.put("backendType", backendType);
|
||||
|
||||
Map<String, Object> fields = new TreeMap<>();
|
||||
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
|
||||
{
|
||||
Map<String, Object> f = new LinkedHashMap<>();
|
||||
f.put("type", field.getType().name());
|
||||
f.put("required", field.getIsRequired());
|
||||
f.put("maxLength", field.getMaxLength());
|
||||
f.put("pvs", field.getPossibleValueSourceName());
|
||||
f.put("heavy", field.getIsHeavy());
|
||||
f.put("sensitive", field.getIsSensitive());
|
||||
fields.put(field.getName(), f);
|
||||
}
|
||||
d.put("fields", fields);
|
||||
|
||||
List<String> uniqueKeys = new ArrayList<>();
|
||||
if(table.getUniqueKeys() != null)
|
||||
{
|
||||
for(UniqueKey key : table.getUniqueKeys())
|
||||
{
|
||||
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
|
||||
}
|
||||
}
|
||||
uniqueKeys.sort(String::compareTo);
|
||||
d.put("uniqueKeys", uniqueKeys);
|
||||
|
||||
List<String> associations = new ArrayList<>();
|
||||
if(table.getAssociations() != null)
|
||||
{
|
||||
for(Association association : table.getAssociations())
|
||||
{
|
||||
associations.add(association.getName() + "->" + association.getAssociatedTableName());
|
||||
}
|
||||
}
|
||||
associations.sort(String::compareTo);
|
||||
d.put("associations", associations);
|
||||
return (d);
|
||||
return (TableSchemaFingerprint.of(table, backendType).getDescriptor());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a table: sha256:hex of the canonical descriptor.
|
||||
** Retain the manifest's sha256 prefix around the core hash.
|
||||
*******************************************************************************/
|
||||
public static String of(QTableMetaData table, String backendType)
|
||||
{
|
||||
return (fingerprintOf(descriptor(table, backendType)));
|
||||
return ("sha256:" + TableSchemaFingerprint.of(table, backendType).getHash());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
|
||||
** An absent algorithm identifies manifests written before RC.6. Unknown
|
||||
** algorithms must not silently fall back to a different fingerprint.
|
||||
*******************************************************************************/
|
||||
public static String fingerprintOf(Map<String, Object> descriptor)
|
||||
public static boolean matches(BackupManifest.TableEntry entry, QTableMetaData table, String backendType)
|
||||
{
|
||||
StringBuilder sb = new StringBuilder();
|
||||
canonical(descriptor, sb);
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
|
||||
if(entry.fingerprint() == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
String algorithm = entry.fingerprintAlgorithm();
|
||||
if(algorithm == null || LEGACY_ALGORITHM.equals(algorithm))
|
||||
{
|
||||
return (entry.fingerprint().equals(LegacyTableFingerprint.of(table, backendType)));
|
||||
}
|
||||
return (CORE_ALGORITHM.equals(algorithm) && entry.fingerprint().equals(of(table, backendType)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a whole instance's tables: a hash of the table fingerprints.
|
||||
** Normalize both descriptor versions to field-name maps for restore's
|
||||
** compatibility analysis. Core fields use a list and explicit flag names.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
public static Map<String, Object> fields(Map<String, Object> descriptor)
|
||||
{
|
||||
Object fields = descriptor == null ? null : descriptor.get("fields");
|
||||
if(fields instanceof Map<?, ?> map)
|
||||
{
|
||||
return ((Map<String, Object>) map);
|
||||
}
|
||||
Map<String, Object> result = new LinkedHashMap<>();
|
||||
if(fields instanceof List<?> list)
|
||||
{
|
||||
for(Object item : list)
|
||||
{
|
||||
Map<String, Object> field = (Map<String, Object>) item;
|
||||
Map<String, Object> normalized = new LinkedHashMap<>();
|
||||
normalized.put("type", field.get("type"));
|
||||
normalized.put("required", field.get("isRequired"));
|
||||
normalized.put("maxLength", field.get("maxLength"));
|
||||
normalized.put("pvs", field.get("possibleValueSource"));
|
||||
normalized.put("heavy", field.get("isHeavy"));
|
||||
normalized.put("sensitive", field.get("isSensitive"));
|
||||
result.put((String) field.get("name"), normalized);
|
||||
}
|
||||
}
|
||||
return (result);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The instance fingerprint remains a hash of the ordered table fingerprints.
|
||||
*******************************************************************************/
|
||||
public static String combine(List<String> tableFingerprints)
|
||||
{
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(String.join("\n", tableFingerprints).getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private static void canonical(Object o, StringBuilder sb)
|
||||
{
|
||||
if(o == null)
|
||||
{
|
||||
sb.append("null");
|
||||
}
|
||||
else if(o instanceof Map<?, ?> map)
|
||||
{
|
||||
sb.append('{');
|
||||
boolean first = true;
|
||||
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
|
||||
{
|
||||
if(!first)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
first = false;
|
||||
sb.append('"').append(e.getKey()).append("\":");
|
||||
canonical(e.getValue(), sb);
|
||||
}
|
||||
sb.append('}');
|
||||
}
|
||||
else if(o instanceof List<?> list)
|
||||
{
|
||||
sb.append('[');
|
||||
for(int i = 0; i < list.size(); i++)
|
||||
{
|
||||
if(i > 0)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
canonical(list.get(i), sb);
|
||||
}
|
||||
sb.append(']');
|
||||
}
|
||||
else if(o instanceof Number || o instanceof Boolean)
|
||||
{
|
||||
sb.append(o);
|
||||
}
|
||||
else
|
||||
{
|
||||
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.model;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Applies help/qbit-backups.yaml (the QQQ help-bundle format) to the Backups app
|
||||
** and its backups_* tables: what each table is for and how its rows get created,
|
||||
** shown on the table screen and when it is empty.
|
||||
** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with HelpBundles.register once the pinned QQQ
|
||||
** release includes it; the bundle file stays the same.
|
||||
*******************************************************************************/
|
||||
public final class BackupsHelp
|
||||
{
|
||||
public static final String RESOURCE = "/help/qbit-backups.yaml";
|
||||
public static final String SLOT = "help";
|
||||
public static final String APP_KEY = "app:backups;slot:help";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private BackupsHelp()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The bundle's entries by key, validated (fails startup if malformed).
|
||||
*******************************************************************************/
|
||||
public static Map<String, JsonNode> entries() throws QException
|
||||
{
|
||||
try(InputStream stream = BackupsHelp.class.getResourceAsStream(RESOURCE))
|
||||
{
|
||||
if(stream == null)
|
||||
{
|
||||
throw (new IOException("missing resource"));
|
||||
}
|
||||
JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
|
||||
if(!"qbit-backups".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
|
||||
{
|
||||
throw (new IOException("invalid bundle header"));
|
||||
}
|
||||
Map<String, JsonNode> entries = new LinkedHashMap<>();
|
||||
for(JsonNode entry : bundle.path("entries"))
|
||||
{
|
||||
String key = entry.path("key").asText();
|
||||
String summary = entry.path("summary").asText();
|
||||
if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
|
||||
{
|
||||
throw (new IOException("invalid or duplicate entry: " + key));
|
||||
}
|
||||
}
|
||||
return (entries);
|
||||
}
|
||||
catch(IOException | RuntimeException e)
|
||||
{
|
||||
throw (new QException("Could not load " + RESOURCE, e));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Apply the help to the app and every backups_* table in the instance. A host's
|
||||
** own help in the slot is kept (the bundle only fills it).
|
||||
*******************************************************************************/
|
||||
public static void apply(QInstance qInstance, String appName) throws QException
|
||||
{
|
||||
Map<String, JsonNode> entries = entries();
|
||||
for(QTableMetaData table : qInstance.getTables().values())
|
||||
{
|
||||
JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
|
||||
if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
|
||||
{
|
||||
table.withHelpContent(SLOT, content(entry));
|
||||
}
|
||||
}
|
||||
QAppMetaData app = qInstance.getApp(appName);
|
||||
JsonNode appEntry = entries.get(APP_KEY);
|
||||
if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
|
||||
{
|
||||
app.withHelpContent(SLOT, content(appEntry));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Summary as the first paragraph, then the content, as markdown.
|
||||
*******************************************************************************/
|
||||
static QHelpContent content(JsonNode entry)
|
||||
{
|
||||
return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
|
||||
@@ -446,8 +447,8 @@ public final class BackupsTables
|
||||
.withField(new QFieldMetaData("pruneBucketAlias", QFieldType.STRING).withMaxLength(100))
|
||||
.withField(new QFieldMetaData("prefix", QFieldType.STRING).withMaxLength(255))
|
||||
.withField(new QFieldMetaData("critical", QFieldType.BOOLEAN).withDefaultValue(false))
|
||||
.withField(enumField("format", "format", "JSONL"))
|
||||
.withField(enumField("compression", "compression", "GZIP"))
|
||||
.withField(enumField("format", "format", "JSONL").withHelpContent(new QHelpContent("JSONL, CSV and JSONL_AND_CSV are available. PARQUET is not available yet and is refused when the plan is saved.")))
|
||||
.withField(enumField("compression", "compression", "GZIP").withHelpContent(new QHelpContent("GZIP and NONE are available. ZSTD is not available yet and is refused when the plan is saved.")))
|
||||
.withField(enumField("encryptionMode", "encryption_mode", "NONE"))
|
||||
.withField(new QFieldMetaData("encryptionKeyRef", QFieldType.STRING).withMaxLength(100))
|
||||
.withField(enumField("sensitiveFieldPolicy", "sensitive_policy", "EXCLUDE"))
|
||||
@@ -455,12 +456,15 @@ public final class BackupsTables
|
||||
.withField(enumField("fileObjectPolicy", "file_policy", "COPY"))
|
||||
.withField(enumField("fileMissingPolicy", "file_missing_policy", "WARN"))
|
||||
.withField(enumField("consistency", "consistency", "SNAPSHOT_PREFERRED"))
|
||||
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false))
|
||||
.withField(enumField("incrementalMode", "incremental_mode", "FULL"))
|
||||
.withField(enumField("nativeDump", "native_dump", "NONE"))
|
||||
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false)
|
||||
.withHelpContent(new QHelpContent("Not available yet: backups always read from the application's own backend. Leave this off; turning it on is refused when the plan is saved.")))
|
||||
.withField(enumField("incrementalMode", "incremental_mode", "FULL")
|
||||
.withHelpContent(new QHelpContent("Only FULL is available yet. CHANGED_SINCE_LAST is not available yet and is refused when the plan is saved.")))
|
||||
.withField(enumField("nativeDump", "native_dump", "NONE")
|
||||
.withHelpContent(new QHelpContent("Only NONE is available yet. Native database dumps (PG_DUMP_CUSTOM) are not available yet and are refused when the plan is saved.")))
|
||||
.withField(new QFieldMetaData("cronExpression", QFieldType.STRING).withMaxLength(100))
|
||||
.withField(new QFieldMetaData("cronTimeZoneId", QFieldType.STRING).withMaxLength(60))
|
||||
.withField(roInt("scheduledJobId"))
|
||||
.withField(roInt("scheduledJobId").withHelpContent(new QHelpContent("Not used: a plan's schedules are scheduled jobs found by their foreign key (backups_plan for the backup, backups_plan_verify for the verification), so look there.")))
|
||||
.withField(new QFieldMetaData("slaHours", QFieldType.INTEGER).withDefaultValue(26))
|
||||
.withField(new QFieldMetaData("verifySlaDays", QFieldType.INTEGER).withDefaultValue(8))
|
||||
.withField(new QFieldMetaData("verifyFileSamplePercent", QFieldType.INTEGER).withDefaultValue(100))
|
||||
|
||||
@@ -48,7 +48,7 @@ import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
|
||||
public class BackupPlanTableCustomizer implements TableCustomizerInterface
|
||||
{
|
||||
private static final List<String> CONFIGURATION_FIELDS = List.of("planKey", "isActive", "scopeType", "includeRoles", "storageBucketAlias", "pruneBucketAlias",
|
||||
"format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "incrementalMode", "nativeDump", "cronExpression", "cronTimeZoneId",
|
||||
"format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "incrementalMode", "nativeDump", "useReadReplica", "cronExpression", "cronTimeZoneId",
|
||||
"verifyCronExpression", "retainDaily", "retainWeekly", "retainMonthly", "retainYearly", "retainMinimum", "objectLockMode", "objectLockDays",
|
||||
"slaHours", "pageSize", "ownerUserId");
|
||||
|
||||
|
||||
@@ -176,6 +176,10 @@ public final class BackupPlanValidator
|
||||
{
|
||||
errors.add("Native database dumps are not available yet. Use NONE.");
|
||||
}
|
||||
if(settings.bool("useReadReplica", false))
|
||||
{
|
||||
errors.add("Reading from a replica is not available yet: backups read from the application's own backend. Turn Use read replica off.");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -72,6 +72,7 @@ public final class BackupsProcesses
|
||||
List<QProcessMetaData> processes = new ArrayList<>();
|
||||
processes.add(permission(etl(RUN_PLAN, "Run backup now", BackupsTables.PLAN, RunPlanTransform.class, RunPlanLoad.class,
|
||||
"This is a preview of what the backup will include. Nothing is written until you continue."), BackupsPermissions.RUN));
|
||||
processes.add(DefaultPlanProcess.process());
|
||||
processes.addAll(VerifyProcesses.processes());
|
||||
processes.addAll(RestoreProcesses.processes());
|
||||
processes.addAll(CatalogProcesses.processes());
|
||||
|
||||
@@ -0,0 +1,314 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.processes;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
|
||||
import com.kingsrook.qbits.backups.plan.BackupPlanValidator;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qbits.backups.scope.PlanSettings;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** "Create default plan": a table action on backups_plan that adds one working
|
||||
** plan, "Nightly full", so a fresh install has something to run. The only thing
|
||||
** it asks for is the storage bucket alias, because that is the one value the qbit
|
||||
** cannot know (it names a bucket the host's storage resolver provides). Review
|
||||
** shows the plan that will be created and checks it against the same rules a
|
||||
** hand-made plan must pass; Result links the new plan.
|
||||
**
|
||||
** Insert-only: when a plan with the key nightly-full exists (edited or not), nothing
|
||||
** is created or changed. The plan goes through the normal insert, so the owner is
|
||||
** the user running the action and the schedule syncs like any other plan.
|
||||
*******************************************************************************/
|
||||
public final class DefaultPlanProcess
|
||||
{
|
||||
public static final String NAME = "backupsPlanCreateDefault";
|
||||
public static final String FIELD_ALIAS = "storageBucketAlias";
|
||||
public static final String PLAN_KEY = "nightly-full";
|
||||
public static final String PLAN_NAME = "Nightly full";
|
||||
public static final String CRON = "0 0 2 * * ?";
|
||||
public static final String VERIFY_CRON = "0 0 4 ? * SUN";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private DefaultPlanProcess()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The process: a table-level action with one question (the bucket alias).
|
||||
*******************************************************************************/
|
||||
public static QProcessMetaData process()
|
||||
{
|
||||
Map<String, Serializable> values = new HashMap<>();
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, BackupsTables.PLAN);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, BackupsTables.PLAN);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
|
||||
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, "This plan will be created. Nothing is written until you continue. You can edit every setting afterwards.");
|
||||
QProcessMetaData process = StreamedETLWithFrontendProcess.defineProcessMetaData(Extract.class, Transform.class, Load.class, values);
|
||||
process.setName(NAME);
|
||||
process.setLabel("Create default plan");
|
||||
process.setIcon(new QIcon("playlist_add"));
|
||||
process.setTableName(BackupsTables.PLAN);
|
||||
process.withStep(0, new QFrontendStepMetaData()
|
||||
.withName("input")
|
||||
.withLabel("Where should the backups go?")
|
||||
.withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM))
|
||||
.withFormField(new QFieldMetaData(FIELD_ALIAS, QFieldType.STRING).withLabel("Storage bucket alias").withIsRequired(true)
|
||||
.withHelpContent(new QHelpContent("The alias of a bucket your application provides to Backups (never a key or an endpoint). It becomes the plan's Storage bucket alias."))));
|
||||
process.setMinInputRecords(0);
|
||||
process.setPermissionRules(BackupsTables.permissionRules());
|
||||
return (BackupsProcesses.permission(process, BackupsPermissions.ADMIN));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The values of the default plan for a bucket alias. Every setting is explicit
|
||||
** so the plan does not depend on the backend applying field defaults.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Serializable> planValues(String alias)
|
||||
{
|
||||
Map<String, Serializable> plan = new LinkedHashMap<>();
|
||||
plan.put("name", PLAN_NAME);
|
||||
plan.put("planKey", PLAN_KEY);
|
||||
plan.put("description", "Every night: all data and audit tables of the application, and the files they reference, to the storage bucket. Edit any setting on this plan.");
|
||||
plan.put("isActive", true);
|
||||
plan.put("scopeType", "INSTANCE");
|
||||
plan.put("includeRoles", "DATA,AUDIT");
|
||||
plan.put("storageBucketAlias", alias);
|
||||
plan.put("critical", true);
|
||||
plan.put("format", "JSONL");
|
||||
plan.put("compression", "GZIP");
|
||||
plan.put("encryptionMode", "NONE");
|
||||
plan.put("sensitiveFieldPolicy", "EXCLUDE");
|
||||
plan.put("heavyFieldPolicy", "INLINE");
|
||||
plan.put("fileObjectPolicy", "COPY");
|
||||
plan.put("fileMissingPolicy", "WARN");
|
||||
plan.put("consistency", "SNAPSHOT_PREFERRED");
|
||||
plan.put("useReadReplica", false);
|
||||
plan.put("incrementalMode", "FULL");
|
||||
plan.put("nativeDump", "NONE");
|
||||
plan.put("cronExpression", CRON);
|
||||
plan.put("cronTimeZoneId", "UTC");
|
||||
plan.put("slaHours", 26);
|
||||
plan.put("verifySlaDays", 8);
|
||||
plan.put("verifyFileSamplePercent", 100);
|
||||
plan.put("verifyCronExpression", VERIFY_CRON);
|
||||
plan.put("verifyLevel", "CONTENT");
|
||||
plan.put("retainDaily", 7);
|
||||
plan.put("retainWeekly", 4);
|
||||
plan.put("retainMonthly", 12);
|
||||
plan.put("retainYearly", 0);
|
||||
plan.put("retainMinimum", 3);
|
||||
plan.put("objectLockMode", "NONE");
|
||||
plan.put("maxDurationMinutes", 240);
|
||||
plan.put("pageSize", 5000);
|
||||
plan.put("targetPartBytes", 268435456L);
|
||||
return (plan);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether the default plan already exists.
|
||||
*******************************************************************************/
|
||||
private static boolean exists() throws QException
|
||||
{
|
||||
return (!new BackupCatalog().where(BackupsTables.PLAN, "planKey", PLAN_KEY).isEmpty());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Extract: the plan to create, or nothing when it exists already.
|
||||
*******************************************************************************/
|
||||
public static class Extract extends AbstractExtractStep
|
||||
{
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
String alias = input.getValueString(FIELD_ALIAS);
|
||||
if(alias == null || alias.isBlank())
|
||||
{
|
||||
throw (new QUserFacingException("Enter the storage bucket alias the backups should go to."));
|
||||
}
|
||||
if(!exists())
|
||||
{
|
||||
QRecord record = new QRecord().withTableName(BackupsTables.PLAN);
|
||||
record.setValues(new LinkedHashMap<>(planValues(alias.strip())));
|
||||
getRecordPipe().addRecord(record);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Review: the plan, checked against the rules a plan must pass.
|
||||
*******************************************************************************/
|
||||
public static class Transform extends AbstractTransformStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private final ProcessSummaryLine create = BackupsProcesses.line(Status.OK, "plan will be created", "plans will be created", "plan was created", "plans were created");
|
||||
private final ProcessSummaryLine invalid = BackupsProcesses.line(Status.ERROR, "plan cannot be created", "plans cannot be created", "plan could not be created", "plans could not be created");
|
||||
private final List<String> problems = new ArrayList<>();
|
||||
private boolean sawRecord;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
for(QRecord record : input.getRecords())
|
||||
{
|
||||
sawRecord = true;
|
||||
BackupPlanValidator.Result result = BackupPlanValidator.validate(PlanSettings.of(record), null, false, List.of());
|
||||
boolean nameTaken = !new BackupCatalog().where(BackupsTables.PLAN, "name", record.getValueString("name")).isEmpty();
|
||||
if(nameTaken)
|
||||
{
|
||||
problems.add("A plan named " + record.getValueString("name") + " already exists. Rename it or edit that plan.");
|
||||
}
|
||||
if(result.ok() && !nameTaken)
|
||||
{
|
||||
create.incrementCountAndAddPrimaryKey(record.getValueString("planKey"));
|
||||
output.getRecords().add(record);
|
||||
}
|
||||
else
|
||||
{
|
||||
invalid.incrementCount();
|
||||
problems.addAll(result.errors());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
|
||||
create.addSelfToListIfAnyCount(lines);
|
||||
invalid.setBulletsOfText(new ArrayList<>(problems));
|
||||
invalid.addSelfToListIfAnyCount(lines);
|
||||
if(!sawRecord)
|
||||
{
|
||||
lines.add(new ProcessSummaryLine(Status.INFO, null, "A plan with the key " + PLAN_KEY + " already exists, so nothing will be created. Edit that plan instead."));
|
||||
}
|
||||
return (lines);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Load: insert through the normal insert (validation, owner, schedule).
|
||||
*******************************************************************************/
|
||||
public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface
|
||||
{
|
||||
private final ProcessSummaryLine created = BackupsProcesses.line(Status.OK, "plan will be created", "plans will be created", "plan was created", "plans were created");
|
||||
private final ProcessSummaryLine failed = BackupsProcesses.line(Status.ERROR, "plan could not be created", "plans could not be created", "plan could not be created", "plans could not be created");
|
||||
private final List<String> problems = new ArrayList<>();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
InsertInput insert = new InsertInput(BackupsTables.PLAN);
|
||||
insert.setRecords(input.getRecords());
|
||||
InsertOutput inserted = new InsertAction().execute(insert);
|
||||
for(QRecord record : inserted.getRecords())
|
||||
{
|
||||
if(record.getErrors() == null || record.getErrors().isEmpty())
|
||||
{
|
||||
created.incrementCountAndAddPrimaryKey(record.getValue("id"));
|
||||
output.getRecords().add(record);
|
||||
}
|
||||
else
|
||||
{
|
||||
failed.incrementCount();
|
||||
record.getErrors().forEach(error -> problems.add(error.getMessage()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
|
||||
{
|
||||
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
|
||||
created.addSelfToListIfAnyCount(lines);
|
||||
failed.setBulletsOfText(new ArrayList<>(problems));
|
||||
failed.addSelfToListIfAnyCount(lines);
|
||||
return (lines);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -53,6 +53,7 @@ public final class LifecycleProcesses
|
||||
{
|
||||
List<QProcessMetaData> processes = new ArrayList<>();
|
||||
processes.add(BackupScheduledRunProcess.processMetaData());
|
||||
processes.add(com.kingsrook.qbits.backups.schedule.BackupScheduledVerifyProcess.processMetaData());
|
||||
processes.addAll(CancelRunProcess.processes());
|
||||
processes.add(ResumeRunProcess.process());
|
||||
processes.add(TestDestinationProcess.process());
|
||||
|
||||
@@ -105,7 +105,8 @@ public class RestoreEngine
|
||||
Map<String, String> targets = new LinkedHashMap<>();
|
||||
QInstance instance = QContext.getQInstance();
|
||||
|
||||
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE))
|
||||
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE)
|
||||
|| (request.getWriter().usesCoreRestoreFlag() && (QContext.getQSession() == null || !QContext.getQSession().hasPermission(BackupsPermissions.RESTORE))))
|
||||
{
|
||||
errors.add("Restoring needs the " + BackupsPermissions.RESTORE + " permission.");
|
||||
}
|
||||
|
||||
@@ -341,14 +341,14 @@ public class RestorePass
|
||||
String backendType = instance.getBackend(c.target.getBackendName()) == null ? null : instance.getBackend(c.target.getBackendName()).getBackendType();
|
||||
RestoreOutcome.TableOutcome o = c.outcome;
|
||||
|
||||
if(c.entry.fingerprint() != null && c.entry.fingerprint().equals(TableFingerprints.of(c.target, backendType)))
|
||||
if(TableFingerprints.matches(c.entry, c.target, backendType))
|
||||
{
|
||||
o.setSchemaStatus("MATCH");
|
||||
return;
|
||||
}
|
||||
|
||||
Map<String, Object> backupFields = c.entry.schema() == null ? Map.of() : (Map<String, Object>) c.entry.schema().getOrDefault("fields", Map.of());
|
||||
Map<String, Object> targetFields = (Map<String, Object>) TableFingerprints.descriptor(c.target, backendType).get("fields");
|
||||
Map<String, Object> backupFields = TableFingerprints.fields(c.entry.schema());
|
||||
Map<String, Object> targetFields = TableFingerprints.fields(TableFingerprints.descriptor(c.target, backendType));
|
||||
boolean incompatible = false;
|
||||
boolean different = false;
|
||||
|
||||
|
||||
@@ -28,9 +28,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
|
||||
/*******************************************************************************
|
||||
** How a restore writes (spec 5.3 and 5.4). The default implementation uses the
|
||||
** plain Insert, Update and Delete actions, and selects the core RestoreWriteFlag
|
||||
** (QRun-IO/qqq#1111) and the core key-sequence resync (#1112) reflectively when the
|
||||
** running QQQ has them. A backend specific writer can be supplied on the request.
|
||||
** plain Insert, Update and Delete actions with core RestoreWriteFlag and typed
|
||||
** key-sequence resync. A backend specific writer can be supplied on the request.
|
||||
**
|
||||
** Whatever the implementation, record security, field security, unique keys and
|
||||
** required-field validation still apply to every write.
|
||||
|
||||
@@ -18,7 +18,6 @@ package com.kingsrook.qbits.backups.restore;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.lang.reflect.Method;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.LinkedHashMap;
|
||||
@@ -30,8 +29,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.ActionFlag;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
@@ -41,48 +40,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.SequenceResyncResult;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The default RestoreWriter: the standard Insert, Update and Delete actions.
|
||||
**
|
||||
** Stand-ins (replace with the core API when QQQ 4.1.0-RC.5 ships):
|
||||
** <ul>
|
||||
** <li>RestoreWriteFlag (QRun-IO/qqq#1111) is created reflectively when the class is
|
||||
** present. Without it, the DML audit is omitted (a summary audit is written per
|
||||
** table instead), but created and modified dates may be set by the normal defaults.</li>
|
||||
** <li>resyncKeySequences (QRun-IO/qqq#1112) is called reflectively on the backend
|
||||
** module when it exists; otherwise it is a no-op that the Review reports.</li>
|
||||
** </ul>
|
||||
** The default RestoreWriter using core restore flags and key sequence resync.
|
||||
*******************************************************************************/
|
||||
public class StandardRestoreWriter implements RestoreWriter
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(StandardRestoreWriter.class);
|
||||
|
||||
private static final String FLAG_CLASS = "com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag";
|
||||
|
||||
private static final Class<?> FLAG = lookup(FLAG_CLASS);
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Class<?> lookup(String name)
|
||||
{
|
||||
try
|
||||
{
|
||||
return (Class.forName(name));
|
||||
}
|
||||
catch(ClassNotFoundException | LinkageError e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@@ -148,27 +113,15 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create the core flag (when the running QQQ has it) and hand it to the input.
|
||||
** Apply the core restore behavior requested by the caller.
|
||||
*******************************************************************************/
|
||||
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink) throws QException
|
||||
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink)
|
||||
{
|
||||
if(FLAG == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
Object flag = FLAG.getDeclaredConstructor().newInstance();
|
||||
FLAG.getMethod("withSuppressAutomations", boolean.class).invoke(flag, options.getSuppressAutomations());
|
||||
FLAG.getMethod("withRunCustomizers", boolean.class).invoke(flag, options.getRunCustomizers());
|
||||
FLAG.getMethod("withPerRecordAudit", boolean.class).invoke(flag, options.getPerRecordAudit());
|
||||
FLAG.getMethod("withPublishEvents", boolean.class).invoke(flag, options.getPublishEvents());
|
||||
sink.accept((ActionFlag) flag);
|
||||
}
|
||||
catch(ReflectiveOperationException | ClassCastException e)
|
||||
{
|
||||
throw (new QException("Could not use the core restore flag: " + e.getMessage(), e));
|
||||
}
|
||||
sink.accept(new RestoreWriteFlag()
|
||||
.withSuppressAutomations(options.getSuppressAutomations())
|
||||
.withRunCustomizers(options.getRunCustomizers())
|
||||
.withPerRecordAudit(options.getPerRecordAudit())
|
||||
.withPublishEvents(options.getPublishEvents()));
|
||||
}
|
||||
|
||||
|
||||
@@ -179,7 +132,7 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public boolean usesCoreRestoreFlag()
|
||||
{
|
||||
return (FLAG != null);
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
@@ -190,24 +143,7 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public boolean canResyncSequences()
|
||||
{
|
||||
return (resyncMethod() != null);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Method resyncMethod()
|
||||
{
|
||||
try
|
||||
{
|
||||
return (QBackendModuleInterface.class.getMethod("resyncKeySequences", QBackendMetaData.class, Collection.class));
|
||||
}
|
||||
catch(NoSuchMethodException | LinkageError e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
@@ -218,12 +154,6 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public List<String> resyncSequences(Collection<QTableMetaData> tables) throws QException
|
||||
{
|
||||
Method method = resyncMethod();
|
||||
if(method == null)
|
||||
{
|
||||
return (List.of());
|
||||
}
|
||||
|
||||
Map<String, List<QTableMetaData>> byBackend = new LinkedHashMap<>();
|
||||
for(QTableMetaData table : tables)
|
||||
{
|
||||
@@ -234,23 +164,14 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
for(Map.Entry<String, List<QTableMetaData>> entry : byBackend.entrySet())
|
||||
{
|
||||
QBackendMetaData backend = QContext.getQInstance().getBackend(entry.getKey());
|
||||
try
|
||||
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
|
||||
for(SequenceResyncResult result : module.resyncKeySequences(backend, entry.getValue()))
|
||||
{
|
||||
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
|
||||
for(Object result : (List<?>) method.invoke(module, backend, entry.getValue()))
|
||||
if(result.getStatus() == SequenceResyncResult.Status.RESYNCED)
|
||||
{
|
||||
Object status = result.getClass().getMethod("getStatus").invoke(result);
|
||||
if("RESYNCED".equals(String.valueOf(status)))
|
||||
{
|
||||
resynced.add(String.valueOf(result.getClass().getMethod("getTableName").invoke(result)));
|
||||
}
|
||||
resynced.add(result.getTableName());
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException e)
|
||||
{
|
||||
LOG.warn("Could not resync key sequences", e, logPair("backend", entry.getKey()));
|
||||
throw (new QException("Could not resync key sequences on backend " + entry.getKey() + ": " + e.getMessage(), e));
|
||||
}
|
||||
}
|
||||
return (resynced);
|
||||
}
|
||||
|
||||
@@ -35,8 +35,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.events.BackupEvents;
|
||||
import com.kingsrook.qbits.backups.codec.TableArtifactWriter;
|
||||
import com.kingsrook.qbits.backups.events.BackupEvents;
|
||||
import com.kingsrook.qbits.backups.files.FileCopier;
|
||||
import com.kingsrook.qbits.backups.files.FileReference;
|
||||
import com.kingsrook.qbits.backups.files.FileReferenceProvider;
|
||||
@@ -398,7 +398,7 @@ public class BackupEngine
|
||||
keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
|
||||
keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
|
||||
manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
|
||||
fingerprint, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
|
||||
fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
|
||||
done++;
|
||||
catalogProgress(result, done, rows, bytesRaw, bytesStored);
|
||||
if(request.getCancelRequested().getAsBoolean())
|
||||
|
||||
@@ -18,30 +18,20 @@ package com.kingsrook.qbits.backups.run;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.List;
|
||||
import java.util.function.Consumer;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.KeysetPager;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reads the rows of one table in primary key order, in keyset pages (WHERE pk >
|
||||
** lastKey ORDER BY pk LIMIT n), so the cost stays flat with depth and a read can
|
||||
** resume from a key. A read raw: hidden fields, passwords and heavy fields are
|
||||
** returned as stored, and possible values are not translated.
|
||||
**
|
||||
** This is an internal stand-in for the QQQ core KeysetPager and QueryInput
|
||||
** withRawRead (core PR feature/keyset-pager-raw-read); the run engine only
|
||||
** depends on this interface, so it is swapped when that ships in the next RC.
|
||||
** Reads rows in primary key order through core KeysetPager. Each bounded page
|
||||
** uses the supplied transaction and returns hidden, password and heavy fields
|
||||
** as stored, without possible-value translation or display generation.
|
||||
*******************************************************************************/
|
||||
public interface TableReader
|
||||
{
|
||||
@@ -81,48 +71,25 @@ public interface TableReader
|
||||
@Override
|
||||
public void read(QTableMetaData table, String filterJson, Serializable startAfterKey, int pageSize, QBackendTransaction transaction, Consumer<QRecord> sink) throws QException
|
||||
{
|
||||
String pk = table.getPrimaryKeyField();
|
||||
Serializable lastKey = startAfterKey;
|
||||
QQueryFilter user = parseFilter(filterJson);
|
||||
|
||||
while(true)
|
||||
KeysetPager pager = KeysetPager.forTable(table.getName())
|
||||
.withFilter(parseFilter(filterJson))
|
||||
.startingAfter(startAfterKey)
|
||||
.withPageSize(pageSize)
|
||||
.withTransaction(transaction)
|
||||
.withQueryInputCustomizer(input ->
|
||||
{
|
||||
input.setShouldOmitHiddenFields(false);
|
||||
input.setShouldMaskPasswords(false);
|
||||
input.setShouldFetchHeavyFields(true);
|
||||
input.setShouldTranslatePossibleValues(false);
|
||||
input.setShouldGenerateDisplayValues(false);
|
||||
input.withPermissionBypass("backups run");
|
||||
});
|
||||
if(!pager.isPrimaryKeyOrderable())
|
||||
{
|
||||
QQueryFilter filter = new QQueryFilter();
|
||||
if(lastKey != null)
|
||||
{
|
||||
filter.addCriteria(new QFilterCriteria(pk, QCriteriaOperator.GREATER_THAN, List.of(lastKey)));
|
||||
}
|
||||
if(user != null)
|
||||
{
|
||||
filter.addSubFilter(user);
|
||||
}
|
||||
filter.withOrderBy(new QFilterOrderBy(pk, true));
|
||||
filter.setLimit(pageSize);
|
||||
|
||||
QueryInput queryInput = new QueryInput(table.getName());
|
||||
queryInput.setFilter(filter);
|
||||
queryInput.setShouldOmitHiddenFields(false);
|
||||
queryInput.setShouldMaskPasswords(false);
|
||||
queryInput.setShouldFetchHeavyFields(true);
|
||||
queryInput.setShouldTranslatePossibleValues(false);
|
||||
queryInput.setShouldGenerateDisplayValues(false);
|
||||
if(transaction != null)
|
||||
{
|
||||
queryInput.setTransaction(transaction);
|
||||
}
|
||||
queryInput.withPermissionBypass("backups run");
|
||||
|
||||
List<QRecord> page = new QueryAction().execute(queryInput).getRecords();
|
||||
for(QRecord record : page)
|
||||
{
|
||||
sink.accept(record);
|
||||
lastKey = record.getValue(pk);
|
||||
}
|
||||
if(page.size() < pageSize)
|
||||
{
|
||||
return;
|
||||
}
|
||||
throw (new QException("Backup requires an orderable primary key on table " + table.getName()));
|
||||
}
|
||||
pager.forEachPage(page -> page.forEach(sink));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,7 +50,7 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
** run is a full-instance, system-session backup (securityScope FULL, requested by
|
||||
** system:scheduler), which is why the plan owner is validated first: a plan whose
|
||||
** owner has gone, or lost backups.admin, is not run silently. It raises
|
||||
** backups.plan.owner_invalid, marks the plan OWNER_INVALID, and stops.
|
||||
** backups.plan.owner-invalid, marks the plan OWNER_INVALID, and stops.
|
||||
**
|
||||
** Everything goes through the run coordinator, so a trigger that finds the plan
|
||||
** already running is skipped visibly, and a full instance queues the run.
|
||||
@@ -141,11 +141,11 @@ public class BackupScheduledRunProcess implements BackendStep
|
||||
if(problem != null)
|
||||
{
|
||||
catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaStatus", "OWNER_INVALID"));
|
||||
BackupsRuntime.getAlertSink().raise("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
|
||||
BackupsRuntime.getAlertSink().raise("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
|
||||
"A scheduled backup did not run: " + problem, Map.of("plan", plan.getValueString("planKey")));
|
||||
throw (new QUserFacingException("The scheduled backup did not run: " + problem + "."));
|
||||
}
|
||||
BackupsRuntime.getAlertSink().resolve("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"));
|
||||
BackupsRuntime.getAlertSink().resolve("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"));
|
||||
|
||||
BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
|
||||
output.addValue("status", result.getStatus());
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.schedule;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
|
||||
import com.kingsrook.qbits.backups.processes.BackupsProcesses;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qbits.backups.verify.BackupVerifier;
|
||||
import com.kingsrook.qbits.backups.verify.VerificationOutcome;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.processes.QRecordListMetaData;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The scheduled verify entry point: the process the verify scheduled job of a
|
||||
** plan (backups_plan.verifyCronExpression) runs. Until this class existed the
|
||||
** verify job was created but pointed at a process nobody had registered, so a
|
||||
** verify cron never verified anything.
|
||||
**
|
||||
** It verifies the plan's latest successful run, through the level the plan names
|
||||
** (verifyLevel, CONTENT when unset), stopping at the first level that does not
|
||||
** pass. Findings, alerts and the plan's last-verified fields are written by the
|
||||
** verifier, exactly as for Verify backup on a run. A plan that has no
|
||||
** successful run yet has nothing to verify, and says so.
|
||||
*******************************************************************************/
|
||||
public class BackupScheduledVerifyProcess implements BackendStep
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(BackupScheduledVerifyProcess.class);
|
||||
|
||||
public static final String NAME = BackupPlanVerifySyncToScheduledJobProcess.VERIFY_PROCESS_NAME;
|
||||
|
||||
public static final String SCHEDULER_USER = "system:scheduler";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The process metadata: not in the nav, a record process on backups_plan.
|
||||
*******************************************************************************/
|
||||
public static QProcessMetaData processMetaData()
|
||||
{
|
||||
QProcessMetaData process = new QProcessMetaData()
|
||||
.withName(NAME)
|
||||
.withLabel("Run scheduled verification")
|
||||
.withTableName(BackupsTables.PLAN)
|
||||
.withIcon(new QIcon().withName("verified"))
|
||||
.withStep(new QBackendStepMetaData()
|
||||
.withName("execute")
|
||||
.withInputData(new QFunctionInputMetaData().withRecordListMetaData(new QRecordListMetaData().withTableName(BackupsTables.PLAN)))
|
||||
.withCode(new QCodeReference(BackupScheduledVerifyProcess.class)));
|
||||
process.setPermissionRules(BackupsTables.permissionRules());
|
||||
return (BackupsProcesses.permission(process, BackupsPermissions.RUN));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The level a plan verifies to: its verifyLevel, CONTENT when unset or unknown.
|
||||
*******************************************************************************/
|
||||
public static String levelOf(QRecord plan)
|
||||
{
|
||||
String level = plan.getValueString("verifyLevel");
|
||||
return (level != null && BackupVerifier.LEVELS.contains(level) ? level : BackupVerifier.CONTENT);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Verify the latest successful run of the plan.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
|
||||
{
|
||||
List<QRecord> records = input.getRecords();
|
||||
if(records == null || records.isEmpty())
|
||||
{
|
||||
throw (new QUserFacingException("No backup plan was selected or found."));
|
||||
}
|
||||
QRecord plan = new BackupCatalog().get(BackupsTables.PLAN, records.get(0).getValueInteger("id"))
|
||||
.orElseThrow(() -> new QUserFacingException("The backup plan no longer exists."));
|
||||
|
||||
if(!Boolean.TRUE.equals(plan.getValueBoolean("isActive")))
|
||||
{
|
||||
LOG.info("A scheduled verification fired for a plan that is not active; nothing was verified", logPair("plan", plan.getValueString("planKey")));
|
||||
output.addValue("status", "INACTIVE");
|
||||
return;
|
||||
}
|
||||
|
||||
Integer runId = plan.getValueInteger("lastSuccessRunId");
|
||||
if(runId == null)
|
||||
{
|
||||
LOG.info("A scheduled verification fired for a plan with no successful backup yet; nothing was verified", logPair("plan", plan.getValueString("planKey")));
|
||||
output.addValue("status", "NO_BACKUP");
|
||||
return;
|
||||
}
|
||||
|
||||
String level = levelOf(plan);
|
||||
List<VerificationOutcome> outcomes = new BackupVerifier().verifyThrough(runId, level, "SCHEDULED", SCHEDULER_USER);
|
||||
VerificationOutcome last = outcomes.get(outcomes.size() - 1);
|
||||
output.addValue("status", last.status());
|
||||
output.addValue("runId", runId);
|
||||
output.addValue("level", level);
|
||||
if(!last.passed())
|
||||
{
|
||||
LOG.info("A scheduled verification did not pass", logPair("plan", plan.getValueString("planKey")), logPair("runId", runId), logPair("status", last.status()));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -33,8 +33,8 @@ import java.util.Optional;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.kingsrook.qbits.backups.BackupsQBitConfig;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
|
||||
import com.kingsrook.qbits.backups.codec.ArtifactTransform;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.codec.JsonlRowCodec;
|
||||
@@ -274,7 +274,7 @@ public class BackupVerifier
|
||||
List<String> excluded = entry.excludedFields() == null ? List.of() : entry.excludedFields().stream().map(BackupManifest.ExcludedField::name).toList();
|
||||
codec = new JsonlRowCodec(table, excluded);
|
||||
String backendType = qInstance.getBackend(table.getBackendName()) == null ? null : qInstance.getBackend(table.getBackendName()).getBackendType();
|
||||
roundTrip = TableFingerprints.of(table, backendType).equals(entry.fingerprint());
|
||||
roundTrip = TableFingerprints.matches(entry, table, backendType);
|
||||
if(!roundTrip)
|
||||
{
|
||||
findings.warn("Table " + entry.name() + " has changed since this backup; rows were parsed but not checked for an exact round trip");
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
bundle: qbit-backups
|
||||
version: 1
|
||||
layer: QBIT
|
||||
# Help for the Backups app and every backups_* table, in the QQQ help-bundle format
|
||||
# (qqq#1198). Each table entry says what the table is for and how its rows get
|
||||
# created, so an empty table explains itself. BackupsHelp applies it until the core
|
||||
# HelpBundles registry ships in a QQQ release (STAND-IN(core=#1198)).
|
||||
# Summaries stay under 120 characters (BackupsTablesGuardrailTest).
|
||||
entries:
|
||||
- key: app:backups;slot:help
|
||||
summary: "Back up, verify and restore the application's data to S3-compatible storage."
|
||||
content: |
|
||||
The Setup card on the home lists what a working backup needs (a plan, a storage bucket, a schedule, verification) with the next step for each. A new installation has no plan: run Create default plan on Backup plans, which asks only for the storage bucket alias, or add a plan with New. Everything else (runs, artifacts, verifications, restores) is written by the backups themselves.
|
||||
|
||||
- key: table:backups_plan;slot:help
|
||||
summary: "A backup plan: what is backed up, where it goes, when it runs, how long it is kept and how it is verified."
|
||||
content: |
|
||||
**How rows get created:** an administrator adds one with New, or runs Create default plan, which adds "Nightly full" (the whole instance, data and audit tables, JSONL and gzip, every night at 02:00 UTC, verified on Sundays) and asks only for the storage bucket alias. The alias names a bucket your application provides to Backups; no key or endpoint is ever stored here. A plan with a cron expression runs on its own once it is active; Run backup now runs it by hand. A few settings (PARQUET, ZSTD, incremental, native dump, read replica) are not available yet and are refused when the plan is saved.
|
||||
|
||||
- key: table:backups_plan_scope;slot:help
|
||||
summary: "One include or exclude rule of a plan: an app, a qbit, a table or a table role."
|
||||
content: |
|
||||
**How rows get created:** an administrator adds them as Scope rules on a plan record. A plan whose scope is Instance backs up everything allowed by its roles and needs no rules; a plan limited to apps, qbits or tables is made of these rules, and an Exclude rule always wins over an Include.
|
||||
|
||||
- key: table:backups_run;slot:help
|
||||
summary: "One execution of a plan, or an ad-hoc export: its status, size, rows, files and storage location."
|
||||
content: |
|
||||
**How rows get created:** only the backups write it. Run backup now, a plan's schedule, a pre-restore safety backup, Resume and Import backups from storage each add a row. Nothing is added by hand; Verify backup, Download, Restore and Prune are actions on a run.
|
||||
|
||||
- key: table:backups_artifact;slot:help
|
||||
summary: "One stored object of a run: a table part, its manifest, checksums or a file index, with its hash and size."
|
||||
content: |
|
||||
**How rows get created:** only the backups write it, one row per object a run stores. They appear as Artifacts on a run record, and Restore this table and Download work from here.
|
||||
|
||||
- key: table:backups_verification;slot:help
|
||||
summary: "The result of checking a backup: that it is in storage, that it decodes, or that it restores into scratch."
|
||||
content: |
|
||||
**How rows get created:** only verification writes it. Run Verify backup on a run (storage, content or test restore), or give a plan a verify cron expression so it verifies its latest backup on a schedule. The newest result also shows on the plan and the home.
|
||||
|
||||
- key: table:backups_restore_run;slot:help
|
||||
summary: "One restore of a backup into the application: mode, id strategy, row and file counts and its safety backup."
|
||||
content: |
|
||||
**How rows get created:** only Restore writes it. Run Restore on a backup run, or Restore this table on an artifact: Review previews what would change before anything is written. A restore that replaces data takes a safety backup first and links it here.
|
||||
|
||||
- key: table:backups_restore_run_table;slot:help
|
||||
summary: "Per-table results of a restore: rows inserted, updated and skipped and how the schema matched."
|
||||
content: |
|
||||
**How rows get created:** only Restore writes it, one row per table restored. They appear as Tables restored on a restore record.
|
||||
|
||||
- key: table:backups_file_object;slot:help
|
||||
summary: "The content-addressed store of copied files, shared by runs, with reference counts and verification."
|
||||
content: |
|
||||
**How rows get created:** only backups write it. When a plan copies the files its rows reference, each distinct file (by SHA-256) is stored once and every run that uses it adds a reference; Prune lowers the counts and removes a file when no run needs it.
|
||||
|
||||
- key: table:backups_id_map;slot:help
|
||||
summary: "Old id to new id pairs recorded while restoring with new ids, so related rows can be rewritten."
|
||||
content: |
|
||||
**How rows get created:** only Restore writes it, and only when a restore uses the Remap new ids strategy. The rows are working data for that restore and are cleared when it finishes, so this table is usually empty.
|
||||
@@ -106,7 +106,7 @@ public final class BackupsTestUtils
|
||||
new BackupsQBitProducer().withQBitConfig(config).withNamespace("backups").produce(qInstance);
|
||||
|
||||
new QInstanceValidator().validate(qInstance);
|
||||
QContext.init(qInstance, new QSession());
|
||||
QContext.init(qInstance, new QSession().withPermissions("backups.restore"));
|
||||
return (qInstance);
|
||||
}
|
||||
|
||||
|
||||
@@ -18,16 +18,24 @@ package com.kingsrook.qbits.backups.alerts;
|
||||
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.VerifyTestSupport;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlertTypeMetaData;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlerts;
|
||||
import com.kingsrook.qbits.backups.sla.SlaChecker;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -35,14 +43,10 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Alerts qbit adapter, bound reflectively. The SPI is not in QQQ 4.1.0-RC.4,
|
||||
** so these tests run it against a test double with the same shape as the real
|
||||
** com.kingsrook.qqq.backend.core.alerts package (QRun-IO/qqq PR #1108).
|
||||
** The backup adapter against the released core alerts API and a registered sink.
|
||||
*******************************************************************************/
|
||||
class AlertsQBitAlertSinkTest
|
||||
{
|
||||
private static final String FAKE_API = "com.kingsrook.qbits.backups.alerts.fakeapi";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
@@ -52,9 +56,11 @@ class AlertsQBitAlertSinkTest
|
||||
void setUp() throws Exception
|
||||
{
|
||||
VerifyTestSupport.setUp(3);
|
||||
QAlerts.reset();
|
||||
QAlertTypeMetaData.ADDED.clear();
|
||||
AlertsInstanceMetaData.installed = true;
|
||||
RecordingSink.raised.clear();
|
||||
RecordingSink.resolved.clear();
|
||||
RecordingSink.heartbeats.clear();
|
||||
RecordingSink.failWith = null;
|
||||
AlertsInstanceMetaData.ofOrWithNew(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
|
||||
}
|
||||
|
||||
|
||||
@@ -65,8 +71,8 @@ class AlertsQBitAlertSinkTest
|
||||
@AfterEach
|
||||
void tearDown()
|
||||
{
|
||||
AlertsInstanceMetaData.installed = false;
|
||||
QAlerts.reset();
|
||||
BackupsRuntime.reset();
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
|
||||
@@ -78,7 +84,7 @@ class AlertsQBitAlertSinkTest
|
||||
@Test
|
||||
void testRaiseResolveHeartbeat()
|
||||
{
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
Map<String, String> detail = new LinkedHashMap<>();
|
||||
detail.put("plan", "nightly");
|
||||
detail.put("slaHours", "26");
|
||||
@@ -87,60 +93,90 @@ class AlertsQBitAlertSinkTest
|
||||
sink.resolve("backups.plan.missed", "backups.plan.missed:nightly");
|
||||
sink.heartbeat("backups.plan:nightly", 93600, 0);
|
||||
|
||||
assertThat(QAlerts.RAISED).hasSize(1);
|
||||
var signal = QAlerts.RAISED.get(0);
|
||||
assertThat(signal.alertTypeName).isEqualTo("backups.plan.missed");
|
||||
assertThat(signal.dedupKey).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(signal.severity.name()).isEqualTo("CRITICAL");
|
||||
assertThat(signal.title).isEqualTo("A backup did not run in time");
|
||||
assertThat(signal.summary).isEqualTo("plan=nightly, slaHours=26");
|
||||
assertThat(signal.attributes).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
|
||||
assertThat(QAlerts.RESOLVED).containsExactly("backups.plan.missed|backups.plan.missed:nightly|Cleared by the Backups qbit");
|
||||
assertThat(QAlerts.HEARTBEATS).containsExactly("backups.plan:nightly");
|
||||
assertThat(RecordingSink.raised).hasSize(1);
|
||||
var signal = RecordingSink.raised.get(0);
|
||||
assertThat(signal.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(signal.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(signal.getSeverity().name()).isEqualTo("CRITICAL");
|
||||
assertThat(signal.getTitle()).isEqualTo("A backup did not run in time");
|
||||
assertThat(signal.getSummary()).isEqualTo("plan=nightly, slaHours=26");
|
||||
assertThat(signal.getAttributes()).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
|
||||
assertThat(RecordingSink.resolved).singleElement().satisfies(resolution ->
|
||||
{
|
||||
assertThat(resolution.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(resolution.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(resolution.getNote()).isEqualTo("Cleared by the Backups qbit");
|
||||
});
|
||||
assertThat(RecordingSink.heartbeats).containsExactly("backups.plan:nightly");
|
||||
|
||||
sink.raise("backups.storage.debris", "k", BackupAlertSink.Severity.INFO, "no detail", null);
|
||||
assertThat(QAlerts.RAISED.get(1).summary).isNull();
|
||||
assertThat(RecordingSink.raised.get(1).getSummary()).isNull();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Without the Alerts qbit installed (no alerts meta-data on the instance) every
|
||||
** Without a configured sink, even with alert type metadata present, every
|
||||
** call is a no-op.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testNoOpWithoutTheAlertsQBit()
|
||||
{
|
||||
AlertsInstanceMetaData.installed = false;
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
|
||||
sink.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
sink.resolve("t", "k");
|
||||
sink.heartbeat("m", 1, 1);
|
||||
assertThat(QAlerts.RAISED).isEmpty();
|
||||
assertThat(QAlerts.RESOLVED).isEmpty();
|
||||
assertThat(QAlerts.HEARTBEATS).isEmpty();
|
||||
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(null);
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
long failures = QAlerts.getProducerFailureCount();
|
||||
sink.raise(null, "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
sink.resolve(null, "k");
|
||||
sink.heartbeat(null, 1, 1);
|
||||
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures);
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
assertThat(RecordingSink.resolved).isEmpty();
|
||||
assertThat(RecordingSink.heartbeats).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A broken or missing SPI never throws into the backup.
|
||||
** The same adapter follows the current instance and a sink installed later.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLateInstallationAndInstanceSwitch()
|
||||
{
|
||||
var original = QContext.getQInstance();
|
||||
QContext.setQInstance(new QInstance());
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
assertThat(sink.isAlertsQBitInstalled()).isFalse();
|
||||
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
sink.registerAlertTypes(QContext.getQInstance());
|
||||
assertThat(sink.isAlertsQBitInstalled()).isFalse();
|
||||
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
|
||||
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
assertThat(RecordingSink.raised).hasSize(1);
|
||||
assertThat(RecordingSink.raised.get(0).getSummary()).isNull();
|
||||
QContext.setQInstance(original);
|
||||
assertThat(sink.isAlertsQBitInstalled()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Sink and adapter failures never throw into the backup.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFailuresAreSwallowed()
|
||||
{
|
||||
AlertsQBitAlertSink missing = new AlertsQBitAlertSink("no.such.package");
|
||||
missing.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
missing.resolve("t", "k");
|
||||
missing.heartbeat("m", 1, 1);
|
||||
assertThat(missing.registerAlertTypes(QContext.getQInstance())).isZero();
|
||||
|
||||
QAlerts.failWith = new IllegalStateException("alerts down");
|
||||
AlertsQBitAlertSink broken = new AlertsQBitAlertSink(FAKE_API);
|
||||
long failures = QAlerts.getProducerFailureCount();
|
||||
RecordingSink.failWith = new IllegalStateException("alerts down");
|
||||
AlertsQBitAlertSink broken = new AlertsQBitAlertSink();
|
||||
broken.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
broken.resolve("t", "k");
|
||||
broken.heartbeat("m", 1, 1);
|
||||
assertThat(QAlerts.RAISED).isEmpty();
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures + 3);
|
||||
broken.raise("t", "k", null, "bad severity", Map.of());
|
||||
broken.raise("t", "k", null, "bad severity again", Map.of());
|
||||
assertThat(broken.registerAlertTypes(null)).isZero();
|
||||
}
|
||||
|
||||
|
||||
@@ -149,41 +185,41 @@ class AlertsQBitAlertSinkTest
|
||||
** The backup alert types are declared with the Alerts qbit.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testRegisterAlertTypes()
|
||||
void testRegisterAlertTypes() throws Exception
|
||||
{
|
||||
int added = new AlertsQBitAlertSink(FAKE_API).registerAlertTypes(QContext.getQInstance());
|
||||
QInstance instance = new QInstance();
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
int added = sink.registerAlertTypes(instance);
|
||||
var types = new ArrayList<>(AlertsInstanceMetaData.of(instance).getAlertTypes().values());
|
||||
assertThat(added).isEqualTo(BackupAlerts.alertTypes().size()).isEqualTo(9);
|
||||
assertThat(QAlertTypeMetaData.ADDED).extracting(t -> t.name).containsExactly(
|
||||
"backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing", "backups.verify.failed",
|
||||
"backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(QAlertTypeMetaData.ADDED).allSatisfy(t ->
|
||||
assertThat(types).extracting(t -> t.getName()).containsExactly(
|
||||
"backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing", "backups.verify.failed",
|
||||
"backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(types).allSatisfy(t ->
|
||||
{
|
||||
assertThat(t.groupName).isEqualTo("Backups");
|
||||
assertThat(t.howToFix).isNotBlank();
|
||||
assertThat(t.label).isNotBlank();
|
||||
assertThat(t.getGroupName()).isEqualTo("Backups");
|
||||
assertThat(t.getHowToFix()).isNotBlank();
|
||||
assertThat(t.getLabel()).isNotBlank();
|
||||
});
|
||||
assertThat(QAlertTypeMetaData.ADDED.get(2).defaultSeverity.name()).isEqualTo("CRITICAL");
|
||||
assertThat(types.get(2).getDefaultSeverity().name()).isEqualTo("CRITICAL");
|
||||
assertThat(sink.registerAlertTypes(instance)).isZero();
|
||||
new QInstanceValidator().validate(QContext.getQInstance());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Detection: the SPI is present or not by class name; the real SPI is not in
|
||||
** RC.4, so nothing is bound by default there.
|
||||
** Default binding preserves a host-provided backup sink.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testDetectionAndBinding()
|
||||
{
|
||||
assertThat(AlertsQBitAlertSink.isPresent(FAKE_API)).isTrue();
|
||||
assertThat(AlertsQBitAlertSink.isPresent("no.such.package")).isFalse();
|
||||
|
||||
boolean realSpiPresent = AlertsQBitAlertSink.isPresent(AlertsQBitAlertSink.DEFAULT_API_PACKAGE);
|
||||
assertThat(AlertsQBitAlertSink.bindIfPresent()).isEqualTo(realSpiPresent);
|
||||
if(realSpiPresent)
|
||||
{
|
||||
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
|
||||
assertThat(AlertsQBitAlertSink.bindIfPresent()).as("an already bound sink is not replaced").isFalse();
|
||||
}
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
|
||||
assertThat(BackupsRuntime.getAlertSink()).isSameAs(VerifyTestSupport.sink);
|
||||
BackupsRuntime.setAlertSink(null);
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isTrue();
|
||||
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
|
||||
}
|
||||
|
||||
|
||||
@@ -194,22 +230,80 @@ class AlertsQBitAlertSinkTest
|
||||
@Test
|
||||
void testSlaAlertsThroughTheAdapter() throws Exception
|
||||
{
|
||||
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink(FAKE_API));
|
||||
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink());
|
||||
QRecord plan = VerifyTestSupport.plan(Map.of());
|
||||
VerifyTestSupport.backup(plan);
|
||||
assertThat(QAlerts.HEARTBEATS).contains("backups.plan:nightly");
|
||||
assertThat(RecordingSink.heartbeats).contains("backups.plan:nightly");
|
||||
|
||||
VerifyTestSupport.clock.advance(Duration.ofHours(30));
|
||||
new SlaChecker().check(VerifyTestSupport.reload(plan));
|
||||
assertThat(QAlerts.RAISED).anySatisfy(s ->
|
||||
assertThat(RecordingSink.raised).anySatisfy(s ->
|
||||
{
|
||||
assertThat(s.alertTypeName).isEqualTo("backups.plan.missed");
|
||||
assertThat(s.dedupKey).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(s.severity.name()).isEqualTo("ERROR");
|
||||
assertThat(s.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(s.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(s.getSeverity().name()).isEqualTo("ERROR");
|
||||
});
|
||||
|
||||
VerifyTestSupport.backup(plan);
|
||||
new SlaChecker().check(VerifyTestSupport.reload(plan));
|
||||
assertThat(QAlerts.RESOLVED).anyMatch(r -> r.startsWith("backups.plan.missed|backups.plan.missed:nightly|"));
|
||||
assertThat(RecordingSink.resolved).anyMatch(r -> r.getAlertTypeName().equals("backups.plan.missed") && r.getDedupKey().equals("backups.plan.missed:nightly"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Sink loaded by the real QCodeLoader through AlertsInstanceMetaData.
|
||||
*******************************************************************************/
|
||||
public static class RecordingSink implements AlertSinkInterface
|
||||
{
|
||||
static final List<AlertSignal> raised = new ArrayList<>();
|
||||
static final List<AlertResolution> resolved = new ArrayList<>();
|
||||
static final List<String> heartbeats = new ArrayList<>();
|
||||
static RuntimeException failWith;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void raise(AlertSignal signal)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
raised.add(signal);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void resolve(AlertResolution resolution)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
resolved.add(resolution);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void heartbeat(String heartbeatKey, Instant at)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
heartbeats.add(heartbeatKey);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -70,8 +70,8 @@ class BackupAlertsTest
|
||||
void testTheSpecTable()
|
||||
{
|
||||
Set<String> names = BackupAlerts.alertTypes().stream().map(BackupAlerts.AlertType::name).collect(Collectors.toSet());
|
||||
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing",
|
||||
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing",
|
||||
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(BackupAlerts.alertTypes()).allSatisfy(t -> assertThat(t.howToFix()).isNotBlank());
|
||||
assertThat(BackupAlerts.alertTypes().stream().filter(t -> t.name().equals(BackupAlerts.RUN_MISSING)).findFirst().orElseThrow().defaultSeverity())
|
||||
.isEqualTo(BackupAlertSink.Severity.CRITICAL);
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI (QRun-IO/qqq PR #1108): the same shape as
|
||||
** com.kingsrook.qqq.backend.core.alerts.AlertSeverity.
|
||||
*******************************************************************************/
|
||||
public enum AlertSeverity
|
||||
{
|
||||
INFO,
|
||||
WARNING,
|
||||
ERROR,
|
||||
CRITICAL
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI AlertSignal (only what the adapter uses).
|
||||
*******************************************************************************/
|
||||
public class AlertSignal
|
||||
{
|
||||
public final String alertTypeName;
|
||||
public String dedupKey;
|
||||
public AlertSeverity severity;
|
||||
public String title;
|
||||
public String summary;
|
||||
public final Map<String, String> attributes = new LinkedHashMap<>();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor.
|
||||
*******************************************************************************/
|
||||
public AlertSignal(String alertTypeName)
|
||||
{
|
||||
this.alertTypeName = alertTypeName;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setDedupKey(String dedupKey)
|
||||
{
|
||||
this.dedupKey = dedupKey;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setSeverity(AlertSeverity severity)
|
||||
{
|
||||
this.severity = severity;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setTitle(String title)
|
||||
{
|
||||
this.title = title;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setSummary(String summary)
|
||||
{
|
||||
this.summary = summary;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent attribute.
|
||||
*******************************************************************************/
|
||||
public AlertSignal withAttribute(String key, String value)
|
||||
{
|
||||
attributes.put(key, value);
|
||||
return (this);
|
||||
}
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI AlertsInstanceMetaData: of(instance) is
|
||||
** non-null only when a test says the Alerts qbit is installed.
|
||||
*******************************************************************************/
|
||||
public class AlertsInstanceMetaData
|
||||
{
|
||||
public static boolean installed = false;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The alerts meta-data of an instance, or null.
|
||||
*******************************************************************************/
|
||||
public static AlertsInstanceMetaData of(QInstance qInstance)
|
||||
{
|
||||
return (installed ? new AlertsInstanceMetaData() : null);
|
||||
}
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI QAlertTypeMetaData: records the types
|
||||
** added to an instance.
|
||||
*******************************************************************************/
|
||||
public class QAlertTypeMetaData
|
||||
{
|
||||
public static final List<QAlertTypeMetaData> ADDED = new ArrayList<>();
|
||||
|
||||
public String name;
|
||||
public String label;
|
||||
public String groupName;
|
||||
public String howToFix;
|
||||
public AlertSeverity defaultSeverity;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withName(String name)
|
||||
{
|
||||
this.name = name;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withLabel(String label)
|
||||
{
|
||||
this.label = label;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withGroupName(String groupName)
|
||||
{
|
||||
this.groupName = groupName;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withHowToFix(String howToFix)
|
||||
{
|
||||
this.howToFix = howToFix;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withDefaultSeverity(AlertSeverity defaultSeverity)
|
||||
{
|
||||
this.defaultSeverity = defaultSeverity;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Add to the instance (recorded).
|
||||
*******************************************************************************/
|
||||
public void addSelfToInstance(QInstance qInstance)
|
||||
{
|
||||
ADDED.add(this);
|
||||
}
|
||||
}
|
||||
@@ -1,104 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI QAlerts: static calls that record what
|
||||
** they were given.
|
||||
*******************************************************************************/
|
||||
public final class QAlerts
|
||||
{
|
||||
public static final List<AlertSignal> RAISED = new ArrayList<>();
|
||||
public static final List<String> RESOLVED = new ArrayList<>();
|
||||
public static final List<String> HEARTBEATS = new ArrayList<>();
|
||||
|
||||
/** When set, every call throws it (the SPI is meant not to, but the adapter must survive). */
|
||||
public static RuntimeException failWith;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private QAlerts()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Forget everything.
|
||||
*******************************************************************************/
|
||||
public static void reset()
|
||||
{
|
||||
RAISED.clear();
|
||||
RESOLVED.clear();
|
||||
HEARTBEATS.clear();
|
||||
failWith = null;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Raise.
|
||||
*******************************************************************************/
|
||||
public static void raise(AlertSignal signal)
|
||||
{
|
||||
maybeFail();
|
||||
RAISED.add(signal);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve.
|
||||
*******************************************************************************/
|
||||
public static void resolve(String alertTypeName, String dedupKey, String note)
|
||||
{
|
||||
maybeFail();
|
||||
RESOLVED.add(alertTypeName + "|" + dedupKey + "|" + note);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Heartbeat.
|
||||
*******************************************************************************/
|
||||
public static void heartbeat(String heartbeatKey)
|
||||
{
|
||||
maybeFail();
|
||||
HEARTBEATS.add(heartbeatKey);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static void maybeFail()
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.app;
|
||||
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.app.BackupsSetupStatus.Item;
|
||||
import com.kingsrook.qbits.backups.app.BackupsSetupStatus.State;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Setup card of the Backups home: with no plan it tells the user to run Create
|
||||
** default plan; with a plan it checks the storage alias, the schedule, the
|
||||
** verification and the prune bucket, and links each to its table.
|
||||
*******************************************************************************/
|
||||
class BackupsSetupStatusTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Item item(List<Item> items, String label)
|
||||
{
|
||||
return (items.stream().filter(i -> i.label().equals(label)).findFirst().orElseThrow());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static QRecord plan(Map<String, Object> values)
|
||||
{
|
||||
QRecord plan = new QRecord().withValue("name", "Nightly").withValue("isActive", true).withValue("scopeType", "INSTANCE").withValue("storageBucketAlias", "backups");
|
||||
values.forEach((k, v) -> plan.setValue(k, (java.io.Serializable) v));
|
||||
return (plan);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** No plans: the first step is the Create default plan action.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void noPlansPointsAtTheDefaultAction()
|
||||
{
|
||||
List<Item> items = BackupsSetupStatus.load(List.of());
|
||||
assertThat(item(items, "Backup plans").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Backup plans").nextStep()).contains("Create default plan");
|
||||
assertThat(item(items, "Plan scope rules").state()).isEqualTo(State.OPTIONAL);
|
||||
|
||||
String html = BackupsHomeView.render(new BackupsHomeData(List.of(), List.of(), List.of(), List.of(), List.of(), Instant.now()), "/app");
|
||||
assertThat(html).contains("Setup").contains("1 item needs attention").contains("Create default plan").contains("href=\"/app/backups_plan\"");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A bare plan: storage resolves, but no schedule, no verification and no prune bucket.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aBarePlanNeedsASchedule()
|
||||
{
|
||||
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of())), true);
|
||||
assertThat(item(items, "Backup plans").state()).isEqualTo(State.OK);
|
||||
assertThat(item(items, "Storage").state()).isEqualTo(State.OK);
|
||||
assertThat(item(items, "Schedule").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Verification").state()).isEqualTo(State.OPTIONAL);
|
||||
assertThat(item(items, "Prune bucket").state()).isEqualTo(State.DEFAULT);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A complete plan: everything is in place.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aCompletePlanIsSetUp()
|
||||
{
|
||||
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"))), true);
|
||||
assertThat(items).extracting(Item::state).doesNotContain(State.TODO, State.DEFAULT);
|
||||
assertThat(items).extracting(Item::tableName).contains(BackupsTables.PLAN, BackupsTables.PLAN_SCOPE, BackupsTables.VERIFICATION);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Without a scheduler the crons never fire: Schedule and Verification are TODO, and a
|
||||
** plan whose isActive is null is not counted active (as at runtime).
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void noSchedulerMeansCronsNeverFire()
|
||||
{
|
||||
QRecord complete = plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"));
|
||||
List<Item> items = BackupsSetupStatus.load(List.of(complete), false);
|
||||
assertThat(item(items, "Schedule").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Schedule").detail()).contains("No scheduler is configured");
|
||||
assertThat(item(items, "Verification").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Verification").nextStep()).contains("schedulerName");
|
||||
|
||||
QRecord unset = new QRecord().withValue("name", "N").withValue("storageBucketAlias", "backups");
|
||||
assertThat(item(BackupsSetupStatus.load(List.of(unset), true), "Backup plans").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(BackupsSetupStatus.load(List.of(complete)), "Schedule").state()).as("the test instance has no scheduler").isEqualTo(State.TODO);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A host that ticks plans itself is not told it needs a scheduler; a verify cron its
|
||||
** tick does not run is called out, and one it does run is fine.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void hostSchedulerCountsButUnrunVerifyIsCalledOut()
|
||||
{
|
||||
QRecord complete = plan(Map.of("cronExpression", DefaultPlanProcess.CRON, "verifyCronExpression", DefaultPlanProcess.VERIFY_CRON, "pruneBucketAlias", "backups"));
|
||||
List<Item> items = BackupsSetupStatus.load(List.of(complete), false, "the website ticker", false);
|
||||
assertThat(item(items, "Schedule").state()).isEqualTo(State.OK);
|
||||
assertThat(item(items, "Schedule").detail()).contains("the website ticker");
|
||||
assertThat(item(items, "Verification").state()).isEqualTo(State.OPTIONAL);
|
||||
assertThat(item(items, "Verification").detail()).contains("Not run by the website ticker");
|
||||
|
||||
assertThat(item(BackupsSetupStatus.load(List.of(complete), false, "the website ticker", true), "Verification").state()).isEqualTo(State.OK);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An alias that does not resolve, an inactive plan and a scoped plan are called out.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void problemsAreCalledOut()
|
||||
{
|
||||
List<Item> items = BackupsSetupStatus.load(List.of(plan(Map.of("storageBucketAlias", "nowhere", "isActive", false, "scopeType", "TABLES"))), true);
|
||||
assertThat(item(items, "Backup plans").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Storage").state()).isEqualTo(State.TODO);
|
||||
assertThat(item(items, "Storage").detail()).contains("nowhere");
|
||||
assertThat(item(items, "Plan scope rules").detail()).contains("limited by scope rules");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fingerprint algorithm dispatch is explicit; a marker cannot be ignored just
|
||||
** because its hash happens to match a different supported algorithm.
|
||||
*******************************************************************************/
|
||||
class TableFingerprintsTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** A frozen old hash matches only the legacy algorithm, and new hashes only
|
||||
** match the declared core algorithm. Cosmetic changes preserve both.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testAlgorithmDispatch() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
var table = QContext.getQInstance().getTable("blogPost");
|
||||
ObjectMapper mapper = new ObjectMapper();
|
||||
ObjectNode entry;
|
||||
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
|
||||
{
|
||||
entry = (ObjectNode) mapper.readTree(in).get("blogPost");
|
||||
}
|
||||
String oldHash = entry.get("fingerprint").asText();
|
||||
String coreHash = TableFingerprints.of(table, "memory");
|
||||
assertThat(coreHash).isNotEqualTo(oldHash);
|
||||
table.setLabel("Cosmetic label");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", TableFingerprints.LEGACY_ALGORITHM);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", TableFingerprints.CORE_ALGORITHM);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
entry.put("fingerprint", coreHash);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", "unknown-v2");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
entry.remove("fingerprint");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.restore;
|
||||
|
||||
|
||||
import java.util.Map;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.manifest.BackupManifest;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qbits.backups.run.BackupRunResult;
|
||||
import com.kingsrook.qbits.backups.verify.BackupVerifier;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise pre-RC.6 manifest fingerprints through verification and restore.
|
||||
** The schema fixture was frozen from the original stand-in, not recomputed
|
||||
** with the implementation being tested.
|
||||
*******************************************************************************/
|
||||
class LegacyManifestRestoreTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Load the blog and grant the permission required by the core restore flag.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupsTestUtils.loadBlog(30);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build an old-format manifest around real backup parts and re-seal its hash.
|
||||
*******************************************************************************/
|
||||
private BackupRunResult legacyBackup() throws Exception
|
||||
{
|
||||
BackupRunResult run = RestoreTestSupport.backup(Map.of());
|
||||
ObjectMapper mapper = new ObjectMapper();
|
||||
ObjectNode manifest = (ObjectNode) mapper.readTree(BackupsTestUtils.BACKUPS.getBytes(run.getManifestKey()));
|
||||
JsonNode legacy;
|
||||
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
|
||||
{
|
||||
legacy = mapper.readTree(in);
|
||||
}
|
||||
for(JsonNode table : manifest.get("tables"))
|
||||
{
|
||||
ObjectNode entry = (ObjectNode) table;
|
||||
JsonNode old = legacy.get(entry.get("name").asText());
|
||||
entry.remove("fingerprintAlgorithm");
|
||||
entry.set("schema", old.get("schema"));
|
||||
entry.set("fingerprint", old.get("fingerprint"));
|
||||
}
|
||||
byte[] bytes = mapper.writeValueAsBytes(manifest);
|
||||
BackupsTestUtils.BACKUPS.putBytes(run.getManifestKey(), bytes);
|
||||
new BackupCatalog().update(BackupsTables.RUN, run.getRunId(), Map.of("manifestSha256", HashingOutputStream.sha256Hex(bytes)));
|
||||
assertThat(BackupManifest.read(bytes).table("blogPost").fingerprintAlgorithm()).isNull();
|
||||
return (run);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Old hashes still receive exact codec verification and restore all data,
|
||||
** including original timestamps, after deletes, edits and additions.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLegacyManifestVerifiesAndRestores() throws Exception
|
||||
{
|
||||
String before = RestoreTestSupport.contentHash("blogPost");
|
||||
BackupRunResult run = legacyBackup();
|
||||
var verified = new BackupVerifier().verify(run.getRunId(), BackupVerifier.CONTENT, "MANUAL", "tester");
|
||||
assertThat(verified.status()).isEqualTo("PASSED");
|
||||
assertThat(verified.messages()).noneMatch(message -> message.contains("has changed since"));
|
||||
RestoreTestSupport.mutate();
|
||||
RestoreOutcome outcome = new RestoreEngine().execute(RestoreTestSupport.request(run, RestoreRequest.MODE_REPLACE, "blogPost").withConfirmationText("blogPost"));
|
||||
assertThat(outcome.getErrors()).isEmpty();
|
||||
assertThat(outcome.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MATCH");
|
||||
assertThat(RestoreTestSupport.contentHash("blogPost")).isEqualTo(before);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Legacy field maps and core field lists also compare correctly when hashes
|
||||
** differ: optional additions are compatible and type changes are rejected.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLegacySchemaChangesAreAnalyzed() throws Exception
|
||||
{
|
||||
BackupRunResult run = legacyBackup();
|
||||
QContext.getQInstance().getTable("author").withField(new QFieldMetaData("nickname", QFieldType.STRING));
|
||||
QContext.getQInstance().getTable("blogPost").getField("title").setType(QFieldType.INTEGER);
|
||||
RestoreOutcome review = new RestoreEngine().review(RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "author", "blogPost"));
|
||||
assertThat(review.getTables().get("author").getSchemaStatus()).isEqualTo("COMPATIBLE");
|
||||
assertThat(review.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MISMATCH");
|
||||
}
|
||||
}
|
||||
@@ -187,4 +187,22 @@ class RestoreSafetyTest
|
||||
QContext.setQSession(new QSession().withPermissions("backups.view", "backups.restore"));
|
||||
assertThat(new RestoreEngine().plan(request).errors()).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The core restore permission is required even in anonymous demo mode, and
|
||||
** Review reports it before a restore can create a run or write any data.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testCoreRestorePermissionIsRequiredWhenAnonymous() throws Exception
|
||||
{
|
||||
BackupRunResult run = RestoreTestSupport.backup(Map.of());
|
||||
QContext.setQSession(new QSession());
|
||||
RestoreRequest request = RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "blogPost");
|
||||
assertThat(new RestoreEngine().plan(request).errors()).anyMatch(e -> e.contains("backups.restore"));
|
||||
assertThatThrownBy(() -> new RestoreEngine().execute(request)).hasMessageContaining("backups.restore");
|
||||
assertThatThrownBy(() -> new StandardRestoreWriter().insert(QContext.getQInstance().getTable("blogPost"),
|
||||
List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", 99)), request, null)).hasMessageContaining("backups.restore");
|
||||
}
|
||||
|
||||
}
|
||||
@@ -109,7 +109,9 @@ class BackupEngineTest
|
||||
BackupManifest.TableEntry posts = manifest.table("blogPost");
|
||||
assertThat(posts.rowCount()).isEqualTo(7);
|
||||
assertThat(posts.excludedFields()).extracting(BackupManifest.ExcludedField::name).containsExactly("secretNote");
|
||||
assertThat(posts.fingerprint()).startsWith("sha256:");
|
||||
assertThat(posts.fingerprintAlgorithm()).isEqualTo(com.kingsrook.qbits.backups.manifest.TableFingerprints.CORE_ALGORITHM);
|
||||
assertThat(posts.fingerprint()).isEqualTo("sha256:" + com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getHash());
|
||||
assertThat(posts.schema()).isEqualTo(com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getDescriptor());
|
||||
|
||||
BackupManifest.Part part = posts.parts().get(0);
|
||||
byte[] stored = BackupsTestUtils.BACKUPS.getBytes(result.getRunPrefix() + part.key());
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.run;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.consistentread.ConsistentReadInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Integration coverage for the core pager adapter's ordering, bounded fetches,
|
||||
** raw fields and transaction lifetime.
|
||||
*******************************************************************************/
|
||||
class TableReaderTest
|
||||
{
|
||||
private QTableMetaData table;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Seed more records than fit in a page.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupsTestUtils.loadBlog(7);
|
||||
table = QContext.getQInstance().getTable("blogPost");
|
||||
table.getField("body").setIsHeavy(true);
|
||||
table.getField("secretNote").setIsHidden(true);
|
||||
table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(CaptureQueries.class));
|
||||
CaptureQueries.inputs.clear();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An OR filter and a resume cursor are ANDed, with ascending keys even when
|
||||
** the caller requests descending order. Every query has the requested bound.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFilterResumeOrderingAndRawFields() throws Exception
|
||||
{
|
||||
String filter = "{\"booleanOperator\":\"OR\",\"criteria\":[{\"fieldName\":\"id\",\"operator\":\"LESS_THAN_OR_EQUALS\",\"values\":[3]},{\"fieldName\":\"id\",\"operator\":\"GREATER_THAN\",\"values\":[5]}],\"orderBys\":[{\"fieldName\":\"id\",\"isAscending\":false}]}";
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, filter, 2, 2, null, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(3, 6, 7);
|
||||
assertThat(rows.get(0).getValueString("body")).isEqualTo("Body of 3, with a comma");
|
||||
assertThat(rows.get(0).getValueString("secretNote")).isEqualTo("do-not-leak-3");
|
||||
assertThat(CaptureQueries.inputs).hasSize(2).allSatisfy(input ->
|
||||
{
|
||||
assertThat(input.getFilter().getLimit()).isEqualTo(2);
|
||||
assertThat(input.getFilter().getSkip()).isNull();
|
||||
assertThat(input.getShouldOmitHiddenFields()).isFalse();
|
||||
assertThat(input.getShouldMaskPasswords()).isFalse();
|
||||
assertThat(input.getShouldFetchHeavyFields()).isTrue();
|
||||
assertThat(input.getShouldTranslatePossibleValues()).isFalse();
|
||||
assertThat(input.getShouldGenerateDisplayValues()).isFalse();
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Changes made after a snapshot was opened are invisible on every page. The
|
||||
** reader leaves the transaction open, with its commit callbacks unexecuted.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testConsistentSnapshotIsRetainedAcrossPages() throws Exception
|
||||
{
|
||||
List<Boolean> commits = new ArrayList<>();
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openConsistentReadFor(new ConsistentReadInput().withBackendName(table.getBackendName())))
|
||||
{
|
||||
transaction.addAfterCommitCallback(() -> commits.add(true));
|
||||
new UpdateAction().execute(new UpdateInput("blogPost").withRecords(List.of(new QRecord().withValue("id", 7).withValue("title", "Changed after snapshot"))));
|
||||
CaptureQueries.inputs.clear();
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, null, null, 2, transaction, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(1, 2, 3, 4, 5, 6, 7);
|
||||
assertThat(rows.get(6).getValueString("title")).isEqualTo("Post 7");
|
||||
assertThat(CaptureQueries.inputs).hasSize(4).allSatisfy(input -> assertThat(input.getTransaction()).isSameAs(transaction));
|
||||
assertThat(commits).isEmpty();
|
||||
transaction.commit();
|
||||
assertThat(commits).containsExactly(true);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exact page multiples, empty scans, invalid bounds and unsupported key types
|
||||
** must terminate or fail explicitly rather than drop rows or stream unordered.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testBoundariesAndInvalidPaging() throws Exception
|
||||
{
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, null, 3, 2, null, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(4, 5, 6, 7);
|
||||
rows.clear();
|
||||
new TableReader.Keyset().read(table, null, 7, 2, null, rows::add);
|
||||
assertThat(rows).isEmpty();
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 0, null, rows::add)).hasMessageContaining("Page size");
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{\"limit\":1}", null, 2, null, rows::add)).hasMessageContaining("skip or limit");
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{", null, 2, null, rows::add)).hasMessageContaining("scope filter");
|
||||
table.getField("id").setType(QFieldType.BOOLEAN);
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 2, null, rows::add)).hasMessageContaining("orderable primary key");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Observe actual QueryAction inputs without replacing the backend or pager.
|
||||
*******************************************************************************/
|
||||
public static class CaptureQueries implements TableCustomizerInterface
|
||||
{
|
||||
static final List<QueryInput> inputs = new ArrayList<>();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Record each page and preserve the result unchanged.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records)
|
||||
{
|
||||
inputs.add((QueryInput) input);
|
||||
return (records);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -306,7 +306,7 @@ class BackupPlanScheduleTest
|
||||
assertThat(catalog.where(BackupsTables.RUN, "backupPlanId", planId)).isEmpty();
|
||||
assertThat(BackupsTestUtils.BACKUPS.list("")).isEmpty();
|
||||
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValueString("slaStatus")).isEqualTo("OWNER_INVALID");
|
||||
assertThat(alerts.raised).containsExactly("backups.plan.owner_invalid|backups.plan.owner_invalid:nightly");
|
||||
assertThat(alerts.raised).containsExactly("backups.plan.owner-invalid|backups.plan.owner-invalid:nightly");
|
||||
|
||||
BackupScheduledRunProcess.setOwnerHasAdminCheck(null);
|
||||
Integer noOwner = insertPlan(Map.of("name", "Ownerless", "planKey", "ownerless", "isActive", true));
|
||||
|
||||
@@ -180,12 +180,12 @@ class SlaCheckerTest
|
||||
|
||||
clock.advance(Duration.ofMinutes(55));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
BackupAlertsAssert.assertOpen("backups.run.missing_in_storage:nightly", BackupAlertSink.Severity.CRITICAL);
|
||||
BackupAlertsAssert.assertOpen("backups.run.missing-in-storage:nightly", BackupAlertSink.Severity.CRITICAL);
|
||||
assertThat(check(plan).status()).as("a negative answer is never cached").isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
|
||||
BackupsTestUtils.BACKUPS.putBytes(result.getManifestKey(), bytes);
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
|
||||
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isFalse();
|
||||
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isFalse();
|
||||
}
|
||||
|
||||
|
||||
@@ -260,11 +260,11 @@ class SlaCheckerTest
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> false);
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OWNER_INVALID);
|
||||
BackupAlertsAssert.assertOpen("backups.plan.owner_invalid:nightly", BackupAlertSink.Severity.WARNING);
|
||||
BackupAlertsAssert.assertOpen("backups.plan.owner-invalid:nightly", BackupAlertSink.Severity.WARNING);
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> "bob".equals(owner));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
|
||||
assertThat(sink.isOpen("backups.plan.owner_invalid:nightly")).isFalse();
|
||||
assertThat(sink.isOpen("backups.plan.owner-invalid:nightly")).isFalse();
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner ->
|
||||
{
|
||||
@@ -287,7 +287,7 @@ class SlaCheckerTest
|
||||
clock.advance(Duration.ofDays(2));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
assertThat(sink.isOpen("backups.plan.missed:nightly")).as("the alerts are independent of the status shown").isTrue();
|
||||
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isTrue();
|
||||
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.tables;
|
||||
|
||||
|
||||
import java.lang.reflect.Field;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.regex.Pattern;
|
||||
import java.util.stream.Stream;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.model.BackupsHelp;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Guardrail (owner requirement 2026-10-14: "all tables should be populated and
|
||||
** the qbit should be fully using them"). For every backups_* table:
|
||||
** - runtime code (not just its model, metadata or home screen) reads or writes it,
|
||||
** so no table exists only on paper;
|
||||
** - it is listed here with why it starts empty and, where a user sets it up, how
|
||||
** (this qbit ships no DDL and its one setup value, the storage bucket alias, is
|
||||
** the host's, so nothing is seeded by a changelog);
|
||||
** - it has bundle help that says what it is for and how its rows get created
|
||||
** (shown on the table screen and when it is empty).
|
||||
** A new table fails this test until it is wired and explained.
|
||||
*******************************************************************************/
|
||||
class BackupsTablesGuardrailTest
|
||||
{
|
||||
private static final Path MAIN = Path.of("src", "main", "java", "com", "kingsrook", "qbits", "backups");
|
||||
|
||||
/*******************************************************************************
|
||||
** Why each table legitimately starts empty, and where its rows come from.
|
||||
*******************************************************************************/
|
||||
static final Map<String, String> STARTS_EMPTY = Map.ofEntries(
|
||||
Map.entry(BackupsTables.PLAN, "host-specific: its storage bucket alias names a bucket the host provides (the Create default plan action adds Nightly full for an alias)"),
|
||||
Map.entry(BackupsTables.PLAN_SCOPE, "user content: include/exclude rules; an INSTANCE plan needs none"),
|
||||
Map.entry(BackupsTables.RUN, "runtime: a plan runs, or Import backups from storage"),
|
||||
Map.entry(BackupsTables.ARTIFACT, "runtime: every object a run stores"),
|
||||
Map.entry(BackupsTables.VERIFICATION, "runtime: Verify backup, or a plan verify schedule"),
|
||||
Map.entry(BackupsTables.RESTORE_RUN, "runtime: a restore"),
|
||||
Map.entry(BackupsTables.RESTORE_RUN_TABLE, "runtime: each table of a restore"),
|
||||
Map.entry(BackupsTables.FILE_OBJECT, "runtime: each distinct file a plan copies"),
|
||||
Map.entry(BackupsTables.ID_MAP, "runtime: working data of a restore with new ids, cleared when it ends"));
|
||||
|
||||
private QInstance qInstance;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
qInstance = BackupsTestUtils.init();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every table is used by runtime code.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void everyTableIsUsedAtRuntime() throws Exception
|
||||
{
|
||||
List<String> runtimeSource = new ArrayList<>();
|
||||
try(Stream<Path> files = Files.walk(MAIN))
|
||||
{
|
||||
for(Path file : files.filter(p -> p.toString().endsWith(".java")).filter(p -> !p.startsWith(MAIN.resolve("model")) && !p.startsWith(MAIN.resolve("app"))).toList())
|
||||
{
|
||||
runtimeSource.add(Files.readString(file, StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
for(String table : backupsTables())
|
||||
{
|
||||
String constant = constantFor(table);
|
||||
Pattern use = Pattern.compile("\\bBackupsTables\\." + constant + "\\b");
|
||||
assertThat(runtimeSource.stream().anyMatch(source -> use.matcher(source).find()))
|
||||
.as(table + " (BackupsTables." + constant + ") must be read or written by runtime code, not only declared").isTrue();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every table is explained: where its rows come from.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void everyTableIsExplained()
|
||||
{
|
||||
assertThat(backupsTables()).containsExactlyInAnyOrderElementsOf(STARTS_EMPTY.keySet());
|
||||
assertThat(BackupsTables.ALL).containsExactlyInAnyOrderElementsOf(STARTS_EMPTY.keySet());
|
||||
assertThat(STARTS_EMPTY.values()).allMatch(reason -> !reason.isBlank());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every table and the app have bundle help that says how rows get created, and
|
||||
** the plan help names the action that sets it up.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void everyTableHasEmptyStateHelp() throws Exception
|
||||
{
|
||||
assertThat(BackupsHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
|
||||
for(String table : backupsTables())
|
||||
{
|
||||
List<QHelpContent> help = qInstance.getTable(table).getHelpContent() == null ? null : qInstance.getTable(table).getHelpContent().get(BackupsHelp.SLOT);
|
||||
assertThat(help).as(table + " table help").isNotEmpty();
|
||||
assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created");
|
||||
}
|
||||
assertThat(qInstance.getApp("backups").getHelpContent()).containsKey(BackupsHelp.SLOT);
|
||||
assertThat(BackupsHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || qInstance.getTable(key.substring(6, key.indexOf(';'))) != null,
|
||||
"help names only real tables");
|
||||
|
||||
assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("Create default plan");
|
||||
assertThat(qInstance.getProcess(DefaultPlanProcess.NAME)).as("the action the plan help names").isNotNull();
|
||||
assertThat(qInstance.getProcess(DefaultPlanProcess.NAME).getLabel()).isEqualTo("Create default plan");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A host's own help in the slot is never replaced by the bundle.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void hostHelpIsKept() throws Exception
|
||||
{
|
||||
QTableMetaData table = qInstance.getTable(BackupsTables.RUN);
|
||||
table.withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for runs."));
|
||||
qInstance.getApp("backups").withHelpContent(BackupsHelp.SLOT, new QHelpContent("Host help for the app."));
|
||||
BackupsHelp.apply(qInstance, "backups");
|
||||
assertThat(table.getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for runs."));
|
||||
assertThat(qInstance.getApp("backups").getHelpContent().get(BackupsHelp.SLOT)).singleElement().satisfies(h -> assertThat(h.getContent()).isEqualTo("Host help for the app."));
|
||||
assertThat(qInstance.getTable(BackupsTables.PLAN).getHelpContent().get(BackupsHelp.SLOT).get(0).getContent()).contains("How rows get created");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private List<String> backupsTables()
|
||||
{
|
||||
List<String> tables = qInstance.getTables().values().stream().map(QTableMetaData::getName).filter(n -> n.startsWith("backups_")).sorted().toList();
|
||||
assertThat(tables).hasSize(9);
|
||||
return (tables);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The BackupsTables constant whose value is the table name.
|
||||
*******************************************************************************/
|
||||
private static String constantFor(String table) throws Exception
|
||||
{
|
||||
for(Field field : BackupsTables.class.getFields())
|
||||
{
|
||||
if(field.getType() == String.class && table.equals(field.get(null)) && !field.getName().equals("PVS_PREFIX"))
|
||||
{
|
||||
return (field.getName());
|
||||
}
|
||||
}
|
||||
throw (new IllegalStateException("no constant for " + table));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.tables;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
|
||||
import com.kingsrook.qbits.backups.plan.BackupValidatorReset;
|
||||
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The "Create default plan" table action: it asks for a storage bucket alias,
|
||||
** creates the Nightly full plan through the normal insert, refuses an alias that
|
||||
** does not resolve, and is insert-only (a second run changes nothing, an edited
|
||||
** plan survives).
|
||||
*******************************************************************************/
|
||||
class DefaultPlanProcessTest
|
||||
{
|
||||
private final BackupCatalog catalog = new BackupCatalog();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupValidatorReset.reset();
|
||||
QContext.getQSession().withPermissions(BackupsPermissions.ADMIN);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Run the action through Review and then Result.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private List<ProcessSummaryLine> run(String alias, boolean execute) throws QException
|
||||
{
|
||||
RunProcessInput input = new RunProcessInput();
|
||||
input.setProcessName(DefaultPlanProcess.NAME);
|
||||
input.addValue(DefaultPlanProcess.FIELD_ALIAS, alias);
|
||||
//////////////////////////////////////////////////////////////////////////
|
||||
// the first call stops at the input form; the next runs Review (full //
|
||||
// validation), the last executes and gives the Result //
|
||||
//////////////////////////////////////////////////////////////////////////
|
||||
RunProcessOutput output = new RunProcessAction().execute(input);
|
||||
input.addValue("processUUID", output.getProcessUUID());
|
||||
input.setStartAfterStep("input");
|
||||
output = new RunProcessAction().execute(input);
|
||||
if(!execute)
|
||||
{
|
||||
return ((List<ProcessSummaryLine>) output.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY));
|
||||
}
|
||||
input.setStartAfterStep("review");
|
||||
output = new RunProcessAction().execute(input);
|
||||
return ((List<ProcessSummaryLine>) output.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The action is a table action that needs no selected records and backups.admin.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void isATableActionForAdmins() throws Exception
|
||||
{
|
||||
var process = QContext.getQInstance().getProcess(DefaultPlanProcess.NAME);
|
||||
assertThat(process.getTableName()).isEqualTo(BackupsTables.PLAN);
|
||||
assertThat(process.getMinInputRecords()).isEqualTo(0);
|
||||
assertThat(((com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData) process.getStep("input")).getFormFields().stream().map(f -> f.getName())).containsExactly(DefaultPlanProcess.FIELD_ALIAS);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Review previews without writing; Result creates the plan, and the plan works.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void createsTheNightlyFullPlan() throws Exception
|
||||
{
|
||||
List<ProcessSummaryLine> review = run("backups", false);
|
||||
assertThat(review).anySatisfy(line -> assertThat(line.getPluralFutureMessage()).contains("will be created"));
|
||||
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).as("Review writes nothing").isEmpty();
|
||||
|
||||
List<ProcessSummaryLine> result = run("backups", true);
|
||||
assertThat(result).singleElement().satisfies(line -> assertThat(line.getStatus().name()).isEqualTo("OK"));
|
||||
|
||||
QRecord plan = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0);
|
||||
assertThat(plan.getValueString("name")).isEqualTo("Nightly full");
|
||||
assertThat(plan.getValueString("storageBucketAlias")).isEqualTo("backups");
|
||||
assertThat(plan.getValueBoolean("isActive")).isTrue();
|
||||
assertThat(plan.getValueString("scopeType")).isEqualTo("INSTANCE");
|
||||
assertThat(plan.getValueString("cronExpression")).isEqualTo(DefaultPlanProcess.CRON);
|
||||
assertThat(plan.getValueString("verifyCronExpression")).isEqualTo(DefaultPlanProcess.VERIFY_CRON);
|
||||
assertThat(plan.getValueInteger("slaHours")).isEqualTo(26);
|
||||
assertThat(plan.getValueString("format")).isEqualTo("JSONL");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A plan made by the action is a working plan: it runs and backs up the instance.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theCreatedPlanRuns() throws Exception
|
||||
{
|
||||
BackupsTestUtils.loadBlog(3);
|
||||
run("backups", true);
|
||||
QRecord plan = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0);
|
||||
var result = new com.kingsrook.qbits.backups.run.BackupRunCoordinator().runPlan(plan, "MANUAL");
|
||||
assertThat(result.getStatus()).isIn("SUCCEEDED", "SUCCEEDED_WITH_WARNINGS");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Insert-only: a second run creates nothing, and an edited plan keeps its edit.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void isInsertOnly() throws Exception
|
||||
{
|
||||
run("backups", true);
|
||||
Integer id = catalog.where(BackupsTables.PLAN, "planKey", DefaultPlanProcess.PLAN_KEY).get(0).getValueInteger("id");
|
||||
catalog.update(BackupsTables.PLAN, id, Map.of("slaHours", 48, "name", "Our nightly"));
|
||||
|
||||
List<ProcessSummaryLine> review = run("backups", false);
|
||||
assertThat(review).singleElement().satisfies(line -> assertThat(line.getMessage()).contains("already exists"));
|
||||
run("backups", true);
|
||||
|
||||
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).hasSize(1);
|
||||
QRecord plan = catalog.get(BackupsTables.PLAN, id).orElseThrow();
|
||||
assertThat(plan.getValueInteger("slaHours")).isEqualTo(48);
|
||||
assertThat(plan.getValueString("name")).isEqualTo("Our nightly");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An alias the host does not provide is refused in Review, and nothing is created.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void refusesAnUnknownAlias() throws Exception
|
||||
{
|
||||
List<ProcessSummaryLine> review = run("no-such-bucket", false);
|
||||
assertThat(review).anySatisfy(line ->
|
||||
{
|
||||
assertThat(line.getStatus().name()).isEqualTo("ERROR");
|
||||
assertThat(String.join(" ", line.getBulletsOfText())).contains("no-such-bucket");
|
||||
});
|
||||
run("no-such-bucket", true);
|
||||
assertThat(catalog.query(BackupsTables.PLAN, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A plan that already has the name Nightly full (under another key) is refused in
|
||||
** Review, never "will be created".
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void reviewRefusesATakenName() throws Exception
|
||||
{
|
||||
catalog.insert(BackupsTables.PLAN, new java.util.LinkedHashMap<>(Map.of("name", DefaultPlanProcess.PLAN_NAME, "planKey", "other-key", "isActive", false, "scopeType", "INSTANCE", "storageBucketAlias", "backups")));
|
||||
List<ProcessSummaryLine> review = run("backups", false);
|
||||
assertThat(review).noneSatisfy(line -> assertThat(line.getPluralFutureMessage()).contains("will be created"));
|
||||
assertThat(review).anySatisfy(line ->
|
||||
{
|
||||
assertThat(line.getStatus().name()).isEqualTo("ERROR");
|
||||
assertThat(String.join(" ", line.getBulletsOfText())).contains("already exists");
|
||||
});
|
||||
}
|
||||
}
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.tables;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.plan.BackupValidatorReset;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qbits.backups.run.BackupRunCoordinator;
|
||||
import com.kingsrook.qbits.backups.schedule.BackupPlanVerifySyncToScheduledJobProcess;
|
||||
import com.kingsrook.qbits.backups.schedule.BackupScheduledVerifyProcess;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory;
|
||||
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Two half-wired settings of backups_plan, made honest or made to work:
|
||||
** - the settings that are not available yet (read replica, native dump,
|
||||
** incremental, PARQUET, ZSTD) are refused at save with a clear message and say so
|
||||
** in their field help;
|
||||
** - verifyCronExpression creates a scheduled job that runs the process
|
||||
** backupVerifyPlan; that process now exists, and verifies the plan's latest
|
||||
** successful backup through the plan's verifyLevel.
|
||||
*******************************************************************************/
|
||||
class UnavailableSettingsAndScheduledVerifyTest
|
||||
{
|
||||
private final BackupCatalog catalog = new BackupCatalog();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupsTestUtils.loadBlog(3);
|
||||
BackupValidatorReset.reset();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Save a plan through the table (validation runs), returning the record.
|
||||
*******************************************************************************/
|
||||
private QRecord save(Map<String, Serializable> overrides) throws QException
|
||||
{
|
||||
Map<String, Serializable> values = new LinkedHashMap<>(Map.of("name", "P" + System.nanoTime(), "planKey", "p" + System.nanoTime(), "isActive", true, "scopeType", "INSTANCE", "storageBucketAlias", "backups"));
|
||||
values.putAll(overrides);
|
||||
QRecord record = new QRecord();
|
||||
record.setValues(values);
|
||||
InsertInput input = new InsertInput(BackupsTables.PLAN);
|
||||
input.setRecords(List.of(record));
|
||||
input.withPermissionBypass("test");
|
||||
return (new InsertAction().execute(input).getRecords().get(0));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every setting that is not available yet is refused, and the field help says so.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void unavailableSettingsAreRefusedAndSayWhy() throws Exception
|
||||
{
|
||||
assertThat(save(Map.of()).getErrors()).as("the defaults are fine").isEmpty();
|
||||
|
||||
Map<String, Map<String, Serializable>> refused = Map.of(
|
||||
"Reading from a replica is not available yet", Map.of("useReadReplica", true),
|
||||
"Native database dumps are not available yet", Map.of("nativeDump", "PG_DUMP_CUSTOM"),
|
||||
"Incremental backups are not available yet", Map.of("incrementalMode", "CHANGED_SINCE_LAST"),
|
||||
"PARQUET format is not available yet", Map.of("format", "PARQUET"),
|
||||
"ZSTD compression is not available yet", Map.of("compression", "ZSTD"));
|
||||
for(Map.Entry<String, Map<String, Serializable>> entry : refused.entrySet())
|
||||
{
|
||||
QRecord record = save(entry.getValue());
|
||||
assertThat(record.getErrors()).as(entry.getValue().toString()).isNotEmpty();
|
||||
assertThat(record.getErrors().toString()).contains(entry.getKey());
|
||||
}
|
||||
|
||||
var table = QContext.getQInstance().getTable(BackupsTables.PLAN);
|
||||
for(String field : List.of("useReadReplica", "nativeDump", "incrementalMode", "format", "compression", "scheduledJobId"))
|
||||
{
|
||||
assertThat(table.getField(field).getHelpContents()).as(field + " help").isNotEmpty();
|
||||
assertThat(table.getField(field).getHelpContents().get(0).getContent()).as(field + " help").matches("(?is).*(not available yet|not used).*");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The process a verify cron schedules exists, and verifies the plan's latest
|
||||
** successful run, as the scheduler, through the plan's verifyLevel.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theVerifyCronRunsARealProcess() throws Exception
|
||||
{
|
||||
assertThat(QContext.getQInstance().getProcess(BackupPlanVerifySyncToScheduledJobProcess.VERIFY_PROCESS_NAME)).as("the process the scheduled verify job names").isNotNull();
|
||||
|
||||
Integer planId = catalog.insert(BackupsTables.PLAN, new LinkedHashMap<>(Map.of("name", "Nightly", "planKey", "nightly", "isActive", true, "scopeType", "INSTANCE",
|
||||
"storageBucketAlias", "backups", "verifyLevel", "STORAGE")));
|
||||
assertThat(run(planId).getValues().get("status")).as("nothing to verify before the first backup").isEqualTo("NO_BACKUP");
|
||||
|
||||
QRecord plan = catalog.get(BackupsTables.PLAN, planId).orElseThrow();
|
||||
var result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
|
||||
assertThat(result.getStatus()).isIn("SUCCEEDED", "SUCCEEDED_WITH_WARNINGS");
|
||||
|
||||
RunProcessOutput output = run(planId);
|
||||
assertThat(output.getException()).isEmpty();
|
||||
assertThat(output.getValues().get("status")).isEqualTo("PASSED");
|
||||
assertThat(output.getValues().get("level")).isEqualTo("STORAGE");
|
||||
|
||||
List<QRecord> verifications = catalog.where(BackupsTables.VERIFICATION, "backupRunId", result.getRunId());
|
||||
assertThat(verifications).singleElement().satisfies(v ->
|
||||
{
|
||||
assertThat(v.getValueString("level")).isEqualTo("STORAGE");
|
||||
assertThat(v.getValueString("trigger")).isEqualTo("SCHEDULED");
|
||||
assertThat(v.getValueString("requestedByUserId")).isEqualTo(BackupScheduledVerifyProcess.SCHEDULER_USER);
|
||||
});
|
||||
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValue("lastVerifiedDate")).as("a storage check alone does not count as verified").isNull();
|
||||
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
// the plan's level decides how far it goes; an inactive plan is skipped //
|
||||
/////////////////////////////////////////////////////////////////////
|
||||
catalog.update(BackupsTables.PLAN, planId, Map.of("verifyLevel", "CONTENT"));
|
||||
assertThat(run(planId).getValues().get("level")).isEqualTo("CONTENT");
|
||||
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValue("lastVerifiedDate")).isNotNull();
|
||||
catalog.update(BackupsTables.PLAN, planId, Map.of("isActive", false));
|
||||
assertThat(run(planId).getValues().get("status")).isEqualTo("INACTIVE");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An unset or unknown level falls back to CONTENT.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void levelFallsBackToContent()
|
||||
{
|
||||
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord())).isEqualTo("CONTENT");
|
||||
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord().withValue("verifyLevel", "BOGUS"))).isEqualTo("CONTENT");
|
||||
assertThat(BackupScheduledVerifyProcess.levelOf(new QRecord().withValue("verifyLevel", "TEST_RESTORE"))).isEqualTo("TEST_RESTORE");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Run the scheduled verify entry point for a plan, as the scheduler would.
|
||||
*******************************************************************************/
|
||||
private RunProcessOutput run(Integer planId) throws QException
|
||||
{
|
||||
RunProcessInput input = new RunProcessInput();
|
||||
input.setProcessName(BackupScheduledVerifyProcess.NAME);
|
||||
input.setCallback(QProcessCallbackFactory.forPrimaryKey("id", planId));
|
||||
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
|
||||
return (new RunProcessAction().execute(input));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"author" : {
|
||||
"schema" : {
|
||||
"name" : "author",
|
||||
"primaryKey" : "id",
|
||||
"backendType" : "memory",
|
||||
"fields" : {
|
||||
"bio" : {
|
||||
"type" : "TEXT",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"id" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"name" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
}
|
||||
},
|
||||
"uniqueKeys" : [ ],
|
||||
"associations" : [ ]
|
||||
},
|
||||
"fingerprint" : "sha256:c0fce31286898a15a98f9f46515b15c6eee4ffeb1f1eb39d6100833dfe3f760e"
|
||||
},
|
||||
"blogPost" : {
|
||||
"schema" : {
|
||||
"name" : "blogPost",
|
||||
"primaryKey" : "id",
|
||||
"backendType" : "memory",
|
||||
"fields" : {
|
||||
"authorId" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : "author",
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"body" : {
|
||||
"type" : "TEXT",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"createDate" : {
|
||||
"type" : "DATE_TIME",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"id" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"mediaKey" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"price" : {
|
||||
"type" : "DECIMAL",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"published" : {
|
||||
"type" : "BOOLEAN",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"publishedOn" : {
|
||||
"type" : "DATE",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"secretNote" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : true
|
||||
},
|
||||
"title" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
}
|
||||
},
|
||||
"uniqueKeys" : [ ],
|
||||
"associations" : [ ]
|
||||
},
|
||||
"fingerprint" : "sha256:3a4de7479908cd4b4ce8eaa7a7af43e7cbaabbdbc2b9741bebeb24a6bd41b9ea"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user