Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a5297e709d | ||
|
|
f2af36ec5f | ||
|
|
054f6d90d7
|
||
|
|
5b7106b981
|
||
|
|
32034e5762 | ||
|
|
bc97d59b96
|
||
|
|
506741bd90 | ||
|
|
95b971a5fb
|
||
|
|
e56cc1b021 |
No files matched your search
+2
-13
@@ -1,7 +1,7 @@
|
||||
version: 2.1
|
||||
|
||||
orbs:
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.8
|
||||
qqq-orb: kingsrook/qqq-orb@0.6.9
|
||||
|
||||
commands:
|
||||
configure-package-install:
|
||||
@@ -22,7 +22,7 @@ workflows:
|
||||
context: [qqq-maven-registry-credentials]
|
||||
filters:
|
||||
branches:
|
||||
ignore: /(develop|release\/.*|hotfix\/.*)/
|
||||
ignore: /(develop|release\/.*)/
|
||||
|
||||
publish_snapshot:
|
||||
jobs:
|
||||
@@ -58,14 +58,3 @@ workflows:
|
||||
ignore: /.*/
|
||||
tags:
|
||||
only: [/^v\d+\.\d+\.\d+$/]
|
||||
|
||||
publish_hotfix_release:
|
||||
jobs:
|
||||
- qqq-orb/mvn_publish:
|
||||
pre-steps:
|
||||
- configure-package-install
|
||||
context: [qqq-maven-registry-credentials]
|
||||
branch_type: hotfix
|
||||
filters:
|
||||
branches:
|
||||
only: [/hotfix\/.*/]
|
||||
@@ -0,0 +1,25 @@
|
||||
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
|
||||
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
|
||||
# the release line and the release/* push runs CircleCI publish_release_candidate.
|
||||
name: Release candidate
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
description: Print the plan without merging or creating branches
|
||||
type: boolean
|
||||
default: false
|
||||
pull_request:
|
||||
types: [closed]
|
||||
branches: [develop]
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
jobs:
|
||||
release-rc:
|
||||
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
|
||||
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
|
||||
with:
|
||||
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
|
||||
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
|
||||
## Dependencies
|
||||
|
||||
- Java 21.
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
|
||||
- Tests: JUnit 5, AssertJ.
|
||||
|
||||
## Release
|
||||
|
||||
@@ -6,3 +6,12 @@ All notable changes to this project are documented here. The format follows Keep
|
||||
|
||||
### Added
|
||||
- Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.
|
||||
|
||||
### Changed
|
||||
- Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
|
||||
- Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.
|
||||
- Use RC.5-valid alert names `backups.run.missing-in-storage` and `backups.plan.owner-invalid` (and matching deduplication keys). Hosts using the old underscore names must update routing rules.
|
||||
- Target QQQ 4.1.0-RC.6 and replace the fingerprint, keyset pager and reflective restore bridges with typed core APIs.
|
||||
- New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
|
||||
- Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
|
||||
- Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
|
||||
@@ -10,7 +10,7 @@
|
||||
## Requirements
|
||||
|
||||
- Java 21.
|
||||
- QQQ 4.1 (built against `4.1.0-RC.4`).
|
||||
- QQQ 4.1 (targets `4.1.0-RC.6`, not yet released).
|
||||
|
||||
## Install
|
||||
|
||||
@@ -36,8 +36,32 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
|
||||
./mvnw verify
|
||||
```
|
||||
|
||||
Before RC.6 is released, verify against the locally installed core build only:
|
||||
|
||||
```bash
|
||||
./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
|
||||
```
|
||||
|
||||
The POM stays on RC.6. See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for
|
||||
fingerprint compatibility, paging guarantees and restore permissions.
|
||||
|
||||
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
|
||||
|
||||
### Alerts integration
|
||||
|
||||
Backups binds directly to core QAlerts in QQQ 4.1.0-RC.5 and declares its nine
|
||||
alert types on each instance. Hosts install an `AlertSinkInterface` through
|
||||
`AlertsInstanceMetaData.withSink(QCodeReference)`. Without a configured sink,
|
||||
backup alert calls remain no-ops (they do not use core's logging fallback).
|
||||
Custom `BackupAlertSink` bindings are preserved. Core handles dispatch failures;
|
||||
adapter failures are caught and logged once, so alerting cannot fail a backup.
|
||||
Raise and resolve follow core's transaction-aware dispatch; heartbeats run immediately.
|
||||
The backup SLA checker still owns expected intervals and grace periods.
|
||||
|
||||
RC.5 requires hyphenated alert names: `backups.run.missing-in-storage` and
|
||||
`backups.plan.owner-invalid` replace the former underscore spellings, including
|
||||
their deduplication keys. Update host routing rules that used those spellings.
|
||||
@@ -0,0 +1,100 @@
|
||||
# RC.6 core migration (issue #19, F3)
|
||||
|
||||
The POM targets `4.1.0-RC.6`. Until that release is available, local verification
|
||||
uses `./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` against the
|
||||
installed build from `qqq-wt-archrc6`. No core source changes are needed.
|
||||
|
||||
## Fingerprint compatibility
|
||||
|
||||
The algorithms are **not byte-compatible**, even for identical table metadata:
|
||||
|
||||
| Descriptor property | Pre-RC.6 backup algorithm | Core algorithm |
|
||||
| --- | --- | --- |
|
||||
| Table and key | `name`, `primaryKey` | `table`, `primaryKeyField` |
|
||||
| Fields | Map keyed by name | Sorted list with explicit `name` |
|
||||
| Field flags | `required`, `pvs`, `heavy`, `sensitive` | `isRequired`, `possibleValueSource`, `isHeavy`, `isSensitive` |
|
||||
| Unique keys | Sorted `+`-joined strings | Sorted lists of field names |
|
||||
| Associations | Sorted `name->table` strings | Objects with name, associatedTable and joinName |
|
||||
| Null backend type | JSON null | Empty string |
|
||||
| Escaping | Historical hand-written escaping | Jackson JSON escaping |
|
||||
|
||||
New table entries contain `fingerprintAlgorithm: "qqq-table-schema-v1"`, the core
|
||||
`schema` descriptor, and `fingerprint: "sha256:" + core.getHash()`. The outer
|
||||
manifest remains version 1; row parts, checksums, compression and storage paths
|
||||
are unchanged. The instance/catalog hash still combines the ordered table
|
||||
fingerprint strings, so its value changes too. New manifests require this RC.6
|
||||
reader; older readers do not understand the new descriptor shape.
|
||||
|
||||
An absent algorithm identifies the old `qbit-backups-table-v1` algorithm. The
|
||||
reader also accepts that explicit identifier. `LegacyTableFingerprint` freezes
|
||||
the original descriptor and hash implementation solely for comparison against
|
||||
existing backups; no new backup uses it. Restore and content verification select
|
||||
the declared algorithm. Unknown identifiers never match a known fingerprint.
|
||||
Restore normalizes either field shape for its existing compatibility analysis
|
||||
when a table has changed. Fingerprint equality is not required for a compatible
|
||||
schema migration, as before.
|
||||
|
||||
`src/test/resources/manifests/legacy-schema.json` freezes descriptors and hashes
|
||||
produced by the original stand-in for the blog fixture. Tests place these into a
|
||||
real backup manifest without the algorithm field, verify its content, mutate the
|
||||
live data, then restore and compare the complete content hash including dates.
|
||||
They also cover optional additions and incompatible type changes in old schemas.
|
||||
|
||||
## Paging and consistency
|
||||
|
||||
`TableReader.Keyset` delegates to core `KeysetPager`. Every query uses
|
||||
`(scope filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize`, with no offset.
|
||||
The resume key is exclusive. Page size is a row bound, **not a JDBC fetch-size
|
||||
setting**. Core's normal bounded QueryAction path retains backend driver defaults;
|
||||
this adapter does not opt into the large-result streaming query hint or change
|
||||
JDBC fetch size. Heavy/hidden/password fields and untranslated stored values keep
|
||||
the same query settings and existing backup permission boundary.
|
||||
|
||||
Every page receives the caller's transaction. The pager does not open, commit,
|
||||
close, or replace that transaction. A test opens a real core memory snapshot,
|
||||
changes a later page's data, and proves every page still sees the original state.
|
||||
The existing BackupEngine continues to report BEST_EFFORT and reject
|
||||
SNAPSHOT_REQUIRED: adopting the consistent-read opening API at engine level is
|
||||
separate work. This migration does not claim snapshot guarantees for ordinary
|
||||
backup runs.
|
||||
|
||||
Core can stream non-orderable keys without deterministic key order; backups
|
||||
explicitly reject those keys instead. Core rejects scope filters carrying skip
|
||||
or limit, and replaces user ordering with ascending primary-key ordering. Tests
|
||||
cover OR filters, resume, full and partial pages, empty scans and invalid inputs.
|
||||
|
||||
## Restore writes
|
||||
|
||||
`StandardRestoreWriter` now constructs `RestoreWriteFlag` directly and calls
|
||||
`QBackendModuleInterface.resyncKeySequences`, consuming typed `SequenceResyncResult`
|
||||
values. Only RESYNCED results are reported as sequence changes; NOT_NEEDED and
|
||||
NOT_APPLICABLE remain normal outcomes. There is no reflection or unavailable-core
|
||||
fallback. The core flag preserves supplied dates and carries the requested
|
||||
customizer, automation, audit and event settings.
|
||||
|
||||
Core always requires an actual session permission `backups.restore` for its
|
||||
restore flag, including hosts with general core enforcement disabled. The qbit's
|
||||
anonymous-mode convenience setting does not grant that permission. Review now
|
||||
reports this requirement before executing a restore; test sessions explicitly
|
||||
hold it. Custom writers that do not use the core flag keep their existing qbit
|
||||
permission checks.
|
||||
|
||||
## HTTP operations and wire changes
|
||||
|
||||
No HTTP operations are added, removed or modified: method, path and operationId
|
||||
are unchanged (no operation list entries for this migration). The RC.6 UI API
|
||||
extension documentation was reviewed; this task introduces no Javalin route or
|
||||
middleware extension. Wire changes are confined to the manifest fields and
|
||||
fingerprint values described above, plus an earlier restore permission error in
|
||||
Review for anonymous sessions lacking `backups.restore`.
|
||||
|
||||
## Local verification (2026-10-13)
|
||||
|
||||
- `./mvnw -B clean verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`:
|
||||
BUILD SUCCESS; 158 tests, zero failures/errors/skips; both JaCoCo gates passed.
|
||||
Existing backup, restore and MinIO integration tests ran.
|
||||
- Separate Checkstyle 14.3.0 audit of all 16 changed Java files (main and test):
|
||||
zero violations, using the shared Kingsrook-derived rules with this repository's
|
||||
Apache file header. This repository has no Checkstyle execution configured in
|
||||
its POM; audit configuration and results are local under `.cache/style/`.
|
||||
- `git diff --check`: passed. No commit, push or core source modification.
|
||||
@@ -18,6 +18,8 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
|
||||
<properties>
|
||||
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
|
||||
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
|
||||
<revision>0.1.0-SNAPSHOT</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
@@ -27,7 +29,7 @@
|
||||
<parent>
|
||||
<groupId>com.kingsrook</groupId>
|
||||
<artifactId>qbit-build-parent</artifactId>
|
||||
<version>2.0.0</version>
|
||||
<version>2.1.0</version>
|
||||
</parent>
|
||||
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
@@ -192,14 +194,14 @@
|
||||
<profiles>
|
||||
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
|
||||
<!-- imports the released 4.1 BOM, this profile is active by default and -->
|
||||
<!-- imports qqq-bom-pom 4.1.0-RC.4 ahead of the parent's 4.0.0 BOM. -->
|
||||
<!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM. -->
|
||||
<profile>
|
||||
<id>qqq-snapshot</id>
|
||||
<activation>
|
||||
<activeByDefault>true</activeByDefault>
|
||||
</activation>
|
||||
<properties>
|
||||
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
|
||||
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
|
||||
</properties>
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
|
||||
@@ -119,13 +119,8 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
|
||||
qInstance.addQBit(qBit);
|
||||
BackupsRuntime.install(qBitConfig);
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
// bind the Alerts qbit producer SPI when it is on the class path (optional) //
|
||||
///////////////////////////////////////////////////////////////////////////////
|
||||
if(com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfPresent())
|
||||
{
|
||||
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
|
||||
}
|
||||
com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfAbsent();
|
||||
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
|
||||
|
||||
com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput output = new com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput();
|
||||
for(var possibleValueSource : com.kingsrook.qbits.backups.model.BackupsTables.possibleValueSources())
|
||||
|
||||
@@ -17,105 +17,39 @@
|
||||
package com.kingsrook.qbits.backups.alerts;
|
||||
|
||||
|
||||
import java.lang.reflect.Method;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The BackupAlertSink adapter for the Alerts qbit producer SPI (QRun-IO/qqq
|
||||
** PR #1108, com.kingsrook.qqq.backend.core.alerts: QAlerts, AlertSignal,
|
||||
** AlertResolution, AlertSeverity, QAlertTypeMetaData; design spec
|
||||
** docs/specs/2026-10-12-qbit-alerts-design.md).
|
||||
**
|
||||
** STAND-IN: the SPI is an open core PR and is not in QQQ 4.1.0-RC.4, so this
|
||||
** adapter is bound REFLECTIVELY and optionally. It compiles against nothing in
|
||||
** the SPI; it looks the classes up by name, and every call is guarded:
|
||||
** <ul>
|
||||
** <li>no SPI classes on the class path: {@link #bindIfPresent()} binds nothing
|
||||
** and the sink stays the no-op;</li>
|
||||
** <li>the SPI is present but the instance has no alerts meta-data (the Alerts
|
||||
** qbit is not installed): every call is a no-op;</li>
|
||||
** <li>any reflective or SPI failure is logged once and swallowed, because an
|
||||
** alerting problem must never fail a backup.</li>
|
||||
** </ul>
|
||||
** Replace with a direct binding (QAlerts.raise and friends) when RC.5 ships.
|
||||
**
|
||||
** Mapping: raise to QAlerts.raise(AlertSignal), resolve to QAlerts.resolve(type,
|
||||
** dedupKey, note), heartbeat to QAlerts.heartbeat(key). The SPI has no per-plan
|
||||
** heartbeat registration (monitors are declared on the alert type), so the
|
||||
** expected interval and grace are not passed on; the SLA checker raises
|
||||
** backups.plan.missed itself, independent of any heartbeat (spec 10.4).
|
||||
** Direct adapter to core QAlerts. Calls are no-ops without a configured alerts
|
||||
** sink, including when only the backup alert types have been registered.
|
||||
** QAlerts catches and logs dispatch failures; adapter failures are logged once.
|
||||
** Heartbeat intervals remain the responsibility of the backup SLA checker.
|
||||
*******************************************************************************/
|
||||
public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(AlertsQBitAlertSink.class);
|
||||
|
||||
/** The package of the Alerts producer SPI in qqq-backend-core. */
|
||||
public static final String DEFAULT_API_PACKAGE = "com.kingsrook.qqq.backend.core.alerts";
|
||||
|
||||
private final String apiPackage;
|
||||
|
||||
private boolean warned = false;
|
||||
|
||||
private Class<?> qAlerts;
|
||||
private Class<?> alertSignal;
|
||||
private Class<?> alertSeverity;
|
||||
private Class<?> instanceMetaData;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor for the real SPI package.
|
||||
** Bind the core adapter unless the host already supplied a backup sink.
|
||||
*******************************************************************************/
|
||||
public AlertsQBitAlertSink()
|
||||
public static boolean bindIfAbsent()
|
||||
{
|
||||
this(DEFAULT_API_PACKAGE);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor for another package (a test double of the SPI shape).
|
||||
*******************************************************************************/
|
||||
public AlertsQBitAlertSink(String apiPackage)
|
||||
{
|
||||
this.apiPackage = apiPackage;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether the SPI classes are on the class path.
|
||||
*******************************************************************************/
|
||||
public static boolean isPresent(String apiPackage)
|
||||
{
|
||||
try
|
||||
{
|
||||
Class.forName(apiPackage + ".QAlerts");
|
||||
Class.forName(apiPackage + ".AlertSignal");
|
||||
return (true);
|
||||
}
|
||||
catch(ClassNotFoundException | LinkageError e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bind this adapter as the runtime's alert sink when the SPI is present and no
|
||||
** other sink was bound already. Returns whether it bound.
|
||||
*******************************************************************************/
|
||||
public static boolean bindIfPresent()
|
||||
{
|
||||
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink) || !isPresent(DEFAULT_API_PACKAGE))
|
||||
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
@@ -126,38 +60,32 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Declare the backup alert types with the Alerts qbit (QAlertTypeMetaData), when
|
||||
** the SPI is present. Returns how many were added.
|
||||
** Declare missing backup alert types on each instance, independently of sink
|
||||
** installation order. Returns how many were added.
|
||||
*******************************************************************************/
|
||||
public int registerAlertTypes(QInstance qInstance)
|
||||
{
|
||||
int added = 0;
|
||||
try
|
||||
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
|
||||
{
|
||||
Class<?> typeClass = Class.forName(apiPackage + ".QAlertTypeMetaData");
|
||||
load();
|
||||
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
|
||||
try
|
||||
{
|
||||
try
|
||||
if(QAlertTypeMetaData.of(qInstance, alertType.name()) == null)
|
||||
{
|
||||
Object type = typeClass.getConstructor().newInstance();
|
||||
typeClass.getMethod("withName", String.class).invoke(type, alertType.name());
|
||||
typeClass.getMethod("withLabel", String.class).invoke(type, alertType.label());
|
||||
typeClass.getMethod("withGroupName", String.class).invoke(type, "Backups");
|
||||
typeClass.getMethod("withHowToFix", String.class).invoke(type, alertType.howToFix());
|
||||
typeClass.getMethod("withDefaultSeverity", alertSeverity).invoke(type, severity(alertType.defaultSeverity()));
|
||||
typeClass.getMethod("addSelfToInstance", QInstance.class).invoke(type, qInstance);
|
||||
new QAlertTypeMetaData()
|
||||
.withName(alertType.name())
|
||||
.withLabel(alertType.label())
|
||||
.withGroupName("Backups")
|
||||
.withHowToFix(alertType.howToFix())
|
||||
.withDefaultSeverity(AlertSeverity.valueOf(alertType.defaultSeverity().name()))
|
||||
.addSelfToInstance(qInstance);
|
||||
added++;
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
{
|
||||
warnOnce("Could not declare the alert type " + alertType.name(), e);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | LinkageError e)
|
||||
{
|
||||
warnOnce("The alerts SPI is not available", e);
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not declare the alert type " + alertType.name(), e);
|
||||
}
|
||||
}
|
||||
return (added);
|
||||
}
|
||||
@@ -170,28 +98,24 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void raise(String type, String dedupeKey, Severity severity, String title, Map<String, String> detail)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
Object signal = alertSignal.getConstructor(String.class).newInstance(type);
|
||||
alertSignal.getMethod("setDedupKey", String.class).invoke(signal, dedupeKey);
|
||||
alertSignal.getMethod("setSeverity", alertSeverity).invoke(signal, severity(severity));
|
||||
alertSignal.getMethod("setTitle", String.class).invoke(signal, title);
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
AlertSignal signal = new AlertSignal(type)
|
||||
.withDedupKey(dedupeKey)
|
||||
.withSeverity(AlertSeverity.valueOf(severity.name()))
|
||||
.withTitle(title);
|
||||
if(detail != null && !detail.isEmpty())
|
||||
{
|
||||
alertSignal.getMethod("setSummary", String.class).invoke(signal, detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
|
||||
Method withAttribute = alertSignal.getMethod("withAttribute", String.class, String.class);
|
||||
for(Map.Entry<String, String> entry : detail.entrySet())
|
||||
{
|
||||
withAttribute.invoke(signal, entry.getKey(), entry.getValue());
|
||||
}
|
||||
signal.withSummary(detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
|
||||
detail.forEach(signal::withAttribute);
|
||||
}
|
||||
qAlerts.getMethod("raise", alertSignal).invoke(null, signal);
|
||||
QAlerts.raise(signal);
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not raise an alert", e);
|
||||
}
|
||||
@@ -205,15 +129,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void resolve(String type, String dedupeKey)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
qAlerts.getMethod("resolve", String.class, String.class, String.class).invoke(null, type, dedupeKey, "Cleared by the Backups qbit");
|
||||
if(isAlertsQBitInstalled())
|
||||
{
|
||||
QAlerts.resolve(type, dedupeKey, "Cleared by the Backups qbit");
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not resolve an alert", e);
|
||||
}
|
||||
@@ -227,15 +150,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
@Override
|
||||
public void heartbeat(String monitorKey, long expectedEverySeconds, long graceSeconds)
|
||||
{
|
||||
if(!isAlertsQBitInstalled())
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
qAlerts.getMethod("heartbeat", String.class).invoke(null, monitorKey);
|
||||
if(isAlertsQBitInstalled())
|
||||
{
|
||||
QAlerts.heartbeat(monitorKey);
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException e)
|
||||
catch(RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("Could not check in a heartbeat", e);
|
||||
}
|
||||
@@ -244,67 +166,26 @@ public class AlertsQBitAlertSink implements BackupAlertSink
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The SPI classes load, and the instance has alerts meta-data (the Alerts qbit
|
||||
** is installed). Checked on every call so the order qbits are produced in
|
||||
** does not matter.
|
||||
** Check on every call so installing the alerts sink after this qbit works.
|
||||
** Type declarations alone do not mean an alerts implementation is installed.
|
||||
*******************************************************************************/
|
||||
boolean isAlertsQBitInstalled()
|
||||
{
|
||||
try
|
||||
{
|
||||
load();
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
if(qInstance == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
return (instanceMetaData.getMethod("of", QInstance.class).invoke(null, qInstance) != null);
|
||||
}
|
||||
catch(ReflectiveOperationException | RuntimeException | LinkageError e)
|
||||
{
|
||||
warnOnce("The alerts SPI is not usable", e);
|
||||
return (false);
|
||||
}
|
||||
AlertsInstanceMetaData metaData = AlertsInstanceMetaData.of(QContext.getQInstance());
|
||||
return (metaData != null && metaData.getSink() != null);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
** Log adapter failures once; dispatch failures are already logged by core.
|
||||
*******************************************************************************/
|
||||
private synchronized void load() throws ClassNotFoundException
|
||||
{
|
||||
if(qAlerts == null)
|
||||
{
|
||||
qAlerts = Class.forName(apiPackage + ".QAlerts");
|
||||
alertSignal = Class.forName(apiPackage + ".AlertSignal");
|
||||
alertSeverity = Class.forName(apiPackage + ".AlertSeverity");
|
||||
instanceMetaData = Class.forName(apiPackage + ".AlertsInstanceMetaData");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Map our severity to the SPI's enum constant of the same name.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings({ "unchecked", "rawtypes" })
|
||||
private Object severity(Severity severity)
|
||||
{
|
||||
return (Enum.valueOf((Class<Enum>) alertSeverity, severity.name()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private void warnOnce(String message, Throwable t)
|
||||
private synchronized void warnOnce(String message, Throwable t)
|
||||
{
|
||||
if(!warned)
|
||||
{
|
||||
warned = true;
|
||||
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t, logPair("apiPackage", apiPackage));
|
||||
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -36,11 +36,11 @@ public final class BackupAlerts
|
||||
{
|
||||
public static final String PLAN_MISSED = "backups.plan.missed";
|
||||
public static final String RUN_FAILED = "backups.run.failed";
|
||||
public static final String RUN_MISSING = "backups.run.missing_in_storage";
|
||||
public static final String RUN_MISSING = "backups.run.missing-in-storage";
|
||||
public static final String FILES_MISSING = "backups.files.missing";
|
||||
public static final String VERIFY_FAILED = "backups.verify.failed";
|
||||
public static final String VERIFY_OVERDUE = "backups.verify.overdue";
|
||||
public static final String PLAN_OWNER_INVALID = "backups.plan.owner_invalid";
|
||||
public static final String PLAN_OWNER_INVALID = "backups.plan.owner-invalid";
|
||||
public static final String STORAGE_DEBRIS = "backups.storage.debris";
|
||||
public static final String RESTORE_COMPLETED = "backups.restore.completed";
|
||||
|
||||
|
||||
@@ -137,6 +137,7 @@ public record BackupManifest(
|
||||
String backend,
|
||||
String primaryKey,
|
||||
String fingerprint,
|
||||
String fingerprintAlgorithm,
|
||||
Map<String, Object> schema,
|
||||
String filter,
|
||||
List<ExcludedField> excludedFields,
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Frozen pre-RC.6 fingerprint algorithm, used only to read existing manifests.
|
||||
** Keep its descriptor and escaping byte-compatible with the original writer.
|
||||
*******************************************************************************/
|
||||
final class LegacyTableFingerprint
|
||||
{
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private LegacyTableFingerprint()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
|
||||
** what goes in the manifest schema).
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
|
||||
{
|
||||
Map<String, Object> d = new LinkedHashMap<>();
|
||||
d.put("name", table.getName());
|
||||
d.put("primaryKey", table.getPrimaryKeyField());
|
||||
d.put("backendType", backendType);
|
||||
|
||||
Map<String, Object> fields = new TreeMap<>();
|
||||
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
|
||||
{
|
||||
Map<String, Object> f = new LinkedHashMap<>();
|
||||
f.put("type", field.getType().name());
|
||||
f.put("required", field.getIsRequired());
|
||||
f.put("maxLength", field.getMaxLength());
|
||||
f.put("pvs", field.getPossibleValueSourceName());
|
||||
f.put("heavy", field.getIsHeavy());
|
||||
f.put("sensitive", field.getIsSensitive());
|
||||
fields.put(field.getName(), f);
|
||||
}
|
||||
d.put("fields", fields);
|
||||
|
||||
List<String> uniqueKeys = new ArrayList<>();
|
||||
if(table.getUniqueKeys() != null)
|
||||
{
|
||||
for(UniqueKey key : table.getUniqueKeys())
|
||||
{
|
||||
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
|
||||
}
|
||||
}
|
||||
uniqueKeys.sort(String::compareTo);
|
||||
d.put("uniqueKeys", uniqueKeys);
|
||||
|
||||
List<String> associations = new ArrayList<>();
|
||||
if(table.getAssociations() != null)
|
||||
{
|
||||
for(Association association : table.getAssociations())
|
||||
{
|
||||
associations.add(association.getName() + "->" + association.getAssociatedTableName());
|
||||
}
|
||||
}
|
||||
associations.sort(String::compareTo);
|
||||
d.put("associations", associations);
|
||||
return (d);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a table: sha256:hex of the canonical descriptor.
|
||||
*******************************************************************************/
|
||||
public static String of(QTableMetaData table, String backendType)
|
||||
{
|
||||
return (fingerprintOf(descriptor(table, backendType)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
|
||||
*******************************************************************************/
|
||||
public static String fingerprintOf(Map<String, Object> descriptor)
|
||||
{
|
||||
StringBuilder sb = new StringBuilder();
|
||||
canonical(descriptor, sb);
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private static void canonical(Object o, StringBuilder sb)
|
||||
{
|
||||
if(o == null)
|
||||
{
|
||||
sb.append("null");
|
||||
}
|
||||
else if(o instanceof Map<?, ?> map)
|
||||
{
|
||||
sb.append('{');
|
||||
boolean first = true;
|
||||
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
|
||||
{
|
||||
if(!first)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
first = false;
|
||||
sb.append('"').append(e.getKey()).append("\":");
|
||||
canonical(e.getValue(), sb);
|
||||
}
|
||||
sb.append('}');
|
||||
}
|
||||
else if(o instanceof List<?> list)
|
||||
{
|
||||
sb.append('[');
|
||||
for(int i = 0; i < list.size(); i++)
|
||||
{
|
||||
if(i > 0)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
canonical(list.get(i), sb);
|
||||
}
|
||||
sb.append(']');
|
||||
}
|
||||
else if(o instanceof Number || o instanceof Boolean)
|
||||
{
|
||||
sb.append(o);
|
||||
}
|
||||
else
|
||||
{
|
||||
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -18,28 +18,23 @@ package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A stable schema descriptor and SHA-256 fingerprint of a table (spec 5.5).
|
||||
**
|
||||
** This is an internal stand-in for the QQQ core TableSchemaFingerprint (core PR
|
||||
** feature/schema-fingerprint). When that ships in the next 4.1.x RC, this class
|
||||
** delegates to it and the descriptor below is kept only for reading old manifests.
|
||||
** Labels, adornments and help text are excluded, so cosmetic changes do not count.
|
||||
** Core schema fingerprints with read compatibility for pre-RC.6 manifests.
|
||||
*******************************************************************************/
|
||||
public final class TableFingerprints
|
||||
{
|
||||
public static final String CORE_ALGORITHM = "qqq-table-schema-v1";
|
||||
public static final String LEGACY_ALGORITHM = "qbit-backups-table-v1";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
@@ -51,134 +46,83 @@ public final class TableFingerprints
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
|
||||
** what goes in the manifest schema).
|
||||
** The core descriptor stored in new manifests.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
|
||||
{
|
||||
Map<String, Object> d = new LinkedHashMap<>();
|
||||
d.put("name", table.getName());
|
||||
d.put("primaryKey", table.getPrimaryKeyField());
|
||||
d.put("backendType", backendType);
|
||||
|
||||
Map<String, Object> fields = new TreeMap<>();
|
||||
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
|
||||
{
|
||||
Map<String, Object> f = new LinkedHashMap<>();
|
||||
f.put("type", field.getType().name());
|
||||
f.put("required", field.getIsRequired());
|
||||
f.put("maxLength", field.getMaxLength());
|
||||
f.put("pvs", field.getPossibleValueSourceName());
|
||||
f.put("heavy", field.getIsHeavy());
|
||||
f.put("sensitive", field.getIsSensitive());
|
||||
fields.put(field.getName(), f);
|
||||
}
|
||||
d.put("fields", fields);
|
||||
|
||||
List<String> uniqueKeys = new ArrayList<>();
|
||||
if(table.getUniqueKeys() != null)
|
||||
{
|
||||
for(UniqueKey key : table.getUniqueKeys())
|
||||
{
|
||||
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
|
||||
}
|
||||
}
|
||||
uniqueKeys.sort(String::compareTo);
|
||||
d.put("uniqueKeys", uniqueKeys);
|
||||
|
||||
List<String> associations = new ArrayList<>();
|
||||
if(table.getAssociations() != null)
|
||||
{
|
||||
for(Association association : table.getAssociations())
|
||||
{
|
||||
associations.add(association.getName() + "->" + association.getAssociatedTableName());
|
||||
}
|
||||
}
|
||||
associations.sort(String::compareTo);
|
||||
d.put("associations", associations);
|
||||
return (d);
|
||||
return (TableSchemaFingerprint.of(table, backendType).getDescriptor());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a table: sha256:hex of the canonical descriptor.
|
||||
** Retain the manifest's sha256 prefix around the core hash.
|
||||
*******************************************************************************/
|
||||
public static String of(QTableMetaData table, String backendType)
|
||||
{
|
||||
return (fingerprintOf(descriptor(table, backendType)));
|
||||
return ("sha256:" + TableSchemaFingerprint.of(table, backendType).getHash());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
|
||||
** An absent algorithm identifies manifests written before RC.6. Unknown
|
||||
** algorithms must not silently fall back to a different fingerprint.
|
||||
*******************************************************************************/
|
||||
public static String fingerprintOf(Map<String, Object> descriptor)
|
||||
public static boolean matches(BackupManifest.TableEntry entry, QTableMetaData table, String backendType)
|
||||
{
|
||||
StringBuilder sb = new StringBuilder();
|
||||
canonical(descriptor, sb);
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
|
||||
if(entry.fingerprint() == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
String algorithm = entry.fingerprintAlgorithm();
|
||||
if(algorithm == null || LEGACY_ALGORITHM.equals(algorithm))
|
||||
{
|
||||
return (entry.fingerprint().equals(LegacyTableFingerprint.of(table, backendType)));
|
||||
}
|
||||
return (CORE_ALGORITHM.equals(algorithm) && entry.fingerprint().equals(of(table, backendType)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The fingerprint of a whole instance's tables: a hash of the table fingerprints.
|
||||
** Normalize both descriptor versions to field-name maps for restore's
|
||||
** compatibility analysis. Core fields use a list and explicit flag names.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
public static Map<String, Object> fields(Map<String, Object> descriptor)
|
||||
{
|
||||
Object fields = descriptor == null ? null : descriptor.get("fields");
|
||||
if(fields instanceof Map<?, ?> map)
|
||||
{
|
||||
return ((Map<String, Object>) map);
|
||||
}
|
||||
Map<String, Object> result = new LinkedHashMap<>();
|
||||
if(fields instanceof List<?> list)
|
||||
{
|
||||
for(Object item : list)
|
||||
{
|
||||
Map<String, Object> field = (Map<String, Object>) item;
|
||||
Map<String, Object> normalized = new LinkedHashMap<>();
|
||||
normalized.put("type", field.get("type"));
|
||||
normalized.put("required", field.get("isRequired"));
|
||||
normalized.put("maxLength", field.get("maxLength"));
|
||||
normalized.put("pvs", field.get("possibleValueSource"));
|
||||
normalized.put("heavy", field.get("isHeavy"));
|
||||
normalized.put("sensitive", field.get("isSensitive"));
|
||||
result.put((String) field.get("name"), normalized);
|
||||
}
|
||||
}
|
||||
return (result);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The instance fingerprint remains a hash of the ordered table fingerprints.
|
||||
*******************************************************************************/
|
||||
public static String combine(List<String> tableFingerprints)
|
||||
{
|
||||
return ("sha256:" + HashingOutputStream.sha256Hex(String.join("\n", tableFingerprints).getBytes(StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private static void canonical(Object o, StringBuilder sb)
|
||||
{
|
||||
if(o == null)
|
||||
{
|
||||
sb.append("null");
|
||||
}
|
||||
else if(o instanceof Map<?, ?> map)
|
||||
{
|
||||
sb.append('{');
|
||||
boolean first = true;
|
||||
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
|
||||
{
|
||||
if(!first)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
first = false;
|
||||
sb.append('"').append(e.getKey()).append("\":");
|
||||
canonical(e.getValue(), sb);
|
||||
}
|
||||
sb.append('}');
|
||||
}
|
||||
else if(o instanceof List<?> list)
|
||||
{
|
||||
sb.append('[');
|
||||
for(int i = 0; i < list.size(); i++)
|
||||
{
|
||||
if(i > 0)
|
||||
{
|
||||
sb.append(',');
|
||||
}
|
||||
canonical(list.get(i), sb);
|
||||
}
|
||||
sb.append(']');
|
||||
}
|
||||
else if(o instanceof Number || o instanceof Boolean)
|
||||
{
|
||||
sb.append(o);
|
||||
}
|
||||
else
|
||||
{
|
||||
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -105,7 +105,8 @@ public class RestoreEngine
|
||||
Map<String, String> targets = new LinkedHashMap<>();
|
||||
QInstance instance = QContext.getQInstance();
|
||||
|
||||
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE))
|
||||
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE)
|
||||
|| (request.getWriter().usesCoreRestoreFlag() && (QContext.getQSession() == null || !QContext.getQSession().hasPermission(BackupsPermissions.RESTORE))))
|
||||
{
|
||||
errors.add("Restoring needs the " + BackupsPermissions.RESTORE + " permission.");
|
||||
}
|
||||
|
||||
@@ -341,14 +341,14 @@ public class RestorePass
|
||||
String backendType = instance.getBackend(c.target.getBackendName()) == null ? null : instance.getBackend(c.target.getBackendName()).getBackendType();
|
||||
RestoreOutcome.TableOutcome o = c.outcome;
|
||||
|
||||
if(c.entry.fingerprint() != null && c.entry.fingerprint().equals(TableFingerprints.of(c.target, backendType)))
|
||||
if(TableFingerprints.matches(c.entry, c.target, backendType))
|
||||
{
|
||||
o.setSchemaStatus("MATCH");
|
||||
return;
|
||||
}
|
||||
|
||||
Map<String, Object> backupFields = c.entry.schema() == null ? Map.of() : (Map<String, Object>) c.entry.schema().getOrDefault("fields", Map.of());
|
||||
Map<String, Object> targetFields = (Map<String, Object>) TableFingerprints.descriptor(c.target, backendType).get("fields");
|
||||
Map<String, Object> backupFields = TableFingerprints.fields(c.entry.schema());
|
||||
Map<String, Object> targetFields = TableFingerprints.fields(TableFingerprints.descriptor(c.target, backendType));
|
||||
boolean incompatible = false;
|
||||
boolean different = false;
|
||||
|
||||
|
||||
@@ -28,9 +28,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
|
||||
/*******************************************************************************
|
||||
** How a restore writes (spec 5.3 and 5.4). The default implementation uses the
|
||||
** plain Insert, Update and Delete actions, and selects the core RestoreWriteFlag
|
||||
** (QRun-IO/qqq#1111) and the core key-sequence resync (#1112) reflectively when the
|
||||
** running QQQ has them. A backend specific writer can be supplied on the request.
|
||||
** plain Insert, Update and Delete actions with core RestoreWriteFlag and typed
|
||||
** key-sequence resync. A backend specific writer can be supplied on the request.
|
||||
**
|
||||
** Whatever the implementation, record security, field security, unique keys and
|
||||
** required-field validation still apply to every write.
|
||||
|
||||
@@ -18,7 +18,6 @@ package com.kingsrook.qbits.backups.restore;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.lang.reflect.Method;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.LinkedHashMap;
|
||||
@@ -30,8 +29,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.ActionFlag;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
@@ -41,48 +40,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.SequenceResyncResult;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The default RestoreWriter: the standard Insert, Update and Delete actions.
|
||||
**
|
||||
** Stand-ins (replace with the core API when QQQ 4.1.0-RC.5 ships):
|
||||
** <ul>
|
||||
** <li>RestoreWriteFlag (QRun-IO/qqq#1111) is created reflectively when the class is
|
||||
** present. Without it, the DML audit is omitted (a summary audit is written per
|
||||
** table instead), but created and modified dates may be set by the normal defaults.</li>
|
||||
** <li>resyncKeySequences (QRun-IO/qqq#1112) is called reflectively on the backend
|
||||
** module when it exists; otherwise it is a no-op that the Review reports.</li>
|
||||
** </ul>
|
||||
** The default RestoreWriter using core restore flags and key sequence resync.
|
||||
*******************************************************************************/
|
||||
public class StandardRestoreWriter implements RestoreWriter
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(StandardRestoreWriter.class);
|
||||
|
||||
private static final String FLAG_CLASS = "com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag";
|
||||
|
||||
private static final Class<?> FLAG = lookup(FLAG_CLASS);
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Class<?> lookup(String name)
|
||||
{
|
||||
try
|
||||
{
|
||||
return (Class.forName(name));
|
||||
}
|
||||
catch(ClassNotFoundException | LinkageError e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@@ -148,27 +113,15 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create the core flag (when the running QQQ has it) and hand it to the input.
|
||||
** Apply the core restore behavior requested by the caller.
|
||||
*******************************************************************************/
|
||||
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink) throws QException
|
||||
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink)
|
||||
{
|
||||
if(FLAG == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
Object flag = FLAG.getDeclaredConstructor().newInstance();
|
||||
FLAG.getMethod("withSuppressAutomations", boolean.class).invoke(flag, options.getSuppressAutomations());
|
||||
FLAG.getMethod("withRunCustomizers", boolean.class).invoke(flag, options.getRunCustomizers());
|
||||
FLAG.getMethod("withPerRecordAudit", boolean.class).invoke(flag, options.getPerRecordAudit());
|
||||
FLAG.getMethod("withPublishEvents", boolean.class).invoke(flag, options.getPublishEvents());
|
||||
sink.accept((ActionFlag) flag);
|
||||
}
|
||||
catch(ReflectiveOperationException | ClassCastException e)
|
||||
{
|
||||
throw (new QException("Could not use the core restore flag: " + e.getMessage(), e));
|
||||
}
|
||||
sink.accept(new RestoreWriteFlag()
|
||||
.withSuppressAutomations(options.getSuppressAutomations())
|
||||
.withRunCustomizers(options.getRunCustomizers())
|
||||
.withPerRecordAudit(options.getPerRecordAudit())
|
||||
.withPublishEvents(options.getPublishEvents()));
|
||||
}
|
||||
|
||||
|
||||
@@ -179,7 +132,7 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public boolean usesCoreRestoreFlag()
|
||||
{
|
||||
return (FLAG != null);
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
@@ -190,24 +143,7 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public boolean canResyncSequences()
|
||||
{
|
||||
return (resyncMethod() != null);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Method resyncMethod()
|
||||
{
|
||||
try
|
||||
{
|
||||
return (QBackendModuleInterface.class.getMethod("resyncKeySequences", QBackendMetaData.class, Collection.class));
|
||||
}
|
||||
catch(NoSuchMethodException | LinkageError e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
@@ -218,12 +154,6 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
@Override
|
||||
public List<String> resyncSequences(Collection<QTableMetaData> tables) throws QException
|
||||
{
|
||||
Method method = resyncMethod();
|
||||
if(method == null)
|
||||
{
|
||||
return (List.of());
|
||||
}
|
||||
|
||||
Map<String, List<QTableMetaData>> byBackend = new LinkedHashMap<>();
|
||||
for(QTableMetaData table : tables)
|
||||
{
|
||||
@@ -234,23 +164,14 @@ public class StandardRestoreWriter implements RestoreWriter
|
||||
for(Map.Entry<String, List<QTableMetaData>> entry : byBackend.entrySet())
|
||||
{
|
||||
QBackendMetaData backend = QContext.getQInstance().getBackend(entry.getKey());
|
||||
try
|
||||
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
|
||||
for(SequenceResyncResult result : module.resyncKeySequences(backend, entry.getValue()))
|
||||
{
|
||||
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
|
||||
for(Object result : (List<?>) method.invoke(module, backend, entry.getValue()))
|
||||
if(result.getStatus() == SequenceResyncResult.Status.RESYNCED)
|
||||
{
|
||||
Object status = result.getClass().getMethod("getStatus").invoke(result);
|
||||
if("RESYNCED".equals(String.valueOf(status)))
|
||||
{
|
||||
resynced.add(String.valueOf(result.getClass().getMethod("getTableName").invoke(result)));
|
||||
}
|
||||
resynced.add(result.getTableName());
|
||||
}
|
||||
}
|
||||
catch(ReflectiveOperationException e)
|
||||
{
|
||||
LOG.warn("Could not resync key sequences", e, logPair("backend", entry.getKey()));
|
||||
throw (new QException("Could not resync key sequences on backend " + entry.getKey() + ": " + e.getMessage(), e));
|
||||
}
|
||||
}
|
||||
return (resynced);
|
||||
}
|
||||
|
||||
@@ -35,8 +35,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.events.BackupEvents;
|
||||
import com.kingsrook.qbits.backups.codec.TableArtifactWriter;
|
||||
import com.kingsrook.qbits.backups.events.BackupEvents;
|
||||
import com.kingsrook.qbits.backups.files.FileCopier;
|
||||
import com.kingsrook.qbits.backups.files.FileReference;
|
||||
import com.kingsrook.qbits.backups.files.FileReferenceProvider;
|
||||
@@ -398,7 +398,7 @@ public class BackupEngine
|
||||
keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
|
||||
keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
|
||||
manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
|
||||
fingerprint, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
|
||||
fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
|
||||
done++;
|
||||
catalogProgress(result, done, rows, bytesRaw, bytesStored);
|
||||
if(request.getCancelRequested().getAsBoolean())
|
||||
|
||||
@@ -18,30 +18,20 @@ package com.kingsrook.qbits.backups.run;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.util.List;
|
||||
import java.util.function.Consumer;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.KeysetPager;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reads the rows of one table in primary key order, in keyset pages (WHERE pk >
|
||||
** lastKey ORDER BY pk LIMIT n), so the cost stays flat with depth and a read can
|
||||
** resume from a key. A read raw: hidden fields, passwords and heavy fields are
|
||||
** returned as stored, and possible values are not translated.
|
||||
**
|
||||
** This is an internal stand-in for the QQQ core KeysetPager and QueryInput
|
||||
** withRawRead (core PR feature/keyset-pager-raw-read); the run engine only
|
||||
** depends on this interface, so it is swapped when that ships in the next RC.
|
||||
** Reads rows in primary key order through core KeysetPager. Each bounded page
|
||||
** uses the supplied transaction and returns hidden, password and heavy fields
|
||||
** as stored, without possible-value translation or display generation.
|
||||
*******************************************************************************/
|
||||
public interface TableReader
|
||||
{
|
||||
@@ -81,48 +71,25 @@ public interface TableReader
|
||||
@Override
|
||||
public void read(QTableMetaData table, String filterJson, Serializable startAfterKey, int pageSize, QBackendTransaction transaction, Consumer<QRecord> sink) throws QException
|
||||
{
|
||||
String pk = table.getPrimaryKeyField();
|
||||
Serializable lastKey = startAfterKey;
|
||||
QQueryFilter user = parseFilter(filterJson);
|
||||
|
||||
while(true)
|
||||
KeysetPager pager = KeysetPager.forTable(table.getName())
|
||||
.withFilter(parseFilter(filterJson))
|
||||
.startingAfter(startAfterKey)
|
||||
.withPageSize(pageSize)
|
||||
.withTransaction(transaction)
|
||||
.withQueryInputCustomizer(input ->
|
||||
{
|
||||
input.setShouldOmitHiddenFields(false);
|
||||
input.setShouldMaskPasswords(false);
|
||||
input.setShouldFetchHeavyFields(true);
|
||||
input.setShouldTranslatePossibleValues(false);
|
||||
input.setShouldGenerateDisplayValues(false);
|
||||
input.withPermissionBypass("backups run");
|
||||
});
|
||||
if(!pager.isPrimaryKeyOrderable())
|
||||
{
|
||||
QQueryFilter filter = new QQueryFilter();
|
||||
if(lastKey != null)
|
||||
{
|
||||
filter.addCriteria(new QFilterCriteria(pk, QCriteriaOperator.GREATER_THAN, List.of(lastKey)));
|
||||
}
|
||||
if(user != null)
|
||||
{
|
||||
filter.addSubFilter(user);
|
||||
}
|
||||
filter.withOrderBy(new QFilterOrderBy(pk, true));
|
||||
filter.setLimit(pageSize);
|
||||
|
||||
QueryInput queryInput = new QueryInput(table.getName());
|
||||
queryInput.setFilter(filter);
|
||||
queryInput.setShouldOmitHiddenFields(false);
|
||||
queryInput.setShouldMaskPasswords(false);
|
||||
queryInput.setShouldFetchHeavyFields(true);
|
||||
queryInput.setShouldTranslatePossibleValues(false);
|
||||
queryInput.setShouldGenerateDisplayValues(false);
|
||||
if(transaction != null)
|
||||
{
|
||||
queryInput.setTransaction(transaction);
|
||||
}
|
||||
queryInput.withPermissionBypass("backups run");
|
||||
|
||||
List<QRecord> page = new QueryAction().execute(queryInput).getRecords();
|
||||
for(QRecord record : page)
|
||||
{
|
||||
sink.accept(record);
|
||||
lastKey = record.getValue(pk);
|
||||
}
|
||||
if(page.size() < pageSize)
|
||||
{
|
||||
return;
|
||||
}
|
||||
throw (new QException("Backup requires an orderable primary key on table " + table.getName()));
|
||||
}
|
||||
pager.forEachPage(page -> page.forEach(sink));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,7 +50,7 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
** run is a full-instance, system-session backup (securityScope FULL, requested by
|
||||
** system:scheduler), which is why the plan owner is validated first: a plan whose
|
||||
** owner has gone, or lost backups.admin, is not run silently. It raises
|
||||
** backups.plan.owner_invalid, marks the plan OWNER_INVALID, and stops.
|
||||
** backups.plan.owner-invalid, marks the plan OWNER_INVALID, and stops.
|
||||
**
|
||||
** Everything goes through the run coordinator, so a trigger that finds the plan
|
||||
** already running is skipped visibly, and a full instance queues the run.
|
||||
@@ -141,11 +141,11 @@ public class BackupScheduledRunProcess implements BackendStep
|
||||
if(problem != null)
|
||||
{
|
||||
catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaStatus", "OWNER_INVALID"));
|
||||
BackupsRuntime.getAlertSink().raise("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
|
||||
BackupsRuntime.getAlertSink().raise("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"), BackupAlertSink.Severity.WARNING,
|
||||
"A scheduled backup did not run: " + problem, Map.of("plan", plan.getValueString("planKey")));
|
||||
throw (new QUserFacingException("The scheduled backup did not run: " + problem + "."));
|
||||
}
|
||||
BackupsRuntime.getAlertSink().resolve("backups.plan.owner_invalid", "backups.plan.owner_invalid:" + plan.getValueString("planKey"));
|
||||
BackupsRuntime.getAlertSink().resolve("backups.plan.owner-invalid", "backups.plan.owner-invalid:" + plan.getValueString("planKey"));
|
||||
|
||||
BackupRunResult result = new BackupRunCoordinator().runPlan(plan, "SCHEDULED");
|
||||
output.addValue("status", result.getStatus());
|
||||
|
||||
@@ -33,8 +33,8 @@ import java.util.Optional;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.kingsrook.qbits.backups.BackupsQBitConfig;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
|
||||
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
|
||||
import com.kingsrook.qbits.backups.codec.ArtifactTransform;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.codec.JsonlRowCodec;
|
||||
@@ -274,7 +274,7 @@ public class BackupVerifier
|
||||
List<String> excluded = entry.excludedFields() == null ? List.of() : entry.excludedFields().stream().map(BackupManifest.ExcludedField::name).toList();
|
||||
codec = new JsonlRowCodec(table, excluded);
|
||||
String backendType = qInstance.getBackend(table.getBackendName()) == null ? null : qInstance.getBackend(table.getBackendName()).getBackendType();
|
||||
roundTrip = TableFingerprints.of(table, backendType).equals(entry.fingerprint());
|
||||
roundTrip = TableFingerprints.matches(entry, table, backendType);
|
||||
if(!roundTrip)
|
||||
{
|
||||
findings.warn("Table " + entry.name() + " has changed since this backup; rows were parsed but not checked for an exact round trip");
|
||||
|
||||
@@ -106,7 +106,7 @@ public final class BackupsTestUtils
|
||||
new BackupsQBitProducer().withQBitConfig(config).withNamespace("backups").produce(qInstance);
|
||||
|
||||
new QInstanceValidator().validate(qInstance);
|
||||
QContext.init(qInstance, new QSession());
|
||||
QContext.init(qInstance, new QSession().withPermissions("backups.restore"));
|
||||
return (qInstance);
|
||||
}
|
||||
|
||||
|
||||
@@ -18,16 +18,24 @@ package com.kingsrook.qbits.backups.alerts;
|
||||
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.backups.BackupsRuntime;
|
||||
import com.kingsrook.qbits.backups.VerifyTestSupport;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlertTypeMetaData;
|
||||
import com.kingsrook.qbits.backups.alerts.fakeapi.QAlerts;
|
||||
import com.kingsrook.qbits.backups.sla.SlaChecker;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
|
||||
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
|
||||
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
@@ -35,14 +43,10 @@ import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Alerts qbit adapter, bound reflectively. The SPI is not in QQQ 4.1.0-RC.4,
|
||||
** so these tests run it against a test double with the same shape as the real
|
||||
** com.kingsrook.qqq.backend.core.alerts package (QRun-IO/qqq PR #1108).
|
||||
** The backup adapter against the released core alerts API and a registered sink.
|
||||
*******************************************************************************/
|
||||
class AlertsQBitAlertSinkTest
|
||||
{
|
||||
private static final String FAKE_API = "com.kingsrook.qbits.backups.alerts.fakeapi";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
@@ -52,9 +56,11 @@ class AlertsQBitAlertSinkTest
|
||||
void setUp() throws Exception
|
||||
{
|
||||
VerifyTestSupport.setUp(3);
|
||||
QAlerts.reset();
|
||||
QAlertTypeMetaData.ADDED.clear();
|
||||
AlertsInstanceMetaData.installed = true;
|
||||
RecordingSink.raised.clear();
|
||||
RecordingSink.resolved.clear();
|
||||
RecordingSink.heartbeats.clear();
|
||||
RecordingSink.failWith = null;
|
||||
AlertsInstanceMetaData.ofOrWithNew(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
|
||||
}
|
||||
|
||||
|
||||
@@ -65,8 +71,8 @@ class AlertsQBitAlertSinkTest
|
||||
@AfterEach
|
||||
void tearDown()
|
||||
{
|
||||
AlertsInstanceMetaData.installed = false;
|
||||
QAlerts.reset();
|
||||
BackupsRuntime.reset();
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
|
||||
@@ -78,7 +84,7 @@ class AlertsQBitAlertSinkTest
|
||||
@Test
|
||||
void testRaiseResolveHeartbeat()
|
||||
{
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
Map<String, String> detail = new LinkedHashMap<>();
|
||||
detail.put("plan", "nightly");
|
||||
detail.put("slaHours", "26");
|
||||
@@ -87,60 +93,90 @@ class AlertsQBitAlertSinkTest
|
||||
sink.resolve("backups.plan.missed", "backups.plan.missed:nightly");
|
||||
sink.heartbeat("backups.plan:nightly", 93600, 0);
|
||||
|
||||
assertThat(QAlerts.RAISED).hasSize(1);
|
||||
var signal = QAlerts.RAISED.get(0);
|
||||
assertThat(signal.alertTypeName).isEqualTo("backups.plan.missed");
|
||||
assertThat(signal.dedupKey).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(signal.severity.name()).isEqualTo("CRITICAL");
|
||||
assertThat(signal.title).isEqualTo("A backup did not run in time");
|
||||
assertThat(signal.summary).isEqualTo("plan=nightly, slaHours=26");
|
||||
assertThat(signal.attributes).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
|
||||
assertThat(QAlerts.RESOLVED).containsExactly("backups.plan.missed|backups.plan.missed:nightly|Cleared by the Backups qbit");
|
||||
assertThat(QAlerts.HEARTBEATS).containsExactly("backups.plan:nightly");
|
||||
assertThat(RecordingSink.raised).hasSize(1);
|
||||
var signal = RecordingSink.raised.get(0);
|
||||
assertThat(signal.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(signal.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(signal.getSeverity().name()).isEqualTo("CRITICAL");
|
||||
assertThat(signal.getTitle()).isEqualTo("A backup did not run in time");
|
||||
assertThat(signal.getSummary()).isEqualTo("plan=nightly, slaHours=26");
|
||||
assertThat(signal.getAttributes()).containsEntry("plan", "nightly").containsEntry("slaHours", "26");
|
||||
assertThat(RecordingSink.resolved).singleElement().satisfies(resolution ->
|
||||
{
|
||||
assertThat(resolution.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(resolution.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(resolution.getNote()).isEqualTo("Cleared by the Backups qbit");
|
||||
});
|
||||
assertThat(RecordingSink.heartbeats).containsExactly("backups.plan:nightly");
|
||||
|
||||
sink.raise("backups.storage.debris", "k", BackupAlertSink.Severity.INFO, "no detail", null);
|
||||
assertThat(QAlerts.RAISED.get(1).summary).isNull();
|
||||
assertThat(RecordingSink.raised.get(1).getSummary()).isNull();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Without the Alerts qbit installed (no alerts meta-data on the instance) every
|
||||
** Without a configured sink, even with alert type metadata present, every
|
||||
** call is a no-op.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testNoOpWithoutTheAlertsQBit()
|
||||
{
|
||||
AlertsInstanceMetaData.installed = false;
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink(FAKE_API);
|
||||
sink.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
sink.resolve("t", "k");
|
||||
sink.heartbeat("m", 1, 1);
|
||||
assertThat(QAlerts.RAISED).isEmpty();
|
||||
assertThat(QAlerts.RESOLVED).isEmpty();
|
||||
assertThat(QAlerts.HEARTBEATS).isEmpty();
|
||||
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(null);
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
long failures = QAlerts.getProducerFailureCount();
|
||||
sink.raise(null, "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
sink.resolve(null, "k");
|
||||
sink.heartbeat(null, 1, 1);
|
||||
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures);
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
assertThat(RecordingSink.resolved).isEmpty();
|
||||
assertThat(RecordingSink.heartbeats).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A broken or missing SPI never throws into the backup.
|
||||
** The same adapter follows the current instance and a sink installed later.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLateInstallationAndInstanceSwitch()
|
||||
{
|
||||
var original = QContext.getQInstance();
|
||||
QContext.setQInstance(new QInstance());
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
assertThat(sink.isAlertsQBitInstalled()).isFalse();
|
||||
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
sink.registerAlertTypes(QContext.getQInstance());
|
||||
assertThat(sink.isAlertsQBitInstalled()).isFalse();
|
||||
AlertsInstanceMetaData.of(QContext.getQInstance()).withSink(new QCodeReference(RecordingSink.class));
|
||||
sink.raise("backups.plan.missed", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
assertThat(RecordingSink.raised).hasSize(1);
|
||||
assertThat(RecordingSink.raised.get(0).getSummary()).isNull();
|
||||
QContext.setQInstance(original);
|
||||
assertThat(sink.isAlertsQBitInstalled()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Sink and adapter failures never throw into the backup.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFailuresAreSwallowed()
|
||||
{
|
||||
AlertsQBitAlertSink missing = new AlertsQBitAlertSink("no.such.package");
|
||||
missing.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
missing.resolve("t", "k");
|
||||
missing.heartbeat("m", 1, 1);
|
||||
assertThat(missing.registerAlertTypes(QContext.getQInstance())).isZero();
|
||||
|
||||
QAlerts.failWith = new IllegalStateException("alerts down");
|
||||
AlertsQBitAlertSink broken = new AlertsQBitAlertSink(FAKE_API);
|
||||
long failures = QAlerts.getProducerFailureCount();
|
||||
RecordingSink.failWith = new IllegalStateException("alerts down");
|
||||
AlertsQBitAlertSink broken = new AlertsQBitAlertSink();
|
||||
broken.raise("t", "k", BackupAlertSink.Severity.INFO, "x", Map.of());
|
||||
broken.resolve("t", "k");
|
||||
broken.heartbeat("m", 1, 1);
|
||||
assertThat(QAlerts.RAISED).isEmpty();
|
||||
assertThat(RecordingSink.raised).isEmpty();
|
||||
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures + 3);
|
||||
broken.raise("t", "k", null, "bad severity", Map.of());
|
||||
broken.raise("t", "k", null, "bad severity again", Map.of());
|
||||
assertThat(broken.registerAlertTypes(null)).isZero();
|
||||
}
|
||||
|
||||
|
||||
@@ -149,41 +185,41 @@ class AlertsQBitAlertSinkTest
|
||||
** The backup alert types are declared with the Alerts qbit.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testRegisterAlertTypes()
|
||||
void testRegisterAlertTypes() throws Exception
|
||||
{
|
||||
int added = new AlertsQBitAlertSink(FAKE_API).registerAlertTypes(QContext.getQInstance());
|
||||
QInstance instance = new QInstance();
|
||||
AlertsQBitAlertSink sink = new AlertsQBitAlertSink();
|
||||
int added = sink.registerAlertTypes(instance);
|
||||
var types = new ArrayList<>(AlertsInstanceMetaData.of(instance).getAlertTypes().values());
|
||||
assertThat(added).isEqualTo(BackupAlerts.alertTypes().size()).isEqualTo(9);
|
||||
assertThat(QAlertTypeMetaData.ADDED).extracting(t -> t.name).containsExactly(
|
||||
"backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing", "backups.verify.failed",
|
||||
"backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(QAlertTypeMetaData.ADDED).allSatisfy(t ->
|
||||
assertThat(types).extracting(t -> t.getName()).containsExactly(
|
||||
"backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing", "backups.verify.failed",
|
||||
"backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(types).allSatisfy(t ->
|
||||
{
|
||||
assertThat(t.groupName).isEqualTo("Backups");
|
||||
assertThat(t.howToFix).isNotBlank();
|
||||
assertThat(t.label).isNotBlank();
|
||||
assertThat(t.getGroupName()).isEqualTo("Backups");
|
||||
assertThat(t.getHowToFix()).isNotBlank();
|
||||
assertThat(t.getLabel()).isNotBlank();
|
||||
});
|
||||
assertThat(QAlertTypeMetaData.ADDED.get(2).defaultSeverity.name()).isEqualTo("CRITICAL");
|
||||
assertThat(types.get(2).getDefaultSeverity().name()).isEqualTo("CRITICAL");
|
||||
assertThat(sink.registerAlertTypes(instance)).isZero();
|
||||
new QInstanceValidator().validate(QContext.getQInstance());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Detection: the SPI is present or not by class name; the real SPI is not in
|
||||
** RC.4, so nothing is bound by default there.
|
||||
** Default binding preserves a host-provided backup sink.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testDetectionAndBinding()
|
||||
{
|
||||
assertThat(AlertsQBitAlertSink.isPresent(FAKE_API)).isTrue();
|
||||
assertThat(AlertsQBitAlertSink.isPresent("no.such.package")).isFalse();
|
||||
|
||||
boolean realSpiPresent = AlertsQBitAlertSink.isPresent(AlertsQBitAlertSink.DEFAULT_API_PACKAGE);
|
||||
assertThat(AlertsQBitAlertSink.bindIfPresent()).isEqualTo(realSpiPresent);
|
||||
if(realSpiPresent)
|
||||
{
|
||||
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
|
||||
assertThat(AlertsQBitAlertSink.bindIfPresent()).as("an already bound sink is not replaced").isFalse();
|
||||
}
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
|
||||
assertThat(BackupsRuntime.getAlertSink()).isSameAs(VerifyTestSupport.sink);
|
||||
BackupsRuntime.setAlertSink(null);
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isTrue();
|
||||
assertThat(BackupsRuntime.getAlertSink()).isInstanceOf(AlertsQBitAlertSink.class);
|
||||
assertThat(AlertsQBitAlertSink.bindIfAbsent()).isFalse();
|
||||
}
|
||||
|
||||
|
||||
@@ -194,22 +230,80 @@ class AlertsQBitAlertSinkTest
|
||||
@Test
|
||||
void testSlaAlertsThroughTheAdapter() throws Exception
|
||||
{
|
||||
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink(FAKE_API));
|
||||
BackupsRuntime.setAlertSink(new AlertsQBitAlertSink());
|
||||
QRecord plan = VerifyTestSupport.plan(Map.of());
|
||||
VerifyTestSupport.backup(plan);
|
||||
assertThat(QAlerts.HEARTBEATS).contains("backups.plan:nightly");
|
||||
assertThat(RecordingSink.heartbeats).contains("backups.plan:nightly");
|
||||
|
||||
VerifyTestSupport.clock.advance(Duration.ofHours(30));
|
||||
new SlaChecker().check(VerifyTestSupport.reload(plan));
|
||||
assertThat(QAlerts.RAISED).anySatisfy(s ->
|
||||
assertThat(RecordingSink.raised).anySatisfy(s ->
|
||||
{
|
||||
assertThat(s.alertTypeName).isEqualTo("backups.plan.missed");
|
||||
assertThat(s.dedupKey).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(s.severity.name()).isEqualTo("ERROR");
|
||||
assertThat(s.getAlertTypeName()).isEqualTo("backups.plan.missed");
|
||||
assertThat(s.getDedupKey()).isEqualTo("backups.plan.missed:nightly");
|
||||
assertThat(s.getSeverity().name()).isEqualTo("ERROR");
|
||||
});
|
||||
|
||||
VerifyTestSupport.backup(plan);
|
||||
new SlaChecker().check(VerifyTestSupport.reload(plan));
|
||||
assertThat(QAlerts.RESOLVED).anyMatch(r -> r.startsWith("backups.plan.missed|backups.plan.missed:nightly|"));
|
||||
assertThat(RecordingSink.resolved).anyMatch(r -> r.getAlertTypeName().equals("backups.plan.missed") && r.getDedupKey().equals("backups.plan.missed:nightly"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Sink loaded by the real QCodeLoader through AlertsInstanceMetaData.
|
||||
*******************************************************************************/
|
||||
public static class RecordingSink implements AlertSinkInterface
|
||||
{
|
||||
static final List<AlertSignal> raised = new ArrayList<>();
|
||||
static final List<AlertResolution> resolved = new ArrayList<>();
|
||||
static final List<String> heartbeats = new ArrayList<>();
|
||||
static RuntimeException failWith;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void raise(AlertSignal signal)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
raised.add(signal);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void resolve(AlertResolution resolution)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
resolved.add(resolution);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** {@inheritDoc}
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public void heartbeat(String heartbeatKey, Instant at)
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
heartbeats.add(heartbeatKey);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -70,8 +70,8 @@ class BackupAlertsTest
|
||||
void testTheSpecTable()
|
||||
{
|
||||
Set<String> names = BackupAlerts.alertTypes().stream().map(BackupAlerts.AlertType::name).collect(Collectors.toSet());
|
||||
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing_in_storage", "backups.files.missing",
|
||||
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner_invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(names).containsExactlyInAnyOrder("backups.plan.missed", "backups.run.failed", "backups.run.missing-in-storage", "backups.files.missing",
|
||||
"backups.verify.failed", "backups.verify.overdue", "backups.plan.owner-invalid", "backups.storage.debris", "backups.restore.completed");
|
||||
assertThat(BackupAlerts.alertTypes()).allSatisfy(t -> assertThat(t.howToFix()).isNotBlank());
|
||||
assertThat(BackupAlerts.alertTypes().stream().filter(t -> t.name().equals(BackupAlerts.RUN_MISSING)).findFirst().orElseThrow().defaultSeverity())
|
||||
.isEqualTo(BackupAlertSink.Severity.CRITICAL);
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI (QRun-IO/qqq PR #1108): the same shape as
|
||||
** com.kingsrook.qqq.backend.core.alerts.AlertSeverity.
|
||||
*******************************************************************************/
|
||||
public enum AlertSeverity
|
||||
{
|
||||
INFO,
|
||||
WARNING,
|
||||
ERROR,
|
||||
CRITICAL
|
||||
}
|
||||
@@ -1,96 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI AlertSignal (only what the adapter uses).
|
||||
*******************************************************************************/
|
||||
public class AlertSignal
|
||||
{
|
||||
public final String alertTypeName;
|
||||
public String dedupKey;
|
||||
public AlertSeverity severity;
|
||||
public String title;
|
||||
public String summary;
|
||||
public final Map<String, String> attributes = new LinkedHashMap<>();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor.
|
||||
*******************************************************************************/
|
||||
public AlertSignal(String alertTypeName)
|
||||
{
|
||||
this.alertTypeName = alertTypeName;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setDedupKey(String dedupKey)
|
||||
{
|
||||
this.dedupKey = dedupKey;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setSeverity(AlertSeverity severity)
|
||||
{
|
||||
this.severity = severity;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setTitle(String title)
|
||||
{
|
||||
this.title = title;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter.
|
||||
*******************************************************************************/
|
||||
public void setSummary(String summary)
|
||||
{
|
||||
this.summary = summary;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent attribute.
|
||||
*******************************************************************************/
|
||||
public AlertSignal withAttribute(String key, String value)
|
||||
{
|
||||
attributes.put(key, value);
|
||||
return (this);
|
||||
}
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI AlertsInstanceMetaData: of(instance) is
|
||||
** non-null only when a test says the Alerts qbit is installed.
|
||||
*******************************************************************************/
|
||||
public class AlertsInstanceMetaData
|
||||
{
|
||||
public static boolean installed = false;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The alerts meta-data of an instance, or null.
|
||||
*******************************************************************************/
|
||||
public static AlertsInstanceMetaData of(QInstance qInstance)
|
||||
{
|
||||
return (installed ? new AlertsInstanceMetaData() : null);
|
||||
}
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI QAlertTypeMetaData: records the types
|
||||
** added to an instance.
|
||||
*******************************************************************************/
|
||||
public class QAlertTypeMetaData
|
||||
{
|
||||
public static final List<QAlertTypeMetaData> ADDED = new ArrayList<>();
|
||||
|
||||
public String name;
|
||||
public String label;
|
||||
public String groupName;
|
||||
public String howToFix;
|
||||
public AlertSeverity defaultSeverity;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withName(String name)
|
||||
{
|
||||
this.name = name;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withLabel(String label)
|
||||
{
|
||||
this.label = label;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withGroupName(String groupName)
|
||||
{
|
||||
this.groupName = groupName;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withHowToFix(String howToFix)
|
||||
{
|
||||
this.howToFix = howToFix;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent.
|
||||
*******************************************************************************/
|
||||
public QAlertTypeMetaData withDefaultSeverity(AlertSeverity defaultSeverity)
|
||||
{
|
||||
this.defaultSeverity = defaultSeverity;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Add to the instance (recorded).
|
||||
*******************************************************************************/
|
||||
public void addSelfToInstance(QInstance qInstance)
|
||||
{
|
||||
ADDED.add(this);
|
||||
}
|
||||
}
|
||||
@@ -1,104 +0,0 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.alerts.fakeapi;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Test double of the Alerts producer SPI QAlerts: static calls that record what
|
||||
** they were given.
|
||||
*******************************************************************************/
|
||||
public final class QAlerts
|
||||
{
|
||||
public static final List<AlertSignal> RAISED = new ArrayList<>();
|
||||
public static final List<String> RESOLVED = new ArrayList<>();
|
||||
public static final List<String> HEARTBEATS = new ArrayList<>();
|
||||
|
||||
/** When set, every call throws it (the SPI is meant not to, but the adapter must survive). */
|
||||
public static RuntimeException failWith;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Utility class.
|
||||
*******************************************************************************/
|
||||
private QAlerts()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Forget everything.
|
||||
*******************************************************************************/
|
||||
public static void reset()
|
||||
{
|
||||
RAISED.clear();
|
||||
RESOLVED.clear();
|
||||
HEARTBEATS.clear();
|
||||
failWith = null;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Raise.
|
||||
*******************************************************************************/
|
||||
public static void raise(AlertSignal signal)
|
||||
{
|
||||
maybeFail();
|
||||
RAISED.add(signal);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve.
|
||||
*******************************************************************************/
|
||||
public static void resolve(String alertTypeName, String dedupKey, String note)
|
||||
{
|
||||
maybeFail();
|
||||
RESOLVED.add(alertTypeName + "|" + dedupKey + "|" + note);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Heartbeat.
|
||||
*******************************************************************************/
|
||||
public static void heartbeat(String heartbeatKey)
|
||||
{
|
||||
maybeFail();
|
||||
HEARTBEATS.add(heartbeatKey);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static void maybeFail()
|
||||
{
|
||||
if(failWith != null)
|
||||
{
|
||||
throw (failWith);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.manifest;
|
||||
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fingerprint algorithm dispatch is explicit; a marker cannot be ignored just
|
||||
** because its hash happens to match a different supported algorithm.
|
||||
*******************************************************************************/
|
||||
class TableFingerprintsTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** A frozen old hash matches only the legacy algorithm, and new hashes only
|
||||
** match the declared core algorithm. Cosmetic changes preserve both.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testAlgorithmDispatch() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
var table = QContext.getQInstance().getTable("blogPost");
|
||||
ObjectMapper mapper = new ObjectMapper();
|
||||
ObjectNode entry;
|
||||
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
|
||||
{
|
||||
entry = (ObjectNode) mapper.readTree(in).get("blogPost");
|
||||
}
|
||||
String oldHash = entry.get("fingerprint").asText();
|
||||
String coreHash = TableFingerprints.of(table, "memory");
|
||||
assertThat(coreHash).isNotEqualTo(oldHash);
|
||||
table.setLabel("Cosmetic label");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", TableFingerprints.LEGACY_ALGORITHM);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", TableFingerprints.CORE_ALGORITHM);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
entry.put("fingerprint", coreHash);
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
|
||||
entry.put("fingerprintAlgorithm", "unknown-v2");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
entry.remove("fingerprint");
|
||||
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.restore;
|
||||
|
||||
|
||||
import java.util.Map;
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
|
||||
import com.kingsrook.qbits.backups.manifest.BackupManifest;
|
||||
import com.kingsrook.qbits.backups.model.BackupsTables;
|
||||
import com.kingsrook.qbits.backups.run.BackupCatalog;
|
||||
import com.kingsrook.qbits.backups.run.BackupRunResult;
|
||||
import com.kingsrook.qbits.backups.verify.BackupVerifier;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exercise pre-RC.6 manifest fingerprints through verification and restore.
|
||||
** The schema fixture was frozen from the original stand-in, not recomputed
|
||||
** with the implementation being tested.
|
||||
*******************************************************************************/
|
||||
class LegacyManifestRestoreTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Load the blog and grant the permission required by the core restore flag.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupsTestUtils.loadBlog(30);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build an old-format manifest around real backup parts and re-seal its hash.
|
||||
*******************************************************************************/
|
||||
private BackupRunResult legacyBackup() throws Exception
|
||||
{
|
||||
BackupRunResult run = RestoreTestSupport.backup(Map.of());
|
||||
ObjectMapper mapper = new ObjectMapper();
|
||||
ObjectNode manifest = (ObjectNode) mapper.readTree(BackupsTestUtils.BACKUPS.getBytes(run.getManifestKey()));
|
||||
JsonNode legacy;
|
||||
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
|
||||
{
|
||||
legacy = mapper.readTree(in);
|
||||
}
|
||||
for(JsonNode table : manifest.get("tables"))
|
||||
{
|
||||
ObjectNode entry = (ObjectNode) table;
|
||||
JsonNode old = legacy.get(entry.get("name").asText());
|
||||
entry.remove("fingerprintAlgorithm");
|
||||
entry.set("schema", old.get("schema"));
|
||||
entry.set("fingerprint", old.get("fingerprint"));
|
||||
}
|
||||
byte[] bytes = mapper.writeValueAsBytes(manifest);
|
||||
BackupsTestUtils.BACKUPS.putBytes(run.getManifestKey(), bytes);
|
||||
new BackupCatalog().update(BackupsTables.RUN, run.getRunId(), Map.of("manifestSha256", HashingOutputStream.sha256Hex(bytes)));
|
||||
assertThat(BackupManifest.read(bytes).table("blogPost").fingerprintAlgorithm()).isNull();
|
||||
return (run);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Old hashes still receive exact codec verification and restore all data,
|
||||
** including original timestamps, after deletes, edits and additions.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLegacyManifestVerifiesAndRestores() throws Exception
|
||||
{
|
||||
String before = RestoreTestSupport.contentHash("blogPost");
|
||||
BackupRunResult run = legacyBackup();
|
||||
var verified = new BackupVerifier().verify(run.getRunId(), BackupVerifier.CONTENT, "MANUAL", "tester");
|
||||
assertThat(verified.status()).isEqualTo("PASSED");
|
||||
assertThat(verified.messages()).noneMatch(message -> message.contains("has changed since"));
|
||||
RestoreTestSupport.mutate();
|
||||
RestoreOutcome outcome = new RestoreEngine().execute(RestoreTestSupport.request(run, RestoreRequest.MODE_REPLACE, "blogPost").withConfirmationText("blogPost"));
|
||||
assertThat(outcome.getErrors()).isEmpty();
|
||||
assertThat(outcome.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MATCH");
|
||||
assertThat(RestoreTestSupport.contentHash("blogPost")).isEqualTo(before);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Legacy field maps and core field lists also compare correctly when hashes
|
||||
** differ: optional additions are compatible and type changes are rejected.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testLegacySchemaChangesAreAnalyzed() throws Exception
|
||||
{
|
||||
BackupRunResult run = legacyBackup();
|
||||
QContext.getQInstance().getTable("author").withField(new QFieldMetaData("nickname", QFieldType.STRING));
|
||||
QContext.getQInstance().getTable("blogPost").getField("title").setType(QFieldType.INTEGER);
|
||||
RestoreOutcome review = new RestoreEngine().review(RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "author", "blogPost"));
|
||||
assertThat(review.getTables().get("author").getSchemaStatus()).isEqualTo("COMPATIBLE");
|
||||
assertThat(review.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MISMATCH");
|
||||
}
|
||||
}
|
||||
@@ -187,4 +187,22 @@ class RestoreSafetyTest
|
||||
QContext.setQSession(new QSession().withPermissions("backups.view", "backups.restore"));
|
||||
assertThat(new RestoreEngine().plan(request).errors()).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The core restore permission is required even in anonymous demo mode, and
|
||||
** Review reports it before a restore can create a run or write any data.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testCoreRestorePermissionIsRequiredWhenAnonymous() throws Exception
|
||||
{
|
||||
BackupRunResult run = RestoreTestSupport.backup(Map.of());
|
||||
QContext.setQSession(new QSession());
|
||||
RestoreRequest request = RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "blogPost");
|
||||
assertThat(new RestoreEngine().plan(request).errors()).anyMatch(e -> e.contains("backups.restore"));
|
||||
assertThatThrownBy(() -> new RestoreEngine().execute(request)).hasMessageContaining("backups.restore");
|
||||
assertThatThrownBy(() -> new StandardRestoreWriter().insert(QContext.getQInstance().getTable("blogPost"),
|
||||
List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", 99)), request, null)).hasMessageContaining("backups.restore");
|
||||
}
|
||||
|
||||
}
|
||||
@@ -109,7 +109,9 @@ class BackupEngineTest
|
||||
BackupManifest.TableEntry posts = manifest.table("blogPost");
|
||||
assertThat(posts.rowCount()).isEqualTo(7);
|
||||
assertThat(posts.excludedFields()).extracting(BackupManifest.ExcludedField::name).containsExactly("secretNote");
|
||||
assertThat(posts.fingerprint()).startsWith("sha256:");
|
||||
assertThat(posts.fingerprintAlgorithm()).isEqualTo(com.kingsrook.qbits.backups.manifest.TableFingerprints.CORE_ALGORITHM);
|
||||
assertThat(posts.fingerprint()).isEqualTo("sha256:" + com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getHash());
|
||||
assertThat(posts.schema()).isEqualTo(com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getDescriptor());
|
||||
|
||||
BackupManifest.Part part = posts.parts().get(0);
|
||||
byte[] stored = BackupsTestUtils.BACKUPS.getBytes(result.getRunPrefix() + part.key());
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.backups.run;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.backups.BackupsTestUtils;
|
||||
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.consistentread.ConsistentReadInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Integration coverage for the core pager adapter's ordering, bounded fetches,
|
||||
** raw fields and transaction lifetime.
|
||||
*******************************************************************************/
|
||||
class TableReaderTest
|
||||
{
|
||||
private QTableMetaData table;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Seed more records than fit in a page.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
BackupsTestUtils.init();
|
||||
BackupsTestUtils.loadBlog(7);
|
||||
table = QContext.getQInstance().getTable("blogPost");
|
||||
table.getField("body").setIsHeavy(true);
|
||||
table.getField("secretNote").setIsHidden(true);
|
||||
table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(CaptureQueries.class));
|
||||
CaptureQueries.inputs.clear();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An OR filter and a resume cursor are ANDed, with ascending keys even when
|
||||
** the caller requests descending order. Every query has the requested bound.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFilterResumeOrderingAndRawFields() throws Exception
|
||||
{
|
||||
String filter = "{\"booleanOperator\":\"OR\",\"criteria\":[{\"fieldName\":\"id\",\"operator\":\"LESS_THAN_OR_EQUALS\",\"values\":[3]},{\"fieldName\":\"id\",\"operator\":\"GREATER_THAN\",\"values\":[5]}],\"orderBys\":[{\"fieldName\":\"id\",\"isAscending\":false}]}";
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, filter, 2, 2, null, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(3, 6, 7);
|
||||
assertThat(rows.get(0).getValueString("body")).isEqualTo("Body of 3, with a comma");
|
||||
assertThat(rows.get(0).getValueString("secretNote")).isEqualTo("do-not-leak-3");
|
||||
assertThat(CaptureQueries.inputs).hasSize(2).allSatisfy(input ->
|
||||
{
|
||||
assertThat(input.getFilter().getLimit()).isEqualTo(2);
|
||||
assertThat(input.getFilter().getSkip()).isNull();
|
||||
assertThat(input.getShouldOmitHiddenFields()).isFalse();
|
||||
assertThat(input.getShouldMaskPasswords()).isFalse();
|
||||
assertThat(input.getShouldFetchHeavyFields()).isTrue();
|
||||
assertThat(input.getShouldTranslatePossibleValues()).isFalse();
|
||||
assertThat(input.getShouldGenerateDisplayValues()).isFalse();
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Changes made after a snapshot was opened are invisible on every page. The
|
||||
** reader leaves the transaction open, with its commit callbacks unexecuted.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testConsistentSnapshotIsRetainedAcrossPages() throws Exception
|
||||
{
|
||||
List<Boolean> commits = new ArrayList<>();
|
||||
try(QBackendTransaction transaction = QBackendTransaction.openConsistentReadFor(new ConsistentReadInput().withBackendName(table.getBackendName())))
|
||||
{
|
||||
transaction.addAfterCommitCallback(() -> commits.add(true));
|
||||
new UpdateAction().execute(new UpdateInput("blogPost").withRecords(List.of(new QRecord().withValue("id", 7).withValue("title", "Changed after snapshot"))));
|
||||
CaptureQueries.inputs.clear();
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, null, null, 2, transaction, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(1, 2, 3, 4, 5, 6, 7);
|
||||
assertThat(rows.get(6).getValueString("title")).isEqualTo("Post 7");
|
||||
assertThat(CaptureQueries.inputs).hasSize(4).allSatisfy(input -> assertThat(input.getTransaction()).isSameAs(transaction));
|
||||
assertThat(commits).isEmpty();
|
||||
transaction.commit();
|
||||
assertThat(commits).containsExactly(true);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Exact page multiples, empty scans, invalid bounds and unsupported key types
|
||||
** must terminate or fail explicitly rather than drop rows or stream unordered.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testBoundariesAndInvalidPaging() throws Exception
|
||||
{
|
||||
List<QRecord> rows = new ArrayList<>();
|
||||
new TableReader.Keyset().read(table, null, 3, 2, null, rows::add);
|
||||
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(4, 5, 6, 7);
|
||||
rows.clear();
|
||||
new TableReader.Keyset().read(table, null, 7, 2, null, rows::add);
|
||||
assertThat(rows).isEmpty();
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 0, null, rows::add)).hasMessageContaining("Page size");
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{\"limit\":1}", null, 2, null, rows::add)).hasMessageContaining("skip or limit");
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{", null, 2, null, rows::add)).hasMessageContaining("scope filter");
|
||||
table.getField("id").setType(QFieldType.BOOLEAN);
|
||||
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 2, null, rows::add)).hasMessageContaining("orderable primary key");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Observe actual QueryAction inputs without replacing the backend or pager.
|
||||
*******************************************************************************/
|
||||
public static class CaptureQueries implements TableCustomizerInterface
|
||||
{
|
||||
static final List<QueryInput> inputs = new ArrayList<>();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Record each page and preserve the result unchanged.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records)
|
||||
{
|
||||
inputs.add((QueryInput) input);
|
||||
return (records);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -306,7 +306,7 @@ class BackupPlanScheduleTest
|
||||
assertThat(catalog.where(BackupsTables.RUN, "backupPlanId", planId)).isEmpty();
|
||||
assertThat(BackupsTestUtils.BACKUPS.list("")).isEmpty();
|
||||
assertThat(catalog.get(BackupsTables.PLAN, planId).orElseThrow().getValueString("slaStatus")).isEqualTo("OWNER_INVALID");
|
||||
assertThat(alerts.raised).containsExactly("backups.plan.owner_invalid|backups.plan.owner_invalid:nightly");
|
||||
assertThat(alerts.raised).containsExactly("backups.plan.owner-invalid|backups.plan.owner-invalid:nightly");
|
||||
|
||||
BackupScheduledRunProcess.setOwnerHasAdminCheck(null);
|
||||
Integer noOwner = insertPlan(Map.of("name", "Ownerless", "planKey", "ownerless", "isActive", true));
|
||||
|
||||
@@ -180,12 +180,12 @@ class SlaCheckerTest
|
||||
|
||||
clock.advance(Duration.ofMinutes(55));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
BackupAlertsAssert.assertOpen("backups.run.missing_in_storage:nightly", BackupAlertSink.Severity.CRITICAL);
|
||||
BackupAlertsAssert.assertOpen("backups.run.missing-in-storage:nightly", BackupAlertSink.Severity.CRITICAL);
|
||||
assertThat(check(plan).status()).as("a negative answer is never cached").isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
|
||||
BackupsTestUtils.BACKUPS.putBytes(result.getManifestKey(), bytes);
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
|
||||
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isFalse();
|
||||
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isFalse();
|
||||
}
|
||||
|
||||
|
||||
@@ -260,11 +260,11 @@ class SlaCheckerTest
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> false);
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OWNER_INVALID);
|
||||
BackupAlertsAssert.assertOpen("backups.plan.owner_invalid:nightly", BackupAlertSink.Severity.WARNING);
|
||||
BackupAlertsAssert.assertOpen("backups.plan.owner-invalid:nightly", BackupAlertSink.Severity.WARNING);
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner -> "bob".equals(owner));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.OK);
|
||||
assertThat(sink.isOpen("backups.plan.owner_invalid:nightly")).isFalse();
|
||||
assertThat(sink.isOpen("backups.plan.owner-invalid:nightly")).isFalse();
|
||||
|
||||
BackupsRuntime.getConfig().withPlanOwnerValidator(owner ->
|
||||
{
|
||||
@@ -287,7 +287,7 @@ class SlaCheckerTest
|
||||
clock.advance(Duration.ofDays(2));
|
||||
assertThat(check(plan).status()).isEqualTo(SlaChecker.MISSING_IN_STORAGE);
|
||||
assertThat(sink.isOpen("backups.plan.missed:nightly")).as("the alerts are independent of the status shown").isTrue();
|
||||
assertThat(sink.isOpen("backups.run.missing_in_storage:nightly")).isTrue();
|
||||
assertThat(sink.isOpen("backups.run.missing-in-storage:nightly")).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"author" : {
|
||||
"schema" : {
|
||||
"name" : "author",
|
||||
"primaryKey" : "id",
|
||||
"backendType" : "memory",
|
||||
"fields" : {
|
||||
"bio" : {
|
||||
"type" : "TEXT",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"id" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"name" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
}
|
||||
},
|
||||
"uniqueKeys" : [ ],
|
||||
"associations" : [ ]
|
||||
},
|
||||
"fingerprint" : "sha256:c0fce31286898a15a98f9f46515b15c6eee4ffeb1f1eb39d6100833dfe3f760e"
|
||||
},
|
||||
"blogPost" : {
|
||||
"schema" : {
|
||||
"name" : "blogPost",
|
||||
"primaryKey" : "id",
|
||||
"backendType" : "memory",
|
||||
"fields" : {
|
||||
"authorId" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : "author",
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"body" : {
|
||||
"type" : "TEXT",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"createDate" : {
|
||||
"type" : "DATE_TIME",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"id" : {
|
||||
"type" : "INTEGER",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"mediaKey" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"price" : {
|
||||
"type" : "DECIMAL",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"published" : {
|
||||
"type" : "BOOLEAN",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"publishedOn" : {
|
||||
"type" : "DATE",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
},
|
||||
"secretNote" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : true
|
||||
},
|
||||
"title" : {
|
||||
"type" : "STRING",
|
||||
"required" : false,
|
||||
"maxLength" : null,
|
||||
"pvs" : null,
|
||||
"heavy" : false,
|
||||
"sensitive" : false
|
||||
}
|
||||
},
|
||||
"uniqueKeys" : [ ],
|
||||
"associations" : [ ]
|
||||
},
|
||||
"fingerprint" : "sha256:3a4de7479908cd4b4ce8eaa7a7af43e7cbaabbdbc2b9741bebeb24a6bd41b9ea"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user