Compare commits

...
Author SHA1 Message Date
james.maesandClaude Opus 5.5 acf51b6749 feat(help): ship the qbit-backups help bundle through QQQ help bundles, with coverage gate
Host-neutral help (deployment-specific facts removed), registered with
HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:17:05 -05:00
james.maes a825360ba1 Merge pull request #24 from QRun-IO/feature/health-system-session
fix(health): run the context-less health check as the system user
2026-10-14 17:36:41 -05:00
james.maes 01db11b123 Merge pull request #25 from QRun-IO/feature/tables-populated-defaults
feat: populated defaults, guided empty states, scheduled verify
2026-10-14 17:13:31 -05:00
james.maesandClaude Opus 5.5 6dea78881d feat: host-scheduler config so a host that ticks plans itself is not shown a false needs-setup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:10:21 -05:00
james.maesandClaude Opus 5.5 89f218937a fix: setup card reflects the scheduler, review checks the plan name, host help test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:07:54 -05:00
james.maesandClaude Opus 5.5 85f4ca0a6f feat: guided empty states, setup card, Create default plan, scheduled verify process
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:49:20 -05:00
james.maesandClaude Opus 5.5 e2d07b275a fix(health): run the context-less health check as the system user
BackupsHealthIndicator made its own context with a plain tokenless
QSession. Token-based authentication modules (OAuth2, Auth0) reject any
tokenless session other than QSystemUserSession, so the plan query failed
session validation and the check errored on such hosts (kof22). Use core
QSystemUserSession(instance).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:14:13 -05:00
james.maes 48e7320228 Merge pull request #22 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:56:39 -05:00
james.maesandClaude Opus 5.5 a5297e709d ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:06:12 -05:00
james.maes f2af36ec5f Merge pull request #21 from QRun-IO/feature/arch-rc6-keyset
refactor(backups): core TableSchemaFingerprint + KeysetPager on RC.6 (#19 part 2)
2026-10-14 07:19:52 -05:00
james.maesandClaude Opus 5.5 054f6d90d7 build: qbit-build-parent 2.1.0 with architecture guardrails enforced
Enforce the qbit-bom#10 guardrails (G1, G3, G4, G7) and run tests in
strict raw-route mode (G2). All pass on the released QQQ 4.1.0-RC.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:15:24 -05:00
james.maesandClaude Opus 5.5 5b7106b981 refactor(backups): core TableSchemaFingerprint and KeysetPager (QQQ 4.1.0-RC.6)
Replace the RC.6 stand-ins in TableFingerprints, TableReader and
StandardRestoreWriter with core TableSchemaFingerprint and KeysetPager.

Core fingerprints differ from the local ones: descriptor names, unique
keys as lists, association joinName, and empty-string backend types. New
manifests therefore record fingerprintAlgorithm "qqq-table-schema-v1".
A manifest without the field uses the frozen LegacyTableFingerprint, so
old backups still verify and restore.

Paging keeps ascending primary-key order, exclusive resume cursors,
bounded pages and the caller's transaction. Restore writes now require
the explicit backups.restore permission, checked during Review.

Closes #19 (architecture review QRun-IO/qqq#1150, finding F3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 20:36:44 -05:00
james.maes 32034e5762 Merge pull request #20 from QRun-IO/feature/arch-alerts-core
refactor(alerts): direct core QAlerts binding on QQQ RC.5 (#19 part 1)
2026-10-13 19:37:01 -05:00
james.maesandClaude Opus 5.5 bc97d59b96 refactor(alerts): bind core QAlerts directly (QQQ 4.1.0-RC.5)
Core QAlerts shipped in 4.1.0-RC.5. Replace the reflective
AlertsQBitAlertSink stand-in with a compiled binding:
- raise maps to QAlerts.raise, resolve to QAlerts.resolve and heartbeat to
  QAlerts.heartbeat;
- the backup alert types register as QAlertTypeMetaData.
It stays quiet when no alerts implementation is installed, and an alerting
failure never fails a backup. The fake SPI test classes are deleted.

RC.5 rejects underscores in alert type names, so two types are renamed:
- backups.run.missing_in_storage becomes backups.run.missing-in-storage;
- backups.plan.owner_invalid becomes backups.plan.owner-invalid.

TableFingerprints, TableReader and StandardRestoreWriter are tagged
STAND-IN(..., removeBy=4.1.0-RC.6) for TableSchemaFingerprint and
KeysetPager.

Refs #19 (architecture review QRun-IO/qqq#1150, finding F3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:33:55 -05:00
james.maes 506741bd90 Merge pull request #18 from QRun-IO/feature/release-rc-automation
ci: automate RC releases; hotfix/* pushes no longer publish
2026-10-13 14:42:59 -05:00
james.maesandClaude Opus 5.5 95b971a5fb ci: automate RC releases; hotfix/* pushes no longer publish
Removes publish_hotfix_release: a hotfix/* branch push published a GA
release to Maven Central with no tag or review (QRun-IO/qbit-cdn#12,
QRun-IO/qqq#1121). hotfix/* branches now run test_only; a hotfix
release publishes from a reviewed vX.Y.Z tag through publish_release.
Adds .github/workflows/release-rc.yml (reusable QRun-IO/.github
qbit-release-rc): a manual dispatch, or merging a develop PR labelled
release:rc, merges develop into the release line and the release/* push
runs the existing CircleCI RC publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:29:57 -05:00
james.maes e56cc1b021 Merge pull request #17 from QRun-IO/feature/object-storage-rc2
build(deps): qbit-object-storage 0.1.0-RC.2 (AWS SDK security fix)
2026-10-13 11:07:46 -05:00
james.maesandClaude Opus 5.5 664f467d11 build(deps): qbit-object-storage 0.1.0-RC.2 (AWS SDK GHSA-443w-3rq3-5m5h)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 11:05:06 -05:00
james.maes 4fb182e824 Merge pull request #16 from QRun-IO/chore/object-storage-rc1
build: depend on qbit-object-storage 0.1.0-RC.1 (no SNAPSHOT dependency)
2026-10-12 19:15:20 -05:00
james.maesandClaude Sonnet 5.5 1de38f03ae build: depend on qbit-object-storage 0.1.0-RC.1 (Central), not a SNAPSHOT
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:11:25 -05:00
james.maes cf2d027f9a Merge pull request #9 from QRun-IO/feature/home-polish
feat(home): readable Backups home, record screens, runnable demo
2026-10-12 19:10:47 -05:00
james.maesandClaude Sonnet 5.5 714ef906fe feat(home): drop duplicate app section, bounded trend bars, screenshots
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:07:45 -05:00
james.maesandClaude Sonnet 5.5 26c174f7bf feat(home): null-safe home, readable plan rows, record sections and child lists, runnable demo
- Fix the NPE in BackupsHomeData.plansNeedingAttention when slaStatus is null (and null plan ids)
- Home: full-width plan status rows, derived lateness, links that work in the Next UI (tablePathPrefix), bars for the size trend
- The Backups app attaches itself to its parent app (Admin) so the nav shows it
- Nav lists the main tables only; artifacts, verifications, restores and per-table results are child lists on their parents
- Record screens grouped into sections
- Test-scope demo server (memory backend, real engine history incl. a failed run)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:07:45 -05:00
james.maes 284f0cec8c Merge pull request #8 from QRun-IO/feature/object-storage-adapter
feat(storage): BackupStorage adapter over qbit-object-storage (MinIO tested)
2026-10-12 19:07:11 -05:00
james.maesandClaude Sonnet 5.5 2f16bd6515 test(storage): skip, do not fail, when the MinIO image cannot be started
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:04:44 -05:00
james.maesandClaude Sonnet 5.5 d4ad39f19d test(storage): skip the MinIO tests when Docker is not available
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:02:10 -05:00
james.maesandClaude Sonnet 5.5 ba3959e7dc feat(storage): BackupStorage adapter over qbit-object-storage, tested on MinIO
A streamed write of unknown length is bridged to the storage qbit put; permanent
delete reports locked objects; failures map to BackupStorageException reasons.
Object storage is an optional dependency. Tested against a MinIO container with
object lock: the adapter contract and a full backup run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 19:00:59 -05:00
james.maes a67e4c4a57 Merge pull request #7 from QRun-IO/feature/prune-schedule
feat: run lifecycle - schedule, lease, cancel, resume, GFS prune, object lock, plan validation
2026-10-12 19:00:25 -05:00
james.maesandClaude Sonnet 5.5 4022fa1eee test(verify): do not depend on which mismatch is reported first for a flipped byte
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 18:57:50 -05:00
james.maesandClaude Sonnet 5.5 6017f3ed2d test(sla): save the plan with a bucket that exists, then remove it from the config
Plan save now validates the bucket alias (lifecycle), so the SLA test creates the
plan while the bucket is configured and removes it afterwards.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 18:55:21 -05:00
james.maesandClaude Sonnet 5.5 01b60708f8 feat: run lifecycle - plan schedule, run lease, cancel, resume, GFS prune, object lock, plan validation
- BackupPlanSyncToScheduledJobProcess (+ verify variant) and the plan table wiring
- BackupRunCoordinator: one run per plan (live-heartbeat lease, QUEUED to RUNNING,
  SKIPPED_CONCURRENT warning), maxConcurrentRuns queueing, scheduled entry point
  (backupRunScheduledPlan) as the system session with plan-owner validation
- plan save validation (admin and read on resolved tables, object lock vs bucket
  capabilities, encryption rules, unsupported formats, planKey immutable after a run)
- PruneCalculator (GFS) and BackupPruner (retention classes, deletePermanent on the
  prune alias, kept: locked until, debris, file store mark-and-sweep)
- object lock applied at write time in the engine, parts catalogued as they complete
- Back up these records, Cancel, Resume (PER_TABLE, from the last complete part),
  Test destination and Prune processes with Review and Result lines

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-12 18:54:24 -05:00
james.maes 06c87afa20 Merge pull request #6 from QRun-IO/feature/restore
feat(restore): restore, catalog import and download (spec 9.3-9.7)
2026-10-12 18:53:53 -05:00
158 changed files with 12162 additions and 1003 deletions

No files matched your search

+2 -13
View File
@@ -1,7 +1,7 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
@@ -22,7 +22,7 @@ workflows:
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|release\/.*|hotfix\/.*)/
ignore: /(develop|release\/.*)/
publish_snapshot:
jobs:
@@ -58,14 +58,3 @@ workflows:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
publish_hotfix_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: hotfix
filters:
branches:
only: [/hotfix\/.*/]
+25
View File
@@ -0,0 +1,25 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
type: boolean
default: false
pull_request:
types: [closed]
branches: [develop]
permissions:
contents: write
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+1 -1
View File
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
## Dependencies
- Java 21.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Tests: JUnit 5, AssertJ.
## Release
+15
View File
@@ -5,4 +5,19 @@ All notable changes to this project are documented here. The format follows Keep
## [0.1.0] - Unreleased
### Added
- Complete host-neutral in-app help bundle with live references and a help coverage gate. Hosts can override entries with a HOST bundle.
- Guided empty states: help bundle `help/qbit-backups.yaml` (app and all nine tables, with "How rows get created") applied by `BackupsHelp`, and a Setup card on the Backups home (plan, scope rules, storage, schedule, verification, prune bucket).
- "Create default plan" table action on Backup plans: adds "Nightly full" for a storage bucket alias; insert-only.
- `backupVerifyPlan`: the scheduled verification process. A plan verify cron created a scheduled job for a process that did not exist, so scheduled verification never ran.
- Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.
### Changed
- Pin QQQ to released 4.1.0-RC.7 and replace the temporary help loader with core HelpBundles registration.
- Saving a plan with Use read replica on is refused (it was never read); the not-yet-available settings (read replica, native dump, incremental, PARQUET, ZSTD) say so in their field help; scheduledJobId is documented as unused.
- Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
- Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.
- Use RC.5-valid alert names `backups.run.missing-in-storage` and `backups.plan.owner-invalid` (and matching deduplication keys). Hosts using the old underscore names must update routing rules.
- Target QQQ 4.1.0-RC.6 and replace the fingerprint, keyset pager and reflective restore bridges with typed core APIs.
- New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
- Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
- Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
+35 -1
View File
@@ -10,7 +10,7 @@
## Requirements
- Java 21.
- QQQ 4.1 (built against `4.1.0-RC.4`).
- QQQ `4.1.0-RC.7` or later.
## Install
@@ -22,6 +22,10 @@ new BackupsQBitProducer()
.produce(qInstance);
```
## Setup
This qbit ships no DDL and seeds nothing from a changelog: its one setup value, the storage bucket alias, belongs to your application. On a fresh install open Backups, read the Setup card, and run **Create default plan** on Backup plans (it asks only for the alias of a bucket your storage resolver provides). It creates the insert-only "Nightly full" plan: whole instance, data and audit tables, JSONL and gzip, 02:00 UTC, verified Sundays 04:00 UTC. Edit any setting afterwards. Help for every table, including how its rows get created, ships in `help/qbit-backups.yaml`.
## Naming
Tables are `backups_<name>` (lowercase snake_case, backend name equal to the table name). Permissions are `backups.<action>`.
@@ -36,8 +40,38 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
./mvnw verify
```
See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for fingerprint compatibility,
paging guarantees and restore permissions.
## In-app help
The producer registers [help/qbit-backups.yaml](src/main/resources/help/qbit-backups.yaml)
as a QBIT bundle. Hosts can register a `layer: HOST` bundle with matching entry keys
and roles to override its text. Core's help startup sync applies the bundle when the
host installs the help content table; registration also works without that table.
`BackupPlanScheduleTest` checks coverage with `HelpBundleAssertions`, including
live references and optional scheduler processes. `BackupsTablesGuardrailTest`
checks empty-state, widget and possible-value-source coverage. Run it with
`./mvnw verify`. Keep the bundle's literal version aligned with the POM version.
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
## License
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
### Alerts integration
Backups binds directly to core QAlerts in QQQ 4.1.0-RC.5 and declares its nine
alert types on each instance. Hosts install an `AlertSinkInterface` through
`AlertsInstanceMetaData.withSink(QCodeReference)`. Without a configured sink,
backup alert calls remain no-ops (they do not use core's logging fallback).
Custom `BackupAlertSink` bindings are preserved. Core handles dispatch failures;
adapter failures are caught and logged once, so alerting cannot fail a backup.
Raise and resolve follow core's transaction-aware dispatch; heartbeats run immediately.
The backup SLA checker still owns expected intervals and grace periods.
RC.5 requires hyphenated alert names: `backups.run.missing-in-storage` and
`backups.plan.owner-invalid` replace the former underscore spellings, including
their deduplication keys. Update host routing rules that used those spellings.
+7
View File
@@ -0,0 +1,7 @@
# Upgrading
## Unreleased
- No schema change and no new include line. Hosts get the help text and the Setup card automatically.
- Plans with `use_read_replica` true are now refused on the next save (the setting was never used). Turn it off.
- Hosts that set a plan `verifyCronExpression` and a scheduler now get working scheduled verification: the `backupVerifyPlan` process exists. It verifies the plan's latest successful run through the plan's `verifyLevel` (CONTENT when unset). Expect verification rows and, on failure, alerts.
+100
View File
@@ -0,0 +1,100 @@
# RC.6 core migration (issue #19, F3)
The POM targets `4.1.0-RC.6`. Until that release is available, local verification
uses `./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` against the
installed build from `qqq-wt-archrc6`. No core source changes are needed.
## Fingerprint compatibility
The algorithms are **not byte-compatible**, even for identical table metadata:
| Descriptor property | Pre-RC.6 backup algorithm | Core algorithm |
| --- | --- | --- |
| Table and key | `name`, `primaryKey` | `table`, `primaryKeyField` |
| Fields | Map keyed by name | Sorted list with explicit `name` |
| Field flags | `required`, `pvs`, `heavy`, `sensitive` | `isRequired`, `possibleValueSource`, `isHeavy`, `isSensitive` |
| Unique keys | Sorted `+`-joined strings | Sorted lists of field names |
| Associations | Sorted `name->table` strings | Objects with name, associatedTable and joinName |
| Null backend type | JSON null | Empty string |
| Escaping | Historical hand-written escaping | Jackson JSON escaping |
New table entries contain `fingerprintAlgorithm: "qqq-table-schema-v1"`, the core
`schema` descriptor, and `fingerprint: "sha256:" + core.getHash()`. The outer
manifest remains version 1; row parts, checksums, compression and storage paths
are unchanged. The instance/catalog hash still combines the ordered table
fingerprint strings, so its value changes too. New manifests require this RC.6
reader; older readers do not understand the new descriptor shape.
An absent algorithm identifies the old `qbit-backups-table-v1` algorithm. The
reader also accepts that explicit identifier. `LegacyTableFingerprint` freezes
the original descriptor and hash implementation solely for comparison against
existing backups; no new backup uses it. Restore and content verification select
the declared algorithm. Unknown identifiers never match a known fingerprint.
Restore normalizes either field shape for its existing compatibility analysis
when a table has changed. Fingerprint equality is not required for a compatible
schema migration, as before.
`src/test/resources/manifests/legacy-schema.json` freezes descriptors and hashes
produced by the original stand-in for the blog fixture. Tests place these into a
real backup manifest without the algorithm field, verify its content, mutate the
live data, then restore and compare the complete content hash including dates.
They also cover optional additions and incompatible type changes in old schemas.
## Paging and consistency
`TableReader.Keyset` delegates to core `KeysetPager`. Every query uses
`(scope filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize`, with no offset.
The resume key is exclusive. Page size is a row bound, **not a JDBC fetch-size
setting**. Core's normal bounded QueryAction path retains backend driver defaults;
this adapter does not opt into the large-result streaming query hint or change
JDBC fetch size. Heavy/hidden/password fields and untranslated stored values keep
the same query settings and existing backup permission boundary.
Every page receives the caller's transaction. The pager does not open, commit,
close, or replace that transaction. A test opens a real core memory snapshot,
changes a later page's data, and proves every page still sees the original state.
The existing BackupEngine continues to report BEST_EFFORT and reject
SNAPSHOT_REQUIRED: adopting the consistent-read opening API at engine level is
separate work. This migration does not claim snapshot guarantees for ordinary
backup runs.
Core can stream non-orderable keys without deterministic key order; backups
explicitly reject those keys instead. Core rejects scope filters carrying skip
or limit, and replaces user ordering with ascending primary-key ordering. Tests
cover OR filters, resume, full and partial pages, empty scans and invalid inputs.
## Restore writes
`StandardRestoreWriter` now constructs `RestoreWriteFlag` directly and calls
`QBackendModuleInterface.resyncKeySequences`, consuming typed `SequenceResyncResult`
values. Only RESYNCED results are reported as sequence changes; NOT_NEEDED and
NOT_APPLICABLE remain normal outcomes. There is no reflection or unavailable-core
fallback. The core flag preserves supplied dates and carries the requested
customizer, automation, audit and event settings.
Core always requires an actual session permission `backups.restore` for its
restore flag, including hosts with general core enforcement disabled. The qbit's
anonymous-mode convenience setting does not grant that permission. Review now
reports this requirement before executing a restore; test sessions explicitly
hold it. Custom writers that do not use the core flag keep their existing qbit
permission checks.
## HTTP operations and wire changes
No HTTP operations are added, removed or modified: method, path and operationId
are unchanged (no operation list entries for this migration). The RC.6 UI API
extension documentation was reviewed; this task introduces no Javalin route or
middleware extension. Wire changes are confined to the manifest fields and
fingerprint values described above, plus an earlier restore permission error in
Review for anonymous sessions lacking `backups.restore`.
## Local verification (2026-10-13)
- `./mvnw -B clean verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`:
BUILD SUCCESS; 158 tests, zero failures/errors/skips; both JaCoCo gates passed.
Existing backup, restore and MinIO integration tests ran.
- Separate Checkstyle 14.3.0 audit of all 16 changed Java files (main and test):
zero violations, using the shared Kingsrook-derived rules with this repository's
Apache file header. This repository has no Checkstyle execution configured in
its POM; audit configuration and results are local under `.cache/style/`.
- `git diff --check`: passed. No commit, push or core source modification.
+7
View File
@@ -0,0 +1,7 @@
# Backups home and record screenshots
Taken 2026-10-12 against `BackupsDemoServer` (src/test/.../demo, memory backend, real engine history) and the Next UI 1.0.0-RC.14 jar.
Light and dark at 390, 600, 768, 1024, 1440, 1920 px (home) and 390, 1440 px (records).
- `.` : this branch (foundation base): home, plan, run, failed run, lists.
- `with-restore-verify/` : a scratch merge of PRs #5 (verify), #6 (restore), #7 (prune/schedule) onto this branch, so verifications, a restore run and retention classes exist. Not part of this PR.
Binary file not shown.

After

Width:  |  Height:  |  Size: 163 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 160 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 165 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 143 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 163 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 161 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 166 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 138 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 144 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 168 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 146 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 147 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 45 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 161 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 141 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 163 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 221 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 135 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 216 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 166 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 169 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 144 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 176 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 172 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 144 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 151 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 177 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 177 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 178 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 169 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 92 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

+45 -3
View File
@@ -18,6 +18,8 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
@@ -27,7 +29,7 @@
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
<version>2.1.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -86,6 +88,35 @@
<optional>true</optional>
</dependency>
<!-- Optional: the production BackupStorage adapter. Apps that install this qbit -->
<!-- with object storage already have it; others use their own BackupStorage. -->
<dependency>
<groupId>com.kingsrook.qbits</groupId>
<artifactId>qbit-object-storage</artifactId>
<version>0.1.0-RC.2</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>2.0.3</version>
<scope>test</scope>
</dependency>
<!-- Demo server only (src/test/.../demo): never shipped. -->
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-middleware-javalin</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-frontend-next</artifactId>
<version>1.0.0-RC.14</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
@@ -101,6 +132,17 @@
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- the MinIO test container default credentials, resolved through ${env.X} refs like production -->
<environmentVariables>
<STORAGE_TEST_ACCESS_KEY>minioadmin</STORAGE_TEST_ACCESS_KEY>
<STORAGE_TEST_SECRET_KEY>minioadmin</STORAGE_TEST_SECRET_KEY>
</environmentVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
@@ -152,14 +194,14 @@
<profiles>
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
<!-- imports the released 4.1 BOM, this profile is active by default and -->
<!-- imports qqq-bom-pom 4.1.0-RC.4 ahead of the parent's 4.0.0 BOM. -->
<!-- imports qqq-bom-pom 4.1.0-RC.7 ahead of the parent's 4.0.0 BOM. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -45,7 +45,10 @@ public class BackupsQBitConfig implements QBitConfig
private String appName = "backups";
private String appLabel = "Backups";
private String parentAppName;
private String tablePathPrefix;
private String schedulerName;
private String hostSchedulerDescription;
private Boolean hostSchedulerRunsVerification = false;
private String instanceName;
private Integer maxConcurrentRuns = 1;
@@ -179,6 +182,73 @@ public class BackupsQBitConfig implements QBitConfig
/*******************************************************************************
** Getter for tablePathPrefix (the front of the links the home makes to tables and
** records, "/app" for the Next dashboard, which routes /app/{table}/{id}).
*******************************************************************************/
public String getTablePathPrefix()
{
return (tablePathPrefix == null ? "/app" : tablePathPrefix);
}
/*******************************************************************************
** Fluent setter for tablePathPrefix
*******************************************************************************/
public BackupsQBitConfig withTablePathPrefix(String tablePathPrefix)
{
this.tablePathPrefix = tablePathPrefix;
return (this);
}
/*******************************************************************************
** Getter for hostSchedulerDescription: when the application runs plan crons with
** its own tick instead of a QQQ scheduler (schedulerName), a short name for it.
** The Setup card then treats plan crons as firing.
*******************************************************************************/
public String getHostSchedulerDescription()
{
return (hostSchedulerDescription);
}
/*******************************************************************************
** Fluent setter for hostSchedulerDescription.
*******************************************************************************/
public BackupsQBitConfig withHostSchedulerDescription(String hostSchedulerDescription)
{
this.hostSchedulerDescription = hostSchedulerDescription;
return (this);
}
/*******************************************************************************
** Getter for hostSchedulerRunsVerification: whether the host's own tick also runs
** the verify cron (by calling the backupVerifyPlan process). False by default.
*******************************************************************************/
public Boolean getHostSchedulerRunsVerification()
{
return (hostSchedulerRunsVerification);
}
/*******************************************************************************
** Fluent setter for hostSchedulerRunsVerification.
*******************************************************************************/
public BackupsQBitConfig withHostSchedulerRunsVerification(Boolean hostSchedulerRunsVerification)
{
this.hostSchedulerRunsVerification = hostSchedulerRunsVerification;
return (this);
}
/*******************************************************************************
** Getter for schedulerName (the QQQ scheduler used for plan jobs and the SLA checker)
*******************************************************************************/
@@ -17,6 +17,7 @@
package com.kingsrook.qbits.backups;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
@@ -119,13 +120,8 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
qInstance.addQBit(qBit);
BackupsRuntime.install(qBitConfig);
///////////////////////////////////////////////////////////////////////////////
// bind the Alerts qbit producer SPI when it is on the class path (optional) //
///////////////////////////////////////////////////////////////////////////////
if(com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfPresent())
{
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
}
com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink.bindIfAbsent();
new com.kingsrook.qbits.backups.alerts.AlertsQBitAlertSink().registerAlertTypes(qInstance);
com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput output = new com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput();
for(var possibleValueSource : com.kingsrook.qbits.backups.model.BackupsTables.possibleValueSources())
@@ -149,12 +145,31 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
process.setSourceQBitName(qBit.getName());
output.add(process);
}
for(var process : com.kingsrook.qbits.backups.processes.ExportTableProcess.exportProcesses(qInstance))
{
process.setSourceQBitName(qBit.getName());
output.add(process);
}
for(var widget : com.kingsrook.qbits.backups.app.BackupsApp.widgets())
{
output.add(widget);
}
output.add(com.kingsrook.qbits.backups.app.BackupsApp.app(qBitConfig, tables));
for(var widget : com.kingsrook.qbits.backups.model.BackupsTables.widgets())
{
output.add(widget);
}
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData app = com.kingsrook.qbits.backups.app.BackupsApp.app(qBitConfig, tables);
output.add(app);
output.addSelfToInstance(qInstance);
// Registration is metadata-only; core guards startup sync when helpContent is absent.
HelpBundles.register(qInstance, "help/qbit-backups.yaml");
// A parent app only lists the apps that are its children, so attach ourselves (for example Admin, then Backups).
com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData parent = qBitConfig.getParentAppName() == null ? null : qInstance.getApp(qBitConfig.getParentAppName());
if(parent != null && (parent.getChildren() == null || parent.getChildren().stream().noneMatch(child -> app.getName().equals(child.getName()))))
{
parent.withChild(app);
}
return (new com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput());
}
@@ -17,105 +17,39 @@
package com.kingsrook.qbits.backups.alerts;
import java.lang.reflect.Method;
import java.util.Map;
import java.util.stream.Collectors;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** The BackupAlertSink adapter for the Alerts qbit producer SPI (QRun-IO/qqq
** PR #1108, com.kingsrook.qqq.backend.core.alerts: QAlerts, AlertSignal,
** AlertResolution, AlertSeverity, QAlertTypeMetaData; design spec
** docs/specs/2026-10-12-qbit-alerts-design.md).
**
** STAND-IN: the SPI is an open core PR and is not in QQQ 4.1.0-RC.4, so this
** adapter is bound REFLECTIVELY and optionally. It compiles against nothing in
** the SPI; it looks the classes up by name, and every call is guarded:
** <ul>
** <li>no SPI classes on the class path: {@link #bindIfPresent()} binds nothing
** and the sink stays the no-op;</li>
** <li>the SPI is present but the instance has no alerts meta-data (the Alerts
** qbit is not installed): every call is a no-op;</li>
** <li>any reflective or SPI failure is logged once and swallowed, because an
** alerting problem must never fail a backup.</li>
** </ul>
** Replace with a direct binding (QAlerts.raise and friends) when RC.5 ships.
**
** Mapping: raise to QAlerts.raise(AlertSignal), resolve to QAlerts.resolve(type,
** dedupKey, note), heartbeat to QAlerts.heartbeat(key). The SPI has no per-plan
** heartbeat registration (monitors are declared on the alert type), so the
** expected interval and grace are not passed on; the SLA checker raises
** backups.plan.missed itself, independent of any heartbeat (spec 10.4).
** Direct adapter to core QAlerts. Calls are no-ops without a configured alerts
** sink, including when only the backup alert types have been registered.
** QAlerts catches and logs dispatch failures; adapter failures are logged once.
** Heartbeat intervals remain the responsibility of the backup SLA checker.
*******************************************************************************/
public class AlertsQBitAlertSink implements BackupAlertSink
{
private static final QLogger LOG = QLogger.getLogger(AlertsQBitAlertSink.class);
/** The package of the Alerts producer SPI in qqq-backend-core. */
public static final String DEFAULT_API_PACKAGE = "com.kingsrook.qqq.backend.core.alerts";
private final String apiPackage;
private boolean warned = false;
private Class<?> qAlerts;
private Class<?> alertSignal;
private Class<?> alertSeverity;
private Class<?> instanceMetaData;
/*******************************************************************************
** Constructor for the real SPI package.
** Bind the core adapter unless the host already supplied a backup sink.
*******************************************************************************/
public AlertsQBitAlertSink()
public static boolean bindIfAbsent()
{
this(DEFAULT_API_PACKAGE);
}
/*******************************************************************************
** Constructor for another package (a test double of the SPI shape).
*******************************************************************************/
public AlertsQBitAlertSink(String apiPackage)
{
this.apiPackage = apiPackage;
}
/*******************************************************************************
** Whether the SPI classes are on the class path.
*******************************************************************************/
public static boolean isPresent(String apiPackage)
{
try
{
Class.forName(apiPackage + ".QAlerts");
Class.forName(apiPackage + ".AlertSignal");
return (true);
}
catch(ClassNotFoundException | LinkageError e)
{
return (false);
}
}
/*******************************************************************************
** Bind this adapter as the runtime's alert sink when the SPI is present and no
** other sink was bound already. Returns whether it bound.
*******************************************************************************/
public static boolean bindIfPresent()
{
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink) || !isPresent(DEFAULT_API_PACKAGE))
if(!(BackupsRuntime.getAlertSink() instanceof NoOpBackupAlertSink))
{
return (false);
}
@@ -126,38 +60,32 @@ public class AlertsQBitAlertSink implements BackupAlertSink
/*******************************************************************************
** Declare the backup alert types with the Alerts qbit (QAlertTypeMetaData), when
** the SPI is present. Returns how many were added.
** Declare missing backup alert types on each instance, independently of sink
** installation order. Returns how many were added.
*******************************************************************************/
public int registerAlertTypes(QInstance qInstance)
{
int added = 0;
try
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
{
Class<?> typeClass = Class.forName(apiPackage + ".QAlertTypeMetaData");
load();
for(BackupAlerts.AlertType alertType : BackupAlerts.alertTypes())
try
{
try
if(QAlertTypeMetaData.of(qInstance, alertType.name()) == null)
{
Object type = typeClass.getConstructor().newInstance();
typeClass.getMethod("withName", String.class).invoke(type, alertType.name());
typeClass.getMethod("withLabel", String.class).invoke(type, alertType.label());
typeClass.getMethod("withGroupName", String.class).invoke(type, "Backups");
typeClass.getMethod("withHowToFix", String.class).invoke(type, alertType.howToFix());
typeClass.getMethod("withDefaultSeverity", alertSeverity).invoke(type, severity(alertType.defaultSeverity()));
typeClass.getMethod("addSelfToInstance", QInstance.class).invoke(type, qInstance);
new QAlertTypeMetaData()
.withName(alertType.name())
.withLabel(alertType.label())
.withGroupName("Backups")
.withHowToFix(alertType.howToFix())
.withDefaultSeverity(AlertSeverity.valueOf(alertType.defaultSeverity().name()))
.addSelfToInstance(qInstance);
added++;
}
catch(ReflectiveOperationException | RuntimeException e)
{
warnOnce("Could not declare the alert type " + alertType.name(), e);
}
}
}
catch(ReflectiveOperationException | LinkageError e)
{
warnOnce("The alerts SPI is not available", e);
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not declare the alert type " + alertType.name(), e);
}
}
return (added);
}
@@ -170,28 +98,24 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void raise(String type, String dedupeKey, Severity severity, String title, Map<String, String> detail)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
Object signal = alertSignal.getConstructor(String.class).newInstance(type);
alertSignal.getMethod("setDedupKey", String.class).invoke(signal, dedupeKey);
alertSignal.getMethod("setSeverity", alertSeverity).invoke(signal, severity(severity));
alertSignal.getMethod("setTitle", String.class).invoke(signal, title);
if(!isAlertsQBitInstalled())
{
return;
}
AlertSignal signal = new AlertSignal(type)
.withDedupKey(dedupeKey)
.withSeverity(AlertSeverity.valueOf(severity.name()))
.withTitle(title);
if(detail != null && !detail.isEmpty())
{
alertSignal.getMethod("setSummary", String.class).invoke(signal, detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
Method withAttribute = alertSignal.getMethod("withAttribute", String.class, String.class);
for(Map.Entry<String, String> entry : detail.entrySet())
{
withAttribute.invoke(signal, entry.getKey(), entry.getValue());
}
signal.withSummary(detail.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.joining(", ")));
detail.forEach(signal::withAttribute);
}
qAlerts.getMethod("raise", alertSignal).invoke(null, signal);
QAlerts.raise(signal);
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not raise an alert", e);
}
@@ -205,15 +129,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void resolve(String type, String dedupeKey)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
qAlerts.getMethod("resolve", String.class, String.class, String.class).invoke(null, type, dedupeKey, "Cleared by the Backups qbit");
if(isAlertsQBitInstalled())
{
QAlerts.resolve(type, dedupeKey, "Cleared by the Backups qbit");
}
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not resolve an alert", e);
}
@@ -227,15 +150,14 @@ public class AlertsQBitAlertSink implements BackupAlertSink
@Override
public void heartbeat(String monitorKey, long expectedEverySeconds, long graceSeconds)
{
if(!isAlertsQBitInstalled())
{
return;
}
try
{
qAlerts.getMethod("heartbeat", String.class).invoke(null, monitorKey);
if(isAlertsQBitInstalled())
{
QAlerts.heartbeat(monitorKey);
}
}
catch(ReflectiveOperationException | RuntimeException e)
catch(RuntimeException | LinkageError e)
{
warnOnce("Could not check in a heartbeat", e);
}
@@ -244,67 +166,26 @@ public class AlertsQBitAlertSink implements BackupAlertSink
/*******************************************************************************
** The SPI classes load, and the instance has alerts meta-data (the Alerts qbit
** is installed). Checked on every call so the order qbits are produced in
** does not matter.
** Check on every call so installing the alerts sink after this qbit works.
** Type declarations alone do not mean an alerts implementation is installed.
*******************************************************************************/
boolean isAlertsQBitInstalled()
{
try
{
load();
QInstance qInstance = QContext.getQInstance();
if(qInstance == null)
{
return (false);
}
return (instanceMetaData.getMethod("of", QInstance.class).invoke(null, qInstance) != null);
}
catch(ReflectiveOperationException | RuntimeException | LinkageError e)
{
warnOnce("The alerts SPI is not usable", e);
return (false);
}
AlertsInstanceMetaData metaData = AlertsInstanceMetaData.of(QContext.getQInstance());
return (metaData != null && metaData.getSink() != null);
}
/*******************************************************************************
**
** Log adapter failures once; dispatch failures are already logged by core.
*******************************************************************************/
private synchronized void load() throws ClassNotFoundException
{
if(qAlerts == null)
{
qAlerts = Class.forName(apiPackage + ".QAlerts");
alertSignal = Class.forName(apiPackage + ".AlertSignal");
alertSeverity = Class.forName(apiPackage + ".AlertSeverity");
instanceMetaData = Class.forName(apiPackage + ".AlertsInstanceMetaData");
}
}
/*******************************************************************************
** Map our severity to the SPI's enum constant of the same name.
*******************************************************************************/
@SuppressWarnings({ "unchecked", "rawtypes" })
private Object severity(Severity severity)
{
return (Enum.valueOf((Class<Enum>) alertSeverity, severity.name()));
}
/*******************************************************************************
**
*******************************************************************************/
private void warnOnce(String message, Throwable t)
private synchronized void warnOnce(String message, Throwable t)
{
if(!warned)
{
warned = true;
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t, logPair("apiPackage", apiPackage));
LOG.warn(message + " (the Alerts qbit adapter will keep failing quietly)", t);
}
}
}
@@ -36,11 +36,11 @@ public final class BackupAlerts
{
public static final String PLAN_MISSED = "backups.plan.missed";
public static final String RUN_FAILED = "backups.run.failed";
public static final String RUN_MISSING = "backups.run.missing_in_storage";
public static final String RUN_MISSING = "backups.run.missing-in-storage";
public static final String FILES_MISSING = "backups.files.missing";
public static final String VERIFY_FAILED = "backups.verify.failed";
public static final String VERIFY_OVERDUE = "backups.verify.overdue";
public static final String PLAN_OWNER_INVALID = "backups.plan.owner_invalid";
public static final String PLAN_OWNER_INVALID = "backups.plan.owner-invalid";
public static final String STORAGE_DEBRIS = "backups.storage.debris";
public static final String RESTORE_COMPLETED = "backups.restore.completed";
@@ -29,7 +29,6 @@ import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
@@ -62,7 +61,7 @@ public final class BackupsApp
public static List<QWidgetMetaData> widgets()
{
return (List.of(
new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Backups").withType(WidgetType.HTML.getType())
new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType())
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false).withPermissionRules(BackupsTables.permissionRules()),
new QWidgetMetaData().withName(HEALTH_CARD_WIDGET).withLabel("Backups").withType(WidgetType.HTML.getType())
.withCodeReference(new QCodeReference(HealthCardRenderer.class)).withGridColumns(12).withIsCard(true).withPermissionRules(BackupsTables.permissionRules())));
@@ -80,13 +79,7 @@ public final class BackupsApp
.withLabel(config.getAppLabel())
.withIcon(new QIcon("backup"))
.withPermissionRules(BackupsTables.permissionRules())
.withWidgets(List.of(HOME_WIDGET))
.withSection(new QAppSection().withName("backups").withLabel("Backups").withIcon(new QIcon("backup"))
.withTables(List.of(BackupsTables.PLAN, BackupsTables.RUN)))
.withSection(new QAppSection().withName("recovery").withLabel("Restore and verify").withIcon(new QIcon("restore"))
.withTables(List.of(BackupsTables.RESTORE_RUN, BackupsTables.VERIFICATION)))
.withSection(new QAppSection().withName("files").withLabel("Files").withIcon(new QIcon("attach_file"))
.withTables(List.of(BackupsTables.FILE_OBJECT)));
.withWidgets(List.of(HOME_WIDGET));
if(config.getParentAppName() != null)
{
app.setParentAppName(config.getParentAppName());
@@ -143,10 +136,6 @@ public final class BackupsApp
static String pathPrefix()
{
var config = com.kingsrook.qbits.backups.BackupsRuntime.getConfig();
if(config == null)
{
return ("/backups");
}
return (config.getParentAppName() != null ? "/" + config.getParentAppName() + "/" + config.getAppName() : "/" + config.getAppName());
return (config == null ? "/app" : config.getTablePathPrefix());
}
}
@@ -159,12 +159,53 @@ public class BackupsHomeData
/*******************************************************************************
** The status to show for a plan: the stored slaStatus when it is a problem state,
** otherwise LATE when the age of the last success is past the SLA (the SLA
** checker may not have run yet), otherwise the stored status (UNKNOWN when none).
*******************************************************************************/
public String effectiveStatus(QRecord plan)
{
String stored = plan.getValueString("slaStatus");
if(stored != null && ATTENTION.contains(stored))
{
return (stored);
}
Long age = ageHours(plan);
Integer sla = plan.getValueInteger("slaHours");
if(age != null && sla != null && age > sla)
{
return ("LATE");
}
return (stored == null ? "UNKNOWN" : stored);
}
/*******************************************************************************
** Plans that need attention.
*******************************************************************************/
public List<QRecord> plansNeedingAttention()
{
return (plans.stream().filter(p -> Boolean.TRUE.equals(p.getValueBoolean("isActive")) && ATTENTION.contains(p.getValueString("slaStatus"))).toList());
return (plans.stream().filter(p -> Boolean.TRUE.equals(p.getValueBoolean("isActive")) && ATTENTION.contains(effectiveStatus(p))).toList());
}
/*******************************************************************************
** The last run of a plan (from the recent runs), or null.
*******************************************************************************/
public QRecord lastRun(QRecord plan)
{
Integer lastRunId = plan.getValueInteger("lastRunId");
for(QRecord run : recentRuns)
{
if(lastRunId != null && lastRunId.equals(run.getValueInteger("id")))
{
return (run);
}
}
return (null);
}
@@ -206,7 +247,7 @@ public class BackupsHomeData
Integer runId = verification.getValueInteger("backupRunId");
for(QRecord run : recentRuns)
{
if(runId != null && runId.equals(run.getValueInteger("id")) && planId.equals(run.getValueInteger("backupPlanId")))
if(runId != null && planId != null && runId.equals(run.getValueInteger("id")) && planId.equals(run.getValueInteger("backupPlanId")))
{
return (verification);
}
@@ -224,6 +265,10 @@ public class BackupsHomeData
public List<QRecord> successfulRuns(Integer planId)
{
List<QRecord> rs = new ArrayList<>();
if(planId == null)
{
return (rs);
}
for(QRecord run : recentRuns)
{
if(planId.equals(run.getValueInteger("backupPlanId")) && run.getValueString("status") != null && run.getValueString("status").startsWith("SUCCEEDED"))
@@ -242,6 +287,10 @@ public class BackupsHomeData
public Map<String, Integer> retentionCounts(Integer planId)
{
Map<String, Integer> counts = new LinkedHashMap<>();
if(planId == null)
{
return (counts);
}
for(QRecord run : recentRuns)
{
if(planId.equals(run.getValueInteger("backupPlanId")) && run.getValueString("status") != null && run.getValueString("status").startsWith("SUCCEEDED")
@@ -27,21 +27,30 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Renders the Backups home: one card per concern, each with an honest empty
** state. Cards stack to one column on narrow screens, long names wrap, and
** there are no tables and no inner horizontal scroll (definition lists and
** lists only), so it reads at every width in light and dark mode (colors are
** inherited, not set).
** Renders the Backups home: a plan status row per plan across the full width,
** then cards for what needs attention, the latest test restore, recent activity,
** the size trend and retention. Every card has an honest empty state. Layout
** is flow and grid only (no tables, no fixed widths, no inner scrolling): plan
** rows wrap their cells and the cards stack to one column on narrow screens,
** long names wrap, and colors are inherited (only the status dot is colored), so
** it reads at every width in light and dark mode.
*******************************************************************************/
public final class BackupsHomeView
{
private static final DateTimeFormatter WHEN = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC);
private static final String GRID = "display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,22rem),1fr));gap:1rem;";
private static final String ROOT = "display:grid;gap:1rem;min-width:0;";
private static final String GRID = "display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,24rem),1fr));gap:1rem;align-items:start;min-width:0;";
private static final String CARD = "border:1px solid rgba(127,127,127,.3);border-radius:8px;padding:1rem;min-width:0;overflow-wrap:anywhere;";
private static final String ROW = "border:1px solid rgba(127,127,127,.25);border-radius:6px;padding:.75rem;margin:0 0 .75rem;min-width:0;";
private static final String CELLS = "display:grid;grid-template-columns:repeat(auto-fit,minmax(9.5rem,1fr));gap:.6rem 1rem;margin:.6rem 0 0;font-size:.9rem;";
private static final String LIST = "display:grid;grid-template-columns:minmax(7rem,auto) 1fr;gap:.25rem .75rem;margin:0 0 .75rem;font-size:.9rem;";
private static final String ITEMS = "margin:0 0 .75rem;padding-left:1.25rem;font-size:.9rem;";
private static final String ITEMS = "margin:0 0 .75rem;padding-left:1.25rem;list-style:disc;font-size:.9rem;display:grid;gap:.35rem;";
private static final String MUTED = "opacity:.75;";
private static final String LINK = "color:inherit;text-decoration:underline;text-underline-offset:2px;";
private static final String CHIP = "display:inline-flex;align-items:center;gap:.4rem;padding:.05rem .6rem;border-radius:999px;border:1px solid rgba(127,127,127,.4);font-size:.8rem;white-space:nowrap;";
private static final int TREND_BARS = 30;
@@ -59,14 +68,16 @@ public final class BackupsHomeView
*******************************************************************************/
public static String render(BackupsHomeData data, String pathPrefix)
{
StringBuilder html = new StringBuilder("<div class=\"backups-home\" style=\"" + GRID + "\">");
StringBuilder html = new StringBuilder("<div class=\"backups-home\" style=\"" + ROOT + "\">");
planStatus(html, data, pathPrefix);
setup(html, data, pathPrefix);
html.append("<div style=\"").append(GRID).append("\">");
needsAttention(html, data, pathPrefix);
latestTestRestore(html, data, pathPrefix);
trend(html, data);
recentActivity(html, data, pathPrefix);
trend(html, data);
retention(html, data);
return (html.append("</div>").toString());
return (html.append("</div></div>").toString());
}
@@ -82,13 +93,13 @@ public final class BackupsHomeView
case "ATTENTION" -> "Needs attention: see the Backups home.";
default -> "All backups are within their SLA.";
};
return ("<div style=\"" + CARD + "\"><strong>Backups</strong><p>" + esc(text) + "</p><a href=\"" + esc(pathPrefix) + "\">Open the Backups home</a></div>");
return ("<div style=\"" + CARD + "\"><strong>Backups</strong><p>" + esc(text) + "</p><a style=\"" + LINK + "\" href=\"" + esc(pathPrefix) + "\">Open the Backups home</a></div>");
}
/*******************************************************************************
** Plan status: last successful backup, age against the SLA, last run, verification.
** Plan status: one row per plan, the cells wrap to the width.
*******************************************************************************/
private static void planStatus(StringBuilder html, BackupsHomeData data, String prefix)
{
@@ -99,24 +110,29 @@ public final class BackupsHomeView
}
for(QRecord plan : data.plans())
{
Long age = data.ageHours(plan);
Integer sla = plan.getValueInteger("slaHours");
html.append("<dl style=\"").append(LIST).append("\">");
html.append("<dt><strong>Plan</strong></dt><dd><a href=\"").append(esc(prefix + "/" + BackupsTables.PLAN + "/" + plan.getValue("id"))).append("\">")
.append(esc(plan.getValueString("name"))).append("</a>").append(Boolean.TRUE.equals(plan.getValueBoolean("isActive")) ? "" : " (not active)").append("</dd>");
row(html, "Status", label(plan.getValueString("slaStatus")));
row(html, "Last backup", plan.getValueInstant("lastSuccessDate") == null ? "No successful backup yet" : WHEN.format(plan.getValueInstant("lastSuccessDate")));
if(age != null && sla != null)
Long age = data.ageHours(plan);
Integer sla = plan.getValueInteger("slaHours");
String status = data.effectiveStatus(plan);
QRecord last = data.lastRun(plan);
html.append("<div style=\"").append(ROW).append("\"><div style=\"display:flex;flex-wrap:wrap;align-items:center;gap:.5rem .75rem;\">");
html.append("<strong>").append(recordLink(prefix, BackupsTables.PLAN, plan.getValue("id"), plan.getValueString("name"))).append("</strong>");
html.append(chip(status));
if(!Boolean.TRUE.equals(plan.getValueBoolean("isActive")))
{
row(html, "Age against SLA", age + " h of " + sla + " h" + (age > sla ? " - late" : ""));
html.append("<span style=\"").append(MUTED).append("\">Not active</span>");
}
row(html, "Last verified", plan.getValueInstant("lastVerifiedDate") == null ? "Not verified yet" : WHEN.format(plan.getValueInstant("lastVerifiedDate")));
row(html, "Schedule", plan.getValueString("cronExpression") == null ? "Manual only" : plan.getValueString("cronExpression"));
html.append("</div><div style=\"").append(CELLS).append("\">");
cell(html, "Last backup", plan.getValueInstant("lastSuccessDate") == null ? "No successful backup yet" : WHEN.format(plan.getValueInstant("lastSuccessDate")));
cell(html, "Age against SLA", age != null && sla != null ? age + " h of " + sla + " h" : (sla == null ? "No SLA set" : "Not run yet"));
cell(html, "Last run", last == null ? "None yet" : label(last.getValueString("status")) + (last.getValueInstant("startDate") == null ? "" : ", " + WHEN.format(last.getValueInstant("startDate"))));
cell(html, "Schedule", plan.getValueString("cronExpression") == null ? "Manual only" : plan.getValueString("cronExpression"));
cell(html, "Last verified", plan.getValueInstant("lastVerifiedDate") == null ? "Not verified yet" : WHEN.format(plan.getValueInstant("lastVerifiedDate")));
if(plan.getValueString("objectLockMode") != null && !"NONE".equals(plan.getValueString("objectLockMode")))
{
row(html, "Object lock", plan.getValueString("objectLockMode") + ", " + plan.getValueInteger("objectLockDays") + " days");
cell(html, "Object lock", plan.getValueString("objectLockMode") + (plan.getValueInteger("objectLockDays") == null ? "" : ", " + plan.getValueInteger("objectLockDays") + " days"));
}
html.append("</dl>");
html.append("</div></div>");
}
link(html, prefix, BackupsTables.PLAN, "Backup plans");
close(html);
@@ -124,6 +140,50 @@ public final class BackupsHomeView
/*******************************************************************************
** The Setup card: what a working backup needs, whether each part is in place,
** and the next step, linked to its table.
*******************************************************************************/
private static void setup(StringBuilder html, BackupsHomeData data, String prefix)
{
List<BackupsSetupStatus.Item> items = BackupsSetupStatus.load(data.plans());
open(html, "Setup");
long todo = items.stream().filter(i -> i.state() == BackupsSetupStatus.State.TODO).count();
html.append("<p style=\"margin:0 0 .6rem;font-size:.9rem;\">")
.append(todo == 0 ? "Everything backups need is set up." : esc(todo + (todo == 1 ? " item needs" : " items need") + " attention."))
.append("</p><ul style=\"list-style:none;margin:0;padding:0;\">");
for(BackupsSetupStatus.Item item : items)
{
String color = switch(item.state())
{
case OK -> "#16a34a";
case TODO -> "#dc2626";
case DEFAULT -> "#d97706";
case OPTIONAL -> "#9ca3af";
};
String label = switch(item.state())
{
case OK -> "Set up";
case TODO -> "Needs setup";
case DEFAULT -> "Using defaults";
case OPTIONAL -> "Optional";
};
html.append("<li style=\"").append(ROW).append("\"><div style=\"display:flex;flex-wrap:wrap;align-items:center;gap:.5rem .75rem;\">")
.append("<strong>").append("<a style=\"").append(LINK).append("\" href=\"").append(esc(prefix + "/" + item.tableName())).append("\">").append(esc(item.label())).append("</a></strong>")
.append("<span style=\"").append(CHIP).append("\"><span aria-hidden=\"true\" style=\"width:.55rem;height:.55rem;border-radius:50%;background:").append(color).append(";display:inline-block;\"></span>").append(esc(label)).append("</span>")
.append("</div><div style=\"font-size:.9rem;margin-top:.35rem;\">").append(esc(item.detail())).append("</div>");
if(item.nextStep() != null)
{
html.append("<div style=\"").append(MUTED).append("font-size:.9rem;margin-top:.2rem;\">Next: ").append(esc(item.nextStep())).append("</div>");
}
html.append("</li>");
}
html.append("</ul>");
close(html);
}
/*******************************************************************************
** Needs attention.
*******************************************************************************/
@@ -144,13 +204,13 @@ public final class BackupsHomeView
html.append("<ul style=\"").append(ITEMS).append("\">");
for(QRecord plan : plans)
{
html.append("<li><a href=\"").append(esc(prefix + "/" + BackupsTables.PLAN + "/" + plan.getValue("id"))).append("\">").append(esc(plan.getValueString("name")))
.append("</a>: ").append(esc(label(plan.getValueString("slaStatus")))).append("</li>");
html.append("<li>").append(recordLink(prefix, BackupsTables.PLAN, plan.getValue("id"), plan.getValueString("name")))
.append(": ").append(esc(label(data.effectiveStatus(plan)))).append("</li>");
}
for(QRecord run : data.failedRuns())
{
html.append("<li><a href=\"").append(esc(prefix + "/" + BackupsTables.RUN + "/" + run.getValue("id"))).append("\">Run ").append(esc(run.getValueString("runKey")))
.append("</a> ").append(esc(label(run.getValueString("status")))).append("</li>");
html.append("<li>Run ").append(recordLink(prefix, BackupsTables.RUN, run.getValue("id"), run.getValueString("runKey")))
.append(": ").append(esc(label(run.getValueString("status")))).append("</li>");
}
html.append("</ul>");
}
@@ -178,7 +238,8 @@ public final class BackupsHomeView
html.append("<dl style=\"").append(LIST).append("\">");
row(html, "Plan", plan.getValueString("name"));
row(html, "Result", label(v.getValueString("status")));
row(html, "Compared", v.getValueInteger("tablesChecked") + " tables, " + v.getValueLong("rowsChecked") + " rows, " + v.getValueInteger("mismatches") + " mismatches");
row(html, "Compared", number(v.getValueLong("rowsChecked")) + " rows in " + number(toLong(v.getValueInteger("tablesChecked"))) + " tables, "
+ number(toLong(v.getValueInteger("mismatches"))) + " mismatches");
row(html, "Scratch target", v.getValueString("scratchTarget") == null ? "Not recorded" : v.getValueString("scratchTarget"));
html.append("</dl>");
}
@@ -193,7 +254,8 @@ public final class BackupsHomeView
/*******************************************************************************
** Size and duration trend per plan, as numbers next to each other.
** Size trend per plan: bars for the stored size of each successful run (oldest
** left), then the numbers.
*******************************************************************************/
private static void trend(StringBuilder html, BackupsHomeData data)
{
@@ -209,13 +271,14 @@ public final class BackupsHomeView
any = true;
QRecord first = runs.get(0);
QRecord last = runs.get(runs.size() - 1);
html.append("<div style=\"margin:0 0 .75rem;\"><strong style=\"font-size:.9rem;\">").append(esc(plan.getValueString("name"))).append("</strong>");
bars(html, runs);
html.append("<dl style=\"").append(LIST).append("\">");
row(html, "Plan", plan.getValueString("name"));
row(html, "Successful runs", String.valueOf(runs.size()));
row(html, "Stored (first, latest)", bytes(first.getValueLong("bytesStored")) + ", " + bytes(last.getValueLong("bytesStored")));
row(html, "Rows (first, latest)", number(first.getValueLong("rowsTotal")) + ", " + number(last.getValueLong("rowsTotal")));
row(html, "Duration (latest)", last.getValueLong("durationMs") == null ? "Not recorded" : last.getValueLong("durationMs") / 1000 + " s");
html.append("</dl>");
row(html, "Stored", bytes(first.getValueLong("bytesStored")) + " at first, " + bytes(last.getValueLong("bytesStored")) + " latest");
row(html, "Rows", number(first.getValueLong("rowsTotal")) + " at first, " + number(last.getValueLong("rowsTotal")) + " latest");
row(html, "Duration", last.getValueLong("durationMs") == null ? "Not recorded" : duration(last.getValueLong("durationMs")) + " latest");
html.append("</dl></div>");
}
if(!any)
{
@@ -227,7 +290,31 @@ public final class BackupsHomeView
/*******************************************************************************
** Recent activity: the last runs and restores.
** A row of bars, one per run, tall in proportion to the stored bytes.
*******************************************************************************/
private static void bars(StringBuilder html, List<QRecord> runs)
{
List<QRecord> shown = runs.size() > TREND_BARS ? runs.subList(runs.size() - TREND_BARS, runs.size()) : runs;
long max = 1;
for(QRecord run : shown)
{
max = Math.max(max, run.getValueLong("bytesStored") == null ? 0 : run.getValueLong("bytesStored"));
}
html.append("<div role=\"img\" aria-label=\"Stored size of the last ").append(shown.size()).append(" successful runs\" style=\"display:flex;align-items:flex-end;gap:2px;height:2.5rem;margin:.4rem 0 .6rem;\">");
for(QRecord run : shown)
{
long value = run.getValueLong("bytesStored") == null ? 0 : run.getValueLong("bytesStored");
long pct = Math.max(4, Math.round(100.0 * value / max));
String title = (run.getValueInstant("startDate") == null ? "" : WHEN.format(run.getValueInstant("startDate")) + ": ") + bytes(run.getValueLong("bytesStored"));
html.append("<span title=\"").append(esc(title)).append("\" style=\"flex:1 1 0;min-width:2px;max-width:1.5rem;height:").append(pct).append("%;background:currentColor;opacity:.55;border-radius:2px 2px 0 0;\"></span>");
}
html.append("</div>");
}
/*******************************************************************************
** Recent activity: the last runs, restores and verifications.
*******************************************************************************/
private static void recentActivity(StringBuilder html, BackupsHomeData data, String prefix)
{
@@ -242,13 +329,13 @@ public final class BackupsHomeView
data.recentRuns().stream().limit(6).forEach(run ->
{
Instant start = run.getValueInstant("startDate");
html.append("<li>Backup <a href=\"").append(esc(prefix + "/" + BackupsTables.RUN + "/" + run.getValue("id"))).append("\">").append(esc(run.getValueString("runKey")))
.append("</a>: ").append(esc(label(run.getValueString("status")))).append(start == null ? "" : ", " + WHEN.format(start)).append("</li>");
html.append("<li>Backup ").append(recordLink(prefix, BackupsTables.RUN, run.getValue("id"), run.getValueString("runKey")))
.append(": ").append(esc(label(run.getValueString("status")))).append(start == null ? "" : ", " + esc(WHEN.format(start))).append("</li>");
});
data.restores().stream().limit(2).forEach(restore -> html.append("<li>Restore <a href=\"").append(esc(prefix + "/" + BackupsTables.RESTORE_RUN + "/" + restore.getValue("id")))
.append("\">#").append(esc(String.valueOf(restore.getValue("id")))).append("</a>: ").append(esc(label(restore.getValueString("status")))).append("</li>"));
data.verifications().stream().limit(2).forEach(v -> html.append("<li>Verify <a href=\"").append(esc(prefix + "/" + BackupsTables.VERIFICATION + "/" + v.getValue("id")))
.append("\">").append(esc(label(v.getValueString("level")))).append("</a>: ").append(esc(label(v.getValueString("status")))).append("</li>"));
data.restores().stream().limit(2).forEach(restore -> html.append("<li>Restore ").append(recordLink(prefix, BackupsTables.RESTORE_RUN, restore.getValue("id"), "#" + restore.getValue("id")))
.append(": ").append(esc(label(restore.getValueString("status")))).append("</li>"));
data.verifications().stream().limit(2).forEach(v -> html.append("<li>Verify ").append(recordLink(prefix, BackupsTables.VERIFICATION, v.getValue("id"), label(v.getValueString("level"))))
.append(": ").append(esc(label(v.getValueString("status")))).append("</li>"));
html.append("</ul>");
}
link(html, prefix, BackupsTables.RUN, "All runs");
@@ -274,9 +361,10 @@ public final class BackupsHomeView
any = true;
html.append("<dl style=\"").append(LIST).append("\">");
row(html, "Plan", plan.getValueString("name"));
row(html, "Kept", counts.entrySet().stream().map(e -> e.getValue() + " " + label(e.getKey()).toLowerCase()).reduce((a, b) -> a + ", " + b).orElse(""));
List<QRecord> runs = data.successfulRuns(plan.getValueInteger("id"));
Instant oldest = runs.isEmpty() ? null : runs.get(0).getValueInstant("startDate");
row(html, "Kept", counts.entrySet().stream().map(e -> e.getValue() + " " + ("UNCLASSIFIED".equals(e.getKey()) ? "not classified yet (no prune has run)" : label(e.getKey()).toLowerCase()))
.reduce((a, b) -> a + ", " + b).orElse(""));
List<QRecord> runs = data.successfulRuns(plan.getValueInteger("id"));
Instant oldest = runs.isEmpty() ? null : runs.get(0).getValueInstant("startDate");
row(html, "Oldest restorable", oldest == null ? "Not known" : WHEN.format(oldest));
html.append("</dl>");
}
@@ -290,17 +378,17 @@ public final class BackupsHomeView
/*******************************************************************************
**
** Open a card.
*******************************************************************************/
private static void open(StringBuilder html, String title)
{
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0 0 .75rem;font-size:1rem;\">").append(esc(title)).append("</h3>");
html.append("<section style=\"").append(CARD).append("\"><h3 style=\"margin:0 0 .75rem;font-size:1rem;font-weight:600;\">").append(esc(title)).append("</h3>");
}
/*******************************************************************************
**
** Close a card.
*******************************************************************************/
private static void close(StringBuilder html)
{
@@ -310,17 +398,17 @@ public final class BackupsHomeView
/*******************************************************************************
**
** A muted empty-state line.
*******************************************************************************/
private static void empty(StringBuilder html, String text)
{
html.append("<p style=\"").append(MUTED).append("margin:0 0 .75rem;\">").append(esc(text)).append("</p>");
html.append("<p style=\"").append(MUTED).append("margin:0 0 .75rem;font-size:.9rem;\">").append(esc(text)).append("</p>");
}
/*******************************************************************************
**
** A name and value pair in a definition list.
*******************************************************************************/
private static void row(StringBuilder html, String name, String value)
{
@@ -330,17 +418,53 @@ public final class BackupsHomeView
/*******************************************************************************
**
** A small label over its value (a plan row cell).
*******************************************************************************/
private static void link(StringBuilder html, String prefix, String table, String text)
private static void cell(StringBuilder html, String name, String value)
{
html.append("<a href=\"").append(esc(prefix + "/" + table)).append("\">").append(esc(text)).append("</a>");
html.append("<div style=\"min-width:0;\"><div style=\"").append(MUTED).append("font-size:.75rem;\">").append(esc(name)).append("</div><div>").append(esc(value)).append("</div></div>");
}
/*******************************************************************************
** SOME_STATUS to Some status.
** A status chip: a colored dot (the only color) and the status label.
*******************************************************************************/
private static String chip(String status)
{
String color = switch(status == null ? "UNKNOWN" : status)
{
case "OK" -> "#16a34a";
case "LATE", "UNVERIFIED", "UNKNOWN" -> "#d97706";
default -> "#dc2626";
};
return ("<span style=\"" + CHIP + "\"><span aria-hidden=\"true\" style=\"width:.55rem;height:.55rem;border-radius:50%;background:" + color + ";display:inline-block;\"></span>" + esc(label(status)) + "</span>");
}
/*******************************************************************************
** A link to the table list at the end of a card.
*******************************************************************************/
private static void link(StringBuilder html, String prefix, String table, String text)
{
html.append("<a style=\"").append(LINK).append("font-size:.9rem;\" href=\"").append(esc(prefix + "/" + table)).append("\">").append(esc(text)).append("</a>");
}
/*******************************************************************************
** A link to a record.
*******************************************************************************/
private static String recordLink(String prefix, String table, Object id, String text)
{
return ("<a style=\"" + LINK + "\" href=\"" + esc(prefix + "/" + table + "/" + id) + "\">" + esc(text == null ? String.valueOf(id) : text) + "</a>");
}
/*******************************************************************************
** SOME_STATUS to Some status (OK stays OK).
*******************************************************************************/
static String label(String status)
{
@@ -348,6 +472,10 @@ public final class BackupsHomeView
{
return ("Unknown");
}
if("OK".equals(status))
{
return ("OK");
}
String s = status.replace('_', ' ').toLowerCase();
return (Character.toUpperCase(s.charAt(0)) + s.substring(1));
}
@@ -355,7 +483,32 @@ public final class BackupsHomeView
/*******************************************************************************
**
** An Integer as a Long, null-safe.
*******************************************************************************/
private static Long toLong(Integer value)
{
return (value == null ? null : value.longValue());
}
/*******************************************************************************
** Milliseconds as a short duration.
*******************************************************************************/
static String duration(long millis)
{
if(millis < 1000)
{
return (millis + " ms");
}
long seconds = millis / 1000;
return (seconds < 60 ? seconds + " s" : seconds / 60 + " min " + seconds % 60 + " s");
}
/*******************************************************************************
** Bytes as a short size.
*******************************************************************************/
static String bytes(Long value)
{
@@ -363,7 +516,7 @@ public final class BackupsHomeView
{
return ("Not recorded");
}
double v = value;
double v = value;
String[] units = { "B", "KB", "MB", "GB", "TB" };
int i = 0;
while(v >= 1024 && i < units.length - 1)
@@ -377,7 +530,7 @@ public final class BackupsHomeView
/*******************************************************************************
**
** A number with separators.
*******************************************************************************/
static String number(Long value)
{
@@ -387,7 +540,7 @@ public final class BackupsHomeView
/*******************************************************************************
**
** HTML-escape.
*******************************************************************************/
static String esc(String s)
{
@@ -0,0 +1,207 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.app;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.processes.DefaultPlanProcess;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** The Setup card on the Backups home: what a working backup needs (a plan, a
** storage bucket that resolves, a schedule, verification, a separate prune
** bucket), whether each is in place, and the next step. Computed from the plans
** the home already loaded and the runtime's storage resolver; it reads nothing
** else.
*******************************************************************************/
public final class BackupsSetupStatus
{
/*******************************************************************************
** OK: in place. TODO: backups need it. DEFAULT: working, with a default worth a
** look. OPTIONAL: a feature that is off until set up.
*******************************************************************************/
public enum State
{
OK, TODO, DEFAULT, OPTIONAL
}
/*******************************************************************************
** One line of the card.
*******************************************************************************/
public record Item(String label, String tableName, State state, String detail, String nextStep)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BackupsSetupStatus()
{
}
/*******************************************************************************
** Build the items from the plans.
*******************************************************************************/
public static List<Item> load(List<QRecord> plans)
{
BackupsQBitConfig config = BackupsRuntime.isInstalled() ? BackupsRuntime.getConfig() : null;
return (load(plans, config != null && hasContent(config.getSchedulerName()), config == null ? null : config.getHostSchedulerDescription(),
config != null && Boolean.TRUE.equals(config.getHostSchedulerRunsVerification())));
}
/*******************************************************************************
** Build the items from the plans, given whether a scheduler is configured (a plan
** cron only fires through the scheduler the qbit config names).
*******************************************************************************/
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured)
{
return (load(plans, schedulerConfigured, null, false));
}
/*******************************************************************************
** Build the items from the plans. A host that runs plan crons with its own tick
** (hostScheduler names it) has crons that fire; hostRunsVerification says whether
** that tick also runs the verify cron.
*******************************************************************************/
public static List<Item> load(List<QRecord> plans, boolean schedulerConfigured, String hostScheduler, boolean hostRunsVerification)
{
boolean host = hasContent(hostScheduler);
List<Item> items = new ArrayList<>();
List<QRecord> active = plans.stream().filter(p -> Boolean.TRUE.equals(p.getValueBoolean("isActive"))).toList();
///////////
// plans //
///////////
if(plans.isEmpty())
{
items.add(new Item("Backup plans", BackupsTables.PLAN, State.TODO, "No plan: nothing is backed up.",
"Run Create default plan on Backup plans (it asks only for the storage bucket alias), or add a plan with New."));
items.add(new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "None needed yet.",
"A plan that backs up everything (scope Instance) needs no rules. Add them on a plan that is limited to apps, qbits or tables."));
return (items);
}
items.add(active.isEmpty()
? new Item("Backup plans", BackupsTables.PLAN, State.TODO, plans.size() + (plans.size() == 1 ? " plan, but it is" : " plans, but none is") + " active: nothing runs on its own.", "Turn a plan on (Active).")
: new Item("Backup plans", BackupsTables.PLAN, State.OK, active.size() + " active plan" + (active.size() == 1 ? "" : "s") + " of " + plans.size() + ".", null));
/////////////////////////////////////////
// scope rules: needed unless INSTANCE //
/////////////////////////////////////////
List<String> limited = plans.stream().filter(p -> p.getValueString("scopeType") != null && !"INSTANCE".equals(p.getValueString("scopeType"))).map(p -> p.getValueString("name")).toList();
items.add(limited.isEmpty()
? new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OPTIONAL, "Every plan backs up the whole instance, so no rules are needed.",
"Add rules to a plan only to include or leave out apps, qbits, tables or roles.")
: new Item("Plan scope rules", BackupsTables.PLAN_SCOPE, State.OK, limited.size() + (limited.size() == 1 ? " plan is" : " plans are") + " limited by scope rules: " + String.join(", ", limited) + ".",
"Check each plan's Scope rules to see what it covers."));
///////////////////////////////////////////////
// storage: every alias a plan names resolves //
///////////////////////////////////////////////
Set<String> aliases = new LinkedHashSet<>();
for(QRecord plan : plans)
{
for(String field : List.of("storageBucketAlias", "pruneBucketAlias"))
{
if(plan.getValueString(field) != null)
{
aliases.add(plan.getValueString(field));
}
}
}
List<String> broken = new ArrayList<>();
for(String alias : aliases)
{
try
{
BackupsRuntime.storageFor(alias);
}
catch(RuntimeException e)
{
broken.add(alias);
}
}
items.add(broken.isEmpty()
? new Item("Storage", BackupsTables.PLAN, State.OK, aliases.size() + " storage bucket alias" + (aliases.size() == 1 ? "" : "es") + " in use, and each resolves.", "Run Test destination on a plan to prove it can write.")
: new Item("Storage", BackupsTables.PLAN, State.TODO, "These aliases do not resolve to a bucket: " + String.join(", ", broken) + ".",
"Fix the alias on the plan, or add the bucket to the application's storage resolver."));
/////////////////////////////////////////
// schedule: an active plan has a cron //
/////////////////////////////////////////
long scheduled = active.stream().filter(p -> hasContent(p.getValueString("cronExpression"))).count();
String noScheduler = "No scheduler is configured (schedulerName on the qbit config), so plan crons are saved but never fire";
items.add(!schedulerConfigured && !host
? new Item("Schedule", BackupsTables.PLAN, State.TODO, noScheduler + ". Backups run only when someone clicks Run backup now.",
"Set schedulerName on the qbit config (or run your own tick if the application does), then give each plan a cron expression.")
: scheduled == 0
? new Item("Schedule", BackupsTables.PLAN, State.TODO, "No active plan has a schedule: backups run only when someone clicks Run backup now.", "Set a Cron expression on a plan (for example " + DefaultPlanProcess.CRON + ").")
: new Item("Schedule", BackupsTables.PLAN, State.OK, scheduled + " active plan" + (scheduled == 1 ? " runs" : "s run") + " on a schedule"
+ (host && !schedulerConfigured ? ", run by " + hostScheduler + " (the application scheduler)." : "."), null));
//////////////////
// verification //
//////////////////
long verified = active.stream().filter(p -> hasContent(p.getValueString("verifyCronExpression"))).count();
items.add(host && !schedulerConfigured && !hostRunsVerification
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "Not run by " + hostScheduler + ": verify cron expressions are saved but nothing fires them.",
"Run Verify backup on a run by hand, or have the application scheduler run the backupVerifyPlan process for each plan.")
: !schedulerConfigured && !host
? new Item("Verification", BackupsTables.VERIFICATION, State.TODO, noScheduler + ", so no verification is scheduled.",
"Set schedulerName on the qbit config, then a verify cron expression on a plan; or run Verify backup on a run by hand.")
: verified == 0
? new Item("Verification", BackupsTables.VERIFICATION, State.OPTIONAL, "No plan verifies its backups on a schedule.",
"Set a Verify cron expression on a plan, or run Verify backup on a run. A backup nobody has restored is a guess.")
: new Item("Verification", BackupsTables.VERIFICATION, State.OK, verified + " active plan" + (verified == 1 ? " verifies" : "s verify") + " its latest backup on a schedule.", null));
////////////////////////////////////////////
// prune bucket: separate delete authority //
////////////////////////////////////////////
long noPrune = plans.stream().filter(p -> !hasContent(p.getValueString("pruneBucketAlias"))).count();
items.add(noPrune == 0
? new Item("Prune bucket", BackupsTables.PLAN, State.OK, "Every plan prunes through its own delete-capable bucket alias.", null)
: new Item("Prune bucket", BackupsTables.PLAN, State.DEFAULT, noPrune + (noPrune == 1 ? " plan has" : " plans have") + " no prune bucket: Prune uses the backup bucket, so one key can both write and delete backups.",
"Give the plan a separate delete-capable Prune bucket alias."));
return (items);
}
/*******************************************************************************
**
*******************************************************************************/
private static boolean hasContent(String value)
{
return (value != null && !value.isBlank());
}
}
@@ -22,9 +22,11 @@ import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.function.Consumer;
import java.util.zip.GZIPOutputStream;
import com.kingsrook.qbits.backups.storage.BackupStorage;
import com.kingsrook.qbits.backups.storage.WriteOptions;
@@ -65,6 +67,10 @@ public class TableArtifactWriter implements Closeable
private String firstKeyOfTable;
private String lastKeyOfTable;
private Consumer<PartResult> partListener;
private String objectLockMode;
private Instant objectLockUntil;
/***************************************************************************
@@ -122,6 +128,63 @@ public class TableArtifactWriter implements Closeable
/*******************************************************************************
** Be told about each part as it is committed to storage (so the catalog records
** a part the moment it is complete, which is what a resume starts from).
*******************************************************************************/
public TableArtifactWriter withPartListener(Consumer<PartResult> partListener)
{
this.partListener = partListener;
return (this);
}
/*******************************************************************************
** Apply an object lock to every part written.
*******************************************************************************/
public TableArtifactWriter withObjectLock(String mode, Instant until)
{
this.objectLockMode = mode;
this.objectLockUntil = until;
return (this);
}
/*******************************************************************************
** Continue after parts that were already written (resume): numbering carries on
** after them and the table totals include them. The caller then feeds the rows
** hash with seedRowsHash.
*******************************************************************************/
public void resumeFrom(List<PartResult> priorParts)
{
if(priorParts.isEmpty())
{
return;
}
parts.addAll(priorParts);
partNumber = priorParts.get(priorParts.size() - 1).partNumber();
for(PartResult part : priorParts)
{
totalRows += part.rows();
}
firstKeyOfTable = priorParts.get(0).firstKey();
lastKeyOfTable = priorParts.get(priorParts.size() - 1).lastKey();
}
/*******************************************************************************
** Feed already-written canonical JSONL bytes into the rows hash (resume).
*******************************************************************************/
public void seedRowsHash(byte[] bytes, int offset, int length)
{
rowsDigest.update(bytes, offset, length);
}
/*******************************************************************************
** Write one record.
*******************************************************************************/
@@ -245,6 +308,10 @@ public class TableArtifactWriter implements Closeable
WriteOptions options = WriteOptions.create().withContentType("application/octet-stream")
.withMetadata("qqq-backup-run", runKey)
.withMetadata("qqq-backup-table", table.getName());
if(objectLockMode != null)
{
options.withObjectLock(objectLockMode, objectLockUntil);
}
OutputStream toStorage = storage.openWrite(current.key, options);
current.stored = new HashingOutputStream(toStorage);
OutputStream transformed = transform.wrap(current.stored);
@@ -266,9 +333,14 @@ public class TableArtifactWriter implements Closeable
private void closePart() throws IOException
{
current.raw.close();
parts.add(new PartResult(partNumber, current.key, current.rows, current.raw.getCount(), current.stored.getCount(),
PartResult result = new PartResult(partNumber, current.key, current.rows, current.raw.getCount(), current.stored.getCount(),
current.raw.getSha256(), current.stored.getSha256(), current.firstKey, current.lastKey, transform.encrypts(),
csv ? "CSV" : "JSONL", gzip ? "GZIP" : "NONE"));
csv ? "CSV" : "JSONL", gzip ? "GZIP" : "NONE");
parts.add(result);
current = null;
if(partListener != null)
{
partListener.accept(result);
}
}
}
@@ -57,6 +57,8 @@ public class FileCopier
private final FileStoreRegistry registry;
private final boolean referenceOnly;
private final String runKey;
private String objectLockMode;
private java.time.Instant objectLockUntil;
@@ -119,6 +121,19 @@ public class FileCopier
/*******************************************************************************
** Lock every object this copier writes (a server-side copy cannot set a lock,
** so with a lock the bytes are streamed instead).
*******************************************************************************/
public FileCopier withObjectLock(String mode, java.time.Instant until)
{
this.objectLockMode = mode;
this.objectLockUntil = until;
return (this);
}
/*******************************************************************************
** The content-addressed key for a hash: prefix/ab/cd/sha256.
*******************************************************************************/
@@ -177,7 +192,7 @@ public class FileCopier
*******************************************************************************/
private String copy(BackupStorage source, FileReference ref, String key) throws IOException
{
if(source == destination && !transform.encrypts())
if(source == destination && !transform.encrypts() && objectLockMode == null)
{
destination.copy(ref.key(), key);
return (null);
@@ -185,6 +200,10 @@ public class FileCopier
WriteOptions options = WriteOptions.create().withContentType(ref.contentType() == null ? "application/octet-stream" : ref.contentType())
.withMetadata("qqq-backup-run", runKey);
if(objectLockMode != null)
{
options.withObjectLock(objectLockMode, objectLockUntil);
}
try(InputStream in = source.openRead(ref.key());
HashingOutputStream plain = new HashingOutputStream(transform.wrap(destination.openWrite(key, options))))
{
@@ -32,6 +32,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.middleware.health.HealthIndicator;
import com.kingsrook.qqq.middleware.health.model.metadata.HealthCheckResult;
import com.kingsrook.qqq.middleware.health.model.metadata.HealthStatus;
@@ -79,7 +80,7 @@ public class BackupsHealthIndicator implements HealthIndicator
boolean ownsContext = QContext.getQInstance() == null;
if(ownsContext)
{
QContext.init(qInstance, new QSession());
QContext.init(qInstance, systemSession(qInstance));
}
try
{
@@ -117,4 +118,16 @@ public class BackupsHealthIndicator implements HealthIndicator
}
}
}
/*******************************************************************************
** The session for a health request that has no QQQ context. It must be the
** core system user: token-based authentication modules (OAuth2, Auth0)
** reject any other tokenless session, which failed the check on such hosts.
*******************************************************************************/
static QSession systemSession(QInstance qInstance)
{
return (new QSystemUserSession(qInstance));
}
}
@@ -137,6 +137,7 @@ public record BackupManifest(
String backend,
String primaryKey,
String fingerprint,
String fingerprintAlgorithm,
Map<String, Object> schema,
String filter,
List<ExcludedField> excludedFields,
@@ -0,0 +1,170 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
/*******************************************************************************
** Frozen pre-RC.6 fingerprint algorithm, used only to read existing manifests.
** Keep its descriptor and escaping byte-compatible with the original writer.
*******************************************************************************/
final class LegacyTableFingerprint
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private LegacyTableFingerprint()
{
}
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -18,28 +18,23 @@ package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint;
/*******************************************************************************
** A stable schema descriptor and SHA-256 fingerprint of a table (spec 5.5).
**
** This is an internal stand-in for the QQQ core TableSchemaFingerprint (core PR
** feature/schema-fingerprint). When that ships in the next 4.1.x RC, this class
** delegates to it and the descriptor below is kept only for reading old manifests.
** Labels, adornments and help text are excluded, so cosmetic changes do not count.
** Core schema fingerprints with read compatibility for pre-RC.6 manifests.
*******************************************************************************/
public final class TableFingerprints
{
public static final String CORE_ALGORITHM = "qqq-table-schema-v1";
public static final String LEGACY_ALGORITHM = "qbit-backups-table-v1";
/*******************************************************************************
** Utility class.
@@ -51,134 +46,83 @@ public final class TableFingerprints
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
** The core descriptor stored in new manifests.
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
return (TableSchemaFingerprint.of(table, backendType).getDescriptor());
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
** Retain the manifest's sha256 prefix around the core hash.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
return ("sha256:" + TableSchemaFingerprint.of(table, backendType).getHash());
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
** An absent algorithm identifies manifests written before RC.6. Unknown
** algorithms must not silently fall back to a different fingerprint.
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
public static boolean matches(BackupManifest.TableEntry entry, QTableMetaData table, String backendType)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
if(entry.fingerprint() == null)
{
return (false);
}
String algorithm = entry.fingerprintAlgorithm();
if(algorithm == null || LEGACY_ALGORITHM.equals(algorithm))
{
return (entry.fingerprint().equals(LegacyTableFingerprint.of(table, backendType)));
}
return (CORE_ALGORITHM.equals(algorithm) && entry.fingerprint().equals(of(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a whole instance's tables: a hash of the table fingerprints.
** Normalize both descriptor versions to field-name maps for restore's
** compatibility analysis. Core fields use a list and explicit flag names.
*******************************************************************************/
@SuppressWarnings("unchecked")
public static Map<String, Object> fields(Map<String, Object> descriptor)
{
Object fields = descriptor == null ? null : descriptor.get("fields");
if(fields instanceof Map<?, ?> map)
{
return ((Map<String, Object>) map);
}
Map<String, Object> result = new LinkedHashMap<>();
if(fields instanceof List<?> list)
{
for(Object item : list)
{
Map<String, Object> field = (Map<String, Object>) item;
Map<String, Object> normalized = new LinkedHashMap<>();
normalized.put("type", field.get("type"));
normalized.put("required", field.get("isRequired"));
normalized.put("maxLength", field.get("maxLength"));
normalized.put("pvs", field.get("possibleValueSource"));
normalized.put("heavy", field.get("isHeavy"));
normalized.put("sensitive", field.get("isSensitive"));
result.put((String) field.get("name"), normalized);
}
}
return (result);
}
/*******************************************************************************
** The instance fingerprint remains a hash of the ordered table fingerprints.
*******************************************************************************/
public static String combine(List<String> tableFingerprints)
{
return ("sha256:" + HashingOutputStream.sha256Hex(String.join("\n", tableFingerprints).getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -23,9 +23,22 @@ import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.permissions.BackupsPermissionChecker;
import com.kingsrook.qbits.backups.plan.BackupPlanTableCustomizer;
import com.kingsrook.qbits.backups.schedule.BackupPlanSyncToScheduledJobProcess;
import com.kingsrook.qbits.backups.schedule.BackupPlanVerifySyncToScheduledJobProcess;
import com.kingsrook.qqq.backend.core.actions.customizers.MultiCustomizer;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceWithProperties;
import com.kingsrook.qqq.backend.core.scheduler.processes.BaseSyncToScheduledJobTableCustomizer;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
@@ -63,7 +76,7 @@ public final class BackupsTables
public static final List<String> PROCESS_WRITTEN = List.of(RUN, ARTIFACT, VERIFICATION, RESTORE_RUN, RESTORE_RUN_TABLE, FILE_OBJECT, ID_MAP);
/** The tables shown in the nav (sub-tables appear as relations on their parents). */
public static final List<String> NAV_TABLES = List.of(PLAN, RUN, RESTORE_RUN, VERIFICATION, FILE_OBJECT);
public static final List<String> NAV_TABLES = List.of(PLAN, RUN, RESTORE_RUN, VERIFICATION);
public static final List<String> ALL = List.of(PLAN, PLAN_SCOPE, RUN, ARTIFACT, VERIFICATION, RESTORE_RUN, RESTORE_RUN_TABLE, FILE_OBJECT, ID_MAP);
@@ -91,14 +104,14 @@ public final class BackupsTables
ENUMS.put("lock_mode", List.of("NONE", "GOVERNANCE", "COMPLIANCE"));
ENUMS.put("sla_status", List.of("OK", "LATE", "FAILING", "MISSING_IN_STORAGE", "UNVERIFIED", "ABANDONED", "OWNER_INVALID", "UNKNOWN"));
ENUMS.put("run_trigger", List.of("SCHEDULED", "MANUAL", "PRE_RESTORE", "RESUME", "IMPORTED"));
ENUMS.put("run_status", List.of("QUEUED", "RUNNING", "SUCCEEDED", "SUCCEEDED_WITH_WARNINGS", "FAILED", "TIMED_OUT", "CANCELLED", "ABANDONED"));
ENUMS.put("run_status", List.of("QUEUED", "RUNNING", "SUCCEEDED", "SUCCEEDED_WITH_WARNINGS", "FAILED", "TIMED_OUT", "CANCELLING", "CANCELLED", "ABANDONED"));
ENUMS.put("retention_class", List.of("DAILY", "WEEKLY", "MONTHLY", "YEARLY", "NONE"));
ENUMS.put("verification_status", List.of("UNVERIFIED", "STORAGE_OK", "CONTENT_OK", "TEST_RESTORE_OK", "FAILED"));
ENUMS.put("artifact_type", List.of("TABLE_DATA", "TABLE_BLOBS", "FILE_OBJECTS_INDEX", "NATIVE_DUMP", "MANIFEST", "CHECKSUMS"));
ENUMS.put("artifact_status", List.of("WRITING", "COMPLETE", "FAILED"));
ENUMS.put("verify_level", List.of("STORAGE", "CONTENT", "TEST_RESTORE"));
ENUMS.put("verify_result", List.of("PASSED", "PASSED_WITH_WARNINGS", "FAILED", "RUNNING", "ERROR"));
ENUMS.put("restore_status", List.of("PREVIEWED", "QUEUED", "RUNNING", "SUCCEEDED", "SUCCEEDED_WITH_WARNINGS", "FAILED", "CANCELLED"));
ENUMS.put("restore_status", List.of("PREVIEWED", "QUEUED", "RUNNING", "SUCCEEDED", "SUCCEEDED_WITH_WARNINGS", "FAILED", "CANCELLING", "CANCELLED"));
ENUMS.put("restore_mode", List.of("INSERT_NEW", "UPSERT", "REPLACE"));
ENUMS.put("id_strategy", List.of("PRESERVE", "REMAP_NEW_IDS"));
ENUMS.put("schema_status", List.of("MATCH", "COMPATIBLE", "MISMATCH"));
@@ -203,7 +216,111 @@ public final class BackupsTables
public static List<QTableMetaData> tables(BackupsQBitConfig config)
{
return (List.of(plan(config), planScope(config), run(config), artifact(config), verification(config),
restoreRun(config), restoreRunTable(config), fileObject(config), idMap(config)));
restoreRun(config), restoreRunTable(config), fileObject(config), idMap(config)).stream().map(BackupsTables::withSections).toList());
}
/*******************************************************************************
** Group a table's fields into record-screen sections (the identity and status
** first, then one section per concern, then the system fields), so a record reads
** as a few short groups instead of one long list. Fields not named here land in
** the last section.
*******************************************************************************/
static QTableMetaData withSections(QTableMetaData table)
{
List<Object[]> layout = switch(table.getName())
{
case PLAN -> List.of(
new Object[] { "plan", "Plan", "backup", Tier.T1, "name", "description", "planKey", "isActive", "scopeType", "slaStatus", "lastSuccessDate", "lastVerifiedDate" },
new Object[] { "schedule", "Schedule and SLA", "schedule", Tier.T2, "cronExpression", "cronTimeZoneId", "scheduledJobId", "slaHours", "verifySlaDays", "verifyCronExpression", "verifyLevel" },
new Object[] { "where", "Scope and storage", "storage", Tier.T2, "includeRoles", "storageBucketAlias", "pruneBucketAlias", "prefix", "critical", "consistency", "useReadReplica", "incrementalMode", "nativeDump" },
new Object[] { "format", "Format and safety", "shield", Tier.T2, "format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "heavyFieldPolicy", "fileObjectPolicy", "fileMissingPolicy" },
new Object[] { "retention", "Retention and lock", "lock_clock", Tier.T2, "retainDaily", "retainWeekly", "retainMonthly", "retainYearly", "retainMinimum", "objectLockMode", "objectLockDays" },
new Object[] { "system", "Limits and history", "history", Tier.T3, "maxDurationMinutes", "pageSize", "targetPartBytes", "ownerUserId", "lastRunId", "lastSuccessRunId", "lastVerifiedRunId", "id", "createDate", "modifyDate" });
case RUN -> List.of(
new Object[] { "run", "Run", "history", Tier.T1, "runKey", "backupPlanId", "status", "trigger", "startDate", "endDate", "durationMs", "errorMessage" },
new Object[] { "volume", "Rows and size", "data_usage", Tier.T2, "tablesPlanned", "tablesDone", "rowsTotal", "bytesRaw", "bytesStored", "consistencyAchieved", "warningCount" },
new Object[] { "files", "Files", "attach_file", Tier.T2, "filesReferenced", "filesCopied", "filesDeduplicated", "filesMissing", "fileBytesCopied", "fileBytesReferenced" },
new Object[] { "storage", "Storage and integrity", "storage", Tier.T2, "storageBucketAlias", "runPrefix", "manifestKey", "manifestSha256", "snapshotId", "schemaFingerprint", "securityScope", "encryptionKeyRef", "encryptionKeyFingerprint", "verificationStatus" },
new Object[] { "retention", "Retention", "lock_clock", Tier.T2, "retentionClass", "lockedUntil", "prunedDate", "prunedByUserId" },
new Object[] { "system", "System", "settings", Tier.T3, "requestedByUserId", "heartbeatDate", "qqqVersion", "appVersion", "instanceName", "errorDetail", "id", "createDate", "modifyDate" });
case VERIFICATION -> List.of(
new Object[] { "result", "Result", "verified", Tier.T1, "backupRunId", "level", "status", "tablesChecked", "rowsChecked", "mismatches", "scratchTarget", "startDate", "endDate" },
new Object[] { "system", "Detail", "settings", Tier.T3, "trigger", "requestedByUserId", "detailJson", "id", "createDate", "modifyDate" });
case RESTORE_RUN -> List.of(
new Object[] { "restore", "Restore", "restore", Tier.T1, "sourceBackupRunId", "status", "mode", "idStrategy", "matchOn", "targetInstanceName", "startDate", "endDate", "errorMessage" },
new Object[] { "rows", "Rows", "table_rows", Tier.T2, "rowsInserted", "rowsUpdated", "rowsUnchanged", "rowsDeleted", "rowsSkipped", "rowsErrored", "sequencesResynced" },
new Object[] { "files", "Files", "attach_file", Tier.T2, "filesRestored", "filesAlreadyPresent", "filesConflicting", "filesMissingFromBackup", "fileBytesRestored" },
new Object[] { "system", "Safety and system", "settings", Tier.T3, "preRestoreBackupRunId", "scopeSummary", "automationsSuppressed", "esbSuppressed", "confirmationText", "requestedByUserId", "id", "createDate", "modifyDate" });
default -> List.of();
};
if(layout.isEmpty())
{
return (table);
}
java.util.Set<String> placed = new java.util.HashSet<>();
for(int i = 0; i < layout.size(); i++)
{
Object[] spec = layout.get(i);
List<String> fields = new ArrayList<>();
for(int f = 4; f < spec.length; f++)
{
fields.add((String) spec[f]);
}
if(i == layout.size() - 1)
{
table.getFields().keySet().stream().filter(name -> !placed.contains(name) && !fields.contains(name)).forEach(fields::add);
}
placed.addAll(fields);
table.withSection(new QFieldSection((String) spec[0], (String) spec[1], new QIcon((String) spec[2]), (Tier) spec[3], fields));
}
for(ChildList childList : CHILD_LISTS)
{
if(childList.parent().equals(table.getName()))
{
table.withSection(new QFieldSection(childList.key(), new QIcon(childList.icon()), Tier.T2).withLabel(childList.label()).withWidgetName(childList.joinName()));
}
}
return (table);
}
/** A child list on a parent record: its join, the columns shown, and the order. */
record ChildList(String parent, String joinName, String key, String label, String icon, int maxRows, boolean canAdd, String orderField, List<String> columns)
{
}
/** The relations shown on the record screens (sub-tables are child lists here, not in the nav). */
static final List<ChildList> CHILD_LISTS = List.of(
new ChildList(PLAN, "backupsPlanScopeJoinPlan", "scopeRules", "Scope rules", "rule", 50, true, "id", List.of("action", "selectorType", "selectorValue", "includeAssociatedChildren", "fieldExclusions")),
new ChildList(PLAN, "backupsRunJoinPlan", "runs", "Runs", "history", 10, false, "id", List.of("runKey", "status", "startDate", "durationMs", "rowsTotal", "bytesStored")),
new ChildList(RUN, "backupsArtifactJoinRun", "artifacts", "Artifacts", "inventory_2", 100, false, "id", List.of("tableName", "artifactType", "partNumber", "rowCount", "bytesStored", "status")),
new ChildList(RUN, "backupsVerificationJoinRun", "verifications", "Verifications", "verified", 20, false, "id", List.of("level", "status", "tablesChecked", "rowsChecked", "mismatches", "startDate")),
new ChildList(RUN, "backupsRestoreRunJoinRun", "restores", "Restores", "restore", 20, false, "id", List.of("status", "mode", "startDate", "rowsInserted", "rowsUpdated", "rowsErrored")),
new ChildList(RESTORE_RUN, "backupsRestoreRunTableJoinRestoreRun", "restoreTables", "Tables restored", "table_rows", 100, false, "id",
List.of("tableName", "targetTableName", "rowsInserted", "rowsUpdated", "rowsUnchanged", "rowsErrored", "schemaStatus")));
/*******************************************************************************
** The child-list widgets for the record screens (see CHILD_LISTS). Each shows a
** few readable columns, newest first, and never scrolls sideways.
*******************************************************************************/
public static List<QWidgetMetaData> widgets()
{
Map<String, QJoinMetaData> byName = new LinkedHashMap<>();
joins().forEach(join -> byName.put(join.getName(), join));
List<QWidgetMetaData> rs = new ArrayList<>();
for(ChildList childList : CHILD_LISTS)
{
rs.add(ChildRecordListRenderer.widgetMetaDataBuilder(byName.get(childList.joinName())).withLabel(childList.label()).withMaxRows(childList.maxRows())
.withCanAddChildRecord(childList.canAdd()).withOnlyIncludeFieldNames(childList.columns())
.withOrderBys(List.of(new QFilterOrderBy(childList.orderField(), !PLAN.equals(childList.parent()) || childList.canAdd())))
.getWidgetMetaData().withPermissionRules(permissionRules()));
}
return (rs);
}
@@ -286,6 +403,34 @@ public final class BackupsTables
** backups_plan: what, how, where, when and how long (spec 4.1).
*******************************************************************************/
static QTableMetaData plan(BackupsQBitConfig config)
{
QTableMetaData table = planBase(config);
////////////////////////////////////////////////////////////////////////////////////////
// save validation (spec 4.1, 11.1), and the plan -> scheduled job sync when the qbit //
// has a scheduler (spec 9.1): one job per active plan, and one for the verify cron //
////////////////////////////////////////////////////////////////////////////////////////
QCodeReference validation = new QCodeReference(BackupPlanTableCustomizer.class);
table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, validation);
table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, validation);
if(StringUtils.hasContent(config.getSchedulerName()))
{
QCodeReferenceWithProperties sync = MultiCustomizer.of(
BaseSyncToScheduledJobTableCustomizer.makeCodeReference(table, new BackupPlanSyncToScheduledJobProcess()),
BaseSyncToScheduledJobTableCustomizer.makeCodeReference(table, new BackupPlanVerifySyncToScheduledJobProcess()));
table.withCustomizer(TableCustomizers.POST_INSERT_RECORD, sync);
table.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, sync);
table.withCustomizer(TableCustomizers.POST_DELETE_RECORD, sync);
}
return (table);
}
/*******************************************************************************
** The plan table before the customizers.
*******************************************************************************/
private static QTableMetaData planBase(BackupsQBitConfig config)
{
return (base(config, PLAN, "Backup Plan", "backup")
.withRecordLabelFormatAndFields("%s", "name")
@@ -302,8 +447,8 @@ public final class BackupsTables
.withField(new QFieldMetaData("pruneBucketAlias", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("prefix", QFieldType.STRING).withMaxLength(255))
.withField(new QFieldMetaData("critical", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(enumField("format", "format", "JSONL"))
.withField(enumField("compression", "compression", "GZIP"))
.withField(enumField("format", "format", "JSONL").withHelpContent(new QHelpContent("JSONL, CSV and JSONL_AND_CSV are available. PARQUET is not available yet and is refused when the plan is saved.")))
.withField(enumField("compression", "compression", "GZIP").withHelpContent(new QHelpContent("GZIP and NONE are available. ZSTD is not available yet and is refused when the plan is saved.")))
.withField(enumField("encryptionMode", "encryption_mode", "NONE"))
.withField(new QFieldMetaData("encryptionKeyRef", QFieldType.STRING).withMaxLength(100))
.withField(enumField("sensitiveFieldPolicy", "sensitive_policy", "EXCLUDE"))
@@ -311,12 +456,15 @@ public final class BackupsTables
.withField(enumField("fileObjectPolicy", "file_policy", "COPY"))
.withField(enumField("fileMissingPolicy", "file_missing_policy", "WARN"))
.withField(enumField("consistency", "consistency", "SNAPSHOT_PREFERRED"))
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(enumField("incrementalMode", "incremental_mode", "FULL"))
.withField(enumField("nativeDump", "native_dump", "NONE"))
.withField(new QFieldMetaData("useReadReplica", QFieldType.BOOLEAN).withDefaultValue(false)
.withHelpContent(new QHelpContent("Not available yet: backups always read from the application's own backend. Leave this off; turning it on is refused when the plan is saved.")))
.withField(enumField("incrementalMode", "incremental_mode", "FULL")
.withHelpContent(new QHelpContent("Only FULL is available yet. CHANGED_SINCE_LAST is not available yet and is refused when the plan is saved.")))
.withField(enumField("nativeDump", "native_dump", "NONE")
.withHelpContent(new QHelpContent("Only NONE is available yet. Native database dumps (PG_DUMP_CUSTOM) are not available yet and are refused when the plan is saved.")))
.withField(new QFieldMetaData("cronExpression", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("cronTimeZoneId", QFieldType.STRING).withMaxLength(60))
.withField(roInt("scheduledJobId"))
.withField(roInt("scheduledJobId").withHelpContent(new QHelpContent("Not used: a plan's schedules are scheduled jobs found by their foreign key (backups_plan for the backup, backups_plan_verify for the verification), so look there.")))
.withField(new QFieldMetaData("slaHours", QFieldType.INTEGER).withDefaultValue(26))
.withField(new QFieldMetaData("verifySlaDays", QFieldType.INTEGER).withDefaultValue(8))
.withField(new QFieldMetaData("verifyFileSamplePercent", QFieldType.INTEGER).withDefaultValue(100))
@@ -369,8 +517,6 @@ public final class BackupsTables
return (base(config, RUN, "Backup Run", "history")
.withRecordLabelFormatAndFields("%s", "runKey")
.withUniqueKey(new UniqueKey("runKey"))
.withAssociation(new Association().withName("artifacts").withAssociatedTableName(ARTIFACT).withJoinName("backupsArtifactJoinRun"))
.withAssociation(new Association().withName("verifications").withAssociatedTableName(VERIFICATION).withJoinName("backupsVerificationJoinRun"))
.withField(ro("backupPlanId", QFieldType.INTEGER).withPossibleValueSourceName(PLAN))
.withField(ro("runKey", QFieldType.STRING))
.withField(roEnum("trigger", "run_trigger"))
@@ -476,7 +622,6 @@ public final class BackupsTables
{
return (base(config, RESTORE_RUN, "Backup Restore", "restore")
.withRecordLabelFormatAndFields("Restore %s", "id")
.withAssociation(new Association().withName("tables").withAssociatedTableName(RESTORE_RUN_TABLE).withJoinName("backupsRestoreRunTableJoinRestoreRun"))
.withField(ro("sourceBackupRunId", QFieldType.INTEGER).withPossibleValueSourceName(RUN))
.withField(ro("requestedByUserId", QFieldType.STRING))
.withField(ro("startDate", QFieldType.DATE_TIME))
@@ -0,0 +1,144 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.plan;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.run.BackupRunFactory;
import com.kingsrook.qbits.backups.scope.PlanSettings;
import com.kingsrook.qbits.backups.scope.ScopeResolver;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
/*******************************************************************************
** Plan save validation (pre insert and pre update of backups_plan). See
** BackupPlanValidator for the rules. An update that touches none of the plan
** configuration (the processes write the last run and SLA fields) is not
** validated again.
**
** The saving user becomes the plan owner when none is given.
*******************************************************************************/
public class BackupPlanTableCustomizer implements TableCustomizerInterface
{
private static final List<String> CONFIGURATION_FIELDS = List.of("planKey", "isActive", "scopeType", "includeRoles", "storageBucketAlias", "pruneBucketAlias",
"format", "compression", "encryptionMode", "encryptionKeyRef", "sensitiveFieldPolicy", "incrementalMode", "nativeDump", "useReadReplica", "cronExpression", "cronTimeZoneId",
"verifyCronExpression", "retainDaily", "retainWeekly", "retainMonthly", "retainYearly", "retainMinimum", "objectLockMode", "objectLockDays",
"slaHours", "pageSize", "ownerUserId");
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean isPreview, Optional<List<QRecord>> oldRecordList) throws QException
{
Map<Serializable, QRecord> oldById = oldRecordList.isPresent() ? oldRecordListToMap("id", oldRecordList).orElse(Map.of()) : Map.of();
BackupCatalog catalog = new BackupCatalog();
for(QRecord record : records)
{
QRecord old = oldRecordList.isPresent() ? oldById.get(record.getValue("id")) : null;
if(old != null && !touchesConfiguration(record))
{
continue;
}
QRecord merged = new QRecord();
if(old != null)
{
merged.getValues().putAll(old.getValues());
}
merged.getValues().putAll(record.getValues());
if(merged.getValue("ownerUserId") == null && Boolean.TRUE.equals(merged.getValueBoolean("isActive")))
{
QSession session = QContext.getQSession();
String owner = session == null || session.getUser() == null ? null : session.getUser().getIdReference();
if(owner != null)
{
record.setValue("ownerUserId", owner);
merged.setValue("ownerUserId", owner);
}
}
boolean hasRun = old != null && !catalog.where(BackupsTables.RUN, "backupPlanId", old.getValueInteger("id")).isEmpty();
BackupPlanValidator.Result result = BackupPlanValidator.validate(PlanSettings.of(merged), old == null ? null : old.getValueString("planKey"), hasRun, scopeRules(record, old));
for(String error : result.errors())
{
record.addError(new BadInputStatusMessage(error));
}
for(String warning : result.warnings())
{
record.addWarning(new QWarningMessage(warning));
}
}
return (records);
}
/*******************************************************************************
**
*******************************************************************************/
private static boolean touchesConfiguration(QRecord record)
{
for(String field : CONFIGURATION_FIELDS)
{
if(record.getValues().containsKey(field))
{
return (true);
}
}
return (false);
}
/*******************************************************************************
** The scope rules to resolve: the ones sent with the plan (associated rows), or
** the saved ones.
*******************************************************************************/
private static List<ScopeResolver.ScopeRule> scopeRules(QRecord record, QRecord old) throws QException
{
List<QRecord> sent = record.getAssociatedRecords() == null ? null : record.getAssociatedRecords().get("scopes");
if(sent != null)
{
List<ScopeResolver.ScopeRule> rules = new ArrayList<>();
for(QRecord row : sent)
{
rules.add(new ScopeResolver.ScopeRule(!"EXCLUDE".equals(row.getValueString("action")), row.getValueString("selectorType"), row.getValueString("selectorValue"),
row.getValueString("filterJson"), !Boolean.FALSE.equals(row.getValueBoolean("includeAssociatedChildren")), row.getValueString("fieldExclusions")));
}
return (rules);
}
return (old == null ? List.of() : BackupRunFactory.scopeRules(old.getValueInteger("id")));
}
}
@@ -0,0 +1,358 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.plan;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.regex.Pattern;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.permissions.BackupsPermissionChecker;
import com.kingsrook.qbits.backups.permissions.BackupsPermissions;
import com.kingsrook.qbits.backups.scope.PlanSettings;
import com.kingsrook.qbits.backups.scope.ScopeResolver;
import com.kingsrook.qbits.backups.storage.BackupStorage;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
/*******************************************************************************
** The rules a backup plan must satisfy when it is saved or activated (spec 4.1,
** 6.2, 11.1, 11.3): the refusals are clear messages, not run-time failures.
**
** - the saving user has backups.admin and may read every table the plan resolves
** to (a scheduled run is full-fidelity, so whoever activates the plan must be
** allowed to read all of it)
** - an object lock mode is checked against what the storage bucket can do
** - INCLUDE_ENCRYPTED_ONLY needs encryption
** - formats, compressions and options that are not built yet are refused
** - the plan key is a slug, and cannot change after the first run (it is in every
** object key)
*******************************************************************************/
public final class BackupPlanValidator
{
private static final Pattern PLAN_KEY = Pattern.compile("[a-z0-9][a-z0-9_-]{0,59}");
private static final Set<String> FORMATS = Set.of("JSONL", "CSV", "JSONL_AND_CSV");
private static final Set<String> COMPRESSIONS = Set.of("GZIP", "NONE");
private static final Set<String> LOCK_MODES = Set.of("NONE", "GOVERNANCE", "COMPLIANCE");
/** Replace with the real check when client-side encryption ships (spec 11.2). */
private static volatile boolean encryptionAvailable = false;
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BackupPlanValidator()
{
}
/*******************************************************************************
** Tell the validator that client-side encryption is available (the encryption
** work calls this when it installs a cipher).
*******************************************************************************/
public static void setEncryptionAvailable(boolean available)
{
encryptionAvailable = available;
}
/*******************************************************************************
** The outcome of a validation.
*******************************************************************************/
public record Result(List<String> errors, List<String> warnings)
{
/*******************************************************************************
**
*******************************************************************************/
public boolean ok()
{
return (errors.isEmpty());
}
}
/*******************************************************************************
** Validate the complete settings of a plan.
**
** @param settings the plan values as they will be after the save
** @param oldPlanKey the plan key before the save, null for a new plan
** @param hasRun whether the plan already has a run
** @param rules the scope rules of the plan
*******************************************************************************/
public static Result validate(PlanSettings settings, String oldPlanKey, boolean hasRun, List<ScopeResolver.ScopeRule> rules)
{
List<String> errors = new ArrayList<>();
List<String> warnings = new ArrayList<>();
validateKey(settings, oldPlanKey, hasRun, errors);
validateFormats(settings, errors);
validateEncryption(settings, errors);
validateStorage(settings, errors, warnings);
validateSchedule(settings, errors);
validateNumbers(settings, errors);
if(settings.bool("isActive", true))
{
validateOwner(settings, rules, errors);
}
return (new Result(errors, warnings));
}
/*******************************************************************************
**
*******************************************************************************/
private static void validateKey(PlanSettings settings, String oldPlanKey, boolean hasRun, List<String> errors)
{
String planKey = settings.string("planKey", null);
if(planKey == null || !PLAN_KEY.matcher(planKey).matches())
{
errors.add("The plan key must be lowercase letters, digits, dash or underscore (up to 60 characters): it is part of every object key.");
}
if(oldPlanKey != null && planKey != null && !oldPlanKey.equals(planKey) && hasRun)
{
errors.add("The plan key cannot change after the plan has run: it is part of the object keys of its backups. Create a new plan instead.");
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void validateFormats(PlanSettings settings, List<String> errors)
{
String format = settings.string("format", "JSONL");
if("PARQUET".equals(format))
{
errors.add("The PARQUET format is not available yet. Use JSONL, CSV or JSONL_AND_CSV.");
}
else if(!FORMATS.contains(format))
{
errors.add("Unknown format " + format + ". Use JSONL, CSV or JSONL_AND_CSV.");
}
String compression = settings.string("compression", "GZIP");
if("ZSTD".equals(compression))
{
errors.add("ZSTD compression is not available yet. Use GZIP or NONE.");
}
else if(!COMPRESSIONS.contains(compression))
{
errors.add("Unknown compression " + compression + ". Use GZIP or NONE.");
}
if("CHANGED_SINCE_LAST".equals(settings.string("incrementalMode", "FULL")))
{
errors.add("Incremental backups are not available yet. Use FULL.");
}
if(!"NONE".equals(settings.string("nativeDump", "NONE")))
{
errors.add("Native database dumps are not available yet. Use NONE.");
}
if(settings.bool("useReadReplica", false))
{
errors.add("Reading from a replica is not available yet: backups read from the application's own backend. Turn Use read replica off.");
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void validateEncryption(PlanSettings settings, List<String> errors)
{
String mode = settings.string("encryptionMode", "NONE");
String policy = settings.string("sensitiveFieldPolicy", "EXCLUDE");
if("CLIENT_AES256_GCM".equals(mode))
{
if(!encryptionAvailable)
{
errors.add("Client-side encryption is not available in this version. Use NONE (and keep the sensitive field policy at EXCLUDE).");
}
else if(settings.string("encryptionKeyRef", null) == null)
{
errors.add("Client-side encryption needs an encryption key reference (the name of the key, never the key itself).");
}
}
if("INCLUDE_ENCRYPTED_ONLY".equals(policy) && !"CLIENT_AES256_GCM".equals(mode))
{
errors.add("The sensitive field policy INCLUDE_ENCRYPTED_ONLY requires client-side encryption (CLIENT_AES256_GCM).");
}
if("INCLUDE".equals(policy) && !BackupsPermissionChecker.has(BackupsPermissions.ADMIN))
{
errors.add("Including sensitive fields in the clear needs the backups.admin permission.");
}
}
/*******************************************************************************
** The destination: the bucket exists, and an object lock is something the bucket can do.
*******************************************************************************/
private static void validateStorage(PlanSettings settings, List<String> errors, List<String> warnings)
{
String alias = settings.string("storageBucketAlias", null);
if(alias == null)
{
errors.add("Choose a storage bucket for the backups.");
return;
}
BackupStorage storage = null;
try
{
storage = BackupsRuntime.storageFor(alias);
}
catch(RuntimeException e)
{
errors.add("The storage bucket " + alias + " cannot be used: " + e.getMessage());
}
String lockMode = settings.string("objectLockMode", "NONE");
if(!LOCK_MODES.contains(lockMode))
{
errors.add("Unknown object lock mode " + lockMode + ". Use NONE, GOVERNANCE or COMPLIANCE.");
}
else if(!"NONE".equals(lockMode))
{
Integer days = settings.integer("objectLockDays", null);
if(days == null || days < 1)
{
errors.add("Object lock needs a number of days to keep each object locked (objectLockDays).");
}
if(storage != null && !storage.capabilities().contains(BackupStorage.StorageCapability.OBJECT_LOCK))
{
errors.add("Object lock " + lockMode + " was asked for, but the storage bucket " + alias + " does not support object lock (its capabilities: "
+ storage.capabilities() + "). Choose NONE, or a bucket whose Test connection found object lock supported.");
}
}
String pruneAlias = settings.string("pruneBucketAlias", null);
if(pruneAlias == null)
{
warnings.add("No prune bucket is set: Prune will use the backup bucket, so one key can both write and delete backups. Give the plan a separate delete-capable alias.");
}
else
{
try
{
BackupsRuntime.storageFor(pruneAlias);
}
catch(RuntimeException e)
{
errors.add("The prune bucket " + pruneAlias + " cannot be used: " + e.getMessage());
}
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void validateSchedule(PlanSettings settings, List<String> errors)
{
for(String field : List.of("cronExpression", "verifyCronExpression"))
{
String cron = settings.string(field, null);
if(cron != null && !org.quartz.CronExpression.isValidExpression(cron))
{
errors.add("The " + field + " is not a valid cron expression: " + cron);
}
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void validateNumbers(PlanSettings settings, List<String> errors)
{
for(String field : List.of("retainDaily", "retainWeekly", "retainMonthly", "retainYearly", "retainMinimum"))
{
Integer value = settings.integer(field, 0);
if(value < 0)
{
errors.add(field + " cannot be negative.");
}
}
if(settings.integer("slaHours", 26) < 1)
{
errors.add("slaHours must be at least 1.");
}
if(settings.integer("pageSize", 5000) < 1)
{
errors.add("pageSize must be at least 1.");
}
}
/*******************************************************************************
** Saving or activating a plan: the saving user must have backups.admin and read
** access to every table the plan resolves to (spec 11.1).
*******************************************************************************/
private static void validateOwner(PlanSettings settings, List<ScopeResolver.ScopeRule> rules, List<String> errors)
{
if(!BackupsPermissionChecker.has(BackupsPermissions.ADMIN))
{
errors.add("Saving an active plan needs the backups.admin permission: the plan runs on its own with full access.");
return;
}
ScopeResolver.ResolvedScope scope;
try
{
scope = new ScopeResolver(QContext.getQInstance()).resolve(settings, rules);
}
catch(RuntimeException e)
{
errors.add("The plan scope cannot be resolved: " + e.getMessage());
return;
}
List<String> unreadable = new ArrayList<>();
for(ScopeResolver.ResolvedTable table : scope.tables())
{
if(!PermissionsHelper.hasTablePermission(new QueryInput(table.name()), table.name(), TablePermissionSubType.READ))
{
unreadable.add(table.name());
}
}
if(!unreadable.isEmpty())
{
errors.add("The plan covers tables that its owner is not allowed to read, so it could not back them up in full: "
+ String.join(", ", unreadable.stream().limit(10).toList()) + (unreadable.size() > 10 ? " and " + (unreadable.size() - 10) + " more" : "")
+ ". Remove them from the scope, or save the plan as a user who can read them.");
}
}
}
@@ -0,0 +1,260 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.plan;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.prune.BackupPruner;
import com.kingsrook.qbits.backups.storage.BackupStorage;
import com.kingsrook.qbits.backups.storage.StoredObjectInfo;
import com.kingsrook.qbits.backups.storage.WriteOptions;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Test destination (spec 9): proves the plan destination works before the first
** night depends on it. Writes a small probe object, reads it back, lists it, and
** deletes it through the prune alias (so the delete credential is exercised too).
**
** On a bucket that is object-locked the probe is head only: a probe object could
** not be deleted again, so nothing is written; the prefix is listed, and the
** newest manifest is looked at with a head. A capability report always comes with
** it.
*******************************************************************************/
public final class DestinationProbe
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private DestinationProbe()
{
}
/*******************************************************************************
** One thing that was checked.
*******************************************************************************/
public record Check(String name, boolean ok, String detail)
{
}
/*******************************************************************************
** The whole report.
*******************************************************************************/
public record Report(boolean headOnly, List<Check> checks, List<String> capabilities, List<String> warnings)
{
/*******************************************************************************
**
*******************************************************************************/
public boolean ok()
{
return (checks.stream().allMatch(Check::ok));
}
}
/*******************************************************************************
** Probe the destination of a plan.
*******************************************************************************/
public static Report probe(QRecord plan)
{
List<Check> checks = new ArrayList<>();
List<String> warnings = new ArrayList<>();
String alias = plan.getValueString("storageBucketAlias");
BackupStorage storage;
try
{
storage = BackupsRuntime.storageFor(alias);
}
catch(RuntimeException e)
{
checks.add(new Check("Open the backup bucket " + alias, false, e.getMessage()));
return (new Report(false, checks, List.of(), warnings));
}
checks.add(new Check("Open the backup bucket " + alias, true, "found"));
BackupStorage deleteSide = storage;
String pruneAlias = plan.getValueString("pruneBucketAlias");
if(pruneAlias == null || pruneAlias.isBlank())
{
warnings.add("No prune bucket is set, so one key both writes and deletes backups.");
}
else
{
try
{
deleteSide = BackupsRuntime.storageFor(pruneAlias);
checks.add(new Check("Open the prune bucket " + pruneAlias, true, "found"));
}
catch(RuntimeException e)
{
checks.add(new Check("Open the prune bucket " + pruneAlias, false, e.getMessage()));
}
}
List<String> capabilities = new ArrayList<>();
for(BackupStorage.StorageCapability capability : BackupStorage.StorageCapability.values())
{
capabilities.add(capability.name() + ": " + (storage.capabilities().contains(capability) ? "supported" : "not supported"));
}
String lockMode = plan.getValueString("objectLockMode");
boolean locked = lockMode != null && !"NONE".equals(lockMode);
if(locked && !storage.capabilities().contains(BackupStorage.StorageCapability.OBJECT_LOCK))
{
checks.add(new Check("Object lock " + lockMode, false, "the plan asks for object lock, but this bucket does not support it"));
}
String prefix = BackupPruner.planPrefix(plan);
if(locked)
{
headOnly(plan, storage, prefix, checks);
}
else
{
writeReadListDelete(storage, deleteSide, prefix + ".qqq-probe/", checks);
}
return (new Report(locked, checks, capabilities, warnings));
}
/*******************************************************************************
** The probe for a locked bucket: list the prefix, and head the newest manifest.
*******************************************************************************/
private static void headOnly(QRecord plan, BackupStorage storage, String prefix, List<Check> checks)
{
checks.add(attempt("Write and delete a probe object", () -> "skipped: object lock is on, a probe object could not be deleted again"));
checks.add(attempt("List the plan prefix", () ->
{
List<StoredObjectInfo> objects = storage.list(prefix);
return (objects.size() + " objects under " + prefix);
}));
Integer lastSuccess = plan.getValueInteger("lastSuccessRunId");
if(lastSuccess == null)
{
checks.add(new Check("Head the newest manifest", true, "there is no successful run yet"));
return;
}
checks.add(attempt("Head the newest manifest", () ->
{
QRecord run = new com.kingsrook.qbits.backups.run.BackupCatalog().get(com.kingsrook.qbits.backups.model.BackupsTables.RUN, lastSuccess).orElseThrow();
String key = run.getValueString("manifestKey");
if(key == null || storage.head(key).isEmpty())
{
throw (new IllegalStateException("the manifest of the newest successful run is not in the bucket"));
}
return ("found " + key);
}));
}
/*******************************************************************************
** Write, read back, list and delete a probe object.
*******************************************************************************/
private static void writeReadListDelete(BackupStorage storage, BackupStorage deleteSide, String probePrefix, List<Check> checks)
{
String key = probePrefix + UUID.randomUUID();
byte[] content = ("qqq backups probe " + UUID.randomUUID()).getBytes(StandardCharsets.UTF_8);
Check write = attempt("Write a probe object", () ->
{
try(OutputStream out = storage.openWrite(key, WriteOptions.create().withContentType("text/plain")))
{
out.write(content);
}
return ("wrote " + content.length + " bytes");
});
checks.add(write);
if(!write.ok())
{
return;
}
checks.add(attempt("Read it back", () ->
{
try(InputStream in = storage.openRead(key))
{
if(!java.util.Arrays.equals(content, in.readAllBytes()))
{
throw (new IllegalStateException("the bytes read back are not the bytes written"));
}
}
return ("the same bytes came back");
}));
checks.add(attempt("List it", () ->
{
if(storage.list(probePrefix).stream().noneMatch(o -> o.key().equals(key)))
{
throw (new IllegalStateException("the probe object is not in the listing"));
}
return ("found in the listing");
}));
checks.add(attempt("Delete it", () ->
{
BackupStorage.DeleteResult result = deleteSide.deletePermanent(key);
if(result != BackupStorage.DeleteResult.DELETED)
{
throw (new IllegalStateException("the delete returned " + result));
}
return ("deleted");
}));
}
/***************************************************************************
**
***************************************************************************/
@FunctionalInterface
private interface Attempt
{
String run() throws Exception;
}
/*******************************************************************************
**
*******************************************************************************/
private static Check attempt(String name, Attempt attempt)
{
try
{
return (new Check(name, true, attempt.run()));
}
catch(Exception e)
{
return (new Check(name, false, e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage()));
}
}
}
Loaded 100 of 158 files, more files were not shown because too many files have changed in this diff. Show more