refactor(backups): core TableSchemaFingerprint + KeysetPager on RC.6 (#19 part 2) #21

Merged
KofTwentyTwo merged 2 commits from feature/arch-rc6-keyset into develop 2026-10-14 12:19:52 +00:00
22 changed files with 899 additions and 285 deletions

No files matched your search

+1 -1
View File
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
## Dependencies
- Java 21.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.5` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Tests: JUnit 5, AssertJ.
## Release
+4 -1
View File
@@ -11,4 +11,7 @@ All notable changes to this project are documented here. The format follows Keep
- Build against the released QQQ BOM 4.1.0-RC.5; replace reflective alerts binding with core QAlerts and real sink integration tests.
- Register backup alert types for every instance while preserving custom backup sinks and no-op behavior without a configured core sink.
- Use RC.5-valid alert names `backups.run.missing-in-storage` and `backups.plan.owner-invalid` (and matching deduplication keys). Hosts using the old underscore names must update routing rules.
- Mark fingerprint, keyset reader, and restore bridges for removal in RC.6.
- Target QQQ 4.1.0-RC.6 and replace the fingerprint, keyset pager and reflective restore bridges with typed core APIs.
- New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
- Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
- Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
+10 -1
View File
@@ -10,7 +10,7 @@
## Requirements
- Java 21.
- QQQ 4.1 (built against `4.1.0-RC.5`).
- QQQ 4.1 (targets `4.1.0-RC.6`, not yet released).
## Install
@@ -36,6 +36,15 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
./mvnw verify
```
Before RC.6 is released, verify against the locally installed core build only:
```bash
./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
```
The POM stays on RC.6. See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for
fingerprint compatibility, paging guarantees and restore permissions.
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
## License
+100
View File
@@ -0,0 +1,100 @@
# RC.6 core migration (issue #19, F3)
The POM targets `4.1.0-RC.6`. Until that release is available, local verification
uses `./mvnw -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` against the
installed build from `qqq-wt-archrc6`. No core source changes are needed.
## Fingerprint compatibility
The algorithms are **not byte-compatible**, even for identical table metadata:
| Descriptor property | Pre-RC.6 backup algorithm | Core algorithm |
| --- | --- | --- |
| Table and key | `name`, `primaryKey` | `table`, `primaryKeyField` |
| Fields | Map keyed by name | Sorted list with explicit `name` |
| Field flags | `required`, `pvs`, `heavy`, `sensitive` | `isRequired`, `possibleValueSource`, `isHeavy`, `isSensitive` |
| Unique keys | Sorted `+`-joined strings | Sorted lists of field names |
| Associations | Sorted `name->table` strings | Objects with name, associatedTable and joinName |
| Null backend type | JSON null | Empty string |
| Escaping | Historical hand-written escaping | Jackson JSON escaping |
New table entries contain `fingerprintAlgorithm: "qqq-table-schema-v1"`, the core
`schema` descriptor, and `fingerprint: "sha256:" + core.getHash()`. The outer
manifest remains version 1; row parts, checksums, compression and storage paths
are unchanged. The instance/catalog hash still combines the ordered table
fingerprint strings, so its value changes too. New manifests require this RC.6
reader; older readers do not understand the new descriptor shape.
An absent algorithm identifies the old `qbit-backups-table-v1` algorithm. The
reader also accepts that explicit identifier. `LegacyTableFingerprint` freezes
the original descriptor and hash implementation solely for comparison against
existing backups; no new backup uses it. Restore and content verification select
the declared algorithm. Unknown identifiers never match a known fingerprint.
Restore normalizes either field shape for its existing compatibility analysis
when a table has changed. Fingerprint equality is not required for a compatible
schema migration, as before.
`src/test/resources/manifests/legacy-schema.json` freezes descriptors and hashes
produced by the original stand-in for the blog fixture. Tests place these into a
real backup manifest without the algorithm field, verify its content, mutate the
live data, then restore and compare the complete content hash including dates.
They also cover optional additions and incompatible type changes in old schemas.
## Paging and consistency
`TableReader.Keyset` delegates to core `KeysetPager`. Every query uses
`(scope filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize`, with no offset.
The resume key is exclusive. Page size is a row bound, **not a JDBC fetch-size
setting**. Core's normal bounded QueryAction path retains backend driver defaults;
this adapter does not opt into the large-result streaming query hint or change
JDBC fetch size. Heavy/hidden/password fields and untranslated stored values keep
the same query settings and existing backup permission boundary.
Every page receives the caller's transaction. The pager does not open, commit,
close, or replace that transaction. A test opens a real core memory snapshot,
changes a later page's data, and proves every page still sees the original state.
The existing BackupEngine continues to report BEST_EFFORT and reject
SNAPSHOT_REQUIRED: adopting the consistent-read opening API at engine level is
separate work. This migration does not claim snapshot guarantees for ordinary
backup runs.
Core can stream non-orderable keys without deterministic key order; backups
explicitly reject those keys instead. Core rejects scope filters carrying skip
or limit, and replaces user ordering with ascending primary-key ordering. Tests
cover OR filters, resume, full and partial pages, empty scans and invalid inputs.
## Restore writes
`StandardRestoreWriter` now constructs `RestoreWriteFlag` directly and calls
`QBackendModuleInterface.resyncKeySequences`, consuming typed `SequenceResyncResult`
values. Only RESYNCED results are reported as sequence changes; NOT_NEEDED and
NOT_APPLICABLE remain normal outcomes. There is no reflection or unavailable-core
fallback. The core flag preserves supplied dates and carries the requested
customizer, automation, audit and event settings.
Core always requires an actual session permission `backups.restore` for its
restore flag, including hosts with general core enforcement disabled. The qbit's
anonymous-mode convenience setting does not grant that permission. Review now
reports this requirement before executing a restore; test sessions explicitly
hold it. Custom writers that do not use the core flag keep their existing qbit
permission checks.
## HTTP operations and wire changes
No HTTP operations are added, removed or modified: method, path and operationId
are unchanged (no operation list entries for this migration). The RC.6 UI API
extension documentation was reviewed; this task introduces no Javalin route or
middleware extension. Wire changes are confined to the manifest fields and
fingerprint values described above, plus an earlier restore permission error in
Review for anonymous sessions lacking `backups.restore`.
## Local verification (2026-10-13)
- `./mvnw -B clean verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`:
BUILD SUCCESS; 158 tests, zero failures/errors/skips; both JaCoCo gates passed.
Existing backup, restore and MinIO integration tests ran.
- Separate Checkstyle 14.3.0 audit of all 16 changed Java files (main and test):
zero violations, using the shared Kingsrook-derived rules with this repository's
Apache file header. This repository has no Checkstyle execution configured in
its POM; audit configuration and results are local under `.cache/style/`.
- `git diff --check`: passed. No commit, push or core source modification.
+5 -3
View File
@@ -18,6 +18,8 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
@@ -27,7 +29,7 @@
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
<version>2.1.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -192,14 +194,14 @@
<profiles>
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
<!-- imports the released 4.1 BOM, this profile is active by default and -->
<!-- imports qqq-bom-pom 4.1.0-RC.5 ahead of the parent's 4.0.0 BOM. -->
<!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.5</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -137,6 +137,7 @@ public record BackupManifest(
String backend,
String primaryKey,
String fingerprint,
String fingerprintAlgorithm,
Map<String, Object> schema,
String filter,
List<ExcludedField> excludedFields,
@@ -0,0 +1,170 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
/*******************************************************************************
** Frozen pre-RC.6 fingerprint algorithm, used only to read existing manifests.
** Keep its descriptor and escaping byte-compatible with the original writer.
*******************************************************************************/
final class LegacyTableFingerprint
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private LegacyTableFingerprint()
{
}
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -18,29 +18,23 @@ package com.kingsrook.qbits.backups.manifest;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint;
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1113, removeBy=4.1.0-RC.6)
** A stable schema descriptor and SHA-256 fingerprint of a table (spec 5.5).
**
** This is an internal stand-in for the QQQ core TableSchemaFingerprint (core PR
** feature/schema-fingerprint). When that ships in the next 4.1.x RC, this class
** delegates to it and the descriptor below is kept only for reading old manifests.
** Labels, adornments and help text are excluded, so cosmetic changes do not count.
** Core schema fingerprints with read compatibility for pre-RC.6 manifests.
*******************************************************************************/
public final class TableFingerprints
{
public static final String CORE_ALGORITHM = "qqq-table-schema-v1";
public static final String LEGACY_ALGORITHM = "qbit-backups-table-v1";
/*******************************************************************************
** Utility class.
@@ -52,134 +46,83 @@ public final class TableFingerprints
/*******************************************************************************
** The descriptor of a table (sorted fields; plain maps and lists, so it is also
** what goes in the manifest schema).
** The core descriptor stored in new manifests.
*******************************************************************************/
public static Map<String, Object> descriptor(QTableMetaData table, String backendType)
{
Map<String, Object> d = new LinkedHashMap<>();
d.put("name", table.getName());
d.put("primaryKey", table.getPrimaryKeyField());
d.put("backendType", backendType);
Map<String, Object> fields = new TreeMap<>();
for(QFieldMetaData field : new TreeMap<>(table.getFields()).values())
{
Map<String, Object> f = new LinkedHashMap<>();
f.put("type", field.getType().name());
f.put("required", field.getIsRequired());
f.put("maxLength", field.getMaxLength());
f.put("pvs", field.getPossibleValueSourceName());
f.put("heavy", field.getIsHeavy());
f.put("sensitive", field.getIsSensitive());
fields.put(field.getName(), f);
}
d.put("fields", fields);
List<String> uniqueKeys = new ArrayList<>();
if(table.getUniqueKeys() != null)
{
for(UniqueKey key : table.getUniqueKeys())
{
uniqueKeys.add(String.join("+", key.getFieldNames().stream().sorted().toList()));
}
}
uniqueKeys.sort(String::compareTo);
d.put("uniqueKeys", uniqueKeys);
List<String> associations = new ArrayList<>();
if(table.getAssociations() != null)
{
for(Association association : table.getAssociations())
{
associations.add(association.getName() + "->" + association.getAssociatedTableName());
}
}
associations.sort(String::compareTo);
d.put("associations", associations);
return (d);
return (TableSchemaFingerprint.of(table, backendType).getDescriptor());
}
/*******************************************************************************
** The fingerprint of a table: sha256:hex of the canonical descriptor.
** Retain the manifest's sha256 prefix around the core hash.
*******************************************************************************/
public static String of(QTableMetaData table, String backendType)
{
return (fingerprintOf(descriptor(table, backendType)));
return ("sha256:" + TableSchemaFingerprint.of(table, backendType).getHash());
}
/*******************************************************************************
** The fingerprint of a descriptor (maps sorted by key at every level, no whitespace).
** An absent algorithm identifies manifests written before RC.6. Unknown
** algorithms must not silently fall back to a different fingerprint.
*******************************************************************************/
public static String fingerprintOf(Map<String, Object> descriptor)
public static boolean matches(BackupManifest.TableEntry entry, QTableMetaData table, String backendType)
{
StringBuilder sb = new StringBuilder();
canonical(descriptor, sb);
return ("sha256:" + HashingOutputStream.sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)));
if(entry.fingerprint() == null)
{
return (false);
}
String algorithm = entry.fingerprintAlgorithm();
if(algorithm == null || LEGACY_ALGORITHM.equals(algorithm))
{
return (entry.fingerprint().equals(LegacyTableFingerprint.of(table, backendType)));
}
return (CORE_ALGORITHM.equals(algorithm) && entry.fingerprint().equals(of(table, backendType)));
}
/*******************************************************************************
** The fingerprint of a whole instance's tables: a hash of the table fingerprints.
** Normalize both descriptor versions to field-name maps for restore's
** compatibility analysis. Core fields use a list and explicit flag names.
*******************************************************************************/
@SuppressWarnings("unchecked")
public static Map<String, Object> fields(Map<String, Object> descriptor)
{
Object fields = descriptor == null ? null : descriptor.get("fields");
if(fields instanceof Map<?, ?> map)
{
return ((Map<String, Object>) map);
}
Map<String, Object> result = new LinkedHashMap<>();
if(fields instanceof List<?> list)
{
for(Object item : list)
{
Map<String, Object> field = (Map<String, Object>) item;
Map<String, Object> normalized = new LinkedHashMap<>();
normalized.put("type", field.get("type"));
normalized.put("required", field.get("isRequired"));
normalized.put("maxLength", field.get("maxLength"));
normalized.put("pvs", field.get("possibleValueSource"));
normalized.put("heavy", field.get("isHeavy"));
normalized.put("sensitive", field.get("isSensitive"));
result.put((String) field.get("name"), normalized);
}
}
return (result);
}
/*******************************************************************************
** The instance fingerprint remains a hash of the ordered table fingerprints.
*******************************************************************************/
public static String combine(List<String> tableFingerprints)
{
return ("sha256:" + HashingOutputStream.sha256Hex(String.join("\n", tableFingerprints).getBytes(StandardCharsets.UTF_8)));
}
/*******************************************************************************
**
*******************************************************************************/
@SuppressWarnings("unchecked")
private static void canonical(Object o, StringBuilder sb)
{
if(o == null)
{
sb.append("null");
}
else if(o instanceof Map<?, ?> map)
{
sb.append('{');
boolean first = true;
for(Map.Entry<String, Object> e : new TreeMap<>((Map<String, Object>) map).entrySet())
{
if(!first)
{
sb.append(',');
}
first = false;
sb.append('"').append(e.getKey()).append("\":");
canonical(e.getValue(), sb);
}
sb.append('}');
}
else if(o instanceof List<?> list)
{
sb.append('[');
for(int i = 0; i < list.size(); i++)
{
if(i > 0)
{
sb.append(',');
}
canonical(list.get(i), sb);
}
sb.append(']');
}
else if(o instanceof Number || o instanceof Boolean)
{
sb.append(o);
}
else
{
sb.append('"').append(String.valueOf(o).replace("\\", "\\\\").replace("\"", "\\\"")).append('"');
}
}
}
@@ -105,7 +105,8 @@ public class RestoreEngine
Map<String, String> targets = new LinkedHashMap<>();
QInstance instance = QContext.getQInstance();
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE))
if(!BackupsPermissionChecker.has(BackupsPermissions.RESTORE)
|| (request.getWriter().usesCoreRestoreFlag() && (QContext.getQSession() == null || !QContext.getQSession().hasPermission(BackupsPermissions.RESTORE))))
{
errors.add("Restoring needs the " + BackupsPermissions.RESTORE + " permission.");
}
@@ -341,14 +341,14 @@ public class RestorePass
String backendType = instance.getBackend(c.target.getBackendName()) == null ? null : instance.getBackend(c.target.getBackendName()).getBackendType();
RestoreOutcome.TableOutcome o = c.outcome;
if(c.entry.fingerprint() != null && c.entry.fingerprint().equals(TableFingerprints.of(c.target, backendType)))
if(TableFingerprints.matches(c.entry, c.target, backendType))
{
o.setSchemaStatus("MATCH");
return;
}
Map<String, Object> backupFields = c.entry.schema() == null ? Map.of() : (Map<String, Object>) c.entry.schema().getOrDefault("fields", Map.of());
Map<String, Object> targetFields = (Map<String, Object>) TableFingerprints.descriptor(c.target, backendType).get("fields");
Map<String, Object> backupFields = TableFingerprints.fields(c.entry.schema());
Map<String, Object> targetFields = TableFingerprints.fields(TableFingerprints.descriptor(c.target, backendType));
boolean incompatible = false;
boolean different = false;
@@ -28,9 +28,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** How a restore writes (spec 5.3 and 5.4). The default implementation uses the
** plain Insert, Update and Delete actions, and selects the core RestoreWriteFlag
** (QRun-IO/qqq#1111) and the core key-sequence resync (#1112) reflectively when the
** running QQQ has them. A backend specific writer can be supplied on the request.
** plain Insert, Update and Delete actions with core RestoreWriteFlag and typed
** key-sequence resync. A backend specific writer can be supplied on the request.
**
** Whatever the implementation, record security, field security, unique keys and
** required-field validation still apply to every write.
@@ -18,7 +18,6 @@ package com.kingsrook.qbits.backups.restore;
import java.io.Serializable;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashMap;
@@ -30,8 +29,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.ActionFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
@@ -41,50 +40,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleInterface;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
import com.kingsrook.qqq.backend.core.modules.backend.SequenceResyncResult;
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1111, removeBy=4.1.0-RC.6)
** STAND-IN(core=QRun-IO/qqq#1112, removeBy=4.1.0-RC.6)
** The default RestoreWriter: the standard Insert, Update and Delete actions.
**
** Stand-ins (replace with the core API when QQQ 4.1.0-RC.5 ships):
** <ul>
** <li>RestoreWriteFlag (QRun-IO/qqq#1111) is created reflectively when the class is
** present. Without it, the DML audit is omitted (a summary audit is written per
** table instead), but created and modified dates may be set by the normal defaults.</li>
** <li>resyncKeySequences (QRun-IO/qqq#1112) is called reflectively on the backend
** module when it exists; otherwise it is a no-op that the Review reports.</li>
** </ul>
** The default RestoreWriter using core restore flags and key sequence resync.
*******************************************************************************/
public class StandardRestoreWriter implements RestoreWriter
{
private static final QLogger LOG = QLogger.getLogger(StandardRestoreWriter.class);
private static final String FLAG_CLASS = "com.kingsrook.qqq.backend.core.model.actions.tables.RestoreWriteFlag";
private static final Class<?> FLAG = lookup(FLAG_CLASS);
/*******************************************************************************
**
*******************************************************************************/
private static Class<?> lookup(String name)
{
try
{
return (Class.forName(name));
}
catch(ClassNotFoundException | LinkageError e)
{
return (null);
}
}
/*******************************************************************************
** {@inheritDoc}
*******************************************************************************/
@@ -150,27 +113,15 @@ public class StandardRestoreWriter implements RestoreWriter
/*******************************************************************************
** Create the core flag (when the running QQQ has it) and hand it to the input.
** Apply the core restore behavior requested by the caller.
*******************************************************************************/
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink) throws QException
private void addFlag(RestoreRequest options, java.util.function.Consumer<ActionFlag> sink)
{
if(FLAG == null)
{
return;
}
try
{
Object flag = FLAG.getDeclaredConstructor().newInstance();
FLAG.getMethod("withSuppressAutomations", boolean.class).invoke(flag, options.getSuppressAutomations());
FLAG.getMethod("withRunCustomizers", boolean.class).invoke(flag, options.getRunCustomizers());
FLAG.getMethod("withPerRecordAudit", boolean.class).invoke(flag, options.getPerRecordAudit());
FLAG.getMethod("withPublishEvents", boolean.class).invoke(flag, options.getPublishEvents());
sink.accept((ActionFlag) flag);
}
catch(ReflectiveOperationException | ClassCastException e)
{
throw (new QException("Could not use the core restore flag: " + e.getMessage(), e));
}
sink.accept(new RestoreWriteFlag()
.withSuppressAutomations(options.getSuppressAutomations())
.withRunCustomizers(options.getRunCustomizers())
.withPerRecordAudit(options.getPerRecordAudit())
.withPublishEvents(options.getPublishEvents()));
}
@@ -181,7 +132,7 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public boolean usesCoreRestoreFlag()
{
return (FLAG != null);
return (true);
}
@@ -192,24 +143,7 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public boolean canResyncSequences()
{
return (resyncMethod() != null);
}
/*******************************************************************************
**
*******************************************************************************/
private static Method resyncMethod()
{
try
{
return (QBackendModuleInterface.class.getMethod("resyncKeySequences", QBackendMetaData.class, Collection.class));
}
catch(NoSuchMethodException | LinkageError e)
{
return (null);
}
return (true);
}
@@ -220,12 +154,6 @@ public class StandardRestoreWriter implements RestoreWriter
@Override
public List<String> resyncSequences(Collection<QTableMetaData> tables) throws QException
{
Method method = resyncMethod();
if(method == null)
{
return (List.of());
}
Map<String, List<QTableMetaData>> byBackend = new LinkedHashMap<>();
for(QTableMetaData table : tables)
{
@@ -236,23 +164,14 @@ public class StandardRestoreWriter implements RestoreWriter
for(Map.Entry<String, List<QTableMetaData>> entry : byBackend.entrySet())
{
QBackendMetaData backend = QContext.getQInstance().getBackend(entry.getKey());
try
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
for(SequenceResyncResult result : module.resyncKeySequences(backend, entry.getValue()))
{
QBackendModuleInterface module = new QBackendModuleDispatcher().getQBackendModule(backend);
for(Object result : (List<?>) method.invoke(module, backend, entry.getValue()))
if(result.getStatus() == SequenceResyncResult.Status.RESYNCED)
{
Object status = result.getClass().getMethod("getStatus").invoke(result);
if("RESYNCED".equals(String.valueOf(status)))
{
resynced.add(String.valueOf(result.getClass().getMethod("getTableName").invoke(result)));
}
resynced.add(result.getTableName());
}
}
catch(ReflectiveOperationException e)
{
LOG.warn("Could not resync key sequences", e, logPair("backend", entry.getKey()));
throw (new QException("Could not resync key sequences on backend " + entry.getKey() + ": " + e.getMessage(), e));
}
}
return (resynced);
}
@@ -35,8 +35,8 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.events.BackupEvents;
import com.kingsrook.qbits.backups.codec.TableArtifactWriter;
import com.kingsrook.qbits.backups.events.BackupEvents;
import com.kingsrook.qbits.backups.files.FileCopier;
import com.kingsrook.qbits.backups.files.FileReference;
import com.kingsrook.qbits.backups.files.FileReferenceProvider;
@@ -398,7 +398,7 @@ public class BackupEngine
keyRange.put("min", allResults.isEmpty() ? null : allResults.get(0).firstKey());
keyRange.put("max", allResults.isEmpty() ? null : allResults.get(allResults.size() - 1).lastKey());
manifestTables.add(new BackupManifest.TableEntry(table.getName(), resolved.role().name(), table.getBackendName(), table.getPrimaryKeyField(),
fingerprint, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
fingerprint, TableFingerprints.CORE_ALGORITHM, TableFingerprints.descriptor(table, backendType), resolved.filterJson(), excluded, counter[0], rowsHash, keyRange, parts, consistency));
done++;
catalogProgress(result, done, rows, bytesRaw, bytesStored);
if(request.getCancelRequested().getAsBoolean())
@@ -18,31 +18,20 @@ package com.kingsrook.qbits.backups.run;
import java.io.Serializable;
import java.util.List;
import java.util.function.Consumer;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.KeysetPager;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1114, removeBy=4.1.0-RC.6)
** Reads the rows of one table in primary key order, in keyset pages (WHERE pk >
** lastKey ORDER BY pk LIMIT n), so the cost stays flat with depth and a read can
** resume from a key. A read raw: hidden fields, passwords and heavy fields are
** returned as stored, and possible values are not translated.
**
** This is an internal stand-in for the QQQ core KeysetPager and QueryInput
** withRawRead (core PR feature/keyset-pager-raw-read); the run engine only
** depends on this interface, so it is swapped when that ships in the next RC.
** Reads rows in primary key order through core KeysetPager. Each bounded page
** uses the supplied transaction and returns hidden, password and heavy fields
** as stored, without possible-value translation or display generation.
*******************************************************************************/
public interface TableReader
{
@@ -82,48 +71,25 @@ public interface TableReader
@Override
public void read(QTableMetaData table, String filterJson, Serializable startAfterKey, int pageSize, QBackendTransaction transaction, Consumer<QRecord> sink) throws QException
{
String pk = table.getPrimaryKeyField();
Serializable lastKey = startAfterKey;
QQueryFilter user = parseFilter(filterJson);
while(true)
KeysetPager pager = KeysetPager.forTable(table.getName())
.withFilter(parseFilter(filterJson))
.startingAfter(startAfterKey)
.withPageSize(pageSize)
.withTransaction(transaction)
.withQueryInputCustomizer(input ->
{
input.setShouldOmitHiddenFields(false);
input.setShouldMaskPasswords(false);
input.setShouldFetchHeavyFields(true);
input.setShouldTranslatePossibleValues(false);
input.setShouldGenerateDisplayValues(false);
input.withPermissionBypass("backups run");
});
if(!pager.isPrimaryKeyOrderable())
{
QQueryFilter filter = new QQueryFilter();
if(lastKey != null)
{
filter.addCriteria(new QFilterCriteria(pk, QCriteriaOperator.GREATER_THAN, List.of(lastKey)));
}
if(user != null)
{
filter.addSubFilter(user);
}
filter.withOrderBy(new QFilterOrderBy(pk, true));
filter.setLimit(pageSize);
QueryInput queryInput = new QueryInput(table.getName());
queryInput.setFilter(filter);
queryInput.setShouldOmitHiddenFields(false);
queryInput.setShouldMaskPasswords(false);
queryInput.setShouldFetchHeavyFields(true);
queryInput.setShouldTranslatePossibleValues(false);
queryInput.setShouldGenerateDisplayValues(false);
if(transaction != null)
{
queryInput.setTransaction(transaction);
}
queryInput.withPermissionBypass("backups run");
List<QRecord> page = new QueryAction().execute(queryInput).getRecords();
for(QRecord record : page)
{
sink.accept(record);
lastKey = record.getValue(pk);
}
if(page.size() < pageSize)
{
return;
}
throw (new QException("Backup requires an orderable primary key on table " + table.getName()));
}
pager.forEachPage(page -> page.forEach(sink));
}
}
}
@@ -33,8 +33,8 @@ import java.util.Optional;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.backups.BackupsQBitConfig;
import com.kingsrook.qbits.backups.BackupsRuntime;
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
import com.kingsrook.qbits.backups.alerts.BackupAlerts;
import com.kingsrook.qbits.backups.codec.ArtifactTransform;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.codec.JsonlRowCodec;
@@ -274,7 +274,7 @@ public class BackupVerifier
List<String> excluded = entry.excludedFields() == null ? List.of() : entry.excludedFields().stream().map(BackupManifest.ExcludedField::name).toList();
codec = new JsonlRowCodec(table, excluded);
String backendType = qInstance.getBackend(table.getBackendName()) == null ? null : qInstance.getBackend(table.getBackendName()).getBackendType();
roundTrip = TableFingerprints.of(table, backendType).equals(entry.fingerprint());
roundTrip = TableFingerprints.matches(entry, table, backendType);
if(!roundTrip)
{
findings.warn("Table " + entry.name() + " has changed since this backup; rows were parsed but not checked for an exact round trip");
@@ -106,7 +106,7 @@ public final class BackupsTestUtils
new BackupsQBitProducer().withQBitConfig(config).withNamespace("backups").produce(qInstance);
new QInstanceValidator().validate(qInstance);
QContext.init(qInstance, new QSession());
QContext.init(qInstance, new QSession().withPermissions("backups.restore"));
return (qInstance);
}
@@ -0,0 +1,65 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.manifest;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qqq.backend.core.context.QContext;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Fingerprint algorithm dispatch is explicit; a marker cannot be ignored just
** because its hash happens to match a different supported algorithm.
*******************************************************************************/
class TableFingerprintsTest
{
/*******************************************************************************
** A frozen old hash matches only the legacy algorithm, and new hashes only
** match the declared core algorithm. Cosmetic changes preserve both.
*******************************************************************************/
@Test
void testAlgorithmDispatch() throws Exception
{
BackupsTestUtils.init();
var table = QContext.getQInstance().getTable("blogPost");
ObjectMapper mapper = new ObjectMapper();
ObjectNode entry;
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
{
entry = (ObjectNode) mapper.readTree(in).get("blogPost");
}
String oldHash = entry.get("fingerprint").asText();
String coreHash = TableFingerprints.of(table, "memory");
assertThat(coreHash).isNotEqualTo(oldHash);
table.setLabel("Cosmetic label");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", TableFingerprints.LEGACY_ALGORITHM);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", TableFingerprints.CORE_ALGORITHM);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
entry.put("fingerprint", coreHash);
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isTrue();
entry.put("fingerprintAlgorithm", "unknown-v2");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
entry.remove("fingerprint");
assertThat(TableFingerprints.matches(mapper.treeToValue(entry, BackupManifest.TableEntry.class), table, "memory")).isFalse();
}
}
@@ -0,0 +1,123 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.restore;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qbits.backups.codec.HashingOutputStream;
import com.kingsrook.qbits.backups.manifest.BackupManifest;
import com.kingsrook.qbits.backups.model.BackupsTables;
import com.kingsrook.qbits.backups.run.BackupCatalog;
import com.kingsrook.qbits.backups.run.BackupRunResult;
import com.kingsrook.qbits.backups.verify.BackupVerifier;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Exercise pre-RC.6 manifest fingerprints through verification and restore.
** The schema fixture was frozen from the original stand-in, not recomputed
** with the implementation being tested.
*******************************************************************************/
class LegacyManifestRestoreTest
{
/*******************************************************************************
** Load the blog and grant the permission required by the core restore flag.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupsTestUtils.loadBlog(30);
}
/*******************************************************************************
** Build an old-format manifest around real backup parts and re-seal its hash.
*******************************************************************************/
private BackupRunResult legacyBackup() throws Exception
{
BackupRunResult run = RestoreTestSupport.backup(Map.of());
ObjectMapper mapper = new ObjectMapper();
ObjectNode manifest = (ObjectNode) mapper.readTree(BackupsTestUtils.BACKUPS.getBytes(run.getManifestKey()));
JsonNode legacy;
try(var in = getClass().getResourceAsStream("/manifests/legacy-schema.json"))
{
legacy = mapper.readTree(in);
}
for(JsonNode table : manifest.get("tables"))
{
ObjectNode entry = (ObjectNode) table;
JsonNode old = legacy.get(entry.get("name").asText());
entry.remove("fingerprintAlgorithm");
entry.set("schema", old.get("schema"));
entry.set("fingerprint", old.get("fingerprint"));
}
byte[] bytes = mapper.writeValueAsBytes(manifest);
BackupsTestUtils.BACKUPS.putBytes(run.getManifestKey(), bytes);
new BackupCatalog().update(BackupsTables.RUN, run.getRunId(), Map.of("manifestSha256", HashingOutputStream.sha256Hex(bytes)));
assertThat(BackupManifest.read(bytes).table("blogPost").fingerprintAlgorithm()).isNull();
return (run);
}
/*******************************************************************************
** Old hashes still receive exact codec verification and restore all data,
** including original timestamps, after deletes, edits and additions.
*******************************************************************************/
@Test
void testLegacyManifestVerifiesAndRestores() throws Exception
{
String before = RestoreTestSupport.contentHash("blogPost");
BackupRunResult run = legacyBackup();
var verified = new BackupVerifier().verify(run.getRunId(), BackupVerifier.CONTENT, "MANUAL", "tester");
assertThat(verified.status()).isEqualTo("PASSED");
assertThat(verified.messages()).noneMatch(message -> message.contains("has changed since"));
RestoreTestSupport.mutate();
RestoreOutcome outcome = new RestoreEngine().execute(RestoreTestSupport.request(run, RestoreRequest.MODE_REPLACE, "blogPost").withConfirmationText("blogPost"));
assertThat(outcome.getErrors()).isEmpty();
assertThat(outcome.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MATCH");
assertThat(RestoreTestSupport.contentHash("blogPost")).isEqualTo(before);
}
/*******************************************************************************
** Legacy field maps and core field lists also compare correctly when hashes
** differ: optional additions are compatible and type changes are rejected.
*******************************************************************************/
@Test
void testLegacySchemaChangesAreAnalyzed() throws Exception
{
BackupRunResult run = legacyBackup();
QContext.getQInstance().getTable("author").withField(new QFieldMetaData("nickname", QFieldType.STRING));
QContext.getQInstance().getTable("blogPost").getField("title").setType(QFieldType.INTEGER);
RestoreOutcome review = new RestoreEngine().review(RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "author", "blogPost"));
assertThat(review.getTables().get("author").getSchemaStatus()).isEqualTo("COMPATIBLE");
assertThat(review.getTables().get("blogPost").getSchemaStatus()).isEqualTo("MISMATCH");
}
}
@@ -187,4 +187,22 @@ class RestoreSafetyTest
QContext.setQSession(new QSession().withPermissions("backups.view", "backups.restore"));
assertThat(new RestoreEngine().plan(request).errors()).isEmpty();
}
/*******************************************************************************
** The core restore permission is required even in anonymous demo mode, and
** Review reports it before a restore can create a run or write any data.
*******************************************************************************/
@Test
void testCoreRestorePermissionIsRequiredWhenAnonymous() throws Exception
{
BackupRunResult run = RestoreTestSupport.backup(Map.of());
QContext.setQSession(new QSession());
RestoreRequest request = RestoreTestSupport.request(run, RestoreRequest.MODE_UPSERT, "blogPost");
assertThat(new RestoreEngine().plan(request).errors()).anyMatch(e -> e.contains("backups.restore"));
assertThatThrownBy(() -> new RestoreEngine().execute(request)).hasMessageContaining("backups.restore");
assertThatThrownBy(() -> new StandardRestoreWriter().insert(QContext.getQInstance().getTable("blogPost"),
List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", 99)), request, null)).hasMessageContaining("backups.restore");
}
}
@@ -109,7 +109,9 @@ class BackupEngineTest
BackupManifest.TableEntry posts = manifest.table("blogPost");
assertThat(posts.rowCount()).isEqualTo(7);
assertThat(posts.excludedFields()).extracting(BackupManifest.ExcludedField::name).containsExactly("secretNote");
assertThat(posts.fingerprint()).startsWith("sha256:");
assertThat(posts.fingerprintAlgorithm()).isEqualTo(com.kingsrook.qbits.backups.manifest.TableFingerprints.CORE_ALGORITHM);
assertThat(posts.fingerprint()).isEqualTo("sha256:" + com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getHash());
assertThat(posts.schema()).isEqualTo(com.kingsrook.qqq.backend.core.model.metadata.tables.TableSchemaFingerprint.of(QContext.getQInstance().getTable("blogPost"), "memory").getDescriptor());
BackupManifest.Part part = posts.parts().get(0);
byte[] stored = BackupsTestUtils.BACKUPS.getBytes(result.getRunPrefix() + part.key());
@@ -0,0 +1,163 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.backups.run;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.backups.BackupsTestUtils;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.consistentread.ConsistentReadInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Integration coverage for the core pager adapter's ordering, bounded fetches,
** raw fields and transaction lifetime.
*******************************************************************************/
class TableReaderTest
{
private QTableMetaData table;
/*******************************************************************************
** Seed more records than fit in a page.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
BackupsTestUtils.init();
BackupsTestUtils.loadBlog(7);
table = QContext.getQInstance().getTable("blogPost");
table.getField("body").setIsHeavy(true);
table.getField("secretNote").setIsHidden(true);
table.withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(CaptureQueries.class));
CaptureQueries.inputs.clear();
}
/*******************************************************************************
** An OR filter and a resume cursor are ANDed, with ascending keys even when
** the caller requests descending order. Every query has the requested bound.
*******************************************************************************/
@Test
void testFilterResumeOrderingAndRawFields() throws Exception
{
String filter = "{\"booleanOperator\":\"OR\",\"criteria\":[{\"fieldName\":\"id\",\"operator\":\"LESS_THAN_OR_EQUALS\",\"values\":[3]},{\"fieldName\":\"id\",\"operator\":\"GREATER_THAN\",\"values\":[5]}],\"orderBys\":[{\"fieldName\":\"id\",\"isAscending\":false}]}";
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, filter, 2, 2, null, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(3, 6, 7);
assertThat(rows.get(0).getValueString("body")).isEqualTo("Body of 3, with a comma");
assertThat(rows.get(0).getValueString("secretNote")).isEqualTo("do-not-leak-3");
assertThat(CaptureQueries.inputs).hasSize(2).allSatisfy(input ->
{
assertThat(input.getFilter().getLimit()).isEqualTo(2);
assertThat(input.getFilter().getSkip()).isNull();
assertThat(input.getShouldOmitHiddenFields()).isFalse();
assertThat(input.getShouldMaskPasswords()).isFalse();
assertThat(input.getShouldFetchHeavyFields()).isTrue();
assertThat(input.getShouldTranslatePossibleValues()).isFalse();
assertThat(input.getShouldGenerateDisplayValues()).isFalse();
});
}
/*******************************************************************************
** Changes made after a snapshot was opened are invisible on every page. The
** reader leaves the transaction open, with its commit callbacks unexecuted.
*******************************************************************************/
@Test
void testConsistentSnapshotIsRetainedAcrossPages() throws Exception
{
List<Boolean> commits = new ArrayList<>();
try(QBackendTransaction transaction = QBackendTransaction.openConsistentReadFor(new ConsistentReadInput().withBackendName(table.getBackendName())))
{
transaction.addAfterCommitCallback(() -> commits.add(true));
new UpdateAction().execute(new UpdateInput("blogPost").withRecords(List.of(new QRecord().withValue("id", 7).withValue("title", "Changed after snapshot"))));
CaptureQueries.inputs.clear();
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, null, null, 2, transaction, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(1, 2, 3, 4, 5, 6, 7);
assertThat(rows.get(6).getValueString("title")).isEqualTo("Post 7");
assertThat(CaptureQueries.inputs).hasSize(4).allSatisfy(input -> assertThat(input.getTransaction()).isSameAs(transaction));
assertThat(commits).isEmpty();
transaction.commit();
assertThat(commits).containsExactly(true);
}
}
/*******************************************************************************
** Exact page multiples, empty scans, invalid bounds and unsupported key types
** must terminate or fail explicitly rather than drop rows or stream unordered.
*******************************************************************************/
@Test
void testBoundariesAndInvalidPaging() throws Exception
{
List<QRecord> rows = new ArrayList<>();
new TableReader.Keyset().read(table, null, 3, 2, null, rows::add);
assertThat(rows).extracting(r -> r.getValueInteger("id")).containsExactly(4, 5, 6, 7);
rows.clear();
new TableReader.Keyset().read(table, null, 7, 2, null, rows::add);
assertThat(rows).isEmpty();
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 0, null, rows::add)).hasMessageContaining("Page size");
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{\"limit\":1}", null, 2, null, rows::add)).hasMessageContaining("skip or limit");
assertThatThrownBy(() -> new TableReader.Keyset().read(table, "{", null, 2, null, rows::add)).hasMessageContaining("scope filter");
table.getField("id").setType(QFieldType.BOOLEAN);
assertThatThrownBy(() -> new TableReader.Keyset().read(table, null, null, 2, null, rows::add)).hasMessageContaining("orderable primary key");
}
/*******************************************************************************
** Observe actual QueryAction inputs without replacing the backend or pager.
*******************************************************************************/
public static class CaptureQueries implements TableCustomizerInterface
{
static final List<QueryInput> inputs = new ArrayList<>();
/*******************************************************************************
** Record each page and preserve the result unchanged.
*******************************************************************************/
@Override
public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records)
{
inputs.add((QueryInput) input);
return (records);
}
}
}
@@ -0,0 +1,130 @@
{
"author" : {
"schema" : {
"name" : "author",
"primaryKey" : "id",
"backendType" : "memory",
"fields" : {
"bio" : {
"type" : "TEXT",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"id" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"name" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
}
},
"uniqueKeys" : [ ],
"associations" : [ ]
},
"fingerprint" : "sha256:c0fce31286898a15a98f9f46515b15c6eee4ffeb1f1eb39d6100833dfe3f760e"
},
"blogPost" : {
"schema" : {
"name" : "blogPost",
"primaryKey" : "id",
"backendType" : "memory",
"fields" : {
"authorId" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : "author",
"heavy" : false,
"sensitive" : false
},
"body" : {
"type" : "TEXT",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"createDate" : {
"type" : "DATE_TIME",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"id" : {
"type" : "INTEGER",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"mediaKey" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"price" : {
"type" : "DECIMAL",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"published" : {
"type" : "BOOLEAN",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"publishedOn" : {
"type" : "DATE",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
},
"secretNote" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : true
},
"title" : {
"type" : "STRING",
"required" : false,
"maxLength" : null,
"pvs" : null,
"heavy" : false,
"sensitive" : false
}
},
"uniqueKeys" : [ ],
"associations" : [ ]
},
"fingerprint" : "sha256:3a4de7479908cd4b4ce8eaa7a7af43e7cbaabbdbc2b9741bebeb24a6bd41b9ea"
}
}