Compare commits

..
Author SHA1 Message Date
james.maesandCodex eb429f7cbb ci: centralize QBit source scanning
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 10:19:34 -05:00
14 changed files with 35 additions and 464 deletions

No files matched your search

+2 -2
View File
@@ -1,12 +1,12 @@
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.8
qqq-orb: kingsrook/qqq-orb@0.7.12
workflows:
setup:
jobs:
- qqq-orb/generate_pipeline:
orb_version: '0.8'
orb_version: 0.7.12
filters:
branches:
ignore: automation/qbit-rc-pending
-7
View File
@@ -1,7 +0,0 @@
name: build
on:
push:
jobs:
maven:
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@v1
secrets: inherit
+1 -1
View File
@@ -1,5 +1,5 @@
profile: qbit
orb_version: '0.8'
orb_version: 0.7.12
default_branch: develop
java_version: '21'
browsers: true
-1
View File
@@ -11,7 +11,6 @@ All notable changes to this project are documented here. The format follows Keep
Requires QQQ 4.1.0-RC.15 or later (central health checks, help bundles, the spec-managed UI API extension contract, and `ClusterCaches` for multi-node deployments).
### Fixed
- Table standard: every `quick_search_*` table now has `createDate` and `modifyDate` (`DATE_TIME`, not editable, `CREATE_DATE` and `MODIFY_DATE` behaviors). `quick_search_config_tombstone`, `quick_search_diagnostic_event` and `quick_search_validation_run` gain `modifyDate`; the other five declared the fields but never filled them, because they had no behaviors. All writes go through QQQ's own insert and update actions, so no writer changed. Run the additive DDL in `docs/MIGRATION-1.1.md` ("Standard dates on every table") before deploying; it is safe while the previous release runs.
- Single-node clusters: the Quick Search index no longer asks for a replica it cannot place, so a one-node OpenSearch stays green instead of yellow and `quick_search.opensearch` is no longer DEGRADED. New `indexReplicas` (default null: automatic, `min(maxAutoReplicas, data nodes - 1)`) and `maxAutoReplicas` (default 1) config; zero config gives 0 replicas on one node and 1 on larger clusters. The target is set on every new physical index and reconciled on existing ones at startup with `PUT /<index>/_settings` when it differs (idempotent, logged; a failure is logged and reported, never blocks startup; indexes with `auto_expand_replicas` and OpenSearch Serverless are left alone). The status snapshot and the `quick_search.opensearch` health details gain `unassignedReplicaShards`, `targetReplicas` and `replicaSettingsFailed`; yellow on a single node with every primary active (only replicas unassigned) is reported UP with a note. Yellow with an unassigned primary, or on a multi-node cluster, stays DEGRADED; red stays DOWN.
- Priming config records at startup, and recording diagnostic events, build their system session with the instance, so it has its all-access keys. Startup logged "QInstance was not supplied when creating a QSystemUserSession" once per node.
- Multi-node deployments: a Quick Search config change (an index row, its fields or the settings row, from the admin UI or a config process) now clears the cached config on every node, not only the node that made the change. The runtime's config cache (the config store snapshot and settings, and the per-table enabled flags) is registered with QQQ `ClusterCaches` as `qbit-quick-search.config`, and `QuickSearchRuntime.invalidateConfig()` publishes a cluster invalidation. The 30 s and 60 s TTLs remain as the safety net. `QuickSearchFieldRules`' warn-once table set is registered as node-local. `QuickSearchClusterCachesTest` audits the package for unregistered static caches and covers a remote invalidation.
+2 -2
View File
@@ -255,8 +255,8 @@ The panel's status is Indexed, Stale, Not indexed, Disabled (the table is turned
- `quick_search_settings`: one row: `indexingPaused` (with `pausedAt`, `pausedBy`, `pauseReason`) and the last self-priming report (`lastPrimedAt`, `lastPrimingSummary`).
- `quick_search_config_tombstone`: tables (blank `fieldName`) and fields an admin removed; priming never re-adds them.
- `quick_search_index_run`: `runType` (`BASEPULL`, `RECONCILE`, `FULL_REINDEX`), `status` (`RUNNING`, `COMPLETED`, `FAILED`, `CANCELLED`), counts, `errorMessage`, `cancelRequested`, `cancelRequestedBy`.
- `quick_search_diagnostic_event` (with `enableDiagnosticTools`): `eventType` (`SEARCH_FAILED`, `TOOL_RUN`, `REINDEX`, `RATE_LIMITED`), `source`, `outcome`, `userName`, `tableNames`, `searchTerm` (tool runs only), `durationMillis`, `errorClass`, `errorMessage` (text), `detail` (text), `hostName`, `createDate`, `modifyDate`. Purged after `runHistoryRetentionDays`.
- `quick_search_validation_run` (with `enableDiagnosticTools`): `status` (`PASS`, `WARN`, `FAIL`), `triggerType` (`MANUAL`, `SCHEDULED`), `userName`, `tableNames`, `sampleSize`, `tablesChecked`, `failCount`, `warnCount`, `summary`, `reportHtml` (HTML), `reportJson` (text), `startTime`, `endTime`, `createDate`, `modifyDate`. Purged after `runHistoryRetentionDays`.
- `quick_search_diagnostic_event` (with `enableDiagnosticTools`): `eventType` (`SEARCH_FAILED`, `TOOL_RUN`, `REINDEX`, `RATE_LIMITED`), `source`, `outcome`, `userName`, `tableNames`, `searchTerm` (tool runs only), `durationMillis`, `errorClass`, `errorMessage` (text), `detail` (text), `hostName`, `createDate`. Purged after `runHistoryRetentionDays`.
- `quick_search_validation_run` (with `enableDiagnosticTools`): `status` (`PASS`, `WARN`, `FAIL`), `triggerType` (`MANUAL`, `SCHEDULED`), `userName`, `tableNames`, `sampleSize`, `tablesChecked`, `failCount`, `warnCount`, `summary`, `reportHtml` (HTML), `reportJson` (text), `startTime`, `endTime`, `createDate`. Purged after `runHistoryRetentionDays`.
- `quick_search_failed_event`: real-time events that could not be applied; `status` `PENDING` or `EXHAUSTED` after 10 attempts. Deletes made while a full reindex of all tables runs are also recorded here with status `AWAITING_REINDEX`: the reindex applies them to the new index after the alias swap (basepull does not replay them). While a table is `REBUILDING`, drift detection is suspended for it. Only one full reindex runs at a time: starting one while another is running fails. A full reindex shows it is running through its `FULL_REINDEX` run records, which stay `RUNNING` with `modifyDate` refreshed after every page. If it is killed mid-run (pod restart, OOM, deploy), recovery is automatic once its run records have shown no sign of life for `fullReindexStaleMinutes` (30 by default): the next basepull, or the next full reindex, marks those run records `FAILED`, returns the tables to `ACTIVE` with `lastBasepullTime` moved back to the run's start, and turns the `AWAITING_REINDEX` rows `PENDING` so basepull replays the deletes. No data is lost. A run that was only slow and is judged stopped notices before its alias swap and fails without swapping, or, after its swap, stops applying its captured deletes; basepull skips a replayed delete whose record exists again.
### Starter defaults and empty-state help
-30
View File
@@ -238,33 +238,3 @@ The qbit now sets bundle help (`help/quick-search.yaml`) on the Quick Search Adm
2. Grant `quick_search_index_field`, `quick_search_settings` and `quick_search_config_tombstone` permissions (read, and edit where admins should change them) next to the existing Quick Search grants.
3. Upgrade and start. The Search Diagnostics widget should not say the searchable tables come from code, and shows the first priming's summary.
4. Optionally move code-declared search config into the records and delete the annotations: the rows stay and are now the owner's.
## Standard dates on every table (after 1.1.0-RC.16)
Every `quick_search_*` table now has a `createDate` and a `modifyDate`: `DATE_TIME`, not editable, set by QQQ's `CREATE_DATE` and `MODIFY_DATE` behaviors. The QBit writes all of its tables with QQQ's own insert and update actions (no raw SQL), so the behaviors set them on every write; nothing else about the tables, the index documents or how a document is identified in the index changes.
- `quick_search_config_tombstone`, `quick_search_diagnostic_event` and `quick_search_validation_run` gain `modifyDate` (they had `createDate` only). For the tombstone and the two logs it equals `createDate` and never moves, since the QBit only inserts and deletes their rows.
- The other five tables already declared both fields, but without the behaviors, so the QBit never filled them. They are filled now. On `quick_search_index_run`, `modifyDate` stays the full reindex heartbeat (the heartbeat writes it explicitly); any other update of a run row, such as completing it, now moves it too, which only makes the run look more recently alive.
- The Search Diagnostics "last error" time reads `quick_search_index.modifyDate`, which now moves with every update of the row (including document counts), so it is the time of the last update of an index that has an error message, not strictly of the error.
### DDL (run before deploying)
A host that wrote its own DDL must have the columns before this release starts, because the QBit now writes them. Run `HOST-DDL-postgresql.sql`'s equivalent, one block per table (safe to run while the previous release is running, and safe to repeat):
```sql
-- repeat for quick_search_index, quick_search_index_field, quick_search_index_run, quick_search_failed_event,
-- quick_search_settings, quick_search_config_tombstone, quick_search_diagnostic_event, quick_search_validation_run
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'quick_search_config_tombstone' AND column_name = 'create_date') THEN
ALTER TABLE quick_search_config_tombstone ADD COLUMN create_date TIMESTAMP WITHOUT TIME ZONE DEFAULT now();
END IF;
IF NOT EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'quick_search_config_tombstone' AND column_name = 'modify_date') THEN
ALTER TABLE quick_search_config_tombstone ADD COLUMN modify_date TIMESTAMP WITHOUT TIME ZONE DEFAULT now();
UPDATE quick_search_config_tombstone SET modify_date = create_date WHERE create_date IS NOT NULL;
END IF;
END
$$;
```
The columns are nullable with a default of `now()`, which PostgreSQL 11 and later adds without rewriting the table (the previous release, which does not write them, gets the current time from the default). Existing rows take their own creation time where the table has one (`create_date` for the new `modify_date`; `start_time` for a newly added `create_date` on `quick_search_index_run` and `quick_search_validation_run`), otherwise the time of the migration. The one-time `UPDATE` touches every row, so on a very large log table run it in id ranges. Names are the snake_case a host gets when it derives backend names from field names; without that, the columns are `"createDate"` and `"modifyDate"`. MySQL: `DATETIME DEFAULT CURRENT_TIMESTAMP`, and drop the `DO` wrapper. No setting, customizer or deployment order is needed beyond running the DDL first.
+14 -3
View File
@@ -18,7 +18,6 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<qqq.version>4.1.0-RC.26</qqq.version> <!-- feeds qbit-build-parent's qqq-bom-pom import -->
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
<revision>1.1.0-SNAPSHOT</revision>
@@ -33,7 +32,7 @@
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.2.1</version>
<version>2.1.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -284,7 +283,7 @@
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.26</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.15</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -297,6 +296,18 @@
</dependency>
</dependencies>
</dependencyManagement>
<repositories>
<repository>
<id>sonatype-snapshots</id>
<url>https://central.sonatype.com/repository/maven-snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
</profile>
</profiles>
</project>
@@ -29,7 +29,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
@@ -120,8 +119,8 @@ public class QuickSearchConfigMetaDataHelper
.withField(new QFieldMetaData("sortOrder", QFieldType.INTEGER))
.withField(new QFieldMetaData("source", QFieldType.STRING).withIsEditable(false).withPossibleValueSourceName(config.applyTablePrefix(CONFIG_SOURCE_PVS)))
.withField(new QFieldMetaData("ownerEdited", QFieldType.BOOLEAN).withLabel("Edited by Owner").withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("identity", "Field", new QIcon("text_fields"), Tier.T1, List.of("id", "quickSearchIndexId", "fieldName", "weight", "role")))
.withSection(new QFieldSection("matching", "Matching", new QIcon("tune"), Tier.T2, List.of("analyzer", "includeInSnippet", "includeLabel", "sortOrder")))
.withSection(new QFieldSection("origin", "Origin", new QIcon("history"), Tier.T3, List.of("source", "ownerEdited", "createDate", "modifyDate"))));
@@ -156,8 +155,8 @@ public class QuickSearchConfigMetaDataHelper
.withField(new QFieldMetaData("lastPrimedAt", QFieldType.DATE_TIME).withLabel("Last Primed").withIsEditable(false))
.withField(new QFieldMetaData("lastPrimingSummary", QFieldType.TEXT).withLabel("Last Priming").withIsEditable(false))
.withField(new QFieldMetaData("lastPrimingNewTables", QFieldType.INTEGER).withLabel("Tables Added by Last Priming").withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("pause", "Indexing", new QIcon("pause_circle"), Tier.T1, List.of("id", "indexingPaused", "pauseReason", "pausedAt", "pausedBy")))
.withSection(new QFieldSection("priming", "Self-priming", new QIcon("auto_mode"), Tier.T2, List.of("lastPrimedAt", "lastPrimingSummary", "lastPrimingNewTables")))
.withSection(new QFieldSection("dates", "Dates", new QIcon("calendar_today"), Tier.T3, List.of("createDate", "modifyDate"))));
@@ -189,8 +188,7 @@ public class QuickSearchConfigMetaDataHelper
.withField(new QFieldMetaData("fieldName", QFieldType.STRING).withLabel("Field (blank: whole table)").withIsEditable(false))
.withField(new QFieldMetaData("removedBy", QFieldType.STRING).withIsEditable(false))
.withField(new QFieldMetaData("reason", QFieldType.STRING).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withLabel("Removed").withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false)));
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withLabel("Removed").withIsEditable(false)));
}
@@ -23,7 +23,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
@@ -104,8 +103,8 @@ public class QuickSearchTableMetaDataHelper
.withField(new QFieldMetaData("realTimeErrorCount", QFieldType.INTEGER).withIsEditable(false)
.withHelpContent(new QHelpContent("Real-time indexing failures since the last full reindex or basepull")))
.withField(new QFieldMetaData("status", QFieldType.STRING).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("identity", "Search Index", new QIcon("manage_search"), Tier.T1,
List.of("id", "tableName", "enabled", "status", "basepullIntervalMinutes", "basepullTimestampField")))
.withSection(new QFieldSection("fields", "Searchable Fields", new QIcon("text_fields"), Tier.T2, null)
@@ -154,8 +153,8 @@ public class QuickSearchTableMetaDataHelper
.withField(new QFieldMetaData("errorMessage", QFieldType.TEXT).withIsEditable(false))
.withField(new QFieldMetaData("cancelRequested", QFieldType.BOOLEAN).withIsEditable(false))
.withField(new QFieldMetaData("cancelRequestedBy", QFieldType.STRING).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withLabel("Last Heartbeat").withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withLabel("Last Heartbeat").withIsEditable(false))
.withSection(new QFieldSection("run", "Run", new QIcon("history"), Tier.T1, List.of("id", "quickSearchIndexId", "runType", "status", "startTime", "endTime")))
.withSection(new QFieldSection("counts", "Counts", new QIcon("functions"), Tier.T2, List.of("recordsProcessed", "recordsIndexed", "errorCount", "errorMessage")))
.withSection(new QFieldSection("cancel", "Cancel", new QIcon("cancel"), Tier.T2, List.of("cancelRequested", "cancelRequestedBy")))
@@ -194,8 +193,8 @@ public class QuickSearchTableMetaDataHelper
.withField(new QFieldMetaData("attempts", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("status", QFieldType.STRING).withIsEditable(false)
.withPossibleValueSourceName(config.applyTablePrefix(QuickSearchConfigMetaDataHelper.EVENT_STATUS_PVS)))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("event", "Event", new QIcon("error_outline"), Tier.T1, List.of("id", "tableName", "recordId", "action", "status")))
.withSection(new QFieldSection("error", "Error and retries", new QIcon("replay"), Tier.T2, List.of("errorMessage", "attempts")))
.withSection(new QFieldSection("dates", "Dates", new QIcon("calendar_today"), Tier.T3, List.of("createDate", "modifyDate"))));
@@ -233,9 +232,8 @@ public class QuickSearchTableMetaDataHelper
.withField(new QFieldMetaData("errorMessage", QFieldType.TEXT).withIsEditable(false))
.withField(new QFieldMetaData("detail", QFieldType.TEXT).withIsEditable(false))
.withField(new QFieldMetaData("hostName", QFieldType.STRING).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withSection(new QFieldSection("identity", "Event", new QIcon("bug_report"), Tier.T1, List.of("id", "createDate", "modifyDate", "eventType", "source", "outcome")))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("identity", "Event", new QIcon("bug_report"), Tier.T1, List.of("id", "createDate", "eventType", "source", "outcome")))
.withSection(new QFieldSection("request", "Request", new QIcon("search"), Tier.T2, List.of("userName", "tableNames", "searchTerm", "durationMillis", "hostName")))
.withSection(new QFieldSection("error", "Error and detail", new QIcon("error_outline"), Tier.T2, List.of("errorClass", "errorMessage", "detail"))));
}
@@ -275,10 +273,9 @@ public class QuickSearchTableMetaDataHelper
.withFieldAdornment(new FieldAdornment(AdornmentType.CODE_EDITOR).withValue(AdornmentType.CodeEditorValues.languageMode("json"))))
.withField(new QFieldMetaData("startTime", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("endTime", QFieldType.DATE_TIME).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.CREATE_DATE).withIsEditable(false))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE).withIsEditable(false))
.withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withSection(new QFieldSection("identity", "Run", new QIcon("fact_check"), Tier.T1, List.of("id", "status", "summary", "triggerType", "startTime", "endTime")))
.withSection(new QFieldSection("scope", "Scope", new QIcon("table_chart"), Tier.T2, List.of("userName", "tableNames", "sampleSize", "tablesChecked", "failCount", "warnCount", "createDate", "modifyDate")))
.withSection(new QFieldSection("scope", "Scope", new QIcon("table_chart"), Tier.T2, List.of("userName", "tableNames", "sampleSize", "tablesChecked", "failCount", "warnCount", "createDate")))
.withSection(new QFieldSection("report", "Report", new QIcon("description"), Tier.T2, List.of("reportHtml")))
.withSection(new QFieldSection("data", "Report data (JSON)", new QIcon("data_object"), Tier.T3, List.of("reportJson"))));
}
@@ -45,9 +45,6 @@ public class QuickSearchConfigTombstone extends QRecordEntity
@QField()
private Instant createDate;
@QField()
private Instant modifyDate;
/*******************************************************************************
** Getter for id
@@ -227,35 +224,4 @@ public class QuickSearchConfigTombstone extends QRecordEntity
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate
*******************************************************************************/
public Instant getModifyDate()
{
return (this.modifyDate);
}
/*******************************************************************************
** Setter for modifyDate
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Fluent setter for modifyDate
*******************************************************************************/
public QuickSearchConfigTombstone withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
}
@@ -87,9 +87,6 @@ public class QuickSearchDiagnosticEvent extends QRecordEntity
@QField()
private Instant createDate;
@QField()
private Instant modifyDate;
/*******************************************************************************
@@ -512,35 +509,4 @@ public class QuickSearchDiagnosticEvent extends QRecordEntity
return (this);
}
/*******************************************************************************
** Getter for modifyDate
*******************************************************************************/
public Instant getModifyDate()
{
return (this.modifyDate);
}
/*******************************************************************************
** Setter for modifyDate
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Fluent setter for modifyDate
*******************************************************************************/
public QuickSearchDiagnosticEvent withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
}
@@ -89,9 +89,6 @@ public class QuickSearchValidationRun extends QRecordEntity
@QField()
private Instant createDate;
@QField()
private Instant modifyDate;
/*******************************************************************************
@@ -576,35 +573,4 @@ public class QuickSearchValidationRun extends QRecordEntity
return (this);
}
/*******************************************************************************
** Getter for modifyDate
*******************************************************************************/
public Instant getModifyDate()
{
return (this.modifyDate);
}
/*******************************************************************************
** Setter for modifyDate
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Fluent setter for modifyDate
*******************************************************************************/
public QuickSearchValidationRun withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
}
@@ -1,292 +0,0 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.quicksearch;
import java.time.Instant;
import java.util.List;
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField;
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** QQQ table standard: every table has a createDate and a modifyDate, typed
** DATE_TIME, not editable, maintained by the CREATE_DATE and MODIFY_DATE
** behaviors. The QBit writes all of its tables with QQQ's own insert and
** update actions (no raw SQL), so the behaviors are what sets the values;
** these tests walk every table the producer declares, so a table added later
** without the two dates fails here.
*******************************************************************************/
class QuickSearchTableStandardDatesTest
{
/*******************************************************************************
** A searchable entity for the produced instance.
*******************************************************************************/
@QuickSearchable(tableName = "person")
static class PersonEntity
{
@QuickSearchField(weight = 3)
private String name;
}
/*******************************************************************************
**
*******************************************************************************/
@BeforeEach
void setUp()
{
MemoryRecordStore.getInstance().reset();
}
/*******************************************************************************
**
*******************************************************************************/
@AfterEach
void tearDown()
{
QContext.clear();
}
/*******************************************************************************
** Every table the producer declares has both dates, as the standard says.
*******************************************************************************/
@Test
void everyTableHasTheTwoStandardDates()
{
for(QTableMetaData table : quickSearchTables(producedInstance()))
{
QFieldMetaData createDate = table.getFields().get("createDate");
QFieldMetaData modifyDate = table.getFields().get("modifyDate");
assertThat(createDate).as(table.getName() + " createDate").isNotNull();
assertThat(createDate.getType()).as(table.getName() + " createDate type").isEqualTo(QFieldType.DATE_TIME);
assertThat(createDate.getIsEditable()).as(table.getName() + " createDate editable").isFalse();
assertThat(createDate.getBehaviors()).as(table.getName() + " createDate behaviors").contains(DynamicDefaultValueBehavior.CREATE_DATE);
assertThat(modifyDate).as(table.getName() + " modifyDate").isNotNull();
assertThat(modifyDate.getType()).as(table.getName() + " modifyDate type").isEqualTo(QFieldType.DATE_TIME);
assertThat(modifyDate.getIsEditable()).as(table.getName() + " modifyDate editable").isFalse();
assertThat(modifyDate.getBehaviors()).as(table.getName() + " modifyDate behaviors").contains(DynamicDefaultValueBehavior.MODIFY_DATE);
}
}
/*******************************************************************************
** The entities that stand for the tables carry both dates too, so a row read
** back as an entity shows them.
*******************************************************************************/
@Test
void everyEntityHasTheTwoStandardDates()
{
List<Class<?>> entities = List.of(
com.kingsrook.qbits.quicksearch.model.QuickSearchIndex.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchIndexField.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchIndexRun.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchSettings.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchConfigTombstone.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent.class,
com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun.class);
assertThat(entities).hasSize(quickSearchTables(producedInstance()).size());
for(Class<?> entity : entities)
{
assertThat(entity.getDeclaredFields()).extracting(java.lang.reflect.Field::getName)
.as(entity.getSimpleName()).contains("createDate", "modifyDate");
}
}
/*******************************************************************************
** Inserting a row, with QQQ's own insert action (the only way the QBit writes
** these tables), sets both dates; an update moves modifyDate and leaves
** createDate alone.
*******************************************************************************/
@Test
void insertSetsBothDatesAndUpdateMovesModifyDateOnly() throws Exception
{
QInstance qInstance = producedInstance();
for(QTableMetaData table : quickSearchTables(qInstance))
{
QRecord inserted = new InsertAction().execute(new InsertInput(table.getName()).withRecord(minimalRecord(table))).getRecords().get(0);
assertThat(inserted.getErrors()).as(table.getName() + " insert errors").isEmpty();
Instant createDate = inserted.getValueInstant("createDate");
Instant modifyDate = inserted.getValueInstant("modifyDate");
assertThat(createDate).as(table.getName() + " createDate after insert").isNotNull();
assertThat(modifyDate).as(table.getName() + " modifyDate after insert").isNotNull();
String changeField = changeableField(table);
if(changeField == null)
{
continue;
}
Thread.sleep(20);
QRecord update = new QRecord().withValue(table.getPrimaryKeyField(), inserted.getValue(table.getPrimaryKeyField())).withValue(changeField, changedValue(table.getField(changeField)));
QRecord updated = new UpdateAction().execute(new UpdateInput(table.getName()).withRecord(update)).getRecords().get(0);
assertThat(updated.getErrors()).as(table.getName() + " update errors").isEmpty();
assertThat(updated.getValueInstant("modifyDate")).as(table.getName() + " modifyDate after update").isAfter(modifyDate);
assertThat(updated.getValueInstant("createDate") == null || updated.getValueInstant("createDate").equals(createDate))
.as(table.getName() + " createDate after update").isTrue();
}
}
/*******************************************************************************
** The full reindex heartbeat writes a bare modifyDate: that still works, and
** the value is not older than what was there.
*******************************************************************************/
@Test
void heartbeatWriteOfModifyDateOnlyStillMovesModifyDate() throws Exception
{
QInstance qInstance = producedInstance();
QTableMetaData runTable = qInstance.getTable("quick_search_index_run");
QRecord inserted = new InsertAction().execute(new InsertInput(runTable.getName()).withRecord(minimalRecord(runTable))).getRecords().get(0);
Instant before = inserted.getValueInstant("modifyDate");
Thread.sleep(20);
QRecord heartbeat = new QRecord().withValue("id", inserted.getValue("id")).withValue("modifyDate", Instant.now());
QRecord updated = new UpdateAction().execute(new UpdateInput(runTable.getName()).withRecord(heartbeat)).getRecords().get(0);
assertThat(updated.getValueInstant("modifyDate")).isAfter(before);
assertThat(updated.getErrors()).isEmpty();
}
/*******************************************************************************
** A record with only what the table requires.
*******************************************************************************/
private QRecord minimalRecord(QTableMetaData table)
{
QRecord record = new QRecord();
switch(table.getName())
{
case "quick_search_index" -> record.withValue("tableName", "person");
case "quick_search_index_field" -> record.withValue("quickSearchIndexId", 1).withValue("fieldName", "name");
default ->
{
}
}
return (record);
}
/*******************************************************************************
** A plain text or number field, not a key, a date or a possible-value
** picker, that an update can change; null if the table has none.
*******************************************************************************/
private String changeableField(QTableMetaData table)
{
return (table.getFields().values().stream()
.filter(f -> !f.getName().equals(table.getPrimaryKeyField()))
.filter(f -> f.getPossibleValueSourceName() == null)
.filter(f -> f.getType() == QFieldType.STRING || f.getType() == QFieldType.INTEGER)
.map(QFieldMetaData::getName)
.findFirst().orElse(null));
}
/*******************************************************************************
**
*******************************************************************************/
private java.io.Serializable changedValue(QFieldMetaData field)
{
return (field.getType() == QFieldType.INTEGER ? Integer.valueOf(7) : "changed");
}
/*******************************************************************************
**
*******************************************************************************/
private List<QTableMetaData> quickSearchTables(QInstance qInstance)
{
List<QTableMetaData> tables = qInstance.getTables().values().stream()
.filter(t -> t.getName().startsWith("quick_search_"))
.sorted((a, b) -> a.getName().compareTo(b.getName()))
.toList();
assertThat(tables).as("the tables this QBit declares").hasSize(8);
return (tables);
}
/*******************************************************************************
**
*******************************************************************************/
private QInstance producedInstance()
{
try
{
QInstance qInstance = new QInstance();
qInstance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
qInstance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
qInstance.addTable(new QTableMetaData().withName("person").withBackendName("memory").withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
QContext.init(qInstance, new QSession());
new QuickSearchQBitProducer().withConfig(new QuickSearchQBitConfig()
.withStartupMode(QuickSearchStartupMode.DEGRADED)
.withBackendName("memory")
.withOpensearchHost("localhost")
.withOpensearchPort(9200)
.withOpensearchIndexName("table-standard")
.withSearchableEntityClasses(List.of(PersonEntity.class))).produce(qInstance);
return (qInstance);
}
catch(QException e)
{
throw (new IllegalStateException(e));
}
}
}
@@ -989,10 +989,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
}
};
//////////////////////////////////////////////////////////////////////////////////////
// the MODIFY_DATE behavior stamps modifyDate to the millisecond, so compare at that //
//////////////////////////////////////////////////////////////////////////////////////
Instant before = Instant.now().truncatedTo(ChronoUnit.MILLIS);
Instant before = Instant.now();
step.indexAllRecords(mockClient, QuickSearchQBitContext.getTableConfig(TEST_ENTITY_TABLE), List.of(), PHYSICAL_INDEX,
new QuickSearchIndexRun().withId(runId).withRunType(FullReindexStep.RUN_TYPE));