feat: serve QQQ /qqq/v1/search from OpenSearch (record-search provider)

Add QuickSearchRecordSearchProvider, an implementation of QQQ core's
RecordSearchProviderInterface (QRun-IO/qqq#1042), so RecordSearchAction
and POST /qqq/v1/search (the Next UI global search) are served from the
Quick Search index for the tables this QBit indexes (audit F-11, D1).

- Claims indexed, enabled tables; one _msearch request with a per-table
  top-N search (same query, AND semantics and field weights), returning
  record ids only, which core re-reads through QueryAction.
- recordSearchTimeoutMillis (default 2500) bounds the request; any
  failure makes core fall back to its own search and starts a 30 s
  backoff during which no tables are claimed.
- serveCoreRecordSearch (default true) registers the provider and gives
  indexed tables core searchFields when they have none; host
  searchFields and a host-registered provider are kept.
- PASSWORD fields are no longer indexed (like hidden fields).
- Build against qqq 4.1.0-SNAPSHOT until the SPI is released; version
  1.1.0-SNAPSHOT; README, MIGRATION-1.1, CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
James MaesandClaude Opus 5.5 committed 2026-10-10 19:18:05 -05:00
1 parent 2b627e767a
commit 75dc2b3b3d
16 files changed
+1416 -31

No files matched your search

+3 -2
View File
@@ -15,7 +15,7 @@ This file provides guidance to Codex when working with code in this repository.
./mvnw package # Build JAR
CI=true ./mvnw verify # Full verify: integration tests (Docker) and the JaCoCo gate
./mvnw verify -Dopensearch.test.image=opensearchproject/opensearch:3.9.0 # other server line
./mvnw verify -Dqqq.snapshot.version=4.1.0-SNAPSHOT # other qqq 4.1 build (profile is active by default)
./mvnw verify -Dqqq.snapshot.version=4.1.0-RC.1 # other qqq 4.1 build (profile is active by default)
```
Integration tests (require Docker) run via `mvn verify` using maven-failsafe-plugin.
@@ -34,6 +34,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
- `listeners/QuickSearchRecordChangeListener` - Real-time path: builds events in the write's transaction, publishes after commit, records failures in `quickSearchFailedEvent`.
- `processes/AbstractIndexingStep` - Keyset paging by primary key with display values, run records, drift detection, run purge. `BasepullIndexStep` (replays failed events, start-anchored watermark with overlap), `ReconcileIndexStep` (in place, orphan sweep), `FullReindexStep` (all tables: new physical index plus alias swap; one table: reconcile algorithm).
- `actions/QuickSearchAction` - Permission-aware search (table READ filter, `QueryAction` post-filter), bounds, `tableNames`, `limitPerTable`, `tableLabel`.
- `actions/QuickSearchRecordSearchProvider` - QQQ core `RecordSearchProviderInterface` (`POST /qqq/v1/search`): claims indexed, enabled tables; one `_msearch` per call returning ordered record ids per table; bounded by `recordSearchTimeoutMillis`; failures make core fall back and start a 30 s backoff (`QuickSearchRuntime.markRecordSearchFailed`). Registered, with default table `searchFields`, when `serveCoreRecordSearch` is true.
- `metadata/*` - Component producers and the table/process helpers.
**Tables Produced:** `quickSearchIndex`, `quickSearchIndexRun`, `quickSearchFailedEvent` (plus a PVS over `quickSearchIndex`).
@@ -50,7 +51,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
## Dependencies
- Java 21
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom:4.1.0-RC.1` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007)
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007). 1.1 temporarily imports `4.1.0-SNAPSHOT` for the record-search provider SPI (QRun-IO/qqq#1042); re-pin to the QQQ release that carries it before releasing 1.1.0
- OpenSearch Java Client 3.10.0; optional AWS SDK v2 `apache-client` and `sts` for `AWS_SIGV4`
- Apache HttpClient5 and jackson-datatype-jsr310, versions managed by the qqq BOM (keep them unpinned so they match its httpcore5 and jackson)
- JUnit 5 + AssertJ + Mockito for testing
+13
View File
@@ -2,6 +2,19 @@
All notable changes to this project are documented here. The format follows Keep a Changelog; versions follow semantic versioning.
## [1.1.0] - Unreleased
Requires the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042); built against 4.1.0-SNAPSHOT until it is released. See `docs/MIGRATION-1.1.md`.
### Added
- `QuickSearchRecordSearchProvider`: serves QQQ core record search (`RecordSearchAction`, `POST /qqq/v1/search`, the Next UI global search) from OpenSearch for the tables Quick Search indexes. One `_msearch` request with a per-table top-N search (same query, AND semantics and field weights as `QuickSearchAction`); returns record ids only, which core re-reads through `QueryAction` (permissions, record locks, labels). Audit finding F-11, decision D1.
- `serveCoreRecordSearch` (default `true`): registers the provider and gives indexed tables core `searchFields` (visible string and integer searchable fields) when they have none. Host `searchFields` and a host-registered provider are never replaced.
- `recordSearchTimeoutMillis` (default 2500): wall-clock limit on the OpenSearch request; on timeout or any failure core falls back to its own search, and the provider claims no tables for 30 seconds.
- `QuickSearchOpenSearchClient.searchRecordIdsPerTable`.
### Security
- Fields of type `PASSWORD` are no longer indexed (as hidden fields already were). Reconcile or fully reindex once to remove values a 1.0 index may hold.
## [1.0.0-RC.2] - 2026-10-06
Second release candidate, published to Maven Central from `release/1.0`. Fixes the five pre-GA findings from the PR #10 review (#13 to #17); the changes are listed under 1.0.0 below. Requires QQQ 4.1. No schema change, but two new status values: `REBUILDING` on `quickSearchIndex.status` and `AWAITING_REINDEX` on `quickSearchFailedEvent.status`; a host that sized its own `quickSearchFailedEvent.status` column needs at least 16 characters. Known gaps before GA are tracked in #12.
+3 -2
View File
@@ -15,7 +15,7 @@ This file provides guidance to Claude Code when working with code in this reposi
./mvnw package # Build JAR
CI=true ./mvnw verify # Full verify: integration tests (Docker) and the JaCoCo gate
./mvnw verify -Dopensearch.test.image=opensearchproject/opensearch:3.9.0 # other server line
./mvnw verify -Dqqq.snapshot.version=4.1.0-SNAPSHOT # other qqq 4.1 build (profile is active by default)
./mvnw verify -Dqqq.snapshot.version=4.1.0-RC.1 # other qqq 4.1 build (profile is active by default)
```
Integration tests (require Docker) run via `mvn verify` using maven-failsafe-plugin.
@@ -34,6 +34,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
- `listeners/QuickSearchRecordChangeListener` - Real-time path: builds events in the write's transaction, publishes after commit, records failures in `quickSearchFailedEvent`.
- `processes/AbstractIndexingStep` - Keyset paging by primary key with display values, run records, drift detection, run purge. `BasepullIndexStep` (replays failed events, start-anchored watermark with overlap), `ReconcileIndexStep` (in place, orphan sweep), `FullReindexStep` (all tables: new physical index plus alias swap; one table: reconcile algorithm).
- `actions/QuickSearchAction` - Permission-aware search (table READ filter, `QueryAction` post-filter), bounds, `tableNames`, `limitPerTable`, `tableLabel`.
- `actions/QuickSearchRecordSearchProvider` - QQQ core `RecordSearchProviderInterface` (`POST /qqq/v1/search`): claims indexed, enabled tables; one `_msearch` per call returning ordered record ids per table; bounded by `recordSearchTimeoutMillis`; failures make core fall back and start a 30 s backoff (`QuickSearchRuntime.markRecordSearchFailed`). Registered, with default table `searchFields`, when `serveCoreRecordSearch` is true.
- `metadata/*` - Component producers and the table/process helpers.
**Tables Produced:** `quickSearchIndex`, `quickSearchIndexRun`, `quickSearchFailedEvent` (plus a PVS over `quickSearchIndex`).
@@ -50,7 +51,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
## Dependencies
- Java 21
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom:4.1.0-RC.1` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007)
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007). 1.1 temporarily imports `4.1.0-SNAPSHOT` for the record-search provider SPI (QRun-IO/qqq#1042); re-pin to the QQQ release that carries it before releasing 1.1.0
- OpenSearch Java Client 3.10.0; optional AWS SDK v2 `apache-client` and `sts` for `AWS_SIGV4`
- Apache HttpClient5 and jackson-datatype-jsr310, versions managed by the qqq BOM (keep them unpinned so they match its httpcore5 and jackson)
- JUnit 5 + AssertJ + Mockito for testing
+20 -2
View File
@@ -8,7 +8,7 @@
Annotate entity classes or declare tables in config, point the QBit at an OpenSearch cluster, produce it into your `QInstance`, and you get real-time indexing of committed writes, a scheduled catch-up and reconcile, a blackout-free full rebuild, and a permission-aware search API with relevance scoring, typeahead, highlights and pagination.
Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit callbacks and runtime services).
Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit callbacks and runtime services). Serving core search (`POST /qqq/v1/search`, below) needs the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042); until it is released, 1.1 builds against 4.1.0-SNAPSHOT.
## What it does
@@ -18,6 +18,7 @@ Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit call
| Basepull catch-up | Scheduled process (`schedulerName` plus `basepullRepeatSeconds`) that re-indexes rows changed since the previous run's start, with a configurable overlap, paging by primary key. |
| Reconcile | Scheduled or on-demand process that re-indexes everything in place, then removes documents whose source row is gone, including documents of tables no longer configured. No search blackout. |
| Full rebuild | Indexes every table into a fresh physical index and atomically swaps the alias; the old index keeps serving until the new one is complete. Also upgrades an index created by an older release. |
| Core search | `QuickSearchRecordSearchProvider` serves QQQ's `RecordSearchAction` (`POST /qqq/v1/search`, the Next UI global search box) from OpenSearch for the indexed tables, with a timeout and an automatic fallback to core's own search. |
| Search | `QuickSearchAction`: AND semantics, edge-ngram typeahead with a plain search analyzer, per-field weights, highlights, `tableNames` and `limitPerTable`, bounded paging, `tableLabel` on results. Only tables the session may read are searched; hits are re-read through `QueryAction` so record security locks apply. |
| Connections | HTTP or HTTPS, basic auth, AWS IAM SigV4, custom CA or trust store, mTLS, timeouts, pool size, a transport customizer SPI for anything else. Secrets by `${env.X}` reference. |
| Operations | `quickSearchIndex` (per-table status, counts, last run, last error), `quickSearchIndexRun` (history, purged after `runHistoryRetentionDays`), `quickSearchFailedEvent`, a `quickSearchAdmin` app, explicit permission rules, fail-fast or degraded startup. |
@@ -99,7 +100,7 @@ config.withSearchableTable(new SearchableTableConfig("crmContact", List.of(
.withRecordLabelFormat("%s %s", "firstName", "lastName"));
```
Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull.
Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden and password fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull.
### Searching
@@ -124,6 +125,17 @@ output.getTotalHits(); output.getTotalHitsIsLowerBound(); output.getHasMore();
Search runs as the current `QContext` session: tables the session cannot read are skipped, and with `applyRecordSecurityLocks` (default) hits are re-read through `QueryAction` so record security locks apply.
### Core search (`POST /qqq/v1/search`)
QQQ core has its own record search (`RecordSearchAction`, served at `POST /qqq/v1/search` and used by the Next UI's global search box): a case-insensitive contains-query over each table's `searchFields`. With `serveCoreRecordSearch` (default `true`) the producer registers `QuickSearchRecordSearchProvider` on the instance, and core asks it to search every table Quick Search indexes. The API and the UI do not change.
- **What is served from OpenSearch.** Tables this QBit indexes and that are enabled in `quickSearchIndex`. Core offers a table only when it has `searchFields`, the session may read it, and it supports query, so the producer gives each indexed table that has no `searchFields` its visible string and integer searchable fields (never hidden or password fields). `searchFields` you set yourself are kept. A table must be in the `QInstance` before the QBit is produced to get them.
- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights); each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order. Core re-reads them by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, and records deleted since they were indexed drop out.
- **Failure.** The OpenSearch request is cancelled after `recordSearchTimeoutMillis` (default 2500). Any failure (timeout, cluster down, index missing) makes core search those tables with its own contains-query instead, so search degrades but never fails. After a failure, Quick Search claims no tables for 30 seconds, so a down cluster does not add a timeout to every search.
- **Opting out.** `withServeCoreRecordSearch(false)` leaves core search and `searchFields` untouched. If the host already registered another record-search provider, the QBit keeps it and logs a warning (an instance has one provider).
`QuickSearchAction` remains the richer API (scores, highlights, pagination, totals).
### Processes
| Process | Trigger | What it does |
@@ -159,6 +171,8 @@ Without `schedulerName` the processes exist but nothing runs them; the QBit logs
| `maxFieldLength` | 10000 | Truncate each indexed value |
| `maxSearchLimit` | 100 | Largest page size |
| `applyRecordSecurityLocks` | `true` | Re-read hits through `QueryAction` |
| `serveCoreRecordSearch` | `true` | Register `QuickSearchRecordSearchProvider` so core `POST /qqq/v1/search` uses OpenSearch for indexed tables, and give those tables `searchFields` when they have none |
| `recordSearchTimeoutMillis` | 2500 | Wall-clock limit on the OpenSearch request behind a core search; past it, core falls back to its own search |
| `adminPermissionRules` | `HAS_ACCESS_PERMISSION`, base name `quickSearchAdmin` | Rules for the app and processes |
| `tablePermissionRules` | `READ_WRITE_PERMISSIONS` | Rules for the operational tables |
| `tableMetaDataCustomizer` | | Applied to every produced table (backend details, for example) |
@@ -187,6 +201,10 @@ CI=true ./mvnw verify # plus Testcontainers integration te
Use the wrapper: Maven 3.10 cannot read the published parent POM. The build imports qqq 4.1.0-RC.1 through the default-active `qqq-snapshot` profile until `qbit-build-parent` 2.1.0 ships (override with `-Dqqq.snapshot.version=...`). Without Docker the integration tests are skipped locally; with `CI=true` they fail instead.
## Upgrading from 1.0
See [docs/MIGRATION-1.1.md](docs/MIGRATION-1.1.md). In short: 1.1 needs the QQQ release with the record-search provider SPI; core search starts using OpenSearch for indexed tables by default, and indexed tables gain `searchFields` (set `serveCoreRecordSearch(false)` to keep 1.0 behaviour).
## Upgrading from 0.x
See [docs/MIGRATION-1.0.md](docs/MIGRATION-1.0.md). In short: QQQ 4.1 is required, remove any manual `addQBit` for this QBit, set `schedulerName`, prefer `opensearchUrl` and `${env.}` secrets, and run a full reindex once so the index gets the new mapping.
+30
View File
@@ -0,0 +1,30 @@
# Migrating from 1.0 to 1.1
1.1 lets QQQ core's record search (`POST /qqq/v1/search`, the Next UI global search box) use the Quick Search index. Hosts that do nothing get it by default.
## Prerequisites
- The QQQ release that carries the record-search provider SPI (`RecordSearchProviderInterface`, `QInstance.withRecordSearchProvider`), added in QRun-IO/qqq#1042. Until that release exists, 1.1 builds against `4.1.0-SNAPSHOT` through the `qqq-snapshot` profile; it is re-pinned to the release before 1.1.0 ships.
## What changes at startup
1. **The QBit registers a record-search provider.** `QuickSearchRecordSearchProvider` becomes the instance's `recordSearchProvider`. If the host already registered a different provider, the QBit keeps it and logs a warning; core search then does not use OpenSearch.
2. **Indexed tables get `searchFields`.** Core only searches tables with `searchFields`. Each indexed table that has none gets its visible string and integer searchable fields. Tables that already have `searchFields` keep them. A table added to the instance after the QBit is produced gets none, so add tables first (as for field validation) or set `searchFields` yourself.
3. **Password fields are no longer indexed.** Fields of type `PASSWORD` are now dropped from the searchable fields like hidden fields. If a 1.0 index holds such values, run "Quick Search Reconcile Index" (or a full reindex) to rewrite the documents without them.
## Behaviour to be aware of
- Next UI search results for indexed tables now come from OpenSearch: all words must match (in any searchable field), with prefix matching and field weights, instead of a contains-match on one field. Tables not indexed by Quick Search keep core's search.
- Results still pass core's security: core re-reads every hit through `QueryAction`, so table permissions and record security locks apply, and records deleted since they were indexed do not appear.
- Records are only as fresh as the index: a record updated in a way that is not yet indexed (for example while real-time indexing is off) is found by its indexed text.
- When OpenSearch is slow or down, core waits at most `recordSearchTimeoutMillis` (default 2500) and then searches with its own contains-query; for the next 30 seconds it skips OpenSearch entirely.
## Opting out
```java
new QuickSearchQBitConfig()
// ...
.withServeCoreRecordSearch(false);
```
restores 1.0 behaviour: no provider, no `searchFields` changes.
+6 -1
View File
@@ -9,7 +9,7 @@
- [ ] qqq-all follow-up PR: remove the manual `instance.addQBit(...)` for quick-search in `DemoQBits`, pass `withSchedulerName("demoScheduler")`, set `withStartupMode(QuickSearchStartupMode.DEGRADED)` for the full profile test, pin 1.0.0.
- [ ] Voyage: pin 1.0.0, set `schedulerName`, switch to `opensearchUrl` and `${env.}` credentials.
- [ ] CI: add a second test job with `-Dopensearch.test.image=opensearchproject/opensearch:3.9.0` (needs a qqq-orb parameter for extra Maven args, or a plain job).
- [ ] Open the qqq issue proposing `RecordSearchProviderInterface` (D1); quick-search adapter lands in 1.1.
- [x] Core SPI proposed as QRun-IO/qqq#1042 (D1); the 1.1 adapter `QuickSearchRecordSearchProvider` is on `feature/record-search-provider`.
- [x] Tracking issue #12 created and recorded in the PRD (2026-10-06).
- [x] PR #8 merged 2026-10-05; Dependabot PR #1 closed as superseded.
- [x] PR #10 merged and `1.0.0-RC.1` published to Maven Central from `release/1.0` (2026-10-05).
@@ -26,6 +26,11 @@
- [ ] Stuck `REBUILDING` after a killed or overlapping full reindex (from the PR #20 review): only another all-tables reindex clears it, `AWAITING_REINDEX` rows accumulate and drift detection stays off for those tables. No data is lost. Fix needs a liveness check on the `FULL_REINDEX` run record before reconcile may reset the status, plus refusing a second concurrent full reindex.
- [ ] Drift detection still writes `NEEDS_REINDEX` into `status`, so a few-millisecond window can overwrite `REBUILDING`; the durable fix is a separate drift field (schema change, so 1.1 at the earliest).
## Before releasing 1.1.0
- [ ] Re-point the `qqq-snapshot` profile from `4.1.0-SNAPSHOT` to the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042).
- [ ] qqq-all / Website-Backend: confirm the Next UI search box is served from OpenSearch (provider registered, indexed tables have `searchFields`).
## Later (1.x)
- [ ] Health widget for the admin app (data exists: `isHealthy`, `documentCount`, `lastRunStatus`).
+6 -2
View File
@@ -18,7 +18,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<revision>1.0.0-SNAPSHOT</revision>
<revision>1.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<opensearch.version>3.10.0</opensearch.version>
@@ -264,13 +264,17 @@
<!-- qqq-bom-pom 4.1.0-RC.1 ahead of the parent's 4.0.0 BOM. Override with -->
<!-- -Dqqq.snapshot.version=... ; deactivate with -P!qqq-snapshot. Remove the -->
<!-- activation when the parent moves to 4.1 (ADR-0007). -->
<!-- 1.1 TEMPORARY: the record-search provider needs the core SPI from -->
<!-- QRun-IO/qqq#1042 (RecordSearchProviderInterface), which is only in -->
<!-- 4.1.0-SNAPSHOT once #1042 merges. Re-point this to the QQQ release that -->
<!-- carries the SPI before releasing 1.1.0. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.1</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-SNAPSHOT</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -52,14 +52,20 @@ import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher;
** enableMaintenanceProcesses, schedulerName plus basepull/reconcile schedules,
** permission rules for the admin surface, search bounds, and indexing limits.
**
** Core search: serveCoreRecordSearch (default true) registers
** QuickSearchRecordSearchProvider, so QQQ's POST /qqq/v1/search (the Next UI
** global search) is served from OpenSearch for indexed tables, bounded by
** recordSearchTimeoutMillis.
**
** Tables: searchableEntityClasses (annotated entities) and/or searchableTables
** (config-driven). validate() is list-based and never throws; the producer
** fails production when it reports errors.
*******************************************************************************/
public class QuickSearchQBitConfig implements QBitConfig
{
public static final String DEFAULT_ADMIN_PERMISSION_BASE_NAME = "quickSearchAdmin";
public static final String DEFAULT_BASEPULL_TIMESTAMP_FIELD = "modifyDate";
public static final String DEFAULT_ADMIN_PERMISSION_BASE_NAME = "quickSearchAdmin";
public static final String DEFAULT_BASEPULL_TIMESTAMP_FIELD = "modifyDate";
public static final int DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS = 2500;
private static final Pattern INDEX_NAME_PATTERN = Pattern.compile("^[a-z0-9][a-z0-9_+.-]*$");
private static final Set<String> AWS_SERVICE_NAMES = Set.of("es", "aoss");
@@ -100,6 +106,8 @@ public class QuickSearchQBitConfig implements QBitConfig
private Integer maxFieldLength = 10000;
private Integer maxSearchLimit = 100;
private Boolean applyRecordSecurityLocks = true;
private Boolean serveCoreRecordSearch = true;
private Integer recordSearchTimeoutMillis = DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS;
private List<Class<?>> searchableEntityClasses;
private List<SearchableTableConfig> searchableTables;
@@ -387,6 +395,7 @@ public class QuickSearchQBitConfig implements QBitConfig
requirePositive(maxBulkRequestBytes, "maxBulkRequestBytes", errors);
requirePositive(basepullRepeatSeconds, "basepullRepeatSeconds", errors);
requirePositive(maxSearchLimit, "maxSearchLimit", errors);
requirePositive(recordSearchTimeoutMillis, "recordSearchTimeoutMillis", errors);
requirePositive(maxFieldLength, "maxFieldLength", errors);
requirePositive(runHistoryRetentionDays, "runHistoryRetentionDays", errors);
@@ -1897,6 +1906,72 @@ public class QuickSearchQBitConfig implements QBitConfig
/*******************************************************************************
** Getter for serveCoreRecordSearch: register QuickSearchRecordSearchProvider
** so core RecordSearchAction (POST /qqq/v1/search) searches indexed tables in
** OpenSearch, and give indexed tables core searchFields when they have none
*******************************************************************************/
public Boolean getServeCoreRecordSearch()
{
return (this.serveCoreRecordSearch);
}
/*******************************************************************************
** Setter for serveCoreRecordSearch
*******************************************************************************/
public void setServeCoreRecordSearch(Boolean serveCoreRecordSearch)
{
this.serveCoreRecordSearch = serveCoreRecordSearch;
}
/*******************************************************************************
** Fluent setter for serveCoreRecordSearch
*******************************************************************************/
public QuickSearchQBitConfig withServeCoreRecordSearch(Boolean serveCoreRecordSearch)
{
this.serveCoreRecordSearch = serveCoreRecordSearch;
return (this);
}
/*******************************************************************************
** Getter for recordSearchTimeoutMillis: wall-clock limit on the OpenSearch
** request behind a core record search; past it the provider fails and core
** falls back to its own search (default 2500)
*******************************************************************************/
public Integer getRecordSearchTimeoutMillis()
{
return (this.recordSearchTimeoutMillis);
}
/*******************************************************************************
** Setter for recordSearchTimeoutMillis
*******************************************************************************/
public void setRecordSearchTimeoutMillis(Integer recordSearchTimeoutMillis)
{
this.recordSearchTimeoutMillis = recordSearchTimeoutMillis;
}
/*******************************************************************************
** Fluent setter for recordSearchTimeoutMillis
*******************************************************************************/
public QuickSearchQBitConfig withRecordSearchTimeoutMillis(Integer recordSearchTimeoutMillis)
{
this.recordSearchTimeoutMillis = recordSearchTimeoutMillis;
return (this);
}
/*******************************************************************************
** Getter for searchableEntityClasses: entity classes annotated with @QuickSearchable
@@ -28,12 +28,16 @@ import java.util.Properties;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField;
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
@@ -50,7 +54,9 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
** processes, admin app), applies the config's table customizer and default
** backend, stamps sourceQBitName, then calls {@link #postProduceActions}, which
** discovers searchable tables, builds the runtime, registers the record-change
** listener and runtime service, and (in FAIL_FAST mode) prepares the index.
** listener and runtime service, registers the core record-search provider
** (serveCoreRecordSearch) and gives indexed tables core searchFields, and (in
** FAIL_FAST mode) prepares the index.
**
** Host usage, unchanged from 0.x:
** new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
@@ -254,6 +260,12 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
qInstance.withRuntimeService(new QCodeReference(QuickSearchRuntimeService.class));
if(Boolean.TRUE.equals(qBitConfig.getServeCoreRecordSearch()))
{
registerRecordSearchProvider(qInstance);
defaultCoreSearchFields(qInstance, discoveredTables);
}
if(!StringUtils.hasContent(qBitConfig.getSchedulerName()) && Boolean.TRUE.equals(qBitConfig.getEnableBasepullProcess()))
{
LOG.warn("Quick Search has no schedulerName; the basepull and reconcile processes are registered but will not run on a schedule");
@@ -270,6 +282,82 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
/*******************************************************************************
** Register QuickSearchRecordSearchProvider as the instance's core record
** search provider. An instance holds one provider; a different one the host
** already registered is kept (with a warning).
*******************************************************************************/
static void registerRecordSearchProvider(QInstance qInstance)
{
QCodeReference existing = qInstance.getRecordSearchProvider();
if(existing != null && !QuickSearchRecordSearchProvider.class.getName().equals(existing.getName()))
{
LOG.warn("The instance already has a record search provider; Quick Search does not replace it, so core search does not use OpenSearch",
logPair("provider", existing.getName()));
return;
}
qInstance.withRecordSearchProvider(new QCodeReference(QuickSearchRecordSearchProvider.class));
}
/*******************************************************************************
** Core record search only searches tables with searchFields, so give each
** indexed table that has none the searchable fields core accepts (visible
** string or integer fields; never password fields). Fields the host set are
** never replaced. A table that is not in the instance yet, uses backend
** variants, or has no such field is skipped with a warning.
*******************************************************************************/
static void defaultCoreSearchFields(QInstance qInstance, List<QuickSearchableTableConfig> tableConfigs)
{
for(QuickSearchableTableConfig tableConfig : tableConfigs)
{
QTableMetaData table = qInstance.getTable(tableConfig.getTableName());
if(table == null)
{
LOG.warn("Indexed table is not in the instance when Quick Search is produced, so it gets no core searchFields; set them on the table for core search to use it",
logPair("tableName", tableConfig.getTableName()));
continue;
}
if(table.getSearchFields() != null)
{
continue;
}
QBackendMetaData backend = qInstance.getBackend(table.getBackendName());
if(backend != null && Boolean.TRUE.equals(backend.getUsesVariants()))
{
LOG.warn("Indexed table uses backend variants, which core record search does not support; it gets no core searchFields", logPair("tableName", table.getName()));
continue;
}
List<String> searchFields = new ArrayList<>();
for(String fieldName : CollectionUtils.nonNullList(tableConfig.getSearchableFields()))
{
QFieldMetaData field = table.getFields() == null ? null : table.getFields().get(fieldName);
if(field != null && !Boolean.TRUE.equals(field.getIsHidden()) && field.getType() != null
&& (field.getType().isStringLike() || field.getType().isIntegral()) && !field.getType().needsMasked()
&& !searchFields.contains(fieldName))
{
searchFields.add(fieldName);
}
}
if(searchFields.isEmpty())
{
LOG.warn("Indexed table has no string or integer searchable field, so it gets no core searchFields; set them on the table for core search to use it",
logPair("tableName", table.getName()));
continue;
}
table.setSearchFields(searchFields);
}
}
/*******************************************************************************
** Discover searchable tables from @QuickSearchable entity classes. Fails when
** a configured field does not exist on a table that is in the instance.
@@ -455,22 +543,23 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
/*******************************************************************************
** Hidden fields are never indexed: their values would leak through
** searchable text and highlights.
** Hidden and password fields are never indexed: their values would leak
** through searchable text and highlights.
*******************************************************************************/
private static void removeHiddenFields(QTableMetaData table, List<String> searchableFields, Map<String, Integer> fieldWeights, Map<String, Boolean> fieldIncludeLabels)
{
List<String> hidden = new ArrayList<>();
for(String fieldName : searchableFields)
{
if(table.getFields().containsKey(fieldName) && Boolean.TRUE.equals(table.getField(fieldName).getIsHidden()))
QFieldMetaData field = table.getFields().get(fieldName);
if(field != null && (Boolean.TRUE.equals(field.getIsHidden()) || (field.getType() != null && field.getType().needsMasked())))
{
hidden.add(fieldName);
}
}
if(!hidden.isEmpty())
{
LOG.warn("Hidden fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden));
LOG.warn("Hidden and password fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden));
searchableFields.removeAll(hidden);
hidden.forEach(fieldWeights::remove);
hidden.forEach(fieldIncludeLabels::remove);
@@ -57,6 +57,8 @@ public class QuickSearchRuntime
private static final long ENABLED_CACHE_TTL_MILLIS = 60_000L;
public static final long RECORD_SEARCH_BACKOFF_MILLIS = 30_000L;
private final QuickSearchQBitConfig config;
private List<QuickSearchableTableConfig> discoveredTables = new ArrayList<>();
@@ -64,8 +66,9 @@ public class QuickSearchRuntime
private volatile QuickSearchOpenSearchClient client;
private volatile IndexEventPublisher publisher;
private volatile boolean indexReady = false;
private volatile boolean closed = false;
private volatile boolean indexReady = false;
private volatile boolean closed = false;
private volatile long recordSearchBackoffUntilMillis = 0L;
private final Map<String, CachedFlag> enabledCache = new ConcurrentHashMap<>();
@@ -302,6 +305,40 @@ public class QuickSearchRuntime
/*******************************************************************************
** Record that a core record search through OpenSearch failed. For the next
** RECORD_SEARCH_BACKOFF_MILLIS the record-search provider claims no tables,
** so core searches them itself at once instead of waiting for another
** timeout on every search while OpenSearch is down.
*******************************************************************************/
public void markRecordSearchFailed()
{
recordSearchBackoffUntilMillis = System.currentTimeMillis() + RECORD_SEARCH_BACKOFF_MILLIS;
}
/*******************************************************************************
** Whether core record search is backing off after a recent failure.
*******************************************************************************/
public boolean isRecordSearchBackingOff()
{
return (System.currentTimeMillis() < recordSearchBackoffUntilMillis);
}
/*******************************************************************************
** End a record-search backoff early (an operator or test knows OpenSearch is
** back).
*******************************************************************************/
public void clearRecordSearchBackoff()
{
recordSearchBackoffUntilMillis = 0L;
}
/*******************************************************************************
** Read the enabled flag from the quickSearchIndex row.
*******************************************************************************/
@@ -0,0 +1,153 @@
/*
* Copyright 2024 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.quicksearch.actions;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
/*******************************************************************************
** Serves QQQ core record search (RecordSearchAction, POST /qqq/v1/search,
** the Next UI global search) from the Quick Search index, for the tables
** this QBit indexes. Registered on the QInstance by QuickSearchQBitProducer
** when serveCoreRecordSearch is on (the default).
**
** Core creates a new instance for every search, so this class holds no
** state: it resolves the QBit's QuickSearchRuntime (client, tables, config)
** from QContext.
**
** A table is claimed when it is indexed by this QBit and enabled, and core
** record search is not backing off after a recent failure. search() sends
** one _msearch request with a top-N search per table and returns record ids
** only, best first; core re-reads them by primary key through QueryAction,
** so table permissions, record security locks and record labels are core's
** job, and ids of deleted records drop out. Each table gets
** IDS_PER_LIMIT_MULTIPLIER x limitPerTable ids (at most
** RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE) so dropouts do not leave
** the table short.
**
** The request is bounded by recordSearchTimeoutMillis. Any failure throws,
** and core then searches the claimed tables with its own LIKE search; the
** failure also starts a short backoff (QuickSearchRuntime
** RECORD_SEARCH_BACKOFF_MILLIS) during which no tables are claimed.
*******************************************************************************/
public class QuickSearchRecordSearchProvider implements RecordSearchProviderInterface
{
private static final QLogger LOG = QLogger.getLogger(QuickSearchRecordSearchProvider.class);
public static final int IDS_PER_LIMIT_MULTIPLIER = 4;
/*******************************************************************************
** Claim tables this QBit indexes and that are enabled, unless core record
** search is off or backing off. Cheap: an in-memory lookup plus the
** runtime's cached enabled flag.
*******************************************************************************/
@Override
public boolean claimsTable(QTableMetaData table)
{
QuickSearchRuntime runtime = QuickSearchRuntime.get();
if(runtime == null || table == null || !Boolean.TRUE.equals(runtime.getConfig().getServeCoreRecordSearch()))
{
return (false);
}
if(runtime.getTableConfig(table.getName()) == null || runtime.isRecordSearchBackingOff())
{
return (false);
}
return (runtime.isTableEnabled(table.getName()));
}
/*******************************************************************************
** Ids of the best matches in each requested table, in relevance order.
*******************************************************************************/
@Override
public RecordSearchOutput search(RecordSearchInput input) throws QException
{
QuickSearchRuntime runtime = QuickSearchRuntime.getOrThrow();
QuickSearchQBitConfig config = runtime.getConfig();
List<RecordSearchResult> results = new ArrayList<>();
String term = IndexingUtils.normalizeSearchText(input.getSearchTerm());
if(term.length() < QuickSearchAction.MIN_TERM_LENGTH)
{
return (new RecordSearchOutput().withResults(results));
}
List<String> tableNames = new ArrayList<>();
for(String tableName : new LinkedHashSet<>(CollectionUtils.nonNullList(input.getTableNames())))
{
if(runtime.getTableConfig(tableName) != null)
{
tableNames.add(tableName);
}
}
if(tableNames.isEmpty())
{
return (new RecordSearchOutput().withResults(results));
}
int limitPerTable = input.getLimitPerTable() == null || input.getLimitPerTable() <= 0 ? RecordSearchAction.DEFAULT_LIMIT_PER_TABLE : input.getLimitPerTable();
int idsPerTable = Math.min(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE, limitPerTable * IDS_PER_LIMIT_MULTIPLIER);
int timeoutMillis = config.getRecordSearchTimeoutMillis() == null ? QuickSearchQBitConfig.DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS : config.getRecordSearchTimeoutMillis();
Map<String, List<String>> idsByTable;
try
{
idsByTable = runtime.getClient().searchRecordIdsPerTable(term, tableNames, idsPerTable, runtime.getDiscoveredTables(), timeoutMillis);
}
catch(QException | RuntimeException e)
{
runtime.markRecordSearchFailed();
LOG.info("Quick Search record search failed; core searches these tables itself, and Quick Search claims none for a while", logPair("tables", tableNames),
logPair("backoffMillis", QuickSearchRuntime.RECORD_SEARCH_BACKOFF_MILLIS), logPair("error", e.getMessage()));
throw (e instanceof QException qe ? qe : new QException("Quick Search record search failed: " + e.getMessage(), e));
}
for(String tableName : tableNames)
{
for(String recordId : CollectionUtils.nonNullList(idsByTable.get(tableName)))
{
results.add(new RecordSearchResult().withTableName(tableName).withRecordId(recordId));
}
}
return (new RecordSearchOutput().withResults(results));
}
}
@@ -21,9 +21,14 @@ import java.io.Closeable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
@@ -32,6 +37,7 @@ import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
import org.opensearch.client.json.JsonData;
import org.opensearch.client.opensearch.OpenSearchAsyncClient;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.Conflicts;
import org.opensearch.client.opensearch._types.FieldValue;
@@ -51,12 +57,17 @@ import org.opensearch.client.opensearch.core.DeleteByQueryRequest;
import org.opensearch.client.opensearch.core.DeleteByQueryResponse;
import org.opensearch.client.opensearch.core.DeleteRequest;
import org.opensearch.client.opensearch.core.IndexRequest;
import org.opensearch.client.opensearch.core.MsearchRequest;
import org.opensearch.client.opensearch.core.MsearchResponse;
import org.opensearch.client.opensearch.core.SearchRequest;
import org.opensearch.client.opensearch.core.SearchResponse;
import org.opensearch.client.opensearch.core.bulk.BulkOperation;
import org.opensearch.client.opensearch.core.bulk.BulkResponseItem;
import org.opensearch.client.opensearch.core.bulk.DeleteOperation;
import org.opensearch.client.opensearch.core.bulk.IndexOperation;
import org.opensearch.client.opensearch.core.msearch.MultiSearchResponseItem;
import org.opensearch.client.opensearch.core.msearch.RequestItem;
import org.opensearch.client.opensearch.core.search.Hit;
import org.opensearch.client.opensearch.core.search.Highlight;
import org.opensearch.client.opensearch.core.search.HighlightField;
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
@@ -92,11 +103,12 @@ public class QuickSearchOpenSearchClient implements Closeable
public static final String SEARCH_ANALYZER = "quick_search_search_analyzer";
public static final int MAX_RESULT_WINDOW = 10_000;
private final OpenSearchClient client;
private final OpenSearchTransport transport;
private final String indexName;
private final int maxBulkRequestBytes;
private final ObjectMapper sizingMapper;
private final OpenSearchClient client;
private final OpenSearchAsyncClient asyncClient;
private final OpenSearchTransport transport;
private final String indexName;
private final int maxBulkRequestBytes;
private final ObjectMapper sizingMapper;
private volatile Boolean mappingOutdated = false;
private Boolean closed = false;
@@ -110,6 +122,7 @@ public class QuickSearchOpenSearchClient implements Closeable
{
this.transport = OpenSearchTransportFactory.build(config);
this.client = new OpenSearchClient(transport);
this.asyncClient = new OpenSearchAsyncClient(transport);
this.indexName = config.getOpensearchIndexName();
this.maxBulkRequestBytes = config.getMaxBulkRequestBytes() == null ? 5 * 1024 * 1024 : config.getMaxBulkRequestBytes();
this.sizingMapper = new ObjectMapper().registerModule(new JavaTimeModule()).disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
@@ -652,14 +665,7 @@ public class QuickSearchOpenSearchClient implements Closeable
{
try
{
List<FieldValue> allowed = allowedTables.stream().map(FieldValue::of).toList();
BoolQuery.Builder boolBuilder = new BoolQuery.Builder()
.must(Query.of(q -> q.multiMatch(MultiMatchQuery.of(mm -> mm.query(searchTerm).fields("searchableText").operator(Operator.And)))))
.filter(Query.of(q -> q.terms(t -> t.field("sourceTable").terms(tv -> tv.value(allowed)))))
.should(buildFieldBoostQueries(searchTerm, tableConfigs, allowedTables));
Query finalQuery = Query.of(q -> q.bool(boolBuilder.build()));
Query finalQuery = buildSearchQuery(searchTerm, allowedTables, tableConfigs);
Highlight highlight = Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf)));
SearchRequest request = SearchRequest.of(r -> r
@@ -681,6 +687,117 @@ public class QuickSearchOpenSearchClient implements Closeable
/*******************************************************************************
** Record ids of the best matches in each table, best first, for core record
** search. One _msearch request with one search per table (the same query and
** ranking as {@link #search}, restricted to that table), so every table gets
** its own top idsPerTable hits. Only sourceTable and recordId are fetched.
**
** The whole request is bounded by timeoutMillis of wall-clock time (and the
** same server-side search timeout): when it runs out, the request is
** cancelled and a QException is thrown. A failed search for any table fails
** the call. Returns every requested table as a key, in the given order.
*******************************************************************************/
public Map<String, List<String>> searchRecordIdsPerTable(String searchTerm, List<String> tableNames, int idsPerTable, List<QuickSearchableTableConfig> tableConfigs, int timeoutMillis) throws QException
{
Map<String, List<String>> idsByTable = new LinkedHashMap<>();
if(tableNames == null || tableNames.isEmpty())
{
return (idsByTable);
}
List<RequestItem> searches = new ArrayList<>();
for(String tableName : tableNames)
{
Query query = buildSearchQuery(searchTerm, List.of(tableName), tableConfigs);
searches.add(RequestItem.of(ri -> ri
.header(h -> h.index(indexName))
.body(b -> b
.query(query)
.size(idsPerTable)
.trackTotalHits(t -> t.enabled(false))
.source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId")))
.timeout(timeoutMillis + "ms"))));
}
CompletableFuture<MsearchResponse<OpenSearchDocument>> future = null;
try
{
future = asyncClient.msearch(MsearchRequest.of(r -> r.searches(searches)), OpenSearchDocument.class);
MsearchResponse<OpenSearchDocument> response = future.get(timeoutMillis, TimeUnit.MILLISECONDS);
List<MultiSearchResponseItem<OpenSearchDocument>> items = response.responses();
if(items == null || items.size() != tableNames.size())
{
throw (new QException("Record search failed: OpenSearch returned " + (items == null ? 0 : items.size()) + " responses for " + tableNames.size() + " searches"));
}
for(int i = 0; i < tableNames.size(); i++)
{
MultiSearchResponseItem<OpenSearchDocument> item = items.get(i);
if(item.isFailure())
{
String reason = item.failure() != null && item.failure().error() != null ? item.failure().error().reason() : "unknown error";
throw (new QException("Record search failed for table [" + tableNames.get(i) + "]: " + reason));
}
List<String> ids = new ArrayList<>();
for(Hit<OpenSearchDocument> hit : item.result().hits().hits())
{
if(hit.source() != null && hit.source().getRecordId() != null)
{
ids.add(hit.source().getRecordId());
}
}
idsByTable.put(tableNames.get(i), ids);
}
return (idsByTable);
}
catch(TimeoutException e)
{
future.cancel(true);
throw (new QException("Record search timed out after " + timeoutMillis + " ms", e));
}
catch(InterruptedException e)
{
if(future != null)
{
future.cancel(true);
}
Thread.currentThread().interrupt();
throw (new QException("Record search was interrupted", e));
}
catch(ExecutionException e)
{
throw wrap("Record search failed", e.getCause() instanceof Exception cause ? cause : e);
}
catch(OpenSearchException | java.io.IOException e)
{
throw wrap("Record search failed", e);
}
}
/*******************************************************************************
** The search query: every term of the query (operator AND) must match
** searchableText, only the allowed tables match (an empty set matches
** nothing), and per-field should clauses boost by configured weight.
*******************************************************************************/
private Query buildSearchQuery(String searchTerm, Collection<String> allowedTables, List<QuickSearchableTableConfig> tableConfigs)
{
List<FieldValue> allowed = allowedTables.stream().map(FieldValue::of).toList();
BoolQuery.Builder boolBuilder = new BoolQuery.Builder()
.must(Query.of(q -> q.multiMatch(MultiMatchQuery.of(mm -> mm.query(searchTerm).fields("searchableText").operator(Operator.And)))))
.filter(Query.of(q -> q.terms(t -> t.field("sourceTable").terms(tv -> tv.value(allowed)))))
.should(buildFieldBoostQueries(searchTerm, tableConfigs, allowedTables));
return (Query.of(q -> q.bool(boolBuilder.build())));
}
/*******************************************************************************
**
*******************************************************************************/
@@ -276,6 +276,7 @@ class QuickSearchQBitConfigValidationTest
.withResponseTimeoutMillis(-1)
.withMaxConnections(0)
.withMaxSearchLimit(0)
.withRecordSearchTimeoutMillis(0)
.withBasepullRepeatSeconds(0)
.withBasepullOverlapSeconds(-5)
.withRunHistoryRetentionDays(0)
@@ -284,6 +285,7 @@ class QuickSearchQBitConfigValidationTest
assertThat(errors).anyMatch(e -> e.startsWith("responseTimeoutMillis"));
assertThat(errors).anyMatch(e -> e.startsWith("maxConnections"));
assertThat(errors).anyMatch(e -> e.startsWith("maxSearchLimit"));
assertThat(errors).anyMatch(e -> e.startsWith("recordSearchTimeoutMillis"));
assertThat(errors).anyMatch(e -> e.startsWith("basepullRepeatSeconds"));
assertThat(errors).anyMatch(e -> e.startsWith("basepullOverlapSeconds"));
assertThat(errors).anyMatch(e -> e.startsWith("runHistoryRetentionDays"));
@@ -16,10 +16,14 @@
package com.kingsrook.qbits.quicksearch;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -35,6 +39,7 @@ import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.simple.SimpleSchedulerMetaData;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
@@ -99,6 +104,43 @@ class QuickSearchQBitProducerTest
/***************************************************************************
** Test entity with fields core search must not get: hidden, password and
** non-text, plus an integer field it may get.
***************************************************************************/
@QuickSearchable(tableName = "account")
static class AccountEntity
{
@QuickSearchField
private String name;
@QuickSearchField
private Integer accountNumber;
@QuickSearchField
private String secretNote;
@QuickSearchField
private String password;
@QuickSearchField
private Instant openedAt;
}
/***************************************************************************
** Test entity whose only searchable field is a date.
***************************************************************************/
@QuickSearchable(tableName = "event")
static class EventEntity
{
@QuickSearchField
private Instant occurredAt;
}
@QuickSearchable(tableName = "person")
static class BadFieldEntity
{
@@ -289,6 +331,141 @@ class QuickSearchQBitProducerTest
/***************************************************************************
** By default the producer registers the core record-search provider and
** gives indexed tables core searchFields from their searchable fields.
***************************************************************************/
@Test
void testProduce_registersRecordSearchProviderAndSearchFields() throws QException
{
QInstance qInstance = buildQInstance();
QContext.init(qInstance, new QSession());
new QuickSearchQBitProducer().withConfig(buildValidConfig()).produce(qInstance);
assertThat(qInstance.getRecordSearchProvider()).isNotNull();
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("firstName", "lastName", "email");
}
/***************************************************************************
** serveCoreRecordSearch false leaves core search alone.
***************************************************************************/
@Test
void testProduce_serveCoreRecordSearchOff_noProviderNoSearchFields() throws QException
{
QInstance qInstance = buildQInstance();
QContext.init(qInstance, new QSession());
new QuickSearchQBitProducer().withConfig(buildValidConfig().withServeCoreRecordSearch(false)).produce(qInstance);
assertThat(qInstance.getRecordSearchProvider()).isNull();
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).isNull();
}
/***************************************************************************
** searchFields the host set are kept, and a provider the host registered
** is not replaced.
***************************************************************************/
@Test
void testProduce_keepsHostSearchFieldsAndProvider() throws QException
{
QInstance qInstance = buildQInstance();
qInstance.getTable(SOURCE_TABLE).withSearchFields("email");
QCodeReference hostProvider = new QCodeReference(HostRecordSearchProvider.class);
qInstance.withRecordSearchProvider(hostProvider);
QContext.init(qInstance, new QSession());
new QuickSearchQBitProducer().withConfig(buildValidConfig()).produce(qInstance);
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("email");
assertThat(qInstance.getRecordSearchProvider()).isSameAs(hostProvider);
}
/***************************************************************************
** Producing twice (same provider already registered) keeps one provider.
***************************************************************************/
@Test
void testRegisterRecordSearchProvider_idempotent()
{
QInstance qInstance = new QInstance();
QuickSearchQBitProducer.registerRecordSearchProvider(qInstance);
QuickSearchQBitProducer.registerRecordSearchProvider(qInstance);
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
}
/***************************************************************************
** Hidden and password fields are never indexed; core searchFields only get
** visible string or integer fields; a table with none gets no searchFields.
***************************************************************************/
@Test
void testProduce_searchFieldsAndIndexing_skipHiddenPasswordAndNonText() throws QException
{
QInstance qInstance = buildQInstance();
qInstance.addTable(new QTableMetaData()
.withName("account")
.withBackendName(BACKEND_NAME)
.withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withField(new QFieldMetaData("accountNumber", QFieldType.INTEGER))
.withField(new QFieldMetaData("secretNote", QFieldType.STRING).withIsHidden(true))
.withField(new QFieldMetaData("password", QFieldType.PASSWORD))
.withField(new QFieldMetaData("openedAt", QFieldType.DATE_TIME))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
qInstance.addTable(new QTableMetaData()
.withName("event")
.withBackendName(BACKEND_NAME)
.withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("occurredAt", QFieldType.DATE_TIME))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
QContext.init(qInstance, new QSession());
QuickSearchQBitConfig config = buildValidConfig().withSearchableEntityClasses(List.of(AccountEntity.class, EventEntity.class));
new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
QuickSearchableTableConfig account = config.getRuntime().getTableConfig("account");
assertThat(account.getSearchableFields()).containsExactly("name", "accountNumber", "openedAt");
assertThat(qInstance.getTable("account").getSearchFields()).containsExactly("name", "accountNumber");
assertThat(qInstance.getTable("event").getSearchFields()).isNull();
}
/***************************************************************************
** A config-driven table that is not in the instance yet, or whose backend
** uses variants, gets no searchFields (and production does not fail).
***************************************************************************/
@Test
void testDefaultCoreSearchFields_missingTableAndVariantBackend()
{
QInstance qInstance = buildQInstance();
qInstance.addBackend(new QBackendMetaData().withName("variants").withBackendType(MemoryBackendModule.class).withUsesVariants(true));
qInstance.addTable(new QTableMetaData()
.withName("variantTable")
.withBackendName("variants")
.withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("name", QFieldType.STRING)));
QuickSearchQBitProducer.defaultCoreSearchFields(qInstance, List.of(
new QuickSearchableTableConfig().withTableName("notThere").withSearchableFields(List.of("name")),
new QuickSearchableTableConfig().withTableName("variantTable").withSearchableFields(List.of("name")),
new QuickSearchableTableConfig().withTableName(SOURCE_TABLE).withSearchableFields(List.of("lastName", "nope", "lastName"))));
assertThat(qInstance.getTable("variantTable").getSearchFields()).isNull();
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("lastName");
}
@Test
void testProduce_realTimeIndexingDisabled_noListener() throws QException
{
@@ -914,4 +1091,26 @@ class QuickSearchQBitProducerTest
.withSearchableEntityClasses(List.of(PersonEntity.class));
}
/***************************************************************************
** A record search provider a host registered itself.
***************************************************************************/
public static class HostRecordSearchProvider implements RecordSearchProviderInterface
{
@Override
public boolean claimsTable(QTableMetaData table)
{
return (false);
}
@Override
public RecordSearchOutput search(RecordSearchInput input)
{
return (new RecordSearchOutput());
}
}
}
@@ -0,0 +1,281 @@
/*
* Copyright 2024 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.quicksearch;
import java.io.Serializable;
import java.time.Instant;
import java.util.List;
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
import com.kingsrook.qbits.quicksearch.actions.QuickSearchResult;
import com.kingsrook.qbits.quicksearch.processes.FullReindexStep;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.extension.ExtendWith;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** End-to-end test of core record search (RecordSearchAction, the code behind
** POST /qqq/v1/search) served by QuickSearchRecordSearchProvider from a real
** OpenSearch (Testcontainers): the producer registers the provider and
** searchFields, hits come back in Quick Search's relevance order and are
** re-read by primary key (stale ids drop out, limitPerTable applies),
** unindexed tables keep core's own search, and with OpenSearch stopped core
** falls back to its own search without failing.
*******************************************************************************/
@ExtendWith(RequiresDockerCondition.class)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class RecordSearchProviderIntegrationTest
{
private static final String BACKEND_NAME = "testMemoryBackend";
private static final String PRODUCT_TABLE = "rsProduct";
private static final String NOTE_TABLE = "rsNote";
private static final String INDEX_NAME = "quick_search_record_search_it";
@Container
static GenericContainer<?> opensearch = new GenericContainer<>(System.getProperty("opensearch.test.image", "opensearchproject/opensearch:2.19.6"))
.withExposedPorts(9200)
.withEnv("discovery.type", "single-node")
.withEnv("DISABLE_SECURITY_PLUGIN", "true")
.withEnv("DISABLE_INSTALL_DEMO_CONFIG", "true")
.waitingFor(Wait.forHttp("/_cluster/health").forStatusCode(200));
private static QuickSearchQBitConfig config;
/*******************************************************************************
** Build the instance (an indexed product table and an unindexed note
** table with its own searchFields), produce the QBit, index the products.
*******************************************************************************/
@BeforeAll
static void setUpAll() throws Exception
{
MemoryRecordStore.getInstance().reset();
QInstance qInstance = new QInstance();
qInstance.addBackend(new QBackendMetaData().withName(BACKEND_NAME).withBackendType(MemoryBackendModule.class));
qInstance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
qInstance.addTable(new QTableMetaData()
.withName(PRODUCT_TABLE)
.withLabel("Product")
.withBackendName(BACKEND_NAME)
.withPrimaryKeyField("id")
.withRecordLabelFormat("%s")
.withRecordLabelFields("name")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withField(new QFieldMetaData("description", QFieldType.STRING))
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
qInstance.addTable(new QTableMetaData()
.withName(NOTE_TABLE)
.withBackendName(BACKEND_NAME)
.withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("text", QFieldType.STRING))
.withSearchFields("text"));
config = new QuickSearchQBitConfig()
.withBackendName(BACKEND_NAME)
.withOpensearchHost(opensearch.getHost())
.withOpensearchPort(opensearch.getMappedPort(9200))
.withOpensearchIndexName(INDEX_NAME)
.withSearchableTable(new SearchableTableConfig(PRODUCT_TABLE, List.of(
new SearchableFieldConfig("name").withWeight(3),
new SearchableFieldConfig("description"))))
.withEnableRealTimeIndexing(false);
new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
QContext.init(qInstance, new QSession());
new InsertAction().execute(new InsertInput(PRODUCT_TABLE).withRecords(List.of(
new QRecord().withValue("id", 1).withValue("name", "Blue Widget").withValue("description", "the original").withValue("modifyDate", Instant.now()),
new QRecord().withValue("id", 2).withValue("name", "Widget").withValue("description", "comes in blue and green").withValue("modifyDate", Instant.now()),
new QRecord().withValue("id", 3).withValue("name", "Red Widget").withValue("description", "not that colour").withValue("modifyDate", Instant.now()),
new QRecord().withValue("id", 4).withValue("name", "Gadget").withValue("description", "no match here").withValue("modifyDate", Instant.now()))));
new InsertAction().execute(new InsertInput(NOTE_TABLE).withRecords(List.of(
new QRecord().withValue("id", 1).withValue("text", "a blue widget note"))));
new FullReindexStep().run(new RunBackendStepInput(), new RunBackendStepOutput());
config.getRuntime().getClient().refreshIndex();
}
/*******************************************************************************
**
*******************************************************************************/
@AfterAll
static void tearDownAll()
{
if(config != null && config.getRuntime() != null)
{
config.getRuntime().close();
}
QContext.clear();
QuickSearchQBitContext.clear();
MemoryRecordStore.getInstance().reset();
}
/*******************************************************************************
**
*******************************************************************************/
private static RecordSearchOutput coreSearch(String term, Integer limitPerTable) throws Exception
{
return (new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm(term).withLimitPerTable(limitPerTable)));
}
/*******************************************************************************
**
*******************************************************************************/
private static List<Serializable> idsFor(RecordSearchOutput output, String tableName)
{
return (output.getResults().stream().filter(r -> tableName.equals(r.getTableName())).map(RecordSearchResult::getRecordId).toList());
}
@Test
@Order(1)
void testProducerRegisteredProviderAndSearchFields()
{
QInstance qInstance = QContext.getQInstance();
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
assertThat(qInstance.getTable(PRODUCT_TABLE).getSearchFields()).containsExactly("name", "description");
assertThat(qInstance.getTable(NOTE_TABLE).getSearchFields()).containsExactly("text");
}
/*******************************************************************************
** "blue widget" matches products 1 and 2 in OpenSearch (all terms, any
** field); core's contains-search would only find product 1. The order is
** Quick Search's relevance order; labels come from core's re-read.
*******************************************************************************/
@Test
@Order(2)
void testCoreSearchServedFromOpenSearch() throws Exception
{
RecordSearchOutput output = coreSearch("blue widget", 5);
List<String> quickSearchOrder = new QuickSearchAction().execute(new QuickSearchInput().withSearchTerm("blue widget").withTableNames(List.of(PRODUCT_TABLE)))
.getResults().stream().map(QuickSearchResult::getRecordId).toList();
assertThat(quickSearchOrder).containsExactlyInAnyOrder("1", "2");
assertThat(idsFor(output, PRODUCT_TABLE)).containsExactlyElementsOf(quickSearchOrder.stream().map(Integer::valueOf).toList());
assertThat(output.getResults()).filteredOn(r -> PRODUCT_TABLE.equals(r.getTableName())).extracting(RecordSearchResult::getRecordLabel).contains("Blue Widget", "Widget");
assertThat(output.getResults()).filteredOn(r -> PRODUCT_TABLE.equals(r.getTableName())).extracting(RecordSearchResult::getTableLabel).containsOnly("Product");
////////////////////////////////////////////////
// the unindexed table keeps core's own search //
////////////////////////////////////////////////
assertThat(idsFor(output, NOTE_TABLE)).containsExactly(1);
assertThat(config.getRuntime().isRecordSearchBackingOff()).isFalse();
}
@Test
@Order(3)
void testLimitPerTable() throws Exception
{
assertThat(idsFor(coreSearch("widget", 1), PRODUCT_TABLE)).hasSize(1);
assertThat(idsFor(coreSearch("widget", 5), PRODUCT_TABLE)).containsExactlyInAnyOrder(1, 2, 3);
}
/*******************************************************************************
** A record deleted from the table but still in the index is dropped by
** core's re-read.
*******************************************************************************/
@Test
@Order(4)
void testStaleIndexHitDropped() throws Exception
{
new DeleteAction().execute(new DeleteInput(PRODUCT_TABLE).withPrimaryKeys(List.of(1)));
assertThat(config.getRuntime().getClient().countDocumentsForTable(PRODUCT_TABLE)).as("the deleted record is still indexed").isEqualTo(4L);
assertThat(idsFor(coreSearch("blue widget", 5), PRODUCT_TABLE)).containsExactly(2);
}
/*******************************************************************************
** With OpenSearch stopped, core search still answers, from core's own
** contains-search over the searchFields Quick Search set, and the
** provider backs off.
*******************************************************************************/
@Test
@Order(5)
void testFallbackWhenOpenSearchStopped() throws Exception
{
opensearch.stop();
long start = System.currentTimeMillis();
RecordSearchOutput output = coreSearch("widget", 5);
assertThat(System.currentTimeMillis() - start).isLessThan(15_000);
assertThat(idsFor(output, PRODUCT_TABLE)).containsExactly(2, 3);
assertThat(config.getRuntime().isRecordSearchBackingOff()).isTrue();
assertThat(idsFor(coreSearch("blue", 5), PRODUCT_TABLE)).containsExactly(2);
}
}
@@ -0,0 +1,360 @@
/*
* Copyright 2024 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.quicksearch.actions;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest;
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.assertj.core.api.Assertions.tuple;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
/*******************************************************************************
** Unit tests for QuickSearchRecordSearchProvider: claim logic, result
** mapping, order and per-table id cap, failure and timeout handling, and
** core RecordSearchAction end to end (re-read by primary key, fallback to
** core's own search) over the memory backend with a mock OpenSearch client.
*******************************************************************************/
class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
{
private static final String OTHER_TABLE = "otherEntity";
private static final String UNINDEXED_TABLE = "unindexedEntity";
private QuickSearchQBitConfig config;
private QuickSearchRuntime runtime;
private QuickSearchOpenSearchClient mockClient;
private QuickSearchRecordSearchProvider provider;
/*******************************************************************************
** Register a Quick Search QBit whose runtime returns a mock client, with
** testEntity and otherEntity indexed; add the provider to the instance.
*******************************************************************************/
@BeforeEach
void setUp() throws QException
{
QInstance qInstance = QContext.getQInstance();
qInstance.addTable(simpleTable(OTHER_TABLE));
qInstance.addTable(simpleTable(UNINDEXED_TABLE));
qInstance.getTable(TEST_ENTITY_TABLE).withSearchFields("name", "description");
qInstance.withRecordSearchProvider(new QCodeReference(QuickSearchRecordSearchProvider.class));
config = new QuickSearchQBitConfig()
.withBackendName(TEST_BACKEND_NAME)
.withOpensearchHost("localhost")
.withOpensearchPort(9200)
.withOpensearchIndexName("test-index")
.withRecordSearchTimeoutMillis(1234);
mockClient = mock(QuickSearchOpenSearchClient.class);
runtime = registerRuntime(config, mockClient);
provider = new QuickSearchRecordSearchProvider();
}
/*******************************************************************************
** Register the QBit with a runtime (a spy whose client is the given one,
** or the real lazily built client when null).
*******************************************************************************/
private QuickSearchRuntime registerRuntime(QuickSearchQBitConfig config, QuickSearchOpenSearchClient client) throws QException
{
QuickSearchRuntime runtime = new QuickSearchRuntime(config);
if(client != null)
{
runtime = spy(runtime);
doReturn(client).when(runtime).getClient();
}
runtime.setDiscoveredTables(List.of(
new QuickSearchableTableConfig().withTableName(TEST_ENTITY_TABLE).withPrimaryKeyField("id").withSearchableFields(List.of("name", "description")),
new QuickSearchableTableConfig().withTableName(OTHER_TABLE).withPrimaryKeyField("id").withSearchableFields(List.of("name"))));
config.setRuntime(runtime);
QInstance qInstance = QContext.getQInstance();
if(qInstance.getQBits() != null)
{
qInstance.getQBits().clear();
}
qInstance.addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config));
return (runtime);
}
/*******************************************************************************
**
*******************************************************************************/
private static QTableMetaData simpleTable(String name)
{
return (new QTableMetaData()
.withName(name)
.withBackendName(TEST_BACKEND_NAME)
.withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withSearchFields("name"));
}
/*******************************************************************************
** Insert testEntity rows with ids 1..n named "<prefix> <i>".
*******************************************************************************/
private void insertEntities(String prefix, int count) throws QException
{
List<QRecord> records = new ArrayList<>();
for(int i = 1; i <= count; i++)
{
records.add(new QRecord().withValue("id", i).withValue("name", prefix + " " + i).withValue("description", "row " + i));
}
new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecords(records));
}
@Test
void testClaimsTable_indexedEnabledTablesOnly()
{
QInstance qInstance = QContext.getQInstance();
assertThat(provider.claimsTable(qInstance.getTable(TEST_ENTITY_TABLE))).isTrue();
assertThat(provider.claimsTable(qInstance.getTable(OTHER_TABLE))).isTrue();
assertThat(provider.claimsTable(qInstance.getTable(UNINDEXED_TABLE))).isFalse();
assertThat(provider.claimsTable(null)).isFalse();
}
@Test
void testClaimsTable_disabledTable_notClaimed() throws QException
{
new InsertAction().execute(new InsertInput(config.getQuickSearchIndexTableName()).withRecords(List.of(
new QRecord().withValue("tableName", TEST_ENTITY_TABLE).withValue("enabled", false))));
runtime.invalidateEnabledCache();
assertThat(provider.claimsTable(QContext.getQInstance().getTable(TEST_ENTITY_TABLE))).isFalse();
assertThat(provider.claimsTable(QContext.getQInstance().getTable(OTHER_TABLE))).isTrue();
}
@Test
void testClaimsTable_serveCoreRecordSearchOff_orBackingOff_orNoRuntime()
{
QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
config.setServeCoreRecordSearch(false);
assertThat(provider.claimsTable(table)).isFalse();
config.setServeCoreRecordSearch(true);
runtime.markRecordSearchFailed();
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
assertThat(provider.claimsTable(table)).isFalse();
runtime.clearRecordSearchBackoff();
assertThat(provider.claimsTable(table)).isTrue();
QContext.getQInstance().getQBits().clear();
assertThat(provider.claimsTable(table)).isFalse();
}
@Test
void testSearch_mapsIdsPerTableInOrder_andCapsIdsPerTable() throws QException
{
Map<String, List<String>> ids = new LinkedHashMap<>();
ids.put(OTHER_TABLE, List.of("7"));
ids.put(TEST_ENTITY_TABLE, List.of("3", "1", "2"));
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
RecordSearchOutput output = provider.search(new RecordSearchInput()
.withSearchTerm(" widget blue ")
.withTableNames(List.of(TEST_ENTITY_TABLE, UNINDEXED_TABLE, OTHER_TABLE, TEST_ENTITY_TABLE))
.withLimitPerTable(5));
assertThat(output.getResults()).extracting(RecordSearchResult::getTableName, RecordSearchResult::getRecordId).containsExactly(
tuple(TEST_ENTITY_TABLE, "3"),
tuple(TEST_ENTITY_TABLE, "1"),
tuple(TEST_ENTITY_TABLE, "2"),
tuple(OTHER_TABLE, "7"));
verify(mockClient).searchRecordIdsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(25));
verify(mockClient).searchRecordIdsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), anyList(), eq(1234));
provider.search(new RecordSearchInput().withSearchTerm("gadget").withTableNames(List.of(TEST_ENTITY_TABLE)));
verify(mockClient).searchRecordIdsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
}
@Test
void testSearch_shortTermOrNoIndexedTables_returnsEmptyWithoutOpenSearch() throws QException
{
assertThat(provider.search(new RecordSearchInput().withSearchTerm("x").withTableNames(List.of(TEST_ENTITY_TABLE))).getResults()).isEmpty();
assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(UNINDEXED_TABLE))).getResults()).isEmpty();
assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget")).getResults()).isEmpty();
verify(mockClient, never()).searchRecordIdsPerTable(any(), any(), anyInt(), any(), anyInt());
}
@Test
void testSearch_clientFailure_throwsAndBacksOff() throws QException
{
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("boom"));
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
.isInstanceOf(QException.class).hasMessageContaining("boom");
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
}
@Test
void testSearch_uncheckedClientFailure_wrappedAsQException() throws QException
{
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed"));
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
.isInstanceOf(QException.class).hasMessageContaining("closed");
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
}
/*******************************************************************************
** A server that accepts connections and never answers (a hung OpenSearch):
** the real client gives up after recordSearchTimeoutMillis with a
** QException, well before the transport's response timeout.
*******************************************************************************/
@Test
void testSearch_hungOpenSearch_timesOutWithQException() throws Exception
{
try(ServerSocket hung = new ServerSocket(0, 50, InetAddress.getLoopbackAddress()))
{
QuickSearchQBitConfig hungConfig = new QuickSearchQBitConfig()
.withBackendName(TEST_BACKEND_NAME)
.withOpensearchHost("127.0.0.1")
.withOpensearchPort(hung.getLocalPort())
.withOpensearchIndexName("test-index")
.withResponseTimeoutMillis(60_000)
.withRecordSearchTimeoutMillis(300);
QuickSearchRuntime hungRuntime = registerRuntime(hungConfig, null);
long start = System.currentTimeMillis();
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
.isInstanceOf(QException.class).hasMessageContaining("timed out after 300 ms");
assertThat(System.currentTimeMillis() - start).isLessThan(10_000);
assertThat(hungRuntime.isRecordSearchBackingOff()).isTrue();
assertThat(provider.claimsTable(QContext.getQInstance().getTable(TEST_ENTITY_TABLE))).isFalse();
hungRuntime.close();
}
}
/*******************************************************************************
** Core RecordSearchAction re-reads the provider's ids by primary key, keeps
** its order, drops ids that no longer exist, applies limitPerTable, and
** fills record labels; unclaimed tables keep core's own search.
*******************************************************************************/
@Test
void testCoreRecordSearchAction_usesProviderIds() throws QException
{
insertEntities("Widget", 6);
new InsertAction().execute(new InsertInput(UNINDEXED_TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("name", "Widget elsewhere"))));
Map<String, List<String>> ids = new LinkedHashMap<>();
ids.put(TEST_ENTITY_TABLE, List.of("5", "999", "2", "4"));
ids.put(OTHER_TABLE, List.of());
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withLimitPerTable(2));
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(5, 2);
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(UNINDEXED_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(1);
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(OTHER_TABLE)).isEmpty();
}
/*******************************************************************************
** When the provider fails, core searches the claimed tables itself.
*******************************************************************************/
@Test
void testCoreRecordSearchAction_providerFailure_fallsBackToCoreSearch() throws QException
{
insertEntities("Widget", 3);
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down"));
RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId).containsExactly(1, 2, 3);
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
//////////////////////////////////////////////////////////////////////
// while backing off, the provider claims nothing: OpenSearch is not //
// called again and core still answers //
//////////////////////////////////////////////////////////////////////
RecordSearchOutput again = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
assertThat(again.getResults()).hasSize(3);
verify(mockClient, times(1)).searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt());
}
}