feat: serve QQQ /qqq/v1/search from OpenSearch (record-search provider)
Add QuickSearchRecordSearchProvider, an implementation of QQQ core's RecordSearchProviderInterface (QRun-IO/qqq#1042), so RecordSearchAction and POST /qqq/v1/search (the Next UI global search) are served from the Quick Search index for the tables this QBit indexes (audit F-11, D1). - Claims indexed, enabled tables; one _msearch request with a per-table top-N search (same query, AND semantics and field weights), returning record ids only, which core re-reads through QueryAction. - recordSearchTimeoutMillis (default 2500) bounds the request; any failure makes core fall back to its own search and starts a 30 s backoff during which no tables are claimed. - serveCoreRecordSearch (default true) registers the provider and gives indexed tables core searchFields when they have none; host searchFields and a host-registered provider are kept. - PASSWORD fields are no longer indexed (like hidden fields). - Build against qqq 4.1.0-SNAPSHOT until the SPI is released; version 1.1.0-SNAPSHOT; README, MIGRATION-1.1, CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2b627e767a
commit
75dc2b3b3d
16 files changed
+1416
-31
No files matched your search
@@ -15,7 +15,7 @@ This file provides guidance to Codex when working with code in this repository.
|
||||
./mvnw package # Build JAR
|
||||
CI=true ./mvnw verify # Full verify: integration tests (Docker) and the JaCoCo gate
|
||||
./mvnw verify -Dopensearch.test.image=opensearchproject/opensearch:3.9.0 # other server line
|
||||
./mvnw verify -Dqqq.snapshot.version=4.1.0-SNAPSHOT # other qqq 4.1 build (profile is active by default)
|
||||
./mvnw verify -Dqqq.snapshot.version=4.1.0-RC.1 # other qqq 4.1 build (profile is active by default)
|
||||
```
|
||||
|
||||
Integration tests (require Docker) run via `mvn verify` using maven-failsafe-plugin.
|
||||
@@ -34,6 +34,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
|
||||
- `listeners/QuickSearchRecordChangeListener` - Real-time path: builds events in the write's transaction, publishes after commit, records failures in `quickSearchFailedEvent`.
|
||||
- `processes/AbstractIndexingStep` - Keyset paging by primary key with display values, run records, drift detection, run purge. `BasepullIndexStep` (replays failed events, start-anchored watermark with overlap), `ReconcileIndexStep` (in place, orphan sweep), `FullReindexStep` (all tables: new physical index plus alias swap; one table: reconcile algorithm).
|
||||
- `actions/QuickSearchAction` - Permission-aware search (table READ filter, `QueryAction` post-filter), bounds, `tableNames`, `limitPerTable`, `tableLabel`.
|
||||
- `actions/QuickSearchRecordSearchProvider` - QQQ core `RecordSearchProviderInterface` (`POST /qqq/v1/search`): claims indexed, enabled tables; one `_msearch` per call returning ordered record ids per table; bounded by `recordSearchTimeoutMillis`; failures make core fall back and start a 30 s backoff (`QuickSearchRuntime.markRecordSearchFailed`). Registered, with default table `searchFields`, when `serveCoreRecordSearch` is true.
|
||||
- `metadata/*` - Component producers and the table/process helpers.
|
||||
|
||||
**Tables Produced:** `quickSearchIndex`, `quickSearchIndexRun`, `quickSearchFailedEvent` (plus a PVS over `quickSearchIndex`).
|
||||
@@ -50,7 +51,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
|
||||
## Dependencies
|
||||
|
||||
- Java 21
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom:4.1.0-RC.1` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007)
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007). 1.1 temporarily imports `4.1.0-SNAPSHOT` for the record-search provider SPI (QRun-IO/qqq#1042); re-pin to the QQQ release that carries it before releasing 1.1.0
|
||||
- OpenSearch Java Client 3.10.0; optional AWS SDK v2 `apache-client` and `sts` for `AWS_SIGV4`
|
||||
- Apache HttpClient5 and jackson-datatype-jsr310, versions managed by the qqq BOM (keep them unpinned so they match its httpcore5 and jackson)
|
||||
- JUnit 5 + AssertJ + Mockito for testing
|
||||
|
||||
@@ -2,6 +2,19 @@
|
||||
|
||||
All notable changes to this project are documented here. The format follows Keep a Changelog; versions follow semantic versioning.
|
||||
|
||||
## [1.1.0] - Unreleased
|
||||
|
||||
Requires the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042); built against 4.1.0-SNAPSHOT until it is released. See `docs/MIGRATION-1.1.md`.
|
||||
|
||||
### Added
|
||||
- `QuickSearchRecordSearchProvider`: serves QQQ core record search (`RecordSearchAction`, `POST /qqq/v1/search`, the Next UI global search) from OpenSearch for the tables Quick Search indexes. One `_msearch` request with a per-table top-N search (same query, AND semantics and field weights as `QuickSearchAction`); returns record ids only, which core re-reads through `QueryAction` (permissions, record locks, labels). Audit finding F-11, decision D1.
|
||||
- `serveCoreRecordSearch` (default `true`): registers the provider and gives indexed tables core `searchFields` (visible string and integer searchable fields) when they have none. Host `searchFields` and a host-registered provider are never replaced.
|
||||
- `recordSearchTimeoutMillis` (default 2500): wall-clock limit on the OpenSearch request; on timeout or any failure core falls back to its own search, and the provider claims no tables for 30 seconds.
|
||||
- `QuickSearchOpenSearchClient.searchRecordIdsPerTable`.
|
||||
|
||||
### Security
|
||||
- Fields of type `PASSWORD` are no longer indexed (as hidden fields already were). Reconcile or fully reindex once to remove values a 1.0 index may hold.
|
||||
|
||||
## [1.0.0-RC.2] - 2026-10-06
|
||||
|
||||
Second release candidate, published to Maven Central from `release/1.0`. Fixes the five pre-GA findings from the PR #10 review (#13 to #17); the changes are listed under 1.0.0 below. Requires QQQ 4.1. No schema change, but two new status values: `REBUILDING` on `quickSearchIndex.status` and `AWAITING_REINDEX` on `quickSearchFailedEvent.status`; a host that sized its own `quickSearchFailedEvent.status` column needs at least 16 characters. Known gaps before GA are tracked in #12.
|
||||
|
||||
@@ -15,7 +15,7 @@ This file provides guidance to Claude Code when working with code in this reposi
|
||||
./mvnw package # Build JAR
|
||||
CI=true ./mvnw verify # Full verify: integration tests (Docker) and the JaCoCo gate
|
||||
./mvnw verify -Dopensearch.test.image=opensearchproject/opensearch:3.9.0 # other server line
|
||||
./mvnw verify -Dqqq.snapshot.version=4.1.0-SNAPSHOT # other qqq 4.1 build (profile is active by default)
|
||||
./mvnw verify -Dqqq.snapshot.version=4.1.0-RC.1 # other qqq 4.1 build (profile is active by default)
|
||||
```
|
||||
|
||||
Integration tests (require Docker) run via `mvn verify` using maven-failsafe-plugin.
|
||||
@@ -34,6 +34,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
|
||||
- `listeners/QuickSearchRecordChangeListener` - Real-time path: builds events in the write's transaction, publishes after commit, records failures in `quickSearchFailedEvent`.
|
||||
- `processes/AbstractIndexingStep` - Keyset paging by primary key with display values, run records, drift detection, run purge. `BasepullIndexStep` (replays failed events, start-anchored watermark with overlap), `ReconcileIndexStep` (in place, orphan sweep), `FullReindexStep` (all tables: new physical index plus alias swap; one table: reconcile algorithm).
|
||||
- `actions/QuickSearchAction` - Permission-aware search (table READ filter, `QueryAction` post-filter), bounds, `tableNames`, `limitPerTable`, `tableLabel`.
|
||||
- `actions/QuickSearchRecordSearchProvider` - QQQ core `RecordSearchProviderInterface` (`POST /qqq/v1/search`): claims indexed, enabled tables; one `_msearch` per call returning ordered record ids per table; bounded by `recordSearchTimeoutMillis`; failures make core fall back and start a 30 s backoff (`QuickSearchRuntime.markRecordSearchFailed`). Registered, with default table `searchFields`, when `serveCoreRecordSearch` is true.
|
||||
- `metadata/*` - Component producers and the table/process helpers.
|
||||
|
||||
**Tables Produced:** `quickSearchIndex`, `quickSearchIndexRun`, `quickSearchFailedEvent` (plus a PVS over `quickSearchIndex`).
|
||||
@@ -50,7 +51,7 @@ Integration tests use `@ExtendWith(RequiresDockerCondition.class)`: skipped with
|
||||
## Dependencies
|
||||
|
||||
- Java 21
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom:4.1.0-RC.1` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007)
|
||||
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). 1.0 needs qqq 4.1, so the `qqq-snapshot` profile is active by default and imports `qqq-bom-pom` ahead of the parent's BOM; remove that activation when the parent moves to 4.1 (ADR-0007). 1.1 temporarily imports `4.1.0-SNAPSHOT` for the record-search provider SPI (QRun-IO/qqq#1042); re-pin to the QQQ release that carries it before releasing 1.1.0
|
||||
- OpenSearch Java Client 3.10.0; optional AWS SDK v2 `apache-client` and `sts` for `AWS_SIGV4`
|
||||
- Apache HttpClient5 and jackson-datatype-jsr310, versions managed by the qqq BOM (keep them unpinned so they match its httpcore5 and jackson)
|
||||
- JUnit 5 + AssertJ + Mockito for testing
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
Annotate entity classes or declare tables in config, point the QBit at an OpenSearch cluster, produce it into your `QInstance`, and you get real-time indexing of committed writes, a scheduled catch-up and reconcile, a blackout-free full rebuild, and a permission-aware search API with relevance scoring, typeahead, highlights and pagination.
|
||||
|
||||
Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit callbacks and runtime services).
|
||||
Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit callbacks and runtime services). Serving core search (`POST /qqq/v1/search`, below) needs the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042); until it is released, 1.1 builds against 4.1.0-SNAPSHOT.
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -18,6 +18,7 @@ Requires QQQ 4.1 (the QBit uses 4.1's record-change listeners, after-commit call
|
||||
| Basepull catch-up | Scheduled process (`schedulerName` plus `basepullRepeatSeconds`) that re-indexes rows changed since the previous run's start, with a configurable overlap, paging by primary key. |
|
||||
| Reconcile | Scheduled or on-demand process that re-indexes everything in place, then removes documents whose source row is gone, including documents of tables no longer configured. No search blackout. |
|
||||
| Full rebuild | Indexes every table into a fresh physical index and atomically swaps the alias; the old index keeps serving until the new one is complete. Also upgrades an index created by an older release. |
|
||||
| Core search | `QuickSearchRecordSearchProvider` serves QQQ's `RecordSearchAction` (`POST /qqq/v1/search`, the Next UI global search box) from OpenSearch for the indexed tables, with a timeout and an automatic fallback to core's own search. |
|
||||
| Search | `QuickSearchAction`: AND semantics, edge-ngram typeahead with a plain search analyzer, per-field weights, highlights, `tableNames` and `limitPerTable`, bounded paging, `tableLabel` on results. Only tables the session may read are searched; hits are re-read through `QueryAction` so record security locks apply. |
|
||||
| Connections | HTTP or HTTPS, basic auth, AWS IAM SigV4, custom CA or trust store, mTLS, timeouts, pool size, a transport customizer SPI for anything else. Secrets by `${env.X}` reference. |
|
||||
| Operations | `quickSearchIndex` (per-table status, counts, last run, last error), `quickSearchIndexRun` (history, purged after `runHistoryRetentionDays`), `quickSearchFailedEvent`, a `quickSearchAdmin` app, explicit permission rules, fail-fast or degraded startup. |
|
||||
@@ -99,7 +100,7 @@ config.withSearchableTable(new SearchableTableConfig("crmContact", List.of(
|
||||
.withRecordLabelFormat("%s %s", "firstName", "lastName"));
|
||||
```
|
||||
|
||||
Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull.
|
||||
Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden and password fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull.
|
||||
|
||||
### Searching
|
||||
|
||||
@@ -124,6 +125,17 @@ output.getTotalHits(); output.getTotalHitsIsLowerBound(); output.getHasMore();
|
||||
|
||||
Search runs as the current `QContext` session: tables the session cannot read are skipped, and with `applyRecordSecurityLocks` (default) hits are re-read through `QueryAction` so record security locks apply.
|
||||
|
||||
### Core search (`POST /qqq/v1/search`)
|
||||
|
||||
QQQ core has its own record search (`RecordSearchAction`, served at `POST /qqq/v1/search` and used by the Next UI's global search box): a case-insensitive contains-query over each table's `searchFields`. With `serveCoreRecordSearch` (default `true`) the producer registers `QuickSearchRecordSearchProvider` on the instance, and core asks it to search every table Quick Search indexes. The API and the UI do not change.
|
||||
|
||||
- **What is served from OpenSearch.** Tables this QBit indexes and that are enabled in `quickSearchIndex`. Core offers a table only when it has `searchFields`, the session may read it, and it supports query, so the producer gives each indexed table that has no `searchFields` its visible string and integer searchable fields (never hidden or password fields). `searchFields` you set yourself are kept. A table must be in the `QInstance` before the QBit is produced to get them.
|
||||
- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights); each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order. Core re-reads them by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, and records deleted since they were indexed drop out.
|
||||
- **Failure.** The OpenSearch request is cancelled after `recordSearchTimeoutMillis` (default 2500). Any failure (timeout, cluster down, index missing) makes core search those tables with its own contains-query instead, so search degrades but never fails. After a failure, Quick Search claims no tables for 30 seconds, so a down cluster does not add a timeout to every search.
|
||||
- **Opting out.** `withServeCoreRecordSearch(false)` leaves core search and `searchFields` untouched. If the host already registered another record-search provider, the QBit keeps it and logs a warning (an instance has one provider).
|
||||
|
||||
`QuickSearchAction` remains the richer API (scores, highlights, pagination, totals).
|
||||
|
||||
### Processes
|
||||
|
||||
| Process | Trigger | What it does |
|
||||
@@ -159,6 +171,8 @@ Without `schedulerName` the processes exist but nothing runs them; the QBit logs
|
||||
| `maxFieldLength` | 10000 | Truncate each indexed value |
|
||||
| `maxSearchLimit` | 100 | Largest page size |
|
||||
| `applyRecordSecurityLocks` | `true` | Re-read hits through `QueryAction` |
|
||||
| `serveCoreRecordSearch` | `true` | Register `QuickSearchRecordSearchProvider` so core `POST /qqq/v1/search` uses OpenSearch for indexed tables, and give those tables `searchFields` when they have none |
|
||||
| `recordSearchTimeoutMillis` | 2500 | Wall-clock limit on the OpenSearch request behind a core search; past it, core falls back to its own search |
|
||||
| `adminPermissionRules` | `HAS_ACCESS_PERMISSION`, base name `quickSearchAdmin` | Rules for the app and processes |
|
||||
| `tablePermissionRules` | `READ_WRITE_PERMISSIONS` | Rules for the operational tables |
|
||||
| `tableMetaDataCustomizer` | | Applied to every produced table (backend details, for example) |
|
||||
@@ -187,6 +201,10 @@ CI=true ./mvnw verify # plus Testcontainers integration te
|
||||
|
||||
Use the wrapper: Maven 3.10 cannot read the published parent POM. The build imports qqq 4.1.0-RC.1 through the default-active `qqq-snapshot` profile until `qbit-build-parent` 2.1.0 ships (override with `-Dqqq.snapshot.version=...`). Without Docker the integration tests are skipped locally; with `CI=true` they fail instead.
|
||||
|
||||
## Upgrading from 1.0
|
||||
|
||||
See [docs/MIGRATION-1.1.md](docs/MIGRATION-1.1.md). In short: 1.1 needs the QQQ release with the record-search provider SPI; core search starts using OpenSearch for indexed tables by default, and indexed tables gain `searchFields` (set `serveCoreRecordSearch(false)` to keep 1.0 behaviour).
|
||||
|
||||
## Upgrading from 0.x
|
||||
|
||||
See [docs/MIGRATION-1.0.md](docs/MIGRATION-1.0.md). In short: QQQ 4.1 is required, remove any manual `addQBit` for this QBit, set `schedulerName`, prefer `opensearchUrl` and `${env.}` secrets, and run a full reindex once so the index gets the new mapping.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# Migrating from 1.0 to 1.1
|
||||
|
||||
1.1 lets QQQ core's record search (`POST /qqq/v1/search`, the Next UI global search box) use the Quick Search index. Hosts that do nothing get it by default.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The QQQ release that carries the record-search provider SPI (`RecordSearchProviderInterface`, `QInstance.withRecordSearchProvider`), added in QRun-IO/qqq#1042. Until that release exists, 1.1 builds against `4.1.0-SNAPSHOT` through the `qqq-snapshot` profile; it is re-pinned to the release before 1.1.0 ships.
|
||||
|
||||
## What changes at startup
|
||||
|
||||
1. **The QBit registers a record-search provider.** `QuickSearchRecordSearchProvider` becomes the instance's `recordSearchProvider`. If the host already registered a different provider, the QBit keeps it and logs a warning; core search then does not use OpenSearch.
|
||||
2. **Indexed tables get `searchFields`.** Core only searches tables with `searchFields`. Each indexed table that has none gets its visible string and integer searchable fields. Tables that already have `searchFields` keep them. A table added to the instance after the QBit is produced gets none, so add tables first (as for field validation) or set `searchFields` yourself.
|
||||
3. **Password fields are no longer indexed.** Fields of type `PASSWORD` are now dropped from the searchable fields like hidden fields. If a 1.0 index holds such values, run "Quick Search Reconcile Index" (or a full reindex) to rewrite the documents without them.
|
||||
|
||||
## Behaviour to be aware of
|
||||
|
||||
- Next UI search results for indexed tables now come from OpenSearch: all words must match (in any searchable field), with prefix matching and field weights, instead of a contains-match on one field. Tables not indexed by Quick Search keep core's search.
|
||||
- Results still pass core's security: core re-reads every hit through `QueryAction`, so table permissions and record security locks apply, and records deleted since they were indexed do not appear.
|
||||
- Records are only as fresh as the index: a record updated in a way that is not yet indexed (for example while real-time indexing is off) is found by its indexed text.
|
||||
- When OpenSearch is slow or down, core waits at most `recordSearchTimeoutMillis` (default 2500) and then searches with its own contains-query; for the next 30 seconds it skips OpenSearch entirely.
|
||||
|
||||
## Opting out
|
||||
|
||||
```java
|
||||
new QuickSearchQBitConfig()
|
||||
// ...
|
||||
.withServeCoreRecordSearch(false);
|
||||
```
|
||||
|
||||
restores 1.0 behaviour: no provider, no `searchFields` changes.
|
||||
+6
-1
@@ -9,7 +9,7 @@
|
||||
- [ ] qqq-all follow-up PR: remove the manual `instance.addQBit(...)` for quick-search in `DemoQBits`, pass `withSchedulerName("demoScheduler")`, set `withStartupMode(QuickSearchStartupMode.DEGRADED)` for the full profile test, pin 1.0.0.
|
||||
- [ ] Voyage: pin 1.0.0, set `schedulerName`, switch to `opensearchUrl` and `${env.}` credentials.
|
||||
- [ ] CI: add a second test job with `-Dopensearch.test.image=opensearchproject/opensearch:3.9.0` (needs a qqq-orb parameter for extra Maven args, or a plain job).
|
||||
- [ ] Open the qqq issue proposing `RecordSearchProviderInterface` (D1); quick-search adapter lands in 1.1.
|
||||
- [x] Core SPI proposed as QRun-IO/qqq#1042 (D1); the 1.1 adapter `QuickSearchRecordSearchProvider` is on `feature/record-search-provider`.
|
||||
- [x] Tracking issue #12 created and recorded in the PRD (2026-10-06).
|
||||
- [x] PR #8 merged 2026-10-05; Dependabot PR #1 closed as superseded.
|
||||
- [x] PR #10 merged and `1.0.0-RC.1` published to Maven Central from `release/1.0` (2026-10-05).
|
||||
@@ -26,6 +26,11 @@
|
||||
- [ ] Stuck `REBUILDING` after a killed or overlapping full reindex (from the PR #20 review): only another all-tables reindex clears it, `AWAITING_REINDEX` rows accumulate and drift detection stays off for those tables. No data is lost. Fix needs a liveness check on the `FULL_REINDEX` run record before reconcile may reset the status, plus refusing a second concurrent full reindex.
|
||||
- [ ] Drift detection still writes `NEEDS_REINDEX` into `status`, so a few-millisecond window can overwrite `REBUILDING`; the durable fix is a separate drift field (schema change, so 1.1 at the earliest).
|
||||
|
||||
## Before releasing 1.1.0
|
||||
|
||||
- [ ] Re-point the `qqq-snapshot` profile from `4.1.0-SNAPSHOT` to the QQQ release that carries the record-search provider SPI (QRun-IO/qqq#1042).
|
||||
- [ ] qqq-all / Website-Backend: confirm the Next UI search box is served from OpenSearch (provider registered, indexed tables have `searchFields`).
|
||||
|
||||
## Later (1.x)
|
||||
|
||||
- [ ] Health widget for the admin app (data exists: `isHealthy`, `documentCount`, `lastRunStatus`).
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
|
||||
<properties>
|
||||
<revision>1.0.0-SNAPSHOT</revision>
|
||||
<revision>1.1.0-SNAPSHOT</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<opensearch.version>3.10.0</opensearch.version>
|
||||
@@ -264,13 +264,17 @@
|
||||
<!-- qqq-bom-pom 4.1.0-RC.1 ahead of the parent's 4.0.0 BOM. Override with -->
|
||||
<!-- -Dqqq.snapshot.version=... ; deactivate with -P!qqq-snapshot. Remove the -->
|
||||
<!-- activation when the parent moves to 4.1 (ADR-0007). -->
|
||||
<!-- 1.1 TEMPORARY: the record-search provider needs the core SPI from -->
|
||||
<!-- QRun-IO/qqq#1042 (RecordSearchProviderInterface), which is only in -->
|
||||
<!-- 4.1.0-SNAPSHOT once #1042 merges. Re-point this to the QQQ release that -->
|
||||
<!-- carries the SPI before releasing 1.1.0. -->
|
||||
<profile>
|
||||
<id>qqq-snapshot</id>
|
||||
<activation>
|
||||
<activeByDefault>true</activeByDefault>
|
||||
</activation>
|
||||
<properties>
|
||||
<qqq.snapshot.version>4.1.0-RC.1</qqq.snapshot.version>
|
||||
<qqq.snapshot.version>4.1.0-SNAPSHOT</qqq.snapshot.version>
|
||||
</properties>
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
|
||||
@@ -52,14 +52,20 @@ import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher;
|
||||
** enableMaintenanceProcesses, schedulerName plus basepull/reconcile schedules,
|
||||
** permission rules for the admin surface, search bounds, and indexing limits.
|
||||
**
|
||||
** Core search: serveCoreRecordSearch (default true) registers
|
||||
** QuickSearchRecordSearchProvider, so QQQ's POST /qqq/v1/search (the Next UI
|
||||
** global search) is served from OpenSearch for indexed tables, bounded by
|
||||
** recordSearchTimeoutMillis.
|
||||
**
|
||||
** Tables: searchableEntityClasses (annotated entities) and/or searchableTables
|
||||
** (config-driven). validate() is list-based and never throws; the producer
|
||||
** fails production when it reports errors.
|
||||
*******************************************************************************/
|
||||
public class QuickSearchQBitConfig implements QBitConfig
|
||||
{
|
||||
public static final String DEFAULT_ADMIN_PERMISSION_BASE_NAME = "quickSearchAdmin";
|
||||
public static final String DEFAULT_BASEPULL_TIMESTAMP_FIELD = "modifyDate";
|
||||
public static final String DEFAULT_ADMIN_PERMISSION_BASE_NAME = "quickSearchAdmin";
|
||||
public static final String DEFAULT_BASEPULL_TIMESTAMP_FIELD = "modifyDate";
|
||||
public static final int DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS = 2500;
|
||||
|
||||
private static final Pattern INDEX_NAME_PATTERN = Pattern.compile("^[a-z0-9][a-z0-9_+.-]*$");
|
||||
private static final Set<String> AWS_SERVICE_NAMES = Set.of("es", "aoss");
|
||||
@@ -100,6 +106,8 @@ public class QuickSearchQBitConfig implements QBitConfig
|
||||
private Integer maxFieldLength = 10000;
|
||||
private Integer maxSearchLimit = 100;
|
||||
private Boolean applyRecordSecurityLocks = true;
|
||||
private Boolean serveCoreRecordSearch = true;
|
||||
private Integer recordSearchTimeoutMillis = DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS;
|
||||
private List<Class<?>> searchableEntityClasses;
|
||||
private List<SearchableTableConfig> searchableTables;
|
||||
|
||||
@@ -387,6 +395,7 @@ public class QuickSearchQBitConfig implements QBitConfig
|
||||
requirePositive(maxBulkRequestBytes, "maxBulkRequestBytes", errors);
|
||||
requirePositive(basepullRepeatSeconds, "basepullRepeatSeconds", errors);
|
||||
requirePositive(maxSearchLimit, "maxSearchLimit", errors);
|
||||
requirePositive(recordSearchTimeoutMillis, "recordSearchTimeoutMillis", errors);
|
||||
requirePositive(maxFieldLength, "maxFieldLength", errors);
|
||||
requirePositive(runHistoryRetentionDays, "runHistoryRetentionDays", errors);
|
||||
|
||||
@@ -1897,6 +1906,72 @@ public class QuickSearchQBitConfig implements QBitConfig
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for serveCoreRecordSearch: register QuickSearchRecordSearchProvider
|
||||
** so core RecordSearchAction (POST /qqq/v1/search) searches indexed tables in
|
||||
** OpenSearch, and give indexed tables core searchFields when they have none
|
||||
*******************************************************************************/
|
||||
public Boolean getServeCoreRecordSearch()
|
||||
{
|
||||
return (this.serveCoreRecordSearch);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter for serveCoreRecordSearch
|
||||
*******************************************************************************/
|
||||
public void setServeCoreRecordSearch(Boolean serveCoreRecordSearch)
|
||||
{
|
||||
this.serveCoreRecordSearch = serveCoreRecordSearch;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for serveCoreRecordSearch
|
||||
*******************************************************************************/
|
||||
public QuickSearchQBitConfig withServeCoreRecordSearch(Boolean serveCoreRecordSearch)
|
||||
{
|
||||
this.serveCoreRecordSearch = serveCoreRecordSearch;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for recordSearchTimeoutMillis: wall-clock limit on the OpenSearch
|
||||
** request behind a core record search; past it the provider fails and core
|
||||
** falls back to its own search (default 2500)
|
||||
*******************************************************************************/
|
||||
public Integer getRecordSearchTimeoutMillis()
|
||||
{
|
||||
return (this.recordSearchTimeoutMillis);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Setter for recordSearchTimeoutMillis
|
||||
*******************************************************************************/
|
||||
public void setRecordSearchTimeoutMillis(Integer recordSearchTimeoutMillis)
|
||||
{
|
||||
this.recordSearchTimeoutMillis = recordSearchTimeoutMillis;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for recordSearchTimeoutMillis
|
||||
*******************************************************************************/
|
||||
public QuickSearchQBitConfig withRecordSearchTimeoutMillis(Integer recordSearchTimeoutMillis)
|
||||
{
|
||||
this.recordSearchTimeoutMillis = recordSearchTimeoutMillis;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for searchableEntityClasses: entity classes annotated with @QuickSearchable
|
||||
|
||||
@@ -28,12 +28,16 @@ import java.util.Properties;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
|
||||
import com.kingsrook.qqq.backend.core.utils.StringUtils;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
|
||||
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField;
|
||||
import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
|
||||
import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
|
||||
@@ -50,7 +54,9 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
** processes, admin app), applies the config's table customizer and default
|
||||
** backend, stamps sourceQBitName, then calls {@link #postProduceActions}, which
|
||||
** discovers searchable tables, builds the runtime, registers the record-change
|
||||
** listener and runtime service, and (in FAIL_FAST mode) prepares the index.
|
||||
** listener and runtime service, registers the core record-search provider
|
||||
** (serveCoreRecordSearch) and gives indexed tables core searchFields, and (in
|
||||
** FAIL_FAST mode) prepares the index.
|
||||
**
|
||||
** Host usage, unchanged from 0.x:
|
||||
** new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
|
||||
@@ -254,6 +260,12 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
|
||||
|
||||
qInstance.withRuntimeService(new QCodeReference(QuickSearchRuntimeService.class));
|
||||
|
||||
if(Boolean.TRUE.equals(qBitConfig.getServeCoreRecordSearch()))
|
||||
{
|
||||
registerRecordSearchProvider(qInstance);
|
||||
defaultCoreSearchFields(qInstance, discoveredTables);
|
||||
}
|
||||
|
||||
if(!StringUtils.hasContent(qBitConfig.getSchedulerName()) && Boolean.TRUE.equals(qBitConfig.getEnableBasepullProcess()))
|
||||
{
|
||||
LOG.warn("Quick Search has no schedulerName; the basepull and reconcile processes are registered but will not run on a schedule");
|
||||
@@ -270,6 +282,82 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Register QuickSearchRecordSearchProvider as the instance's core record
|
||||
** search provider. An instance holds one provider; a different one the host
|
||||
** already registered is kept (with a warning).
|
||||
*******************************************************************************/
|
||||
static void registerRecordSearchProvider(QInstance qInstance)
|
||||
{
|
||||
QCodeReference existing = qInstance.getRecordSearchProvider();
|
||||
if(existing != null && !QuickSearchRecordSearchProvider.class.getName().equals(existing.getName()))
|
||||
{
|
||||
LOG.warn("The instance already has a record search provider; Quick Search does not replace it, so core search does not use OpenSearch",
|
||||
logPair("provider", existing.getName()));
|
||||
return;
|
||||
}
|
||||
|
||||
qInstance.withRecordSearchProvider(new QCodeReference(QuickSearchRecordSearchProvider.class));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Core record search only searches tables with searchFields, so give each
|
||||
** indexed table that has none the searchable fields core accepts (visible
|
||||
** string or integer fields; never password fields). Fields the host set are
|
||||
** never replaced. A table that is not in the instance yet, uses backend
|
||||
** variants, or has no such field is skipped with a warning.
|
||||
*******************************************************************************/
|
||||
static void defaultCoreSearchFields(QInstance qInstance, List<QuickSearchableTableConfig> tableConfigs)
|
||||
{
|
||||
for(QuickSearchableTableConfig tableConfig : tableConfigs)
|
||||
{
|
||||
QTableMetaData table = qInstance.getTable(tableConfig.getTableName());
|
||||
if(table == null)
|
||||
{
|
||||
LOG.warn("Indexed table is not in the instance when Quick Search is produced, so it gets no core searchFields; set them on the table for core search to use it",
|
||||
logPair("tableName", tableConfig.getTableName()));
|
||||
continue;
|
||||
}
|
||||
|
||||
if(table.getSearchFields() != null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
QBackendMetaData backend = qInstance.getBackend(table.getBackendName());
|
||||
if(backend != null && Boolean.TRUE.equals(backend.getUsesVariants()))
|
||||
{
|
||||
LOG.warn("Indexed table uses backend variants, which core record search does not support; it gets no core searchFields", logPair("tableName", table.getName()));
|
||||
continue;
|
||||
}
|
||||
|
||||
List<String> searchFields = new ArrayList<>();
|
||||
for(String fieldName : CollectionUtils.nonNullList(tableConfig.getSearchableFields()))
|
||||
{
|
||||
QFieldMetaData field = table.getFields() == null ? null : table.getFields().get(fieldName);
|
||||
if(field != null && !Boolean.TRUE.equals(field.getIsHidden()) && field.getType() != null
|
||||
&& (field.getType().isStringLike() || field.getType().isIntegral()) && !field.getType().needsMasked()
|
||||
&& !searchFields.contains(fieldName))
|
||||
{
|
||||
searchFields.add(fieldName);
|
||||
}
|
||||
}
|
||||
|
||||
if(searchFields.isEmpty())
|
||||
{
|
||||
LOG.warn("Indexed table has no string or integer searchable field, so it gets no core searchFields; set them on the table for core search to use it",
|
||||
logPair("tableName", table.getName()));
|
||||
continue;
|
||||
}
|
||||
|
||||
table.setSearchFields(searchFields);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Discover searchable tables from @QuickSearchable entity classes. Fails when
|
||||
** a configured field does not exist on a table that is in the instance.
|
||||
@@ -455,22 +543,23 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Hidden fields are never indexed: their values would leak through
|
||||
** searchable text and highlights.
|
||||
** Hidden and password fields are never indexed: their values would leak
|
||||
** through searchable text and highlights.
|
||||
*******************************************************************************/
|
||||
private static void removeHiddenFields(QTableMetaData table, List<String> searchableFields, Map<String, Integer> fieldWeights, Map<String, Boolean> fieldIncludeLabels)
|
||||
{
|
||||
List<String> hidden = new ArrayList<>();
|
||||
for(String fieldName : searchableFields)
|
||||
{
|
||||
if(table.getFields().containsKey(fieldName) && Boolean.TRUE.equals(table.getField(fieldName).getIsHidden()))
|
||||
QFieldMetaData field = table.getFields().get(fieldName);
|
||||
if(field != null && (Boolean.TRUE.equals(field.getIsHidden()) || (field.getType() != null && field.getType().needsMasked())))
|
||||
{
|
||||
hidden.add(fieldName);
|
||||
}
|
||||
}
|
||||
if(!hidden.isEmpty())
|
||||
{
|
||||
LOG.warn("Hidden fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden));
|
||||
LOG.warn("Hidden and password fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden));
|
||||
searchableFields.removeAll(hidden);
|
||||
hidden.forEach(fieldWeights::remove);
|
||||
hidden.forEach(fieldIncludeLabels::remove);
|
||||
|
||||
@@ -57,6 +57,8 @@ public class QuickSearchRuntime
|
||||
|
||||
private static final long ENABLED_CACHE_TTL_MILLIS = 60_000L;
|
||||
|
||||
public static final long RECORD_SEARCH_BACKOFF_MILLIS = 30_000L;
|
||||
|
||||
private final QuickSearchQBitConfig config;
|
||||
|
||||
private List<QuickSearchableTableConfig> discoveredTables = new ArrayList<>();
|
||||
@@ -64,8 +66,9 @@ public class QuickSearchRuntime
|
||||
|
||||
private volatile QuickSearchOpenSearchClient client;
|
||||
private volatile IndexEventPublisher publisher;
|
||||
private volatile boolean indexReady = false;
|
||||
private volatile boolean closed = false;
|
||||
private volatile boolean indexReady = false;
|
||||
private volatile boolean closed = false;
|
||||
private volatile long recordSearchBackoffUntilMillis = 0L;
|
||||
|
||||
private final Map<String, CachedFlag> enabledCache = new ConcurrentHashMap<>();
|
||||
|
||||
@@ -302,6 +305,40 @@ public class QuickSearchRuntime
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Record that a core record search through OpenSearch failed. For the next
|
||||
** RECORD_SEARCH_BACKOFF_MILLIS the record-search provider claims no tables,
|
||||
** so core searches them itself at once instead of waiting for another
|
||||
** timeout on every search while OpenSearch is down.
|
||||
*******************************************************************************/
|
||||
public void markRecordSearchFailed()
|
||||
{
|
||||
recordSearchBackoffUntilMillis = System.currentTimeMillis() + RECORD_SEARCH_BACKOFF_MILLIS;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether core record search is backing off after a recent failure.
|
||||
*******************************************************************************/
|
||||
public boolean isRecordSearchBackingOff()
|
||||
{
|
||||
return (System.currentTimeMillis() < recordSearchBackoffUntilMillis);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** End a record-search backoff early (an operator or test knows OpenSearch is
|
||||
** back).
|
||||
*******************************************************************************/
|
||||
public void clearRecordSearchBackoff()
|
||||
{
|
||||
recordSearchBackoffUntilMillis = 0L;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Read the enabled flag from the quickSearchIndex row.
|
||||
*******************************************************************************/
|
||||
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
/*
|
||||
* Copyright 2024 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.quicksearch.actions;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
|
||||
import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
|
||||
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Serves QQQ core record search (RecordSearchAction, POST /qqq/v1/search,
|
||||
** the Next UI global search) from the Quick Search index, for the tables
|
||||
** this QBit indexes. Registered on the QInstance by QuickSearchQBitProducer
|
||||
** when serveCoreRecordSearch is on (the default).
|
||||
**
|
||||
** Core creates a new instance for every search, so this class holds no
|
||||
** state: it resolves the QBit's QuickSearchRuntime (client, tables, config)
|
||||
** from QContext.
|
||||
**
|
||||
** A table is claimed when it is indexed by this QBit and enabled, and core
|
||||
** record search is not backing off after a recent failure. search() sends
|
||||
** one _msearch request with a top-N search per table and returns record ids
|
||||
** only, best first; core re-reads them by primary key through QueryAction,
|
||||
** so table permissions, record security locks and record labels are core's
|
||||
** job, and ids of deleted records drop out. Each table gets
|
||||
** IDS_PER_LIMIT_MULTIPLIER x limitPerTable ids (at most
|
||||
** RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE) so dropouts do not leave
|
||||
** the table short.
|
||||
**
|
||||
** The request is bounded by recordSearchTimeoutMillis. Any failure throws,
|
||||
** and core then searches the claimed tables with its own LIKE search; the
|
||||
** failure also starts a short backoff (QuickSearchRuntime
|
||||
** RECORD_SEARCH_BACKOFF_MILLIS) during which no tables are claimed.
|
||||
*******************************************************************************/
|
||||
public class QuickSearchRecordSearchProvider implements RecordSearchProviderInterface
|
||||
{
|
||||
private static final QLogger LOG = QLogger.getLogger(QuickSearchRecordSearchProvider.class);
|
||||
|
||||
public static final int IDS_PER_LIMIT_MULTIPLIER = 4;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Claim tables this QBit indexes and that are enabled, unless core record
|
||||
** search is off or backing off. Cheap: an in-memory lookup plus the
|
||||
** runtime's cached enabled flag.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean claimsTable(QTableMetaData table)
|
||||
{
|
||||
QuickSearchRuntime runtime = QuickSearchRuntime.get();
|
||||
if(runtime == null || table == null || !Boolean.TRUE.equals(runtime.getConfig().getServeCoreRecordSearch()))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
|
||||
if(runtime.getTableConfig(table.getName()) == null || runtime.isRecordSearchBackingOff())
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
|
||||
return (runtime.isTableEnabled(table.getName()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Ids of the best matches in each requested table, in relevance order.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public RecordSearchOutput search(RecordSearchInput input) throws QException
|
||||
{
|
||||
QuickSearchRuntime runtime = QuickSearchRuntime.getOrThrow();
|
||||
QuickSearchQBitConfig config = runtime.getConfig();
|
||||
|
||||
List<RecordSearchResult> results = new ArrayList<>();
|
||||
String term = IndexingUtils.normalizeSearchText(input.getSearchTerm());
|
||||
if(term.length() < QuickSearchAction.MIN_TERM_LENGTH)
|
||||
{
|
||||
return (new RecordSearchOutput().withResults(results));
|
||||
}
|
||||
|
||||
List<String> tableNames = new ArrayList<>();
|
||||
for(String tableName : new LinkedHashSet<>(CollectionUtils.nonNullList(input.getTableNames())))
|
||||
{
|
||||
if(runtime.getTableConfig(tableName) != null)
|
||||
{
|
||||
tableNames.add(tableName);
|
||||
}
|
||||
}
|
||||
if(tableNames.isEmpty())
|
||||
{
|
||||
return (new RecordSearchOutput().withResults(results));
|
||||
}
|
||||
|
||||
int limitPerTable = input.getLimitPerTable() == null || input.getLimitPerTable() <= 0 ? RecordSearchAction.DEFAULT_LIMIT_PER_TABLE : input.getLimitPerTable();
|
||||
int idsPerTable = Math.min(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE, limitPerTable * IDS_PER_LIMIT_MULTIPLIER);
|
||||
int timeoutMillis = config.getRecordSearchTimeoutMillis() == null ? QuickSearchQBitConfig.DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS : config.getRecordSearchTimeoutMillis();
|
||||
|
||||
Map<String, List<String>> idsByTable;
|
||||
try
|
||||
{
|
||||
idsByTable = runtime.getClient().searchRecordIdsPerTable(term, tableNames, idsPerTable, runtime.getDiscoveredTables(), timeoutMillis);
|
||||
}
|
||||
catch(QException | RuntimeException e)
|
||||
{
|
||||
runtime.markRecordSearchFailed();
|
||||
LOG.info("Quick Search record search failed; core searches these tables itself, and Quick Search claims none for a while", logPair("tables", tableNames),
|
||||
logPair("backoffMillis", QuickSearchRuntime.RECORD_SEARCH_BACKOFF_MILLIS), logPair("error", e.getMessage()));
|
||||
throw (e instanceof QException qe ? qe : new QException("Quick Search record search failed: " + e.getMessage(), e));
|
||||
}
|
||||
|
||||
for(String tableName : tableNames)
|
||||
{
|
||||
for(String recordId : CollectionUtils.nonNullList(idsByTable.get(tableName)))
|
||||
{
|
||||
results.add(new RecordSearchResult().withTableName(tableName).withRecordId(recordId));
|
||||
}
|
||||
}
|
||||
|
||||
return (new RecordSearchOutput().withResults(results));
|
||||
}
|
||||
|
||||
}
|
||||
+130
-13
@@ -21,9 +21,14 @@ import java.io.Closeable;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.SerializationFeature;
|
||||
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
|
||||
@@ -32,6 +37,7 @@ import com.kingsrook.qqq.backend.core.logging.QLogger;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
|
||||
import org.opensearch.client.json.JsonData;
|
||||
import org.opensearch.client.opensearch.OpenSearchAsyncClient;
|
||||
import org.opensearch.client.opensearch.OpenSearchClient;
|
||||
import org.opensearch.client.opensearch._types.Conflicts;
|
||||
import org.opensearch.client.opensearch._types.FieldValue;
|
||||
@@ -51,12 +57,17 @@ import org.opensearch.client.opensearch.core.DeleteByQueryRequest;
|
||||
import org.opensearch.client.opensearch.core.DeleteByQueryResponse;
|
||||
import org.opensearch.client.opensearch.core.DeleteRequest;
|
||||
import org.opensearch.client.opensearch.core.IndexRequest;
|
||||
import org.opensearch.client.opensearch.core.MsearchRequest;
|
||||
import org.opensearch.client.opensearch.core.MsearchResponse;
|
||||
import org.opensearch.client.opensearch.core.SearchRequest;
|
||||
import org.opensearch.client.opensearch.core.SearchResponse;
|
||||
import org.opensearch.client.opensearch.core.bulk.BulkOperation;
|
||||
import org.opensearch.client.opensearch.core.bulk.BulkResponseItem;
|
||||
import org.opensearch.client.opensearch.core.bulk.DeleteOperation;
|
||||
import org.opensearch.client.opensearch.core.bulk.IndexOperation;
|
||||
import org.opensearch.client.opensearch.core.msearch.MultiSearchResponseItem;
|
||||
import org.opensearch.client.opensearch.core.msearch.RequestItem;
|
||||
import org.opensearch.client.opensearch.core.search.Hit;
|
||||
import org.opensearch.client.opensearch.core.search.Highlight;
|
||||
import org.opensearch.client.opensearch.core.search.HighlightField;
|
||||
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
|
||||
@@ -92,11 +103,12 @@ public class QuickSearchOpenSearchClient implements Closeable
|
||||
public static final String SEARCH_ANALYZER = "quick_search_search_analyzer";
|
||||
public static final int MAX_RESULT_WINDOW = 10_000;
|
||||
|
||||
private final OpenSearchClient client;
|
||||
private final OpenSearchTransport transport;
|
||||
private final String indexName;
|
||||
private final int maxBulkRequestBytes;
|
||||
private final ObjectMapper sizingMapper;
|
||||
private final OpenSearchClient client;
|
||||
private final OpenSearchAsyncClient asyncClient;
|
||||
private final OpenSearchTransport transport;
|
||||
private final String indexName;
|
||||
private final int maxBulkRequestBytes;
|
||||
private final ObjectMapper sizingMapper;
|
||||
|
||||
private volatile Boolean mappingOutdated = false;
|
||||
private Boolean closed = false;
|
||||
@@ -110,6 +122,7 @@ public class QuickSearchOpenSearchClient implements Closeable
|
||||
{
|
||||
this.transport = OpenSearchTransportFactory.build(config);
|
||||
this.client = new OpenSearchClient(transport);
|
||||
this.asyncClient = new OpenSearchAsyncClient(transport);
|
||||
this.indexName = config.getOpensearchIndexName();
|
||||
this.maxBulkRequestBytes = config.getMaxBulkRequestBytes() == null ? 5 * 1024 * 1024 : config.getMaxBulkRequestBytes();
|
||||
this.sizingMapper = new ObjectMapper().registerModule(new JavaTimeModule()).disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
|
||||
@@ -652,14 +665,7 @@ public class QuickSearchOpenSearchClient implements Closeable
|
||||
{
|
||||
try
|
||||
{
|
||||
List<FieldValue> allowed = allowedTables.stream().map(FieldValue::of).toList();
|
||||
|
||||
BoolQuery.Builder boolBuilder = new BoolQuery.Builder()
|
||||
.must(Query.of(q -> q.multiMatch(MultiMatchQuery.of(mm -> mm.query(searchTerm).fields("searchableText").operator(Operator.And)))))
|
||||
.filter(Query.of(q -> q.terms(t -> t.field("sourceTable").terms(tv -> tv.value(allowed)))))
|
||||
.should(buildFieldBoostQueries(searchTerm, tableConfigs, allowedTables));
|
||||
|
||||
Query finalQuery = Query.of(q -> q.bool(boolBuilder.build()));
|
||||
Query finalQuery = buildSearchQuery(searchTerm, allowedTables, tableConfigs);
|
||||
Highlight highlight = Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf)));
|
||||
|
||||
SearchRequest request = SearchRequest.of(r -> r
|
||||
@@ -681,6 +687,117 @@ public class QuickSearchOpenSearchClient implements Closeable
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Record ids of the best matches in each table, best first, for core record
|
||||
** search. One _msearch request with one search per table (the same query and
|
||||
** ranking as {@link #search}, restricted to that table), so every table gets
|
||||
** its own top idsPerTable hits. Only sourceTable and recordId are fetched.
|
||||
**
|
||||
** The whole request is bounded by timeoutMillis of wall-clock time (and the
|
||||
** same server-side search timeout): when it runs out, the request is
|
||||
** cancelled and a QException is thrown. A failed search for any table fails
|
||||
** the call. Returns every requested table as a key, in the given order.
|
||||
*******************************************************************************/
|
||||
public Map<String, List<String>> searchRecordIdsPerTable(String searchTerm, List<String> tableNames, int idsPerTable, List<QuickSearchableTableConfig> tableConfigs, int timeoutMillis) throws QException
|
||||
{
|
||||
Map<String, List<String>> idsByTable = new LinkedHashMap<>();
|
||||
if(tableNames == null || tableNames.isEmpty())
|
||||
{
|
||||
return (idsByTable);
|
||||
}
|
||||
|
||||
List<RequestItem> searches = new ArrayList<>();
|
||||
for(String tableName : tableNames)
|
||||
{
|
||||
Query query = buildSearchQuery(searchTerm, List.of(tableName), tableConfigs);
|
||||
searches.add(RequestItem.of(ri -> ri
|
||||
.header(h -> h.index(indexName))
|
||||
.body(b -> b
|
||||
.query(query)
|
||||
.size(idsPerTable)
|
||||
.trackTotalHits(t -> t.enabled(false))
|
||||
.source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId")))
|
||||
.timeout(timeoutMillis + "ms"))));
|
||||
}
|
||||
|
||||
CompletableFuture<MsearchResponse<OpenSearchDocument>> future = null;
|
||||
try
|
||||
{
|
||||
future = asyncClient.msearch(MsearchRequest.of(r -> r.searches(searches)), OpenSearchDocument.class);
|
||||
MsearchResponse<OpenSearchDocument> response = future.get(timeoutMillis, TimeUnit.MILLISECONDS);
|
||||
|
||||
List<MultiSearchResponseItem<OpenSearchDocument>> items = response.responses();
|
||||
if(items == null || items.size() != tableNames.size())
|
||||
{
|
||||
throw (new QException("Record search failed: OpenSearch returned " + (items == null ? 0 : items.size()) + " responses for " + tableNames.size() + " searches"));
|
||||
}
|
||||
|
||||
for(int i = 0; i < tableNames.size(); i++)
|
||||
{
|
||||
MultiSearchResponseItem<OpenSearchDocument> item = items.get(i);
|
||||
if(item.isFailure())
|
||||
{
|
||||
String reason = item.failure() != null && item.failure().error() != null ? item.failure().error().reason() : "unknown error";
|
||||
throw (new QException("Record search failed for table [" + tableNames.get(i) + "]: " + reason));
|
||||
}
|
||||
|
||||
List<String> ids = new ArrayList<>();
|
||||
for(Hit<OpenSearchDocument> hit : item.result().hits().hits())
|
||||
{
|
||||
if(hit.source() != null && hit.source().getRecordId() != null)
|
||||
{
|
||||
ids.add(hit.source().getRecordId());
|
||||
}
|
||||
}
|
||||
idsByTable.put(tableNames.get(i), ids);
|
||||
}
|
||||
return (idsByTable);
|
||||
}
|
||||
catch(TimeoutException e)
|
||||
{
|
||||
future.cancel(true);
|
||||
throw (new QException("Record search timed out after " + timeoutMillis + " ms", e));
|
||||
}
|
||||
catch(InterruptedException e)
|
||||
{
|
||||
if(future != null)
|
||||
{
|
||||
future.cancel(true);
|
||||
}
|
||||
Thread.currentThread().interrupt();
|
||||
throw (new QException("Record search was interrupted", e));
|
||||
}
|
||||
catch(ExecutionException e)
|
||||
{
|
||||
throw wrap("Record search failed", e.getCause() instanceof Exception cause ? cause : e);
|
||||
}
|
||||
catch(OpenSearchException | java.io.IOException e)
|
||||
{
|
||||
throw wrap("Record search failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The search query: every term of the query (operator AND) must match
|
||||
** searchableText, only the allowed tables match (an empty set matches
|
||||
** nothing), and per-field should clauses boost by configured weight.
|
||||
*******************************************************************************/
|
||||
private Query buildSearchQuery(String searchTerm, Collection<String> allowedTables, List<QuickSearchableTableConfig> tableConfigs)
|
||||
{
|
||||
List<FieldValue> allowed = allowedTables.stream().map(FieldValue::of).toList();
|
||||
|
||||
BoolQuery.Builder boolBuilder = new BoolQuery.Builder()
|
||||
.must(Query.of(q -> q.multiMatch(MultiMatchQuery.of(mm -> mm.query(searchTerm).fields("searchableText").operator(Operator.And)))))
|
||||
.filter(Query.of(q -> q.terms(t -> t.field("sourceTable").terms(tv -> tv.value(allowed)))))
|
||||
.should(buildFieldBoostQueries(searchTerm, tableConfigs, allowedTables));
|
||||
|
||||
return (Query.of(q -> q.bool(boolBuilder.build())));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -276,6 +276,7 @@ class QuickSearchQBitConfigValidationTest
|
||||
.withResponseTimeoutMillis(-1)
|
||||
.withMaxConnections(0)
|
||||
.withMaxSearchLimit(0)
|
||||
.withRecordSearchTimeoutMillis(0)
|
||||
.withBasepullRepeatSeconds(0)
|
||||
.withBasepullOverlapSeconds(-5)
|
||||
.withRunHistoryRetentionDays(0)
|
||||
@@ -284,6 +285,7 @@ class QuickSearchQBitConfigValidationTest
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("responseTimeoutMillis"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("maxConnections"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("maxSearchLimit"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("recordSearchTimeoutMillis"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("basepullRepeatSeconds"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("basepullOverlapSeconds"));
|
||||
assertThat(errors).anyMatch(e -> e.startsWith("runHistoryRetentionDays"));
|
||||
|
||||
@@ -16,10 +16,14 @@
|
||||
package com.kingsrook.qbits.quicksearch;
|
||||
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
@@ -35,6 +39,7 @@ import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.simple.SimpleSchedulerMetaData;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
|
||||
import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
@@ -99,6 +104,43 @@ class QuickSearchQBitProducerTest
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Test entity with fields core search must not get: hidden, password and
|
||||
** non-text, plus an integer field it may get.
|
||||
***************************************************************************/
|
||||
@QuickSearchable(tableName = "account")
|
||||
static class AccountEntity
|
||||
{
|
||||
@QuickSearchField
|
||||
private String name;
|
||||
|
||||
@QuickSearchField
|
||||
private Integer accountNumber;
|
||||
|
||||
@QuickSearchField
|
||||
private String secretNote;
|
||||
|
||||
@QuickSearchField
|
||||
private String password;
|
||||
|
||||
@QuickSearchField
|
||||
private Instant openedAt;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Test entity whose only searchable field is a date.
|
||||
***************************************************************************/
|
||||
@QuickSearchable(tableName = "event")
|
||||
static class EventEntity
|
||||
{
|
||||
@QuickSearchField
|
||||
private Instant occurredAt;
|
||||
}
|
||||
|
||||
|
||||
|
||||
@QuickSearchable(tableName = "person")
|
||||
static class BadFieldEntity
|
||||
{
|
||||
@@ -289,6 +331,141 @@ class QuickSearchQBitProducerTest
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** By default the producer registers the core record-search provider and
|
||||
** gives indexed tables core searchFields from their searchable fields.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testProduce_registersRecordSearchProviderAndSearchFields() throws QException
|
||||
{
|
||||
QInstance qInstance = buildQInstance();
|
||||
QContext.init(qInstance, new QSession());
|
||||
new QuickSearchQBitProducer().withConfig(buildValidConfig()).produce(qInstance);
|
||||
|
||||
assertThat(qInstance.getRecordSearchProvider()).isNotNull();
|
||||
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
|
||||
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("firstName", "lastName", "email");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** serveCoreRecordSearch false leaves core search alone.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testProduce_serveCoreRecordSearchOff_noProviderNoSearchFields() throws QException
|
||||
{
|
||||
QInstance qInstance = buildQInstance();
|
||||
QContext.init(qInstance, new QSession());
|
||||
new QuickSearchQBitProducer().withConfig(buildValidConfig().withServeCoreRecordSearch(false)).produce(qInstance);
|
||||
|
||||
assertThat(qInstance.getRecordSearchProvider()).isNull();
|
||||
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).isNull();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** searchFields the host set are kept, and a provider the host registered
|
||||
** is not replaced.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testProduce_keepsHostSearchFieldsAndProvider() throws QException
|
||||
{
|
||||
QInstance qInstance = buildQInstance();
|
||||
qInstance.getTable(SOURCE_TABLE).withSearchFields("email");
|
||||
QCodeReference hostProvider = new QCodeReference(HostRecordSearchProvider.class);
|
||||
qInstance.withRecordSearchProvider(hostProvider);
|
||||
QContext.init(qInstance, new QSession());
|
||||
|
||||
new QuickSearchQBitProducer().withConfig(buildValidConfig()).produce(qInstance);
|
||||
|
||||
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("email");
|
||||
assertThat(qInstance.getRecordSearchProvider()).isSameAs(hostProvider);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Producing twice (same provider already registered) keeps one provider.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testRegisterRecordSearchProvider_idempotent()
|
||||
{
|
||||
QInstance qInstance = new QInstance();
|
||||
QuickSearchQBitProducer.registerRecordSearchProvider(qInstance);
|
||||
QuickSearchQBitProducer.registerRecordSearchProvider(qInstance);
|
||||
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Hidden and password fields are never indexed; core searchFields only get
|
||||
** visible string or integer fields; a table with none gets no searchFields.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testProduce_searchFieldsAndIndexing_skipHiddenPasswordAndNonText() throws QException
|
||||
{
|
||||
QInstance qInstance = buildQInstance();
|
||||
qInstance.addTable(new QTableMetaData()
|
||||
.withName("account")
|
||||
.withBackendName(BACKEND_NAME)
|
||||
.withPrimaryKeyField("id")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(new QFieldMetaData("name", QFieldType.STRING))
|
||||
.withField(new QFieldMetaData("accountNumber", QFieldType.INTEGER))
|
||||
.withField(new QFieldMetaData("secretNote", QFieldType.STRING).withIsHidden(true))
|
||||
.withField(new QFieldMetaData("password", QFieldType.PASSWORD))
|
||||
.withField(new QFieldMetaData("openedAt", QFieldType.DATE_TIME))
|
||||
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
|
||||
qInstance.addTable(new QTableMetaData()
|
||||
.withName("event")
|
||||
.withBackendName(BACKEND_NAME)
|
||||
.withPrimaryKeyField("id")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(new QFieldMetaData("occurredAt", QFieldType.DATE_TIME))
|
||||
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
|
||||
QContext.init(qInstance, new QSession());
|
||||
|
||||
QuickSearchQBitConfig config = buildValidConfig().withSearchableEntityClasses(List.of(AccountEntity.class, EventEntity.class));
|
||||
new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
|
||||
|
||||
QuickSearchableTableConfig account = config.getRuntime().getTableConfig("account");
|
||||
assertThat(account.getSearchableFields()).containsExactly("name", "accountNumber", "openedAt");
|
||||
assertThat(qInstance.getTable("account").getSearchFields()).containsExactly("name", "accountNumber");
|
||||
assertThat(qInstance.getTable("event").getSearchFields()).isNull();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** A config-driven table that is not in the instance yet, or whose backend
|
||||
** uses variants, gets no searchFields (and production does not fail).
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void testDefaultCoreSearchFields_missingTableAndVariantBackend()
|
||||
{
|
||||
QInstance qInstance = buildQInstance();
|
||||
qInstance.addBackend(new QBackendMetaData().withName("variants").withBackendType(MemoryBackendModule.class).withUsesVariants(true));
|
||||
qInstance.addTable(new QTableMetaData()
|
||||
.withName("variantTable")
|
||||
.withBackendName("variants")
|
||||
.withPrimaryKeyField("id")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER))
|
||||
.withField(new QFieldMetaData("name", QFieldType.STRING)));
|
||||
|
||||
QuickSearchQBitProducer.defaultCoreSearchFields(qInstance, List.of(
|
||||
new QuickSearchableTableConfig().withTableName("notThere").withSearchableFields(List.of("name")),
|
||||
new QuickSearchableTableConfig().withTableName("variantTable").withSearchableFields(List.of("name")),
|
||||
new QuickSearchableTableConfig().withTableName(SOURCE_TABLE).withSearchableFields(List.of("lastName", "nope", "lastName"))));
|
||||
|
||||
assertThat(qInstance.getTable("variantTable").getSearchFields()).isNull();
|
||||
assertThat(qInstance.getTable(SOURCE_TABLE).getSearchFields()).containsExactly("lastName");
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testProduce_realTimeIndexingDisabled_noListener() throws QException
|
||||
{
|
||||
@@ -914,4 +1091,26 @@ class QuickSearchQBitProducerTest
|
||||
.withSearchableEntityClasses(List.of(PersonEntity.class));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** A record search provider a host registered itself.
|
||||
***************************************************************************/
|
||||
public static class HostRecordSearchProvider implements RecordSearchProviderInterface
|
||||
{
|
||||
@Override
|
||||
public boolean claimsTable(QTableMetaData table)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Override
|
||||
public RecordSearchOutput search(RecordSearchInput input)
|
||||
{
|
||||
return (new RecordSearchOutput());
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
/*
|
||||
* Copyright 2024 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.quicksearch;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
|
||||
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
|
||||
import com.kingsrook.qbits.quicksearch.actions.QuickSearchResult;
|
||||
import com.kingsrook.qbits.quicksearch.processes.FullReindexStep;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.MethodOrderer;
|
||||
import org.junit.jupiter.api.Order;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.TestMethodOrder;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.testcontainers.containers.GenericContainer;
|
||||
import org.testcontainers.containers.wait.strategy.Wait;
|
||||
import org.testcontainers.junit.jupiter.Container;
|
||||
import org.testcontainers.junit.jupiter.Testcontainers;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** End-to-end test of core record search (RecordSearchAction, the code behind
|
||||
** POST /qqq/v1/search) served by QuickSearchRecordSearchProvider from a real
|
||||
** OpenSearch (Testcontainers): the producer registers the provider and
|
||||
** searchFields, hits come back in Quick Search's relevance order and are
|
||||
** re-read by primary key (stale ids drop out, limitPerTable applies),
|
||||
** unindexed tables keep core's own search, and with OpenSearch stopped core
|
||||
** falls back to its own search without failing.
|
||||
*******************************************************************************/
|
||||
@ExtendWith(RequiresDockerCondition.class)
|
||||
@Testcontainers
|
||||
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
|
||||
class RecordSearchProviderIntegrationTest
|
||||
{
|
||||
private static final String BACKEND_NAME = "testMemoryBackend";
|
||||
private static final String PRODUCT_TABLE = "rsProduct";
|
||||
private static final String NOTE_TABLE = "rsNote";
|
||||
private static final String INDEX_NAME = "quick_search_record_search_it";
|
||||
|
||||
@Container
|
||||
static GenericContainer<?> opensearch = new GenericContainer<>(System.getProperty("opensearch.test.image", "opensearchproject/opensearch:2.19.6"))
|
||||
.withExposedPorts(9200)
|
||||
.withEnv("discovery.type", "single-node")
|
||||
.withEnv("DISABLE_SECURITY_PLUGIN", "true")
|
||||
.withEnv("DISABLE_INSTALL_DEMO_CONFIG", "true")
|
||||
.waitingFor(Wait.forHttp("/_cluster/health").forStatusCode(200));
|
||||
|
||||
private static QuickSearchQBitConfig config;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build the instance (an indexed product table and an unindexed note
|
||||
** table with its own searchFields), produce the QBit, index the products.
|
||||
*******************************************************************************/
|
||||
@BeforeAll
|
||||
static void setUpAll() throws Exception
|
||||
{
|
||||
MemoryRecordStore.getInstance().reset();
|
||||
|
||||
QInstance qInstance = new QInstance();
|
||||
qInstance.addBackend(new QBackendMetaData().withName(BACKEND_NAME).withBackendType(MemoryBackendModule.class));
|
||||
qInstance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
|
||||
|
||||
qInstance.addTable(new QTableMetaData()
|
||||
.withName(PRODUCT_TABLE)
|
||||
.withLabel("Product")
|
||||
.withBackendName(BACKEND_NAME)
|
||||
.withPrimaryKeyField("id")
|
||||
.withRecordLabelFormat("%s")
|
||||
.withRecordLabelFields("name")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(new QFieldMetaData("name", QFieldType.STRING))
|
||||
.withField(new QFieldMetaData("description", QFieldType.STRING))
|
||||
.withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)));
|
||||
|
||||
qInstance.addTable(new QTableMetaData()
|
||||
.withName(NOTE_TABLE)
|
||||
.withBackendName(BACKEND_NAME)
|
||||
.withPrimaryKeyField("id")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(new QFieldMetaData("text", QFieldType.STRING))
|
||||
.withSearchFields("text"));
|
||||
|
||||
config = new QuickSearchQBitConfig()
|
||||
.withBackendName(BACKEND_NAME)
|
||||
.withOpensearchHost(opensearch.getHost())
|
||||
.withOpensearchPort(opensearch.getMappedPort(9200))
|
||||
.withOpensearchIndexName(INDEX_NAME)
|
||||
.withSearchableTable(new SearchableTableConfig(PRODUCT_TABLE, List.of(
|
||||
new SearchableFieldConfig("name").withWeight(3),
|
||||
new SearchableFieldConfig("description"))))
|
||||
.withEnableRealTimeIndexing(false);
|
||||
|
||||
new QuickSearchQBitProducer().withConfig(config).produce(qInstance);
|
||||
QContext.init(qInstance, new QSession());
|
||||
|
||||
new InsertAction().execute(new InsertInput(PRODUCT_TABLE).withRecords(List.of(
|
||||
new QRecord().withValue("id", 1).withValue("name", "Blue Widget").withValue("description", "the original").withValue("modifyDate", Instant.now()),
|
||||
new QRecord().withValue("id", 2).withValue("name", "Widget").withValue("description", "comes in blue and green").withValue("modifyDate", Instant.now()),
|
||||
new QRecord().withValue("id", 3).withValue("name", "Red Widget").withValue("description", "not that colour").withValue("modifyDate", Instant.now()),
|
||||
new QRecord().withValue("id", 4).withValue("name", "Gadget").withValue("description", "no match here").withValue("modifyDate", Instant.now()))));
|
||||
|
||||
new InsertAction().execute(new InsertInput(NOTE_TABLE).withRecords(List.of(
|
||||
new QRecord().withValue("id", 1).withValue("text", "a blue widget note"))));
|
||||
|
||||
new FullReindexStep().run(new RunBackendStepInput(), new RunBackendStepOutput());
|
||||
config.getRuntime().getClient().refreshIndex();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@AfterAll
|
||||
static void tearDownAll()
|
||||
{
|
||||
if(config != null && config.getRuntime() != null)
|
||||
{
|
||||
config.getRuntime().close();
|
||||
}
|
||||
QContext.clear();
|
||||
QuickSearchQBitContext.clear();
|
||||
MemoryRecordStore.getInstance().reset();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static RecordSearchOutput coreSearch(String term, Integer limitPerTable) throws Exception
|
||||
{
|
||||
return (new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm(term).withLimitPerTable(limitPerTable)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static List<Serializable> idsFor(RecordSearchOutput output, String tableName)
|
||||
{
|
||||
return (output.getResults().stream().filter(r -> tableName.equals(r.getTableName())).map(RecordSearchResult::getRecordId).toList());
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
@Order(1)
|
||||
void testProducerRegisteredProviderAndSearchFields()
|
||||
{
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
assertThat(qInstance.getRecordSearchProvider().getName()).isEqualTo(QuickSearchRecordSearchProvider.class.getName());
|
||||
assertThat(qInstance.getTable(PRODUCT_TABLE).getSearchFields()).containsExactly("name", "description");
|
||||
assertThat(qInstance.getTable(NOTE_TABLE).getSearchFields()).containsExactly("text");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** "blue widget" matches products 1 and 2 in OpenSearch (all terms, any
|
||||
** field); core's contains-search would only find product 1. The order is
|
||||
** Quick Search's relevance order; labels come from core's re-read.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
@Order(2)
|
||||
void testCoreSearchServedFromOpenSearch() throws Exception
|
||||
{
|
||||
RecordSearchOutput output = coreSearch("blue widget", 5);
|
||||
|
||||
List<String> quickSearchOrder = new QuickSearchAction().execute(new QuickSearchInput().withSearchTerm("blue widget").withTableNames(List.of(PRODUCT_TABLE)))
|
||||
.getResults().stream().map(QuickSearchResult::getRecordId).toList();
|
||||
assertThat(quickSearchOrder).containsExactlyInAnyOrder("1", "2");
|
||||
|
||||
assertThat(idsFor(output, PRODUCT_TABLE)).containsExactlyElementsOf(quickSearchOrder.stream().map(Integer::valueOf).toList());
|
||||
assertThat(output.getResults()).filteredOn(r -> PRODUCT_TABLE.equals(r.getTableName())).extracting(RecordSearchResult::getRecordLabel).contains("Blue Widget", "Widget");
|
||||
assertThat(output.getResults()).filteredOn(r -> PRODUCT_TABLE.equals(r.getTableName())).extracting(RecordSearchResult::getTableLabel).containsOnly("Product");
|
||||
|
||||
////////////////////////////////////////////////
|
||||
// the unindexed table keeps core's own search //
|
||||
////////////////////////////////////////////////
|
||||
assertThat(idsFor(output, NOTE_TABLE)).containsExactly(1);
|
||||
assertThat(config.getRuntime().isRecordSearchBackingOff()).isFalse();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
@Order(3)
|
||||
void testLimitPerTable() throws Exception
|
||||
{
|
||||
assertThat(idsFor(coreSearch("widget", 1), PRODUCT_TABLE)).hasSize(1);
|
||||
assertThat(idsFor(coreSearch("widget", 5), PRODUCT_TABLE)).containsExactlyInAnyOrder(1, 2, 3);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A record deleted from the table but still in the index is dropped by
|
||||
** core's re-read.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
@Order(4)
|
||||
void testStaleIndexHitDropped() throws Exception
|
||||
{
|
||||
new DeleteAction().execute(new DeleteInput(PRODUCT_TABLE).withPrimaryKeys(List.of(1)));
|
||||
|
||||
assertThat(config.getRuntime().getClient().countDocumentsForTable(PRODUCT_TABLE)).as("the deleted record is still indexed").isEqualTo(4L);
|
||||
|
||||
assertThat(idsFor(coreSearch("blue widget", 5), PRODUCT_TABLE)).containsExactly(2);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** With OpenSearch stopped, core search still answers, from core's own
|
||||
** contains-search over the searchFields Quick Search set, and the
|
||||
** provider backs off.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
@Order(5)
|
||||
void testFallbackWhenOpenSearchStopped() throws Exception
|
||||
{
|
||||
opensearch.stop();
|
||||
|
||||
long start = System.currentTimeMillis();
|
||||
RecordSearchOutput output = coreSearch("widget", 5);
|
||||
assertThat(System.currentTimeMillis() - start).isLessThan(15_000);
|
||||
|
||||
assertThat(idsFor(output, PRODUCT_TABLE)).containsExactly(2, 3);
|
||||
assertThat(config.getRuntime().isRecordSearchBackingOff()).isTrue();
|
||||
|
||||
assertThat(idsFor(coreSearch("blue", 5), PRODUCT_TABLE)).containsExactly(2);
|
||||
}
|
||||
|
||||
}
|
||||
+360
@@ -0,0 +1,360 @@
|
||||
/*
|
||||
* Copyright 2024 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.quicksearch.actions;
|
||||
|
||||
|
||||
import java.net.InetAddress;
|
||||
import java.net.ServerSocket;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
|
||||
import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
|
||||
import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
import static org.assertj.core.api.Assertions.tuple;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.ArgumentMatchers.anyInt;
|
||||
import static org.mockito.ArgumentMatchers.anyList;
|
||||
import static org.mockito.ArgumentMatchers.anyString;
|
||||
import static org.mockito.ArgumentMatchers.eq;
|
||||
import static org.mockito.Mockito.doReturn;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.spy;
|
||||
import static org.mockito.Mockito.times;
|
||||
import static org.mockito.Mockito.verify;
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Unit tests for QuickSearchRecordSearchProvider: claim logic, result
|
||||
** mapping, order and per-table id cap, failure and timeout handling, and
|
||||
** core RecordSearchAction end to end (re-read by primary key, fallback to
|
||||
** core's own search) over the memory backend with a mock OpenSearch client.
|
||||
*******************************************************************************/
|
||||
class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
|
||||
{
|
||||
private static final String OTHER_TABLE = "otherEntity";
|
||||
private static final String UNINDEXED_TABLE = "unindexedEntity";
|
||||
|
||||
private QuickSearchQBitConfig config;
|
||||
private QuickSearchRuntime runtime;
|
||||
private QuickSearchOpenSearchClient mockClient;
|
||||
private QuickSearchRecordSearchProvider provider;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Register a Quick Search QBit whose runtime returns a mock client, with
|
||||
** testEntity and otherEntity indexed; add the provider to the instance.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws QException
|
||||
{
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
qInstance.addTable(simpleTable(OTHER_TABLE));
|
||||
qInstance.addTable(simpleTable(UNINDEXED_TABLE));
|
||||
qInstance.getTable(TEST_ENTITY_TABLE).withSearchFields("name", "description");
|
||||
qInstance.withRecordSearchProvider(new QCodeReference(QuickSearchRecordSearchProvider.class));
|
||||
|
||||
config = new QuickSearchQBitConfig()
|
||||
.withBackendName(TEST_BACKEND_NAME)
|
||||
.withOpensearchHost("localhost")
|
||||
.withOpensearchPort(9200)
|
||||
.withOpensearchIndexName("test-index")
|
||||
.withRecordSearchTimeoutMillis(1234);
|
||||
|
||||
mockClient = mock(QuickSearchOpenSearchClient.class);
|
||||
runtime = registerRuntime(config, mockClient);
|
||||
provider = new QuickSearchRecordSearchProvider();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Register the QBit with a runtime (a spy whose client is the given one,
|
||||
** or the real lazily built client when null).
|
||||
*******************************************************************************/
|
||||
private QuickSearchRuntime registerRuntime(QuickSearchQBitConfig config, QuickSearchOpenSearchClient client) throws QException
|
||||
{
|
||||
QuickSearchRuntime runtime = new QuickSearchRuntime(config);
|
||||
if(client != null)
|
||||
{
|
||||
runtime = spy(runtime);
|
||||
doReturn(client).when(runtime).getClient();
|
||||
}
|
||||
runtime.setDiscoveredTables(List.of(
|
||||
new QuickSearchableTableConfig().withTableName(TEST_ENTITY_TABLE).withPrimaryKeyField("id").withSearchableFields(List.of("name", "description")),
|
||||
new QuickSearchableTableConfig().withTableName(OTHER_TABLE).withPrimaryKeyField("id").withSearchableFields(List.of("name"))));
|
||||
config.setRuntime(runtime);
|
||||
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
if(qInstance.getQBits() != null)
|
||||
{
|
||||
qInstance.getQBits().clear();
|
||||
}
|
||||
qInstance.addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config));
|
||||
return (runtime);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static QTableMetaData simpleTable(String name)
|
||||
{
|
||||
return (new QTableMetaData()
|
||||
.withName(name)
|
||||
.withBackendName(TEST_BACKEND_NAME)
|
||||
.withPrimaryKeyField("id")
|
||||
.withField(new QFieldMetaData("id", QFieldType.INTEGER).withIsEditable(false))
|
||||
.withField(new QFieldMetaData("name", QFieldType.STRING))
|
||||
.withSearchFields("name"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Insert testEntity rows with ids 1..n named "<prefix> <i>".
|
||||
*******************************************************************************/
|
||||
private void insertEntities(String prefix, int count) throws QException
|
||||
{
|
||||
List<QRecord> records = new ArrayList<>();
|
||||
for(int i = 1; i <= count; i++)
|
||||
{
|
||||
records.add(new QRecord().withValue("id", i).withValue("name", prefix + " " + i).withValue("description", "row " + i));
|
||||
}
|
||||
new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecords(records));
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testClaimsTable_indexedEnabledTablesOnly()
|
||||
{
|
||||
QInstance qInstance = QContext.getQInstance();
|
||||
assertThat(provider.claimsTable(qInstance.getTable(TEST_ENTITY_TABLE))).isTrue();
|
||||
assertThat(provider.claimsTable(qInstance.getTable(OTHER_TABLE))).isTrue();
|
||||
assertThat(provider.claimsTable(qInstance.getTable(UNINDEXED_TABLE))).isFalse();
|
||||
assertThat(provider.claimsTable(null)).isFalse();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testClaimsTable_disabledTable_notClaimed() throws QException
|
||||
{
|
||||
new InsertAction().execute(new InsertInput(config.getQuickSearchIndexTableName()).withRecords(List.of(
|
||||
new QRecord().withValue("tableName", TEST_ENTITY_TABLE).withValue("enabled", false))));
|
||||
runtime.invalidateEnabledCache();
|
||||
|
||||
assertThat(provider.claimsTable(QContext.getQInstance().getTable(TEST_ENTITY_TABLE))).isFalse();
|
||||
assertThat(provider.claimsTable(QContext.getQInstance().getTable(OTHER_TABLE))).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testClaimsTable_serveCoreRecordSearchOff_orBackingOff_orNoRuntime()
|
||||
{
|
||||
QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
|
||||
|
||||
config.setServeCoreRecordSearch(false);
|
||||
assertThat(provider.claimsTable(table)).isFalse();
|
||||
config.setServeCoreRecordSearch(true);
|
||||
|
||||
runtime.markRecordSearchFailed();
|
||||
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
|
||||
assertThat(provider.claimsTable(table)).isFalse();
|
||||
runtime.clearRecordSearchBackoff();
|
||||
assertThat(provider.claimsTable(table)).isTrue();
|
||||
|
||||
QContext.getQInstance().getQBits().clear();
|
||||
assertThat(provider.claimsTable(table)).isFalse();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testSearch_mapsIdsPerTableInOrder_andCapsIdsPerTable() throws QException
|
||||
{
|
||||
Map<String, List<String>> ids = new LinkedHashMap<>();
|
||||
ids.put(OTHER_TABLE, List.of("7"));
|
||||
ids.put(TEST_ENTITY_TABLE, List.of("3", "1", "2"));
|
||||
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
|
||||
|
||||
RecordSearchOutput output = provider.search(new RecordSearchInput()
|
||||
.withSearchTerm(" widget blue ")
|
||||
.withTableNames(List.of(TEST_ENTITY_TABLE, UNINDEXED_TABLE, OTHER_TABLE, TEST_ENTITY_TABLE))
|
||||
.withLimitPerTable(5));
|
||||
|
||||
assertThat(output.getResults()).extracting(RecordSearchResult::getTableName, RecordSearchResult::getRecordId).containsExactly(
|
||||
tuple(TEST_ENTITY_TABLE, "3"),
|
||||
tuple(TEST_ENTITY_TABLE, "1"),
|
||||
tuple(TEST_ENTITY_TABLE, "2"),
|
||||
tuple(OTHER_TABLE, "7"));
|
||||
|
||||
verify(mockClient).searchRecordIdsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
|
||||
|
||||
provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(25));
|
||||
verify(mockClient).searchRecordIdsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), anyList(), eq(1234));
|
||||
|
||||
provider.search(new RecordSearchInput().withSearchTerm("gadget").withTableNames(List.of(TEST_ENTITY_TABLE)));
|
||||
verify(mockClient).searchRecordIdsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testSearch_shortTermOrNoIndexedTables_returnsEmptyWithoutOpenSearch() throws QException
|
||||
{
|
||||
assertThat(provider.search(new RecordSearchInput().withSearchTerm("x").withTableNames(List.of(TEST_ENTITY_TABLE))).getResults()).isEmpty();
|
||||
assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(UNINDEXED_TABLE))).getResults()).isEmpty();
|
||||
assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget")).getResults()).isEmpty();
|
||||
verify(mockClient, never()).searchRecordIdsPerTable(any(), any(), anyInt(), any(), anyInt());
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testSearch_clientFailure_throwsAndBacksOff() throws QException
|
||||
{
|
||||
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("boom"));
|
||||
|
||||
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
|
||||
.isInstanceOf(QException.class).hasMessageContaining("boom");
|
||||
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Test
|
||||
void testSearch_uncheckedClientFailure_wrappedAsQException() throws QException
|
||||
{
|
||||
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed"));
|
||||
|
||||
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
|
||||
.isInstanceOf(QException.class).hasMessageContaining("closed");
|
||||
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A server that accepts connections and never answers (a hung OpenSearch):
|
||||
** the real client gives up after recordSearchTimeoutMillis with a
|
||||
** QException, well before the transport's response timeout.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testSearch_hungOpenSearch_timesOutWithQException() throws Exception
|
||||
{
|
||||
try(ServerSocket hung = new ServerSocket(0, 50, InetAddress.getLoopbackAddress()))
|
||||
{
|
||||
QuickSearchQBitConfig hungConfig = new QuickSearchQBitConfig()
|
||||
.withBackendName(TEST_BACKEND_NAME)
|
||||
.withOpensearchHost("127.0.0.1")
|
||||
.withOpensearchPort(hung.getLocalPort())
|
||||
.withOpensearchIndexName("test-index")
|
||||
.withResponseTimeoutMillis(60_000)
|
||||
.withRecordSearchTimeoutMillis(300);
|
||||
QuickSearchRuntime hungRuntime = registerRuntime(hungConfig, null);
|
||||
|
||||
long start = System.currentTimeMillis();
|
||||
assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
|
||||
.isInstanceOf(QException.class).hasMessageContaining("timed out after 300 ms");
|
||||
assertThat(System.currentTimeMillis() - start).isLessThan(10_000);
|
||||
assertThat(hungRuntime.isRecordSearchBackingOff()).isTrue();
|
||||
assertThat(provider.claimsTable(QContext.getQInstance().getTable(TEST_ENTITY_TABLE))).isFalse();
|
||||
|
||||
hungRuntime.close();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Core RecordSearchAction re-reads the provider's ids by primary key, keeps
|
||||
** its order, drops ids that no longer exist, applies limitPerTable, and
|
||||
** fills record labels; unclaimed tables keep core's own search.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testCoreRecordSearchAction_usesProviderIds() throws QException
|
||||
{
|
||||
insertEntities("Widget", 6);
|
||||
new InsertAction().execute(new InsertInput(UNINDEXED_TABLE).withRecords(List.of(new QRecord().withValue("id", 1).withValue("name", "Widget elsewhere"))));
|
||||
|
||||
Map<String, List<String>> ids = new LinkedHashMap<>();
|
||||
ids.put(TEST_ENTITY_TABLE, List.of("5", "999", "2", "4"));
|
||||
ids.put(OTHER_TABLE, List.of());
|
||||
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
|
||||
|
||||
RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withLimitPerTable(2));
|
||||
|
||||
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(5, 2);
|
||||
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(UNINDEXED_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(1);
|
||||
assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(OTHER_TABLE)).isEmpty();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** When the provider fails, core searches the claimed tables itself.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testCoreRecordSearchAction_providerFailure_fallsBackToCoreSearch() throws QException
|
||||
{
|
||||
insertEntities("Widget", 3);
|
||||
when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down"));
|
||||
|
||||
RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
|
||||
|
||||
assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId).containsExactly(1, 2, 3);
|
||||
assertThat(runtime.isRecordSearchBackingOff()).isTrue();
|
||||
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
// while backing off, the provider claims nothing: OpenSearch is not //
|
||||
// called again and core still answers //
|
||||
//////////////////////////////////////////////////////////////////////
|
||||
RecordSearchOutput again = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
|
||||
assertThat(again.getResults()).hasSize(3);
|
||||
verify(mockClient, times(1)).searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt());
|
||||
}
|
||||
|
||||
}
|
||||
Reference in new issue
Block a user