From 5c8320bf29626136a58aa2c1c90c23fb2e4f60ea Mon Sep 17 00:00:00 2001 From: James Maes Date: Mon, 12 Oct 2026 05:49:19 -0500 Subject: [PATCH] feat(search): scores, highlights, totals and paging for core record search QuickSearchRecordSearchProvider now fills the QQQ 4.1.0-RC.4 record search contract: each hit's relevance score, a plain-text highlight of the matching fieldValues.* fields (OpenSearch highlighting with private-use markers, converted to text plus match ranges), each table's total hits (totalCountByTable, counted to 10,000), and the request offset (from). RECENT sort keeps relevance order: the index holds no record modify dates. New client method searchRecordsPerTable; searchRecordIdsPerTable delegates to it. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 +- README.md | 4 +- .../QuickSearchRecordSearchProvider.java | 132 ++++++++++++++++-- .../QuickSearchOpenSearchClient.java | 102 +++++++++++--- .../RecordSearchProviderIntegrationTest.java | 46 ++++++ .../QuickSearchRecordSearchProviderTest.java | 110 +++++++++++++-- 6 files changed, 358 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 382163a..8881a43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,13 +4,14 @@ All notable changes to this project are documented here. The format follows Keep ## [1.1.0] - Unreleased -Requires QQQ 4.1.0-RC.2 or later, the first release that carries the record-search provider SPI; built against 4.1.0-RC.2. See `docs/MIGRATION-1.1.md`. +Requires QQQ 4.1.0-RC.4 or later (the record-search provider SPI arrived in 4.1.0-RC.2; its scores, highlights, totals and paging in 4.1.0-RC.4); built against 4.1.0-RC.4. See `docs/MIGRATION-1.1.md`. ### Added - `QuickSearchRecordSearchProvider`: serves QQQ core record search (`RecordSearchAction`, `POST /qqq/v1/search`, the Next UI global search) from OpenSearch for the tables Quick Search indexes. One `_msearch` request with a per-table top-N search (same query, AND semantics and field weights as `QuickSearchAction`); returns record ids only, which core re-reads through `QueryAction` (permissions, record locks, labels). Audit finding F-11, decision D1. - `serveCoreRecordSearch` (default `true`): registers the provider and gives indexed tables core `searchFields` (visible string and integer searchable fields) when they have none. Host `searchFields` and a host-registered provider are never replaced. - `recordSearchTimeoutMillis` (default 2500): wall-clock limit on the OpenSearch request; on timeout or any failure core falls back to its own search, and the provider claims no tables for 30 seconds. - `QuickSearchOpenSearchClient.searchRecordIdsPerTable`. +- Record search details for QQQ 4.1.0-RC.4's record-search contract: the provider passes each hit's relevance `score`, a plain-text highlight (match ranges, from OpenSearch highlighting of `fieldValues.*` with private-use markers) and each table's total hits (`totalCountByTable`, counted to 10,000), and honors the request `offset` (`from`). `sort: RECENT` keeps relevance order (the index has no record modify dates). New `QuickSearchOpenSearchClient.searchRecordsPerTable`; `searchRecordIdsPerTable` now delegates to it. This requires QQQ 4.1.0-RC.4 or later. ### Security - Fields of type `PASSWORD` are no longer indexed (as hidden fields already were). Reconcile or fully reindex once to remove values a 1.0 index may hold. diff --git a/README.md b/README.md index dbc8080..7255d0a 100644 --- a/README.md +++ b/README.md @@ -130,11 +130,11 @@ Search runs as the current `QContext` session: tables the session cannot read ar QQQ core has its own record search (`RecordSearchAction`, served at `POST /qqq/v1/search` and used by the Next UI's global search box): a case-insensitive contains-query over each table's `searchFields`. With `serveCoreRecordSearch` (default `true`) the producer registers `QuickSearchRecordSearchProvider` on the instance, and core asks it to search every table Quick Search indexes. The API and the UI do not change. - **What is served from OpenSearch.** Tables this QBit indexes and that are enabled in `quickSearchIndex`. Core offers a table only when it has `searchFields`, the session may read it, and it supports query, so the producer gives each indexed table that has no `searchFields` its visible string and integer searchable fields (never hidden or password fields). `searchFields` you set yourself are kept. A table must be in the `QInstance` before the QBit is produced to get them. -- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights); each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order. Core re-reads them by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, and records deleted since they were indexed drop out. +- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights), skipping the request's `offset`; each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order, each with its score and a highlight of the matching `fieldValues.*` fields (plain text plus match ranges), and the table's total hits (counted to 10,000). Core re-reads the ids by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, records deleted since they were indexed drop out, and only highlights of fields the user can see (where core's own highlighting found no match) reach the response, with `tables[].totalCount` and `hasMore`. The index holds no record modify dates, so `sort: RECENT` gets relevance order for indexed tables. - **Failure.** The OpenSearch request is cancelled after `recordSearchTimeoutMillis` (default 2500). Any failure (timeout, cluster down, index missing) makes core search those tables with its own contains-query instead, so search degrades but never fails. After a failure, Quick Search claims no tables for 30 seconds, so a down cluster does not add a timeout to every search. - **Opting out.** `withServeCoreRecordSearch(false)` leaves core search and `searchFields` untouched. If the host already registered another record-search provider, the QBit keeps it and logs a warning (an instance has one provider). -`QuickSearchAction` remains the richer API (scores, highlights, pagination, totals). +`QuickSearchAction` remains the direct API (all hits in one ranked list, `searchableText` highlights, pagination across tables). ### Processes diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java index 409c427..d1eb081 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java @@ -18,6 +18,7 @@ package com.kingsrook.qbits.quicksearch.actions; import java.util.ArrayList; +import java.util.HashMap; import java.util.LinkedHashSet; import java.util.List; import java.util.Map; @@ -25,6 +26,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction; import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult; @@ -32,6 +35,9 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.utils.CollectionUtils; import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits; import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; @@ -48,10 +54,14 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; ** ** A table is claimed when it is indexed by this QBit and enabled, and core ** record search is not backing off after a recent failure. search() sends - ** one _msearch request with a top-N search per table and returns record ids - ** only, best first; core re-reads them by primary key through QueryAction, - ** so table permissions, record security locks and record labels are core's - ** job, and ids of deleted records drop out. Each table gets + ** one _msearch request with a top-N search per table (after skipping the + ** input's offset) and returns record ids, best first, each with its score + ** and highlights (plain text plus match ranges, from the indexed field + ** values), and each table's total hits; core re-reads the ids by primary key + ** through QueryAction, so table permissions, record security locks and + ** record labels are core's job, and ids of deleted records drop out (core + ** also keeps only highlights of fields the session can see). The index has + ** no record modify dates, so a RECENT sort gets relevance order. Each table gets ** IDS_PER_LIMIT_MULTIPLIER x limitPerTable ids (at most ** RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE) so dropouts do not leave ** the table short. @@ -67,6 +77,8 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte public static final int IDS_PER_LIMIT_MULTIPLIER = 4; + private static final String FIELD_VALUES_PREFIX = "fieldValues."; + /******************************************************************************* @@ -126,10 +138,12 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte int idsPerTable = Math.min(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE, limitPerTable * IDS_PER_LIMIT_MULTIPLIER); int timeoutMillis = config.getRecordSearchTimeoutMillis() == null ? QuickSearchQBitConfig.DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS : config.getRecordSearchTimeoutMillis(); - Map> idsByTable; + int offset = input.getOffset() == null ? 0 : Math.max(0, input.getOffset()); + + Map hitsByTable; try { - idsByTable = runtime.getClient().searchRecordIdsPerTable(term, tableNames, idsPerTable, runtime.getDiscoveredTables(), timeoutMillis); + hitsByTable = runtime.getClient().searchRecordsPerTable(term, tableNames, idsPerTable, offset, true, runtime.getDiscoveredTables(), timeoutMillis); } catch(QException | RuntimeException e) { @@ -139,15 +153,115 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte throw (e instanceof QException qe ? qe : new QException("Quick Search record search failed: " + e.getMessage(), e)); } + Map totalCountByTable = new HashMap<>(); for(String tableName : tableNames) { - for(String recordId : CollectionUtils.nonNullList(idsByTable.get(tableName))) + RecordSearchTableHits tableHits = hitsByTable.get(tableName); + if(tableHits == null) { - results.add(new RecordSearchResult().withTableName(tableName).withRecordId(recordId)); + continue; + } + + if(tableHits.totalHits() != null) + { + totalCountByTable.put(tableName, (int) Math.min(Integer.MAX_VALUE, tableHits.totalHits())); + } + + for(RecordSearchHit hit : CollectionUtils.nonNullList(tableHits.hits())) + { + List highlights = highlights(hit.highlights()); + results.add(new RecordSearchResult() + .withTableName(tableName) + .withRecordId(hit.recordId()) + .withScore(hit.score()) + .withHighlights(highlights.isEmpty() ? null : highlights)); } } - return (new RecordSearchOutput().withResults(results)); + return (new RecordSearchOutput().withResults(results).withTotalCountByTable(totalCountByTable)); + } + + + + /******************************************************************************* + ** A hit's highlight fragments (by index field) as core record search + ** highlights: one per fieldValues.{fieldName} field, as plain text with + ** match ranges. Other index fields, and fragments without a match, are + ** skipped. + *******************************************************************************/ + public static List highlights(Map> fragmentsByIndexField) + { + List highlights = new ArrayList<>(); + for(Map.Entry> entry : CollectionUtils.nonNullMap(fragmentsByIndexField).entrySet()) + { + String indexField = entry.getKey(); + if(indexField == null || !indexField.startsWith(FIELD_VALUES_PREFIX) || indexField.length() == FIELD_VALUES_PREFIX.length() || CollectionUtils.nullSafeIsEmpty(entry.getValue())) + { + continue; + } + + RecordSearchHighlight highlight = highlight(indexField.substring(FIELD_VALUES_PREFIX.length()), entry.getValue().get(0)); + if(highlight != null) + { + highlights.add(highlight); + } + } + return (highlights); + } + + + + /******************************************************************************* + ** A highlight from a fragment whose matches are wrapped in the client's + ** pre/post tags: the fragment's text without the tags, and the ranges they + ** wrapped (adjacent ranges merged). Null if the fragment is blank or has + ** no match. + *******************************************************************************/ + public static RecordSearchHighlight highlight(String fieldName, String fragment) + { + if(fragment == null) + { + return (null); + } + + StringBuilder text = new StringBuilder(); + List matches = new ArrayList<>(); + int open = -1; + for(int i = 0; i < fragment.length(); i++) + { + char c = fragment.charAt(i); + if(c == QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG.charAt(0)) + { + open = text.length(); + } + else if(c == QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG.charAt(0)) + { + if(open >= 0 && text.length() > open) + { + RecordSearchHighlightMatch last = matches.isEmpty() ? null : matches.get(matches.size() - 1); + if(last != null && last.getEnd() == open) + { + last.setEnd(text.length()); + } + else + { + matches.add(new RecordSearchHighlightMatch().withStart(open).withEnd(text.length())); + } + } + open = -1; + } + else + { + text.append(c); + } + } + + if(matches.isEmpty() || text.toString().isBlank()) + { + return (null); + } + + return (new RecordSearchHighlight().withFieldName(fieldName).withText(text.toString()).withMatches(matches)); } } diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java index ee1e273..ef24f0b 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java @@ -103,6 +103,14 @@ public class QuickSearchOpenSearchClient implements Closeable public static final String SEARCH_ANALYZER = "quick_search_search_analyzer"; public static final int MAX_RESULT_WINDOW = 10_000; + //////////////////////////////////////////////////////////////////////////// + // record search highlight markers: private-use characters, which do not // + // occur in indexed text, so fragments convert to plain text and ranges // + //////////////////////////////////////////////////////////////////////////// + public static final String HIGHLIGHT_PRE_TAG = "\uE000"; // private use area + public static final String HIGHLIGHT_POST_TAG = "\uE001"; // private use area + public static final int HIGHLIGHT_FRAGMENT_SIZE = 160; + private final OpenSearchClient client; private final OpenSearchAsyncClient asyncClient; private final OpenSearchTransport transport; @@ -688,36 +696,94 @@ public class QuickSearchOpenSearchClient implements Closeable /******************************************************************************* - ** Record ids of the best matches in each table, best first, for core record - ** search. One _msearch request with one search per table (the same query and - ** ranking as {@link #search}, restricted to that table), so every table gets - ** its own top idsPerTable hits. Only sourceTable and recordId are fetched. + ** One hit of a record search: the record id, its relevance score, and its + ** highlight fragments by index field (e.g., "fieldValues.firstName"), with + ** matches between {@link #HIGHLIGHT_PRE_TAG} and {@link #HIGHLIGHT_POST_TAG}. + *******************************************************************************/ + public record RecordSearchHit(String recordId, Double score, Map> highlights) + { + } + + + + /******************************************************************************* + ** The hits of a record search in one table, best first, and the table's + ** total number of matches (counted up to {@link #MAX_RESULT_WINDOW}; null + ** if unknown). + *******************************************************************************/ + public record RecordSearchTableHits(List hits, Long totalHits) + { + } + + + + /******************************************************************************* + ** Record ids of the best matches in each table, best first - see + ** {@link #searchRecordsPerTable}, which also returns scores, highlights and + ** totals. + *******************************************************************************/ + public Map> searchRecordIdsPerTable(String searchTerm, List tableNames, int idsPerTable, List tableConfigs, int timeoutMillis) throws QException + { + Map> idsByTable = new LinkedHashMap<>(); + for(Map.Entry entry : searchRecordsPerTable(searchTerm, tableNames, idsPerTable, 0, false, tableConfigs, timeoutMillis).entrySet()) + { + idsByTable.put(entry.getKey(), entry.getValue().hits().stream().map(RecordSearchHit::recordId).toList()); + } + return (idsByTable); + } + + + + /******************************************************************************* + ** The best matches in each table, best first, for core record search. One + ** _msearch request with one search per table (the same query and ranking as + ** {@link #search}, restricted to that table), so every table gets its own + ** top hitsPerTable hits, after skipping offset. Only sourceTable and + ** recordId are fetched, plus each hit's score, the table's total hits + ** (counted up to MAX_RESULT_WINDOW) and - when highlight is set - one + ** highlight fragment per matching fieldValues.* field. ** ** The whole request is bounded by timeoutMillis of wall-clock time (and the ** same server-side search timeout): when it runs out, the request is ** cancelled and a QException is thrown. A failed search for any table fails ** the call. Returns every requested table as a key, in the given order. *******************************************************************************/ - public Map> searchRecordIdsPerTable(String searchTerm, List tableNames, int idsPerTable, List tableConfigs, int timeoutMillis) throws QException + public Map searchRecordsPerTable(String searchTerm, List tableNames, int hitsPerTable, int offset, boolean highlight, List tableConfigs, int timeoutMillis) throws QException { - Map> idsByTable = new LinkedHashMap<>(); + Map hitsByTable = new LinkedHashMap<>(); if(tableNames == null || tableNames.isEmpty()) { - return (idsByTable); + return (hitsByTable); } + int from = Math.max(0, Math.min(offset, MAX_RESULT_WINDOW - hitsPerTable)); + List searches = new ArrayList<>(); for(String tableName : tableNames) { Query query = buildSearchQuery(searchTerm, List.of(tableName), tableConfigs); searches.add(RequestItem.of(ri -> ri .header(h -> h.index(indexName)) - .body(b -> b - .query(query) - .size(idsPerTable) - .trackTotalHits(t -> t.enabled(false)) - .source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId"))) - .timeout(timeoutMillis + "ms")))); + .body(b -> + { + b.query(query) + .from(from) + .size(hitsPerTable) + .trackTotalHits(t -> t.count(MAX_RESULT_WINDOW)) + .source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId"))) + .timeout(timeoutMillis + "ms"); + if(highlight) + { + b.highlight(h -> h + .preTags(HIGHLIGHT_PRE_TAG) + .postTags(HIGHLIGHT_POST_TAG) + .requireFieldMatch(false) + .numberOfFragments(1) + .fragmentSize(HIGHLIGHT_FRAGMENT_SIZE) + .fields("fieldValues.*", HighlightField.of(hf -> hf))); + } + return (b); + }))); } CompletableFuture> future = null; @@ -741,17 +807,19 @@ public class QuickSearchOpenSearchClient implements Closeable throw (new QException("Record search failed for table [" + tableNames.get(i) + "]: " + reason)); } - List ids = new ArrayList<>(); + List hits = new ArrayList<>(); for(Hit hit : item.result().hits().hits()) { if(hit.source() != null && hit.source().getRecordId() != null) { - ids.add(hit.source().getRecordId()); + hits.add(new RecordSearchHit(hit.source().getRecordId(), hit.score(), hit.highlight())); } } - idsByTable.put(tableNames.get(i), ids); + + Long totalHits = item.result().hits().total() == null ? null : item.result().hits().total().value(); + hitsByTable.put(tableNames.get(i), new RecordSearchTableHits(hits, totalHits)); } - return (idsByTable); + return (hitsByTable); } catch(TimeoutException e) { diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/RecordSearchProviderIntegrationTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/RecordSearchProviderIntegrationTest.java index fd56b2c..4d8647c 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/RecordSearchProviderIntegrationTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/RecordSearchProviderIntegrationTest.java @@ -20,6 +20,7 @@ package com.kingsrook.qbits.quicksearch; import java.io.Serializable; import java.time.Instant; import java.util.List; +import java.util.Map; import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction; @@ -28,9 +29,14 @@ import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInpu import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchProviderState; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchSource; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchTableSummary; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData; @@ -46,6 +52,8 @@ import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction; import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput; import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider; import com.kingsrook.qbits.quicksearch.actions.QuickSearchResult; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits; import com.kingsrook.qbits.quicksearch.processes.FullReindexStep; import org.junit.jupiter.api.AfterAll; import org.junit.jupiter.api.BeforeAll; @@ -59,6 +67,7 @@ import org.testcontainers.containers.wait.strategy.Wait; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.tuple; /******************************************************************************* @@ -230,6 +239,43 @@ class RecordSearchProviderIntegrationTest + /******************************************************************************* + ** Scores, totals and paging come from OpenSearch; the client's highlight + ** fragments are marked on the fieldValues.* fields and convert to plain + ** text and ranges; core highlights the matching non-label field. + *******************************************************************************/ + @Test + @Order(2) + void testScoresTotalsPagingAndHighlights() throws Exception + { + RecordSearchInput input = new RecordSearchInput().withSearchTerm("blue widget").withTableNames(List.of(PRODUCT_TABLE)).withLimitPerTable(1); + RecordSearchOutput page1 = new RecordSearchAction().execute(input); + assertThat(page1.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.OK); + RecordSearchTableSummary products = page1.getTables().get(0); + assertThat(products.getSource()).isEqualTo(RecordSearchSource.PROVIDER); + assertThat(products.getTotalCount()).isEqualTo(2); + assertThat(products.getHasMore()).isTrue(); + assertThat(page1.getResults()).hasSize(1); + assertThat(page1.getResults().get(0).getScore()).isPositive(); + + RecordSearchOutput page2 = new RecordSearchAction().execute(input.withOffset(1)); + assertThat(page2.getTables().get(0).getHasMore()).isFalse(); + assertThat(List.of(page1.getResults().get(0).getRecordId(), page2.getResults().get(0).getRecordId())).containsExactlyInAnyOrder(1, 2); + + RecordSearchResult widget = coreSearch("blue", 5).getResults().stream().filter(r -> Integer.valueOf(2).equals(r.getRecordId())).findFirst().orElseThrow(); + assertThat(widget.getHighlights()).extracting(RecordSearchHighlight::getFieldName).containsExactly("description"); + assertThat(widget.getHighlights().get(0).getText()).isEqualTo("comes in blue and green"); + + Map hits = config.getRuntime().getClient().searchRecordsPerTable("blue", List.of(PRODUCT_TABLE), 5, 0, true, config.getRuntime().getDiscoveredTables(), 10_000); + RecordSearchHit hit = hits.get(PRODUCT_TABLE).hits().stream().filter(h -> "2".equals(h.recordId())).findFirst().orElseThrow(); + List highlights = QuickSearchRecordSearchProvider.highlights(hit.highlights()); + assertThat(highlights).extracting(RecordSearchHighlight::getFieldName).containsExactly("description"); + assertThat(highlights.get(0).getText()).isEqualTo("comes in blue and green"); + assertThat(highlights.get(0).getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(9, 13)); + } + + + @Test @Order(3) void testLimitPerTable() throws Exception diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProviderTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProviderTest.java index 779f93f..d8b363f 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProviderTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProviderTest.java @@ -28,9 +28,14 @@ import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchProviderState; import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchSource; +import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchTableSummary; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; @@ -43,12 +48,15 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.assertj.core.api.Assertions.tuple; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.ArgumentMatchers.anyList; import static org.mockito.ArgumentMatchers.anyString; @@ -218,7 +226,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest Map> ids = new LinkedHashMap<>(); ids.put(OTHER_TABLE, List.of("7")); ids.put(TEST_ENTITY_TABLE, List.of("3", "1", "2")); - when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hits(ids)); RecordSearchOutput output = provider.search(new RecordSearchInput() .withSearchTerm(" widget blue ") @@ -231,13 +239,13 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest tuple(TEST_ENTITY_TABLE, "2"), tuple(OTHER_TABLE, "7")); - verify(mockClient).searchRecordIdsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234)); + verify(mockClient).searchRecordsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(0), eq(true), anyList(), eq(1234)); provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(25)); - verify(mockClient).searchRecordIdsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), anyList(), eq(1234)); + verify(mockClient).searchRecordsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), eq(0), eq(true), anyList(), eq(1234)); provider.search(new RecordSearchInput().withSearchTerm("gadget").withTableNames(List.of(TEST_ENTITY_TABLE))); - verify(mockClient).searchRecordIdsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234)); + verify(mockClient).searchRecordsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(0), eq(true), anyList(), eq(1234)); } @@ -248,7 +256,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest assertThat(provider.search(new RecordSearchInput().withSearchTerm("x").withTableNames(List.of(TEST_ENTITY_TABLE))).getResults()).isEmpty(); assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(UNINDEXED_TABLE))).getResults()).isEmpty(); assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget")).getResults()).isEmpty(); - verify(mockClient, never()).searchRecordIdsPerTable(any(), any(), anyInt(), any(), anyInt()); + verify(mockClient, never()).searchRecordsPerTable(any(), any(), anyInt(), anyInt(), anyBoolean(), any(), anyInt()); } @@ -256,7 +264,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest @Test void testSearch_clientFailure_throwsAndBacksOff() throws QException { - when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("boom")); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new QException("boom")); assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)))) .isInstanceOf(QException.class).hasMessageContaining("boom"); @@ -268,7 +276,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest @Test void testSearch_uncheckedClientFailure_wrappedAsQException() throws QException { - when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed")); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed")); assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)))) .isInstanceOf(QException.class).hasMessageContaining("closed"); @@ -276,6 +284,82 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest } + /******************************************************************************* + ** Client hits for these ids per table (no scores, highlights or totals). + *******************************************************************************/ + private static Map hits(Map> idsByTable) + { + Map hitsByTable = new LinkedHashMap<>(); + idsByTable.forEach((tableName, ids) -> hitsByTable.put(tableName, new RecordSearchTableHits(ids.stream().map(id -> new RecordSearchHit(id, null, null)).toList(), null))); + return (hitsByTable); + } + + + + /******************************************************************************* + ** A fragment with these words marked as matches, as the client returns it. + *******************************************************************************/ + private static String marked(String word) + { + return (QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG + word + QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG); + } + + + + /******************************************************************************* + ** Scores, highlights (fieldValues.* only, as plain text and ranges) and + ** totals reach core; the offset reaches the client. + *******************************************************************************/ + @Test + void testSearch_returnsScoresHighlightsAndTotals_andPassesOffset() throws QException + { + Map hitsByTable = new LinkedHashMap<>(); + hitsByTable.put(TEST_ENTITY_TABLE, new RecordSearchTableHits(List.of( + new RecordSearchHit("3", 7.5, Map.of( + "fieldValues.name", List.of("big " + marked("Widget") + " for " + marked("blue") + marked("s")), + "searchableText", List.of(marked("Widget")))), + new RecordSearchHit("1", 2.25, Map.of("fieldValues.name", List.of("no match here")))), 42L)); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hitsByTable); + + RecordSearchOutput output = provider.search(new RecordSearchInput().withSearchTerm("widget blue").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(5).withOffset(10)); + + assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId, RecordSearchResult::getScore).containsExactly(tuple("3", 7.5), tuple("1", 2.25)); + assertThat(output.getTotalCountByTable()).containsEntry(TEST_ENTITY_TABLE, 42); + + List highlights = output.getResults().get(0).getHighlights(); + assertThat(highlights).hasSize(1); + assertThat(highlights.get(0).getFieldName()).isEqualTo("name"); + assertThat(highlights.get(0).getText()).isEqualTo("big Widget for blues"); + assertThat(highlights.get(0).getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(4, 10), tuple(15, 20)); + assertThat(output.getResults().get(1).getHighlights()).isNull(); + + verify(mockClient).searchRecordsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(10), eq(true), anyList(), eq(1234)); + } + + + + /******************************************************************************* + ** Fragment conversion: markers removed, ranges kept; no match, blank or + ** unbalanced markers give no highlight; non-fieldValues fields are skipped. + *******************************************************************************/ + @Test + void testHighlightConversion() + { + RecordSearchHighlight highlight = QuickSearchRecordSearchProvider.highlight("name", marked("ab") + "c" + marked("d")); + assertThat(highlight.getText()).isEqualTo("abcd"); + assertThat(highlight.getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(0, 2), tuple(3, 4)); + + assertThat(QuickSearchRecordSearchProvider.highlight("name", "plain")).isNull(); + assertThat(QuickSearchRecordSearchProvider.highlight("name", null)).isNull(); + assertThat(QuickSearchRecordSearchProvider.highlight("name", marked(" "))).isNull(); + assertThat(QuickSearchRecordSearchProvider.highlight("name", QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG + "x" + QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG)).isNull(); + + assertThat(QuickSearchRecordSearchProvider.highlights(Map.of("searchableText", List.of(marked("x")), "fieldValues.", List.of(marked("x")), "fieldValues.other", List.of()))) + .isEmpty(); + assertThat(QuickSearchRecordSearchProvider.highlights(null)).isEmpty(); + } + + /******************************************************************************* ** A server that accepts connections and never answers (a hung OpenSearch): @@ -323,13 +407,16 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest Map> ids = new LinkedHashMap<>(); ids.put(TEST_ENTITY_TABLE, List.of("5", "999", "2", "4")); ids.put(OTHER_TABLE, List.of()); - when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hits(ids)); RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withLimitPerTable(2)); assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(5, 2); assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(UNINDEXED_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(1); assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(OTHER_TABLE)).isEmpty(); + assertThat(output.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.OK); + assertThat(output.getTables()).filteredOn(t -> t.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchTableSummary::getSource, RecordSearchTableSummary::getHasMore) + .containsExactly(tuple(RecordSearchSource.PROVIDER, true)); } @@ -341,11 +428,14 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest void testCoreRecordSearchAction_providerFailure_fallsBackToCoreSearch() throws QException { insertEntities("Widget", 3); - when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down")); + when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down")); RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))); assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId).containsExactly(1, 2, 3); + assertThat(output.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.FAILED); + assertThat(output.getProviderStatus().getFailureClass()).isEqualTo("QException"); + assertThat(output.getTables()).extracting(RecordSearchTableSummary::getSource).containsExactly(RecordSearchSource.FALLBACK); assertThat(runtime.isRecordSearchBackingOff()).isTrue(); ////////////////////////////////////////////////////////////////////// @@ -354,7 +444,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest ////////////////////////////////////////////////////////////////////// RecordSearchOutput again = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))); assertThat(again.getResults()).hasSize(3); - verify(mockClient, times(1)).searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt()); + verify(mockClient, times(1)).searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt()); } } -- 2.54.0