CHANGELOG.md

@@ -33,13 +33,14 @@ First release candidate of 1.1, published to Maven Central from `release/1.1`. I
 
 ## [1.1.0] - Unreleased
 
-Requires QQQ 4.1.0-RC.2 or later, the first release that carries the record-search provider SPI; built against 4.1.0-RC.2. See `docs/MIGRATION-1.1.md`.
+Requires QQQ 4.1.0-RC.4 or later (the record-search provider SPI arrived in 4.1.0-RC.2; its scores, highlights, totals and paging in 4.1.0-RC.4); built against 4.1.0-RC.4. See `docs/MIGRATION-1.1.md`.
 
 ### Added
 - `QuickSearchRecordSearchProvider`: serves QQQ core record search (`RecordSearchAction`, `POST /qqq/v1/search`, the Next UI global search) from OpenSearch for the tables Quick Search indexes. One `_msearch` request with a per-table top-N search (same query, AND semantics and field weights as `QuickSearchAction`); returns record ids only, which core re-reads through `QueryAction` (permissions, record locks, labels). Audit finding F-11, decision D1.
 - `serveCoreRecordSearch` (default `true`): registers the provider and gives indexed tables core `searchFields` (visible string and integer searchable fields) when they have none. Host `searchFields` and a host-registered provider are never replaced.
 - `recordSearchTimeoutMillis` (default 2500): wall-clock limit on the OpenSearch request; on timeout or any failure core falls back to its own search, and the provider claims no tables for 30 seconds.
 - `QuickSearchOpenSearchClient.searchRecordIdsPerTable`.
+- Record search details for QQQ 4.1.0-RC.4's record-search contract: the provider passes each hit's relevance `score`, a plain-text highlight (match ranges, from OpenSearch highlighting of `fieldValues.*` with private-use markers) and each table's total hits (`totalCountByTable`, counted to 10,000), and honors the request `offset` (`from`). `sort: RECENT` keeps relevance order (the index has no record modify dates). New `QuickSearchOpenSearchClient.searchRecordsPerTable`; `searchRecordIdsPerTable` now delegates to it. This requires QQQ 4.1.0-RC.4 or later.
 - Diagnostic tools (#35), on by default (`enableDiagnosticTools`): Search Playground, Validate Search Index (schedulable with `validationCronExpression`; reports kept in the new `quickSearchValidationRun` table), Test Search Analyzer, and on every indexed table and on `quickSearchFailedEvent` rows the record actions Inspect Search Document (database vs index, and why a record does or does not match a term) and Reindex in Search. All are table or record actions, gated by the new `debugPermissionRules` (`quickSearchDebug.hasAccess`; Reindex in Search uses `adminPermissionRules`), rate-limited per user (`diagnosticRunsPerMinute`), and recorded in the new `quickSearchDiagnosticEvent` table. Settings: `enableRecordDiagnosticActions`, `validationCronExpression`, `validationCronTimeZoneId`, `validationSampleSize`.
 - Search failure diagnostics: a failed or timed-out OpenSearch request behind core record search or `QuickSearchAction` is recorded as a `SEARCH_FAILED` diagnostic event (tables, duration, error, server; never the term); the runtime counts failures and backoff fallbacks. The Search Diagnostics widget (`quickSearchDiagnostics`, on the admin app home) shows the provider's state, recent failures and the latest validation.
 - `QuickSearchDiagnosticsOperations`: read-only probe search (named clauses, field highlights, request and response JSON), explain, document reads, `_analyze`, document sampling, and raw cluster, mapping and settings reads.

README.md

@@ -130,11 +130,11 @@ Search runs as the current `QContext` session: tables the session cannot read ar
 QQQ core has its own record search (`RecordSearchAction`, served at `POST /qqq/v1/search` and used by the Next UI's global search box): a case-insensitive contains-query over each table's `searchFields`. With `serveCoreRecordSearch` (default `true`) the producer registers `QuickSearchRecordSearchProvider` on the instance, and core asks it to search every table Quick Search indexes. The API and the UI do not change.
 
 - **What is served from OpenSearch.** Tables this QBit indexes and that are enabled in `quick_search_index`. Core offers a table only when it has `searchFields`, the session may read it, and it supports query, so the producer gives each indexed table that has no `searchFields` its visible string and integer searchable fields (never hidden or password fields). `searchFields` you set yourself are kept. A table must be in the `QInstance` before the QBit is produced to get them.
-- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights); each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order. Core re-reads them by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, and records deleted since they were indexed drop out.
+- **How.** One `_msearch` request with one search per table (Quick Search's query, AND semantics and field weights), skipping the request's `offset`; each table returns up to 4 x `limitPerTable` record ids (at most 100) in relevance order, each with its score and a highlight of the matching `fieldValues.*` fields (plain text plus match ranges), and the table's total hits (counted to 10,000). Core re-reads the ids by primary key through `QueryAction`, so table permissions, record security locks and record labels are core's, records deleted since they were indexed drop out, and only highlights of fields the user can see (where core's own highlighting found no match) reach the response, with `tables[].totalCount` and `hasMore`. The index holds no record modify dates, so `sort: RECENT` gets relevance order for indexed tables.
 - **Failure.** The OpenSearch request is cancelled after `recordSearchTimeoutMillis` (default 2500). Any failure (timeout, cluster down, index missing) makes core search those tables with its own contains-query instead, so search degrades but never fails. After a failure, Quick Search claims no tables for 30 seconds, so a down cluster does not add a timeout to every search.
 - **Opting out.** `withServeCoreRecordSearch(false)` leaves core search and `searchFields` untouched. If the host already registered another record-search provider, the QBit keeps it and logs a warning (an instance has one provider).
 
-`QuickSearchAction` remains the richer API (scores, highlights, pagination, totals).
+`QuickSearchAction` remains the direct API (all hits in one ranked list, `searchableText` highlights, pagination across tables).
 
 ### Processes
 

src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java

@@ -18,6 +18,7 @@ package com.kingsrook.qbits.quicksearch.actions;
 
 
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
@@ -25,6 +26,8 @@ import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
 import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchProviderInterface;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
@@ -32,6 +35,9 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits;
 import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
@@ -51,10 +57,14 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
  **
  ** A table is claimed when it is indexed by this QBit and enabled, and core
  ** record search is not backing off after a recent failure. search() sends
- ** one _msearch request with a top-N search per table and returns record ids
- ** only, best first; core re-reads them by primary key through QueryAction,
- ** so table permissions, record security locks and record labels are core's
- ** job, and ids of deleted records drop out. Each table gets
+ ** one _msearch request with a top-N search per table (after skipping the
+ ** input's offset) and returns record ids, best first, each with its score
+ ** and highlights (plain text plus match ranges, from the indexed field
+ ** values), and each table's total hits; core re-reads the ids by primary key
+ ** through QueryAction, so table permissions, record security locks and
+ ** record labels are core's job, and ids of deleted records drop out (core
+ ** also keeps only highlights of fields the session can see). The index has
+ ** no record modify dates, so a RECENT sort gets relevance order. Each table gets
  ** IDS_PER_LIMIT_MULTIPLIER x limitPerTable ids (at most
  ** RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE) so dropouts do not leave
  ** the table short.
@@ -70,6 +80,8 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte
 
    public static final int IDS_PER_LIMIT_MULTIPLIER = 4;
 
+   private static final String FIELD_VALUES_PREFIX = "fieldValues.";
+
 
 
    /*******************************************************************************
@@ -136,11 +148,13 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte
       int idsPerTable   = Math.min(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE, limitPerTable * IDS_PER_LIMIT_MULTIPLIER);
       int timeoutMillis = config.getRecordSearchTimeoutMillis() == null ? QuickSearchQBitConfig.DEFAULT_RECORD_SEARCH_TIMEOUT_MILLIS : config.getRecordSearchTimeoutMillis();
 
-      Map<String, List<String>> idsByTable;
-      long                      start = System.currentTimeMillis();
+      int offset = input.getOffset() == null ? 0 : Math.max(0, input.getOffset());
+
+      Map<String, RecordSearchTableHits> hitsByTable;
+      long                               start = System.currentTimeMillis();
       try
       {
-         idsByTable = runtime.getClient().searchRecordIdsPerTable(term, tableNames, idsPerTable, runtime.getDiscoveredTables(), timeoutMillis);
+         hitsByTable = runtime.getClient().searchRecordsPerTable(term, tableNames, idsPerTable, offset, true, runtime.getDiscoveredTables(), timeoutMillis);
       }
       catch(QException | RuntimeException e)
       {
@@ -157,15 +171,115 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte
          throw (e instanceof QException qe ? qe : new QException("Quick Search record search failed: " + e.getMessage(), e));
       }
 
+      Map<String, Integer> totalCountByTable = new HashMap<>();
       for(String tableName : tableNames)
       {
-         for(String recordId : CollectionUtils.nonNullList(idsByTable.get(tableName)))
+         RecordSearchTableHits tableHits = hitsByTable.get(tableName);
+         if(tableHits == null)
          {
-            results.add(new RecordSearchResult().withTableName(tableName).withRecordId(recordId));
+            continue;
+         }
+
+         if(tableHits.totalHits() != null)
+         {
+            totalCountByTable.put(tableName, (int) Math.min(Integer.MAX_VALUE, tableHits.totalHits()));
+         }
+
+         for(RecordSearchHit hit : CollectionUtils.nonNullList(tableHits.hits()))
+         {
+            List<RecordSearchHighlight> highlights = highlights(hit.highlights());
+            results.add(new RecordSearchResult()
+               .withTableName(tableName)
+               .withRecordId(hit.recordId())
+               .withScore(hit.score())
+               .withHighlights(highlights.isEmpty() ? null : highlights));
          }
       }
 
-      return (new RecordSearchOutput().withResults(results));
+      return (new RecordSearchOutput().withResults(results).withTotalCountByTable(totalCountByTable));
+   }
+
+
+
+   /*******************************************************************************
+    ** A hit's highlight fragments (by index field) as core record search
+    ** highlights: one per fieldValues.{fieldName} field, as plain text with
+    ** match ranges.  Other index fields, and fragments without a match, are
+    ** skipped.
+    *******************************************************************************/
+   public static List<RecordSearchHighlight> highlights(Map<String, List<String>> fragmentsByIndexField)
+   {
+      List<RecordSearchHighlight> highlights = new ArrayList<>();
+      for(Map.Entry<String, List<String>> entry : CollectionUtils.nonNullMap(fragmentsByIndexField).entrySet())
+      {
+         String indexField = entry.getKey();
+         if(indexField == null || !indexField.startsWith(FIELD_VALUES_PREFIX) || indexField.length() == FIELD_VALUES_PREFIX.length() || CollectionUtils.nullSafeIsEmpty(entry.getValue()))
+         {
+            continue;
+         }
+
+         RecordSearchHighlight highlight = highlight(indexField.substring(FIELD_VALUES_PREFIX.length()), entry.getValue().get(0));
+         if(highlight != null)
+         {
+            highlights.add(highlight);
+         }
+      }
+      return (highlights);
+   }
+
+
+
+   /*******************************************************************************
+    ** A highlight from a fragment whose matches are wrapped in the client's
+    ** pre/post tags: the fragment's text without the tags, and the ranges they
+    ** wrapped (adjacent ranges merged).  Null if the fragment is blank or has
+    ** no match.
+    *******************************************************************************/
+   public static RecordSearchHighlight highlight(String fieldName, String fragment)
+   {
+      if(fragment == null)
+      {
+         return (null);
+      }
+
+      StringBuilder                    text    = new StringBuilder();
+      List<RecordSearchHighlightMatch> matches = new ArrayList<>();
+      int                              open    = -1;
+      for(int i = 0; i < fragment.length(); i++)
+      {
+         char c = fragment.charAt(i);
+         if(c == QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG.charAt(0))
+         {
+            open = text.length();
+         }
+         else if(c == QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG.charAt(0))
+         {
+            if(open >= 0 && text.length() > open)
+            {
+               RecordSearchHighlightMatch last = matches.isEmpty() ? null : matches.get(matches.size() - 1);
+               if(last != null && last.getEnd() == open)
+               {
+                  last.setEnd(text.length());
+               }
+               else
+               {
+                  matches.add(new RecordSearchHighlightMatch().withStart(open).withEnd(text.length()));
+               }
+            }
+            open = -1;
+         }
+         else
+         {
+            text.append(c);
+         }
+      }
+
+      if(matches.isEmpty() || text.toString().isBlank())
+      {
+         return (null);
+      }
+
+      return (new RecordSearchHighlight().withFieldName(fieldName).withText(text.toString()).withMatches(matches));
    }
 
 }

src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java

@@ -113,6 +113,14 @@ public class QuickSearchOpenSearchClient implements Closeable
    public static final String PATH_ENGLISH         = "fieldValuesEnglish";
    public static final String PATH_EXACT           = "fieldValuesExact";
 
+   ////////////////////////////////////////////////////////////////////////////
+   // record search highlight markers: private-use characters, which do not //
+   // occur in indexed text, so fragments convert to plain text and ranges  //
+   ////////////////////////////////////////////////////////////////////////////
+   public static final String HIGHLIGHT_PRE_TAG       = "\uE000"; // private use area
+   public static final String HIGHLIGHT_POST_TAG      = "\uE001"; // private use area
+   public static final int    HIGHLIGHT_FRAGMENT_SIZE = 160;
+
    private final OpenSearchClient      client;
    private final OpenSearchAsyncClient asyncClient;
    private final OpenSearchTransport   transport;
@@ -821,36 +829,94 @@ public class QuickSearchOpenSearchClient implements Closeable
 
 
    /*******************************************************************************
-    ** Record ids of the best matches in each table, best first, for core record
-    ** search. One _msearch request with one search per table (the same query and
-    ** ranking as {@link #search}, restricted to that table), so every table gets
-    ** its own top idsPerTable hits. Only sourceTable and recordId are fetched.
+    ** One hit of a record search: the record id, its relevance score, and its
+    ** highlight fragments by index field (e.g., "fieldValues.firstName"), with
+    ** matches between {@link #HIGHLIGHT_PRE_TAG} and {@link #HIGHLIGHT_POST_TAG}.
+    *******************************************************************************/
+   public record RecordSearchHit(String recordId, Double score, Map<String, List<String>> highlights)
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** The hits of a record search in one table, best first, and the table's
+    ** total number of matches (counted up to {@link #MAX_RESULT_WINDOW}; null
+    ** if unknown).
+    *******************************************************************************/
+   public record RecordSearchTableHits(List<RecordSearchHit> hits, Long totalHits)
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Record ids of the best matches in each table, best first - see
+    ** {@link #searchRecordsPerTable}, which also returns scores, highlights and
+    ** totals.
+    *******************************************************************************/
+   public Map<String, List<String>> searchRecordIdsPerTable(String searchTerm, List<String> tableNames, int idsPerTable, List<QuickSearchableTableConfig> tableConfigs, int timeoutMillis) throws QException
+   {
+      Map<String, List<String>> idsByTable = new LinkedHashMap<>();
+      for(Map.Entry<String, RecordSearchTableHits> entry : searchRecordsPerTable(searchTerm, tableNames, idsPerTable, 0, false, tableConfigs, timeoutMillis).entrySet())
+      {
+         idsByTable.put(entry.getKey(), entry.getValue().hits().stream().map(RecordSearchHit::recordId).toList());
+      }
+      return (idsByTable);
+   }
+
+
+
+   /*******************************************************************************
+    ** The best matches in each table, best first, for core record search. One
+    ** _msearch request with one search per table (the same query and ranking as
+    ** {@link #search}, restricted to that table), so every table gets its own
+    ** top hitsPerTable hits, after skipping offset. Only sourceTable and
+    ** recordId are fetched, plus each hit's score, the table's total hits
+    ** (counted up to MAX_RESULT_WINDOW) and - when highlight is set - one
+    ** highlight fragment per matching fieldValues.* field.
     **
     ** The whole request is bounded by timeoutMillis of wall-clock time (and the
     ** same server-side search timeout): when it runs out, the request is
     ** cancelled and a QException is thrown. A failed search for any table fails
     ** the call. Returns every requested table as a key, in the given order.
     *******************************************************************************/
-   public Map<String, List<String>> searchRecordIdsPerTable(String searchTerm, List<String> tableNames, int idsPerTable, List<QuickSearchableTableConfig> tableConfigs, int timeoutMillis) throws QException
+   public Map<String, RecordSearchTableHits> searchRecordsPerTable(String searchTerm, List<String> tableNames, int hitsPerTable, int offset, boolean highlight, List<QuickSearchableTableConfig> tableConfigs, int timeoutMillis) throws QException
    {
-      Map<String, List<String>> idsByTable = new LinkedHashMap<>();
+      Map<String, RecordSearchTableHits> hitsByTable = new LinkedHashMap<>();
       if(tableNames == null || tableNames.isEmpty())
       {
-         return (idsByTable);
+         return (hitsByTable);
       }
 
+      int from = Math.max(0, Math.min(offset, MAX_RESULT_WINDOW - hitsPerTable));
+
       List<RequestItem> searches = new ArrayList<>();
       for(String tableName : tableNames)
       {
          Query query = buildSearchQuery(searchTerm, List.of(tableName), tableConfigs);
          searches.add(RequestItem.of(ri -> ri
             .header(h -> h.index(indexName))
-            .body(b -> b
-               .query(query)
-               .size(idsPerTable)
-               .trackTotalHits(t -> t.enabled(false))
-               .source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId")))
-               .timeout(timeoutMillis + "ms"))));
+            .body(b ->
+            {
+               b.query(query)
+                  .from(from)
+                  .size(hitsPerTable)
+                  .trackTotalHits(t -> t.count(MAX_RESULT_WINDOW))
+                  .source(sc -> sc.filter(f -> f.includes("sourceTable", "recordId")))
+                  .timeout(timeoutMillis + "ms");
+               if(highlight)
+               {
+                  b.highlight(h -> h
+                     .preTags(HIGHLIGHT_PRE_TAG)
+                     .postTags(HIGHLIGHT_POST_TAG)
+                     .requireFieldMatch(false)
+                     .numberOfFragments(1)
+                     .fragmentSize(HIGHLIGHT_FRAGMENT_SIZE)
+                     .fields("fieldValues.*", HighlightField.of(hf -> hf)));
+               }
+               return (b);
+            })));
       }
 
       CompletableFuture<MsearchResponse<OpenSearchDocument>> future = null;
@@ -874,17 +940,19 @@ public class QuickSearchOpenSearchClient implements Closeable
                throw (new QException("Record search failed for table [" + tableNames.get(i) + "]: " + reason));
             }
 
-            List<String> ids = new ArrayList<>();
+            List<RecordSearchHit> hits = new ArrayList<>();
             for(Hit<OpenSearchDocument> hit : item.result().hits().hits())
             {
                if(hit.source() != null && hit.source().getRecordId() != null)
                {
-                  ids.add(hit.source().getRecordId());
+                  hits.add(new RecordSearchHit(hit.source().getRecordId(), hit.score(), hit.highlight()));
                }
             }
-            idsByTable.put(tableNames.get(i), ids);
+
+            Long totalHits = item.result().hits().total() == null ? null : item.result().hits().total().value();
+            hitsByTable.put(tableNames.get(i), new RecordSearchTableHits(hits, totalHits));
          }
-         return (idsByTable);
+         return (hitsByTable);
       }
       catch(TimeoutException e)
       {

src/test/java/com/kingsrook/qbits/quicksearch/RecordSearchProviderIntegrationTest.java

@@ -20,6 +20,7 @@ package com.kingsrook.qbits.quicksearch;
 import java.io.Serializable;
 import java.time.Instant;
 import java.util.List;
+import java.util.Map;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
@@ -28,9 +29,14 @@ import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInpu
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchProviderState;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchSource;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchTableSummary;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
@@ -46,6 +52,8 @@ import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction;
 import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput;
 import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
 import com.kingsrook.qbits.quicksearch.actions.QuickSearchResult;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits;
 import com.kingsrook.qbits.quicksearch.processes.FullReindexStep;
 import org.junit.jupiter.api.AfterAll;
 import org.junit.jupiter.api.BeforeAll;
@@ -59,6 +67,7 @@ import org.testcontainers.containers.wait.strategy.Wait;
 import org.testcontainers.junit.jupiter.Container;
 import org.testcontainers.junit.jupiter.Testcontainers;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.tuple;
 
 
 /*******************************************************************************
@@ -230,6 +239,43 @@ class RecordSearchProviderIntegrationTest
 
 
 
+   /*******************************************************************************
+    ** Scores, totals and paging come from OpenSearch; the client's highlight
+    ** fragments are marked on the fieldValues.* fields and convert to plain
+    ** text and ranges; core highlights the matching non-label field.
+    *******************************************************************************/
+   @Test
+   @Order(2)
+   void testScoresTotalsPagingAndHighlights() throws Exception
+   {
+      RecordSearchInput  input  = new RecordSearchInput().withSearchTerm("blue widget").withTableNames(List.of(PRODUCT_TABLE)).withLimitPerTable(1);
+      RecordSearchOutput page1  = new RecordSearchAction().execute(input);
+      assertThat(page1.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.OK);
+      RecordSearchTableSummary products = page1.getTables().get(0);
+      assertThat(products.getSource()).isEqualTo(RecordSearchSource.PROVIDER);
+      assertThat(products.getTotalCount()).isEqualTo(2);
+      assertThat(products.getHasMore()).isTrue();
+      assertThat(page1.getResults()).hasSize(1);
+      assertThat(page1.getResults().get(0).getScore()).isPositive();
+
+      RecordSearchOutput page2 = new RecordSearchAction().execute(input.withOffset(1));
+      assertThat(page2.getTables().get(0).getHasMore()).isFalse();
+      assertThat(List.of(page1.getResults().get(0).getRecordId(), page2.getResults().get(0).getRecordId())).containsExactlyInAnyOrder(1, 2);
+
+      RecordSearchResult widget = coreSearch("blue", 5).getResults().stream().filter(r -> Integer.valueOf(2).equals(r.getRecordId())).findFirst().orElseThrow();
+      assertThat(widget.getHighlights()).extracting(RecordSearchHighlight::getFieldName).containsExactly("description");
+      assertThat(widget.getHighlights().get(0).getText()).isEqualTo("comes in blue and green");
+
+      Map<String, RecordSearchTableHits> hits = config.getRuntime().getClient().searchRecordsPerTable("blue", List.of(PRODUCT_TABLE), 5, 0, true, config.getRuntime().getDiscoveredTables(), 10_000);
+      RecordSearchHit hit = hits.get(PRODUCT_TABLE).hits().stream().filter(h -> "2".equals(h.recordId())).findFirst().orElseThrow();
+      List<RecordSearchHighlight> highlights = QuickSearchRecordSearchProvider.highlights(hit.highlights());
+      assertThat(highlights).extracting(RecordSearchHighlight::getFieldName).containsExactly("description");
+      assertThat(highlights.get(0).getText()).isEqualTo("comes in blue and green");
+      assertThat(highlights.get(0).getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(9, 13));
+   }
+
+
+
    @Test
    @Order(3)
    void testLimitPerTable() throws Exception

src/test/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProviderTest.java

@@ -28,9 +28,14 @@ import com.kingsrook.qqq.backend.core.actions.tables.RecordSearchAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlight;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchHighlightMatch;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchProviderState;
 import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchResult;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchSource;
+import com.kingsrook.qqq.backend.core.model.actions.tables.search.RecordSearchTableSummary;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
@@ -43,12 +48,15 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
 import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchHit;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient.RecordSearchTableHits;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.assertj.core.api.Assertions.assertThatThrownBy;
 import static org.assertj.core.api.Assertions.tuple;
 import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
 import static org.mockito.ArgumentMatchers.anyInt;
 import static org.mockito.ArgumentMatchers.anyList;
 import static org.mockito.ArgumentMatchers.anyString;
@@ -218,7 +226,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
       Map<String, List<String>> ids = new LinkedHashMap<>();
       ids.put(OTHER_TABLE, List.of("7"));
       ids.put(TEST_ENTITY_TABLE, List.of("3", "1", "2"));
-      when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hits(ids));
 
       RecordSearchOutput output = provider.search(new RecordSearchInput()
          .withSearchTerm("  widget   blue ")
@@ -231,13 +239,13 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
          tuple(TEST_ENTITY_TABLE, "2"),
          tuple(OTHER_TABLE, "7"));
 
-      verify(mockClient).searchRecordIdsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
+      verify(mockClient).searchRecordsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE, OTHER_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(0), eq(true), anyList(), eq(1234));
 
       provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(25));
-      verify(mockClient).searchRecordIdsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), anyList(), eq(1234));
+      verify(mockClient).searchRecordsPerTable(eq("widget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE), eq(0), eq(true), anyList(), eq(1234));
 
       provider.search(new RecordSearchInput().withSearchTerm("gadget").withTableNames(List.of(TEST_ENTITY_TABLE)));
-      verify(mockClient).searchRecordIdsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), anyList(), eq(1234));
+      verify(mockClient).searchRecordsPerTable(eq("gadget"), eq(List.of(TEST_ENTITY_TABLE)), eq(RecordSearchAction.DEFAULT_LIMIT_PER_TABLE * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(0), eq(true), anyList(), eq(1234));
    }
 
 
@@ -248,7 +256,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
       assertThat(provider.search(new RecordSearchInput().withSearchTerm("x").withTableNames(List.of(TEST_ENTITY_TABLE))).getResults()).isEmpty();
       assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(UNINDEXED_TABLE))).getResults()).isEmpty();
       assertThat(provider.search(new RecordSearchInput().withSearchTerm("widget")).getResults()).isEmpty();
-      verify(mockClient, never()).searchRecordIdsPerTable(any(), any(), anyInt(), any(), anyInt());
+      verify(mockClient, never()).searchRecordsPerTable(any(), any(), anyInt(), anyInt(), anyBoolean(), any(), anyInt());
    }
 
 
@@ -256,7 +264,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
    @Test
    void testSearch_clientFailure_throwsAndBacksOff() throws QException
    {
-      when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("boom"));
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new QException("boom"));
 
       assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
          .isInstanceOf(QException.class).hasMessageContaining("boom");
@@ -268,7 +276,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
    @Test
    void testSearch_uncheckedClientFailure_wrappedAsQException() throws QException
    {
-      when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed"));
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new IllegalStateException("closed"));
 
       assertThatThrownBy(() -> provider.search(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE))))
          .isInstanceOf(QException.class).hasMessageContaining("closed");
@@ -276,6 +284,82 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
    }
 
 
+   /*******************************************************************************
+    ** Client hits for these ids per table (no scores, highlights or totals).
+    *******************************************************************************/
+   private static Map<String, RecordSearchTableHits> hits(Map<String, List<String>> idsByTable)
+   {
+      Map<String, RecordSearchTableHits> hitsByTable = new LinkedHashMap<>();
+      idsByTable.forEach((tableName, ids) -> hitsByTable.put(tableName, new RecordSearchTableHits(ids.stream().map(id -> new RecordSearchHit(id, null, null)).toList(), null)));
+      return (hitsByTable);
+   }
+
+
+
+   /*******************************************************************************
+    ** A fragment with these words marked as matches, as the client returns it.
+    *******************************************************************************/
+   private static String marked(String word)
+   {
+      return (QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG + word + QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG);
+   }
+
+
+
+   /*******************************************************************************
+    ** Scores, highlights (fieldValues.* only, as plain text and ranges) and
+    ** totals reach core; the offset reaches the client.
+    *******************************************************************************/
+   @Test
+   void testSearch_returnsScoresHighlightsAndTotals_andPassesOffset() throws QException
+   {
+      Map<String, RecordSearchTableHits> hitsByTable = new LinkedHashMap<>();
+      hitsByTable.put(TEST_ENTITY_TABLE, new RecordSearchTableHits(List.of(
+         new RecordSearchHit("3", 7.5, Map.of(
+            "fieldValues.name", List.of("big " + marked("Widget") + " for " + marked("blue") + marked("s")),
+            "searchableText", List.of(marked("Widget")))),
+         new RecordSearchHit("1", 2.25, Map.of("fieldValues.name", List.of("no match here")))), 42L));
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hitsByTable);
+
+      RecordSearchOutput output = provider.search(new RecordSearchInput().withSearchTerm("widget blue").withTableNames(List.of(TEST_ENTITY_TABLE)).withLimitPerTable(5).withOffset(10));
+
+      assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId, RecordSearchResult::getScore).containsExactly(tuple("3", 7.5), tuple("1", 2.25));
+      assertThat(output.getTotalCountByTable()).containsEntry(TEST_ENTITY_TABLE, 42);
+
+      List<RecordSearchHighlight> highlights = output.getResults().get(0).getHighlights();
+      assertThat(highlights).hasSize(1);
+      assertThat(highlights.get(0).getFieldName()).isEqualTo("name");
+      assertThat(highlights.get(0).getText()).isEqualTo("big Widget for blues");
+      assertThat(highlights.get(0).getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(4, 10), tuple(15, 20));
+      assertThat(output.getResults().get(1).getHighlights()).isNull();
+
+      verify(mockClient).searchRecordsPerTable(eq("widget blue"), eq(List.of(TEST_ENTITY_TABLE)), eq(5 * QuickSearchRecordSearchProvider.IDS_PER_LIMIT_MULTIPLIER), eq(10), eq(true), anyList(), eq(1234));
+   }
+
+
+
+   /*******************************************************************************
+    ** Fragment conversion: markers removed, ranges kept; no match, blank or
+    ** unbalanced markers give no highlight; non-fieldValues fields are skipped.
+    *******************************************************************************/
+   @Test
+   void testHighlightConversion()
+   {
+      RecordSearchHighlight highlight = QuickSearchRecordSearchProvider.highlight("name", marked("ab") + "c" + marked("d"));
+      assertThat(highlight.getText()).isEqualTo("abcd");
+      assertThat(highlight.getMatches()).extracting(RecordSearchHighlightMatch::getStart, RecordSearchHighlightMatch::getEnd).containsExactly(tuple(0, 2), tuple(3, 4));
+
+      assertThat(QuickSearchRecordSearchProvider.highlight("name", "plain")).isNull();
+      assertThat(QuickSearchRecordSearchProvider.highlight("name", null)).isNull();
+      assertThat(QuickSearchRecordSearchProvider.highlight("name", marked(" "))).isNull();
+      assertThat(QuickSearchRecordSearchProvider.highlight("name", QuickSearchOpenSearchClient.HIGHLIGHT_POST_TAG + "x" + QuickSearchOpenSearchClient.HIGHLIGHT_PRE_TAG)).isNull();
+
+      assertThat(QuickSearchRecordSearchProvider.highlights(Map.of("searchableText", List.of(marked("x")), "fieldValues.", List.of(marked("x")), "fieldValues.other", List.of())))
+         .isEmpty();
+      assertThat(QuickSearchRecordSearchProvider.highlights(null)).isEmpty();
+   }
+
+
 
    /*******************************************************************************
     ** A server that accepts connections and never answers (a hung OpenSearch):
@@ -323,13 +407,16 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
       Map<String, List<String>> ids = new LinkedHashMap<>();
       ids.put(TEST_ENTITY_TABLE, List.of("5", "999", "2", "4"));
       ids.put(OTHER_TABLE, List.of());
-      when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenReturn(ids);
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenReturn(hits(ids));
 
       RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withLimitPerTable(2));
 
       assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(5, 2);
       assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(UNINDEXED_TABLE)).extracting(RecordSearchResult::getRecordId).containsExactly(1);
       assertThat(output.getResults()).filteredOn(r -> r.getTableName().equals(OTHER_TABLE)).isEmpty();
+      assertThat(output.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.OK);
+      assertThat(output.getTables()).filteredOn(t -> t.getTableName().equals(TEST_ENTITY_TABLE)).extracting(RecordSearchTableSummary::getSource, RecordSearchTableSummary::getHasMore)
+         .containsExactly(tuple(RecordSearchSource.PROVIDER, true));
    }
 
 
@@ -341,11 +428,14 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
    void testCoreRecordSearchAction_providerFailure_fallsBackToCoreSearch() throws QException
    {
       insertEntities("Widget", 3);
-      when(mockClient.searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down"));
+      when(mockClient.searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt())).thenThrow(new QException("OpenSearch is down"));
 
       RecordSearchOutput output = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
 
       assertThat(output.getResults()).extracting(RecordSearchResult::getRecordId).containsExactly(1, 2, 3);
+      assertThat(output.getProviderStatus().getStatus()).isEqualTo(RecordSearchProviderState.FAILED);
+      assertThat(output.getProviderStatus().getFailureClass()).isEqualTo("QException");
+      assertThat(output.getTables()).extracting(RecordSearchTableSummary::getSource).containsExactly(RecordSearchSource.FALLBACK);
       assertThat(runtime.isRecordSearchBackingOff()).isTrue();
 
       //////////////////////////////////////////////////////////////////////
@@ -354,7 +444,7 @@ class QuickSearchRecordSearchProviderTest extends BaseQuickSearchTest
       //////////////////////////////////////////////////////////////////////
       RecordSearchOutput again = new RecordSearchAction().execute(new RecordSearchInput().withSearchTerm("widget").withTableNames(List.of(TEST_ENTITY_TABLE)));
       assertThat(again.getResults()).hasSize(3);
-      verify(mockClient, times(1)).searchRecordIdsPerTable(anyString(), anyList(), anyInt(), anyList(), anyInt());
+      verify(mockClient, times(1)).searchRecordsPerTable(anyString(), anyList(), anyInt(), anyInt(), anyBoolean(), anyList(), anyInt());
    }
 
 }