diff --git a/CHANGELOG.md b/CHANGELOG.md index e139d4a..4be21e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Requires QQQ 4.1.0-RC.6 or later (the spec-managed UI API extension contract). - Real-time indexing failures now increase the table's error count by the number of failed events. Successful basepulls and full reindexes reset the count. Counter update failures are logged without failing the source write. ### Added +- Session-only typeahead: `GET /qqq/v1/qbit/quick-search/suggest` (`getQuickSearchSuggestions`) and `QuickSearchSuggestService.suggest(...)`. Reuses the permission-aware search action with one bounded candidate batch per table; groups expose only record IDs, filtered labels/subtitles, `hasMore` and `truncated`. Defaults to 5 items across 8 tables (maximum 10/20); short terms and unavailable search return empty groups. The existing UI extension and generated contract now carry both status and suggestions. - Supported host status API (`api.QuickSearchStatusService.get()`), immutable status values, and `QuickSearchApi` home navigation constants (issue #50/F10). Permission-checked, bounded diagnostics refreshes share a short per-runtime cache and reuse saved validation counts. New spec-managed `GET /qqq/v1/qbit/quick-search/status` (`getQuickSearchStatus`) with committed OpenAPI fragment and strict HTTP/contract tests. RC.6 requires the default `/qbit/quick-search` prefix; `/quick-search` is not permitted by core. Existing routes and wire shapes are unchanged. - Record developer panel: `QuickSearchDeveloperPanelProvider` (id `quickSearch`) adds a "Quick Search" panel to the developer view of every record of an indexed table (`/app/{table}/{id}/dev` in the Next UI), for users who may read Quick Search Index. It shows the table's index row (enabled, status, record count, last basepull and full reindex, searchable fields), the latest index run, the record's indexed document compared field by field with the database (Inspect Search Document's comparison; needs `debugPermissionRules`), the record's failed index events, the Inspect Search Document and Reindex in Search actions the user may run, and links to the admin app and the index row. An unreachable OpenSearch shows a warning in the panel. Registered by `QuickSearchDeveloperPanelsMetaDataProducer`. diff --git a/README.md b/README.md index 6c8afb0..16c6198 100644 --- a/README.md +++ b/README.md @@ -291,7 +291,55 @@ See also the [QQQ adoption checklist](https://github.com/QRun-IO/qqq/blob/develo Apache License, Version 2.0. See [LICENSE](LICENSE). -## Status API for hosts +## UI API for hosts + +The `quick-search` extension registers both status and suggestions in the committed +[OpenAPI fragment](src/main/resources/openapi/qqq-v1/quick-search.yaml). + +### Typeahead suggestions + +`GET /qqq/v1/qbit/quick-search/suggest` (`getQuickSearchSuggestions`) requires a +signed-in session. Quick Search admin permission is not required. Java hosts use +`QuickSearchSuggestService.suggest(q, limitPerTable, maxTables, tables)` in the caller's +`QContext`. + +| Query parameter | Meaning | +| --- | --- | +| `q` | Required text, trimmed with consecutive whitespace collapsed. Fewer than 2 characters returns empty groups; more than 100 returns HTTP 400. | +| `limitPerTable` | Integer 1..10; default 5. | +| `maxTables` | Integer 1..20; default 8. | +| `tables` | Optional comma-separated table metadata names. Unknown, disabled and unreadable tables are omitted; an empty filter returns no groups. | + +```json +{ + "query": "blue wid", + "groups": [{ + "tableName": "product", + "tableLabel": "Products", + "hasMore": true, + "items": [{"recordId": "42", "label": "Blue Widget", "subtitle": "Small"}] + }], + "truncated": false +} +``` + +The existing edge-ngram query supports the last word's prefix. Groups are ordered +by their best accessible score, with per-table relevance order preserved. Labels +fall back to `tableLabel #recordId`; subtitles are nullable and come only from the +already-filtered result. Scores, highlights and display values are never returned. +The existing action enforces table READ permissions, enabled-table selection, +configured record security locks (on by default), PHI/PII exclusions and protected +primary keys. + +Suggestions make one bounded OpenSearch request per eligible table, fetching at +most `2 * (limitPerTable + 1)` candidates under record locks (also subject to the +host's `maxSearchLimit`). Locked rows can leave a group partially filled. +`hasMore` confirms an extra accessible hit in this batch; false does not prove +there are no further matches. `truncated` means more tables had accessible hits +than `maxTables`. Missing runtime/client or unavailable search returns empty groups. +Missing `q`, invalid integers and out-of-range limits return HTTP 400. + +### Administrative status Requires QQQ **4.1.0-RC.6**. Hosts should use the supported API rather than reach into `QuickSearchRuntime`, `QuickSearchOpenSearchClient`, diagnostic renderers, or qbit tables: diff --git a/docs/STATUS-API-HOST-MIGRATION.md b/docs/STATUS-API-HOST-MIGRATION.md index 6aa5347..25d8a86 100644 --- a/docs/STATUS-API-HOST-MIGRATION.md +++ b/docs/STATUS-API-HOST-MIGRATION.md @@ -23,26 +23,49 @@ session to bypass `adminPermissionRules`. Cache hits still check permission. | Method | Path | operationId | | --- | --- | --- | | GET | `/qqq/v1/qbit/quick-search/status` | `getQuickSearchStatus` | +| GET | `/qqq/v1/qbit/quick-search/suggest` | `getQuickSearchSuggestions` | + +Suggestions require only a signed-in session, with table READ permissions and +configured record locks applied to results. Do not gate the typeahead UI on the +status endpoint's admin permission. Send `q`, optionally `limitPerTable` (default 5, +maximum 10), `maxTables` (default 8, maximum 20), and comma-separated `tables`. +The response contains `query`, relevance-ordered `groups` and `truncated`; each +group has `tableName`, `tableLabel`, `hasMore` and `items` of `recordId`, `label`, +and nullable `subtitle`. See the [API details](../README.md#typeahead-suggestions) +for normalization, bounded-batch semantics and empty responses during outages. Extension: `quick-search`, version `1.0.0`; core dependency: `4.1.0-RC.6`. The requested `/quick-search` prefix cannot validate on the supplied RC.6 baseline: `MiddlewareSpecExtensionsMetaData.ESTABLISHED_PREFIXES` only admits esb, alerts, assistant, and sql-workbench. We use the default namespace without modifying core. -Wire changes against develop: one additive endpoint and its schemas; no existing -method, path, request, or response changed. Next UI `src/` contains no calls to a +The suggestions change adds a second endpoint and its schemas; the status +method, path, request and response are unchanged. Next UI `src/` contains no calls to a Quick Search-specific endpoint today; its core `/qqq/v1/search` integration is unchanged. Vendor `src/main/resources/openapi/qqq-v1/quick-search.yaml` for clients. Regenerate the fragment with: ```powershell -./mvnw.cmd -B test -Dtest=QuickSearchApiContractTest -Dquicksearch.openapi.update=true -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT +./mvnw.cmd -B -Dtest=QuickSearchApiContractTest -Dquicksearch.openapi.update=true test +./mvnw.cmd -B -Dtest=QuickSearchApiContractTest test ``` -The local verification override is never the committed dependency version. +The fragment covers both registered specs using the committed dependency version. -## Verification and limits +## Suggestions verification (2026-10-14) + +The two-spec contract regenerates and passes without the update flag. Final +`CI=true ./mvnw.cmd -B verify` passes 495 unit/guardrail tests, including strict +HTTP checks for reader access and invalid suggestion parameters. All 9 integration +classes fail their Docker prerequisite because this sandbox cannot access Docker's +named pipe; the 3 new suggest integration tests compile but remain unexecuted. +The configured JaCoCo gates pass when invoked separately with +`./mvnw.cmd -B jacoco:report jacoco:check@check` (74.38% instruction, 91.88% class). +New API files and changed tests pass repository Checkstyle; the full source check +still reports 446 existing violations. + +## Original status verification and limits `./mvnw.cmd -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` passes 468 unit tests and JaCoCo, including strict HTTP server startup, permission checks, diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java index 1797289..d598901 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java @@ -166,7 +166,7 @@ public class QuickSearchAction if(input.getLimitPerTable() != null && input.getLimitPerTable() > 0) { - return (searchPerTable(client, normalizedTerm, allowedTables, offset, Math.min(input.getLimitPerTable(), maxLimit), tableConfigs, applyLocks)); + return (searchPerTable(client, normalizedTerm, allowedTables, offset, Math.min(input.getLimitPerTable(), maxLimit), tableConfigs, applyLocks, Boolean.TRUE.equals(input.getSinglePass()))); } Page page = searchPage(client, normalizedTerm, allowedTables, offset, limit, tableConfigs, applyLocks, new ScanBudget(QuickSearchOpenSearchClient.MAX_RESULT_WINDOW, MAX_LOCKED_ROUND_TRIPS)); @@ -178,8 +178,10 @@ public class QuickSearchAction /******************************************************************************* ** Search each allowed table separately, skipping offset results and ** returning up to limitPerTable hits in each table. + ** singlePass bounds locked searches to one batch of twice the requested + ** page size; it never scans again to fill gaps left by inaccessible rows. *******************************************************************************/ - private QuickSearchOutput searchPerTable(QuickSearchOpenSearchClient client, String term, List allowedTables, int offset, int limitPerTable, List tableConfigs, boolean applyLocks) throws QException + private QuickSearchOutput searchPerTable(QuickSearchOpenSearchClient client, String term, List allowedTables, int offset, int limitPerTable, List tableConfigs, boolean applyLocks, boolean singlePass) throws QException { List results = new ArrayList<>(); long totalHits = 0; @@ -189,6 +191,11 @@ public class QuickSearchAction int window = QuickSearchOpenSearchClient.MAX_RESULT_WINDOW; int tableHits = Math.max(window / allowedTables.size(), Math.min(window, 2 * limitPerTable)); int tableTrips = Math.max(2, MAX_LOCKED_ROUND_TRIPS / allowedTables.size()); + if(singlePass) + { + tableHits = Math.min(window, 2 * limitPerTable); + tableTrips = 1; + } for(String tableName : allowedTables) { diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchInput.java b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchInput.java index 33428d8..95bf943 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchInput.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchInput.java @@ -35,6 +35,28 @@ public class QuickSearchInput extends AbstractActionInput private Integer offset; private List tableNames; private Integer limitPerTable; + private Boolean singlePass = false; + + + + /******************************************************************************* + ** Whether per-table searches stop after one bounded candidate batch. + *******************************************************************************/ + public Boolean getSinglePass() + { + return (singlePass); + } + + + + /******************************************************************************* + ** Opt in to bounded typeahead latency, accepting partially filled pages. + *******************************************************************************/ + public QuickSearchInput withSinglePass(Boolean singlePass) + { + this.singlePass = singlePass; + return (this); + } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchApi.java b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchApi.java index 882de0b..98c0909 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchApi.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchApi.java @@ -49,6 +49,6 @@ public final class QuickSearchApi public static MiddlewareSpecExtension extension() { return (new MiddlewareSpecExtension().withName("quick-search").withSemanticVersion("1.0.0") - .withDescription("Quick Search administrative status").withSpec(QuickSearchStatusSpec.class)); + .withDescription("Quick Search administration and suggestions").withSpec(QuickSearchStatusSpec.class).withSpec(QuickSearchSuggestSpec.class)); } } diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggest.java b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggest.java new file mode 100644 index 0000000..1fd81ae --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggest.java @@ -0,0 +1,64 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.quicksearch.api; + + +import java.util.List; + + +/******************************************************************************* + ** Minimal immutable suggestion response. Search internals never cross this API. + *******************************************************************************/ +public record QuickSearchSuggest(String query, List groups, Boolean truncated) +{ + /******************************************************************************* + ** Freeze the group order for callers. + *******************************************************************************/ + public QuickSearchSuggest + { + groups = List.copyOf(groups); + } + + + + /******************************************************************************* + ** One readable table, ordered by its highest scoring accessible hit. + *******************************************************************************/ + public record Group(String tableName, String tableLabel, Boolean hasMore, List items) + { + /******************************************************************************* + ** Freeze the per-table relevance order. + *******************************************************************************/ + public Group + { + items = List.copyOf(items); + } + } + + + + /******************************************************************************* + ** Only already-filtered labels and subtitles are exposed. + *******************************************************************************/ + public record Item(String recordId, String label, String subtitle) + { + } +} diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestService.java b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestService.java new file mode 100644 index 0000000..230639b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestService.java @@ -0,0 +1,161 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.quicksearch.api; + + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchOutput; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchResult; +import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.logging.QLogger; + + +/******************************************************************************* + ** Session-only typeahead over the existing permission-aware search action. + *******************************************************************************/ +public final class QuickSearchSuggestService +{ + private static final QLogger LOG = QLogger.getLogger(QuickSearchSuggestService.class); + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private QuickSearchSuggestService() + { + } + + + + /******************************************************************************* + ** One candidate batch per table, with one extra visible hit for hasMore. + ** A short term or unavailable search returns an empty response. hasMore only + ** confirms an extra accessible hit in this bounded batch, never hidden rows. + *******************************************************************************/ + public static QuickSearchSuggest suggest(String query, Integer limitPerTable, Integer maxTables, String tables) throws QException + { + if(QContext.getQSession() == null) + { + throw (new QPermissionDeniedException("A session is required for Quick Search suggestions.")); + } + if(query == null) + { + throw (new QBadRequestException("q is required.")); + } + String normalized = IndexingUtils.normalizeSearchText(query); + if(normalized.length() > QuickSearchAction.MAX_TERM_LENGTH) + { + throw (new QBadRequestException("Search term is too long (maximum 100 characters).")); + } + Integer itemLimit = bounded(limitPerTable, 5, 10, "limitPerTable"); + Integer groupLimit = bounded(maxTables, 8, 20, "maxTables"); + QuickSearchSuggest empty = new QuickSearchSuggest(normalized, List.of(), false); + QuickSearchRuntime runtime = QuickSearchRuntime.get(); + if(normalized.length() < QuickSearchAction.MIN_TERM_LENGTH || runtime == null) + { + return (empty); + } + + List tableNames = tables == null ? null : Arrays.stream(tables.split(",")).map(String::trim).filter(name -> !name.isEmpty()).distinct().toList(); + if(tableNames != null && tableNames.isEmpty()) + { + return (empty); + } + + QuickSearchOutput output; + try + { + if(runtime.getClient() == null) + { + return (empty); + } + output = new QuickSearchAction().execute(new QuickSearchInput().withSearchTerm(normalized) + .withTableNames(tableNames).withLimitPerTable(itemLimit + 1).withSinglePass(true)); + } + catch(QPermissionDeniedException e) + { + throw (e); + } + catch(QException | RuntimeException e) + { + LOG.debug("Quick Search suggestions unavailable", e); + return (empty); + } + + Map> byTable = new LinkedHashMap<>(); + for(QuickSearchResult result : output.getResults()) + { + byTable.computeIfAbsent(result.getTableName(), key -> new ArrayList<>()).add(result); + } + List> ranked = new ArrayList<>(byTable.values()); + ranked.sort(Comparator., Float>comparing( + hits -> hits.stream().map(QuickSearchResult::getScore).filter(score -> score != null).max(Float::compare).orElse(Float.NEGATIVE_INFINITY)) + .reversed().thenComparing(hits -> hits.get(0).getTableName())); + + List groups = new ArrayList<>(); + for(List hits : ranked.subList(0, Math.min(groupLimit, ranked.size()))) + { + QuickSearchResult first = hits.get(0); + List items = hits.stream().limit(itemLimit).map(hit -> + { + String label = hit.getRecordLabel(); + if(label == null || label.isBlank()) + { + label = hit.getTableLabel() + " #" + hit.getRecordId(); + } + return (new QuickSearchSuggest.Item(hit.getRecordId(), label, hit.getSubtitle())); + }).toList(); + groups.add(new QuickSearchSuggest.Group(first.getTableName(), first.getTableLabel(), hits.size() > itemLimit, items)); + } + return (new QuickSearchSuggest(normalized, groups, ranked.size() > groupLimit)); + } + + + + /******************************************************************************* + ** Reject invalid ranges instead of silently changing a caller's intent. + *******************************************************************************/ + private static Integer bounded(Integer value, Integer defaultValue, Integer maximum, String name) throws QBadRequestException + { + if(value == null) + { + return (defaultValue); + } + if(value < 1 || value > maximum) + { + throw (new QBadRequestException(name + " must be between 1 and " + maximum + ".")); + } + return (value); + } +} diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestSpec.java b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestSpec.java new file mode 100644 index 0000000..0c52f4a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchSuggestSpec.java @@ -0,0 +1,176 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.quicksearch.api; + + +import java.util.List; +import java.util.Map; +import java.util.TreeMap; +import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation; +import com.kingsrook.qqq.middleware.javalin.specs.BasicResponse; +import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import com.kingsrook.qqq.openapi.model.In; +import com.kingsrook.qqq.openapi.model.Method; +import com.kingsrook.qqq.openapi.model.Parameter; +import com.kingsrook.qqq.openapi.model.Schema; +import com.kingsrook.qqq.openapi.model.Type; +import io.javalin.http.HttpStatus; + + +/******************************************************************************* + ** Permission-filtered suggestions through the spec-managed UI API. + *******************************************************************************/ +public class QuickSearchSuggestSpec extends AbstractJsonEndpointSpec +{ + /******************************************************************************* + ** Describe the session-only operation and its bounded search semantics. + *******************************************************************************/ + @Override + public BasicOperation defineBasicOperation() + { + return (new BasicOperation().withHttpMethod(HttpMethod.GET).withPath("/qbit/quick-search/suggest") + .withShortSummary("Get Quick Search suggestions") + .withLongDescription("Requires a signed-in session, not Quick Search admin permission. Results inherit table READ permissions, configured record security locks and protected-field exclusions. One bounded candidate batch per table; no extra scans. Unavailable search and normalized terms shorter than two characters return empty groups.")); + } + + + + /******************************************************************************* + ** Stable generated-client operation name. + *******************************************************************************/ + @Override + protected void customizeMethod(Method method) + { + method.setOperationId("getQuickSearchSuggestions"); + } + + + + /******************************************************************************* + ** Bounds apply after whitespace normalization; short terms are forgiving. + *******************************************************************************/ + @Override + public List defineRequestParameters() + { + return (List.of( + new Parameter().withName("q").withIn(In.QUERY).withRequired(true) + .withDescription("Search text: trim and collapse whitespace before applying the 2..100 character range. Shorter terms return empty groups; longer terms return 400.") + .withSchema(field(Type.STRING, "Normalized search term.", false)), + new Parameter().withName("limitPerTable").withIn(In.QUERY).withRequired(false) + .withDescription("Items per table: integer 1..10, default 5.").withSchema(field(Type.INTEGER, "Maximum items per table; default 5, maximum 10.", false)), + new Parameter().withName("maxTables").withIn(In.QUERY).withRequired(false) + .withDescription("Groups to return: integer 1..20, default 8.").withSchema(field(Type.INTEGER, "Maximum groups; default 8, maximum 20.", false)), + new Parameter().withName("tables").withIn(In.QUERY).withRequired(false) + .withDescription("Optional comma-separated table metadata names. Unknown, disabled and unreadable tables are omitted.") + .withSchema(field(Type.STRING, "Table-name filter; omitted means all eligible tables.", false)))); + } + + + + /******************************************************************************* + ** Versioned response component. + *******************************************************************************/ + @Override + protected String getResponseSchemaName() + { + return ("QuickSearchSuggestV1"); + } + + + + /******************************************************************************* + ** The HTTP and in-process entry points share validation and authorization. + *******************************************************************************/ + @Override + protected Object handle(JsonInput input) throws Exception + { + return (QuickSearchSuggestService.suggest(input.getQueryParam("q"), input.getQueryParamInteger("limitPerTable"), + input.getQueryParamInteger("maxTables"), input.getQueryParam("tables"))); + } + + + + /******************************************************************************* + ** Reuse the extension's standard framework error shape. + *******************************************************************************/ + @Override + public List defineAdditionalBasicResponses() + { + return (List.of(new BasicResponse(HttpStatus.BAD_REQUEST, "Missing q, normalized term longer than 100 characters, or invalid limits", "QuickSearchErrorV1"), + new BasicResponse(HttpStatus.UNAUTHORIZED, "Sign in required", "QuickSearchErrorV1"), + new BasicResponse(HttpStatus.FORBIDDEN, "Session permission denied", "QuickSearchErrorV1"))); + } + + + + /******************************************************************************* + ** Explicit wire schemas exclude scores, highlights and display values. + *******************************************************************************/ + @Override + public Map defineComponentSchemas() + { + Schema response = object(Map.of( + "query", field(Type.STRING, "Search term after trimming and collapsing whitespace.", false), + "groups", array("QuickSearchSuggestGroupV1", "Readable tables ordered by best accessible score; at most maxTables."), + "truncated", field(Type.BOOLEAN, "More tables had accessible hits than maxTables.", false))); + Schema group = object(Map.of( + "tableName", field(Type.STRING, "Source table metadata name.", false), + "tableLabel", field(Type.STRING, "Source table label, falling back to its metadata name.", false), + "hasMore", field(Type.BOOLEAN, "An extra accessible item was found in the bounded batch. False does not prove the index has no further matches.", false), + "items", array("QuickSearchSuggestItemV1", "At most limitPerTable accessible items in per-table relevance order."))); + Schema item = object(Map.of( + "recordId", field(Type.STRING, "Source record primary key.", false), + "label", field(Type.STRING, "Filtered record label, falling back to tableLabel #recordId when blank.", false), + "subtitle", field(Type.STRING, "Already-filtered result subtitle; null when absent.", true))); + return (Map.of("QuickSearchSuggestV1", response, "QuickSearchSuggestGroupV1", group, "QuickSearchSuggestItemV1", item)); + } + + + + /******************************************************************************* + ** All response objects have explicit properties. + *******************************************************************************/ + private static Schema object(Map properties) + { + return (new Schema().withType(Type.OBJECT).withNullable(false).withProperties(new TreeMap<>(properties))); + } + + + + /******************************************************************************* + ** Scalar fields declare nullability and meaning. + *******************************************************************************/ + private static Schema field(Type type, String description, Boolean nullable) + { + return (new Schema().withType(type).withDescription(description).withNullable(nullable)); + } + + + + /******************************************************************************* + ** Arrays always exist, including when empty. + *******************************************************************************/ + private static Schema array(String itemSchema, String description) + { + return (field(Type.ARRAY, description, false).withItems(new Schema().withRef("#/components/schemas/" + itemSchema))); + } +} diff --git a/src/main/resources/openapi/qqq-v1/quick-search.yaml b/src/main/resources/openapi/qqq-v1/quick-search.yaml index 8424f64..5a867e1 100644 --- a/src/main/resources/openapi/qqq-v1/quick-search.yaml +++ b/src/main/resources/openapi/qqq-v1/quick-search.yaml @@ -132,6 +132,66 @@ components: nullable: true type: "integer" type: "object" + QuickSearchSuggestGroupV1: + nullable: false + properties: + hasMore: + description: "An extra accessible item was found in the bounded batch. False\ + \ does not prove the index has no further matches." + nullable: false + type: "boolean" + items: + description: "At most limitPerTable accessible items in per-table relevance\ + \ order." + items: + $ref: "#/components/schemas/QuickSearchSuggestItemV1" + nullable: false + type: "array" + tableLabel: + description: "Source table label, falling back to its metadata name." + nullable: false + type: "string" + tableName: + description: "Source table metadata name." + nullable: false + type: "string" + type: "object" + QuickSearchSuggestItemV1: + nullable: false + properties: + label: + description: "Filtered record label, falling back to tableLabel #recordId\ + \ when blank." + nullable: false + type: "string" + recordId: + description: "Source record primary key." + nullable: false + type: "string" + subtitle: + description: "Already-filtered result subtitle; null when absent." + nullable: true + type: "string" + type: "object" + QuickSearchSuggestV1: + nullable: false + properties: + groups: + description: "Readable tables ordered by best accessible score; at most\ + \ maxTables." + items: + $ref: "#/components/schemas/QuickSearchSuggestGroupV1" + nullable: false + type: "array" + query: + description: "Search term after trimming and collapsing whitespace." + nullable: false + type: "string" + truncated: + description: "More tables had accessible hits than maxTables." + nullable: false + type: "boolean" + type: "object" securitySchemes: sessionUuidCookie: in: "cookie" @@ -140,7 +200,7 @@ components: info: contact: email: "contact@kingsrook.com" - description: "Quick Search administrative status" + description: "Quick Search administration and suggestions" title: "QQQ Middleware API extension: quick-search" version: "1.0.0" openapi: "3.0.3" @@ -181,3 +241,77 @@ paths: - sessionUuidCookie: - "N/A" summary: "Get Quick Search status" + /qqq/v1/qbit/quick-search/suggest: + get: + description: "Requires a signed-in session, not Quick Search admin permission.\ + \ Results inherit table READ permissions, configured record security locks\ + \ and protected-field exclusions. One bounded candidate batch per table; no\ + \ extra scans. Unavailable search and normalized terms shorter than two characters\ + \ return empty groups." + operationId: "getQuickSearchSuggestions" + parameters: + - description: "Search text: trim and collapse whitespace before applying the\ + \ 2..100 character range. Shorter terms return empty groups; longer terms\ + \ return 400." + in: "query" + name: "q" + required: true + schema: + description: "Normalized search term." + nullable: false + type: "string" + - description: "Items per table: integer 1..10, default 5." + in: "query" + name: "limitPerTable" + required: false + schema: + description: "Maximum items per table; default 5, maximum 10." + nullable: false + type: "integer" + - description: "Groups to return: integer 1..20, default 8." + in: "query" + name: "maxTables" + required: false + schema: + description: "Maximum groups; default 8, maximum 20." + nullable: false + type: "integer" + - description: "Optional comma-separated table metadata names. Unknown, disabled\ + \ and unreadable tables are omitted." + in: "query" + name: "tables" + required: false + schema: + description: "Table-name filter; omitted means all eligible tables." + nullable: false + type: "string" + responses: + 400: + content: + application/json: + schema: + $ref: "#/components/schemas/QuickSearchErrorV1" + description: "Missing q, normalized term longer than 100 characters, or\ + \ invalid limits" + 401: + content: + application/json: + schema: + $ref: "#/components/schemas/QuickSearchErrorV1" + description: "Sign in required" + 403: + content: + application/json: + schema: + $ref: "#/components/schemas/QuickSearchErrorV1" + description: "Session permission denied" + 200: + content: + application/json: + schema: + $ref: "#/components/schemas/QuickSearchSuggestV1" + description: "Successful Response" + security: + - sessionUuidCookie: + - "N/A" + summary: "Get Quick Search suggestions" diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchApiContractTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchApiContractTest.java index 4079995..b2c8908 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchApiContractTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchApiContractTest.java @@ -54,7 +54,13 @@ class QuickSearchApiContractTest "mvnw.cmd -Dtest=QuickSearchApiContractTest -Dquicksearch.openapi.update=true test")).isEmpty(); var model = version.generateExtensionOpenAPIModel("qqq", extension); assertThat(model.getComponents().getSchemas().keySet()).allMatch(name -> name.startsWith("QuickSearch")); - assertThat(model.getPaths().keySet()).containsExactly("/qqq/v1/qbit/quick-search/status"); - assertThat(extension.getSpecs()).hasSize(1); + assertThat(model.getPaths().keySet()).containsExactlyInAnyOrder("/qqq/v1/qbit/quick-search/status", "/qqq/v1/qbit/quick-search/suggest"); + assertThat(extension.getSpecs()).hasSize(2); + assertThat(generated).contains("getQuickSearchSuggestions", "QuickSearchSuggestV1", "QuickSearchSuggestGroupV1", "QuickSearchSuggestItemV1"); + var schemas = model.getComponents().getSchemas(); + assertThat(schemas.get("QuickSearchSuggestV1").getProperties()).containsOnlyKeys("query", "groups", "truncated"); + assertThat(schemas.get("QuickSearchSuggestGroupV1").getProperties()).containsOnlyKeys("tableName", "tableLabel", "hasMore", "items"); + assertThat(schemas.get("QuickSearchSuggestItemV1").getProperties()).containsOnlyKeys("recordId", "label", "subtitle"); + assertThat(schemas.get("QuickSearchSuggestItemV1").getProperties().get("subtitle").getNullable()).isTrue(); } } diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchStatusTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchStatusTest.java index e6461eb..6f71e3a 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchStatusTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchStatusTest.java @@ -362,8 +362,27 @@ class QuickSearchStatusTest assertThat(anonymous.statusCode()).as(anonymous.body()).isEqualTo(403); var denied = client.send(HttpRequest.newBuilder(URI.create(url)).header("Cookie", "sessionUUID=reader-" + suffix).GET().build(), HttpResponse.BodyHandlers.ofString()); assertThat(denied.statusCode()).as(denied.body()).isEqualTo(403); + String suggestUrl = "http://127.0.0.1:" + port + "/qqq/v1/qbit/quick-search/suggest"; + for(String query : List.of("?q=x", "?q=%20x%20", "?q=wi&tables=unknown", "?q=wi&tables=", "?q=wi")) + { + var suggestions = client.send(HttpRequest.newBuilder(URI.create(suggestUrl + query)) + .header("Cookie", "sessionUUID=reader-" + suffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(suggestions.statusCode()).as(suggestions.body()).isEqualTo(200); + var suggestionJson = new ObjectMapper().readTree(suggestions.body()); + assertThat(suggestionJson.path("groups").isArray()).isTrue(); + assertThat(suggestionJson.path("groups").size()).isZero(); + assertThat(suggestionJson.path("truncated").asBoolean()).isFalse(); + } + for(String query : List.of("", "?q=" + "x".repeat(101), "?q=wi&limitPerTable=no", "?q=wi&limitPerTable=11", "?q=wi&maxTables=21")) + { + var invalid = client.send(HttpRequest.newBuilder(URI.create(suggestUrl + query)) + .header("Cookie", "sessionUUID=reader-" + suffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(invalid.statusCode()).as(invalid.body()).isEqualTo(400); + assertThat(new ObjectMapper().readTree(invalid.body()).path("error").isTextual()).isTrue(); + } var openapi = client.send(HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/qqq/v1/openapi.yaml")).GET().build(), HttpResponse.BodyHandlers.ofString()); - assertThat(openapi.body()).contains("/qqq/v1/qbit/quick-search/status", "QuickSearchStatusV1"); + assertThat(openapi.body()).contains("/qqq/v1/qbit/quick-search/status", "QuickSearchStatusV1", + "/qqq/v1/qbit/quick-search/suggest", "getQuickSearchSuggestions", "QuickSearchSuggestV1"); } finally { diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestIntegrationTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestIntegrationTest.java new file mode 100644 index 0000000..61aebcc --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestIntegrationTest.java @@ -0,0 +1,217 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.quicksearch; + + +import java.util.List; +import java.util.Map; +import java.util.UUID; +import java.util.stream.IntStream; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.kingsrook.qbits.quicksearch.api.QuickSearchSuggest; +import com.kingsrook.qbits.quicksearch.api.QuickSearchSuggestService; +import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; +import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType; +import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; + + +/******************************************************************************* + ** Actual edge-ngram matches, grouping, permissions and classified stale documents. + *******************************************************************************/ +@ExtendWith(RequiresDockerCondition.class) +@Testcontainers +class QuickSearchSuggestIntegrationTest extends BaseQuickSearchTest +{ + @Container + static GenericContainer opensearch = new GenericContainer<>(System.getProperty("opensearch.test.image", "opensearchproject/opensearch:2.19.6")) + .withExposedPorts(9200) + .withEnv("discovery.type", "single-node") + .withEnv("DISABLE_SECURITY_PLUGIN", "true") + .withEnv("DISABLE_INSTALL_DEMO_CONFIG", "true") + .waitingFor(Wait.forHttp("/_cluster/health").forStatusCode(200)); + + private QuickSearchRuntime runtime; + private List tables; + + + + /******************************************************************************* + ** Use real indexing and search over two memory-backed source tables. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + QTableMetaData first = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + first.withLabel("Products").withRecordLabelFields("name").withRecordLabelFormat("%s"); + first.withField(new QFieldMetaData("personal", QFieldType.STRING)); + QTableMetaData second = first.clone().withName("other").withLabel("Other Products"); + QContext.getQInstance().addTable(second); + tables = List.of(table(TEST_ENTITY_TABLE), table("other")); + QuickSearchQBitConfig config = QuickSearchQBitContext.getConfig() + .withOpensearchHost(opensearch.getHost()).withOpensearchPort(opensearch.getMappedPort(9200)) + .withOpensearchIndexName("suggest-" + UUID.randomUUID()).withEnableRealTimeIndexing(false); + runtime = spy(new QuickSearchRuntime(config)); + runtime.setDiscoveredTables(tables); + doReturn(tables).when(runtime).getDiscoveredTables(); + config.setRuntime(runtime); + QContext.getQInstance().addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config)); + runtime.getClient().ensureIndexExists(); + for(QuickSearchableTableConfig table : tables) + { + List records = IntStream.rangeClosed(1, 12).mapToObj(id -> new QRecord() + .withValue("id", id).withValue("name", "Blue Widget " + id).withValue("description", "Public details") + .withValue("personal", "PII-CANARY-7f3c")).toList(); + new InsertAction().execute(new InsertInput(table.getTableName()).withRecords(records)); + List documents = records.stream().map(record -> IndexingUtils.buildDocument(record, table)).toList(); + runtime.getClient().indexDocuments(documents, 100); + } + runtime.getClient().refreshIndex(); + } + + + + /******************************************************************************* + ** Explicit title and subtitle roles exercise the indexed presentation fields. + *******************************************************************************/ + private QuickSearchableTableConfig table(String name) + { + return (new QuickSearchableTableConfig().withTableName(name).withPrimaryKeyField("id") + .withSearchableFields(List.of("name", "description", "personal")).withEnabledByDefault(true) + .withRecordLabelFields(List.of("name")).withRecordLabelFormat("%s") + .withFieldRoles(Map.of("name", "TITLE", "description", "SUBTITLE", "personal", "DISPLAY"))); + } + + + + /******************************************************************************* + ** Each test owns its alias and closes its transport. + *******************************************************************************/ + @AfterEach + void tearDown() + { + if(runtime != null) + { + runtime.close(); + } + } + + + + /******************************************************************************* + ** Prefix of the final word matches through the unchanged OpenSearch query. + *******************************************************************************/ + @Test + void prefixGroupsAndCaps() throws Exception + { + QuickSearchSuggest result = QuickSearchSuggestService.suggest(" Blue Wid ", 2, 1, null); + assertThat(result.query()).isEqualTo("Blue Wid"); + assertThat(result.groups()).hasSize(1); + assertThat(result.truncated()).isTrue(); + QuickSearchSuggest.Group group = result.groups().get(0); + assertThat(group.items()).hasSize(2); + assertThat(group.hasMore()).isTrue(); + assertThat(group.items()).allSatisfy(item -> + { + assertThat(item.label()).startsWith("Blue Widget "); + assertThat(item.subtitle()).isEqualTo("Public details"); + }); + assertThat(new ObjectMapper().writeValueAsString(result)).doesNotContain("highlight", "displayValues", "score"); + QuickSearchSuggest filtered = QuickSearchSuggestService.suggest("Blue Wid", 10, 20, TEST_ENTITY_TABLE); + assertThat(filtered.groups()).extracting(QuickSearchSuggest.Group::tableName).containsExactly(TEST_ENTITY_TABLE); + assertThat(filtered.groups().get(0).items()).hasSize(10); + assertThat(filtered.groups().get(0).hasMore()).isTrue(); + assertThat(filtered.truncated()).isFalse(); + } + + + + /******************************************************************************* + ** Search cannot expose unreadable tables or rows excluded by actual QQQ locks. + *******************************************************************************/ + @Test + void tableAndRecordSecurity() throws Exception + { + QContext.getQInstance().getTable("other").withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS)); + QContext.getQInstance().addSecurityKeyType(new QSecurityKeyType().withName("entity")); + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).withRecordSecurityLock(new RecordSecurityLock().withSecurityKeyType("entity").withFieldName("id")); + QContext.getQSession().withSecurityKeyValue("entity", 2); + QuickSearchSuggest result = QuickSearchSuggestService.suggest("Blue Wid", 10, null, null); + assertThat(result.groups()).extracting(QuickSearchSuggest.Group::tableName).containsExactly(TEST_ENTITY_TABLE); + assertThat(result.groups().get(0).items()).extracting(QuickSearchSuggest.Item::recordId).containsExactly("2"); + assertThat(result.groups().get(0).hasMore()).isFalse(); + assertThat(result.truncated()).isFalse(); + } + + + + /******************************************************************************* + ** Classification changes protect old documents before a rebuild is performed. + *******************************************************************************/ + @Test + void classifiedFieldsNeverMatchOrReturn() throws Exception + { + OpenSearchDocument stale = new OpenSearchDocument().withSourceTable(TEST_ENTITY_TABLE).withRecordId("1") + .withRecordLabel("PHI-CANARY-7f3c").withSubtitle("PHI-CANARY-7f3c").withSearchableText("PHI-CANARY-7f3c PII-CANARY-7f3c") + .withFieldValues(Map.of("name", "Public widget", "description", "PHI-CANARY-7f3c", "personal", "PII-CANARY-7f3c")) + .withDisplayValues(Map.of("Personal", "PII-CANARY-7f3c")); + runtime.getClient().indexDocuments(List.of(stale), 100); + runtime.getClient().refreshIndex(); + QTableMetaData metadata = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + metadata.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + metadata.getField("personal").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)); + for(String term : List.of("PHI-CANARY", "PII-CANARY")) + { + assertThat(QuickSearchSuggestService.suggest(term, null, null, TEST_ENTITY_TABLE).groups()).isEmpty(); + } + QuickSearchSuggest result = QuickSearchSuggestService.suggest("Public wid", null, null, TEST_ENTITY_TABLE); + assertThat(result.groups()).hasSize(1); + assertThat(result.groups().get(0).items()).hasSize(1); + assertThat(result.groups().get(0).items().get(0).label()).isEqualTo("Public widget"); + assertThat(result.groups().get(0).items().get(0).subtitle()).isNull(); + assertThat(new ObjectMapper().writeValueAsString(result)).doesNotContain("PHI-CANARY", "PII-CANARY"); + metadata.getField("id").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)); + assertThat(QuickSearchSuggestService.suggest("Public wid", null, null, TEST_ENTITY_TABLE).groups()).isEmpty(); + } +} diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestTest.java new file mode 100644 index 0000000..a8601c9 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchSuggestTest.java @@ -0,0 +1,334 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.quicksearch; + + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.stream.IntStream; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.kingsrook.qbits.quicksearch.api.QuickSearchSuggest; +import com.kingsrook.qbits.quicksearch.api.QuickSearchSuggestService; +import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; +import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType; +import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.opensearch.client.opensearch.core.SearchResponse; +import org.opensearch.client.opensearch.core.search.Hit; +import org.opensearch.client.opensearch.core.search.TotalHitsRelation; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.clearInvocations; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; + + +/******************************************************************************* + ** Exercise real search filtering and bounded requests with a stubbed transport. + *******************************************************************************/ +class QuickSearchSuggestTest extends BaseQuickSearchTest +{ + private QuickSearchRuntime runtime; + private QuickSearchQBitConfig config; + private QuickSearchOpenSearchClient client; + private List tables; + + + + /******************************************************************************* + ** Register an instance runtime while keeping the real permission/search path. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + config = QuickSearchQBitContext.getConfig(); + client = QuickSearchQBitContext.getClient(); + runtime = spy(new QuickSearchRuntime(config)); + tables = new ArrayList<>(QuickSearchQBitContext.getDiscoveredTables()); + doReturn(client).when(runtime).getClient(); + doReturn(tables).when(runtime).getDiscoveredTables(); + doReturn(true).when(runtime).isTableEnabled(anyString()); + config.setRuntime(runtime); + QContext.getQInstance().addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config)); + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).withLabel("Test Entity"); + insert(TEST_ENTITY_TABLE); + stub(3); + } + + + + /******************************************************************************* + ** Populate enough source rows for actual QueryAction authorization. + *******************************************************************************/ + private void insert(String table) throws Exception + { + new InsertAction().execute(new InsertInput(table).withRecords(IntStream.rangeClosed(1, 30) + .mapToObj(id -> new QRecord().withValue("id", id).withValue("name", "Widget " + id)).toList())); + } + + + + /******************************************************************************* + ** Add another configured, readable table. + *******************************************************************************/ + private void addTable(String name) throws Exception + { + QTableMetaData metadata = QContext.getQInstance().getTable(TEST_ENTITY_TABLE).clone(); + metadata.setName(name); + metadata.setLabel(name); + QContext.getQInstance().addTable(metadata); + tables.add(new QuickSearchableTableConfig().withTableName(name).withPrimaryKeyField("id").withSearchableFields(List.of("name")).withEnabledByDefault(true)); + insert(name); + } + + + + /******************************************************************************* + ** Honor fetch bounds and give later table names higher scores for ranking. + *******************************************************************************/ + private void stub(Integer total) throws Exception + { + doAnswer(invocation -> + { + String table = ((List) invocation.getArgument(1)).get(0); + Integer size = invocation.getArgument(2); + List> hits = IntStream.rangeClosed(1, Math.min(total, size)).mapToObj(id -> + new Hit.Builder().index("test").id(table + ":" + id).score((table.equals(TEST_ENTITY_TABLE) ? 10.0 : 100.0) - id) + .source(new OpenSearchDocument().withSourceTable(table).withRecordId(String.valueOf(id)).withRecordLabel("Widget " + id) + .withSubtitle("Public subtitle").withDisplayValues(Map.of("Extra", "not exposed"))) + .highlight(Map.of("searchableText", List.of("Widget"))).build()).toList(); + return (new SearchResponse.Builder().took(1).timedOut(false).shards(sh -> sh.total(1).successful(1).failed(0)) + .hits(h -> h.hits(hits).total(t -> t.value(total).relation(TotalHitsRelation.Eq))).build()); + }).when(client).search(anyString(), anyList(), anyInt(), anyInt(), anyList(), any()); + } + + + + /******************************************************************************* + ** Normalize, group, preserve relevance order and expose only the wire fields. + *******************************************************************************/ + @Test + void groupedResultsAndWireShape() throws Exception + { + addTable("other"); + QuickSearchSuggest result = QuickSearchSuggestService.suggest(" blue wid ", 2, 1, null); + assertThat(result.query()).isEqualTo("blue wid"); + assertThat(result.truncated()).isTrue(); + assertThat(result.groups()).hasSize(1); + QuickSearchSuggest.Group group = result.groups().get(0); + assertThat(group.tableName()).isEqualTo("other"); + assertThat(group.hasMore()).isTrue(); + assertThat(group.items()).extracting(QuickSearchSuggest.Item::recordId).containsExactly("1", "2"); + assertThat(group.items().get(0).label()).isEqualTo("Widget 1"); + assertThat(group.items().get(0).subtitle()).isEqualTo("Public subtitle"); + assertThat(new ObjectMapper().writeValueAsString(result)).doesNotContain("score", "highlight", "displayValues", "", "not exposed"); + verify(client).search(eq("blue wid"), eq(List.of("other")), eq(6), eq(0), anyList(), any()); + verify(client).search(eq("blue wid"), eq(List.of(TEST_ENTITY_TABLE)), eq(6), eq(0), anyList(), any()); + assertThatThrownBy(() -> result.groups().clear()).isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> group.items().clear()).isInstanceOf(UnsupportedOperationException.class); + } + + + + /******************************************************************************* + ** Default and maximum caps include an extra visible item without rescanning. + *******************************************************************************/ + @Test + void defaultAndMaximumCaps() throws Exception + { + stub(30); + for(int i = 0; i < 21; i++) + { + addTable("table" + i); + } + QuickSearchSuggest defaults = QuickSearchSuggestService.suggest("wi", null, null, null); + assertThat(defaults.groups()).hasSize(8).allSatisfy(group -> + { + assertThat(group.items()).hasSize(5); + assertThat(group.hasMore()).isTrue(); + }); + assertThat(defaults.truncated()).isTrue(); + QuickSearchSuggest maximum = QuickSearchSuggestService.suggest("wi", 10, 20, null); + assertThat(maximum.groups()).hasSize(20).allSatisfy(group -> assertThat(group.items()).hasSize(10)); + assertThat(maximum.truncated()).isTrue(); + } + + + + /******************************************************************************* + ** Table selection, permissions, enabled flags and protected primary keys apply. + *******************************************************************************/ + @Test + void tableFilters() throws Exception + { + addTable("other"); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, " other, other,unknown ").groups()) + .extracting(QuickSearchSuggest.Group::tableName).containsExactly("other"); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, " , ").groups()).isEmpty(); + QContext.getQInstance().getTable("other").withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS)); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, "other").groups()).isEmpty(); + doReturn(false).when(runtime).isTableEnabled(TEST_ENTITY_TABLE); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, null).groups()).isEmpty(); + doReturn(true).when(runtime).isTableEnabled(TEST_ENTITY_TABLE); + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).getField("id") + .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, TEST_ENTITY_TABLE).groups()).isEmpty(); + } + + + + /******************************************************************************* + ** Real record locks remove hits; an empty candidate batch never triggers scans. + *******************************************************************************/ + @Test + void recordLocksAndSinglePass() throws Exception + { + QContext.getQInstance().addSecurityKeyType(new QSecurityKeyType().withName("entity")); + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).withRecordSecurityLock(new RecordSecurityLock().withSecurityKeyType("entity").withFieldName("id")); + QContext.getQSession().withSecurityKeyValue("entity", 2); + QuickSearchSuggest result = QuickSearchSuggestService.suggest("wi", 1, null, null); + assertThat(result.groups().get(0).items()).extracting(QuickSearchSuggest.Item::recordId).containsExactly("2"); + assertThat(result.groups().get(0).hasMore()).isFalse(); + clearInvocations(client); + QContext.getQSession().clearSecurityKeyValues(); + stub(30); + assertThat(QuickSearchSuggestService.suggest("wi", 1, null, null).groups()).isEmpty(); + verify(client).search(anyString(), anyList(), eq(4), eq(0), anyList(), any()); + verify(client, never()).search(anyString(), anyList(), anyInt(), eq(4), anyList(), any()); + } + + + + /******************************************************************************* + ** Stale PHI/PII labels and subtitles are rebuilt from allowed fields only. + *******************************************************************************/ + @Test + void classifiedStaleResultAndFallback() throws Exception + { + QTableMetaData metadata = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + metadata.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + tables.get(0).withRecordLabelFields(List.of("description")).withFieldRoles(Map.of("description", "SUBTITLE")); + OpenSearchDocument stale = new OpenSearchDocument().withSourceTable(TEST_ENTITY_TABLE).withRecordId("1") + .withRecordLabel("PHI-CANARY-7f3c").withSubtitle("PHI-CANARY-7f3c").withSearchableText("PHI-CANARY-7f3c") + .withFieldValues(Map.of("name", "Public widget", "description", "PHI-CANARY-7f3c")); + SearchResponse response = new SearchResponse.Builder().took(1).timedOut(false) + .shards(sh -> sh.total(1).successful(1).failed(0)).hits(h -> h.hits(hit -> hit.index("test").id("1").source(stale))).build(); + doReturn(response).when(client).search(anyString(), anyList(), anyInt(), anyInt(), anyList(), any()); + QuickSearchSuggest result = QuickSearchSuggestService.suggest("Public", null, null, null); + assertThat(result.groups().get(0).items().get(0).label()).isEqualTo("Test Entity #1"); + assertThat(result.groups().get(0).items().get(0).subtitle()).isNull(); + assertThat(new ObjectMapper().writeValueAsString(result)).doesNotContain("PHI-CANARY-7f3c"); + assertThat(result.truncated()).isFalse(); + } + + + + /******************************************************************************* + ** Input validation runs even when search is unavailable. + *******************************************************************************/ + @Test + void validationAndShortTerms() throws Exception + { + for(String term : List.of("", " ", " x ")) + { + assertThat(QuickSearchSuggestService.suggest(term, null, null, null).groups()).isEmpty(); + } + verifyNoInteractions(client); + assertThatThrownBy(() -> QuickSearchSuggestService.suggest(null, null, null, null)).isInstanceOf(QBadRequestException.class); + assertThatThrownBy(() -> QuickSearchSuggestService.suggest("x".repeat(101), null, null, null)).isInstanceOf(QBadRequestException.class); + for(Integer limit : List.of(0, -1, 11)) + { + assertThatThrownBy(() -> QuickSearchSuggestService.suggest("wi", limit, null, null)).isInstanceOf(QBadRequestException.class); + } + for(Integer limit : List.of(0, -1, 21)) + { + assertThatThrownBy(() -> QuickSearchSuggestService.suggest("wi", null, limit, null)).isInstanceOf(QBadRequestException.class); + } + assertThat(QuickSearchSuggestService.suggest(" " + "x".repeat(100) + " ", null, null, null).query()).hasSize(100); + } + + + + /******************************************************************************* + ** Hosts that disable record locks still use one request and visible lookahead. + *******************************************************************************/ + @Test + void unlockedSearchAndHostLimit() throws Exception + { + config.withApplyRecordSecurityLocks(false); + QuickSearchSuggest result = QuickSearchSuggestService.suggest("wi", 2, null, null); + assertThat(result.groups().get(0).items()).hasSize(2); + assertThat(result.groups().get(0).hasMore()).isTrue(); + verify(client).search(anyString(), anyList(), eq(3), eq(0), anyList(), any()); + config.withMaxSearchLimit(1); + QuickSearchSuggest bounded = QuickSearchSuggestService.suggest("wi", 2, null, null); + assertThat(bounded.groups().get(0).items()).hasSize(1); + assertThat(bounded.groups().get(0).hasMore()).isFalse(); + } + + + + /******************************************************************************* + ** Missing runtime/client and failed transports are forgiving; permissions are not. + *******************************************************************************/ + @Test + void unavailableAndSessionRequired() throws Exception + { + config.setRuntime(null); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, null).groups()).isEmpty(); + config.setRuntime(runtime); + doReturn(null).when(runtime).getClient(); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, null).groups()).isEmpty(); + doThrow(new QException("unavailable")).when(runtime).getClient(); + assertThat(QuickSearchSuggestService.suggest("wi", null, null, null).groups()).isEmpty(); + doThrow(new QPermissionDeniedException("denied")).when(runtime).getClient(); + assertThatThrownBy(() -> QuickSearchSuggestService.suggest("wi", null, null, null)).isInstanceOf(QPermissionDeniedException.class); + QContext.init(QContext.getQInstance(), (QSession) null); + assertThatThrownBy(() -> QuickSearchSuggestService.suggest("", null, null, null)).isInstanceOf(QPermissionDeniedException.class); + } +}