diff --git a/CHANGELOG.md b/CHANGELOG.md index b5df883..e139d4a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,8 @@ Requires QQQ 4.1.0-RC.6 or later (the spec-managed UI API extension contract). - Possible-value sources `quick_search_eligible_table`, `quick_search_indexable_field`, `quick_search_timestamp_field`, `quick_search_field_role`, `quick_search_field_analyzer`, `quick_search_documents_choice`, `quick_search_config_source`; the join `quick_search_index_join_quick_search_index_field` and the Searchable Fields widget on the index record. Config tables are audited at field level. ### Changed +- Distinguish protected fields from fields that simply cannot be searched. Boolean, blob and hidden fields no longer trigger result rebuilding or erase record labels. Labels are suppressed only when their source fields are protected; unprotected hidden primary keys and timestamps remain usable. +- Exclude PII/PHI-classified fields (including table defaults) and field-security-locked fields everywhere Quick Search indexes or displays values. Configuration rejects protected fields; code defaults drop them with one warning per table. Search and diagnostics omit protected values from older documents. Rebuild existing indexes after upgrading or changing classification metadata. Uses QQQ 4.1.0-RC.6; no dependency changes. - `quick_search_index`: the generic Create and Delete are off (`TABLE_INSERT`, `TABLE_DELETE`); use Add Search Index and Remove Search Index. `quick_search_searchable_table` is now a custom source over the effective tables. `quick_search_index_run` gains `cancelRequested` and `cancelRequestedBy`. - Hidden and `isSensitive` fields are never indexed (previously hidden and password only). - Record screens (record-screen audit): `quick_search_index_run` and `quick_search_failed_event` are written by the QBit only, so Create and Edit are off (Delete stays, for clearing); both get named field sections, record labels and pickers (`quick_search_run_type`, `quick_search_run_status`, `quick_search_failed_event_status`, `quick_search_failed_event_action`, and the table picker for `tableName`). The index record lists its Recent Runs (join `quick_search_index_join_quick_search_index_run`). `quick_search_validation_run.reportJson` uses a JSON code editor. The Basepull, Full Reindex and Reconcile processes run from Quick Search Index's Actions and are no longer app (nav) children; their table input is `searchTableName` (`tableName` still works for API callers, and a Quick Search table name is ignored), so running them from the table no longer limits them to `quick_search_index`. Process input steps now carry `HELP_TEXT` and `EDIT_FORM` components, so the Next UI renders their fields. diff --git a/README.md b/README.md index 63777ce..6c8afb0 100644 --- a/README.md +++ b/README.md @@ -346,3 +346,36 @@ configured metadata prefix. `QuickSearchApi.APP_NAME` (`quickSearchAdmin`) and `QuickSearchApi.HOME_HREF` (`/app/quickSearchAdmin`) are available for default-name navigation. Keep detailed runs, failed events, validation and operator tools in the qbit's own home. [kof22 migration mapping](docs/STATUS-API-HOST-MIGRATION.md). + +## Classified data + +Quick Search excludes fields classified as **PII** or **PHI**, including fields +that inherit the table's classification. It also excludes hidden, sensitive, +password/masked fields and every field with a `FieldSecurityLock`; a shared +search index cannot enforce access to individual fields for each user. PUBLIC, +CONFIDENTIAL and unclassified fields remain eligible under the usual field rules. + +The same rule applies to annotation and code configuration, admin field choices, +real-time writes, basepull, rebuilds, replay, labels, snippets, analyzer copies and +result display fields. An admin trying to add a protected field gets an error. +Protected fields in existing or explicit code configuration are dropped, with one +warning per table metadata lifetime; warnings never contain field values. A +protected primary key prevents indexing the record or sending its ID for deletion. + +Protection is separate from search eligibility: boolean, blob, hidden and other +unsupported fields alone do not trigger result filtering or remove record labels. +Labels are omitted when their source fields are protected; unrelated protected +fields do not erase safe labels. Unprotected hidden keys and timestamps can still +provide record IDs and document versions. + +After upgrading, or after changing a field's classification or security metadata, +run **Full Reindex** across all indexed tables to remove previously indexed values. +Until then, tables containing protected fields are searched through allowed field +values, and results are rebuilt without old combined labels or highlights. +Diagnostic views also omit protected values and the raw response for these tables. +This does not purge old OpenSearch backups or retained physical indexes; apply your +host's retention policy to those copies. + +This release uses the APIs in QQQ **4.1.0-RC.6**. Support for the host's +`AuditValueProtector` classification (`ClassifiedValues`, qqq#1182) and +`FieldAccessHelper.getLockTier` (qqq#1172) is deferred until the qbit moves to RC.7. diff --git a/docs/SESSION-STATE.md b/docs/SESSION-STATE.md index 1585eac..59f36ba 100644 --- a/docs/SESSION-STATE.md +++ b/docs/SESSION-STATE.md @@ -48,3 +48,75 @@ Updated: 2026-10-14 1. qqq-all and Voyage pin `1.0.0-RC.2` and report back; their feedback decides RC.3 or GA. 2. Remaining "Before 1.0.0 GA" items in `docs/TODO.md`: stuck `REBUILDING` after a killed or overlapping full reindex (from the PR #20 review), and the `qbit-build-parent` 2.1.0 re-pin. 3. GA: merge `release/1.0` to `main` and tag `v1.0.0`. + +## 2026-10-14 — secured-data section 15 + +- Implemented a shared RC.6 classification/security policy in `QuickSearchFieldRules`. +- Applied it to config discovery, config writes/choices, document construction, + safe query fields, result presentation and diagnostic reads; protected primary + keys cannot be used as index IDs. +- Added live-listener/basepull/stale-result PHI/PII canaries and config/rule tests. +- Final gate: `mvnw.cmd -o -B verify` **BUILD SUCCESS**. 479 unit tests passed + (7 new tests); 41 Docker integration tests skipped because Docker is unavailable. + Architecture guardrails passed. JaCoCo: 73.84% instruction / 91.61% class coverage, + above the 70% / 90% gates. Checkstyle, PMD and SpotBugs are not configured in this + build. No missing offline artifacts. `git diff --check` passed. +- Reviewed the diff; changes remain uncommitted. Gate log: + `target/secured-data-verify.log`. +- RC.7 host audit-protector classification and lock-tier APIs remain deferred. + +### Changed files + +- Documentation: `README.md`, `CHANGELOG.md`, `docs/SESSION-STATE.md`, `docs/TODO.md`. +- Configuration: `QuickSearchQBitProducer.java`, `config/QuickSearchFieldRules.java`, + `config/QuickSearchConfigPrimer.java`, `config/QuickSearchConfigStore.java`. +- Indexing: `listeners/QuickSearchRecordChangeListener.java`, + `processes/IndexingUtils.java`, `processes/AbstractIndexingStep.java`, + `processes/BasepullIndexStep.java`. +- Search and diagnostics: `actions/QuickSearchAction.java`, + `actions/QuickSearchRecordSearchProvider.java`, + `opensearch/QuickSearchOpenSearchClient.java`, + `opensearch/QuickSearchDiagnosticsOperations.java`. +- Tests: `config/QuickSearchConfigCustomizersTest.java`, + `listeners/QuickSearchRecordChangeListenerTest.java`, + `processes/IndexingUtilsTest.java`, `publisher/SynchronousIndexEventPublisherTest.java`, + new `opensearch/ClassifiedDataCanaryTest.java`. + +Production and test paths above are relative to +`src/{main,test}/java/com/kingsrook/qbits/quicksearch/` respectively. + +### Continuity + +Second-brain project/daily writes are unavailable in this session because the +vault lies outside the writable roots. This local checkpoint holds the durable +outcome for the reviewer. No commit, push, branch creation, dependency version +change, or production filesystem IO was introduced. + + +## 2026-10-14 - protection review follow-up + +- Added isProtected for PII/PHI (including table defaults), field locks, sensitive + values and masked types. Unsupported search types and hidden fields alone are + not protected. Searchable/snippet/display field lists still use isIndexable. +- Labels check their configured/table source fields; unrelated protected fields + preserve safe labels. A copied record prevents unsafe preformatted labels from + returning through QQQ fallback formatting without changing the input record. +- Primary-key guards and timestamp versions now use protection rather than + search eligibility. Added four regression tests and extended rule/canary tests. +- Follow-up files: config/QuickSearchFieldRules.java, processes/IndexingUtils.java, + listeners/QuickSearchRecordChangeListener.java, + opensearch/QuickSearchOpenSearchClient.java, + opensearch/QuickSearchDiagnosticsOperations.java; test files + config/QuickSearchConfigCustomizersTest.java, processes/IndexingUtilsTest.java, + opensearch/ClassifiedDataCanaryTest.java; README.md, CHANGELOG.md, + docs/TODO.md and this file. Java paths are relative to the existing main/test + quicksearch packages. All earlier uncommitted work is preserved. +- Second-brain project/daily synchronization remains unavailable because the vault + is outside writable roots; this checkpoint is the local handoff. Nothing committed + or pushed; no branches or dependency changes. +- Final gate: `mvnw.cmd -o -B verify` BUILD SUCCESS; 483 unit tests passed + (four additional tests); all three PHI/PII canaries passed. All 41 Docker + integration tests skipped because Docker is unavailable. Architecture guards + and JaCoCo passed (73.88% instruction / 91.61% class coverage). + Checkstyle, PMD and SpotBugs are not configured in this build. No offline + artifacts missing. Gate log: `target/protection-review-verify.log`. diff --git a/docs/TODO.md b/docs/TODO.md index e85e254..09d52f5 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -56,3 +56,12 @@ - [x] Status assessment, deep-dive audit (25 blockers), PRD (2026-10-04). - [x] 4.1.0-RC.1 compatibility verified (2026-10-04). - [x] All 25 audit blockers implemented and tested on the release branch (2026-10-04). + +## Secured-data section 15 (2026-10-14) + +- [x] Review fix: separate protection from indexability; preserve safe labels, ordinary results and unprotected hidden keys/timestamps, with regression tests. + +- [x] Exclude PII/PHI, hidden, sensitive, password and field-locked values from indexing and results on RC.6. +- [x] Add canary coverage for real-time indexing, basepull and stale results; refuse classified config fields. +- [ ] On the QQQ RC.7 upgrade, honour `ClassifiedValues.isClassified(table, field)` (qqq#1182) and `FieldAccessHelper.getLockTier` (qqq#1172). +- [ ] Deployment: rebuild existing indexes and apply host retention policy to old physical indexes/backups containing previously indexed values. diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java index 41cc1c6..9152f7d 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java @@ -29,6 +29,7 @@ import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider; import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField; import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable; import com.kingsrook.qbits.quicksearch.api.QuickSearchApi; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener; import com.kingsrook.qbits.quicksearch.metadata.QuickSearchAdminAppMetaDataProducer; import com.kingsrook.qbits.quicksearch.metadata.QuickSearchDiagnosticsProcessMetaDataHelper; @@ -380,8 +381,8 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer searchableFields, Map fieldWeights, Map fieldIncludeLabels) + private static void removeExcludedFields(QTableMetaData table, List searchableFields, Map fieldWeights, Map fieldIncludeLabels) { List hidden = new ArrayList<>(); for(String fieldName : searchableFields) { - QFieldMetaData field = table.getFields().get(fieldName); - if(field != null && (Boolean.TRUE.equals(field.getIsHidden()) || field.getIsSensitive() || (field.getType() != null && field.getType().needsMasked()))) + if(!QuickSearchFieldRules.isIndexable(table, fieldName)) { hidden.add(fieldName); } } if(!hidden.isEmpty()) { - LOG.warn("Hidden, sensitive and password fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden)); + QuickSearchFieldRules.warnExcludedFields(table); searchableFields.removeAll(hidden); hidden.forEach(fieldWeights::remove); hidden.forEach(fieldIncludeLabels::remove); diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java index 7bdb696..1797289 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java @@ -46,6 +46,7 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchQBitContext; import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport; import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent; import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; @@ -221,7 +222,7 @@ public class QuickSearchAction if(!applyLocks) { SearchResponse response = runSearch(client, term, tables, pageSize, offset, tableConfigs, null); - List results = toResults(response); + List results = toResults(response, tableConfigs); long totalHits = response.hits().total() != null ? response.hits().total().value() : results.size(); boolean lowerBound = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte; @@ -255,7 +256,7 @@ public class QuickSearchAction rawFloor = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte; List candidates = new ArrayList<>(); - for(QuickSearchResult result : toResults(response)) + for(QuickSearchResult result : toResults(response, tableConfigs)) { if(seen.add(result.getTableName() + ":" + result.getRecordId())) { @@ -315,7 +316,7 @@ public class QuickSearchAction List allowed = new ArrayList<>(); for(String tableName : new LinkedHashSet<>(requested)) { - if(!configured.contains(tableName)) + if(!configured.contains(tableName) || QuickSearchFieldRules.hasProtectedPrimaryKey(tableName)) { continue; } @@ -407,7 +408,7 @@ public class QuickSearchAction /******************************************************************************* ** *******************************************************************************/ - private static List toResults(SearchResponse response) + private static List toResults(SearchResponse response, List tableConfigs) { QInstance qInstance = QContext.getQInstance(); List results = new ArrayList<>(); @@ -420,9 +421,16 @@ public class QuickSearchAction continue; } - String highlightSnippet = null; + String sourceTable = source.getSourceTable(); + boolean filterFields = QuickSearchFieldRules.requiresFieldFiltering(sourceTable); + source = IndexingUtils.filterResult(source, tableConfigs.stream().filter(c -> sourceTable.equals(c.getTableName())).findFirst().orElse(null)); + if(source == null) + { + continue; + } + String highlightSnippet = filterFields ? source.getSearchableText() : null; Map> highlight = hit.highlight(); - if(highlight != null) + if(highlight != null && !filterFields) { List fragments = highlight.get("searchableText"); if(fragments != null && !fragments.isEmpty()) diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java index e6363c8..9e9dc16 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java @@ -32,6 +32,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.utils.CollectionUtils; import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport; import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent; import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; @@ -85,7 +86,7 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte return (false); } - if(runtime.getTableConfig(table.getName()) == null) + if(runtime.getTableConfig(table.getName()) == null || QuickSearchFieldRules.hasProtectedPrimaryKey(table.getName())) { return (false); } @@ -120,7 +121,8 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte List tableNames = new ArrayList<>(); for(String tableName : new LinkedHashSet<>(CollectionUtils.nonNullList(input.getTableNames()))) { - if(runtime.getTableConfig(tableName) != null) + if(runtime.getTableConfig(tableName) != null + && !QuickSearchFieldRules.hasProtectedPrimaryKey(tableName)) { tableNames.add(tableName); } diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigPrimer.java b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigPrimer.java index 2988ce4..bd19b74 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigPrimer.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigPrimer.java @@ -238,8 +238,7 @@ public class QuickSearchConfigPrimer } if(!QuickSearchFieldRules.isIndexable(table, fieldName)) { - LOG.warn("A field declared searchable in code cannot be indexed; it is not primed: " + QuickSearchFieldRules.whyNotIndexable(table, fieldName), - logPair("tableName", tableName), logPair("fieldName", fieldName)); + QuickSearchFieldRules.warnExcludedFields(table); continue; } codeFields.add(codeFieldRow(code, fieldName, i)); diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigStore.java b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigStore.java index 8884a51..843dcef 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigStore.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigStore.java @@ -377,7 +377,7 @@ public class QuickSearchConfigStore String problem = QuickSearchFieldRules.whyNotIndexable(table, fieldName); if(problem != null) { - LOG.warn("Quick Search field row skipped: " + problem, logPair("tableName", tableName), logPair("fieldName", fieldName)); + QuickSearchFieldRules.warnExcludedFields(table); continue; } if(fields.contains(fieldName)) diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchFieldRules.java b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchFieldRules.java index fd22416..a043e25 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchFieldRules.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchFieldRules.java @@ -23,9 +23,14 @@ import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Set; +import java.util.WeakHashMap; import java.util.regex.Pattern; +import com.kingsrook.qqq.backend.core.actions.ai.AiDataClassifications; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper; import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.logging.QLogger; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; @@ -48,11 +53,14 @@ import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole; ** on backend variants, not produced by a QBit (qbit-internal), not a QQQ ** system table (audits, saved views and reports, scheduled jobs, scripts), ** not named like a secret store, and has at least one indexable field. A - ** field is indexable when it is not hidden, not isSensitive, not a password - ** and not a blob or boolean. + ** field is indexable when it is not PII/PHI-classified, locked, hidden, + ** sensitive, masked, a blob or a boolean. *******************************************************************************/ public final class QuickSearchFieldRules { + private static final QLogger LOG = QLogger.getLogger(QuickSearchFieldRules.class); + private static final WeakHashMap WARNED_TABLES = new WeakHashMap<>(); + public static final int MAX_DEFAULT_FIELDS = 20; public static final int TITLE_WEIGHT = 3; @@ -85,6 +93,9 @@ public final class QuickSearchFieldRules /******************************************************************************* ** Why a field cannot be indexed, or null when it can. + ** When the qbit moves to QQQ 4.1.0-RC.7, also honour core + ** ClassifiedValues.isClassified(table, field) (the host's AuditValueProtector + ** classification, qqq#1182) and FieldAccessHelper.getLockTier (qqq#1172). *******************************************************************************/ public static String whyNotIndexable(QTableMetaData table, String fieldName) { @@ -101,6 +112,15 @@ public final class QuickSearchFieldRules { return ("field [" + fieldName + "] is hidden"); } + DataClassification classification = AiDataClassifications.ofField(table, fieldName); + if(classification == DataClassification.PII || classification == DataClassification.PHI) + { + return ("field [" + fieldName + "] is classified " + classification + " and cannot be indexed"); + } + if(field.getFieldSecurityLock() != null) + { + return ("field [" + fieldName + "] has a field security lock and cannot be indexed"); + } if(field.getIsSensitive() || (field.getType() != null && field.getType().needsMasked())) { return ("field [" + fieldName + "] is sensitive (its values must not reach the search index)"); @@ -124,6 +144,96 @@ public final class QuickSearchFieldRules + /******************************************************************************* + ** Whether a field contains protected values, independently of searchability. + ** Classification falls back to the table. When the qbit moves to QQQ + ** 4.1.0-RC.7, also honour ClassifiedValues.isClassified(table, field) + ** (qqq#1182) and FieldAccessHelper.getLockTier (qqq#1172). + *******************************************************************************/ + public static boolean isProtected(QTableMetaData table, String fieldName) + { + QFieldMetaData field = table == null || fieldName == null ? null : table.getFields().get(fieldName); + if(field == null) + { + return (false); + } + DataClassification classification = AiDataClassifications.ofField(table, fieldName); + return (classification == DataClassification.PII || classification == DataClassification.PHI + || field.getFieldSecurityLock() != null || field.getIsSensitive() + || (field.getType() != null && field.getType().needsMasked())); + } + + + + /******************************************************************************* + ** Warn once per table metadata lifetime; never include record values. + *******************************************************************************/ + public static synchronized void warnExcludedFields(QTableMetaData table) + { + if(table != null && WARNED_TABLES.put(table, Boolean.TRUE) == null) + { + LOG.warn("Quick Search dropped fields that cannot be indexed (classified PII/PHI, secured, hidden, sensitive or unsupported fields) from table [" + table.getName() + "]"); + } + } + + + + /******************************************************************************* + ** Filter explicit field lists as well as cached configurations at use time. + *******************************************************************************/ + public static List indexableFields(QTableMetaData table, List fields) + { + List result = new ArrayList<>(); + for(String field : CollectionUtils.nonNullList(fields)) + { + if(isIndexable(table, field)) + { + result.add(field); + } + else + { + warnExcludedFields(table); + } + } + return (result); + } + + + + /******************************************************************************* + ** Resolve a table's metadata for backend query and result filtering. + *******************************************************************************/ + public static QTableMetaData lookupTable(String tableName) + { + QInstance instance = QContext.getQInstance(); + return (instance == null || tableName == null ? null : instance.getTable(tableName)); + } + + + + /******************************************************************************* + ** Combined text and labels from an older index cannot be attributed to + ** individual fields. Use only allowed field values for these tables. + *******************************************************************************/ + public static boolean requiresFieldFiltering(String tableName) + { + QTableMetaData table = lookupTable(tableName); + return (table != null && table.getFields().keySet().stream().anyMatch(f -> isProtected(table, f))); + } + + + + /******************************************************************************* + ** A protected primary key cannot be sent as an index id, even for deletes. + *******************************************************************************/ + public static boolean hasProtectedPrimaryKey(String tableName) + { + QTableMetaData table = lookupTable(tableName); + return (table != null && isProtected(table, table.getPrimaryKeyField())); + } + + + /******************************************************************************* ** The indexable fields of a table, in the table's order. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java b/src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java index dbf76fd..c2c34ab 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java @@ -120,6 +120,11 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte } String primaryKeyField = tableConfig.getPrimaryKeyField() != null ? tableConfig.getPrimaryKeyField() : "id"; + if(QuickSearchFieldRules.lookupTable(tableName) == null || QuickSearchFieldRules.isProtected(QuickSearchFieldRules.lookupTable(tableName), primaryKeyField)) + { + QuickSearchFieldRules.warnExcludedFields(QuickSearchFieldRules.lookupTable(tableName)); + return; + } /////////////////////////////////////////////////////////////////////////// // paused: index nothing, but keep the written keys (DEFERRED), which // @@ -514,7 +519,17 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte { continue; } - events.add(new IndexEvent().withAction(IndexEventAction.INDEX).withTableName(tableName).withRecordId(recordId).withRecord(record)); + QRecord safeRecord = new QRecord(); + for(var field : QuickSearchFieldRules.indexableFields(QuickSearchFieldRules.lookupTable(tableName))) + { + safeRecord.setValue(field.getName(), record.getValue(field.getName())); + safeRecord.setDisplayValue(field.getName(), record.getDisplayValue(field.getName())); + } + if(!QuickSearchFieldRules.requiresFieldFiltering(tableName)) + { + safeRecord.setRecordLabel(record.getRecordLabel()); + } + events.add(new IndexEvent().withAction(IndexEventAction.INDEX).withTableName(tableName).withRecordId(recordId).withRecord(safeRecord)); } return (events); } diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchDiagnosticsOperations.java b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchDiagnosticsOperations.java index ce16b90..c5928e4 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchDiagnosticsOperations.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchDiagnosticsOperations.java @@ -33,6 +33,8 @@ import com.fasterxml.jackson.databind.node.JsonNodeFactory; import com.fasterxml.jackson.databind.node.ObjectNode; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; +import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; import jakarta.json.stream.JsonGenerator; import org.opensearch.client.json.JsonpMapper; import org.opensearch.client.json.JsonpSerializable; @@ -42,6 +44,7 @@ import org.opensearch.client.opensearch._types.query_dsl.Query; import org.opensearch.client.opensearch.core.ExplainResponse; import org.opensearch.client.opensearch.core.SearchRequest; import org.opensearch.client.opensearch.core.SearchResponse; +import org.opensearch.client.opensearch.core.search.Highlight; import org.opensearch.client.opensearch.core.search.Hit; import org.opensearch.client.opensearch.core.search.HighlightField; import org.opensearch.client.opensearch.core.search.MatchedQueries; @@ -169,13 +172,14 @@ public class QuickSearchDiagnosticsOperations { int boundedSize = Math.max(1, Math.min(size, MAX_PROBE_SIZE)); Query query = quickSearchClient.buildSearchQuery(term, List.of(tableName), tableConfigs, true); + Highlight highlight = QuickSearchFieldRules.requiresFieldFiltering(tableName) ? null : Highlight.of(h -> h + .requireFieldMatch(true) + .fields("searchableText", HighlightField.of(hf -> hf)) + .fields("fieldValues.*", HighlightField.of(hf -> hf))); SearchRequest request = SearchRequest.of(r -> r .index(indexName) .query(query) - .highlight(h -> h - .requireFieldMatch(true) - .fields("searchableText", HighlightField.of(hf -> hf)) - .fields("fieldValues.*", HighlightField.of(hf -> hf))) + .highlight(highlight) .trackTotalHits(t -> t.count(QuickSearchOpenSearchClient.MAX_RESULT_WINDOW)) .size(boundedSize)); @@ -189,17 +193,26 @@ public class QuickSearchDiagnosticsOperations for(Hit hit : response.hits().hits()) { OpenSearchDocument source = hit.source(); + boolean filterFields = QuickSearchFieldRules.requiresFieldFiltering(tableName); + if(source != null) + { + source = IndexingUtils.filterResult(source, tableConfigs.stream().filter(c -> tableName.equals(c.getTableName())).findFirst().orElse(null)); + } + if(filterFields && source == null) + { + continue; + } hits.add(new ProbeHit( source == null ? idFromDocumentId(hit.id()) : source.getRecordId(), source == null ? null : source.getRecordLabel(), hit.score(), matchedClauses(hit.matchedQueries()), - hit.highlight() == null ? Map.of() : new LinkedHashMap<>(hit.highlight()))); + filterFields || hit.highlight() == null ? Map.of() : new LinkedHashMap<>(hit.highlight()))); } Long total = response.hits().total() == null ? (long) hits.size() : response.hits().total().value(); Boolean lowerBound = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte; - return (new SearchProbe(tableName, toJson(request), toJson(response), response.took(), wall, total, lowerBound, hits)); + return (new SearchProbe(tableName, toJson(request), QuickSearchFieldRules.requiresFieldFiltering(tableName) ? "{}" : toJson(response), response.took(), wall, total, lowerBound, hits)); } catch(OpenSearchException | IOException e) { @@ -308,7 +321,7 @@ public class QuickSearchDiagnosticsOperations requireOk(response, "Document read failed"); JsonNode body = response.body(); return (new DocumentView(body.path("found").asBoolean(false), body.has("_version") ? body.get("_version").asLong() : null, - body.path("_index").asText(null), body.get("_source"))); + body.path("_index").asText(null), filterDocumentSource(tableName, body.get("_source")))); } @@ -338,7 +351,7 @@ public class QuickSearchDiagnosticsOperations { if(doc.path("found").asBoolean(false) && doc.has("_source")) { - sources.put(idFromDocumentId(doc.path("_id").asText()), doc.get("_source")); + sources.put(idFromDocumentId(doc.path("_id").asText()), filterDocumentSource(tableName, doc.get("_source"))); } } return (sources); @@ -346,6 +359,39 @@ public class QuickSearchDiagnosticsOperations + /******************************************************************************* + ** Old documents may retain protected values. Diagnostic reads expose only + ** attributable, indexable fields, never the old combined presentation. + *******************************************************************************/ + private static JsonNode filterDocumentSource(String tableName, JsonNode source) + { + if(source == null || !QuickSearchFieldRules.requiresFieldFiltering(tableName)) + { + return (source); + } + ObjectNode filtered = JsonNodeFactory.instance.objectNode(); + filtered.put("sourceTable", tableName); + var table = QuickSearchFieldRules.lookupTable(tableName); + if(!QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField())) + { + filtered.set("recordId", source.get("recordId")); + } + for(String container : List.of("fieldValues", "fieldValuesWord", "fieldValuesEnglish", "fieldValuesExact")) + { + ObjectNode values = filtered.putObject(container); + source.path(container).fields().forEachRemaining(entry -> + { + if(QuickSearchFieldRules.isIndexable(table, entry.getKey())) + { + values.set(entry.getKey(), entry.getValue()); + } + }); + } + return (filtered); + } + + + /******************************************************************************* ** Run _analyze on the index with a named analyzer (an index analyzer or a ** built-in one). Text is truncated to MAX_ANALYZE_TEXT_LENGTH. diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java index 8af15e7..5be70be 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java @@ -36,6 +36,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.logging.QLogger; import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldAnalyzer; import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; import org.opensearch.client.json.JsonData; @@ -53,6 +54,7 @@ import org.opensearch.client.opensearch._types.query_dsl.MultiMatchQuery; import org.opensearch.client.opensearch._types.query_dsl.Operator; import org.opensearch.client.opensearch._types.query_dsl.Query; import org.opensearch.client.opensearch._types.query_dsl.TermQuery; +import org.opensearch.client.opensearch._types.query_dsl.TextQueryType; import org.opensearch.client.opensearch.core.BulkRequest; import org.opensearch.client.opensearch.core.BulkResponse; import org.opensearch.client.opensearch.core.DeleteByQueryRequest; @@ -796,7 +798,8 @@ public class QuickSearchOpenSearchClient implements Closeable try { Query finalQuery = buildSearchQuery(searchTerm, allowedTables, tableConfigs); - Highlight highlight = Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf))); + Highlight highlight = allowedTables.stream().anyMatch(QuickSearchFieldRules::requiresFieldFiltering) ? null + : Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf))); SearchRequest request = SearchRequest.of(r -> r .index(indexName) @@ -941,9 +944,22 @@ public class QuickSearchOpenSearchClient implements Closeable // fields kept out of the snippet (KEYWORD, EXACT, or includeInSnippet // // off) match on their own; with none, the query is exactly as before // //////////////////////////////////////////////////////////////////////////// + List aggregateTables = allowedTables.stream().filter(t -> !QuickSearchFieldRules.requiresFieldFiltering(t)).toList(); + if(aggregateTables.isEmpty()) + { + snippetMatch = Query.of(q -> q.matchNone(m -> m)); + } + else if(aggregateTables.size() != allowedTables.size()) + { + Query aggregateMatch = snippetMatch; + snippetMatch = Query.of(q -> q.bool(b -> b.must(aggregateMatch).filter(f -> f.terms(t -> t.field("sourceTable") + .terms(v -> v.value(aggregateTables.stream().map(FieldValue::of).toList())))))); + } List ownMatches = buildOwnFieldMatchQueries(searchTerm, tableConfigs, allowedTables); - Query match = ownMatches.isEmpty() ? snippetMatch - : Query.of(q -> q.bool(b -> b.should(snippetMatch).should(ownMatches).minimumShouldMatch("1"))); + ownMatches.addAll(buildFilteredSnippetQueries(searchTerm, tableConfigs, allowedTables)); + Query safeSnippetMatch = snippetMatch; + Query match = ownMatches.isEmpty() ? safeSnippetMatch + : Query.of(q -> q.bool(b -> b.should(safeSnippetMatch).should(ownMatches).minimumShouldMatch("1"))); BoolQuery.Builder boolBuilder = new BoolQuery.Builder() .must(match) @@ -955,6 +971,36 @@ public class QuickSearchOpenSearchClient implements Closeable + /******************************************************************************* + ** For tables with excluded fields, match only attributable field values. + ** Old combined text may still contain fields that became classified later. + *******************************************************************************/ + private List buildFilteredSnippetQueries(String term, List configs, Collection allowedTables) + { + List queries = new ArrayList<>(); + for(QuickSearchableTableConfig config : configs == null ? List.of() : configs) + { + if(!allowedTables.contains(config.getTableName()) || !QuickSearchFieldRules.requiresFieldFiltering(config.getTableName())) + { + continue; + } + var table = QuickSearchFieldRules.lookupTable(config.getTableName()); + List fields = QuickSearchFieldRules.indexableFields(table, config.getSearchableFields()).stream() + .filter(config::isIncludedInSnippet).map(f -> PATH_PREFIX + "." + f).toList(); + if(fields.isEmpty() || QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField())) + { + continue; + } + queries.add(Query.of(q -> q.bool(b -> b + .filter(f -> f.term(t -> t.field("sourceTable").value(FieldValue.of(config.getTableName())))) + .must(m -> m.multiMatch(mm -> mm.query(term).fields(fields).type(TextQueryType.CrossFields) + .operator(Operator.And).lenient(true)))))); + } + return (queries); + } + + + /******************************************************************************* ** Matches for the fields that are not in searchableText, each limited to ** its own table: an EXACT field matches the whole (lower-cased) term, any @@ -965,7 +1011,7 @@ public class QuickSearchOpenSearchClient implements Closeable List queries = new ArrayList<>(); for(QuickSearchableTableConfig tableConfig : tableConfigs == null ? List.of() : tableConfigs) { - if(!allowedTables.contains(tableConfig.getTableName()) || tableConfig.isAllPlainFields()) + if(QuickSearchFieldRules.hasProtectedPrimaryKey(tableConfig.getTableName()) || !allowedTables.contains(tableConfig.getTableName()) || tableConfig.isAllPlainFields()) { continue; } @@ -973,7 +1019,8 @@ public class QuickSearchOpenSearchClient implements Closeable Query inTable = Query.of(q -> q.term(t -> t.field("sourceTable").value(FieldValue.of(tableConfig.getTableName())))); for(String field : tableConfig.getSearchableFields() == null ? List.of() : tableConfig.getSearchableFields()) { - if(tableConfig.isIncludedInSnippet(field)) + if(tableConfig.isIncludedInSnippet(field) + || (QuickSearchFieldRules.lookupTable(tableConfig.getTableName()) != null && !QuickSearchFieldRules.isIndexable(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()), field))) { continue; } @@ -1019,7 +1066,7 @@ public class QuickSearchOpenSearchClient implements Closeable for(QuickSearchableTableConfig tableConfig : tableConfigs) { - if(!allowedTables.contains(tableConfig.getTableName())) + if(QuickSearchFieldRules.hasProtectedPrimaryKey(tableConfig.getTableName()) || !allowedTables.contains(tableConfig.getTableName())) { continue; } @@ -1028,6 +1075,10 @@ public class QuickSearchOpenSearchClient implements Closeable List fields = tableConfig.getSearchableFields() == null ? List.of() : tableConfig.getSearchableFields(); for(String field : fields) { + if(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()) != null && !QuickSearchFieldRules.isIndexable(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()), field)) + { + continue; + } float boost = (weights == null || weights.get(field) == null) ? 1f : weights.get(field).floatValue(); QuickSearchFieldAnalyzer analyzer = tableConfig.getAnalyzer(field); if(analyzer == QuickSearchFieldAnalyzer.EXACT) diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/processes/AbstractIndexingStep.java b/src/main/java/com/kingsrook/qbits/quicksearch/processes/AbstractIndexingStep.java index 56b3cb0..f4fa517 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/processes/AbstractIndexingStep.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/processes/AbstractIndexingStep.java @@ -132,7 +132,7 @@ public abstract class AbstractIndexingStep implements BackendStep protected List getDiscoveredTables() { List tables = QuickSearchQBitContext.getDiscoveredTables(); - return (tables == null ? Collections.emptyList() : tables); + return (tables == null ? Collections.emptyList() : tables.stream().filter(t -> !QuickSearchFieldRules.hasProtectedPrimaryKey(t.getTableName())).toList()); } @@ -142,7 +142,7 @@ public abstract class AbstractIndexingStep implements BackendStep *******************************************************************************/ protected QuickSearchableTableConfig getTableConfig(String tableName) { - return (QuickSearchQBitContext.getTableConfig(tableName)); + return (QuickSearchFieldRules.hasProtectedPrimaryKey(tableName) ? null : QuickSearchQBitContext.getTableConfig(tableName)); } diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java b/src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java index b149f2a..fc365be 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java @@ -46,6 +46,7 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigWrites; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndex; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndexRun; @@ -506,6 +507,11 @@ public class BasepullIndexStep extends AbstractIndexingStep for(Map.Entry> entry : deleteByTable.entrySet()) { + if(QuickSearchFieldRules.hasProtectedPrimaryKey(entry.getKey())) + { + markReplayOutcome(entry.getValue(), false, List.of("The primary key cannot be indexed"), succeeded, failed); + continue; + } List toDelete = entry.getValue(); try { diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtils.java b/src/main/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtils.java index 0b72c2c..a686ce7 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtils.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtils.java @@ -33,6 +33,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.utils.StringUtils; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole; import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; @@ -103,6 +104,10 @@ public class IndexingUtils for(String fieldName : fields) { + if(table != null && !QuickSearchFieldRules.isIndexable(table, fieldName)) + { + continue; + } String value = displayValue(record, fieldName, maxFieldLength); if(value == null) { @@ -157,7 +162,7 @@ public class IndexingUtils *******************************************************************************/ public static OpenSearchDocument buildDocument(QRecord record, String tableName, String primaryKeyField, List fields, Map fieldWeights, Map includeLabels) { - return (buildDocument(record, tableName, primaryKeyField, fields, includeLabels, lookupTable(tableName), null, null)); + return (buildDocument(record, tableName, primaryKeyField, fields, includeLabels, lookupTable(tableName), null, null, null)); } @@ -165,13 +170,14 @@ public class IndexingUtils /******************************************************************************* ** Build a document for a configured table: display values, label fallback, ** truncation, and an external version from the basepull timestamp field. - ** Returns null when the record has no primary key. + ** Returns null when metadata is unavailable or the primary key is missing + ** or excluded by the field policy. *******************************************************************************/ public static OpenSearchDocument buildDocument(QRecord record, QuickSearchableTableConfig tableConfig) { QTableMetaData table = lookupTable(tableConfig.getTableName()); OpenSearchDocument doc = buildDocument(record, tableConfig.getTableName(), tableConfig.getPrimaryKeyField(), tableConfig.getSearchableFields(), - tableConfig.getFieldIncludeLabels(), table, tableConfig.getMaxFieldLength(), tableConfig.getBasepullTimestampField()); + tableConfig.getFieldIncludeLabels(), table, tableConfig.getMaxFieldLength(), tableConfig.getBasepullTimestampField(), tableConfig.getRecordLabelFields()); if(doc != null && tableConfig.getRecordLabelFormat() != null && tableConfig.getRecordLabelFields() != null) { @@ -199,7 +205,7 @@ public class IndexingUtils *******************************************************************************/ static void applyRolesAndAnalyzers(OpenSearchDocument doc, QRecord record, QuickSearchableTableConfig tableConfig, QTableMetaData table) { - List fields = tableConfig.getSearchableFields() == null ? List.of() : tableConfig.getSearchableFields(); + List fields = table == null ? List.of() : QuickSearchFieldRules.indexableFields(table, tableConfig.getSearchableFields()); if(!tableConfig.isAllPlainFields()) { List snippetFields = new ArrayList<>(); @@ -299,8 +305,14 @@ public class IndexingUtils ** *******************************************************************************/ private static OpenSearchDocument buildDocument(QRecord record, String tableName, String primaryKeyField, List fields, Map includeLabels, - QTableMetaData table, Integer maxFieldLength, String timestampField) + QTableMetaData table, Integer maxFieldLength, String timestampField, List recordLabelFields) { + if(table == null || primaryKeyField == null || table.getField(primaryKeyField) == null || QuickSearchFieldRules.isProtected(table, primaryKeyField)) + { + QuickSearchFieldRules.warnExcludedFields(table); + return (null); + } + fields = QuickSearchFieldRules.indexableFields(table, fields); String recordId = record.getValueString(primaryKeyField); if(recordId == null) { @@ -318,7 +330,15 @@ public class IndexingUtils } String recordLabel = record.getRecordLabel(); - if(!StringUtils.hasContent(recordLabel) && table != null) + boolean protectedLabel = table.getRecordLabelFields() != null + && table.getRecordLabelFields().stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f)); + if(protectedLabel || (recordLabelFields != null && recordLabelFields.stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f)))) + { + recordLabel = null; + record = new QRecord(record); + record.setRecordLabel(null); + } + if(!StringUtils.hasContent(recordLabel) && !protectedLabel && table.getRecordLabelFields() != null) { try { @@ -331,7 +351,7 @@ public class IndexingUtils } Long version = null; - if(timestampField != null) + if(timestampField != null && !QuickSearchFieldRules.isProtected(table, timestampField)) { try { @@ -361,6 +381,12 @@ public class IndexingUtils *******************************************************************************/ private static String formatLabel(QRecord record, QuickSearchableTableConfig tableConfig, String fallback) { + QTableMetaData table = lookupTable(tableConfig.getTableName()); + if(tableConfig.getRecordLabelFields().stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f))) + { + QuickSearchFieldRules.warnExcludedFields(table); + return (fallback); + } try { Object[] labelValues = new Object[tableConfig.getRecordLabelFields().size()]; @@ -373,13 +399,42 @@ public class IndexingUtils } catch(Exception e) { - LOG.warn("recordLabelFormat could not be applied; using default label", e, logPair("tableName", tableConfig.getTableName())); + LOG.warn("recordLabelFormat could not be applied; using default label", logPair("tableName", tableConfig.getTableName())); return (fallback); } } + /******************************************************************************* + ** Rebuild result presentation from allowed field values, discarding old + ** aggregate labels, snippets and display values that may contain secrets. + *******************************************************************************/ + public static OpenSearchDocument filterResult(OpenSearchDocument source, QuickSearchableTableConfig tableConfig) + { + if(!QuickSearchFieldRules.requiresFieldFiltering(source.getSourceTable())) + { + return (source); + } + QTableMetaData table = lookupTable(source.getSourceTable()); + if(tableConfig == null || QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField())) + { + return (null); + } + QRecord record = new QRecord().withValue(table.getPrimaryKeyField(), source.getRecordId()); + for(String field : QuickSearchFieldRules.indexableFields(table, tableConfig.getSearchableFields())) + { + Object value = source.getFieldValues() == null ? null : source.getFieldValues().get(field); + if(value instanceof String text) + { + record.setValue(field, text); + } + } + return (buildDocument(record, tableConfig)); + } + + + /******************************************************************************* ** *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigCustomizersTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigCustomizersTest.java index af69a48..431948f 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigCustomizersTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigCustomizersTest.java @@ -19,8 +19,13 @@ package com.kingsrook.qbits.quicksearch.config; import java.util.List; import java.util.Map; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.logging.QCollectingLogger; +import com.kingsrook.qqq.backend.core.logging.QLogger; import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; @@ -28,6 +33,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput; import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue; +import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent; import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldAnalyzer; import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole; @@ -47,6 +54,106 @@ import static org.assertj.core.api.Assertions.assertThat; class QuickSearchConfigCustomizersTest extends BaseDataDrivenTest { + /******************************************************************************* + ** Classification falls back to the table; PUBLIC and unclassified fields + ** remain searchable. Password, hidden, sensitive and locked fields do not. + *******************************************************************************/ + @Test + void testClassifiedAndSecuredFieldRules() + { + QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + assertThat(QuickSearchFieldRules.isIndexable(table, "name")).isTrue(); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse(); + for(DataClassification classification : DataClassification.values()) + { + table.getField("name").withSupplementalMetaData(new AiFieldMetaData().withClassification(classification)); + assertThat(QuickSearchFieldRules.isIndexable(table, "name")).as(classification.name()) + .isEqualTo(classification != DataClassification.PII && classification != DataClassification.PHI); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).as(classification.name()) + .isEqualTo(classification == DataClassification.PII || classification == DataClassification.PHI); + } + table.withSupplementalMetaData(new AiTableMetaData().withClassification(DataClassification.PHI)); + assertThat(QuickSearchFieldRules.whyNotIndexable(table, "description")).contains("classified PHI"); + assertThat(QuickSearchFieldRules.isProtected(table, "description")).isTrue(); + table.getField("name").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PUBLIC)); + assertThat(QuickSearchFieldRules.isIndexable(table, "name")).isTrue(); + table.getField("name").setFieldSecurityLock(new FieldSecurityLock()); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue(); + assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("security lock"); + table.getField("name").setFieldSecurityLock(null); + table.getField("name").setType(QFieldType.PASSWORD); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue(); + assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("sensitive"); + table.getField("name").setType(QFieldType.STRING); + table.getField("name").setIsHidden(true); + assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("hidden"); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse(); + for(QFieldType type : List.of(QFieldType.BOOLEAN, QFieldType.BLOB)) + { + table.getField("name").setType(type); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse(); + } + table.getField("name").setType(null); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse(); + table.getField("name").setIsSensitive(true); + assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue(); + assertThat(QuickSearchFieldRules.isProtected(null, "name")).isFalse(); + assertThat(QuickSearchFieldRules.isProtected(table, null)).isFalse(); + assertThat(QuickSearchFieldRules.isProtected(table, "missing")).isFalse(); + } + + + + /******************************************************************************* + ** Repeated config loads and document writes warn only once per table. + *******************************************************************************/ + @Test + void testExcludedFieldsWarnOncePerTable() + { + QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + QCollectingLogger logger = QLogger.activateCollectingLoggerForClass(QuickSearchFieldRules.class); + try + { + for(int i = 0; i < 3; i++) + { + assertThat(QuickSearchFieldRules.indexableFields(table, List.of("name", "description"))).containsExactly("name"); + } + } + finally + { + QLogger.deactivateCollectingLoggerForClass(QuickSearchFieldRules.class); + } + assertThat(logger.getCollectedMessages()).singleElement() + .satisfies(message -> assertThat(message.getMessage()).contains("dropped fields", TEST_ENTITY_TABLE, "PII/PHI")); + } + + + + /******************************************************************************* + ** Config writes refuse classified fields with a useful message; the same + ** fields are absent from the picker and default selection. + *******************************************************************************/ + @Test + void testClassifiedFieldConfigIsRefused() throws QException + { + runtime.getConfigStore().tryPrime(); + Integer indexId = indexRow(TEST_ENTITY_TABLE).getValueInteger("id"); + QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + for(DataClassification classification : List.of(DataClassification.PII, DataClassification.PHI)) + { + String fieldName = classification.name().toLowerCase(java.util.Locale.ROOT); + table.withField(new QFieldMetaData(fieldName, QFieldType.STRING) + .withSupplementalMetaData(new AiFieldMetaData().withClassification(classification))); + assertThat(insertField(indexId, fieldName, 1, null).getErrors()).singleElement() + .satisfies(e -> assertThat(e.getMessage()).contains("classified " + classification, "cannot be indexed")); + assertThat(QuickSearchFieldRules.indexableFields(table)).extracting(QFieldMetaData::getName).doesNotContain(fieldName); + assertThat(QuickSearchFieldRules.defaultFields(table)).extracting(QuickSearchFieldRules.DefaultField::fieldName).doesNotContain(fieldName); + } + } + + + /******************************************************************************* ** Field rows: a field must exist, be indexable, not sensitive or hidden, ** and not be listed twice; weights 1-1000; known role and analyzer. diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java index 5aac4a7..38f1ec9 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java @@ -24,6 +24,8 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; @@ -147,6 +149,28 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest + /******************************************************************************* + ** A custom search publisher also receives only allowed values, so it cannot + ** bypass the document builder's policy by indexing the event record. + *******************************************************************************/ + @Test + void testCustomPublisherReceivesNoClassifiedValues() throws QException + { + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).getField("description") + .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).withRecordLabelFields("description").withRecordLabelFormat("%s"); + new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecord(new QRecord() + .withValue("name", "Public widget").withValue("description", "PHI-CANARY-7f3c"))); + ArgumentCaptor> captor = ArgumentCaptor.forClass(List.class); + verify(publisher).publishIndexEvents(captor.capture()); + QRecord record = captor.getValue().get(0).getRecord(); + assertThat(record.getValues()).containsEntry("name", "Public widget").doesNotContainKey("description"); + assertThat(record.getDisplayValue("description")).isNull(); + assertThat(record.getRecordLabel()).isNull(); + } + + + @Test void testAppliesTo_onlyConfiguredTablesWhenEnabled() { diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/opensearch/ClassifiedDataCanaryTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/opensearch/ClassifiedDataCanaryTest.java new file mode 100644 index 0000000..9e14cad --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/quicksearch/opensearch/ClassifiedDataCanaryTest.java @@ -0,0 +1,193 @@ +/* + * Copyright 2024 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.kingsrook.qbits.quicksearch.opensearch; + + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule; +import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest; +import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig; +import com.kingsrook.qbits.quicksearch.QuickSearchQBitContext; +import com.kingsrook.qbits.quicksearch.QuickSearchRuntime; +import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction; +import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput; +import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener; +import com.kingsrook.qbits.quicksearch.processes.BasepullIndexStep; +import com.kingsrook.qbits.quicksearch.processes.IndexingUtils; +import com.kingsrook.qbits.quicksearch.publisher.SynchronousIndexEventPublisher; +import org.junit.jupiter.api.Test; +import org.opensearch.client.opensearch.core.SearchResponse; +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + + +/******************************************************************************* + ** Canary coverage at the backend seam for live writes, batch indexing and + ** stale search results. No OpenSearch server is required. + *******************************************************************************/ +class ClassifiedDataCanaryTest extends BaseQuickSearchTest +{ + private static final String CANARY = "PHI-CANARY-7f3c"; + + + + /******************************************************************************* + ** Classified values must never reach any document representation or result, + ** even with explicit field lists, label formats, roles and analyzer copies. + *******************************************************************************/ + @Test + void testCanaryAcrossRealtimeBasepullAndResults() throws Exception + { + QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + table.withField(new QFieldMetaData("personal", QFieldType.STRING) + .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII))); + table.withRecordLabelFields("description").withRecordLabelFormat("%s"); + + QuickSearchableTableConfig fields = new QuickSearchableTableConfig().withTableName(TEST_ENTITY_TABLE).withPrimaryKeyField("id") + .withSearchableFields(List.of("name", "description", "personal")).withEnabledByDefault(true) + .withBasepullTimestampField("modifyDate").withBasepullIntervalMinutes(5) + .withRecordLabelFields(List.of("description", "personal")).withRecordLabelFormat("%s %s") + .withFieldRoles(Map.of("name", "TITLE", "description", "SUBTITLE", "personal", "DISPLAY")) + .withFieldAnalyzers(Map.of("description", "ENGLISH", "personal", "EXACT")); + QuickSearchOpenSearchClient client = mock(QuickSearchOpenSearchClient.class); + List captured = new ArrayList<>(); + doAnswer(invocation -> + { + List documents = invocation.getArgument(0); + captured.addAll(documents); + return (new BulkIndexResult().withSuccessCount(documents.size())); + }).when(client).indexDocuments(anyList(), anyInt()); + when(client.indexExists()).thenReturn(true); + when(client.countAllDocuments()).thenReturn(1_000L); + when(client.countDocumentsForTable(anyString())).thenReturn(1L); + + QuickSearchQBitConfig config = QuickSearchQBitContext.getConfig(); + config.withIndexEventPublisher(new SynchronousIndexEventPublisher(client, List.of(fields), 100)); + QuickSearchRuntime runtime = spy(new QuickSearchRuntime(config)); + doReturn(client).when(runtime).getClient(); + runtime.setDiscoveredTables(List.of(fields)); + config.setRuntime(runtime); + QContext.getQInstance().addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config)); + QContext.getQInstance().withRecordChangeListener(new QCodeReference(QuickSearchRecordChangeListener.class)); + QuickSearchQBitContext.setClient(client); + QuickSearchQBitContext.setDiscoveredTables(List.of(fields)); + + new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecord(new QRecord() + .withValue("name", "Public widget").withValue("description", CANARY).withValue("personal", CANARY))); + assertThat(captured).as("real-time indexing reached the backend").hasSize(1); + new BasepullIndexStep().run(new RunBackendStepInput(), new RunBackendStepOutput()); + assertThat(captured.size()).as("basepull also reached the backend").isGreaterThan(1); + ObjectMapper mapper = new ObjectMapper().registerModule(new JavaTimeModule()); + for(OpenSearchDocument document : captured) + { + assertThat(mapper.writeValueAsString(document)).doesNotContain(CANARY).contains("Public widget"); + assertThat(document.getFieldValues()).containsEntry("name", "Public widget").doesNotContainKeys("description", "personal"); + } + + OpenSearchDocument stale = new OpenSearchDocument().withSourceTable(TEST_ENTITY_TABLE).withRecordId("1") + .withRecordLabel(CANARY).withSearchableText(CANARY).withSubtitle(CANARY).withDisplayValues(Map.of("Personal", CANARY)) + .withFieldValues(Map.of("name", "Public widget", "description", CANARY, "personal", CANARY)); + SearchResponse response = new SearchResponse.Builder().took(1).timedOut(false) + .shards(sh -> sh.total(1).successful(1).failed(0)) + .hits(h -> h.hits(hit -> hit.index("test").id("1").source(stale).highlight(Map.of("searchableText", List.of(CANARY))))).build(); + when(client.search(anyString(), anyList(), anyInt(), anyInt(), anyList(), any())).thenReturn(response); + config.setApplyRecordSecurityLocks(false); + var result = new QuickSearchAction().execute(new QuickSearchInput().withSearchTerm("Public")); + assertThat(result.getResults()).hasSize(1); + assertThat(mapper.writeValueAsString(result)).doesNotContain(CANARY).contains("Public widget"); + + try(QuickSearchOpenSearchClient queryClient = new QuickSearchOpenSearchClient(config)) + { + String request = queryClient.buildSearchQuery("Public", List.of(TEST_ENTITY_TABLE), List.of(fields), false).toJsonString(); + assertThat(request).doesNotContain("searchableText", "fieldValues.description", "fieldValuesEnglish.description", "fieldValuesExact.personal", CANARY); + assertThat(request).contains("fieldValues.name"); + } + } + + + + /******************************************************************************* + ** Diagnostic document views must not reveal protected values retained in + ** old index documents, including analyzer copies and combined labels. + *******************************************************************************/ + @Test + void testDiagnosticDocumentReadsOmitClassifiedValues() throws Exception + { + QContext.getQInstance().getTable(TEST_ENTITY_TABLE).getField("description") + .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + OpenSearchDocument stale = new OpenSearchDocument().withRecordId("1").withSourceTable(TEST_ENTITY_TABLE) + .withRecordLabel(CANARY).withSearchableText(CANARY).withSubtitle(CANARY).withDisplayValues(Map.of("Secret", CANARY)) + .withFieldValues(Map.of("name", "Public widget", "description", CANARY)) + .withFieldValuesWord(Map.of("description", CANARY)).withFieldValuesEnglish(Map.of("description", CANARY)) + .withFieldValuesExact(Map.of("description", CANARY)); + ObjectMapper mapper = new ObjectMapper().registerModule(new JavaTimeModule()); + var body = mapper.valueToTree(Map.of("found", true, "_source", stale, + "docs", List.of(Map.of("found", true, "_id", TEST_ENTITY_TABLE + ":1", "_source", stale)))); + QuickSearchOpenSearchClient client = mock(QuickSearchOpenSearchClient.class); + when(client.getIndexName()).thenReturn("test"); + QuickSearchDiagnosticsOperations ops = spy(new QuickSearchDiagnosticsOperations(client)); + doReturn(new QuickSearchDiagnosticsOperations.RawResponse(200, body)).when(ops).raw(anyString(), anyString(), any()); + assertThat(ops.getDocument(TEST_ENTITY_TABLE, "1").source().toString()).doesNotContain(CANARY).contains("Public widget"); + assertThat(ops.getDocuments(TEST_ENTITY_TABLE, List.of("1")).get("1").toString()).doesNotContain(CANARY).contains("Public widget"); + } + + + + /******************************************************************************* + ** Recheck metadata at document construction, including primary keys and + ** untrusted preformatted labels, without depending on config priming. + *******************************************************************************/ + @Test + void testTableClassificationAndProtectedPrimaryKey() throws Exception + { + QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE); + table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + table.withRecordLabelFields("description").withRecordLabelFormat("%s"); + QRecord record = new QRecord().withValue("id", 1).withValue("name", "Public widget").withValue("description", CANARY); + record.setRecordLabel(CANARY); + OpenSearchDocument document = IndexingUtils.buildDocument(record, QuickSearchQBitContext.getDiscoveredTables().get(0)); + assertThat(document.getRecordLabel()).isNull(); + assertThat(document.getSearchableText()).isEqualTo("Public widget"); + table.getField("id").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)); + assertThat(IndexingUtils.buildDocument(record, QuickSearchQBitContext.getDiscoveredTables().get(0))).isNull(); + } +} diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtilsTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtilsTest.java index 1f09ebd..c566fb8 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtilsTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtilsTest.java @@ -21,6 +21,8 @@ import java.time.Instant; import java.util.HashMap; import java.util.List; import java.util.Map; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.model.data.QRecord; import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; @@ -33,8 +35,10 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.model.session.QSession; import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; +import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules; import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import static org.assertj.core.api.Assertions.assertThat; @@ -45,6 +49,143 @@ import static org.assertj.core.api.Assertions.assertThat; class IndexingUtilsTest { + /******************************************************************************* + ** Document building requires field metadata to apply the index policy. + *******************************************************************************/ + @BeforeEach + void setUpMetadata() + { + QInstance instance = new QInstance(); + instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class)); + instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS)); + for(String tableName : List.of("customers", "person", "product", "order")) + { + QTableMetaData table = new QTableMetaData().withName(tableName).withBackendName("memory").withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER)) + .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)); + for(String field : List.of("name", "description", "firstName", "lastName", "email", "statusId", "quantity", "notes", "sku")) + { + table.withField(new QFieldMetaData(field, QFieldType.STRING)); + } + instance.addTable(table); + } + QContext.init(instance, new QSession()); + } + + + + /******************************************************************************* + ** Unsupported search fields must not erase labels or rebuild results. + *******************************************************************************/ + @Test + void testUnprotectedBooleanAndBlobPreserveLabelAndResult() + { + QTableMetaData table = QContext.getQInstance().getTable("product"); + table.withField(new QFieldMetaData("active", QFieldType.BOOLEAN)) + .withField(new QFieldMetaData("attachment", QFieldType.BLOB)) + .withRecordLabelFields("name").withRecordLabelFormat("%s"); + QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget") + .withValue("active", true).withValue("attachment", new byte[] { 1, 2 }); + record.setRecordLabel("Widget label"); + QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id") + .withSearchableFields(List.of("name", "active", "attachment")); + + OpenSearchDocument document = IndexingUtils.buildDocument(record, config); + assertThat(document.getRecordLabel()).isEqualTo("Widget label"); + assertThat(document.getSearchableText()).isEqualTo("Widget"); + assertThat(document.getFieldValues()).containsExactlyEntriesOf(Map.of("name", "Widget")); + assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isFalse(); + assertThat(IndexingUtils.filterResult(document, config)).isSameAs(document); + + record.setRecordLabel(null); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget"); + config.withRecordLabelFields(List.of("name", "active")).withRecordLabelFormat("%s: %s"); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget: true"); + table.withRecordLabelFields("name", "active").withRecordLabelFormat("%s: %s"); + config.withRecordLabelFormat(null); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget: true"); + } + + + + /******************************************************************************* + ** A protected field unrelated to the label must not erase a safe label. + *******************************************************************************/ + @Test + void testUnrelatedProtectedFieldPreservesLabel() + { + QTableMetaData table = QContext.getQInstance().getTable("product"); + table.withRecordLabelFields("name").withRecordLabelFormat("%s"); + table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget").withValue("description", "PHI-CANARY"); + record.setRecordLabel("Widget label"); + QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id") + .withSearchableFields(List.of("name", "description")); + + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget label"); + record.setRecordLabel(null); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget"); + assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isTrue(); + } + + + + /******************************************************************************* + ** Both table and configured label formats must exclude protected fields. + *******************************************************************************/ + @Test + void testProtectedLabelFieldsOmitPreformattedAndComputedLabels() + { + QTableMetaData table = QContext.getQInstance().getTable("person"); + table.getField("email").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)); + table.withRecordLabelFields("name", "email").withRecordLabelFormat("%s %s"); + QRecord record = new QRecord().withValue("id", 1).withValue("name", "Public name").withValue("email", "PII-CANARY"); + record.setRecordLabel("Public name PII-CANARY"); + QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("person").withPrimaryKeyField("id") + .withSearchableFields(List.of("name", "email")) + .withRecordLabelFields(List.of("email")).withRecordLabelFormat("%s"); + + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isNull(); + record.setRecordLabel(null); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isNull(); + table.withRecordLabelFields("name").withRecordLabelFormat("%s"); + record.setRecordLabel("PII-CANARY"); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Public name"); + assertThat(record.getRecordLabel()).isEqualTo("PII-CANARY"); + table.withRecordLabelFormat(null); + assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("1"); + } + + + + /******************************************************************************* + ** Hidden keys and timestamps remain usable unless they are protected. + *******************************************************************************/ + @Test + void testHiddenPrimaryKeyAndTimestampAreNotProtected() + { + QTableMetaData table = QContext.getQInstance().getTable("product"); + table.getField("id").withIsHidden(true); + table.getField("modifyDate").withIsHidden(true); + Instant timestamp = Instant.parse("2026-10-14T12:00:00Z"); + QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget").withValue("modifyDate", timestamp); + QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id") + .withSearchableFields(List.of("name")).withBasepullTimestampField("modifyDate"); + + OpenSearchDocument document = IndexingUtils.buildDocument(record, config); + assertThat(document.getRecordId()).isEqualTo("1"); + assertThat(document.getVersion()).isEqualTo(timestamp.toEpochMilli()); + assertThat(QuickSearchFieldRules.hasProtectedPrimaryKey("product")).isFalse(); + assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isFalse(); + table.getField("modifyDate").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI)); + assertThat(IndexingUtils.buildDocument(record, config).getVersion()).isNull(); + table.getField("id").withIsSensitive(true); + assertThat(QuickSearchFieldRules.hasProtectedPrimaryKey("product")).isTrue(); + assertThat(IndexingUtils.buildDocument(record, config)).isNull(); + } + + + /******************************************************************************* ** Test that normal input is trimmed and its case is preserved (the analyzer ** lower-cases at index and search time). @@ -606,7 +747,7 @@ class IndexingUtilsTest .withSearchableFields(List.of("sku")) .withFieldIncludeLabels(Map.of("sku", true)); - assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("sku: AB-1"); + assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("Sku: AB-1"); initContextWithPersonTable(); assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("SKU Code: AB-1"); diff --git a/src/test/java/com/kingsrook/qbits/quicksearch/publisher/SynchronousIndexEventPublisherTest.java b/src/test/java/com/kingsrook/qbits/quicksearch/publisher/SynchronousIndexEventPublisherTest.java index b0e9208..7b55c66 100644 --- a/src/test/java/com/kingsrook/qbits/quicksearch/publisher/SynchronousIndexEventPublisherTest.java +++ b/src/test/java/com/kingsrook/qbits/quicksearch/publisher/SynchronousIndexEventPublisherTest.java @@ -21,8 +21,13 @@ import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.Map; +import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest; import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig; import com.kingsrook.qbits.quicksearch.opensearch.BulkIndexResult; import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument; @@ -46,7 +51,7 @@ import static org.mockito.Mockito.when; /******************************************************************************* ** Tests for SynchronousIndexEventPublisher. *******************************************************************************/ -class SynchronousIndexEventPublisherTest +class SynchronousIndexEventPublisherTest extends BaseQuickSearchTest { private QuickSearchOpenSearchClient mockClient; private SynchronousIndexEventPublisher publisher; @@ -61,6 +66,13 @@ class SynchronousIndexEventPublisherTest @BeforeEach void setUp() { + QContext.getQInstance().addTable( + new QTableMetaData().withName("orders").withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER)) + .withField(new QFieldMetaData("orderNumber", QFieldType.STRING)) + .withField(new QFieldMetaData("customerName", QFieldType.STRING))); + QContext.getQInstance().addTable(new QTableMetaData().withName("products").withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER)).withField(new QFieldMetaData("name", QFieldType.STRING))); mockClient = mock(QuickSearchOpenSearchClient.class); QuickSearchableTableConfig ordersConfig = new QuickSearchableTableConfig()