Merge remote-tracking branch 'origin/develop' into feature/exclude-classified-fields

Secured-data spec section 15: Quick Search must never put classified
values in the shared search index, snippets or results.

QuickSearchFieldRules decides it in one place on QQQ 4.1.0-RC.6:
isProtected is true for fields classified PII or PHI (field, else the
table's classification), fields with a field security lock, sensitive
fields and masked/password types. Such fields are never indexable;
explicit configs drop them with a warning (never the value); a table
whose primary key is protected (for example a PHI table) is not indexed
at all. Record labels built from protected fields fall back, the
version field is skipped when protected, and results from tables with
protected fields are rebuilt from allowed values only, so documents
indexed before this change cannot leak through labels or snippets.
Tables with only harmless non-indexable fields (booleans, blobs) keep
their labels and are not result-filtered.

A canary test proves a PHI and a PII marker never reach a captured
index request, result or snippet. RC.7 follow-up: also honour core
ClassifiedValues.isClassified (qqq#1182) and FieldAccessHelper lock
tiers (qqq#1172).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.circleci/config.yml

@@ -1,7 +1,7 @@
 version: 2.1
 
 orbs:
-   qqq-orb: kingsrook/qqq-orb@0.6.8
+   qqq-orb: kingsrook/qqq-orb@0.6.9
 
 commands:
    configure-package-install:
@@ -22,7 +22,7 @@ workflows:
               context: [qqq-maven-registry-credentials]
               filters:
                  branches:
-                    ignore: /(develop|release\/.*|hotfix\/.*)/
+                    ignore: /(develop|release\/.*)/
 
    publish_snapshot:
       jobs:
@@ -58,14 +58,3 @@ workflows:
                     ignore: /.*/
                  tags:
                     only: [/^v\d+\.\d+\.\d+$/]
-
-   publish_hotfix_release:
-      jobs:
-         - qqq-orb/mvn_publish:
-              pre-steps:
-                 - configure-package-install
-              context: [qqq-maven-registry-credentials]
-              branch_type: hotfix
-              filters:
-                 branches:
-                    only: [/hotfix\/.*/]

.github/workflows/release-rc.yml

@@ -0,0 +1,25 @@
+# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
+# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
+# the release line and the release/* push runs CircleCI publish_release_candidate.
+name: Release candidate
+
+on:
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: Print the plan without merging or creating branches
+        type: boolean
+        default: false
+  pull_request:
+    types: [closed]
+    branches: [develop]
+
+permissions:
+  contents: write
+
+jobs:
+  release-rc:
+    if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
+    uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
+    with:
+      dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}

CHANGELOG.md

@@ -4,9 +4,13 @@ All notable changes to this project are documented here. The format follows Keep
 
 ## [Unreleased]
 
-Requires QQQ 4.1.0-RC.4 or later (the developer panel contract, QRun-IO/qqq#1076).
+Requires QQQ 4.1.0-RC.6 or later (the spec-managed UI API extension contract).
+
+### Fixed
+- Real-time indexing failures now increase the table's error count by the number of failed events. Successful basepulls and full reindexes reset the count. Counter update failures are logged without failing the source write.
 
 ### Added
+- Supported host status API (`api.QuickSearchStatusService.get()`), immutable status values, and `QuickSearchApi` home navigation constants (issue #50/F10). Permission-checked, bounded diagnostics refreshes share a short per-runtime cache and reuse saved validation counts. New spec-managed `GET /qqq/v1/qbit/quick-search/status` (`getQuickSearchStatus`) with committed OpenAPI fragment and strict HTTP/contract tests. RC.6 requires the default `/qbit/quick-search` prefix; `/quick-search` is not permitted by core. Existing routes and wire shapes are unchanged.
 - Record developer panel: `QuickSearchDeveloperPanelProvider` (id `quickSearch`) adds a "Quick Search" panel to the developer view of every record of an indexed table (`/app/{table}/{id}/dev` in the Next UI), for users who may read Quick Search Index. It shows the table's index row (enabled, status, record count, last basepull and full reindex, searchable fields), the latest index run, the record's indexed document compared field by field with the database (Inspect Search Document's comparison; needs `debugPermissionRules`), the record's failed index events, the Inspect Search Document and Reindex in Search actions the user may run, and links to the admin app and the index row. An unreachable OpenSearch shows a warning in the panel. Registered by `QuickSearchDeveloperPanelsMetaDataProducer`.
 
 - **Data-driven configuration.** The searchable tables and fields are records: `quick_search_index` (the table registry, new columns `source` and `ownerEdited`) and the new `quick_search_index_field` (weight, role `DISPLAY`/`TITLE`/`SUBTITLE`/`BODY`/`KEYWORD`/`EXACT`, analyzer `PREFIX`/`WORD`/`ENGLISH`/`EXACT`, snippet and label flags). Indexing, search, the record-search provider and the diagnostics read the effective config from them (cached 30 s, dropped on change), so changes need no redeploy. Code (`@QuickSearchable`, `searchableTables`) only seeds them; rows an admin edited are never overwritten, and removals are remembered in the new `quick_search_config_tombstone`. Precedence and DDL: `docs/MIGRATION-1.1.md`, "Data-driven configuration".
@@ -16,6 +20,8 @@ Requires QQQ 4.1.0-RC.4 or later (the developer panel contract, QRun-IO/qqq#1076
 - Possible-value sources `quick_search_eligible_table`, `quick_search_indexable_field`, `quick_search_timestamp_field`, `quick_search_field_role`, `quick_search_field_analyzer`, `quick_search_documents_choice`, `quick_search_config_source`; the join `quick_search_index_join_quick_search_index_field` and the Searchable Fields widget on the index record. Config tables are audited at field level.
 
 ### Changed
+- Distinguish protected fields from fields that simply cannot be searched. Boolean, blob and hidden fields no longer trigger result rebuilding or erase record labels. Labels are suppressed only when their source fields are protected; unprotected hidden primary keys and timestamps remain usable.
+- Exclude PII/PHI-classified fields (including table defaults) and field-security-locked fields everywhere Quick Search indexes or displays values. Configuration rejects protected fields; code defaults drop them with one warning per table. Search and diagnostics omit protected values from older documents. Rebuild existing indexes after upgrading or changing classification metadata. Uses QQQ 4.1.0-RC.6; no dependency changes.
 - `quick_search_index`: the generic Create and Delete are off (`TABLE_INSERT`, `TABLE_DELETE`); use Add Search Index and Remove Search Index. `quick_search_searchable_table` is now a custom source over the effective tables. `quick_search_index_run` gains `cancelRequested` and `cancelRequestedBy`.
 - Hidden and `isSensitive` fields are never indexed (previously hidden and password only).
 - Record screens (record-screen audit): `quick_search_index_run` and `quick_search_failed_event` are written by the QBit only, so Create and Edit are off (Delete stays, for clearing); both get named field sections, record labels and pickers (`quick_search_run_type`, `quick_search_run_status`, `quick_search_failed_event_status`, `quick_search_failed_event_action`, and the table picker for `tableName`). The index record lists its Recent Runs (join `quick_search_index_join_quick_search_index_run`). `quick_search_validation_run.reportJson` uses a JSON code editor. The Basepull, Full Reindex and Reconcile processes run from Quick Search Index's Actions and are no longer app (nav) children; their table input is `searchTableName` (`tableName` still works for API callers, and a Quick Search table name is ignored), so running them from the table no longer limits them to `quick_search_index`. Process input steps now carry `HELP_TEXT` and `EDIT_FORM` components, so the Next UI renders their fields.

README.md

@@ -257,6 +257,8 @@ The panel's status is Indexed, Stale, Not indexed, Disabled (the table is turned
 - `quick_search_validation_run` (with `enableDiagnosticTools`): `status` (`PASS`, `WARN`, `FAIL`), `triggerType` (`MANUAL`, `SCHEDULED`), `userName`, `tableNames`, `sampleSize`, `tablesChecked`, `failCount`, `warnCount`, `summary`, `reportHtml` (HTML), `reportJson` (text), `startTime`, `endTime`, `createDate`. Purged after `runHistoryRetentionDays`.
 - `quick_search_failed_event`: real-time events that could not be applied; `status` `PENDING` or `EXHAUSTED` after 10 attempts. Deletes made while a full reindex of all tables runs are also recorded here with status `AWAITING_REINDEX`: the reindex applies them to the new index after the alias swap (basepull does not replay them). While a table is `REBUILDING`, drift detection is suspended for it. Only one full reindex runs at a time: starting one while another is running fails. A full reindex shows it is running through its `FULL_REINDEX` run records, which stay `RUNNING` with `modifyDate` refreshed after every page. If it is killed mid-run (pod restart, OOM, deploy), recovery is automatic once its run records have shown no sign of life for `fullReindexStaleMinutes` (30 by default): the next basepull, or the next full reindex, marks those run records `FAILED`, returns the tables to `ACTIVE` with `lastBasepullTime` moved back to the run's start, and turns the `AWAITING_REINDEX` rows `PENDING` so basepull replays the deletes. No data is lost. A run that was only slow and is judged stopped notices before its alias swap and fails without swapping, or, after its swap, stops applying its captured deletes; basepull skips a replayed delete whose record exists again.
 
+Real-time error counts on Quick Search Index show failures since the last successful full reindex or basepull. Each failed event increases the count; deferred changes while indexing is paused do not. Counter updates are best effort and serialized across listeners in one JVM; concurrent application nodes can still race because QQQ has no arithmetic update API.
+
 ## Extending
 
 `IndexEventPublisher` lets a host replace the synchronous publisher with a queue. In 1.0 the SPI is experimental: `IndexEvent` carries the table, record id, action and the record snapshot, but no sequence number, so an asynchronous consumer must order INDEX and DELETE for the same record itself. `OpenSearchTransportCustomizer` customizes the HTTP transport.
@@ -288,3 +290,92 @@ See also the [QQQ adoption checklist](https://github.com/QRun-IO/qqq/blob/develo
 ## License
 
 Apache License, Version 2.0. See [LICENSE](LICENSE).
+
+## Status API for hosts
+
+Requires QQQ **4.1.0-RC.6**. Hosts should use the supported API rather than reach into
+`QuickSearchRuntime`, `QuickSearchOpenSearchClient`, diagnostic renderers, or qbit tables:
+
+```java
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatusService;
+
+// Runs in the host request's QContext, with the caller's QSession.
+QuickSearchStatus status = QuickSearchStatusService.get();
+String homeHref = status.homeHref();
+```
+
+UI clients use **`GET /qqq/v1/qbit/quick-search/status`**, operationId
+`getQuickSearchStatus`, extension `quick-search` version `1.0.0`. The endpoint returns
+that same immutable value as JSON; it requires the Quick Search app's configured
+`adminPermissionRules` (default `quick_search_admin.hasAccess`). Permission is checked
+before every read, including cache hits. No additional source-table permissions are
+required: like index validation, this is an administrator's aggregate view of the index.
+The committed contract is [quick-search.yaml](src/main/resources/openapi/qqq-v1/quick-search.yaml).
+
+The default core namespace is intentional: RC.6 permits `/qbit/quick-search` but rejects
+`/quick-search`, which is not an established legacy prefix. There were no existing
+Quick Search UI API routes to preserve. Core source is unchanged.
+
+- `state`: `OK`, `DEGRADED` (including paused indexing, backlog, incomplete diagnostics
+  or a refresh timeout), `DOWN` (unavailable health or red health), or `DISABLED`
+  (qbit absent or every persisted index disabled). Pausing indexing does not disable search.
+- `clusterReachable`, `clusterHealth`, `nodes`, `activeShards`, `unassignedShards` come
+  from the existing diagnostics health operation. Null means unknown/not probed.
+- `indexes` contains up to 100 persisted index rows. `documentCount` and `sourceCount`
+  come from the latest saved `IndexValidator` report, with `countsCheckedAt`. Without
+  matching validation, only the last indexing run's document count is available and
+  its observation time is unknown. These are **saved counts, not live counts**. A
+  missing source count must not be displayed as zero or “in sync.”
+- `pending` includes `PENDING` and `AWAITING_REINDEX` failed events; `exhausted` counts
+  failed events that gave up. At most 1000 backlog rows are read; `backlogTruncated`
+  makes both counts lower bounds. `tablesTruncated` indicates incomplete index rows.
+  These counters do not measure an external publisher's queue.
+- `lastSuccessfulSync` is the newest successful indexing watermark among the included
+  tables (basepull is start-anchored), not proof that all tables are current.
+  `lastError` is the newest available indexing/runtime search error or snapshot failure.
+- `checkedAt` is snapshot completion time. A per-runtime five-second cache bounds
+  polling cost; each refresh uses `DiagnosticsRateLimiter`'s per-user `STATUS` key.
+  Rate-limit refusal uses the existing diagnostics error (HTTP 400). A refresh waits
+  at most two seconds and retains at most one in-flight worker, even if a backend is
+  stuck. Runtime shutdown stops the worker. No source counts, record samples, full
+  validation, or per-table OpenSearch calls run on this polling path.
+
+Link to `status.homeHref()` instead of copying the Quick Search home. It includes the
+configured metadata prefix. `QuickSearchApi.APP_NAME` (`quickSearchAdmin`) and
+`QuickSearchApi.HOME_HREF` (`/app/quickSearchAdmin`) are available for default-name
+navigation. Keep detailed runs, failed events, validation and operator tools in the
+qbit's own home. [kof22 migration mapping](docs/STATUS-API-HOST-MIGRATION.md).
+
+## Classified data
+
+Quick Search excludes fields classified as **PII** or **PHI**, including fields
+that inherit the table's classification. It also excludes hidden, sensitive,
+password/masked fields and every field with a `FieldSecurityLock`; a shared
+search index cannot enforce access to individual fields for each user. PUBLIC,
+CONFIDENTIAL and unclassified fields remain eligible under the usual field rules.
+
+The same rule applies to annotation and code configuration, admin field choices,
+real-time writes, basepull, rebuilds, replay, labels, snippets, analyzer copies and
+result display fields. An admin trying to add a protected field gets an error.
+Protected fields in existing or explicit code configuration are dropped, with one
+warning per table metadata lifetime; warnings never contain field values. A
+protected primary key prevents indexing the record or sending its ID for deletion.
+
+Protection is separate from search eligibility: boolean, blob, hidden and other
+unsupported fields alone do not trigger result filtering or remove record labels.
+Labels are omitted when their source fields are protected; unrelated protected
+fields do not erase safe labels. Unprotected hidden keys and timestamps can still
+provide record IDs and document versions.
+
+After upgrading, or after changing a field's classification or security metadata,
+run **Full Reindex** across all indexed tables to remove previously indexed values.
+Until then, tables containing protected fields are searched through allowed field
+values, and results are rebuilt without old combined labels or highlights.
+Diagnostic views also omit protected values and the raw response for these tables.
+This does not purge old OpenSearch backups or retained physical indexes; apply your
+host's retention policy to those copies.
+
+This release uses the APIs in QQQ **4.1.0-RC.6**. Support for the host's
+`AuditValueProtector` classification (`ClassifiedValues`, qqq#1182) and
+`FieldAccessHelper.getLockTier` (qqq#1172) is deferred until the qbit moves to RC.7.

docs/SESSION-STATE.md

@@ -1,6 +1,25 @@
 # Session State
 
-Updated: 2026-10-06
+Updated: 2026-10-14
+
+## Real-time error counter fix (2026-10-14)
+
+- Failed-event recording now increments the table's `realTimeErrorCount` after commit, using QQQ reads and system writes. Null counts start at zero; update failures are logged and cannot fail the source write or prevent replay.
+- QQQ has no arithmetic update API. The read-modify-write fallback is serialized across listeners in this JVM; separate application nodes can still race.
+- Successful basepull and full-reindex status updates reset the count, including the single-table reindex path. Investigation found the previous zero assignments only initialized new rows. Field help, README and Unreleased changelog now describe the reset window.
+- Regression coverage includes event counts, concurrent listeners, counter-update failure, after-commit timing, basepull reset and both full-reindex reset paths.
+- Final gate: `./mvnw.cmd -o -B verify` passed against released QQQ 4.1.0-RC.6: 476 unit tests, zero failures/errors; 41 integration tests skipped because Docker is unavailable; architecture guardrails and JaCoCo coverage gates passed. No dependency changes or offline artifact gaps. Checkstyle, PMD and SpotBugs are not configured in this Maven build.
+- No commit, push or branch creation. Second-brain save is blocked by the vault being outside writable roots; this section retains the durable handoff for later vault synchronization.
+
+## Status API worktree (2026-10-13, issue #50 / F10)
+
+- Implemented `api.QuickSearchStatusService.get()`, immutable status values, default app/home constants and a spec-managed status endpoint. Admin permission is checked on every call; saved validation counts avoid source scans. Five-second per-runtime cache, two-second wait, one in-flight worker, existing diagnostics rate limiter, bounded table/backlog reads.
+- Endpoint uses core's permitted `/qqq/v1/qbit/quick-search/status` namespace (`getQuickSearchStatus`). Requested `/quick-search` is rejected by RC.6's established-prefix allowlist. Core and host clones are read-only; no edits there. No previous Quick Search HTTP operations or Next UI callers found.
+- POM targets released-coordinate `4.1.0-RC.6`; local verification uses only `-Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`.
+- Host file/line replacement mapping: `docs/STATUS-API-HOST-MIGRATION.md`. Contract: `src/main/resources/openapi/qqq-v1/quick-search.yaml`.
+- Verification: 468 unit tests (including strict HTTP, contract, bounds, cache isolation, permissions and timeout recovery); JaCoCo passes. All 41 Docker integration tests skip because Docker is unavailable. Final command: `./mvnw.cmd -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT`.
+- Standalone repository Checkstyle is not green: HEAD has 471 main-source violations; this tree has 438, with zero newly introduced violations. New production files and both new test files pass their style checks. Existing header/import/style cleanup is outside the status API change.
+- No commit, push, merge or deployment authorized. Second-brain vault is outside writable roots; durable notes are retained here for later vault synchronization.
 
 ## Context
 
@@ -29,3 +48,75 @@ Updated: 2026-10-06
 1. qqq-all and Voyage pin `1.0.0-RC.2` and report back; their feedback decides RC.3 or GA.
 2. Remaining "Before 1.0.0 GA" items in `docs/TODO.md`: stuck `REBUILDING` after a killed or overlapping full reindex (from the PR #20 review), and the `qbit-build-parent` 2.1.0 re-pin.
 3. GA: merge `release/1.0` to `main` and tag `v1.0.0`.
+
+## 2026-10-14 — secured-data section 15
+
+- Implemented a shared RC.6 classification/security policy in `QuickSearchFieldRules`.
+- Applied it to config discovery, config writes/choices, document construction,
+  safe query fields, result presentation and diagnostic reads; protected primary
+  keys cannot be used as index IDs.
+- Added live-listener/basepull/stale-result PHI/PII canaries and config/rule tests.
+- Final gate: `mvnw.cmd -o -B verify` **BUILD SUCCESS**. 479 unit tests passed
+  (7 new tests); 41 Docker integration tests skipped because Docker is unavailable.
+  Architecture guardrails passed. JaCoCo: 73.84% instruction / 91.61% class coverage,
+  above the 70% / 90% gates. Checkstyle, PMD and SpotBugs are not configured in this
+  build. No missing offline artifacts. `git diff --check` passed.
+- Reviewed the diff; changes remain uncommitted. Gate log:
+  `target/secured-data-verify.log`.
+- RC.7 host audit-protector classification and lock-tier APIs remain deferred.
+
+### Changed files
+
+- Documentation: `README.md`, `CHANGELOG.md`, `docs/SESSION-STATE.md`, `docs/TODO.md`.
+- Configuration: `QuickSearchQBitProducer.java`, `config/QuickSearchFieldRules.java`,
+  `config/QuickSearchConfigPrimer.java`, `config/QuickSearchConfigStore.java`.
+- Indexing: `listeners/QuickSearchRecordChangeListener.java`,
+  `processes/IndexingUtils.java`, `processes/AbstractIndexingStep.java`,
+  `processes/BasepullIndexStep.java`.
+- Search and diagnostics: `actions/QuickSearchAction.java`,
+  `actions/QuickSearchRecordSearchProvider.java`,
+  `opensearch/QuickSearchOpenSearchClient.java`,
+  `opensearch/QuickSearchDiagnosticsOperations.java`.
+- Tests: `config/QuickSearchConfigCustomizersTest.java`,
+  `listeners/QuickSearchRecordChangeListenerTest.java`,
+  `processes/IndexingUtilsTest.java`, `publisher/SynchronousIndexEventPublisherTest.java`,
+  new `opensearch/ClassifiedDataCanaryTest.java`.
+
+Production and test paths above are relative to
+`src/{main,test}/java/com/kingsrook/qbits/quicksearch/` respectively.
+
+### Continuity
+
+Second-brain project/daily writes are unavailable in this session because the
+vault lies outside the writable roots. This local checkpoint holds the durable
+outcome for the reviewer. No commit, push, branch creation, dependency version
+change, or production filesystem IO was introduced.
+
+
+## 2026-10-14 - protection review follow-up
+
+- Added isProtected for PII/PHI (including table defaults), field locks, sensitive
+  values and masked types. Unsupported search types and hidden fields alone are
+  not protected. Searchable/snippet/display field lists still use isIndexable.
+- Labels check their configured/table source fields; unrelated protected fields
+  preserve safe labels. A copied record prevents unsafe preformatted labels from
+  returning through QQQ fallback formatting without changing the input record.
+- Primary-key guards and timestamp versions now use protection rather than
+  search eligibility. Added four regression tests and extended rule/canary tests.
+- Follow-up files: config/QuickSearchFieldRules.java, processes/IndexingUtils.java,
+  listeners/QuickSearchRecordChangeListener.java,
+  opensearch/QuickSearchOpenSearchClient.java,
+  opensearch/QuickSearchDiagnosticsOperations.java; test files
+  config/QuickSearchConfigCustomizersTest.java, processes/IndexingUtilsTest.java,
+  opensearch/ClassifiedDataCanaryTest.java; README.md, CHANGELOG.md,
+  docs/TODO.md and this file. Java paths are relative to the existing main/test
+  quicksearch packages. All earlier uncommitted work is preserved.
+- Second-brain project/daily synchronization remains unavailable because the vault
+  is outside writable roots; this checkpoint is the local handoff. Nothing committed
+  or pushed; no branches or dependency changes.
+- Final gate: `mvnw.cmd -o -B verify` BUILD SUCCESS; 483 unit tests passed
+  (four additional tests); all three PHI/PII canaries passed. All 41 Docker
+  integration tests skipped because Docker is unavailable. Architecture guards
+  and JaCoCo passed (73.88% instruction / 91.61% class coverage).
+  Checkstyle, PMD and SpotBugs are not configured in this build. No offline
+  artifacts missing. Gate log: `target/protection-review-verify.log`.

docs/STATUS-API-HOST-MIGRATION.md

@@ -0,0 +1,56 @@
+# Quick Search host migration (kof22 #164)
+
+Issue #50 / architecture finding F10. Paths below are under
+`Website-Backend/src/main/java/com/kof22/website/metadata/adminhome/`;
+line numbers refer to the local clone inspected on 2026-10-13. No host files were changed.
+
+| Existing host code | Replacement |
+| --- | --- |
+| `LiveAdminHomeProbes.java:340` `searchConfigured()` | Use `QuickSearchApi.APP_NAME` for the default-name app lookup; for an authorized snapshot use `QuickSearchStatusService.get()` and its state/homeHref. `DISABLED` also includes all indexes disabled, so it is not exactly “qbit absent.” |
+| `LiveAdminHomeProbes.java:351` `searchCluster()` | One `QuickSearchStatusService.get()`; map `clusterReachable`, `clusterHealth`, `nodes`, `activeShards`, `unassignedShards`. Remove direct cluster HTTP/auth/CA handling. |
+| `LiveAdminHomeProbes.java:386` `searchDocumentCounts(...)` | Reuse the same snapshot's `indexes()` and each row's `tableName()` / `documentCount()` / `sourceCount()` / `countsCheckedAt()`. Remove `QuickSearchOpenSearchClient` access and relabel counts as saved diagnostics, not live. Preserve null as unknown. |
+| `QuickSearchHomeWidgets.java:118` `widgets()` | Stop adding the host's duplicate Quick Search home widgets. Link to `status.homeHref()` (or default `QuickSearchApi.HOME_HREF`). Keep the qbit-provided home/diagnostics widget. |
+| `QuickSearchHomeWidgets.java:133` `indexFacts()`, `:160` `summary()` and `:196` `syncTile()` | For a compact host overview, use the same `QuickSearchStatusService.get()` value: `state`, `indexes`, `pending`, `exhausted`, `lastSuccessfulSync`, `lastError`. Remove SQL aggregates and live source counts; use saved count timestamps and truncation flags. |
+| `QuickSearchHomeWidgets.java:267` `runs()`, `:315` `failedEvents()` | Replace duplicate detailed views with the home link; drill down using the qbit's own Index Run / Failed Event tables from that home. The summary API intentionally does not export detailed histories. |
+| `AdminOperationsWidgets.java:209` `quickSearch()` | Render `status.indexes()` for the compact admin overview, and use `status.homeHref()` for its footer link. Remove qbit-table SQL and source-table counts. |
+| `AdminOperationsWidgets.java:343` `searchFooter()` | Use `clusterHealth`, `pending`, `exhausted`, `lastSuccessfulSync`, `lastError`, and `checkedAt` from the same snapshot. Detailed recent-run information belongs in the qbit home. |
+
+Call with the existing request's `QContext`/`QSession`; do not elevate a host user's
+session to bypass `adminPermissionRules`. Cache hits still check permission.
+
+## UI contract
+
+| Method | Path | operationId |
+| --- | --- | --- |
+| GET | `/qqq/v1/qbit/quick-search/status` | `getQuickSearchStatus` |
+
+Extension: `quick-search`, version `1.0.0`; core dependency: `4.1.0-RC.6`.
+The requested `/quick-search` prefix cannot validate on the supplied RC.6 baseline:
+`MiddlewareSpecExtensionsMetaData.ESTABLISHED_PREFIXES` only admits esb, alerts,
+assistant, and sql-workbench. We use the default namespace without modifying core.
+
+Wire changes against develop: one additive endpoint and its schemas; no existing
+method, path, request, or response changed. Next UI `src/` contains no calls to a
+Quick Search-specific endpoint today; its core `/qqq/v1/search` integration is unchanged.
+Vendor `src/main/resources/openapi/qqq-v1/quick-search.yaml` for clients.
+
+Regenerate the fragment with:
+
+```powershell
+./mvnw.cmd -B test -Dtest=QuickSearchApiContractTest -Dquicksearch.openapi.update=true -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT
+```
+
+The local verification override is never the committed dependency version.
+
+## Verification and limits
+
+`./mvnw.cmd -B verify -Dqqq.snapshot.version=4.1.0-ARCHRC6-SNAPSHOT` passes
+468 unit tests and JaCoCo, including strict HTTP server startup, permission checks,
+OpenAPI contract consistency, cached/disabled/degraded/down states, timeout recovery,
+bounded backlog/table lists and prefixed QInstance isolation. All 41 Docker integration
+tests skip locally because Docker is unavailable.
+
+The separate Checkstyle check is **not green**: the existing checked-in rules report
+471 main-source violations at HEAD and 438 in this worktree, with zero new violations.
+New source files and the two new tests pass those rules. No repository-wide header
+rewrite or unrelated style cleanup was included.

docs/TODO.md

@@ -1,5 +1,18 @@
 # TODO
 
+## Real-time error counter
+
+- [x] Count failed real-time events after commit, serialize concurrent listeners within a JVM, tolerate counter-write failures, and reset after successful basepull/full reindex; add regression tests and field/documentation help.
+
+## Architecture status API (#50 / F10)
+
+- [x] Supported Java status API, bounded cache/rate limiting, administrator authorization, home navigation, spec endpoint and committed fragment.
+- [x] Contract and strict HTTP tests; host migration mapping for kof22#164.
+- [ ] kof22#164: replace internal probes/duplicate home using `docs/STATUS-API-HOST-MIGRATION.md`.
+- [ ] Next UI: vendor the additive Quick Search status fragment when adopting RC.6.
+- [ ] Run Docker integration tests in CI; local Docker is unavailable.
+- [ ] Resolve the pre-existing repository Checkstyle baseline (438 remaining main-source violations, down from 471 at HEAD); status API adds none.
+
 ## Before tagging 1.0.0
 
 - [ ] qqq-all and Voyage adopt `1.0.0-RC.2` (items below) and report back.
@@ -43,3 +56,12 @@
 - [x] Status assessment, deep-dive audit (25 blockers), PRD (2026-10-04).
 - [x] 4.1.0-RC.1 compatibility verified (2026-10-04).
 - [x] All 25 audit blockers implemented and tested on the release branch (2026-10-04).
+
+## Secured-data section 15 (2026-10-14)
+
+- [x] Review fix: separate protection from indexability; preserve safe labels, ordinary results and unprotected hidden keys/timestamps, with regression tests.
+
+- [x] Exclude PII/PHI, hidden, sensitive, password and field-locked values from indexing and results on RC.6.
+- [x] Add canary coverage for real-time indexing, basepull and stale results; refuse classified config fields.
+- [ ] On the QQQ RC.7 upgrade, honour `ClassifiedValues.isClassified(table, field)` (qqq#1182) and `FieldAccessHelper.getLockTier` (qqq#1172).
+- [ ] Deployment: rebuild existing indexes and apply host retention policy to old physical indexes/backups containing previously indexed values.

pom.xml

@@ -18,6 +18,8 @@
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
    <properties>
+      <qbit.guardrails.enforce>true</qbit.guardrails.enforce>
+      <qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
       <revision>1.1.0-SNAPSHOT</revision>
       <maven.compiler.source>21</maven.compiler.source>
       <maven.compiler.target>21</maven.compiler.target>
@@ -30,7 +32,7 @@
    <parent>
       <groupId>com.kingsrook</groupId>
       <artifactId>qbit-build-parent</artifactId>
-      <version>2.0.0</version>
+      <version>2.1.0</version>
    </parent>
 
    <groupId>com.kingsrook.qbits</groupId>
@@ -70,6 +72,11 @@
    </scm>
 
    <dependencies>
+      <dependency>
+         <groupId>com.kingsrook.qqq</groupId>
+         <artifactId>qqq-middleware-javalin</artifactId>
+      </dependency>
+
       <dependency>
          <groupId>com.kingsrook.qqq</groupId>
          <artifactId>qqq-backend-core</artifactId>
@@ -274,7 +281,7 @@
             <activeByDefault>true</activeByDefault>
          </activation>
          <properties>
-            <qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
+            <qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
          </properties>
          <dependencyManagement>
             <dependencies>

src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java

@@ -25,6 +25,15 @@ import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Properties;
+import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
+import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField;
+import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchApi;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
+import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
+import com.kingsrook.qbits.quicksearch.metadata.QuickSearchAdminAppMetaDataProducer;
+import com.kingsrook.qbits.quicksearch.metadata.QuickSearchDiagnosticsProcessMetaDataHelper;
+import com.kingsrook.qbits.quicksearch.metadata.QuickSearchProcessMetaDataHelper;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
@@ -38,13 +47,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
-import com.kingsrook.qbits.quicksearch.actions.QuickSearchRecordSearchProvider;
-import com.kingsrook.qbits.quicksearch.annotations.QuickSearchField;
-import com.kingsrook.qbits.quicksearch.annotations.QuickSearchable;
-import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
-import com.kingsrook.qbits.quicksearch.metadata.QuickSearchAdminAppMetaDataProducer;
-import com.kingsrook.qbits.quicksearch.metadata.QuickSearchDiagnosticsProcessMetaDataHelper;
-import com.kingsrook.qbits.quicksearch.metadata.QuickSearchProcessMetaDataHelper;
+import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtensionsMetaData;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -262,6 +265,9 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
 
       qInstance.withRuntimeService(new QCodeReference(QuickSearchRuntimeService.class));
 
+      MiddlewareSpecExtensionsMetaData.ofOrWithNew(qInstance)
+         .withExtension(QuickSearchApi.extension());
+
       addRecordDiagnosticActions(output, qInstance, discoveredTables);
 
       if(Boolean.TRUE.equals(qBitConfig.getServeCoreRecordSearch()))
@@ -375,8 +381,8 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
          for(String fieldName : CollectionUtils.nonNullList(tableConfig.getSearchableFields()))
          {
             QFieldMetaData field = table.getFields() == null ? null : table.getFields().get(fieldName);
-            if(field != null && !Boolean.TRUE.equals(field.getIsHidden()) && field.getType() != null
-               && (field.getType().isStringLike() || field.getType().isIntegral()) && !field.getType().needsMasked() && !field.getIsSensitive()
+            if(QuickSearchFieldRules.isIndexable(table, fieldName)
+               && (field.getType().isStringLike() || field.getType().isIntegral())
                && !searchFields.contains(fieldName))
             {
                searchFields.add(fieldName);
@@ -461,7 +467,7 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
          {
             primaryKeyField        = table.getPrimaryKeyField();
             basepullTimestampField = resolveTimestampField(table, searchableFields, basepullTimestampField);
-            removeHiddenFields(table, searchableFields, fieldWeights, fieldIncludeLabels);
+            removeExcludedFields(table, searchableFields, fieldWeights, fieldIncludeLabels);
          }
 
          tables.add(new QuickSearchableTableConfig()
@@ -518,7 +524,7 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
                warnIncrementalBasepullDisabled(table, basepullTimestampField);
                basepullTimestampField = null;
             }
-            removeHiddenFields(table, searchableFields, fieldWeights, fieldIncludeLabels);
+            removeExcludedFields(table, searchableFields, fieldWeights, fieldIncludeLabels);
          }
 
          Integer basepullInterval = stc.getBasepullIntervalMinutes();
@@ -598,23 +604,22 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer<QuickSearch
 
 
    /*******************************************************************************
-    ** Hidden, isSensitive and password fields are never indexed: their values
+    ** Classified, locked, hidden, sensitive and password fields are excluded:
     ** would leak through searchable text and highlights.
     *******************************************************************************/
-   private static void removeHiddenFields(QTableMetaData table, List<String> searchableFields, Map<String, Integer> fieldWeights, Map<String, Boolean> fieldIncludeLabels)
+   private static void removeExcludedFields(QTableMetaData table, List<String> searchableFields, Map<String, Integer> fieldWeights, Map<String, Boolean> fieldIncludeLabels)
    {
       List<String> hidden = new ArrayList<>();
       for(String fieldName : searchableFields)
       {
-         QFieldMetaData field = table.getFields().get(fieldName);
-         if(field != null && (Boolean.TRUE.equals(field.getIsHidden()) || field.getIsSensitive() || (field.getType() != null && field.getType().needsMasked())))
+         if(!QuickSearchFieldRules.isIndexable(table, fieldName))
          {
             hidden.add(fieldName);
          }
       }
       if(!hidden.isEmpty())
       {
-         LOG.warn("Hidden, sensitive and password fields are excluded from Quick Search indexing", logPair("tableName", table.getName()), logPair("fields", hidden));
+         QuickSearchFieldRules.warnExcludedFields(table);
          searchableFields.removeAll(hidden);
          hidden.forEach(fieldWeights::remove);
          hidden.forEach(fieldIncludeLabels::remove);

src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchRuntime.java

@@ -26,6 +26,13 @@ import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicLong;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigStore;
+import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticEventRecorder;
+import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsRateLimiter;
+import com.kingsrook.qbits.quicksearch.diagnostics.StatusCache;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
+import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher;
+import com.kingsrook.qbits.quicksearch.publisher.SynchronousIndexEventPublisher;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -38,12 +45,6 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
-import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigStore;
-import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticEventRecorder;
-import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsRateLimiter;
-import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
-import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher;
-import com.kingsrook.qbits.quicksearch.publisher.SynchronousIndexEventPublisher;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -71,6 +72,8 @@ public class QuickSearchRuntime
 
    public static final long RECORD_SEARCH_BACKOFF_MILLIS = 30_000L;
 
+   private final StatusCache statusCache = new StatusCache();
+
    private final QuickSearchQBitConfig config;
 
    private List<QuickSearchableTableConfig>        codeTables       = new ArrayList<>();
@@ -99,6 +102,16 @@ public class QuickSearchRuntime
 
 
 
+   /*******************************************************************************
+    ** Instance-owned bounded administrative status cache.
+    *******************************************************************************/
+   public StatusCache getStatusCache()
+   {
+      return (statusCache);
+   }
+
+
+
    /*******************************************************************************
     ** A cached enabled flag with the time it was read.
     *******************************************************************************/
@@ -728,6 +741,7 @@ public class QuickSearchRuntime
          return;
       }
       closed = true;
+      statusCache.close();
 
       if(publisher != null)
       {

src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchAction.java

@@ -46,6 +46,7 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchQBitContext;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
 import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
@@ -221,7 +222,7 @@ public class QuickSearchAction
       if(!applyLocks)
       {
          SearchResponse<OpenSearchDocument> response = runSearch(client, term, tables, pageSize, offset, tableConfigs, null);
-         List<QuickSearchResult>            results  = toResults(response);
+         List<QuickSearchResult>            results  = toResults(response, tableConfigs);
 
          long    totalHits  = response.hits().total() != null ? response.hits().total().value() : results.size();
          boolean lowerBound = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte;
@@ -255,7 +256,7 @@ public class QuickSearchAction
          rawFloor = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte;
 
          List<QuickSearchResult> candidates = new ArrayList<>();
-         for(QuickSearchResult result : toResults(response))
+         for(QuickSearchResult result : toResults(response, tableConfigs))
          {
             if(seen.add(result.getTableName() + ":" + result.getRecordId()))
             {
@@ -315,7 +316,7 @@ public class QuickSearchAction
       List<String> allowed          = new ArrayList<>();
       for(String tableName : new LinkedHashSet<>(requested))
       {
-         if(!configured.contains(tableName))
+         if(!configured.contains(tableName) || QuickSearchFieldRules.hasProtectedPrimaryKey(tableName))
          {
             continue;
          }
@@ -407,7 +408,7 @@ public class QuickSearchAction
    /*******************************************************************************
     **
     *******************************************************************************/
-   private static List<QuickSearchResult> toResults(SearchResponse<OpenSearchDocument> response)
+   private static List<QuickSearchResult> toResults(SearchResponse<OpenSearchDocument> response, List<QuickSearchableTableConfig> tableConfigs)
    {
       QInstance               qInstance = QContext.getQInstance();
       List<QuickSearchResult> results   = new ArrayList<>();
@@ -420,9 +421,16 @@ public class QuickSearchAction
             continue;
          }
 
-         String                    highlightSnippet = null;
+         String sourceTable = source.getSourceTable();
+         boolean filterFields = QuickSearchFieldRules.requiresFieldFiltering(sourceTable);
+         source = IndexingUtils.filterResult(source, tableConfigs.stream().filter(c -> sourceTable.equals(c.getTableName())).findFirst().orElse(null));
+         if(source == null)
+         {
+            continue;
+         }
+         String                    highlightSnippet = filterFields ? source.getSearchableText() : null;
          Map<String, List<String>> highlight        = hit.highlight();
-         if(highlight != null)
+         if(highlight != null && !filterFields)
          {
             List<String> fragments = highlight.get("searchableText");
             if(fragments != null && !fragments.isEmpty())

src/main/java/com/kingsrook/qbits/quicksearch/actions/QuickSearchRecordSearchProvider.java

@@ -32,6 +32,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
 import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
@@ -85,7 +86,7 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte
          return (false);
       }
 
-      if(runtime.getTableConfig(table.getName()) == null)
+      if(runtime.getTableConfig(table.getName()) == null || QuickSearchFieldRules.hasProtectedPrimaryKey(table.getName()))
       {
          return (false);
       }
@@ -120,7 +121,8 @@ public class QuickSearchRecordSearchProvider implements RecordSearchProviderInte
       List<String> tableNames = new ArrayList<>();
       for(String tableName : new LinkedHashSet<>(CollectionUtils.nonNullList(input.getTableNames())))
       {
-         if(runtime.getTableConfig(tableName) != null)
+         if(runtime.getTableConfig(tableName) != null
+            && !QuickSearchFieldRules.hasProtectedPrimaryKey(tableName))
          {
             tableNames.add(tableName);
          }

src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchApi.java

@@ -0,0 +1,54 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.api;
+
+
+import com.kingsrook.qbits.quicksearch.metadata.QuickSearchAdminAppMetaDataProducer;
+import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtension;
+
+
+/*******************************************************************************
+ ** Supported host navigation and UI API extension identity.
+ *******************************************************************************/
+public final class QuickSearchApi
+{
+   public static final String APP_NAME = QuickSearchAdminAppMetaDataProducer.APP_NAME;
+   public static final String HOME_HREF = "/app/" + APP_NAME;
+
+
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private QuickSearchApi()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** The spec registration shared by production and contract generation.
+    *******************************************************************************/
+   public static MiddlewareSpecExtension extension()
+   {
+      return (new MiddlewareSpecExtension().withName("quick-search").withSemanticVersion("1.0.0")
+         .withDescription("Quick Search administrative status").withSpec(QuickSearchStatusSpec.class));
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchStatus.java

@@ -0,0 +1,86 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.api;
+
+
+import java.time.Instant;
+import java.util.List;
+import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema;
+import com.kingsrook.qqq.openapi.model.Schema;
+
+
+/*******************************************************************************
+ ** Immutable administrative snapshot. Null measurements are unknown, never zero.
+ ** Counts come from persisted diagnostics; countsCheckedAt identifies their age.
+ *******************************************************************************/
+public record QuickSearchStatus(State state, Boolean clusterReachable, String clusterHealth, Integer nodes, Integer activeShards, Integer unassignedShards,
+                                Boolean indexingPaused, List<IndexRow> indexes, Integer pending, Integer exhausted, Boolean backlogTruncated,
+                                Boolean tablesTruncated, Instant lastSuccessfulSync, String lastError, Instant checkedAt, String homeHref) implements ToSchema
+{
+   /*******************************************************************************
+    ** Published wire schema for the same immutable value used by Java hosts.
+    *******************************************************************************/
+   @Override
+   public Schema toSchema()
+   {
+      return (new QuickSearchStatusSpec().defineComponentSchemas().get("QuickSearchStatusV1"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Protect the snapshot from mutations by callers.
+    *******************************************************************************/
+   public QuickSearchStatus
+   {
+      indexes = List.copyOf(indexes);
+   }
+
+
+
+   /*******************************************************************************
+    ** Overall availability; DEGRADED also covers incomplete measurements.
+    *******************************************************************************/
+   public enum State
+   {
+      OK, DEGRADED, DOWN, DISABLED
+   }
+
+
+
+   /*******************************************************************************
+    ** A persisted index row with counts from the latest saved validation, when
+    ** available. Otherwise documentCount is the last indexing run's count and
+    ** sourceCount/countsCheckedAt are unknown. Sync time is a successful watermark.
+    *******************************************************************************/
+   public record IndexRow(String tableName, Boolean enabled, String status, Long documentCount, Long sourceCount, Instant countsCheckedAt,
+                          Instant lastSuccessfulSync, String lastError, Integer realTimeErrorCount) implements ToSchema
+   {
+      /*******************************************************************************
+       ** Published per-table schema.
+       *******************************************************************************/
+      @Override
+      public Schema toSchema()
+      {
+         return (new QuickSearchStatusSpec().defineComponentSchemas().get("QuickSearchIndexStatusV1"));
+      }
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchStatusService.java

@@ -0,0 +1,70 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.api;
+
+
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.diagnostics.StatusDiagnostics;
+import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+
+
+/*******************************************************************************
+ ** Supported entry point for hosts. Uses the current QContext and checks the
+ ** Quick Search admin app permission on EVERY call, including cache hits.
+ *******************************************************************************/
+public final class QuickSearchStatusService
+{
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private QuickSearchStatusService()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Return a briefly cached snapshot, waiting at most two seconds for a refresh.
+    ** Call with an authenticated QSession (or an explicitly trusted system session).
+    *******************************************************************************/
+   public static QuickSearchStatus get() throws QException
+   {
+      if(QContext.getQSession() == null)
+      {
+         throw (new QPermissionDeniedException("A session is required for Quick Search status."));
+      }
+      QuickSearchRuntime runtime = QuickSearchRuntime.get();
+      if(runtime == null)
+      {
+         return (StatusDiagnostics.unavailable(QuickSearchStatus.State.DISABLED, null, QuickSearchApi.HOME_HREF));
+      }
+      String appName = runtime.getConfig().applyPrefix(QuickSearchApi.APP_NAME);
+      if(QContext.getQInstance().getApp(appName) == null || !PermissionsHelper.hasAppPermission(new QueryInput(), appName))
+      {
+         throw (new QPermissionDeniedException("Quick Search admin permission is required."));
+      }
+      return (runtime.getStatusCache().get(runtime));
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/api/QuickSearchStatusSpec.java

@@ -0,0 +1,182 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.api;
+
+
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation;
+import com.kingsrook.qqq.middleware.javalin.specs.BasicResponse;
+import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
+import com.kingsrook.qqq.openapi.model.HttpMethod;
+import com.kingsrook.qqq.openapi.model.Method;
+import com.kingsrook.qqq.openapi.model.Parameter;
+import com.kingsrook.qqq.openapi.model.Schema;
+import com.kingsrook.qqq.openapi.model.Type;
+import io.javalin.http.HttpStatus;
+
+
+/*******************************************************************************
+ ** Administrative status, available through the spec-managed UI API only.
+ *******************************************************************************/
+public class QuickSearchStatusSpec extends AbstractJsonEndpointSpec
+{
+   /*******************************************************************************
+    ** Describe the supported operation.
+    *******************************************************************************/
+   @Override
+   public BasicOperation defineBasicOperation()
+   {
+      return (new BasicOperation().withHttpMethod(HttpMethod.GET).withPath("/qbit/quick-search/status")
+         .withShortSummary("Get Quick Search status")
+         .withLongDescription("Requires Quick Search admin permission. Five-second instance cache; two-second refresh wait. Counts are saved diagnostics, not live source scans. Unknown values are null. Backlog counts are lower bounds when backlogTruncated is true."));
+   }
+
+
+
+   /*******************************************************************************
+    ** Stable generated-client operation name.
+    *******************************************************************************/
+   @Override
+   protected void customizeMethod(Method method)
+   {
+      method.setOperationId("getQuickSearchStatus");
+   }
+
+
+
+   /*******************************************************************************
+    ** No request parameters.
+    *******************************************************************************/
+   @Override
+   public List<Parameter> defineRequestParameters()
+   {
+      return (List.of());
+   }
+
+
+
+   /*******************************************************************************
+    ** Versioned response component.
+    *******************************************************************************/
+   @Override
+   protected String getResponseSchemaName()
+   {
+      return ("QuickSearchStatusV1");
+   }
+
+
+
+   /*******************************************************************************
+    ** Use exactly the same authorized service as in-process hosts.
+    *******************************************************************************/
+   @Override
+   protected Object handle(JsonInput input) throws Exception
+   {
+      return (QuickSearchStatusService.get());
+   }
+
+
+
+   /*******************************************************************************
+    ** Standard framework errors, including the diagnostics rate limit.
+    *******************************************************************************/
+   @Override
+   public List<BasicResponse> defineAdditionalBasicResponses()
+   {
+      return (List.of(new BasicResponse(HttpStatus.BAD_REQUEST, "Diagnostic rate limit exceeded", "QuickSearchErrorV1"),
+         new BasicResponse(HttpStatus.UNAUTHORIZED, "Sign in required", "QuickSearchErrorV1"),
+         new BasicResponse(HttpStatus.FORBIDDEN, "Quick Search admin permission required", "QuickSearchErrorV1")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Explicit schemas make nullability, bounds and measurement age discoverable.
+    *******************************************************************************/
+   @Override
+   public Map<String, Schema> defineComponentSchemas()
+   {
+      Map<String, Schema> status = new TreeMap<>();
+      status.put("state", field(Type.STRING, "Overall availability; incomplete diagnostics are DEGRADED.", false)
+         .withEnumValues(List.of("OK", "DEGRADED", "DOWN", "DISABLED")));
+      status.put("clusterReachable", field(Type.BOOLEAN, "Whether OpenSearch answered; null when not probed.", true));
+      status.put("clusterHealth", field(Type.STRING, "OpenSearch health color; null when unknown.", true));
+      for(String name : List.of("nodes", "activeShards", "unassignedShards"))
+      {
+         status.put(name, field(Type.INTEGER, "Cluster health counter; null when unavailable.", true));
+      }
+      status.put("indexingPaused", field(Type.BOOLEAN, "Indexing is paused; search can still serve.", true));
+      status.put("indexes", new Schema().withType(Type.ARRAY).withItems(new Schema().withRef("#/components/schemas/QuickSearchIndexStatusV1"))
+         .withDescription("At most 100 persisted index rows, ordered by id."));
+      status.put("pending", field(Type.INTEGER, "Pending and awaiting-reindex failed events, from at most 1000 backlog rows.", true));
+      status.put("exhausted", field(Type.INTEGER, "Exhausted failed events, from the same bounded backlog read.", true));
+      status.put("backlogTruncated", field(Type.BOOLEAN, "Counts are lower bounds when more than 1000 backlog rows exist.", false));
+      status.put("tablesTruncated", field(Type.BOOLEAN, "More than 100 index rows exist.", false));
+      status.put("lastSuccessfulSync", timestamp("Newest successful indexing watermark; not a guarantee all tables are current."));
+      status.put("lastError", field(Type.STRING, "Last known indexing/search error or snapshot failure; null when absent.", true));
+      status.put("checkedAt", timestamp("Snapshot completion time; remains unchanged on cache hits.").withNullable(false));
+      status.put("homeHref", field(Type.STRING, "Quick Search home link, including the configured metadata prefix.", false));
+      Map<String, Schema> row = new TreeMap<>();
+      row.put("tableName", field(Type.STRING, "Indexed source table metadata name.", false));
+      row.put("enabled", field(Type.BOOLEAN, "Whether this index is enabled.", false));
+      row.put("status", field(Type.STRING, "Persisted indexing state, such as ACTIVE or NEEDS_REINDEX.", true));
+      row.put("documentCount", field(Type.INTEGER, "Saved validation document count, else last indexing run count; not live.", true).withFormat("int64"));
+      row.put("sourceCount", field(Type.INTEGER, "Source count from saved IndexValidator report; null without validation.", true).withFormat("int64"));
+      row.put("countsCheckedAt", timestamp("Saved validation completion; null when validation counts are unavailable."));
+      row.put("lastSuccessfulSync", timestamp("Newest successful basepull/full-reindex/reconcile watermark."));
+      row.put("lastError", field(Type.STRING, "Persisted indexing error; null when absent.", true));
+      row.put("realTimeErrorCount", field(Type.INTEGER, "Persisted real-time indexing error count.", true));
+      return (Map.of("QuickSearchStatusV1", object(status), "QuickSearchIndexStatusV1", object(row),
+         "QuickSearchErrorV1", object(Map.of("error", field(Type.STRING, "Framework error message.", false)))));
+   }
+
+
+
+   /*******************************************************************************
+    ** A response object always includes its properties, including nulls.
+    *******************************************************************************/
+   private static Schema object(Map<String, Schema> properties)
+   {
+      return (new Schema().withType(Type.OBJECT).withProperties(new TreeMap<>(properties)));
+   }
+
+
+
+   /*******************************************************************************
+    ** A scalar with explicit unknown semantics.
+    *******************************************************************************/
+   private static Schema field(Type type, String description, Boolean nullable)
+   {
+      return (new Schema().withType(type).withDescription(description).withNullable(nullable));
+   }
+
+
+
+   /*******************************************************************************
+    ** ISO-8601 instant.
+    *******************************************************************************/
+   private static Schema timestamp(String description)
+   {
+      return (field(Type.STRING, description, true).withFormat("date-time"));
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigPrimer.java

@@ -238,8 +238,7 @@ public class QuickSearchConfigPrimer
             }
             if(!QuickSearchFieldRules.isIndexable(table, fieldName))
             {
-               LOG.warn("A field declared searchable in code cannot be indexed; it is not primed: " + QuickSearchFieldRules.whyNotIndexable(table, fieldName),
-                  logPair("tableName", tableName), logPair("fieldName", fieldName));
+               QuickSearchFieldRules.warnExcludedFields(table);
                continue;
             }
             codeFields.add(codeFieldRow(code, fieldName, i));

src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigStore.java

@@ -377,7 +377,7 @@ public class QuickSearchConfigStore
          String problem   = QuickSearchFieldRules.whyNotIndexable(table, fieldName);
          if(problem != null)
          {
-            LOG.warn("Quick Search field row skipped: " + problem, logPair("tableName", tableName), logPair("fieldName", fieldName));
+            QuickSearchFieldRules.warnExcludedFields(table);
             continue;
          }
          if(fields.contains(fieldName))

src/main/java/com/kingsrook/qbits/quicksearch/config/QuickSearchFieldRules.java

@@ -23,9 +23,14 @@ import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Locale;
 import java.util.Set;
+import java.util.WeakHashMap;
 import java.util.regex.Pattern;
+import com.kingsrook.qqq.backend.core.actions.ai.AiDataClassifications;
+import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
 import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -48,11 +53,14 @@ import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole;
  ** on backend variants, not produced by a QBit (qbit-internal), not a QQQ
  ** system table (audits, saved views and reports, scheduled jobs, scripts),
  ** not named like a secret store, and has at least one indexable field. A
- ** field is indexable when it is not hidden, not isSensitive, not a password
- ** and not a blob or boolean.
+ ** field is indexable when it is not PII/PHI-classified, locked, hidden,
+ ** sensitive, masked, a blob or a boolean.
  *******************************************************************************/
 public final class QuickSearchFieldRules
 {
+   private static final QLogger LOG = QLogger.getLogger(QuickSearchFieldRules.class);
+   private static final WeakHashMap<QTableMetaData, Boolean> WARNED_TABLES = new WeakHashMap<>();
+
    public static final int MAX_DEFAULT_FIELDS = 20;
    public static final int TITLE_WEIGHT       = 3;
 
@@ -85,6 +93,9 @@ public final class QuickSearchFieldRules
 
    /*******************************************************************************
     ** Why a field cannot be indexed, or null when it can.
+    ** When the qbit moves to QQQ 4.1.0-RC.7, also honour core
+    ** ClassifiedValues.isClassified(table, field) (the host's AuditValueProtector
+    ** classification, qqq#1182) and FieldAccessHelper.getLockTier (qqq#1172).
     *******************************************************************************/
    public static String whyNotIndexable(QTableMetaData table, String fieldName)
    {
@@ -101,6 +112,15 @@ public final class QuickSearchFieldRules
       {
          return ("field [" + fieldName + "] is hidden");
       }
+      DataClassification classification = AiDataClassifications.ofField(table, fieldName);
+      if(classification == DataClassification.PII || classification == DataClassification.PHI)
+      {
+         return ("field [" + fieldName + "] is classified " + classification + " and cannot be indexed");
+      }
+      if(field.getFieldSecurityLock() != null)
+      {
+         return ("field [" + fieldName + "] has a field security lock and cannot be indexed");
+      }
       if(field.getIsSensitive() || (field.getType() != null && field.getType().needsMasked()))
       {
          return ("field [" + fieldName + "] is sensitive (its values must not reach the search index)");
@@ -124,6 +144,96 @@ public final class QuickSearchFieldRules
 
 
 
+   /*******************************************************************************
+    ** Whether a field contains protected values, independently of searchability.
+    ** Classification falls back to the table. When the qbit moves to QQQ
+    ** 4.1.0-RC.7, also honour ClassifiedValues.isClassified(table, field)
+    ** (qqq#1182) and FieldAccessHelper.getLockTier (qqq#1172).
+    *******************************************************************************/
+   public static boolean isProtected(QTableMetaData table, String fieldName)
+   {
+      QFieldMetaData field = table == null || fieldName == null ? null : table.getFields().get(fieldName);
+      if(field == null)
+      {
+         return (false);
+      }
+      DataClassification classification = AiDataClassifications.ofField(table, fieldName);
+      return (classification == DataClassification.PII || classification == DataClassification.PHI
+         || field.getFieldSecurityLock() != null || field.getIsSensitive()
+         || (field.getType() != null && field.getType().needsMasked()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Warn once per table metadata lifetime; never include record values.
+    *******************************************************************************/
+   public static synchronized void warnExcludedFields(QTableMetaData table)
+   {
+      if(table != null && WARNED_TABLES.put(table, Boolean.TRUE) == null)
+      {
+         LOG.warn("Quick Search dropped fields that cannot be indexed (classified PII/PHI, secured, hidden, sensitive or unsupported fields) from table [" + table.getName() + "]");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Filter explicit field lists as well as cached configurations at use time.
+    *******************************************************************************/
+   public static List<String> indexableFields(QTableMetaData table, List<String> fields)
+   {
+      List<String> result = new ArrayList<>();
+      for(String field : CollectionUtils.nonNullList(fields))
+      {
+         if(isIndexable(table, field))
+         {
+            result.add(field);
+         }
+         else
+         {
+            warnExcludedFields(table);
+         }
+      }
+      return (result);
+   }
+
+
+
+   /*******************************************************************************
+    ** Resolve a table's metadata for backend query and result filtering.
+    *******************************************************************************/
+   public static QTableMetaData lookupTable(String tableName)
+   {
+      QInstance instance = QContext.getQInstance();
+      return (instance == null || tableName == null ? null : instance.getTable(tableName));
+   }
+
+
+
+   /*******************************************************************************
+    ** Combined text and labels from an older index cannot be attributed to
+    ** individual fields. Use only allowed field values for these tables.
+    *******************************************************************************/
+   public static boolean requiresFieldFiltering(String tableName)
+   {
+      QTableMetaData table = lookupTable(tableName);
+      return (table != null && table.getFields().keySet().stream().anyMatch(f -> isProtected(table, f)));
+   }
+
+
+
+   /*******************************************************************************
+    ** A protected primary key cannot be sent as an index id, even for deletes.
+    *******************************************************************************/
+   public static boolean hasProtectedPrimaryKey(String tableName)
+   {
+      QTableMetaData table = lookupTable(tableName);
+      return (table != null && isProtected(table, table.getPrimaryKeyField()));
+   }
+
+
+
    /*******************************************************************************
     ** The indexable fields of a table, in the table's order.
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/quicksearch/diagnostics/DiagnosticsSupport.java

@@ -18,13 +18,20 @@ package com.kingsrook.qbits.quicksearch.diagnostics;
 
 
 import java.util.Collection;
+import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.exceptions.QUserFacingException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
-import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
-import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -45,6 +52,26 @@ public final class DiagnosticsSupport
 
 
 
+   /*******************************************************************************
+    ** Latest saved validation, shared by the home widget and host status API.
+    ** This does not run validation or scan source tables.
+    *******************************************************************************/
+   public static QRecord latestValidation(QuickSearchQBitConfig config) throws QException
+   {
+      String tableName = config.applyTablePrefix(QuickSearchValidationRun.TABLE_NAME);
+      if(QContext.getQInstance().getTable(tableName) == null)
+      {
+         return (null);
+      }
+      var input = new QueryInput(tableName)
+         .withFilter(new QQueryFilter()
+            .withOrderBy(new QFilterOrderBy("id", false)).withLimit(1));
+      var rows = new QueryAction().execute(input).getRecords();
+      return (rows == null || rows.isEmpty() ? null : rows.get(0));
+   }
+
+
+
    /*******************************************************************************
     ** Utility class - no instances.
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/quicksearch/diagnostics/IndexValidator.java

@@ -28,6 +28,18 @@ import java.util.Set;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsHtml.Level;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
+import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchDiagnosticsOperations;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchDiagnosticsOperations.RawResponse;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
+import com.kingsrook.qbits.quicksearch.processes.AbstractIndexingStep;
+import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
 import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
@@ -46,18 +58,6 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
-import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
-import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
-import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
-import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsHtml.Level;
-import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
-import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
-import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
-import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchDiagnosticsOperations;
-import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchDiagnosticsOperations.RawResponse;
-import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchOpenSearchClient;
-import com.kingsrook.qbits.quicksearch.processes.AbstractIndexingStep;
-import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -293,6 +293,16 @@ public class IndexValidator
 
 
 
+   /*******************************************************************************
+    ** Shared cluster health interpretation for validation and the host status API.
+    *******************************************************************************/
+   public static Level healthLevel(String status)
+   {
+      return (healthFinding("Health", status).level());
+   }
+
+
+
    /*******************************************************************************
     ** The live mapping against the mapping this version creates.
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/quicksearch/diagnostics/QuickSearchDiagnosticsWidgetRenderer.java

@@ -19,6 +19,12 @@ package com.kingsrook.qbits.quicksearch.diagnostics;
 
 import java.util.ArrayList;
 import java.util.List;
+import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchStatus;
+import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsHtml.Level;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
 import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
@@ -34,12 +40,6 @@ import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
 import com.kingsrook.qqq.backend.core.model.dashboard.widgets.RawHTML;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
-import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
-import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
-import com.kingsrook.qbits.quicksearch.config.QuickSearchStatus;
-import com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsHtml.Level;
-import com.kingsrook.qbits.quicksearch.model.QuickSearchDiagnosticEvent;
-import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
 
 
 /*******************************************************************************
@@ -211,18 +211,14 @@ public class QuickSearchDiagnosticsWidgetRenderer extends AbstractWidgetRenderer
 
       try
       {
-         QueryInput queryInput = new QueryInput();
-         queryInput.setTableName(tableName);
-         queryInput.setFilter(new QQueryFilter().withOrderBy(new QFilterOrderBy("id", false)).withLimit(1));
-         List<QRecord> rows = CollectionUtils.nonNullList(new QueryAction().execute(queryInput).getRecords());
-         if(rows.isEmpty())
+         QRecord row = DiagnosticsSupport.latestValidation(config);
+         if(row == null)
          {
             html.paragraph("None yet. Run Validate Search Index from Quick Search Index (Actions)"
                + (config.getValidationCronExpression() == null ? "." : "; it is also scheduled (" + config.getValidationCronExpression() + ")."));
             return;
          }
 
-         QRecord row    = rows.get(0);
          String  status = row.getValueString("status");
          Level   level  = QuickSearchValidationRun.STATUS_PASS.equals(status) ? Level.OK : (QuickSearchValidationRun.STATUS_WARN.equals(status) ? Level.WARN : Level.FAIL);
          html.raw("<p>" + DiagnosticsHtml.badge(level) + " <a href=\"/app/" + DiagnosticsHtml.escape(tableName) + "/" + DiagnosticsHtml.escape(row.getValueString("id")) + "\">"

src/main/java/com/kingsrook/qbits/quicksearch/diagnostics/StatusCache.java

@@ -0,0 +1,125 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.diagnostics;
+
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchApi;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+
+
+/*******************************************************************************
+ ** Instance-owned five-second cache with one in-flight refresh. A stuck backend
+ ** cannot grow a queue or create more workers. Runtime shutdown stops the worker.
+ *******************************************************************************/
+public class StatusCache implements AutoCloseable
+{
+   private final ExecutorService executor = Executors.newSingleThreadExecutor(runnable ->
+   {
+      Thread thread = new Thread(runnable, "quick-search-status");
+      thread.setDaemon(true);
+      return (thread);
+   });
+   private Future<QuickSearchStatus> refresh;
+   private QuickSearchStatus cached;
+   private long expiresAt;
+   private long refreshStartedAt;
+
+
+   /*******************************************************************************
+    ** Authorization belongs to the public service, before reaching this cache.
+    ** Only refreshes consume the existing per-user diagnostic rate limit.
+    *******************************************************************************/
+   public synchronized QuickSearchStatus get(QuickSearchRuntime runtime) throws QException
+   {
+      if(cached != null && System.nanoTime() < expiresAt)
+      {
+         return (cached);
+      }
+      String homeHref = "/app/" + runtime.getConfig().applyPrefix(QuickSearchApi.APP_NAME);
+      // A refresh that finished while nobody was polling must not resurrect an old snapshot.
+      if(refresh != null && refresh.isDone() && System.nanoTime() - refreshStartedAt >= TimeUnit.SECONDS.toNanos(5))
+      {
+         refresh = null;
+      }
+      if(refresh == null)
+      {
+         DiagnosticsSupport.checkRateLimit(runtime, "STATUS");
+         QInstance instance = QContext.getQInstance();
+         refreshStartedAt = System.nanoTime();
+         refresh = executor.submit(() ->
+         {
+            QContext.init(instance, new QSystemUserSession(instance));
+            try
+            {
+               return (StatusDiagnostics.read(runtime));
+            }
+            finally
+            {
+               QContext.clear();
+            }
+         });
+      }
+      try
+      {
+         cached = refresh.get(2, TimeUnit.SECONDS);
+         refresh = null;
+      }
+      catch(TimeoutException e)
+      {
+         // Leave the one worker in flight; never pile up abandoned requests.
+         cached = StatusDiagnostics.unavailable(QuickSearchStatus.State.DEGRADED, "Status refresh timed out; measurements are unknown.", homeHref);
+      }
+      catch(InterruptedException e)
+      {
+         Thread.currentThread().interrupt();
+         throw (new QException("Status refresh interrupted", e));
+      }
+      catch(ExecutionException e)
+      {
+         refresh = null;
+         cached = StatusDiagnostics.unavailable(QuickSearchStatus.State.DEGRADED, "Status diagnostics could not be read.", homeHref);
+      }
+      expiresAt = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
+      return (cached);
+   }
+
+
+
+   /*******************************************************************************
+    ** Stop the bounded worker when its runtime closes.
+    *******************************************************************************/
+   @Override
+   public void close()
+   {
+      executor.shutdownNow();
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/diagnostics/StatusDiagnostics.java

@@ -0,0 +1,213 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch.diagnostics;
+
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchApi;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus.IndexRow;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus.State;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
+import com.kingsrook.qbits.quicksearch.opensearch.QuickSearchDiagnosticsOperations;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
+
+
+/*******************************************************************************
+ ** Cheap projection of existing diagnostics: one health request, bounded metadata
+ ** reads, saved IndexValidator counts. No source scans or per-table remote calls.
+ *******************************************************************************/
+public final class StatusDiagnostics
+{
+   public static final int MAX_TABLES = 100;
+   public static final int MAX_BACKLOG = 1000;
+   private static final ObjectMapper MAPPER = new ObjectMapper();
+
+
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private StatusDiagnostics()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Read as the trusted diagnostics worker after the caller's admin check.
+    *******************************************************************************/
+   public static QuickSearchStatus read(QuickSearchRuntime runtime) throws Exception
+   {
+      String homeHref = "/app/" + runtime.getConfig().applyPrefix(QuickSearchApi.APP_NAME);
+      List<QRecord> rows = query(runtime.getConfig().getQuickSearchIndexTableName(), new QQueryFilter()
+         .withOrderBy(new QFilterOrderBy("id")).withLimit(MAX_TABLES + 1));
+      boolean truncated = rows.size() > MAX_TABLES;
+      QRecord validation = DiagnosticsSupport.latestValidation(runtime.getConfig());
+      String json = validation == null ? null : validation.getValueString("reportJson");
+      JsonNode report = json == null || json.length() > 1_000_000 ? MAPPER.createObjectNode() : MAPPER.readTree(json);
+      Instant countsAt = validation == null ? null : validation.getValueInstant("endTime");
+      List<IndexRow> indexes = new ArrayList<>();
+      Instant lastSync = null;
+      String lastError = runtime.getLastRecordSearchFailureMessage();
+      Instant lastErrorAt = runtime.getLastRecordSearchFailureTime();
+      boolean degraded = truncated || runtime.isRecordSearchBackingOff() || (rows.isEmpty() && !runtime.getCodeTables().isEmpty());
+      for(QRecord row : rows.subList(0, Math.min(rows.size(), MAX_TABLES)))
+      {
+         String tableName = row.getValueString("tableName");
+         Long documents = row.getValueLong("documentCount");
+         Long source = null;
+         Instant measuredAt = null;
+         for(JsonNode stats : report.path("tableStats"))
+         {
+            if(tableName.equals(stats.path("tableName").asText()))
+            {
+               documents = stats.path("indexCount").isNumber() ? stats.path("indexCount").longValue() : null;
+               source = stats.path("databaseCount").isNumber() ? stats.path("databaseCount").longValue() : null;
+               measuredAt = countsAt;
+               break;
+            }
+         }
+         Instant sync = latest(row.getValueInstant("lastBasepullTime"), latest(row.getValueInstant("lastFullReindexTime"), row.getValueInstant("lastReconcileTime")));
+         lastSync = latest(lastSync, sync);
+         String error = row.getValueString("lastErrorMessage");
+         if(error != null && !error.isBlank() && (lastErrorAt == null || (row.getValueInstant("modifyDate") != null && row.getValueInstant("modifyDate").isAfter(lastErrorAt))))
+         {
+            lastError = error;
+            lastErrorAt = row.getValueInstant("modifyDate");
+         }
+         boolean enabled = !Boolean.FALSE.equals(row.getValueBoolean("enabled"));
+         Integer realTimeErrors = row.getValueInteger("realTimeErrorCount");
+         degraded |= enabled && (!"ACTIVE".equals(row.getValueString("status")) || "FAILED".equals(row.getValueString("lastRunStatus"))
+            || (realTimeErrors != null && realTimeErrors > 0) || (source != null && documents != null && !source.equals(documents)));
+         indexes.add(new IndexRow(tableName, enabled, row.getValueString("status"), documents, source, measuredAt, sync, error, realTimeErrors));
+      }
+      List<QRecord> backlog = query(runtime.getConfig().applyTablePrefix(QuickSearchFailedEvent.TABLE_NAME),
+         new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN, List.of("PENDING", "EXHAUSTED", "AWAITING_REINDEX")))
+            .withOrderBy(new QFilterOrderBy("id")).withLimit(MAX_BACKLOG + 1));
+      int pending = 0;
+      int exhausted = 0;
+      for(QRecord row : backlog.subList(0, Math.min(backlog.size(), MAX_BACKLOG)))
+      {
+         if("EXHAUSTED".equals(row.getValueString("status")))
+         {
+            exhausted++;
+         }
+         else
+         {
+            pending++;
+         }
+      }
+      boolean paused = runtime.isIndexingPaused();
+      degraded |= paused || pending > 0 || exhausted > 0;
+      Boolean reachable = null;
+      String health = null;
+      Integer nodes = null;
+      Integer active = null;
+      Integer unassigned = null;
+      State state = degraded ? State.DEGRADED : State.OK;
+      if(!truncated && !indexes.isEmpty() && indexes.stream().noneMatch(row -> Boolean.TRUE.equals(row.enabled())))
+      {
+         state = State.DISABLED;
+      }
+      else
+      {
+         try
+         {
+            var response = new QuickSearchDiagnosticsOperations(runtime.getClient()).clusterHealth();
+            reachable = true;
+            QuickSearchDiagnosticsOperations.requireOk(response, "Cluster health unavailable");
+            JsonNode body = response.body();
+            health = body.path("status").asText("unknown");
+            nodes = number(body, "number_of_nodes");
+            active = number(body, "active_shards");
+            unassigned = number(body, "unassigned_shards");
+            if(IndexValidator.healthLevel(health) != DiagnosticsHtml.Level.OK || body.path("timed_out").asBoolean())
+            {
+               state = "red".equals(health) ? State.DOWN : State.DEGRADED;
+            }
+         }
+         catch(QException e)
+         {
+            if(reachable == null)
+            {
+               reachable = false;
+            }
+            state = State.DOWN;
+            lastError = "OpenSearch health is unavailable; see Quick Search diagnostics.";
+         }
+      }
+      return (new QuickSearchStatus(state, reachable, health, nodes, active, unassigned, paused, indexes, pending, exhausted, backlog.size() > MAX_BACKLOG,
+         truncated, lastSync, lastError, Instant.now(), homeHref));
+   }
+
+
+
+   /*******************************************************************************
+    ** Unknown measurements for disabled or failed snapshots.
+    *******************************************************************************/
+   public static QuickSearchStatus unavailable(State state, String error, String homeHref)
+   {
+      return (new QuickSearchStatus(state, null, null, null, null, null, null, List.of(), null, null, false, false, null, error, Instant.now(), homeHref));
+   }
+
+
+
+   /*******************************************************************************
+    ** Bounded metadata queries; failures propagate rather than looking healthy.
+    *******************************************************************************/
+   private static List<QRecord> query(String tableName, QQueryFilter filter) throws QException
+   {
+      return (CollectionUtils.nonNullList(new QueryAction().execute(new QueryInput(tableName).withFilter(filter)).getRecords()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Preserve unknown counters from incomplete cluster responses.
+    *******************************************************************************/
+   private static Integer number(JsonNode body, String field)
+   {
+      return (body.path(field).isNumber() ? body.path(field).intValue() : null);
+   }
+
+
+
+   /*******************************************************************************
+    ** Newest non-null timestamp.
+    *******************************************************************************/
+   private static Instant latest(Instant first, Instant second)
+   {
+      return (first == null ? second : second == null || first.isAfter(second) ? first : second);
+   }
+}

src/main/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListener.java

@@ -19,7 +19,9 @@ package com.kingsrook.qbits.quicksearch.listeners;
 
 import java.io.Serializable;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
@@ -39,6 +41,7 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigWrites;
 import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
 import com.kingsrook.qbits.quicksearch.processes.AbstractIndexingStep;
@@ -117,6 +120,11 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
       }
 
       String primaryKeyField = tableConfig.getPrimaryKeyField() != null ? tableConfig.getPrimaryKeyField() : "id";
+      if(QuickSearchFieldRules.lookupTable(tableName) == null || QuickSearchFieldRules.isProtected(QuickSearchFieldRules.lookupTable(tableName), primaryKeyField))
+      {
+         QuickSearchFieldRules.warnExcludedFields(QuickSearchFieldRules.lookupTable(tableName));
+         return;
+      }
 
       ///////////////////////////////////////////////////////////////////////////
       // paused: index nothing, but keep the written keys (DEFERRED), which     //
@@ -283,6 +291,7 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
          }
 
          insertFailedEventRows(runtime, events, message, QuickSearchFailedEvent.STATUS_PENDING);
+         incrementRealTimeErrorCounts(runtime, events);
       }
       catch(Exception e)
       {
@@ -292,6 +301,54 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
 
 
 
+   /*******************************************************************************
+    ** QQQ UpdateInput has no arithmetic update. Serialize this read-modify-write
+    ** across listeners in this JVM, after the source transaction has committed.
+    ** Counter failures must not prevent replay or fail the source write.
+    *******************************************************************************/
+   private static synchronized void incrementRealTimeErrorCounts(QuickSearchRuntime runtime, List<IndexEvent> events)
+   {
+      Map<String, Integer> counts = new HashMap<>();
+      for(IndexEvent event : events)
+      {
+         counts.merge(event.getTableName(), 1, Integer::sum);
+      }
+
+      for(Map.Entry<String, Integer> entry : counts.entrySet())
+      {
+         try
+         {
+            String     indexTable = runtime.getConfig().getQuickSearchIndexTableName();
+            QueryInput queryInput = new QueryInput();
+            queryInput.setTableName(indexTable);
+            queryInput.setFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, entry.getKey())).withLimit(1));
+            List<QRecord> rows = new QueryAction().execute(queryInput).getRecords();
+            if(CollectionUtils.nullSafeIsEmpty(rows))
+            {
+               continue;
+            }
+
+            QRecord row      = rows.get(0);
+            Integer previous = row.getValueInteger("realTimeErrorCount");
+            UpdateInput updateInput = new UpdateInput();
+            updateInput.setTableName(indexTable);
+            updateInput.setRecords(List.of(new QRecord().withValue("id", row.getValue("id"))
+               .withValue("realTimeErrorCount", Math.addExact(previous == null ? 0 : previous, entry.getValue()))));
+            List<QRecord> updated = QuickSearchConfigWrites.asSystem(() -> new UpdateAction().execute(updateInput)).getRecords();
+            if(CollectionUtils.nullSafeIsEmpty(updated) || CollectionUtils.nullSafeHasContents(updated.get(0).getErrors()))
+            {
+               throw (new QException("Counter update returned errors"));
+            }
+         }
+         catch(Exception | LinkageError e)
+         {
+            LOG.warn("Could not increment real-time indexing error count", e, logPair("tableName", entry.getKey()));
+         }
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Keep changes made while indexing is paused, for the basepull to replay
     ** when it resumes. Never throws.
@@ -462,7 +519,17 @@ public class QuickSearchRecordChangeListener implements RecordChangeListenerInte
          {
             continue;
          }
-         events.add(new IndexEvent().withAction(IndexEventAction.INDEX).withTableName(tableName).withRecordId(recordId).withRecord(record));
+         QRecord safeRecord = new QRecord();
+         for(var field : QuickSearchFieldRules.indexableFields(QuickSearchFieldRules.lookupTable(tableName)))
+         {
+            safeRecord.setValue(field.getName(), record.getValue(field.getName()));
+            safeRecord.setDisplayValue(field.getName(), record.getDisplayValue(field.getName()));
+         }
+         if(!QuickSearchFieldRules.requiresFieldFiltering(tableName))
+         {
+            safeRecord.setRecordLabel(record.getRecordLabel());
+         }
+         events.add(new IndexEvent().withAction(IndexEventAction.INDEX).withTableName(tableName).withRecordId(recordId).withRecord(safeRecord));
       }
       return (events);
    }

src/main/java/com/kingsrook/qbits/quicksearch/metadata/QuickSearchTableMetaDataHelper.java

@@ -26,6 +26,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.Association;
@@ -99,7 +100,8 @@ public class QuickSearchTableMetaDataHelper
          .withField(new QFieldMetaData("lastErrorMessage", QFieldType.TEXT).withIsEditable(false))
          .withField(new QFieldMetaData("recordCount", QFieldType.INTEGER).withIsEditable(false))
          .withField(new QFieldMetaData("documentCount", QFieldType.INTEGER).withIsEditable(false))
-         .withField(new QFieldMetaData("realTimeErrorCount", QFieldType.INTEGER).withIsEditable(false))
+         .withField(new QFieldMetaData("realTimeErrorCount", QFieldType.INTEGER).withIsEditable(false)
+            .withHelpContent(new QHelpContent("Real-time indexing failures since the last full reindex or basepull")))
          .withField(new QFieldMetaData("status", QFieldType.STRING).withIsEditable(false))
          .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withIsEditable(false))
          .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))

src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchDiagnosticsOperations.java

@@ -33,6 +33,8 @@ import com.fasterxml.jackson.databind.node.JsonNodeFactory;
 import com.fasterxml.jackson.databind.node.ObjectNode;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
+import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
 import jakarta.json.stream.JsonGenerator;
 import org.opensearch.client.json.JsonpMapper;
 import org.opensearch.client.json.JsonpSerializable;
@@ -42,6 +44,7 @@ import org.opensearch.client.opensearch._types.query_dsl.Query;
 import org.opensearch.client.opensearch.core.ExplainResponse;
 import org.opensearch.client.opensearch.core.SearchRequest;
 import org.opensearch.client.opensearch.core.SearchResponse;
+import org.opensearch.client.opensearch.core.search.Highlight;
 import org.opensearch.client.opensearch.core.search.Hit;
 import org.opensearch.client.opensearch.core.search.HighlightField;
 import org.opensearch.client.opensearch.core.search.MatchedQueries;
@@ -169,13 +172,14 @@ public class QuickSearchDiagnosticsOperations
    {
       int           boundedSize = Math.max(1, Math.min(size, MAX_PROBE_SIZE));
       Query         query       = quickSearchClient.buildSearchQuery(term, List.of(tableName), tableConfigs, true);
+      Highlight highlight = QuickSearchFieldRules.requiresFieldFiltering(tableName) ? null : Highlight.of(h -> h
+         .requireFieldMatch(true)
+         .fields("searchableText", HighlightField.of(hf -> hf))
+         .fields("fieldValues.*", HighlightField.of(hf -> hf)));
       SearchRequest request     = SearchRequest.of(r -> r
          .index(indexName)
          .query(query)
-         .highlight(h -> h
-            .requireFieldMatch(true)
-            .fields("searchableText", HighlightField.of(hf -> hf))
-            .fields("fieldValues.*", HighlightField.of(hf -> hf)))
+         .highlight(highlight)
          .trackTotalHits(t -> t.count(QuickSearchOpenSearchClient.MAX_RESULT_WINDOW))
          .size(boundedSize));
 
@@ -189,17 +193,26 @@ public class QuickSearchDiagnosticsOperations
          for(Hit<OpenSearchDocument> hit : response.hits().hits())
          {
             OpenSearchDocument source = hit.source();
+            boolean filterFields = QuickSearchFieldRules.requiresFieldFiltering(tableName);
+            if(source != null)
+            {
+               source = IndexingUtils.filterResult(source, tableConfigs.stream().filter(c -> tableName.equals(c.getTableName())).findFirst().orElse(null));
+            }
+            if(filterFields && source == null)
+            {
+               continue;
+            }
             hits.add(new ProbeHit(
                source == null ? idFromDocumentId(hit.id()) : source.getRecordId(),
                source == null ? null : source.getRecordLabel(),
                hit.score(),
                matchedClauses(hit.matchedQueries()),
-               hit.highlight() == null ? Map.of() : new LinkedHashMap<>(hit.highlight())));
+               filterFields || hit.highlight() == null ? Map.of() : new LinkedHashMap<>(hit.highlight())));
          }
 
          Long    total      = response.hits().total() == null ? (long) hits.size() : response.hits().total().value();
          Boolean lowerBound = response.hits().total() != null && response.hits().total().relation() == TotalHitsRelation.Gte;
-         return (new SearchProbe(tableName, toJson(request), toJson(response), response.took(), wall, total, lowerBound, hits));
+         return (new SearchProbe(tableName, toJson(request), QuickSearchFieldRules.requiresFieldFiltering(tableName) ? "{}" : toJson(response), response.took(), wall, total, lowerBound, hits));
       }
       catch(OpenSearchException | IOException e)
       {
@@ -308,7 +321,7 @@ public class QuickSearchDiagnosticsOperations
       requireOk(response, "Document read failed");
       JsonNode body = response.body();
       return (new DocumentView(body.path("found").asBoolean(false), body.has("_version") ? body.get("_version").asLong() : null,
-         body.path("_index").asText(null), body.get("_source")));
+         body.path("_index").asText(null), filterDocumentSource(tableName, body.get("_source"))));
    }
 
 
@@ -338,7 +351,7 @@ public class QuickSearchDiagnosticsOperations
       {
          if(doc.path("found").asBoolean(false) && doc.has("_source"))
          {
-            sources.put(idFromDocumentId(doc.path("_id").asText()), doc.get("_source"));
+            sources.put(idFromDocumentId(doc.path("_id").asText()), filterDocumentSource(tableName, doc.get("_source")));
          }
       }
       return (sources);
@@ -346,6 +359,39 @@ public class QuickSearchDiagnosticsOperations
 
 
 
+   /*******************************************************************************
+    ** Old documents may retain protected values. Diagnostic reads expose only
+    ** attributable, indexable fields, never the old combined presentation.
+    *******************************************************************************/
+   private static JsonNode filterDocumentSource(String tableName, JsonNode source)
+   {
+      if(source == null || !QuickSearchFieldRules.requiresFieldFiltering(tableName))
+      {
+         return (source);
+      }
+      ObjectNode filtered = JsonNodeFactory.instance.objectNode();
+      filtered.put("sourceTable", tableName);
+      var table = QuickSearchFieldRules.lookupTable(tableName);
+      if(!QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField()))
+      {
+         filtered.set("recordId", source.get("recordId"));
+      }
+      for(String container : List.of("fieldValues", "fieldValuesWord", "fieldValuesEnglish", "fieldValuesExact"))
+      {
+         ObjectNode values = filtered.putObject(container);
+         source.path(container).fields().forEachRemaining(entry ->
+         {
+            if(QuickSearchFieldRules.isIndexable(table, entry.getKey()))
+            {
+               values.set(entry.getKey(), entry.getValue());
+            }
+         });
+      }
+      return (filtered);
+   }
+
+
+
    /*******************************************************************************
     ** Run _analyze on the index with a named analyzer (an index analyzer or a
     ** built-in one). Text is truncated to MAX_ANALYZE_TEXT_LENGTH.

src/main/java/com/kingsrook/qbits/quicksearch/opensearch/QuickSearchOpenSearchClient.java

@@ -36,6 +36,7 @@ import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldAnalyzer;
 import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
 import org.opensearch.client.json.JsonData;
@@ -53,6 +54,7 @@ import org.opensearch.client.opensearch._types.query_dsl.MultiMatchQuery;
 import org.opensearch.client.opensearch._types.query_dsl.Operator;
 import org.opensearch.client.opensearch._types.query_dsl.Query;
 import org.opensearch.client.opensearch._types.query_dsl.TermQuery;
+import org.opensearch.client.opensearch._types.query_dsl.TextQueryType;
 import org.opensearch.client.opensearch.core.BulkRequest;
 import org.opensearch.client.opensearch.core.BulkResponse;
 import org.opensearch.client.opensearch.core.DeleteByQueryRequest;
@@ -796,7 +798,8 @@ public class QuickSearchOpenSearchClient implements Closeable
       try
       {
          Query     finalQuery = buildSearchQuery(searchTerm, allowedTables, tableConfigs);
-         Highlight highlight  = Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf)));
+         Highlight highlight  = allowedTables.stream().anyMatch(QuickSearchFieldRules::requiresFieldFiltering) ? null
+            : Highlight.of(h -> h.fields("searchableText", HighlightField.of(hf -> hf)));
 
          SearchRequest request = SearchRequest.of(r -> r
             .index(indexName)
@@ -941,9 +944,22 @@ public class QuickSearchOpenSearchClient implements Closeable
       // fields kept out of the snippet (KEYWORD, EXACT, or includeInSnippet   //
       // off) match on their own; with none, the query is exactly as before    //
       ////////////////////////////////////////////////////////////////////////////
+      List<String> aggregateTables = allowedTables.stream().filter(t -> !QuickSearchFieldRules.requiresFieldFiltering(t)).toList();
+      if(aggregateTables.isEmpty())
+      {
+         snippetMatch = Query.of(q -> q.matchNone(m -> m));
+      }
+      else if(aggregateTables.size() != allowedTables.size())
+      {
+         Query aggregateMatch = snippetMatch;
+         snippetMatch = Query.of(q -> q.bool(b -> b.must(aggregateMatch).filter(f -> f.terms(t -> t.field("sourceTable")
+            .terms(v -> v.value(aggregateTables.stream().map(FieldValue::of).toList()))))));
+      }
       List<Query> ownMatches = buildOwnFieldMatchQueries(searchTerm, tableConfigs, allowedTables);
-      Query       match      = ownMatches.isEmpty() ? snippetMatch
-         : Query.of(q -> q.bool(b -> b.should(snippetMatch).should(ownMatches).minimumShouldMatch("1")));
+      ownMatches.addAll(buildFilteredSnippetQueries(searchTerm, tableConfigs, allowedTables));
+      Query safeSnippetMatch = snippetMatch;
+      Query       match      = ownMatches.isEmpty() ? safeSnippetMatch
+         : Query.of(q -> q.bool(b -> b.should(safeSnippetMatch).should(ownMatches).minimumShouldMatch("1")));
 
       BoolQuery.Builder boolBuilder = new BoolQuery.Builder()
          .must(match)
@@ -955,6 +971,36 @@ public class QuickSearchOpenSearchClient implements Closeable
 
 
 
+   /*******************************************************************************
+    ** For tables with excluded fields, match only attributable field values.
+    ** Old combined text may still contain fields that became classified later.
+    *******************************************************************************/
+   private List<Query> buildFilteredSnippetQueries(String term, List<QuickSearchableTableConfig> configs, Collection<String> allowedTables)
+   {
+      List<Query> queries = new ArrayList<>();
+      for(QuickSearchableTableConfig config : configs == null ? List.<QuickSearchableTableConfig>of() : configs)
+      {
+         if(!allowedTables.contains(config.getTableName()) || !QuickSearchFieldRules.requiresFieldFiltering(config.getTableName()))
+         {
+            continue;
+         }
+         var table = QuickSearchFieldRules.lookupTable(config.getTableName());
+         List<String> fields = QuickSearchFieldRules.indexableFields(table, config.getSearchableFields()).stream()
+            .filter(config::isIncludedInSnippet).map(f -> PATH_PREFIX + "." + f).toList();
+         if(fields.isEmpty() || QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField()))
+         {
+            continue;
+         }
+         queries.add(Query.of(q -> q.bool(b -> b
+            .filter(f -> f.term(t -> t.field("sourceTable").value(FieldValue.of(config.getTableName()))))
+            .must(m -> m.multiMatch(mm -> mm.query(term).fields(fields).type(TextQueryType.CrossFields)
+               .operator(Operator.And).lenient(true))))));
+      }
+      return (queries);
+   }
+
+
+
    /*******************************************************************************
     ** Matches for the fields that are not in searchableText, each limited to
     ** its own table: an EXACT field matches the whole (lower-cased) term, any
@@ -965,7 +1011,7 @@ public class QuickSearchOpenSearchClient implements Closeable
       List<Query> queries = new ArrayList<>();
       for(QuickSearchableTableConfig tableConfig : tableConfigs == null ? List.<QuickSearchableTableConfig>of() : tableConfigs)
       {
-         if(!allowedTables.contains(tableConfig.getTableName()) || tableConfig.isAllPlainFields())
+         if(QuickSearchFieldRules.hasProtectedPrimaryKey(tableConfig.getTableName()) || !allowedTables.contains(tableConfig.getTableName()) || tableConfig.isAllPlainFields())
          {
             continue;
          }
@@ -973,7 +1019,8 @@ public class QuickSearchOpenSearchClient implements Closeable
          Query inTable = Query.of(q -> q.term(t -> t.field("sourceTable").value(FieldValue.of(tableConfig.getTableName()))));
          for(String field : tableConfig.getSearchableFields() == null ? List.<String>of() : tableConfig.getSearchableFields())
          {
-            if(tableConfig.isIncludedInSnippet(field))
+            if(tableConfig.isIncludedInSnippet(field)
+               || (QuickSearchFieldRules.lookupTable(tableConfig.getTableName()) != null && !QuickSearchFieldRules.isIndexable(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()), field)))
             {
                continue;
             }
@@ -1019,7 +1066,7 @@ public class QuickSearchOpenSearchClient implements Closeable
 
       for(QuickSearchableTableConfig tableConfig : tableConfigs)
       {
-         if(!allowedTables.contains(tableConfig.getTableName()))
+         if(QuickSearchFieldRules.hasProtectedPrimaryKey(tableConfig.getTableName()) || !allowedTables.contains(tableConfig.getTableName()))
          {
             continue;
          }
@@ -1028,6 +1075,10 @@ public class QuickSearchOpenSearchClient implements Closeable
          List<String>         fields  = tableConfig.getSearchableFields() == null ? List.of() : tableConfig.getSearchableFields();
          for(String field : fields)
          {
+            if(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()) != null && !QuickSearchFieldRules.isIndexable(QuickSearchFieldRules.lookupTable(tableConfig.getTableName()), field))
+            {
+               continue;
+            }
             float                    boost    = (weights == null || weights.get(field) == null) ? 1f : weights.get(field).floatValue();
             QuickSearchFieldAnalyzer analyzer = tableConfig.getAnalyzer(field);
             if(analyzer == QuickSearchFieldAnalyzer.EXACT)

src/main/java/com/kingsrook/qbits/quicksearch/processes/AbstractIndexingStep.java

@@ -132,7 +132,7 @@ public abstract class AbstractIndexingStep implements BackendStep
    protected List<QuickSearchableTableConfig> getDiscoveredTables()
    {
       List<QuickSearchableTableConfig> tables = QuickSearchQBitContext.getDiscoveredTables();
-      return (tables == null ? Collections.emptyList() : tables);
+      return (tables == null ? Collections.emptyList() : tables.stream().filter(t -> !QuickSearchFieldRules.hasProtectedPrimaryKey(t.getTableName())).toList());
    }
 
 
@@ -142,7 +142,7 @@ public abstract class AbstractIndexingStep implements BackendStep
     *******************************************************************************/
    protected QuickSearchableTableConfig getTableConfig(String tableName)
    {
-      return (QuickSearchQBitContext.getTableConfig(tableName));
+      return (QuickSearchFieldRules.hasProtectedPrimaryKey(tableName) ? null : QuickSearchQBitContext.getTableConfig(tableName));
    }
 
 

src/main/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStep.java

@@ -46,6 +46,7 @@ import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
 import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
 import com.kingsrook.qbits.quicksearch.config.QuickSearchConfigWrites;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchIndex;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchIndexRun;
@@ -423,6 +424,7 @@ public class BasepullIndexStep extends AbstractIndexingStep
          {
             Map<String, Serializable> values = new HashMap<>();
             values.put("lastBasepullTime", runStart);
+            values.put("realTimeErrorCount", 0);
             Integer documentCount = ReconcileIndexStep.safeCount(client, tableName);
             if(documentCount != null)
             {
@@ -505,6 +507,11 @@ public class BasepullIndexStep extends AbstractIndexingStep
 
       for(Map.Entry<String, List<QRecord>> entry : deleteByTable.entrySet())
       {
+         if(QuickSearchFieldRules.hasProtectedPrimaryKey(entry.getKey()))
+         {
+            markReplayOutcome(entry.getValue(), false, List.of("The primary key cannot be indexed"), succeeded, failed);
+            continue;
+         }
          List<QRecord> toDelete = entry.getValue();
          try
          {

src/main/java/com/kingsrook/qbits/quicksearch/processes/FullReindexStep.java

@@ -239,6 +239,7 @@ public class FullReindexStep extends AbstractIndexingStep
 
             Map<String, Serializable> values = new HashMap<>();
             values.put("lastFullReindexTime", Instant.now());
+            values.put("realTimeErrorCount", 0);
             values.put("lastBasepullTime", startTimes.get(tableName));
             values.put("recordCount", tableCounts.processed());
             Integer documentCount = ReconcileIndexStep.safeCount(client, tableName);

src/main/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtils.java

@@ -33,6 +33,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole;
 import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -103,6 +104,10 @@ public class IndexingUtils
 
       for(String fieldName : fields)
       {
+         if(table != null && !QuickSearchFieldRules.isIndexable(table, fieldName))
+         {
+            continue;
+         }
          String value = displayValue(record, fieldName, maxFieldLength);
          if(value == null)
          {
@@ -157,7 +162,7 @@ public class IndexingUtils
     *******************************************************************************/
    public static OpenSearchDocument buildDocument(QRecord record, String tableName, String primaryKeyField, List<String> fields, Map<String, Integer> fieldWeights, Map<String, Boolean> includeLabels)
    {
-      return (buildDocument(record, tableName, primaryKeyField, fields, includeLabels, lookupTable(tableName), null, null));
+      return (buildDocument(record, tableName, primaryKeyField, fields, includeLabels, lookupTable(tableName), null, null, null));
    }
 
 
@@ -165,13 +170,14 @@ public class IndexingUtils
    /*******************************************************************************
     ** Build a document for a configured table: display values, label fallback,
     ** truncation, and an external version from the basepull timestamp field.
-    ** Returns null when the record has no primary key.
+    ** Returns null when metadata is unavailable or the primary key is missing
+    ** or excluded by the field policy.
     *******************************************************************************/
    public static OpenSearchDocument buildDocument(QRecord record, QuickSearchableTableConfig tableConfig)
    {
       QTableMetaData     table = lookupTable(tableConfig.getTableName());
       OpenSearchDocument doc   = buildDocument(record, tableConfig.getTableName(), tableConfig.getPrimaryKeyField(), tableConfig.getSearchableFields(),
-         tableConfig.getFieldIncludeLabels(), table, tableConfig.getMaxFieldLength(), tableConfig.getBasepullTimestampField());
+         tableConfig.getFieldIncludeLabels(), table, tableConfig.getMaxFieldLength(), tableConfig.getBasepullTimestampField(), tableConfig.getRecordLabelFields());
 
       if(doc != null && tableConfig.getRecordLabelFormat() != null && tableConfig.getRecordLabelFields() != null)
       {
@@ -199,7 +205,7 @@ public class IndexingUtils
     *******************************************************************************/
    static void applyRolesAndAnalyzers(OpenSearchDocument doc, QRecord record, QuickSearchableTableConfig tableConfig, QTableMetaData table)
    {
-      List<String> fields = tableConfig.getSearchableFields() == null ? List.of() : tableConfig.getSearchableFields();
+      List<String> fields = table == null ? List.of() : QuickSearchFieldRules.indexableFields(table, tableConfig.getSearchableFields());
       if(!tableConfig.isAllPlainFields())
       {
          List<String> snippetFields = new ArrayList<>();
@@ -299,8 +305,14 @@ public class IndexingUtils
     **
     *******************************************************************************/
    private static OpenSearchDocument buildDocument(QRecord record, String tableName, String primaryKeyField, List<String> fields, Map<String, Boolean> includeLabels,
-      QTableMetaData table, Integer maxFieldLength, String timestampField)
+      QTableMetaData table, Integer maxFieldLength, String timestampField, List<String> recordLabelFields)
    {
+      if(table == null || primaryKeyField == null || table.getField(primaryKeyField) == null || QuickSearchFieldRules.isProtected(table, primaryKeyField))
+      {
+         QuickSearchFieldRules.warnExcludedFields(table);
+         return (null);
+      }
+      fields = QuickSearchFieldRules.indexableFields(table, fields);
       String recordId = record.getValueString(primaryKeyField);
       if(recordId == null)
       {
@@ -318,7 +330,15 @@ public class IndexingUtils
       }
 
       String recordLabel = record.getRecordLabel();
-      if(!StringUtils.hasContent(recordLabel) && table != null)
+      boolean protectedLabel = table.getRecordLabelFields() != null
+         && table.getRecordLabelFields().stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f));
+      if(protectedLabel || (recordLabelFields != null && recordLabelFields.stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f))))
+      {
+         recordLabel = null;
+         record = new QRecord(record);
+         record.setRecordLabel(null);
+      }
+      if(!StringUtils.hasContent(recordLabel) && !protectedLabel && table.getRecordLabelFields() != null)
       {
          try
          {
@@ -331,7 +351,7 @@ public class IndexingUtils
       }
 
       Long version = null;
-      if(timestampField != null)
+      if(timestampField != null && !QuickSearchFieldRules.isProtected(table, timestampField))
       {
          try
          {
@@ -361,6 +381,12 @@ public class IndexingUtils
     *******************************************************************************/
    private static String formatLabel(QRecord record, QuickSearchableTableConfig tableConfig, String fallback)
    {
+      QTableMetaData table = lookupTable(tableConfig.getTableName());
+      if(tableConfig.getRecordLabelFields().stream().anyMatch(f -> QuickSearchFieldRules.isProtected(table, f)))
+      {
+         QuickSearchFieldRules.warnExcludedFields(table);
+         return (fallback);
+      }
       try
       {
          Object[] labelValues = new Object[tableConfig.getRecordLabelFields().size()];
@@ -373,13 +399,42 @@ public class IndexingUtils
       }
       catch(Exception e)
       {
-         LOG.warn("recordLabelFormat could not be applied; using default label", e, logPair("tableName", tableConfig.getTableName()));
+         LOG.warn("recordLabelFormat could not be applied; using default label", logPair("tableName", tableConfig.getTableName()));
          return (fallback);
       }
    }
 
 
 
+   /*******************************************************************************
+    ** Rebuild result presentation from allowed field values, discarding old
+    ** aggregate labels, snippets and display values that may contain secrets.
+    *******************************************************************************/
+   public static OpenSearchDocument filterResult(OpenSearchDocument source, QuickSearchableTableConfig tableConfig)
+   {
+      if(!QuickSearchFieldRules.requiresFieldFiltering(source.getSourceTable()))
+      {
+         return (source);
+      }
+      QTableMetaData table = lookupTable(source.getSourceTable());
+      if(tableConfig == null || QuickSearchFieldRules.isProtected(table, table.getPrimaryKeyField()))
+      {
+         return (null);
+      }
+      QRecord record = new QRecord().withValue(table.getPrimaryKeyField(), source.getRecordId());
+      for(String field : QuickSearchFieldRules.indexableFields(table, tableConfig.getSearchableFields()))
+      {
+         Object value = source.getFieldValues() == null ? null : source.getFieldValues().get(field);
+         if(value instanceof String text)
+         {
+            record.setValue(field, text);
+         }
+      }
+      return (buildDocument(record, tableConfig));
+   }
+
+
+
    /*******************************************************************************
     **
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/quicksearch/processes/ReconcileIndexStep.java

@@ -169,6 +169,7 @@ public class ReconcileIndexStep extends AbstractIndexingStep
             if(ReconcileIndexStep.RUN_TYPE.equals(runType) == false)
             {
                values.put("lastFullReindexTime", Instant.now());
+               values.put("realTimeErrorCount", 0);
             }
             updateIndexRow(indexId, values);
          }

src/main/resources/openapi/qqq-v1/quick-search.yaml

@@ -0,0 +1,183 @@
+components:
+  schemas:
+    QuickSearchErrorV1:
+      properties:
+        error:
+          description: "Framework error message."
+          nullable: false
+          type: "string"
+      type: "object"
+    QuickSearchIndexStatusV1:
+      properties:
+        countsCheckedAt:
+          description: "Saved validation completion; null when validation counts are\
+            \ unavailable."
+          format: "date-time"
+          nullable: true
+          type: "string"
+        documentCount:
+          description: "Saved validation document count, else last indexing run count;\
+            \ not live."
+          format: "int64"
+          nullable: true
+          type: "integer"
+        enabled:
+          description: "Whether this index is enabled."
+          nullable: false
+          type: "boolean"
+        lastError:
+          description: "Persisted indexing error; null when absent."
+          nullable: true
+          type: "string"
+        lastSuccessfulSync:
+          description: "Newest successful basepull/full-reindex/reconcile watermark."
+          format: "date-time"
+          nullable: true
+          type: "string"
+        realTimeErrorCount:
+          description: "Persisted real-time indexing error count."
+          nullable: true
+          type: "integer"
+        sourceCount:
+          description: "Source count from saved IndexValidator report; null without\
+            \ validation."
+          format: "int64"
+          nullable: true
+          type: "integer"
+        status:
+          description: "Persisted indexing state, such as ACTIVE or NEEDS_REINDEX."
+          nullable: true
+          type: "string"
+        tableName:
+          description: "Indexed source table metadata name."
+          nullable: false
+          type: "string"
+      type: "object"
+    QuickSearchStatusV1:
+      properties:
+        activeShards:
+          description: "Cluster health counter; null when unavailable."
+          nullable: true
+          type: "integer"
+        backlogTruncated:
+          description: "Counts are lower bounds when more than 1000 backlog rows exist."
+          nullable: false
+          type: "boolean"
+        checkedAt:
+          description: "Snapshot completion time; remains unchanged on cache hits."
+          format: "date-time"
+          nullable: false
+          type: "string"
+        clusterHealth:
+          description: "OpenSearch health color; null when unknown."
+          nullable: true
+          type: "string"
+        clusterReachable:
+          description: "Whether OpenSearch answered; null when not probed."
+          nullable: true
+          type: "boolean"
+        exhausted:
+          description: "Exhausted failed events, from the same bounded backlog read."
+          nullable: true
+          type: "integer"
+        homeHref:
+          description: "Quick Search home link, including the configured metadata\
+            \ prefix."
+          nullable: false
+          type: "string"
+        indexes:
+          description: "At most 100 persisted index rows, ordered by id."
+          items:
+            $ref: "#/components/schemas/QuickSearchIndexStatusV1"
+          type: "array"
+        indexingPaused:
+          description: "Indexing is paused; search can still serve."
+          nullable: true
+          type: "boolean"
+        lastError:
+          description: "Last known indexing/search error or snapshot failure; null\
+            \ when absent."
+          nullable: true
+          type: "string"
+        lastSuccessfulSync:
+          description: "Newest successful indexing watermark; not a guarantee all\
+            \ tables are current."
+          format: "date-time"
+          nullable: true
+          type: "string"
+        nodes:
+          description: "Cluster health counter; null when unavailable."
+          nullable: true
+          type: "integer"
+        pending:
+          description: "Pending and awaiting-reindex failed events, from at most 1000\
+            \ backlog rows."
+          nullable: true
+          type: "integer"
+        state:
+          description: "Overall availability; incomplete diagnostics are DEGRADED."
+          enum:
+          - "OK"
+          - "DEGRADED"
+          - "DOWN"
+          - "DISABLED"
+          nullable: false
+          type: "string"
+        tablesTruncated:
+          description: "More than 100 index rows exist."
+          nullable: false
+          type: "boolean"
+        unassignedShards:
+          description: "Cluster health counter; null when unavailable."
+          nullable: true
+          type: "integer"
+      type: "object"
+  securitySchemes:
+    sessionUuidCookie:
+      in: "cookie"
+      name: "sessionUUID"
+      type: "apiKey"
+info:
+  contact:
+    email: "contact@kingsrook.com"
+  description: "Quick Search administrative status"
+  title: "QQQ Middleware API extension: quick-search"
+  version: "1.0.0"
+openapi: "3.0.3"
+paths:
+  /qqq/v1/qbit/quick-search/status:
+    get:
+      description: "Requires Quick Search admin permission. Five-second instance cache;\
+        \ two-second refresh wait. Counts are saved diagnostics, not live source scans.\
+        \ Unknown values are null. Backlog counts are lower bounds when backlogTruncated\
+        \ is true."
+      operationId: "getQuickSearchStatus"
+      responses:
+        400:
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/QuickSearchErrorV1"
+          description: "Diagnostic rate limit exceeded"
+        401:
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/QuickSearchErrorV1"
+          description: "Sign in required"
+        403:
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/QuickSearchErrorV1"
+          description: "Quick Search admin permission required"
+        200:
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/QuickSearchStatusV1"
+          description: "Successful Response"
+      security:
+      - sessionUuidCookie:
+        - "N/A"
+      summary: "Get Quick Search status"

src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchApiContractTest.java

@@ -0,0 +1,60 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch;
+
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchApi;
+import com.kingsrook.qqq.middleware.javalin.specs.v1.MiddlewareVersionV1;
+import com.kingsrook.qqq.middleware.javalin.tools.UiApiContract;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** The committed contract is generated from exactly the registered specs.
+ *******************************************************************************/
+class QuickSearchApiContractTest
+{
+   /*******************************************************************************
+    ** Opt-in regeneration follows core and collaboration's contract workflow.
+    *******************************************************************************/
+   @Test
+   void contract() throws Exception
+   {
+      var version = new MiddlewareVersionV1();
+      var extension = QuickSearchApi.extension();
+      String generated = UiApiContract.generateExtensionYaml(version, extension);
+      Path path = Path.of("src/main/resources/openapi/qqq-v1/quick-search.yaml");
+      if(Boolean.getBoolean("quicksearch.openapi.update"))
+      {
+         Files.createDirectories(path.getParent());
+         Files.writeString(path, generated);
+      }
+      assertThat(UiApiContract.check(generated, Files.readString(path), null, 1,
+         "mvnw.cmd -Dtest=QuickSearchApiContractTest -Dquicksearch.openapi.update=true test")).isEmpty();
+      var model = version.generateExtensionOpenAPIModel("qqq", extension);
+      assertThat(model.getComponents().getSchemas().keySet()).allMatch(name -> name.startsWith("QuickSearch"));
+      assertThat(model.getPaths().keySet()).containsExactly("/qqq/v1/qbit/quick-search/status");
+      assertThat(extension.getSpecs()).hasSize(1);
+   }
+}

src/test/java/com/kingsrook/qbits/quicksearch/QuickSearchStatusTest.java

@@ -0,0 +1,405 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2026.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.quicksearch;
+
+
+import java.net.InetSocketAddress;
+import java.net.ServerSocket;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchApi;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatus;
+import com.kingsrook.qbits.quicksearch.api.QuickSearchStatusService;
+import com.kingsrook.qbits.quicksearch.diagnostics.StatusDiagnostics;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
+import com.kingsrook.qbits.quicksearch.model.QuickSearchValidationRun;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
+import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
+import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+import com.kingsrook.qqq.backend.core.model.session.QUser;
+import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
+import com.kingsrook.qqq.middleware.javalin.QJavalinImplementation;
+import com.sun.net.httpserver.HttpServer;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Real bounded diagnostics over a memory backend and a loopback health endpoint.
+ *******************************************************************************/
+class QuickSearchStatusTest
+{
+   private HttpServer healthServer;
+   private QuickSearchQBitConfig config;
+   private QInstance instance;
+   private volatile String health = "green";
+   private volatile int requests;
+   private CountDownLatch blocked;
+
+
+   /*******************************************************************************
+    ** Only cluster health is exposed: status must never scan the source/index.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws Exception
+   {
+      MemoryRecordStore.getInstance().reset();
+      healthServer = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+      healthServer.createContext("/", exchange ->
+      {
+         requests++;
+         if(blocked != null)
+         {
+            try
+            {
+               blocked.await(10, TimeUnit.SECONDS);
+            }
+            catch(InterruptedException e)
+            {
+               Thread.currentThread().interrupt();
+            }
+         }
+         byte[] body = ("{\"status\":\"" + health + "\",\"number_of_nodes\":1,\"active_shards\":1,\"unassigned_shards\":0}").getBytes(StandardCharsets.UTF_8);
+         exchange.getResponseHeaders().add("Content-Type", "application/json");
+         exchange.sendResponseHeaders(200, body.length);
+         exchange.getResponseBody().write(body);
+         exchange.close();
+      });
+      healthServer.start();
+      instance = new QInstance();
+      instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
+      instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
+      instance.addTable(new QTableMetaData().withName("product").withBackendName("memory").withPrimaryKeyField("id")
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER)).withField(new QFieldMetaData("name", QFieldType.STRING)));
+      QContext.init(instance, new QSystemUserSession());
+      config = new QuickSearchQBitConfig().withBackendName("memory").withStartupMode(QuickSearchStartupMode.DEGRADED)
+         .withOpensearchUrl("http://127.0.0.1:" + healthServer.getAddress().getPort()).withEnableRealTimeIndexing(false)
+         .withOpensearchIndexName("status-test").withSearchableTable("product", List.of(new SearchableFieldConfig("name")));
+      new QuickSearchQBitProducer().withConfig(config).produce(instance);
+      insert(config.getQuickSearchIndexTableName(), new QRecord().withValue("tableName", "product").withValue("enabled", true)
+         .withValue("status", "ACTIVE").withValue("documentCount", 12).withValue("lastBasepullTime", Instant.parse("2026-10-01T00:00:00Z")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Close the qbit worker, transport, and test server.
+    *******************************************************************************/
+   @AfterEach
+   void tearDown()
+   {
+      if(blocked != null)
+      {
+         blocked.countDown();
+      }
+      if(config != null && config.getRuntime() != null)
+      {
+         config.getRuntime().close();
+      }
+      healthServer.stop(0);
+      QContext.clear();
+      MemoryRecordStore.getInstance().reset();
+   }
+
+
+
+   /*******************************************************************************
+    ** Cached values stay immutable and permissions are rechecked on cache hits.
+    *******************************************************************************/
+   @Test
+   void cacheAndPermission() throws Exception
+   {
+      QuickSearchStatus first = QuickSearchStatusService.get();
+      assertThat(first.state()).isEqualTo(QuickSearchStatus.State.OK);
+      assertThat(first.clusterReachable()).isTrue();
+      assertThat(first.indexes().get(0).documentCount()).isEqualTo(12);
+      assertThat(first.indexes().get(0).sourceCount()).isNull();
+      assertThat(first.homeHref()).isEqualTo(QuickSearchApi.HOME_HREF);
+      assertThat(QuickSearchStatusService.get()).isSameAs(first);
+      assertThat(requests).isEqualTo(1);
+      assertThatThrownBy(() -> first.indexes().clear()).isInstanceOf(UnsupportedOperationException.class);
+      QContext.init(instance, new QSession());
+      assertThatThrownBy(QuickSearchStatusService::get).isInstanceOf(QPermissionDeniedException.class);
+      assertThat(requests).isEqualTo(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Saved validator counts and operational backlog cause degradation.
+    *******************************************************************************/
+   @Test
+   void savedValidationAndBacklog() throws Exception
+   {
+      Instant measured = Instant.parse("2026-10-02T00:00:00Z");
+      insert(QuickSearchValidationRun.TABLE_NAME, new QRecord().withValue("status", "WARN").withValue("endTime", measured)
+         .withValue("reportJson", "{\"tableStats\":[{\"tableName\":\"product\",\"databaseCount\":20,\"indexCount\":12}]}"));
+      insert(QuickSearchFailedEvent.TABLE_NAME, new QRecord().withValue("status", "PENDING"));
+      insert(QuickSearchFailedEvent.TABLE_NAME, new QRecord().withValue("status", "EXHAUSTED"));
+      QuickSearchStatus result = QuickSearchStatusService.get();
+      assertThat(result.state()).isEqualTo(QuickSearchStatus.State.DEGRADED);
+      assertThat(result.pending()).isEqualTo(1);
+      assertThat(result.exhausted()).isEqualTo(1);
+      assertThat(result.indexes().get(0).sourceCount()).isEqualTo(20);
+      assertThat(result.indexes().get(0).countsCheckedAt()).isEqualTo(measured);
+      assertThat(result.lastSuccessfulSync()).isEqualTo(Instant.parse("2026-10-01T00:00:00Z"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Reachability differs from healthy primaries; yellow is degraded, red down.
+    *******************************************************************************/
+   @Test
+   void healthColorsAndOffline() throws Exception
+   {
+      health = "yellow";
+      assertThat(StatusDiagnostics.read(config.getRuntime()).state()).isEqualTo(QuickSearchStatus.State.DEGRADED);
+      health = "red";
+      QuickSearchStatus red = StatusDiagnostics.read(config.getRuntime());
+      assertThat(red.state()).isEqualTo(QuickSearchStatus.State.DOWN);
+      assertThat(red.clusterReachable()).isTrue();
+      healthServer.stop(0);
+      QuickSearchStatus down = StatusDiagnostics.read(config.getRuntime());
+      assertThat(down.state()).isEqualTo(QuickSearchStatus.State.DOWN);
+      assertThat(down.clusterReachable()).isFalse();
+   }
+
+
+
+   /*******************************************************************************
+    ** A blocked remote call has bounded caller latency and cannot grow a queue.
+    *******************************************************************************/
+   @Test
+   void timeoutDoesNotQueueMoreWork() throws Exception
+   {
+      blocked = new CountDownLatch(1);
+      long start = System.nanoTime();
+      QuickSearchStatus result = QuickSearchStatusService.get();
+      assertThat(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start)).isLessThan(3500);
+      assertThat(result.state()).isEqualTo(QuickSearchStatus.State.DEGRADED);
+      assertThat(result.clusterReachable()).isNull();
+      assertThat(result.lastError()).contains("timed out");
+      assertThat(QuickSearchStatusService.get()).isSameAs(result);
+      assertThat(requests).isEqualTo(1);
+      blocked.countDown();
+      Thread.sleep(5100);
+      QuickSearchStatus recovered = QuickSearchStatusService.get();
+      assertThat(recovered.state()).isEqualTo(QuickSearchStatus.State.OK);
+      assertThat(recovered.checkedAt()).isAfter(result.checkedAt());
+      assertThat(requests).isEqualTo(2);
+   }
+
+
+
+   /*******************************************************************************
+    ** Rate limiting uses the diagnostics limiter before launching remote work.
+    *******************************************************************************/
+   @Test
+   void rateLimitBeforeRefresh() throws Exception
+   {
+      config.withDiagnosticRunsPerMinute(1);
+      com.kingsrook.qbits.quicksearch.diagnostics.DiagnosticsSupport.checkRateLimit(config.getRuntime(), "STATUS");
+      assertThatThrownBy(QuickSearchStatusService::get).hasMessageContaining("Too many");
+      assertThat(requests).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** Bounds are explicit, not silent truncation or full counts of growing queues.
+    *******************************************************************************/
+   @Test
+   void boundedTablesAndBacklog() throws Exception
+   {
+      List<QRecord> indexes = new java.util.ArrayList<>();
+      for(int i = 0; i < 100; i++)
+      {
+         indexes.add(new QRecord().withValue("tableName", "table" + i).withValue("enabled", true).withValue("status", "ACTIVE"));
+      }
+      new InsertAction().execute(new InsertInput(config.getQuickSearchIndexTableName()).withRecords(indexes));
+      List<QRecord> backlog = new java.util.ArrayList<>();
+      for(int i = 0; i < 1001; i++)
+      {
+         backlog.add(new QRecord().withValue("status", "AWAITING_REINDEX"));
+      }
+      new InsertAction().execute(new InsertInput(QuickSearchFailedEvent.TABLE_NAME).withRecords(backlog));
+      QuickSearchStatus status = QuickSearchStatusService.get();
+      assertThat(status.tablesTruncated()).isTrue();
+      assertThat(status.backlogTruncated()).isTrue();
+      assertThat(status.indexes()).hasSize(100);
+      assertThat(status.pending()).isEqualTo(1000);
+      assertThat(status.state()).isEqualTo(QuickSearchStatus.State.DEGRADED);
+   }
+
+
+
+   /*******************************************************************************
+    ** Separate QInstances cannot reuse each other's snapshots; links honor prefixes.
+    *******************************************************************************/
+   @Test
+   void prefixedDisabledInstance() throws Exception
+   {
+      QuickSearchStatus first = QuickSearchStatusService.get();
+      QInstance other = new QInstance();
+      other.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
+      other.addTable(instance.getTable("product"));
+      other.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
+      QuickSearchQBitConfig otherConfig = new QuickSearchQBitConfig().withBackendName("memory").withTableNamePrefix("host_")
+         .withStartupMode(QuickSearchStartupMode.DEGRADED).withOpensearchUrl("http://127.0.0.1:1")
+         .withOpensearchIndexName("other").withEnableRealTimeIndexing(false)
+         .withSearchableTable("product", List.of(new SearchableFieldConfig("name")));
+      QContext.init(other, new QSystemUserSession());
+      new QuickSearchQBitProducer().withConfig(otherConfig).produce(other);
+      try
+      {
+         insert(otherConfig.getQuickSearchIndexTableName(), new QRecord().withValue("tableName", "product").withValue("enabled", false));
+         QuickSearchStatus second = QuickSearchStatusService.get();
+         assertThat(second).isNotSameAs(first);
+         assertThat(second.state()).isEqualTo(QuickSearchStatus.State.DISABLED);
+         assertThat(second.clusterReachable()).isNull();
+         assertThat(second.homeHref()).isEqualTo("/app/host_quickSearchAdmin");
+         assertThat(requests).isEqualTo(1);
+         assertThat(second.toSchema()).isNotNull();
+         assertThat(second.indexes().get(0).toSchema()).isNotNull();
+      }
+      finally
+      {
+         otherConfig.getRuntime().close();
+         QContext.init(instance, new QSystemUserSession());
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Strict route auditing, real authentication, 403 after cache population,
+    ** and the same serialized snapshot through the production HTTP pipeline.
+    *******************************************************************************/
+   @Test
+   void strictHttpContract() throws Exception
+   {
+      instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("mock").withType(QAuthenticationType.MOCK)
+         .withCustomizer(new QCodeReference(TestSessions.class)));
+      int port;
+      try(ServerSocket socket = new ServerSocket(0))
+      {
+         port = socket.getLocalPort();
+      }
+      QApplicationJavalinServer server = new QApplicationJavalinServer(new AbstractQQQApplication()
+      {
+         @Override
+         public QInstance defineQInstance()
+         {
+            return (instance);
+         }
+      }).withPort(port).withServeFrontendNext(false).withServeFrontendMaterialDashboard(false)
+         .withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true);
+      try(HttpClient client = HttpClient.newHttpClient())
+      {
+         server.start();
+         String url = "http://127.0.0.1:" + port + "/qqq/v1/qbit/quick-search/status";
+         String suffix = UUID.randomUUID().toString();
+         var response = client.send(HttpRequest.newBuilder(URI.create(url)).header("Cookie", "sessionUUID=admin-" + suffix).GET().build(), HttpResponse.BodyHandlers.ofString());
+         assertThat(response.statusCode()).as(response.body()).isEqualTo(200);
+         var json = new ObjectMapper().readTree(response.body());
+         assertThat(json.path("homeHref").asText()).isEqualTo(QuickSearchApi.HOME_HREF);
+         assertThat(json.path("checkedAt").isTextual()).isTrue();
+         assertThat(json.path("indexes").isArray()).isTrue();
+         java.util.Set<String> fields = new java.util.HashSet<>();
+         json.fieldNames().forEachRemaining(fields::add);
+         assertThat(fields).containsExactlyInAnyOrderElementsOf(new com.kingsrook.qbits.quicksearch.api.QuickSearchStatusSpec()
+            .defineComponentSchemas().get("QuickSearchStatusV1").getProperties().keySet());
+         var anonymous = client.send(HttpRequest.newBuilder(URI.create(url)).GET().build(), HttpResponse.BodyHandlers.ofString());
+         // MOCK authentication supplies a session even without cookies; it has no admin permission.
+         assertThat(anonymous.statusCode()).as(anonymous.body()).isEqualTo(403);
+         var denied = client.send(HttpRequest.newBuilder(URI.create(url)).header("Cookie", "sessionUUID=reader-" + suffix).GET().build(), HttpResponse.BodyHandlers.ofString());
+         assertThat(denied.statusCode()).as(denied.body()).isEqualTo(403);
+         var openapi = client.send(HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + port + "/qqq/v1/openapi.yaml")).GET().build(), HttpResponse.BodyHandlers.ofString());
+         assertThat(openapi.body()).contains("/qqq/v1/qbit/quick-search/status", "QuickSearchStatusV1");
+      }
+      finally
+      {
+         server.stop();
+         QJavalinImplementation.setSessionCookieHttpOnly(false);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Production mock-auth customization with distinct administrator and reader.
+    *******************************************************************************/
+   public static class TestSessions implements QAuthenticationModuleCustomizerInterface
+   {
+      /*******************************************************************************
+       ** Grant only the administrator the existing qbit admin permission.
+       *******************************************************************************/
+      @Override
+      public void customizeSession(QInstance instance, QSession session, Map<String, Object> context)
+      {
+         session.withUser(new QUser().withIdReference(session.getUuid()));
+         if(session.getUuid().startsWith("admin-"))
+         {
+            session.withPermissions("quick_search_admin.hasAccess");
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Insert fixture metadata using the same backend actions as production.
+    *******************************************************************************/
+   private void insert(String table, QRecord row) throws Exception
+   {
+      new InsertAction().execute(new InsertInput(table).withRecords(List.of(row)));
+   }
+}

src/test/java/com/kingsrook/qbits/quicksearch/config/QuickSearchConfigCustomizersTest.java

@@ -19,8 +19,13 @@ package com.kingsrook.qbits.quicksearch.config;
 
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.AiTableMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
 import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QCollectingLogger;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
@@ -28,6 +33,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFailedEvent;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldAnalyzer;
 import com.kingsrook.qbits.quicksearch.model.QuickSearchFieldRole;
@@ -47,6 +54,106 @@ import static org.assertj.core.api.Assertions.assertThat;
 class QuickSearchConfigCustomizersTest extends BaseDataDrivenTest
 {
 
+   /*******************************************************************************
+    ** Classification falls back to the table; PUBLIC and unclassified fields
+    ** remain searchable. Password, hidden, sensitive and locked fields do not.
+    *******************************************************************************/
+   @Test
+   void testClassifiedAndSecuredFieldRules()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
+      assertThat(QuickSearchFieldRules.isIndexable(table, "name")).isTrue();
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse();
+      for(DataClassification classification : DataClassification.values())
+      {
+         table.getField("name").withSupplementalMetaData(new AiFieldMetaData().withClassification(classification));
+         assertThat(QuickSearchFieldRules.isIndexable(table, "name")).as(classification.name())
+            .isEqualTo(classification != DataClassification.PII && classification != DataClassification.PHI);
+         assertThat(QuickSearchFieldRules.isProtected(table, "name")).as(classification.name())
+            .isEqualTo(classification == DataClassification.PII || classification == DataClassification.PHI);
+      }
+      table.withSupplementalMetaData(new AiTableMetaData().withClassification(DataClassification.PHI));
+      assertThat(QuickSearchFieldRules.whyNotIndexable(table, "description")).contains("classified PHI");
+      assertThat(QuickSearchFieldRules.isProtected(table, "description")).isTrue();
+      table.getField("name").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PUBLIC));
+      assertThat(QuickSearchFieldRules.isIndexable(table, "name")).isTrue();
+      table.getField("name").setFieldSecurityLock(new FieldSecurityLock());
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue();
+      assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("security lock");
+      table.getField("name").setFieldSecurityLock(null);
+      table.getField("name").setType(QFieldType.PASSWORD);
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue();
+      assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("sensitive");
+      table.getField("name").setType(QFieldType.STRING);
+      table.getField("name").setIsHidden(true);
+      assertThat(QuickSearchFieldRules.whyNotIndexable(table, "name")).contains("hidden");
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse();
+      for(QFieldType type : List.of(QFieldType.BOOLEAN, QFieldType.BLOB))
+      {
+         table.getField("name").setType(type);
+         assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse();
+      }
+      table.getField("name").setType(null);
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isFalse();
+      table.getField("name").setIsSensitive(true);
+      assertThat(QuickSearchFieldRules.isProtected(table, "name")).isTrue();
+      assertThat(QuickSearchFieldRules.isProtected(null, "name")).isFalse();
+      assertThat(QuickSearchFieldRules.isProtected(table, null)).isFalse();
+      assertThat(QuickSearchFieldRules.isProtected(table, "missing")).isFalse();
+   }
+
+
+
+   /*******************************************************************************
+    ** Repeated config loads and document writes warn only once per table.
+    *******************************************************************************/
+   @Test
+   void testExcludedFieldsWarnOncePerTable()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
+      table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      QCollectingLogger logger = QLogger.activateCollectingLoggerForClass(QuickSearchFieldRules.class);
+      try
+      {
+         for(int i = 0; i < 3; i++)
+         {
+            assertThat(QuickSearchFieldRules.indexableFields(table, List.of("name", "description"))).containsExactly("name");
+         }
+      }
+      finally
+      {
+         QLogger.deactivateCollectingLoggerForClass(QuickSearchFieldRules.class);
+      }
+      assertThat(logger.getCollectedMessages()).singleElement()
+         .satisfies(message -> assertThat(message.getMessage()).contains("dropped fields", TEST_ENTITY_TABLE, "PII/PHI"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Config writes refuse classified fields with a useful message; the same
+    ** fields are absent from the picker and default selection.
+    *******************************************************************************/
+   @Test
+   void testClassifiedFieldConfigIsRefused() throws QException
+   {
+      runtime.getConfigStore().tryPrime();
+      Integer indexId = indexRow(TEST_ENTITY_TABLE).getValueInteger("id");
+      QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
+      for(DataClassification classification : List.of(DataClassification.PII, DataClassification.PHI))
+      {
+         String fieldName = classification.name().toLowerCase(java.util.Locale.ROOT);
+         table.withField(new QFieldMetaData(fieldName, QFieldType.STRING)
+            .withSupplementalMetaData(new AiFieldMetaData().withClassification(classification)));
+         assertThat(insertField(indexId, fieldName, 1, null).getErrors()).singleElement()
+            .satisfies(e -> assertThat(e.getMessage()).contains("classified " + classification, "cannot be indexed"));
+         assertThat(QuickSearchFieldRules.indexableFields(table)).extracting(QFieldMetaData::getName).doesNotContain(fieldName);
+         assertThat(QuickSearchFieldRules.defaultFields(table)).extracting(QuickSearchFieldRules.DefaultField::fieldName).doesNotContain(fieldName);
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Field rows: a field must exist, be indexable, not sensitive or hidden,
     ** and not be listed twice; weights 1-1000; known role and analyzer.

src/test/java/com/kingsrook/qbits/quicksearch/listeners/QuickSearchRecordChangeListenerTest.java

@@ -19,7 +19,13 @@ package com.kingsrook.qbits.quicksearch.listeners;
 
 import java.util.ArrayList;
 import java.util.List;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
 import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
@@ -36,6 +42,7 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
@@ -142,6 +149,28 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
 
 
 
+   /*******************************************************************************
+    ** A custom search publisher also receives only allowed values, so it cannot
+    ** bypass the document builder's policy by indexing the event record.
+    *******************************************************************************/
+   @Test
+   void testCustomPublisherReceivesNoClassifiedValues() throws QException
+   {
+      QContext.getQInstance().getTable(TEST_ENTITY_TABLE).getField("description")
+         .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      QContext.getQInstance().getTable(TEST_ENTITY_TABLE).withRecordLabelFields("description").withRecordLabelFormat("%s");
+      new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecord(new QRecord()
+         .withValue("name", "Public widget").withValue("description", "PHI-CANARY-7f3c")));
+      ArgumentCaptor<List<IndexEvent>> captor = ArgumentCaptor.forClass(List.class);
+      verify(publisher).publishIndexEvents(captor.capture());
+      QRecord record = captor.getValue().get(0).getRecord();
+      assertThat(record.getValues()).containsEntry("name", "Public widget").doesNotContainKey("description");
+      assertThat(record.getDisplayValue("description")).isNull();
+      assertThat(record.getRecordLabel()).isNull();
+   }
+
+
+
    @Test
    void testAppliesTo_onlyConfiguredTablesWhenEnabled()
    {
@@ -203,7 +232,9 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
       }
 
       assertThat(queryFailedEvents()).isEmpty();
+      assertThat(realTimeErrorCount()).isZero();
       transaction.commit();
+      assertThat(realTimeErrorCount()).isEqualTo(2);
 
       List<QRecord> failed = queryFailedEvents();
       assertThat(failed).extracting(r -> r.getValueString("recordId")).containsExactlyInAnyOrder("7", "8");
@@ -328,6 +359,102 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
 
 
 
+   /*******************************************************************************
+    ** Count failed events, including repeated failures, rather than batches.
+    *******************************************************************************/
+   @Test
+   void testPublishFailure_incrementsErrorCount() throws QException
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishDeleteEvents(anyList());
+      List<IndexEvent> events = List.of(
+         new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("7").withAction(IndexEventAction.DELETE),
+         new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("8").withAction(IndexEventAction.DELETE));
+
+      QuickSearchRecordChangeListener.publish(runtime, List.of(), events);
+      assertThat(realTimeErrorCount()).isEqualTo(2);
+      QuickSearchRecordChangeListener.publish(runtime, List.of(), events);
+      assertThat(realTimeErrorCount()).isEqualTo(4);
+      assertThat(queryFailedEvents()).hasSize(4);
+   }
+
+
+
+   /*******************************************************************************
+    ** Independent listeners must not overwrite each other's increments.
+    *******************************************************************************/
+   @Test
+   void testConcurrentFailures_preserveErrorCount() throws Exception
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishDeleteEvents(anyList());
+      QInstance instance = QContext.getQInstance();
+      CountDownLatch start = new CountDownLatch(1);
+      try(var executor = Executors.newFixedThreadPool(4))
+      {
+         List<Future<?>> futures = new ArrayList<>();
+         for(Integer worker = 0; worker < 4; worker++)
+         {
+            futures.add(executor.submit(() ->
+            {
+               QContext.init(instance, new QSession());
+               try
+               {
+                  assertThat(start.await(10, TimeUnit.SECONDS)).isTrue();
+                  for(Integer i = 0; i < 5; i++)
+                  {
+                     QuickSearchRecordChangeListener.publish(runtime, List.of(), List.of(
+                        new IndexEvent().withTableName(TEST_ENTITY_TABLE).withRecordId("7").withAction(IndexEventAction.DELETE)));
+                  }
+               }
+               finally
+               {
+                  QContext.clear();
+               }
+               return (null);
+            }));
+         }
+         start.countDown();
+         for(Future<?> future : futures)
+         {
+            future.get(20, TimeUnit.SECONDS);
+         }
+      }
+      assertThat(realTimeErrorCount()).isEqualTo(20);
+   }
+
+
+
+   /*******************************************************************************
+    ** Counter storage failure must leave the source write and replay intact.
+    *******************************************************************************/
+   @Test
+   void testCounterUpdateFails_doesNotFailWrite() throws QException
+   {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
+      doThrow(new QException("cluster down")).when(publisher).publishIndexEvents(anyList());
+      try(MockedConstruction<UpdateAction> ignored = mockConstruction(UpdateAction.class, (mock, context) ->
+         when(mock.execute(any(UpdateInput.class))).thenThrow(new QException("counter unavailable"))))
+      {
+         assertThat(insert("widget", null)).isNotNull();
+      }
+      assertThat(queryFailedEvents()).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Read the persisted counter from the memory backend.
+    *******************************************************************************/
+   private Integer realTimeErrorCount() throws QException
+   {
+      QueryInput input = new QueryInput();
+      input.setTableName(QuickSearchIndex.TABLE_NAME);
+      return (new QueryAction().execute(input).getRecords().get(0).getValueInteger("realTimeErrorCount"));
+   }
+
+
+
    @Test
    void testDisabledTable_isSkipped() throws QException
    {
@@ -410,6 +537,7 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
    @Test
    void testUpdate_reReadFails_recordsFailedEventsAfterCommit() throws QException
    {
+      insertIndexRowWithStatus(AbstractIndexingStep.STATUS_ACTIVE);
       QBackendTransaction transaction = new QBackendTransaction();
       try(MockedConstruction<QueryAction> ignored = failingQueries(new QException("possible-value backend unavailable")))
       {
@@ -417,7 +545,9 @@ class QuickSearchRecordChangeListenerTest extends BaseQuickSearchTest
       }
 
       assertThat(queryFailedEvents()).isEmpty();
+      assertThat(realTimeErrorCount()).isNull();
       transaction.commit();
+      assertThat(realTimeErrorCount()).isEqualTo(2);
 
       List<QRecord> failed = queryFailedEvents();
       assertThat(failed).extracting(r -> r.getValueString("recordId")).containsExactlyInAnyOrder("7", "8");

src/test/java/com/kingsrook/qbits/quicksearch/opensearch/ClassifiedDataCanaryTest.java

@@ -0,0 +1,193 @@
+/*
+ * Copyright 2024 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.kingsrook.qbits.quicksearch.opensearch;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
+import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest;
+import com.kingsrook.qbits.quicksearch.QuickSearchQBitConfig;
+import com.kingsrook.qbits.quicksearch.QuickSearchQBitContext;
+import com.kingsrook.qbits.quicksearch.QuickSearchRuntime;
+import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.actions.QuickSearchAction;
+import com.kingsrook.qbits.quicksearch.actions.QuickSearchInput;
+import com.kingsrook.qbits.quicksearch.listeners.QuickSearchRecordChangeListener;
+import com.kingsrook.qbits.quicksearch.processes.BasepullIndexStep;
+import com.kingsrook.qbits.quicksearch.processes.IndexingUtils;
+import com.kingsrook.qbits.quicksearch.publisher.SynchronousIndexEventPublisher;
+import org.junit.jupiter.api.Test;
+import org.opensearch.client.opensearch.core.SearchResponse;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.when;
+
+
+/*******************************************************************************
+ ** Canary coverage at the backend seam for live writes, batch indexing and
+ ** stale search results. No OpenSearch server is required.
+ *******************************************************************************/
+class ClassifiedDataCanaryTest extends BaseQuickSearchTest
+{
+   private static final String CANARY = "PHI-CANARY-7f3c";
+
+
+
+   /*******************************************************************************
+    ** Classified values must never reach any document representation or result,
+    ** even with explicit field lists, label formats, roles and analyzer copies.
+    *******************************************************************************/
+   @Test
+   void testCanaryAcrossRealtimeBasepullAndResults() throws Exception
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
+      table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      table.withField(new QFieldMetaData("personal", QFieldType.STRING)
+         .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII)));
+      table.withRecordLabelFields("description").withRecordLabelFormat("%s");
+
+      QuickSearchableTableConfig fields = new QuickSearchableTableConfig().withTableName(TEST_ENTITY_TABLE).withPrimaryKeyField("id")
+         .withSearchableFields(List.of("name", "description", "personal")).withEnabledByDefault(true)
+         .withBasepullTimestampField("modifyDate").withBasepullIntervalMinutes(5)
+         .withRecordLabelFields(List.of("description", "personal")).withRecordLabelFormat("%s %s")
+         .withFieldRoles(Map.of("name", "TITLE", "description", "SUBTITLE", "personal", "DISPLAY"))
+         .withFieldAnalyzers(Map.of("description", "ENGLISH", "personal", "EXACT"));
+      QuickSearchOpenSearchClient client = mock(QuickSearchOpenSearchClient.class);
+      List<OpenSearchDocument> captured = new ArrayList<>();
+      doAnswer(invocation ->
+      {
+         List<OpenSearchDocument> documents = invocation.getArgument(0);
+         captured.addAll(documents);
+         return (new BulkIndexResult().withSuccessCount(documents.size()));
+      }).when(client).indexDocuments(anyList(), anyInt());
+      when(client.indexExists()).thenReturn(true);
+      when(client.countAllDocuments()).thenReturn(1_000L);
+      when(client.countDocumentsForTable(anyString())).thenReturn(1L);
+
+      QuickSearchQBitConfig config = QuickSearchQBitContext.getConfig();
+      config.withIndexEventPublisher(new SynchronousIndexEventPublisher(client, List.of(fields), 100));
+      QuickSearchRuntime runtime = spy(new QuickSearchRuntime(config));
+      doReturn(client).when(runtime).getClient();
+      runtime.setDiscoveredTables(List.of(fields));
+      config.setRuntime(runtime);
+      QContext.getQInstance().addQBit(new QBitMetaData().withGroupId("com.kingsrook.qbits").withArtifactId("quick-search").withVersion("test").withConfig(config));
+      QContext.getQInstance().withRecordChangeListener(new QCodeReference(QuickSearchRecordChangeListener.class));
+      QuickSearchQBitContext.setClient(client);
+      QuickSearchQBitContext.setDiscoveredTables(List.of(fields));
+
+      new InsertAction().execute(new InsertInput(TEST_ENTITY_TABLE).withRecord(new QRecord()
+         .withValue("name", "Public widget").withValue("description", CANARY).withValue("personal", CANARY)));
+      assertThat(captured).as("real-time indexing reached the backend").hasSize(1);
+      new BasepullIndexStep().run(new RunBackendStepInput(), new RunBackendStepOutput());
+      assertThat(captured.size()).as("basepull also reached the backend").isGreaterThan(1);
+      ObjectMapper mapper = new ObjectMapper().registerModule(new JavaTimeModule());
+      for(OpenSearchDocument document : captured)
+      {
+         assertThat(mapper.writeValueAsString(document)).doesNotContain(CANARY).contains("Public widget");
+         assertThat(document.getFieldValues()).containsEntry("name", "Public widget").doesNotContainKeys("description", "personal");
+      }
+
+      OpenSearchDocument stale = new OpenSearchDocument().withSourceTable(TEST_ENTITY_TABLE).withRecordId("1")
+         .withRecordLabel(CANARY).withSearchableText(CANARY).withSubtitle(CANARY).withDisplayValues(Map.of("Personal", CANARY))
+         .withFieldValues(Map.of("name", "Public widget", "description", CANARY, "personal", CANARY));
+      SearchResponse<OpenSearchDocument> response = new SearchResponse.Builder<OpenSearchDocument>().took(1).timedOut(false)
+         .shards(sh -> sh.total(1).successful(1).failed(0))
+         .hits(h -> h.hits(hit -> hit.index("test").id("1").source(stale).highlight(Map.of("searchableText", List.of(CANARY))))).build();
+      when(client.search(anyString(), anyList(), anyInt(), anyInt(), anyList(), any())).thenReturn(response);
+      config.setApplyRecordSecurityLocks(false);
+      var result = new QuickSearchAction().execute(new QuickSearchInput().withSearchTerm("Public"));
+      assertThat(result.getResults()).hasSize(1);
+      assertThat(mapper.writeValueAsString(result)).doesNotContain(CANARY).contains("Public widget");
+
+      try(QuickSearchOpenSearchClient queryClient = new QuickSearchOpenSearchClient(config))
+      {
+         String request = queryClient.buildSearchQuery("Public", List.of(TEST_ENTITY_TABLE), List.of(fields), false).toJsonString();
+         assertThat(request).doesNotContain("searchableText", "fieldValues.description", "fieldValuesEnglish.description", "fieldValuesExact.personal", CANARY);
+         assertThat(request).contains("fieldValues.name");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Diagnostic document views must not reveal protected values retained in
+    ** old index documents, including analyzer copies and combined labels.
+    *******************************************************************************/
+   @Test
+   void testDiagnosticDocumentReadsOmitClassifiedValues() throws Exception
+   {
+      QContext.getQInstance().getTable(TEST_ENTITY_TABLE).getField("description")
+         .withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      OpenSearchDocument stale = new OpenSearchDocument().withRecordId("1").withSourceTable(TEST_ENTITY_TABLE)
+         .withRecordLabel(CANARY).withSearchableText(CANARY).withSubtitle(CANARY).withDisplayValues(Map.of("Secret", CANARY))
+         .withFieldValues(Map.of("name", "Public widget", "description", CANARY))
+         .withFieldValuesWord(Map.of("description", CANARY)).withFieldValuesEnglish(Map.of("description", CANARY))
+         .withFieldValuesExact(Map.of("description", CANARY));
+      ObjectMapper mapper = new ObjectMapper().registerModule(new JavaTimeModule());
+      var body = mapper.valueToTree(Map.of("found", true, "_source", stale,
+         "docs", List.of(Map.of("found", true, "_id", TEST_ENTITY_TABLE + ":1", "_source", stale))));
+      QuickSearchOpenSearchClient client = mock(QuickSearchOpenSearchClient.class);
+      when(client.getIndexName()).thenReturn("test");
+      QuickSearchDiagnosticsOperations ops = spy(new QuickSearchDiagnosticsOperations(client));
+      doReturn(new QuickSearchDiagnosticsOperations.RawResponse(200, body)).when(ops).raw(anyString(), anyString(), any());
+      assertThat(ops.getDocument(TEST_ENTITY_TABLE, "1").source().toString()).doesNotContain(CANARY).contains("Public widget");
+      assertThat(ops.getDocuments(TEST_ENTITY_TABLE, List.of("1")).get("1").toString()).doesNotContain(CANARY).contains("Public widget");
+   }
+
+
+
+   /*******************************************************************************
+    ** Recheck metadata at document construction, including primary keys and
+    ** untrusted preformatted labels, without depending on config priming.
+    *******************************************************************************/
+   @Test
+   void testTableClassificationAndProtectedPrimaryKey() throws Exception
+   {
+      QTableMetaData table = QContext.getQInstance().getTable(TEST_ENTITY_TABLE);
+      table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      table.withRecordLabelFields("description").withRecordLabelFormat("%s");
+      QRecord record = new QRecord().withValue("id", 1).withValue("name", "Public widget").withValue("description", CANARY);
+      record.setRecordLabel(CANARY);
+      OpenSearchDocument document = IndexingUtils.buildDocument(record, QuickSearchQBitContext.getDiscoveredTables().get(0));
+      assertThat(document.getRecordLabel()).isNull();
+      assertThat(document.getSearchableText()).isEqualTo("Public widget");
+      table.getField("id").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII));
+      assertThat(IndexingUtils.buildDocument(record, QuickSearchQBitContext.getDiscoveredTables().get(0))).isNull();
+   }
+}

src/test/java/com/kingsrook/qbits/quicksearch/processes/BasepullIndexStepTest.java

@@ -304,6 +304,26 @@ class BasepullIndexStepTest extends BaseQuickSearchTest
 
 
 
+   /*******************************************************************************
+    ** A successful basepull starts a new real-time failure counting window.
+    *******************************************************************************/
+   @Test
+   void testBasepull_resetsRealTimeErrorCount() throws QException
+   {
+      QRecord row = insertIndexRow(TEST_ENTITY_TABLE, true, null, 5);
+      new UpdateAction().execute(new UpdateInput().withTableName(QuickSearchIndex.TABLE_NAME)
+         .withRecord(new QRecord().withValue("id", row.getValue("id")).withValue("realTimeErrorCount", 7)));
+      insertTestEntities(2, Instant.now());
+
+      new BasepullIndexStep().run(new RunBackendStepInput(), new RunBackendStepOutput());
+
+      QueryInput input = new QueryInput();
+      input.setTableName(QuickSearchIndex.TABLE_NAME);
+      assertThat(new QueryAction().execute(input).getRecords().get(0).getValueInteger("realTimeErrorCount")).isZero();
+   }
+
+
+
    /*******************************************************************************
     ** Test: no discovered tables, no quickSearchIndex rows - step completes
     ** without calling the OpenSearch client.

src/test/java/com/kingsrook/qbits/quicksearch/processes/FullReindexStepTest.java

@@ -162,6 +162,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
          .withValue("enabled", enabled)
          .withValue("basepullIntervalMinutes", 5)
          .withValue("documentCount", documentCount)
+         .withValue("realTimeErrorCount", 7)
          .withValue("status", "ACTIVE")));
       new InsertAction().execute(insertInput);
    }
@@ -268,6 +269,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
    @Test
    void testAllTables_updatesIndexRowAfterSwap() throws QException
    {
+      insertIndexRow(true);
       insertTestEntities(2);
       when(mockClient.countDocumentsForTable(TEST_ENTITY_TABLE)).thenReturn(2L);
 
@@ -276,6 +278,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
       QRecord row = queryIndexRow();
       assertThat(row).isNotNull();
       assertThat(row.getValue("lastFullReindexTime")).isNotNull();
+      assertThat(row.getValueInteger("realTimeErrorCount")).isZero();
       assertThat(row.getValue("lastBasepullTime")).isNotNull();
       assertThat(row.getValueInteger("recordCount")).isEqualTo(2);
       assertThat(row.getValueInteger("documentCount")).isEqualTo(2);
@@ -449,6 +452,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
    @Test
    void testSingleTable_reconcilesInPlace_neverWipes() throws QException
    {
+      insertIndexRow(true);
       insertTestEntities(2);
 
       RunBackendStepOutput output = new RunBackendStepOutput();
@@ -471,6 +475,7 @@ class FullReindexStepTest extends BaseQuickSearchTest
       assertThat(runs.get(0).getValueInteger("recordsProcessed")).isEqualTo(2);
 
       assertThat(queryIndexRow().getValue("lastFullReindexTime")).isNotNull();
+      assertThat(queryIndexRow().getValueInteger("realTimeErrorCount")).isZero();
       assertThat(output.getValueInteger("recordsIndexed")).isEqualTo(2);
       assertThat(output.getValueBoolean("aliasSwapped")).isFalse();
    }

src/test/java/com/kingsrook/qbits/quicksearch/processes/IndexingUtilsTest.java

@@ -21,6 +21,8 @@ import java.time.Instant;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
@@ -33,8 +35,10 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
+import com.kingsrook.qbits.quicksearch.config.QuickSearchFieldRules;
 import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
 import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
 
@@ -45,6 +49,143 @@ import static org.assertj.core.api.Assertions.assertThat;
 class IndexingUtilsTest
 {
 
+   /*******************************************************************************
+    ** Document building requires field metadata to apply the index policy.
+    *******************************************************************************/
+   @BeforeEach
+   void setUpMetadata()
+   {
+      QInstance instance = new QInstance();
+      instance.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class));
+      instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS));
+      for(String tableName : List.of("customers", "person", "product", "order"))
+      {
+         QTableMetaData table = new QTableMetaData().withName(tableName).withBackendName("memory").withPrimaryKeyField("id")
+            .withField(new QFieldMetaData("id", QFieldType.INTEGER))
+            .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME));
+         for(String field : List.of("name", "description", "firstName", "lastName", "email", "statusId", "quantity", "notes", "sku"))
+         {
+            table.withField(new QFieldMetaData(field, QFieldType.STRING));
+         }
+         instance.addTable(table);
+      }
+      QContext.init(instance, new QSession());
+   }
+
+
+
+   /*******************************************************************************
+    ** Unsupported search fields must not erase labels or rebuild results.
+    *******************************************************************************/
+   @Test
+   void testUnprotectedBooleanAndBlobPreserveLabelAndResult()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable("product");
+      table.withField(new QFieldMetaData("active", QFieldType.BOOLEAN))
+         .withField(new QFieldMetaData("attachment", QFieldType.BLOB))
+         .withRecordLabelFields("name").withRecordLabelFormat("%s");
+      QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget")
+         .withValue("active", true).withValue("attachment", new byte[] { 1, 2 });
+      record.setRecordLabel("Widget label");
+      QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id")
+         .withSearchableFields(List.of("name", "active", "attachment"));
+
+      OpenSearchDocument document = IndexingUtils.buildDocument(record, config);
+      assertThat(document.getRecordLabel()).isEqualTo("Widget label");
+      assertThat(document.getSearchableText()).isEqualTo("Widget");
+      assertThat(document.getFieldValues()).containsExactlyEntriesOf(Map.of("name", "Widget"));
+      assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isFalse();
+      assertThat(IndexingUtils.filterResult(document, config)).isSameAs(document);
+
+      record.setRecordLabel(null);
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget");
+      config.withRecordLabelFields(List.of("name", "active")).withRecordLabelFormat("%s: %s");
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget: true");
+      table.withRecordLabelFields("name", "active").withRecordLabelFormat("%s: %s");
+      config.withRecordLabelFormat(null);
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget: true");
+   }
+
+
+
+   /*******************************************************************************
+    ** A protected field unrelated to the label must not erase a safe label.
+    *******************************************************************************/
+   @Test
+   void testUnrelatedProtectedFieldPreservesLabel()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable("product");
+      table.withRecordLabelFields("name").withRecordLabelFormat("%s");
+      table.getField("description").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget").withValue("description", "PHI-CANARY");
+      record.setRecordLabel("Widget label");
+      QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id")
+         .withSearchableFields(List.of("name", "description"));
+
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget label");
+      record.setRecordLabel(null);
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Widget");
+      assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isTrue();
+   }
+
+
+
+   /*******************************************************************************
+    ** Both table and configured label formats must exclude protected fields.
+    *******************************************************************************/
+   @Test
+   void testProtectedLabelFieldsOmitPreformattedAndComputedLabels()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable("person");
+      table.getField("email").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PII));
+      table.withRecordLabelFields("name", "email").withRecordLabelFormat("%s %s");
+      QRecord record = new QRecord().withValue("id", 1).withValue("name", "Public name").withValue("email", "PII-CANARY");
+      record.setRecordLabel("Public name PII-CANARY");
+      QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("person").withPrimaryKeyField("id")
+         .withSearchableFields(List.of("name", "email"))
+         .withRecordLabelFields(List.of("email")).withRecordLabelFormat("%s");
+
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isNull();
+      record.setRecordLabel(null);
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isNull();
+      table.withRecordLabelFields("name").withRecordLabelFormat("%s");
+      record.setRecordLabel("PII-CANARY");
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("Public name");
+      assertThat(record.getRecordLabel()).isEqualTo("PII-CANARY");
+      table.withRecordLabelFormat(null);
+      assertThat(IndexingUtils.buildDocument(record, config).getRecordLabel()).isEqualTo("1");
+   }
+
+
+
+   /*******************************************************************************
+    ** Hidden keys and timestamps remain usable unless they are protected.
+    *******************************************************************************/
+   @Test
+   void testHiddenPrimaryKeyAndTimestampAreNotProtected()
+   {
+      QTableMetaData table = QContext.getQInstance().getTable("product");
+      table.getField("id").withIsHidden(true);
+      table.getField("modifyDate").withIsHidden(true);
+      Instant timestamp = Instant.parse("2026-10-14T12:00:00Z");
+      QRecord record = new QRecord().withValue("id", 1).withValue("name", "Widget").withValue("modifyDate", timestamp);
+      QuickSearchableTableConfig config = new QuickSearchableTableConfig().withTableName("product").withPrimaryKeyField("id")
+         .withSearchableFields(List.of("name")).withBasepullTimestampField("modifyDate");
+
+      OpenSearchDocument document = IndexingUtils.buildDocument(record, config);
+      assertThat(document.getRecordId()).isEqualTo("1");
+      assertThat(document.getVersion()).isEqualTo(timestamp.toEpochMilli());
+      assertThat(QuickSearchFieldRules.hasProtectedPrimaryKey("product")).isFalse();
+      assertThat(QuickSearchFieldRules.requiresFieldFiltering("product")).isFalse();
+      table.getField("modifyDate").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.PHI));
+      assertThat(IndexingUtils.buildDocument(record, config).getVersion()).isNull();
+      table.getField("id").withIsSensitive(true);
+      assertThat(QuickSearchFieldRules.hasProtectedPrimaryKey("product")).isTrue();
+      assertThat(IndexingUtils.buildDocument(record, config)).isNull();
+   }
+
+
+
    /*******************************************************************************
     ** Test that normal input is trimmed and its case is preserved (the analyzer
     ** lower-cases at index and search time).
@@ -606,7 +747,7 @@ class IndexingUtilsTest
          .withSearchableFields(List.of("sku"))
          .withFieldIncludeLabels(Map.of("sku", true));
 
-      assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("sku: AB-1");
+      assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("Sku: AB-1");
 
       initContextWithPersonTable();
       assertThat(IndexingUtils.buildDocument(record, tableConfig).getSearchableText()).isEqualTo("SKU Code: AB-1");

src/test/java/com/kingsrook/qbits/quicksearch/publisher/SynchronousIndexEventPublisherTest.java

@@ -21,8 +21,13 @@ import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qbits.quicksearch.BaseQuickSearchTest;
 import com.kingsrook.qbits.quicksearch.QuickSearchableTableConfig;
 import com.kingsrook.qbits.quicksearch.opensearch.BulkIndexResult;
 import com.kingsrook.qbits.quicksearch.opensearch.OpenSearchDocument;
@@ -46,7 +51,7 @@ import static org.mockito.Mockito.when;
 /*******************************************************************************
  ** Tests for SynchronousIndexEventPublisher.
  *******************************************************************************/
-class SynchronousIndexEventPublisherTest
+class SynchronousIndexEventPublisherTest extends BaseQuickSearchTest
 {
    private QuickSearchOpenSearchClient    mockClient;
    private SynchronousIndexEventPublisher publisher;
@@ -61,6 +66,13 @@ class SynchronousIndexEventPublisherTest
    @BeforeEach
    void setUp()
    {
+      QContext.getQInstance().addTable(
+         new QTableMetaData().withName("orders").withPrimaryKeyField("id")
+            .withField(new QFieldMetaData("id", QFieldType.INTEGER))
+            .withField(new QFieldMetaData("orderNumber", QFieldType.STRING))
+            .withField(new QFieldMetaData("customerName", QFieldType.STRING)));
+      QContext.getQInstance().addTable(new QTableMetaData().withName("products").withPrimaryKeyField("id")
+         .withField(new QFieldMetaData("id", QFieldType.INTEGER)).withField(new QFieldMetaData("name", QFieldType.STRING)));
       mockClient = mock(QuickSearchOpenSearchClient.class);
 
       QuickSearchableTableConfig ordersConfig = new QuickSearchableTableConfig()