Owner request (2026-10-12): Admin -> Quick Search should have tooling to validate search and interact with it for debugging. Built generically in the QBit, so every QQQ app using it gets the tools.
Placement (repo rule: processes are table/record actions, never left-nav). Every tool is a process with a tableName, so it shows in that table's Actions menu (query screen and record view), plus one widget. None is added to the admin app's children.
Tool
Process / widget
Lives on
Permission
Writes?
Search Playground
quickSearchPlayground
Quick Search Index (table action; selected rows pre-fill tables)
quickSearchDebug.hasAccess
no
Validate Search Index
quickSearchValidateIndex
Quick Search Index (table action) + optional cron (validationCronExpression)
quickSearchDebug.hasAccess
saves a report row only
Test Search Analyzer
quickSearchAnalyzeText
Quick Search Index (table action)
quickSearchDebug.hasAccess
no
Inspect Search Document
quickSearchInspect_<table>
every indexed table + Quick Search Failed Event (record actions, max 10)
quickSearchDebug.hasAccess
no
Reindex in Search
quickSearchReindexRecords_<table>
every indexed table + Quick Search Failed Event (record actions, max 500, confirm step)
quickSearchAdmin.hasAccess
index only
Search Diagnostics
widget quickSearchDiagnostics
admin app home (hosts with their own home add it by name)
quickSearchDebug.hasAccess
no
Interaction model. Looping tools (playground, analyzer, inspect) use steps prepare (backend) -> form (EDIT_FORM + HTML) -> run (backend). run sets overrideLastStepName("prepare"), so the next step is form again with the results under the fields: change the term, run again, no restart (the same mechanism bulk load uses; Next re-renders via screenInstance). Verified through RunProcessAction in tests. Results are HTML in {step}.html, rendered by Next's sanitized HTML block.
What each tool shows.
Playground, as the current user: the path per table (OpenSearch provider / core database search / not searched) and the reason (not indexed, disabled, no read permission, no core searchFields, serveCoreRecordSearch off, another provider, backing off N s after error X); the provider's own _msearch request timed, without starting a backoff; the real global search (RecordSearchAction) results and time; per indexed table, the production query with scores, highlights on every field, matched fields (named clauses; names change neither matching nor scoring), which hits the user can read and which the global search shows; the query DSL and raw response with unreadable hits redacted to their id; the term's tokens with one-character / longer-than-20 warnings; explain for a hit (table:id, only if the user can read it).
Validation: cluster reachable and health, per-index health, alias points at exactly one physical index, leftover {alias}-v* indexes from interrupted rebuilds; live mapping (version, keyword/text/date types, analyzers on searchableText/recordLabel and every fieldValues.*, the dynamic template, date/numeric detection off); analysis settings (both analyzers, edge n-gram 2..20, shards/replicas); per table: database rows vs documents (tolerance max(5, 1%)), a sample of lowest + highest primary keys checked present and current, a random sample of documents checked to still have a record, Quick Search Index row status/last run, waiting failed events. Findings are OK/INFO/WARN/FAIL with the fix. Report saved in the new quickSearchValidationRun table (HTML field for the record view, JSON). Database reads run as the system user, so counts compare like with like (the index holds every row whatever the record locks); the report holds counts, ids and field names, never record values. Single-flight per server.
Inspect: indexed document vs the document indexing would write now (same read: display values, translated possible values), field by field, versions; with a term, every gate of the global search for this record: term length, table enabled, read permission, record readable, document present and current, each token found among the document's indexed tokens (or why not: one char, > 20 chars, not in the text, only in the newer database values), explain, and rank against the provider's id window.
Reindex: rebuild selected records from the database (versioned external_gte, never overwrites newer), remove documents of records that no longer exist (existence checked as system), skip records the user cannot read; refuses disabled tables.
Analyzer: _analyze on the live index with both analyzers; with a term, whether every term token is among the text's indexed tokens.
Failure diagnostics: the provider (core record search) and QuickSearchAction record SEARCH_FAILED events (tables, duration, root error class, message, server) in the new quickSearchDiagnosticEvent table, written off-thread as the system user, capped at 120/min/server. The user's search term is never stored (only its length): end-user terms can contain personal data. The runtime also counts failures and table searches core ran itself during backoffs. The widget shows the provider state on this server, the last 10 failures from all servers, and the latest validation.
Safety. Read-only by default (only Reindex in Search writes, to the index, with confirm + admin permission). Every tool run is audited (TOOL_RUN / REINDEX events with user, tool, tables, term entered by the admin, duration, outcome; RATE_LIMITED refusals). Rate limit diagnosticRunsPerMinute (default 30) per user and tool. Raw responses redact hits the user cannot read; no credentials are ever in the DSL/response; HTML is escaped (highlight <em> becomes <mark>). Layout: one column capped at 72rem, long values wrap (no horizontal or inner scrollbars) from 360 to 3200 px.
Impersonation ("search as user X") is deliberately skipped. A fabricated session would not carry the real user's security key values, so results would mislead; the playground runs as the current user only and explains permission/lock effects in the path reasons and readable flags.
No QQQ core change.RecordSearchAction gives no per-table "served by" signal, so the playground diagnoses the path itself (mirroring claimsTable, changing no state) and runs the provider request directly plus the real core search. No 4.1 RC needed.
Coordination. The global-search failure fix is with another agent; this PR only adds recording around the provider's existing catch (one call) and a backoff counter in claimsTable. If that fix lands first I merge develop in.
Settings (all optional)
enableDiagnosticTools (true), enableRecordDiagnosticActions (true), validationCronExpression / validationCronTimeZoneId (same rules as reconcile: cron-capable scheduler, valid Quartz expression, zone), validationSampleSize (50, max 1000), diagnosticRunsPerMinute (30), debugPermissionRules (quickSearchDebug). Two new operational tables: quickSearchDiagnosticEvent, quickSearchValidationRun (purged with runHistoryRetentionDays). Migration notes in docs/MIGRATION-1.1.md.
Tests
DiagnosticsMetaDataTest (10): tables, processes, per-table actions, permissions, not in nav, widget on the app, cron schedule and rules, opt-outs, prefixes, and QInstanceValidator.revalidate on the produced instance (caught a hidden-field-in-section bug).
DiagnosticsOfflineTest (13): real client against a closed port: playground reports provider failure + core fallback honestly, validation saves a FAIL report, provider failure events (no term) and counters, recorder cap, rate limit refusal, path diagnosis branches, redaction, escaping, widget states, and the playground loop through RunProcessAction.
DiagnosticsIntegrationTest (7, Testcontainers OpenSearch): playground (paths, provider ids, core results, matched clauses, highlights, DSL, explain, tokens), analyzer, inspect current/stale/missing and every why-no-match reason, validation finding missing/stale/orphan/leftover index with mapping and settings OK and the saved report, reindex fixing stale/missing/orphan, and the inspect/reindex/scheduled-validate processes end to end.
Closes #35.
## Design
Owner request (2026-10-12): Admin -> Quick Search should have tooling to validate search and interact with it for debugging. Built generically in the QBit, so every QQQ app using it gets the tools.
**Placement (repo rule: processes are table/record actions, never left-nav).** Every tool is a process with a `tableName`, so it shows in that table's Actions menu (query screen and record view), plus one widget. None is added to the admin app's children.
| Tool | Process / widget | Lives on | Permission | Writes? |
|---|---|---|---|---|
| Search Playground | `quickSearchPlayground` | Quick Search Index (table action; selected rows pre-fill tables) | `quickSearchDebug.hasAccess` | no |
| Validate Search Index | `quickSearchValidateIndex` | Quick Search Index (table action) + optional cron (`validationCronExpression`) | `quickSearchDebug.hasAccess` | saves a report row only |
| Test Search Analyzer | `quickSearchAnalyzeText` | Quick Search Index (table action) | `quickSearchDebug.hasAccess` | no |
| Inspect Search Document | `quickSearchInspect_<table>` | every indexed table + Quick Search Failed Event (record actions, max 10) | `quickSearchDebug.hasAccess` | no |
| Reindex in Search | `quickSearchReindexRecords_<table>` | every indexed table + Quick Search Failed Event (record actions, max 500, confirm step) | `quickSearchAdmin.hasAccess` | index only |
| Search Diagnostics | widget `quickSearchDiagnostics` | admin app home (hosts with their own home add it by name) | `quickSearchDebug.hasAccess` | no |
**Interaction model.** Looping tools (playground, analyzer, inspect) use steps `prepare` (backend) -> `form` (EDIT_FORM + HTML) -> `run` (backend). `run` sets `overrideLastStepName("prepare")`, so the next step is `form` again with the results under the fields: change the term, run again, no restart (the same mechanism bulk load uses; Next re-renders via `screenInstance`). Verified through `RunProcessAction` in tests. Results are HTML in `{step}.html`, rendered by Next's sanitized HTML block.
**What each tool shows.**
1. *Playground*, as the current user: the path per table (OpenSearch provider / core database search / not searched) and the reason (not indexed, disabled, no read permission, no core searchFields, serveCoreRecordSearch off, another provider, backing off N s after error X); the provider's own `_msearch` request timed, without starting a backoff; the real global search (`RecordSearchAction`) results and time; per indexed table, the production query with scores, highlights on every field, matched fields (named clauses; names change neither matching nor scoring), which hits the user can read and which the global search shows; the query DSL and raw response with unreadable hits redacted to their id; the term's tokens with one-character / longer-than-20 warnings; `explain` for a hit (`table:id`, only if the user can read it).
2. *Validation*: cluster reachable and health, per-index health, alias points at exactly one physical index, leftover `{alias}-v*` indexes from interrupted rebuilds; live mapping (version, keyword/text/date types, analyzers on `searchableText`/`recordLabel` and every `fieldValues.*`, the dynamic template, date/numeric detection off); analysis settings (both analyzers, edge n-gram 2..20, shards/replicas); per table: database rows vs documents (tolerance max(5, 1%)), a sample of lowest + highest primary keys checked present and current, a random sample of documents checked to still have a record, Quick Search Index row status/last run, waiting failed events. Findings are OK/INFO/WARN/FAIL with the fix. Report saved in the new `quickSearchValidationRun` table (HTML field for the record view, JSON). Database reads run as the system user, so counts compare like with like (the index holds every row whatever the record locks); the report holds counts, ids and field names, never record values. Single-flight per server.
3. *Inspect*: indexed document vs the document indexing would write now (same read: display values, translated possible values), field by field, versions; with a term, every gate of the global search for this record: term length, table enabled, read permission, record readable, document present and current, each token found among the document's indexed tokens (or why not: one char, > 20 chars, not in the text, only in the newer database values), explain, and rank against the provider's id window.
4. *Reindex*: rebuild selected records from the database (versioned `external_gte`, never overwrites newer), remove documents of records that no longer exist (existence checked as system), skip records the user cannot read; refuses disabled tables.
5. *Analyzer*: `_analyze` on the live index with both analyzers; with a term, whether every term token is among the text's indexed tokens.
6. *Failure diagnostics*: the provider (core record search) and `QuickSearchAction` record `SEARCH_FAILED` events (tables, duration, root error class, message, server) in the new `quickSearchDiagnosticEvent` table, written off-thread as the system user, capped at 120/min/server. **The user's search term is never stored** (only its length): end-user terms can contain personal data. The runtime also counts failures and table searches core ran itself during backoffs. The widget shows the provider state on this server, the last 10 failures from all servers, and the latest validation.
**Safety.** Read-only by default (only Reindex in Search writes, to the index, with confirm + admin permission). Every tool run is audited (`TOOL_RUN` / `REINDEX` events with user, tool, tables, term entered by the admin, duration, outcome; `RATE_LIMITED` refusals). Rate limit `diagnosticRunsPerMinute` (default 30) per user and tool. Raw responses redact hits the user cannot read; no credentials are ever in the DSL/response; HTML is escaped (highlight `<em>` becomes `<mark>`). Layout: one column capped at 72rem, long values wrap (no horizontal or inner scrollbars) from 360 to 3200 px.
**Impersonation ("search as user X") is deliberately skipped.** A fabricated session would not carry the real user's security key values, so results would mislead; the playground runs as the current user only and explains permission/lock effects in the path reasons and readable flags.
**No QQQ core change.** `RecordSearchAction` gives no per-table "served by" signal, so the playground diagnoses the path itself (mirroring `claimsTable`, changing no state) and runs the provider request directly plus the real core search. No 4.1 RC needed.
**Coordination.** The global-search failure fix is with another agent; this PR only adds recording around the provider's existing catch (one call) and a backoff counter in `claimsTable`. If that fix lands first I merge develop in.
## Settings (all optional)
`enableDiagnosticTools` (true), `enableRecordDiagnosticActions` (true), `validationCronExpression` / `validationCronTimeZoneId` (same rules as reconcile: cron-capable scheduler, valid Quartz expression, zone), `validationSampleSize` (50, max 1000), `diagnosticRunsPerMinute` (30), `debugPermissionRules` (`quickSearchDebug`). Two new operational tables: `quickSearchDiagnosticEvent`, `quickSearchValidationRun` (purged with `runHistoryRetentionDays`). Migration notes in `docs/MIGRATION-1.1.md`.
## Tests
- `DiagnosticsMetaDataTest` (10): tables, processes, per-table actions, permissions, not in nav, widget on the app, cron schedule and rules, opt-outs, prefixes, and `QInstanceValidator.revalidate` on the produced instance (caught a hidden-field-in-section bug).
- `DiagnosticsOfflineTest` (13): real client against a closed port: playground reports provider failure + core fallback honestly, validation saves a FAIL report, provider failure events (no term) and counters, recorder cap, rate limit refusal, path diagnosis branches, redaction, escaping, widget states, and the playground loop through `RunProcessAction`.
- `DiagnosticsIntegrationTest` (7, Testcontainers OpenSearch): playground (paths, provider ids, core results, matched clauses, highlights, DSL, explain, tokens), analyzer, inspect current/stale/missing and every why-no-match reason, validation finding missing/stale/orphan/leftover index with mapping and settings OK and the saved report, reindex fixing stale/missing/orphan, and the inspect/reindex/scheduled-validate processes end to end.
- `CI=true ./mvnw verify`: 400 unit + 32 integration tests green, JaCoCo gate met.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Independent review (code-reviewer agent) found 1 high + 3 medium + 1 low issue; all fixed in be5df8e:
High: the user's term reached errorMessage through the client's wrap message. Failure events now redact the term and its words from message and detail; tests assert the stored event never contains it.
Playground: hits you cannot read are now counted only (no id, score, matched fields). They are removed from the raw response along with the total and max score, and provider ids are filtered to readable records.
Inspect / Reindex run steps take targets from the server-side process records, not client values. Failed-event targets must be indexed tables, and reindex skips invalid ids instead of deleting documents for them.
History tables disable insert, update and delete for users and the API, so report HTML is always the QBit's own output.
A validation counts as scheduled only without a user session.
CI=true ./mvnw verify passes: all unit tests + 33 integration tests, coverage met.
Independent review (code-reviewer agent) found 1 high + 3 medium + 1 low issue; all fixed in be5df8e:
- **High:** the user's term reached `errorMessage` through the client's wrap message. Failure events now redact the term and its words from message and detail; tests assert the stored event never contains it.
- Playground: hits you cannot read are now counted only (no id, score, matched fields). They are removed from the raw response along with the total and max score, and provider ids are filtered to readable records.
- Inspect / Reindex run steps take targets from the server-side process records, not client values. Failed-event targets must be indexed tables, and reindex skips invalid ids instead of deleting documents for them.
- History tables disable insert, update and delete for users and the API, so report HTML is always the QBit's own output.
- A validation counts as scheduled only without a user session.
`CI=true ./mvnw verify` passes: all unit tests + 33 integration tests, coverage met.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #35.
Design
Owner request (2026-10-12): Admin -> Quick Search should have tooling to validate search and interact with it for debugging. Built generically in the QBit, so every QQQ app using it gets the tools.
Placement (repo rule: processes are table/record actions, never left-nav). Every tool is a process with a
tableName, so it shows in that table's Actions menu (query screen and record view), plus one widget. None is added to the admin app's children.quickSearchPlaygroundquickSearchDebug.hasAccessquickSearchValidateIndexvalidationCronExpression)quickSearchDebug.hasAccessquickSearchAnalyzeTextquickSearchDebug.hasAccessquickSearchInspect_<table>quickSearchDebug.hasAccessquickSearchReindexRecords_<table>quickSearchAdmin.hasAccessquickSearchDiagnosticsquickSearchDebug.hasAccessInteraction model. Looping tools (playground, analyzer, inspect) use steps
prepare(backend) ->form(EDIT_FORM + HTML) ->run(backend).runsetsoverrideLastStepName("prepare"), so the next step isformagain with the results under the fields: change the term, run again, no restart (the same mechanism bulk load uses; Next re-renders viascreenInstance). Verified throughRunProcessActionin tests. Results are HTML in{step}.html, rendered by Next's sanitized HTML block.What each tool shows.
_msearchrequest timed, without starting a backoff; the real global search (RecordSearchAction) results and time; per indexed table, the production query with scores, highlights on every field, matched fields (named clauses; names change neither matching nor scoring), which hits the user can read and which the global search shows; the query DSL and raw response with unreadable hits redacted to their id; the term's tokens with one-character / longer-than-20 warnings;explainfor a hit (table:id, only if the user can read it).{alias}-v*indexes from interrupted rebuilds; live mapping (version, keyword/text/date types, analyzers onsearchableText/recordLabeland everyfieldValues.*, the dynamic template, date/numeric detection off); analysis settings (both analyzers, edge n-gram 2..20, shards/replicas); per table: database rows vs documents (tolerance max(5, 1%)), a sample of lowest + highest primary keys checked present and current, a random sample of documents checked to still have a record, Quick Search Index row status/last run, waiting failed events. Findings are OK/INFO/WARN/FAIL with the fix. Report saved in the newquickSearchValidationRuntable (HTML field for the record view, JSON). Database reads run as the system user, so counts compare like with like (the index holds every row whatever the record locks); the report holds counts, ids and field names, never record values. Single-flight per server.external_gte, never overwrites newer), remove documents of records that no longer exist (existence checked as system), skip records the user cannot read; refuses disabled tables._analyzeon the live index with both analyzers; with a term, whether every term token is among the text's indexed tokens.QuickSearchActionrecordSEARCH_FAILEDevents (tables, duration, root error class, message, server) in the newquickSearchDiagnosticEventtable, written off-thread as the system user, capped at 120/min/server. The user's search term is never stored (only its length): end-user terms can contain personal data. The runtime also counts failures and table searches core ran itself during backoffs. The widget shows the provider state on this server, the last 10 failures from all servers, and the latest validation.Safety. Read-only by default (only Reindex in Search writes, to the index, with confirm + admin permission). Every tool run is audited (
TOOL_RUN/REINDEXevents with user, tool, tables, term entered by the admin, duration, outcome;RATE_LIMITEDrefusals). Rate limitdiagnosticRunsPerMinute(default 30) per user and tool. Raw responses redact hits the user cannot read; no credentials are ever in the DSL/response; HTML is escaped (highlight<em>becomes<mark>). Layout: one column capped at 72rem, long values wrap (no horizontal or inner scrollbars) from 360 to 3200 px.Impersonation ("search as user X") is deliberately skipped. A fabricated session would not carry the real user's security key values, so results would mislead; the playground runs as the current user only and explains permission/lock effects in the path reasons and readable flags.
No QQQ core change.
RecordSearchActiongives no per-table "served by" signal, so the playground diagnoses the path itself (mirroringclaimsTable, changing no state) and runs the provider request directly plus the real core search. No 4.1 RC needed.Coordination. The global-search failure fix is with another agent; this PR only adds recording around the provider's existing catch (one call) and a backoff counter in
claimsTable. If that fix lands first I merge develop in.Settings (all optional)
enableDiagnosticTools(true),enableRecordDiagnosticActions(true),validationCronExpression/validationCronTimeZoneId(same rules as reconcile: cron-capable scheduler, valid Quartz expression, zone),validationSampleSize(50, max 1000),diagnosticRunsPerMinute(30),debugPermissionRules(quickSearchDebug). Two new operational tables:quickSearchDiagnosticEvent,quickSearchValidationRun(purged withrunHistoryRetentionDays). Migration notes indocs/MIGRATION-1.1.md.Tests
DiagnosticsMetaDataTest(10): tables, processes, per-table actions, permissions, not in nav, widget on the app, cron schedule and rules, opt-outs, prefixes, andQInstanceValidator.revalidateon the produced instance (caught a hidden-field-in-section bug).DiagnosticsOfflineTest(13): real client against a closed port: playground reports provider failure + core fallback honestly, validation saves a FAIL report, provider failure events (no term) and counters, recorder cap, rate limit refusal, path diagnosis branches, redaction, escaping, widget states, and the playground loop throughRunProcessAction.DiagnosticsIntegrationTest(7, Testcontainers OpenSearch): playground (paths, provider ids, core results, matched clauses, highlights, DSL, explain, tokens), analyzer, inspect current/stale/missing and every why-no-match reason, validation finding missing/stale/orphan/leftover index with mapping and settings OK and the saved report, reindex fixing stale/missing/orphan, and the inspect/reindex/scheduled-validate processes end to end.CI=true ./mvnw verify: 400 unit + 32 integration tests green, JaCoCo gate met.🤖 Generated with Claude Code
Independent review (code-reviewer agent) found 1 high + 3 medium + 1 low issue; all fixed in
be5df8e:errorMessagethrough the client's wrap message. Failure events now redact the term and its words from message and detail; tests assert the stored event never contains it.CI=true ./mvnw verifypasses: all unit tests + 33 integration tests, coverage met.🤖 Generated with Claude Code