feat: data-driven Quick Search config, valid Add Search Index, global pause, cancel (1.1.0-RC.3) #47

Merged

KofTwentyTwo merged 3 commits from feature/quick-search-data-driven-config into develop 2026-10-12 21:10:32 +00:00

Conversation 0 Commits 3 Files Changed 75

+10367 -144

KofTwentyTwo commented 2026-10-12 20:56:06 +00:00 (Migrated from github.com)

Copy link

Copy source

For the release captain, for 1.1.0-RC.3 (or the next RC). Stacked on #42 (snake_case names), which is stacked on #41: merge those first; GitHub then retargets this PR to develop.

Why

Owner: "the 'Create Quick Search Index' screen isnt valid at all", and "make quick search fully data driven, and make it easy to add, remove, etc. and it should still self prime correctly on new stuff". Until now the searchable tables and fields came only from code, and the generic Create screen asked for a free-text table name, so a row created by hand was ignored or failed with "Table is no longer configured".

What changes

Data model (standard QQQ tables, permission-gated, audited at field level)

Table	Role
quick_search_index	The searchable-table registry. New source (CODE/OWNER) and ownerEdited; editable timestamp field (picker). Generic Create/Delete off. Searchable Fields and Recent Runs child lists on the record.
quick_search_index_field (new)	One row per field: weight, role (DISPLAY T1, TITLE, SUBTITLE, BODY, KEYWORD, EXACT), analyzer (PREFIX, WORD, ENGLISH, EXACT), includeInSnippet, includeLabel, sortOrder, source, ownerEdited. Validated: the field exists, is indexable (not hidden, not isSensitive, not password/blob/boolean), no duplicates.
quick_search_settings (new)	One row: the global pause (indexingPaused, pausedAt, pausedBy, pauseReason) and the last self-priming report.
quick_search_config_tombstone (new)	What an admin removed; priming never re-adds it.
quick_search_index_run	cancelRequested, cancelRequestedBy; status CANCELLED.

Precedence (records win; code only seeds) — QuickSearchConfigPrimer:

• New code tables and fields get rows at startup and their build is queued (PENDING_BUILD → built by the next basepull, BUILDING → ACTIVE).
• Rows nobody edited follow code (interval, timestamp field, field weight and label flag); enabled is set only on create.
• Rows an admin added or edited (ownerEdited) are never overwritten; tombstoned tables and fields are never re-added; priming never deletes.
• RC.2 rows get their field rows with no rebuild when the index already matches code.
• Without the config tables (DDL not added yet), or until priming succeeds, the code config is used exactly as before.

Effective config — QuickSearchConfigStore: indexing (real-time, basepull, reconcile, full reindex), search, the core record-search provider and the diagnostics all read it from the records (30 s cache, dropped at once by the tables' customizers), so changes need no redeploy. Runtime-added tables get core searchFields for the global search.

Self-priming of the index — basepull recreates a missing index and rebuilds the tables that had documents in it; an empty index is rebuilt. The Search Diagnostics widget (and QuickSearchStatus.bannerHtml() for hosts) shows "New: N tables added and building (…)", "Paused since … by …", and field changes waiting to apply.

Admin actions — standard streamed-ETL processes with review and result steps, run from table Actions (never the nav), gated by quick_search_admin:

• Add Search Index (the valid create): eligible-table picker (readable, not indexed, not qbit-internal/system/secret), interval, timestamp field, enabled. Review shows the estimated record count, the starting fields (record-label fields as Title, then text fields) and what is never indexed. Saving queues the first build (or builds at once when no scheduler runs basepull).
• Searchable Fields child list + Apply Search Index Changes (review previews added/removed/changed fields and records to read, then rebuilds).
• Disable / Remove Search Index with a "delete or keep documents" choice restated on the review step; Remove writes a tombstone.
• Cancel Index Run: cooperative flag checked between batches; the run ends CANCELLED with its counts (a cancelled all-tables rebuild keeps the old index).
• Pause / Resume Indexing: pauses real-time and scheduled indexing (a running rebuild stops after its batch) while search keeps serving; changes are kept as DEFERRED failed events and replayed after resume.

Roles and analyzers — TITLE fields make the hit label, SUBTITLE → QuickSearchResult.subtitle, DISPLAY → displayValues; KEYWORD/EXACT stay out of the snippet and match on their own; WORD/ENGLISH/EXACT get document objects and mapping templates added to existing indexes at startup (no mapping-version change). Plain tables produce identical documents and queries.

Record-screen audit fixes — run history and failed events: Create/Edit off (Delete kept), sections, record labels, pickers (incl. the table picker for failed events); Recent Runs on the index view; validation reportJson in a JSON code editor; Basepull/Full Reindex/Reconcile are Quick Search Index table actions, not nav children, and their table input is searchTableName (running them from the table no longer limited them to quick_search_index); process input steps carry HELP_TEXT + EDIT_FORM so Next renders their fields.

Upgrade note

docs/MIGRATION-1.1.md, "Data-driven configuration (1.1.0-RC.3)": precedence rules, self-priming, the admin actions, Liquibase-formatted PostgreSQL DDL (and MySQL notes) for the new tables/columns, behaviour changes and a checklist. CHANGELOG and README updated.

Tests

CI=true ./mvnw verify: 459 unit + 41 integration tests, 0 failures, JaCoCo gates met (86% instructions). New:

• QuickSearchConfigPrimerTest — seeding vs owner edits (precedence), new code field → rebuild, tombstones (no resurrection), RC.2 upgrade without rebuild, store records mode.
• QuickSearchConfigProcessesTest — create flow end to end (create → built → indexed), scheduler queue → basepull build, field edit → preview → reindex, disable stops (listener and basepull), remove is clean and not resurrected, global pause (deferred, steps skip, provider still serves) and resume replay, cancel mid-run (CANCELLED with counts), permissions on every process, roles shaping documents.
• QuickSearchConfigCustomizersTest — field validation (missing, duplicate, sensitive, password, weight, role, analyzer, move), ownerEdited marking, NEEDS_REINDEX, enable → build, pause stamping and deferred release, eligibility rules and pickers (incl. a table the user cannot read), system-table screens.
• DataDrivenConfigIntegrationTest (real OpenSearch) — fresh install primes and builds; Add Search Index makes a table searchable (sensitive field never indexed, real-time works); EXACT field edit + apply; disable hides; pause/resume; remove deletes documents; a deleted index is recreated and rebuilt.

Screenshots (demo host, Next UI RC.14, mock auth)

Add Search Index: only eligible tables
Review: estimated records, starting fields, never-indexed fields
Result
Index record with Searchable Fields and Recent Runs
Quick Search Index actions (no generic New)
Pause review
Paused home

Follow-ups

• kof22 adoption: KofTwentyTwo/Website-Backend PR (changeset 0031, RC.3 bump, home banner, grants).
• Tables added at runtime get Inspect Search Document / Reindex in Search record actions only after a restart (processes are produced at startup).

🤖 Generated with Claude Code

For the release captain, for **1.1.0-RC.3** (or the next RC). Stacked on #42 (snake_case names), which is stacked on #41: merge those first; GitHub then retargets this PR to `develop`. ## Why Owner: *"the 'Create Quick Search Index' screen isnt valid at all"*, and *"make quick search fully data driven, and make it easy to add, remove, etc. and it should still self prime correctly on new stuff"*. Until now the searchable tables and fields came only from code, and the generic Create screen asked for a free-text table name, so a row created by hand was ignored or failed with "Table is no longer configured". ## What changes **Data model (standard QQQ tables, permission-gated, audited at field level)** | Table | Role | |---|---| | `quick_search_index` | The searchable-table registry. New `source` (`CODE`/`OWNER`) and `ownerEdited`; editable timestamp field (picker). Generic Create/Delete off. Searchable Fields and Recent Runs child lists on the record. | | `quick_search_index_field` (new) | One row per field: `weight`, `role` (`DISPLAY` T1, `TITLE`, `SUBTITLE`, `BODY`, `KEYWORD`, `EXACT`), `analyzer` (`PREFIX`, `WORD`, `ENGLISH`, `EXACT`), `includeInSnippet`, `includeLabel`, `sortOrder`, `source`, `ownerEdited`. Validated: the field exists, is indexable (not hidden, not `isSensitive`, not password/blob/boolean), no duplicates. | | `quick_search_settings` (new) | One row: the global pause (`indexingPaused`, `pausedAt`, `pausedBy`, `pauseReason`) and the last self-priming report. | | `quick_search_config_tombstone` (new) | What an admin removed; priming never re-adds it. | | `quick_search_index_run` | `cancelRequested`, `cancelRequestedBy`; status `CANCELLED`. | **Precedence (records win; code only seeds)** — `QuickSearchConfigPrimer`: - New code tables and fields get rows at startup and their build is queued (`PENDING_BUILD` → built by the next basepull, `BUILDING` → `ACTIVE`). - Rows nobody edited follow code (interval, timestamp field, field weight and label flag); `enabled` is set only on create. - Rows an admin added or edited (`ownerEdited`) are never overwritten; tombstoned tables and fields are never re-added; priming never deletes. - RC.2 rows get their field rows with no rebuild when the index already matches code. - Without the config tables (DDL not added yet), or until priming succeeds, the code config is used exactly as before. **Effective config** — `QuickSearchConfigStore`: indexing (real-time, basepull, reconcile, full reindex), search, the core record-search provider and the diagnostics all read it from the records (30 s cache, dropped at once by the tables' customizers), so changes need no redeploy. Runtime-added tables get core `searchFields` for the global search. **Self-priming of the index** — basepull recreates a missing index and rebuilds the tables that had documents in it; an empty index is rebuilt. The Search Diagnostics widget (and `QuickSearchStatus.bannerHtml()` for hosts) shows "New: N tables added and building (…)", "Paused since … by …", and field changes waiting to apply. **Admin actions** — standard streamed-ETL processes with review and result steps, run from table Actions (never the nav), gated by `quick_search_admin`: - **Add Search Index** (the valid create): eligible-table picker (readable, not indexed, not qbit-internal/system/secret), interval, timestamp field, enabled. Review shows the estimated record count, the starting fields (record-label fields as Title, then text fields) and what is never indexed. Saving queues the first build (or builds at once when no scheduler runs basepull). - **Searchable Fields** child list + **Apply Search Index Changes** (review previews added/removed/changed fields and records to read, then rebuilds). - **Disable / Remove Search Index** with a "delete or keep documents" choice restated on the review step; Remove writes a tombstone. - **Cancel Index Run**: cooperative flag checked between batches; the run ends `CANCELLED` with its counts (a cancelled all-tables rebuild keeps the old index). - **Pause / Resume Indexing**: pauses real-time and scheduled indexing (a running rebuild stops after its batch) while search keeps serving; changes are kept as `DEFERRED` failed events and replayed after resume. **Roles and analyzers** — `TITLE` fields make the hit label, `SUBTITLE` → `QuickSearchResult.subtitle`, `DISPLAY` → `displayValues`; `KEYWORD`/`EXACT` stay out of the snippet and match on their own; `WORD`/`ENGLISH`/`EXACT` get document objects and mapping templates added to existing indexes at startup (no mapping-version change). Plain tables produce identical documents and queries. **Record-screen audit fixes** — run history and failed events: Create/Edit off (Delete kept), sections, record labels, pickers (incl. the table picker for failed events); Recent Runs on the index view; validation `reportJson` in a JSON code editor; Basepull/Full Reindex/Reconcile are Quick Search Index table actions, not nav children, and their table input is `searchTableName` (running them from the table no longer limited them to `quick_search_index`); process input steps carry `HELP_TEXT` + `EDIT_FORM` so Next renders their fields. ## Upgrade note `docs/MIGRATION-1.1.md`, "Data-driven configuration (1.1.0-RC.3)": precedence rules, self-priming, the admin actions, Liquibase-formatted PostgreSQL DDL (and MySQL notes) for the new tables/columns, behaviour changes and a checklist. CHANGELOG and README updated. ## Tests `CI=true ./mvnw verify`: **459 unit + 41 integration tests, 0 failures**, JaCoCo gates met (86% instructions). New: - `QuickSearchConfigPrimerTest` — seeding vs owner edits (precedence), new code field → rebuild, tombstones (no resurrection), RC.2 upgrade without rebuild, store records mode. - `QuickSearchConfigProcessesTest` — create flow end to end (create → built → indexed), scheduler queue → basepull build, field edit → preview → reindex, disable stops (listener and basepull), remove is clean and not resurrected, global pause (deferred, steps skip, provider still serves) and resume replay, cancel mid-run (CANCELLED with counts), permissions on every process, roles shaping documents. - `QuickSearchConfigCustomizersTest` — field validation (missing, duplicate, sensitive, password, weight, role, analyzer, move), ownerEdited marking, NEEDS_REINDEX, enable → build, pause stamping and deferred release, eligibility rules and pickers (incl. a table the user cannot read), system-table screens. - `DataDrivenConfigIntegrationTest` (real OpenSearch) — fresh install primes and builds; Add Search Index makes a table searchable (sensitive field never indexed, real-time works); EXACT field edit + apply; disable hides; pause/resume; remove deletes documents; a deleted index is recreated and rebuilt. ## Screenshots (demo host, Next UI RC.14, mock auth) | | | |---|---| | Add Search Index: only eligible tables |  | | Review: estimated records, starting fields, never-indexed fields |  | | Result |  | | Index record with Searchable Fields and Recent Runs |  | | Quick Search Index actions (no generic New) |  | | Pause review |  | | Paused home |  | ## Follow-ups - kof22 adoption: KofTwentyTwo/Website-Backend PR (changeset 0031, RC.3 bump, home banner, grants). - Tables added at runtime get Inspect Search Document / Reindex in Search record actions only after a restart (processes are produced at startup). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

Architecture review: coupling, cohesion, UI API

bug

Something isn't working

dependencies

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

enhancement

New feature or request

github_actions

Pull requests that update GitHub Actions code

good first issue

Good for newcomers

help wanted

Extra attention is needed

invalid

This doesn't seem right

java

Pull requests that update java code

question

Further information is requested

release:rc

Merging this develop PR cuts the next release candidate

secuurity

wontfix

This will not be worked on

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

forge-recovery (Forge Recovery) james.maes (James Maes)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-quick-search#47