The index was created without number_of_replicas, so OpenSearch's default
of 1 left an unassignable replica on single-node clusters: the alias was
yellow and quick_search.opensearch reported DEGRADED.
- New config indexReplicas (default null = automatic:
min(maxAutoReplicas, data nodes - 1)) and maxAutoReplicas (default 1).
Zero config gives 0 replicas on one node and 1 on larger clusters.
- New physical indexes get the target; at startup existing indexes are
reconciled with PUT /<index>/_settings when the value differs
(idempotent, logged, failure tolerant, never blocks startup). Indexes
with auto_expand_replicas and OpenSearch Serverless are left alone.
- Status snapshot and quick_search.opensearch health details gain
unassignedReplicaShards, targetReplicas and replicaSettingsFailed.
Yellow on a single node with every primary active is UP with a note;
yellow with an inactive primary or on multiple nodes stays DEGRADED,
red stays DOWN.
- Unit tests (stub server), health/status tests and a Testcontainers
integration test; OpenAPI fragment regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Regenerated against UI API 1.6.0: every operation has an operationId and
the standard error responses (BasicErrorResponseV1: error, code, message,
fieldErrors, traceId); the extension's own error schema is kept but
deprecated; hand-written error bodies go through QJavalinUtils so they carry
code/message/traceId. Extension MINOR bumped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host-neutral help (deployment-specific facts removed), registered with
HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds help/quick-search.yaml (QQQ help bundle) applied by QuickSearchHelp, a
tables guardrail test, and a seeded-defaults test proving the primer fills
the config tables on a fresh backend.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the record-search provider (keep score/highlight/total hits and the failure diagnostics), CHANGELOG and README; requires QQQ 4.1.0-RC.4+ (develop is on RC.6).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Secured-data spec section 15: Quick Search must never put classified
values in the shared search index, snippets or results.
QuickSearchFieldRules decides it in one place on QQQ 4.1.0-RC.6:
isProtected is true for fields classified PII or PHI (field, else the
table's classification), fields with a field security lock, sensitive
fields and masked/password types. Such fields are never indexable;
explicit configs drop them with a warning (never the value); a table
whose primary key is protected (for example a PHI table) is not indexed
at all. Record labels built from protected fields fall back, the
version field is skipped when protected, and results from tables with
protected fields are rebuilt from allowed values only, so documents
indexed before this change cannot leak through labels or snippets.
Tables with only harmless non-indexable fields (booleans, blobs) keep
their labels and are not result-filtered.
A canary test proves a PHI and a PII marker never reach a captured
index request, result or snippet. RC.7 follow-up: also honour core
ClassifiedValues.isClassified (qqq#1182) and FieldAccessHelper lock
tiers (qqq#1172).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
realTimeErrorCount was only ever written as 0, so the status screen,
diagnostics and the status API always said there were no real-time
errors. When the change listener records failed events it now adds
their number to the table's count (after commit, serialized in the JVM,
never failing the user's write). A successful basepull or full reindex
resets it; the field's help text says so.
Fixes#53
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts no longer need QuickSearchOpenSearchClient or a copy of this qbit's
diagnostics. The new public api package holds:
- QuickSearchStatus, an immutable snapshot: overall state, cluster
reachability, per-table index counts, backlog, last sync, last error
and checkedAt;
- QuickSearchStatusService.get(), bounded and cached for five seconds,
built on the existing diagnostics;
- QuickSearchApi, with the app name and home href, so hosts link to the
Quick Search home instead of duplicating it.
The same data is served as UI API extension operation
GET /qqq/v1/qbit/quick-search/status (getQuickSearchStatus), gated by the
admin permission. The fragment is committed and checked by a contract
test. The core baseline becomes 4.1.0-RC.6.
Closes#50 (architecture review QRun-IO/qqq#1150, finding F10).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The searchable tables and fields are now standard QQQ records:
quick_search_index (the registry; new source and ownerEdited columns) and
the new quick_search_index_field (weight, role, analyzer, snippet and
label flags). Indexing, search, the record-search provider and the
diagnostics read the effective config from them through
QuickSearchConfigStore (30 s cache, dropped by the tables' customizers),
so changes need no redeploy.
Code (@QuickSearchable, searchableTables) only seeds the records:
QuickSearchConfigPrimer adds missing tables and fields at startup and
queues their builds, updates rows nobody edited, never overwrites rows an
admin edited (ownerEdited) and never re-adds what an admin removed
(quick_search_config_tombstone). Basepull builds queued tables
(PENDING_BUILD, BUILDING) and recreates a missing index; an empty index
is rebuilt. Without the config tables the code config is used as before.
The generic Create and Delete of quick_search_index are replaced by
streamed-ETL processes with review steps: Add Search Index (eligible
readable tables only, estimated record count, default fields, sensitive
and hidden fields never indexed), Apply Search Index Changes (previews
field changes, then rebuilds), Disable / Remove Search Index (delete or
keep documents), Cancel Index Run (cooperative flag between batches; runs
end CANCELLED with counts) and Pause / Resume Indexing (global pause in
quick_search_settings; search keeps serving; changes are deferred and
replayed). Field roles shape documents (TITLE label, SUBTITLE, DISPLAY
values, KEYWORD and EXACT outside the snippet) and WORD, ENGLISH and EXACT
analyzers get their own document objects and mapping templates.
Config tables are audited at field level and permission-gated; the
processes need quick_search_admin. Upgrade notes and DDL:
docs/MIGRATION-1.1.md, "Data-driven configuration".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner rule (2026-10-12): "table names should always be lowercase and use
_, and qbits should always be prefixed with their name". For 1.1.0-RC.3,
metadata only (the QBit ships no DDL):
- tables: quick_search_index, quick_search_index_run,
quick_search_failed_event, quick_search_diagnostic_event,
quick_search_validation_run (were camelCase)
- possible-value sources: quick_search_index, quick_search_searchable_table
- permission bases: quick_search_admin, quick_search_debug; table
permissions follow the table names
- applyTablePrefix: a host prefix is snake_cased and joined with "_"
(myApp -> my_app_quick_search_index); processes, widgets and apps keep
applyPrefix
- @Deprecated LEGACY_* constants keep the old names for upgrade scripts
- docs/MIGRATION-1.1.md: upgrade steps, ALTER TABLE for camelCase
databases, UPDATEs for audit_table, saved views and reports, permission
grants; no reindex needed
- QuickSearchTableNamesTest covers new names, legacy constants and prefixing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds QuickSearchDeveloperPanelProvider (id quickSearch, order 400) for
QQQ's record developer view, registered by the QBit component producer
QuickSearchDeveloperPanelsMetaDataProducer. Shown on indexed tables to
users who may read Quick Search Index: the index row, latest index run,
the record's indexed document vs the database (RecordInspector; debug
permission), failed index events, the Inspect / Reindex record actions
the user may run, and admin links. An unreachable OpenSearch yields a
warning block, not a failed panel.
Builds against QQQ 4.1.0-RC.4 (developer panel contract, QRun-IO/qqq#1076).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(#24): recover a stuck REBUILDING status after a killed reindex
A full reindex now refuses to start while another is alive, judged by a per-page heartbeat on its FULL_REINDEX run records. A run stale for fullReindexStaleMinutes (default 30) is recovered by the next basepull or full reindex: its records become FAILED, its tables ACTIVE, and its captured deletes PENDING for basepull.
Closes#24
* fix(#24): keep a slow captured-delete drain from touching a newer run's rows
The drain now heartbeats every batch, reads only rows up to the newest one at its start and stops once a recovery takes the run over; recovery marks dead runs FAILED last, basepull skips replayed deletes of re-created records, and the heartbeat is best effort and FULL_REINDEX only.
Refs #24
* fix(#24): fail recovery when a dead run cannot be marked FAILED
Recovery now writes FAILED with an update that must succeed for every stale run record, so a new full reindex refuses to start rather than run beside a misjudged live one. Tests pin each heartbeat point and the unreadable-takeover branch.
Refs #24
Keeps develop's scheduling section and applies its cron rules (cron-capable
scheduler, valid Quartz expression, time zone ID) to validationCronExpression
through one shared check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds diagnostic tools so every QQQ app using the QBit can validate its
search and debug it from the admin UI. All are table or record actions
(processes with a tableName, never added to the app nav), gated by
permissions, rate-limited per user and tool, recorded as diagnostic events,
and read-only except Reindex in Search.
- Search Playground (Quick Search Index): runs a term as the current user
the way the global search does; per-table path and reason (OpenSearch
provider, core database search, not searched), the provider's request
timed, the global search results, the production query's scores,
highlights and matched clauses, readable / shown flags, query DSL and raw
response (unreadable hits redacted), term tokens, explain for a hit.
The form loops back so a user can search again.
- Validate Search Index (Quick Search Index; schedulable with
validationCronExpression): cluster and index health, alias, leftover
physical indexes, live mapping and analysis settings, per-table counts,
sampled missing / stale documents and orphaned documents, index row status
and failed events. Reports kept in the new quickSearchValidationRun table.
- Test Search Analyzer: _analyze with the index's analyzers, and whether a
term's tokens are among a text's indexed tokens.
- Inspect Search Document and Reindex in Search on every indexed table and
on quickSearchFailedEvent rows: database vs index diff, and why a record
does or does not match a term (token-level reasons, explain, rank).
- Search failure diagnostics: provider and QuickSearchAction failures are
recorded as SEARCH_FAILED events (never the term, only its length), with
runtime counters; the Search Diagnostics widget shows provider state,
recent failures from every server and the latest validation.
- New settings: enableDiagnosticTools, enableRecordDiagnosticActions,
validationCronExpression, validationCronTimeZoneId, validationSampleSize,
diagnosticRunsPerMinute, debugPermissionRules (quickSearchDebug).
- History purge covers the two new tables.
Tests: unit (metadata incl. QInstanceValidator, offline tool behaviour
against a closed port, the playground loop through RunProcessAction) and a
Testcontainers OpenSearch integration test for every tool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(#29): validate cron zone, warn when basepull is off, document schedulers
A reconcileCronExpression without reconcileCronTimeZoneId passed the
QBit's validation, then QQQ's QInstanceValidator rejected the reconcile
process schedule at startup ("a cron schedule must specify a
cronTimeZoneId"); QQQ has no default zone. validate() now applies the
same checks QQQ does (Quartz CronExpression.validateExpression, a zone
from TimeZone.getAvailableIDs) and requires the zone, naming both
fields. It also rejects a cron on a scheduler in the instance that does
not support cron: QQQ's SimpleScheduler runs fixed intervals only and
fails when the schedule manager starts.
Config-driven tables without their basepull timestamp field now log the
same startup warning as annotated tables, once per table, through one
shared helper. A field set to null is an opt-out and stays quiet.
README and MIGRATION-1.0 document that schedules run only when the host
starts through QApplicationLauncher (qqq-middleware-javalin), and that a
reconcile cron needs a Quartz scheduler and a time zone ID.
Closes#29
* docs(producer): keep develop's hidden-and-password Javadoc after the merge
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(#26): fix migration guide and README gaps
The migration guide now names the permissions a host must grant and has
a schema section for the 0.x to 1.0 table changes, including the new
quickSearchFailedEvent table and the status value lengths. The
plaintext credentials setter is spelled withAllowPlaintextCredentials,
and the README and PRD point at 1.0.0-RC.2 as the current release.
Closes#26
* docs(#26): name each table in the migration schema step
The schema step now says which table each STRING to TEXT change belongs
to and includes the required, unique quickSearchIndex.tableName, matching
the schema section.
Refs #26
* docs(readme): name 1.1.0-RC.1 as the current release candidate
Maven Central's newest is 1.1.0-RC.1 (develop is the 1.1 line). Keep the
1.0 line's 1.0.0-RC.2 for hosts that stay on 1.0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
QuickSearchRecordSearchProvider now fills the QQQ 4.1.0-RC.4 record
search contract: each hit's relevance score, a plain-text highlight of
the matching fieldValues.* fields (OpenSearch highlighting with
private-use markers, converted to text plus match ranges), each table's
total hits (totalCountByTable, counted to 10,000), and the request
offset (from). RECENT sort keeps relevance order: the index holds no
record modify dates. New client method searchRecordsPerTable;
searchRecordIdsPerTable delegates to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implements QQQ core's RecordSearchProviderInterface so the Next UI global
search (POST /qqq/v1/search) is served from the Quick Search index for the
tables this QBit indexes. The provider returns record ids only; core re-reads
them through QueryAction, so table permissions, record security locks and
labels stay core's job. One bounded _msearch per call, with a short backoff
after a failure during which core falls back to its own LIKE search.
Builds against the released QQQ 4.1.0-RC.2, the first release that carries
the SPI (replaces the 4.1.0-SNAPSHOT pin from PR #32).
Re-signed from PR #32 (originally 75dc2b3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(#14): replay deletes captured during a full reindex
While a full reindex rebuilds, the listener records each delete as an AWAITING_REINDEX failed event (the index row is REBUILDING), and the reindex applies them through the alias after the swap.
Closes#14
* fix(#14): read rebuild status before publish, harden drain
The listener reads REBUILDING before publishing deletes, drift detection re-reads the row before writing, and the drain pages past rows it cannot remove and skips records re-created with the same key.
Refs #14
* fix(#14): judge drift on the re-read row, isolate drain batches
Drift detection compares the configured fields against the freshly read row, and a captured-delete batch that fails no longer stops the later batches.
Refs #14
* fix(#16): page search results by accessible hits
With record locks on, offset now skips accessible results, read in doubling
batches from position 0, and per-table mode honors offset in each table.
Every OpenSearch request keeps from + size within the 10,000 result window.
Closes#16
* fix(#16): bound the locked search scan per request
One search now shares a budget of 10,000 raw hits and 32 OpenSearch round
trips across every table it scans, so locked-out hits cannot amplify work.
Refs #16
* fix(#16): give each table its own scan budget and cap batches
Each locked scan now has its own budget of 10,000 raw hits in batches of at most 1,000, deduplicates hits and pins batches to one shard copy, and reports an exact total once every hit is read.
Refs #16
* fix(#16): share the per-table scan budget across tables
Each of T tables now gets an equal share of the scan budget, so one call reads at most max(10,000, T x 2 x limitPerTable) raw hits over max(32, 2T) requests.
Refs #16
* fix(#16): end paging when a table's scan share runs out
A spent scan budget now ends paging like the result window, so hasMore turns false while the total stays a lower bound.
Refs #16
* fix(#15): harden transport config validation
Warn when tls.hostnameVerification is off, treat empty resolved secrets and paths as missing with the field named, and reject credentials in opensearchUrl.
Closes#15
* fix(#15): never echo literals from secret fallback chains
Only a lone ${env.X} or ${prop.X} reference is named in resolution errors, and a happy-path test guards resolved BASIC credentials.
Refs #15
* docs: add 1.0 PRD, deep-dive audit, plan, and session state
* build: target qqq 4.1.0-RC.1, opensearch-java 3.10.0, wrapper 3.9.11, real JaCoCo gate
* feat: add 1.0 config surface, instance-scoped runtime, and deprecated static facade
* feat: implement the 4.1 QBit contract with a record-change listener and failed-event capture
* feat: opensearch client with auth modes, TLS, timeouts, mapping v2, aliases, versioned bulk writes
* feat: trustworthy indexing steps and permission-aware bounded search
* docs: rewrite README for 1.0, add CHANGELOG and migration guide
* feat: table picker and result screens for the maintenance processes; close out 1.0 planning docs
* docs: add AGENTS.md so Codex shares the CLAUDE.md guidance
* fix: keep the live index after an alias swap, atomic legacy upgrade, exact paging under record locks
From the PR #10 review: never delete the new physical index once the alias points at it, upgrade a legacy concrete index in one aliases request, base hasMore on raw hits so filtered pages stay reachable, and deduplicate requested table names.
* docs: explain the hasMore trade-off under record locks
* docs: session state for the RC.1 hand-off
Add SearchableFieldConfig and SearchableTableConfig classes that let
host apps declare searchable tables via config instead of requiring
annotations on entity classes. Both annotation and config paths merge
during produce(). Includes record label override, null-safe formatting,
within-source duplicate detection, and updated validation.
Add CONTRIBUTING.md, CODE_OF_CONDUCT.md, CODE_STYLE.md, SECURITY.md.
Copy checkstyle config from qbit-webhooks. Switch LICENSE to AGPL v3.0
to match QRun-IO org standard. Update .gitignore to standard template.
Rewrite README following qbit-crm pattern with badges and full docs.
Set version to 0.1.0-SNAPSHOT for develop branch.
james.maesandClaude Opus 5.5