From b90b76e93e8210e5c5e61b6f47de043d4b024f0f Mon Sep 17 00:00:00 2001 From: James Maes Date: Fri, 9 Oct 2026 19:23:10 -0500 Subject: [PATCH 1/2] fix(#29): validate cron zone, warn when basepull is off, document schedulers A reconcileCronExpression without reconcileCronTimeZoneId passed the QBit's validation, then QQQ's QInstanceValidator rejected the reconcile process schedule at startup ("a cron schedule must specify a cronTimeZoneId"); QQQ has no default zone. validate() now applies the same checks QQQ does (Quartz CronExpression.validateExpression, a zone from TimeZone.getAvailableIDs) and requires the zone, naming both fields. It also rejects a cron on a scheduler in the instance that does not support cron: QQQ's SimpleScheduler runs fixed intervals only and fails when the schedule manager starts. Config-driven tables without their basepull timestamp field now log the same startup warning as annotated tables, once per table, through one shared helper. A field set to null is an opt-out and stays quiet. README and MIGRATION-1.0 document that schedules run only when the host starts through QApplicationLauncher (qqq-middleware-javalin), and that a reconcile cron needs a Quartz scheduler and a time zone ID. Closes #29 --- README.md | 18 ++- docs/MIGRATION-1.0.md | 12 +- .../quicksearch/QuickSearchQBitConfig.java | 58 ++++++++-- .../quicksearch/QuickSearchQBitProducer.java | 23 +++- .../QuickSearchQBitConfigValidationTest.java | 73 +++++++++++- .../QuickSearchQBitProducerTest.java | 104 +++++++++++++++++- 6 files changed, 268 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index b6758ed..d45853f 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ config.withSearchableTable(new SearchableTableConfig("crmContact", List.of( .withRecordLabelFormat("%s %s", "firstName", "lastName")); ``` -Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull. +Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull, and the QBit logs a warning for it at startup. Set `basepullTimestampField` to null on a config-driven table to opt out without the warning. ### Searching @@ -134,6 +134,20 @@ Search runs as the current `QContext` session: tables the session cannot read ar Without `schedulerName` the processes exist but nothing runs them; the QBit logs a warning at startup. +### Scheduling + +The schedules run only when the host starts through QQQ's `QApplicationLauncher` (`com.kingsrook.qqq.middleware.javalin.QApplicationLauncher`, in `qqq-middleware-javalin`). `QApplicationLauncher.run(new MyApplication(), new QApplicationLauncherConfig())` starts the Javalin server, then the `QScheduleManager` (because the QBit's processes are scheduled), then the instance's runtime services. A host that starts only a server, or builds the `QInstance` itself, gets the processes but nothing runs them on a schedule, unless it calls `QScheduleManager.initInstance(qInstance, systemSessionSupplier).start()` itself. `qqq.scheduleManager.enabled=false` (or `QQQ_SCHEDULE_MANAGER_ENABLED=false`) turns schedules off on a node. + +Basepull's fixed interval runs on any QQQ scheduler. `reconcileCronExpression` needs a scheduler that supports cron, which in QQQ 4.1 means a `QuartzSchedulerMetaData`; a `SimpleSchedulerMetaData` runs fixed intervals only. The expression uses Quartz syntax, seconds first (`0 0 3 * * ?` is 03:00 every day), and needs `reconcileCronTimeZoneId`, a Java time zone ID such as `UTC` or `America/Chicago`; QQQ has no default zone. `produce()` rejects a cron without a zone, an invalid expression or zone, and a cron on a scheduler in the instance that does not support cron. + +```java +qInstance.addScheduler(new QuartzSchedulerMetaData().withName("quartz").withProperties(quartzProperties)); + +config.withSchedulerName("quartz") + .withReconcileCronExpression("0 0 3 * * ?") + .withReconcileCronTimeZoneId("America/Chicago"); +``` + ## Configuration reference | Field | Default | Description | @@ -151,7 +165,7 @@ Without `schedulerName` the processes exist but nothing runs them; the QBit logs | `startupMode` | `FAIL_FAST` | `FAIL_FAST` fails `produce()` when the cluster is unreachable; `DEGRADED` boots and retries on first use | | `enableRealTimeIndexing` | `true` | Register the record-change listener | | `enableBasepullProcess`, `enableMaintenanceProcesses` | `true` | Register basepull; register full reindex and reconcile. `enableScheduledProcesses` is a deprecated alias for both | -| `schedulerName`, `basepullRepeatSeconds`, `reconcileCronExpression`, `reconcileCronTimeZoneId` | none, 300, none | Scheduling | +| `schedulerName`, `basepullRepeatSeconds`, `reconcileCronExpression`, `reconcileCronTimeZoneId` | none, 300, none, none | Scheduling; a cron needs the time zone ID and a Quartz scheduler, and nothing runs without `QApplicationLauncher` (see [Scheduling](#scheduling)) | | `basepullOverlapSeconds` | 300 | Re-read window before the previous run start | | `defaultBasepullIntervalMinutes` | 5 | Per-table due interval for config-driven tables | | `runHistoryRetentionDays` | 30 | Purge run records older than this; null keeps them | diff --git a/docs/MIGRATION-1.0.md b/docs/MIGRATION-1.0.md index 2eb6971..9576b0d 100644 --- a/docs/MIGRATION-1.0.md +++ b/docs/MIGRATION-1.0.md @@ -12,7 +12,7 @@ 1. **Remove any manual `QBitMetaData` registration.** The producer now registers `com.kingsrook.qbits:quick-search` itself; a second `addQBit` with that name throws at startup. (qqq-all's `DemoQBits` did this in 0.x.) 2. **Decide the startup mode.** `startupMode` defaults to `FAIL_FAST`: `produce()` fails when OpenSearch is unreachable. Demo profiles that may run without a cluster should set `withStartupMode(QuickSearchStartupMode.DEGRADED)`. 3. **Run a full reindex once after upgrading.** The index mapping changed (version 2: search analyzer, text-typed `fieldValues`). The QBit logs a warning when it finds an index with the old mapping; "Quick Search Full Reindex" with no table name builds a new physical index and swaps the configured name to an alias. Search keeps working during the rebuild. The identity the QBit uses needs index create, delete and alias privileges for this (see the README role). -4. **Set `schedulerName`** (and optionally `basepullRepeatSeconds`, `reconcileCronExpression`) so basepull and reconcile actually run. In 0.x they were never scheduled. +4. **Set `schedulerName` and start the host through `QApplicationLauncher`** (and optionally `basepullRepeatSeconds`, or `reconcileCronExpression` with `reconcileCronTimeZoneId` on a Quartz scheduler) so basepull and reconcile actually run. In 0.x they were never scheduled. See [Scheduling](#scheduling). 5. **Credentials over HTTP are refused.** Use `opensearchUrl` with `https://`, or set `allowPlaintextCredentials(true)` for local development. Prefer `${env.OPENSEARCH_PASSWORD}` references over literal secrets; never name an OpenSearch secret `QQQ_ENV_*` (those are sent to the frontend). 6. **Permissions.** Processes and the admin app now require `quickSearchAdmin.hasAccess`; the operational tables require read and write permissions. Grant them to operators, or override `adminPermissionRules` and `tablePermissionRules`. Search now honours table read permissions and record security locks, so users see fewer results than in 0.x if they lacked access. @@ -20,12 +20,20 @@ - Real-time indexing happens after commit. Code that searched for a record inside the same transaction that inserted it will not find it until commit. - Real-time failures no longer vanish: they land in `quickSearchFailedEvent` and are replayed by the next basepull. Review that table after an OpenSearch outage. -- A table without the change-detection field (`modifyDate` by default) is indexed on the first basepull and thereafter only by reconcile. +- A table without the change-detection field (`modifyDate` by default) is indexed on the first basepull and thereafter only by reconcile. The QBit logs a warning for each such table at startup, for annotated and config-driven tables alike; a config-driven table with `basepullTimestampField` set to null has opted out and gets no warning. - `limit` is capped (default 100), `offset + limit` at 10,000, and search terms must be 2 to 100 characters. - Query semantics changed from OR over n-grams to AND over whole terms with prefix matching: "blue widget" now requires both words. - `enableScheduledProcesses` still works and sets both `enableBasepullProcess` and `enableMaintenanceProcesses`. - `QuickSearchQBitContext` is deprecated and removed in 1.1; use `QuickSearchRuntime.get()`. +## Scheduling + +Setting `schedulerName` puts schedules on the basepull and reconcile processes; QQQ runs them only once something starts its `QScheduleManager`. + +- **Start through `QApplicationLauncher`.** In QQQ 4.1 that is `com.kingsrook.qqq.middleware.javalin.QApplicationLauncher` (`qqq-middleware-javalin`). `QApplicationLauncher.run(new MyApplication(), new QApplicationLauncherConfig())` starts the Javalin server, then the `QScheduleManager` when any process is scheduled, then the instance's runtime services, including this QBit's. A host that starts only a `QApplicationJavalinServer`, or builds its `QInstance` without the launcher, gets the processes but no scheduled runs, unless it calls `QScheduleManager.initInstance(qInstance, systemSessionSupplier).start()` itself. `qqq.scheduleManager.enabled=false` (or `QQQ_SCHEDULE_MANAGER_ENABLED=false`) turns schedules off on a node. +- **A reconcile cron needs a Quartz scheduler.** Basepull's fixed interval runs on any QQQ scheduler, but only `QuartzSchedulerMetaData` supports cron; QQQ's `SimpleSchedulerMetaData` runs fixed intervals only, and a cron schedule on it fails when the schedule manager starts. +- **A reconcile cron needs a time zone.** Set `reconcileCronTimeZoneId` to a Java time zone ID (`UTC`, `America/Chicago`). QQQ has no default zone: its instance validation rejects a cron schedule without one at startup. `produce()` now catches this first, along with an invalid Quartz expression (seconds first, for example `0 0 3 * * ?`), an unknown zone ID, and a cron on a scheduler in the instance that does not support cron. + ## Removed - `QuickSearchPostInsertCustomizer`, `QuickSearchPostUpdateCustomizer`, `QuickSearchPostDeleteCustomizer`. Hosts that referenced them (for example to compose with their own customizers) should remove those references; the record-change listener coexists with host customizers. diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java index 01b3da3..23eb5ee 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java @@ -18,10 +18,13 @@ package com.kingsrook.qbits.quicksearch; import java.net.URI; +import java.text.ParseException; import java.util.ArrayList; +import java.util.Arrays; import java.util.HashSet; import java.util.List; import java.util.Set; +import java.util.TimeZone; import java.util.regex.Pattern; import com.fasterxml.jackson.annotation.JsonIgnore; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; @@ -30,12 +33,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QSchedulerMetaData; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.utils.CollectionUtils; import com.kingsrook.qqq.backend.core.utils.StringUtils; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndex; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndexRun; import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher; +import org.quartz.CronExpression; /******************************************************************************* @@ -412,18 +417,53 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** + ** The scheduler must exist when the instance already has schedulers. A + ** reconcile cron gets the checks QQQ's instance validator applies to every + ** cron schedule (Quartz syntax, a required and known time zone, which QQQ + ** never defaults), plus a scheduler that supports cron, so a bad schedule + ** fails produce() naming these fields instead of failing at startup. *******************************************************************************/ private void validateScheduling(QInstance qInstance, List errors) { - if(StringUtils.hasContent(schedulerName) && qInstance != null && !CollectionUtils.nullSafeIsEmpty(qInstance.getSchedulers()) && qInstance.getScheduler(schedulerName) == null) + QSchedulerMetaData scheduler = null; + if(StringUtils.hasContent(schedulerName) && qInstance != null && !CollectionUtils.nullSafeIsEmpty(qInstance.getSchedulers())) { - errors.add("Scheduler not found: " + schedulerName); + scheduler = qInstance.getScheduler(schedulerName); + if(scheduler == null) + { + errors.add("Scheduler not found: " + schedulerName); + } } - if(StringUtils.hasContent(reconcileCronExpression) && !StringUtils.hasContent(schedulerName)) + if(StringUtils.hasContent(reconcileCronExpression)) { - errors.add("reconcileCronExpression requires schedulerName"); + if(!StringUtils.hasContent(schedulerName)) + { + errors.add("reconcileCronExpression requires schedulerName"); + } + else if(scheduler != null && !scheduler.supportsCronSchedules()) + { + errors.add("reconcileCronExpression requires a scheduler that supports cron schedules, such as a QuartzSchedulerMetaData; scheduler [" + schedulerName + "] is of type " + scheduler.getType()); + } + + try + { + CronExpression.validateExpression(reconcileCronExpression); + } + catch(ParseException e) + { + errors.add("reconcileCronExpression [" + reconcileCronExpression + "] is not a valid Quartz cron expression: " + e.getMessage()); + } + + if(!StringUtils.hasContent(reconcileCronTimeZoneId)) + { + errors.add("reconcileCronExpression requires reconcileCronTimeZoneId (a time zone ID such as UTC or America/Chicago)"); + } + } + + if(StringUtils.hasContent(reconcileCronTimeZoneId) && !Arrays.asList(TimeZone.getAvailableIDs()).contains(reconcileCronTimeZoneId)) + { + errors.add("reconcileCronTimeZoneId [" + reconcileCronTimeZoneId + "] is not a recognized time zone ID"); } } @@ -1579,7 +1619,9 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** Getter for reconcileCronExpression: optional cron schedule for reconcile (needs a scheduler that supports cron) + ** Getter for reconcileCronExpression: optional Quartz cron schedule for + ** reconcile; needs reconcileCronTimeZoneId and a scheduler that supports + ** cron (Quartz) *******************************************************************************/ public String getReconcileCronExpression() { @@ -1611,7 +1653,9 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** Getter for reconcileCronTimeZoneId: time zone for reconcileCronExpression + ** Getter for reconcileCronTimeZoneId: time zone ID for + ** reconcileCronExpression (UTC, America/Chicago); required with it, since + ** QQQ has no default zone for cron schedules *******************************************************************************/ public String getReconcileCronTimeZoneId() { diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java index 18c679a..e994124 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java @@ -386,8 +386,12 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer