diff --git a/README.md b/README.md index 0fe93b6..9ef7f2d 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ config.withSearchableTable(new SearchableTableConfig("crmContact", List.of( .withRecordLabelFormat("%s %s", "firstName", "lastName")); ``` -Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden and password fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull. +Annotation and config-driven tables can be mixed; a table may appear in only one of them. Hidden and password fields are never indexed. A table without the change-detection field (`modifyDate` by default) is covered by reconcile instead of incremental basepull, and the QBit logs a warning for it at startup. Set `basepullTimestampField` to null on a config-driven table to opt out without the warning. ### Searching @@ -146,6 +146,20 @@ QQQ core has its own record search (`RecordSearchAction`, served at `POST /qqq/v Without `schedulerName` the processes exist but nothing runs them; the QBit logs a warning at startup. +### Scheduling + +The schedules run only when the host starts through QQQ's `QApplicationLauncher` (`com.kingsrook.qqq.middleware.javalin.QApplicationLauncher`, in `qqq-middleware-javalin`). `QApplicationLauncher.run(new MyApplication(), new QApplicationLauncherConfig())` starts the Javalin server, then the `QScheduleManager` (because the QBit's processes are scheduled), then the instance's runtime services. A host that starts only a server, or builds the `QInstance` itself, gets the processes but nothing runs them on a schedule, unless it calls `QScheduleManager.initInstance(qInstance, systemSessionSupplier).start()` itself. `qqq.scheduleManager.enabled=false` (or `QQQ_SCHEDULE_MANAGER_ENABLED=false`) turns schedules off on a node. + +Basepull's fixed interval runs on any QQQ scheduler. `reconcileCronExpression` needs a scheduler that supports cron, which in QQQ 4.1 means a `QuartzSchedulerMetaData`; a `SimpleSchedulerMetaData` runs fixed intervals only. The expression uses Quartz syntax, seconds first (`0 0 3 * * ?` is 03:00 every day), and needs `reconcileCronTimeZoneId`, a Java time zone ID such as `UTC` or `America/Chicago`; QQQ has no default zone. `produce()` rejects a cron without a zone, an invalid expression or zone, and a cron on a scheduler in the instance that does not support cron. + +```java +qInstance.addScheduler(new QuartzSchedulerMetaData().withName("quartz").withProperties(quartzProperties)); + +config.withSchedulerName("quartz") + .withReconcileCronExpression("0 0 3 * * ?") + .withReconcileCronTimeZoneId("America/Chicago"); +``` + ## Configuration reference | Field | Default | Description | @@ -163,7 +177,7 @@ Without `schedulerName` the processes exist but nothing runs them; the QBit logs | `startupMode` | `FAIL_FAST` | `FAIL_FAST` fails `produce()` when the cluster is unreachable; `DEGRADED` boots and retries on first use | | `enableRealTimeIndexing` | `true` | Register the record-change listener | | `enableBasepullProcess`, `enableMaintenanceProcesses` | `true` | Register basepull; register full reindex and reconcile. `enableScheduledProcesses` is a deprecated alias for both | -| `schedulerName`, `basepullRepeatSeconds`, `reconcileCronExpression`, `reconcileCronTimeZoneId` | none, 300, none | Scheduling | +| `schedulerName`, `basepullRepeatSeconds`, `reconcileCronExpression`, `reconcileCronTimeZoneId` | none, 300, none, none | Scheduling; a cron needs the time zone ID and a Quartz scheduler, and nothing runs without `QApplicationLauncher` (see [Scheduling](#scheduling)) | | `basepullOverlapSeconds` | 300 | Re-read window before the previous run start | | `defaultBasepullIntervalMinutes` | 5 | Per-table due interval for config-driven tables | | `runHistoryRetentionDays` | 30 | Purge run records older than this; null keeps them | diff --git a/docs/MIGRATION-1.0.md b/docs/MIGRATION-1.0.md index 4adb5d8..88a192b 100644 --- a/docs/MIGRATION-1.0.md +++ b/docs/MIGRATION-1.0.md @@ -12,7 +12,7 @@ 1. **Remove any manual `QBitMetaData` registration.** The producer now registers `com.kingsrook.qbits:quick-search` itself; a second `addQBit` with that name throws at startup. (qqq-all's `DemoQBits` did this in 0.x.) 2. **Decide the startup mode.** `startupMode` defaults to `FAIL_FAST`: `produce()` fails when OpenSearch is unreachable. Demo profiles that may run without a cluster should set `withStartupMode(QuickSearchStartupMode.DEGRADED)`. 3. **Run a full reindex once after upgrading.** The index mapping changed (version 2: search analyzer, text-typed `fieldValues`). The QBit logs a warning when it finds an index with the old mapping; "Quick Search Full Reindex" with no table name builds a new physical index and swaps the configured name to an alias. Search keeps working during the rebuild. The identity the QBit uses needs index create, delete and alias privileges for this (see the README role). -4. **Set `schedulerName`** (and optionally `basepullRepeatSeconds`, `reconcileCronExpression`) so basepull and reconcile actually run. In 0.x they were never scheduled. +4. **Set `schedulerName` and start the host through `QApplicationLauncher`** (and optionally `basepullRepeatSeconds`, or `reconcileCronExpression` with `reconcileCronTimeZoneId` on a Quartz scheduler) so basepull and reconcile actually run. In 0.x they were never scheduled. See [Scheduling](#scheduling). 5. **Credentials over HTTP are refused.** Use `opensearchUrl` with `https://`, or set `withAllowPlaintextCredentials(true)` for local development. Prefer `${env.OPENSEARCH_PASSWORD}` references over literal secrets; never name an OpenSearch secret `QQQ_ENV_*` (those are sent to the frontend). 6. **Grant the permissions.** Processes and the admin app now require `quickSearchAdmin.hasAccess`; the operational tables require read and write permissions. Grant the names listed under [Permissions](#permissions) to operators, or override `adminPermissionRules` and `tablePermissionRules`. Search now honours table read permissions and record security locks, so users see fewer results than in 0.x if they lacked access. 7. **Update the operational tables before starting 1.0.** `quickSearchIndex` gains five columns, `searchableFieldsJson` changes from `STRING` to `TEXT`, and `tableName` becomes required with a unique key. `quickSearchIndexRun.errorMessage` changes from `STRING` to `TEXT`. There is a new `quickSearchFailedEvent` table. A host that checks its columns at startup rejects the 0.x tables. See [Schema changes](#schema-changes). @@ -76,12 +76,20 @@ If your backend creates tables from metadata and you do not need the 0.x run his - Real-time indexing happens after commit. Code that searched for a record inside the same transaction that inserted it will not find it until commit. - Real-time failures no longer vanish: they land in `quickSearchFailedEvent` and are replayed by the next basepull. Review that table after an OpenSearch outage. - Inserted and updated records are re-read inside the write's transaction, with display values and possible-value labels, before they are indexed. That is one extra query per write batch on an indexed table, and it runs the host's possible-value providers. If the re-read fails, the write still succeeds and the records are replayed by basepull. -- A table without the change-detection field (`modifyDate` by default) is indexed on the first basepull and thereafter only by reconcile. +- A table without the change-detection field (`modifyDate` by default) is indexed on the first basepull and thereafter only by reconcile. The QBit logs a warning for each such table at startup, for annotated and config-driven tables alike; a config-driven table with `basepullTimestampField` set to null has opted out and gets no warning. - `limit` is capped (default 100), `offset + limit` at 10,000, and search terms must be 2 to 100 characters. - Query semantics changed from OR over n-grams to AND over whole terms with prefix matching: "blue widget" now requires both words. - `enableScheduledProcesses` still works and sets both `enableBasepullProcess` and `enableMaintenanceProcesses`. - `QuickSearchQBitContext` is deprecated and removed in 1.1; use `QuickSearchRuntime.get()`. +## Scheduling + +Setting `schedulerName` puts schedules on the basepull and reconcile processes; QQQ runs them only once something starts its `QScheduleManager`. + +- **Start through `QApplicationLauncher`.** In QQQ 4.1 that is `com.kingsrook.qqq.middleware.javalin.QApplicationLauncher` (`qqq-middleware-javalin`). `QApplicationLauncher.run(new MyApplication(), new QApplicationLauncherConfig())` starts the Javalin server, then the `QScheduleManager` when any process is scheduled, then the instance's runtime services, including this QBit's. A host that starts only a `QApplicationJavalinServer`, or builds its `QInstance` without the launcher, gets the processes but no scheduled runs, unless it calls `QScheduleManager.initInstance(qInstance, systemSessionSupplier).start()` itself. `qqq.scheduleManager.enabled=false` (or `QQQ_SCHEDULE_MANAGER_ENABLED=false`) turns schedules off on a node. +- **A reconcile cron needs a Quartz scheduler.** Basepull's fixed interval runs on any QQQ scheduler, but only `QuartzSchedulerMetaData` supports cron; QQQ's `SimpleSchedulerMetaData` runs fixed intervals only, and a cron schedule on it fails when the schedule manager starts. +- **A reconcile cron needs a time zone.** Set `reconcileCronTimeZoneId` to a Java time zone ID (`UTC`, `America/Chicago`). QQQ has no default zone: its instance validation rejects a cron schedule without one at startup. `produce()` now catches this first, along with an invalid Quartz expression (seconds first, for example `0 0 3 * * ?`), an unknown zone ID, and a cron on a scheduler in the instance that does not support cron. + ## Removed - `QuickSearchPostInsertCustomizer`, `QuickSearchPostUpdateCustomizer`, `QuickSearchPostDeleteCustomizer`. Hosts that referenced them (for example to compose with their own customizers) should remove those references; the record-change listener coexists with host customizers. diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java index b8492de..86a1bed 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitConfig.java @@ -18,10 +18,13 @@ package com.kingsrook.qbits.quicksearch; import java.net.URI; +import java.text.ParseException; import java.util.ArrayList; +import java.util.Arrays; import java.util.HashSet; import java.util.List; import java.util.Set; +import java.util.TimeZone; import java.util.regex.Pattern; import com.fasterxml.jackson.annotation.JsonIgnore; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; @@ -30,12 +33,14 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QSchedulerMetaData; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; import com.kingsrook.qqq.backend.core.utils.CollectionUtils; import com.kingsrook.qqq.backend.core.utils.StringUtils; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndex; import com.kingsrook.qbits.quicksearch.model.QuickSearchIndexRun; import com.kingsrook.qbits.quicksearch.publisher.IndexEventPublisher; +import org.quartz.CronExpression; /******************************************************************************* @@ -421,18 +426,53 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** + ** The scheduler must exist when the instance already has schedulers. A + ** reconcile cron gets the checks QQQ's instance validator applies to every + ** cron schedule (Quartz syntax, a required and known time zone, which QQQ + ** never defaults), plus a scheduler that supports cron, so a bad schedule + ** fails produce() naming these fields instead of failing at startup. *******************************************************************************/ private void validateScheduling(QInstance qInstance, List errors) { - if(StringUtils.hasContent(schedulerName) && qInstance != null && !CollectionUtils.nullSafeIsEmpty(qInstance.getSchedulers()) && qInstance.getScheduler(schedulerName) == null) + QSchedulerMetaData scheduler = null; + if(StringUtils.hasContent(schedulerName) && qInstance != null && !CollectionUtils.nullSafeIsEmpty(qInstance.getSchedulers())) { - errors.add("Scheduler not found: " + schedulerName); + scheduler = qInstance.getScheduler(schedulerName); + if(scheduler == null) + { + errors.add("Scheduler not found: " + schedulerName); + } } - if(StringUtils.hasContent(reconcileCronExpression) && !StringUtils.hasContent(schedulerName)) + if(StringUtils.hasContent(reconcileCronExpression)) { - errors.add("reconcileCronExpression requires schedulerName"); + if(!StringUtils.hasContent(schedulerName)) + { + errors.add("reconcileCronExpression requires schedulerName"); + } + else if(scheduler != null && !scheduler.supportsCronSchedules()) + { + errors.add("reconcileCronExpression requires a scheduler that supports cron schedules, such as a QuartzSchedulerMetaData; scheduler [" + schedulerName + "] is of type " + scheduler.getType()); + } + + try + { + CronExpression.validateExpression(reconcileCronExpression); + } + catch(ParseException e) + { + errors.add("reconcileCronExpression [" + reconcileCronExpression + "] is not a valid Quartz cron expression: " + e.getMessage()); + } + + if(!StringUtils.hasContent(reconcileCronTimeZoneId)) + { + errors.add("reconcileCronExpression requires reconcileCronTimeZoneId (a time zone ID such as UTC or America/Chicago)"); + } + } + + if(StringUtils.hasContent(reconcileCronTimeZoneId) && !Arrays.asList(TimeZone.getAvailableIDs()).contains(reconcileCronTimeZoneId)) + { + errors.add("reconcileCronTimeZoneId [" + reconcileCronTimeZoneId + "] is not a recognized time zone ID"); } } @@ -1588,7 +1628,9 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** Getter for reconcileCronExpression: optional cron schedule for reconcile (needs a scheduler that supports cron) + ** Getter for reconcileCronExpression: optional Quartz cron schedule for + ** reconcile; needs reconcileCronTimeZoneId and a scheduler that supports + ** cron (Quartz) *******************************************************************************/ public String getReconcileCronExpression() { @@ -1620,7 +1662,9 @@ public class QuickSearchQBitConfig implements QBitConfig /******************************************************************************* - ** Getter for reconcileCronTimeZoneId: time zone for reconcileCronExpression + ** Getter for reconcileCronTimeZoneId: time zone ID for + ** reconcileCronExpression (UTC, America/Chicago); required with it, since + ** QQQ has no default zone for cron schedules *******************************************************************************/ public String getReconcileCronTimeZoneId() { diff --git a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java index d50b759..df17b71 100644 --- a/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/quicksearch/QuickSearchQBitProducer.java @@ -474,8 +474,12 @@ public class QuickSearchQBitProducer implements QBitMetaDataProducer