perf: scheduled jobs and the signal queue issue far fewer statements #75

Merged

james.maes merged 2 commits from perf/scheduled-job-statements into develop 2026-11-10 22:11:43 +00:00

Conversation 0 Commits 2 Files Changed 29

+2336 -156

james.maes commented 2026-11-10 21:40:45 +00:00

Owner

Copy link

Copy source

Why

The alerts jobs run every 15 to 60 seconds around the clock. In a host their cost is the number of statements (each is a round trip and a pool check-out), and several paths issued statements per alert, per monitor, per delivery row or per queued signal. A host with per-record read hooks pays again for every row these paths read.

What was found (paths at develop before this change)

Path	What it did	Statements
EscalationAndReminder.runDue (every 60 s)	read every OPEN alert, then per alert: alert, type, record refs, two mute queries, group, alert again, and a checkpoint update	1 per 100 alerts + 9 to 11 per open unacknowledged alert, every minute, whether or not anything was due
MonitorEvaluator.evaluateDue (every 30 s)	read every enabled monitor and filtered in Java; per due monitor: claim update, re-read monitor, type, mutes, one open-alert lookup per signal, run-result update	1 (all monitors) + about 9 per due monitor + 1 per signal
AlertsEngine.maintenance (same tick)	snoozed, one query per quiet-period type, flapping (unindexed)	2 + quiet-period types + 1 per flapping alert
DeliveryWorker.deliverDue (every 15 s)	per row: claim update, alert read, outcome update	1 + about 5 per pending row
DeliverNotifications step	purge (2 queries) on every tick; emailer re-read settings and preferences per page	9 idle
QbitAlertsSink drain	one item at a time: a resolve for an alert that is not open = 1, a heartbeat = 3, a repeated raise = 17 (type, mutes, alert x3, refs x2, count, type count + update, audits)	per signal
AlertsApi.inboxSummary (the bell) and recordPanel	fixed number of statements; the bell reads up to 1,000 recipient rows and their notifications per poll	3 and 6 (not changed)

Before and after

StatementBudgetTest runs each scenario against H2 through StatementCounter (the standard RDBMS strategy, recording every statement) and fails when a budget is exceeded. "Before" is the same test on develop (source identical to 0.1.0-RC.18).

Scenario	Before	After
escalate and remind, 50 open alerts, nothing due	452	1
same, with a policy and reminder not due yet and an unrelated mute	552	1
evaluate monitors, 20 monitors, none due	5	2
evaluate monitors, 20 due, clean	185	84
per-record monitor refreshing 10 open alerts	205	120
deliver, nothing due	9	7
deliver, 200 digest-waiting recipients	11	7
deliver, 60 pending rows in 2 groups	317	22
queue: 200 resolves for alerts not open + 50 heartbeats	369	about 33
20 repeated raises of one open alert	340	200
bell / record panel	3 / 6	3 / 6

How

• Select what is due. Monitors: enabled, kind, nextRunAt and claimedUntil in SQL (existing ix_alerts_monitor_enabled_next_run_at); one claim update for all due monitors. Escalation: OPEN, unacknowledged, not attempted this minute, and (CRITICAL, or a type with a reminder or an enabled policy, or a group with an enabled policy); the decision per alert is made from the row, and only an alert that may be due takes the existing per-alert transaction, which re-reads and re-checks everything as before.
• Batch. One open-alert lookup for a monitor's signals and for consecutive queued resolves; one read and one update action for a batch of heartbeats; one claim, one alert read and one outcome update per delivery group; upkeep in one read.
• Cache reference data: QQQ core's Memoization registered with core ClusterCaches (AlertsReferenceCache): alert types, groups, escalation policies, steps and enabled mutes, each table held whole.
  • Invalidation: an instance RecordChangeListenerInterface clears the table on any insert, update or delete through QQQ, on this node and through the ClusterBus on the others. The engine's own counter updates on alerts_type are ignored and cached type rows do not carry those counters.
  • Safety net: 60 second expiry (lost event, direct SQL, a listener that ran before its transaction committed).
  • Bound: a table over 2,000 rows is not held; lookups query as before. A failed load is not cached and throws as the query did.
  • qbit-redis is not used: the data is a few small tables that every node can hold, core already gives cluster invalidation, and a Redis dependency would add a failure mode and a network round trip to save a 60-second staleness bound.
• Do nothing when there is nothing to do. An alert with nothing due is neither read (when its type has no reminder and no policy) nor written.
• Index. Changeset 55: ix_alerts_alert_flapping (the upkeep read scanned every alert). The other job predicates already had indexes.

Behaviour

Delivery, escalation and reminder semantics are unchanged; all 229 existing tests pass, with one assertion changed (below). Differences a host can see:

• timedNotificationCheckedAt is written on an attempt (escalation step or reminder, including one with no eligible recipient), not on every tick. It still stops a second attempt in the same minute. EscalationAndReminderTest.testAlertPages asserted the old every-tick write on 101 alerts with nothing due; it now makes the 101 alerts due and asserts the same thing. testNothingDueIsNotWritten is new.
• Updates of a type's openCount / lastRaisedAt are not record-audited (same rule as the other RuntimeFields).
• Purge of old notifications is hourly per node.
• A resolve queued behind other resolves is skipped when no alert was open for its key at the start of that run (it used to look once per resolve); order within the queue is preserved (SinkAndScheduleTest.testBatchKeepsOrder).

Not done here

• The bell still reads up to 1,000 recipient rows per poll, and the emailer still re-reads digest-waiting recipients on each tick (statements are constant; the rows are not). Reducing the rows changes when rows are marked SKIPPED, which needs a product decision.
• The monitor claim is still read-then-update, as documented ("run on one scheduler node").
• A raise still costs 10 statements, 3 of them core's DML audit and old-record reads.

Gates

./mvnw -B verify (Checkstyle at validate, 246 tests, JaCoCo check) on this branch head: BUILD SUCCESS, no skip flags.

Release

Merge to develop, then develop into the release branch to cut the next RC; hosts pin that RC. No host changelog change is needed when the host includes alerts.changelog.sql. See UPGRADE.md.

🤖 Generated with Claude Code

## Why The alerts jobs run every 15 to 60 seconds around the clock. In a host their cost is the number of statements (each is a round trip and a pool check-out), and several paths issued statements per alert, per monitor, per delivery row or per queued signal. A host with per-record read hooks pays again for every row these paths read. ## What was found (paths at `develop` before this change) | Path | What it did | Statements | |---|---|---| | `EscalationAndReminder.runDue` (every 60 s) | read every OPEN alert, then per alert: alert, type, record refs, two mute queries, group, alert again, and a checkpoint update | 1 per 100 alerts + 9 to 11 per open unacknowledged alert, every minute, whether or not anything was due | | `MonitorEvaluator.evaluateDue` (every 30 s) | read every enabled monitor and filtered in Java; per due monitor: claim update, re-read monitor, type, mutes, one open-alert lookup per signal, run-result update | 1 (all monitors) + about 9 per due monitor + 1 per signal | | `AlertsEngine.maintenance` (same tick) | snoozed, one query per quiet-period type, flapping (unindexed) | 2 + quiet-period types + 1 per flapping alert | | `DeliveryWorker.deliverDue` (every 15 s) | per row: claim update, alert read, outcome update | 1 + about 5 per pending row | | `DeliverNotifications` step | purge (2 queries) on every tick; emailer re-read settings and preferences per page | 9 idle | | `QbitAlertsSink` drain | one item at a time: a resolve for an alert that is not open = 1, a heartbeat = 3, a repeated raise = 17 (type, mutes, alert x3, refs x2, count, type count + update, audits) | per signal | | `AlertsApi.inboxSummary` (the bell) and `recordPanel` | fixed number of statements; the bell reads up to 1,000 recipient rows and their notifications per poll | 3 and 6 (not changed) | ## Before and after `StatementBudgetTest` runs each scenario against H2 through `StatementCounter` (the standard RDBMS strategy, recording every statement) and fails when a budget is exceeded. "Before" is the same test on `develop` (source identical to 0.1.0-RC.18). | Scenario | Before | After | |---|---|---| | escalate and remind, 50 open alerts, nothing due | 452 | 1 | | same, with a policy and reminder not due yet and an unrelated mute | 552 | 1 | | evaluate monitors, 20 monitors, none due | 5 | 2 | | evaluate monitors, 20 due, clean | 185 | 84 | | per-record monitor refreshing 10 open alerts | 205 | 120 | | deliver, nothing due | 9 | 7 | | deliver, 200 digest-waiting recipients | 11 | 7 | | deliver, 60 pending rows in 2 groups | 317 | 22 | | queue: 200 resolves for alerts not open + 50 heartbeats | 369 | about 33 | | 20 repeated raises of one open alert | 340 | 200 | | bell / record panel | 3 / 6 | 3 / 6 | ## How - **Select what is due.** Monitors: `enabled`, kind, `nextRunAt` and `claimedUntil` in SQL (existing `ix_alerts_monitor_enabled_next_run_at`); one claim update for all due monitors. Escalation: OPEN, unacknowledged, not attempted this minute, and (CRITICAL, or a type with a reminder or an enabled policy, or a group with an enabled policy); the decision per alert is made from the row, and only an alert that may be due takes the existing per-alert transaction, which re-reads and re-checks everything as before. - **Batch.** One open-alert lookup for a monitor's signals and for consecutive queued resolves; one read and one update action for a batch of heartbeats; one claim, one alert read and one outcome update per delivery group; upkeep in one read. - **Cache reference data: QQQ core's `Memoization` registered with core `ClusterCaches`** (`AlertsReferenceCache`): alert types, groups, escalation policies, steps and enabled mutes, each table held whole. - Invalidation: an instance `RecordChangeListenerInterface` clears the table on any insert, update or delete through QQQ, on this node and through the `ClusterBus` on the others. The engine's own counter updates on `alerts_type` are ignored and cached type rows do not carry those counters. - Safety net: 60 second expiry (lost event, direct SQL, a listener that ran before its transaction committed). - Bound: a table over 2,000 rows is not held; lookups query as before. A failed load is not cached and throws as the query did. - qbit-redis is not used: the data is a few small tables that every node can hold, core already gives cluster invalidation, and a Redis dependency would add a failure mode and a network round trip to save a 60-second staleness bound. - **Do nothing when there is nothing to do.** An alert with nothing due is neither read (when its type has no reminder and no policy) nor written. - **Index.** Changeset 55: `ix_alerts_alert_flapping` (the upkeep read scanned every alert). The other job predicates already had indexes. ## Behaviour Delivery, escalation and reminder semantics are unchanged; all 229 existing tests pass, with one assertion changed (below). Differences a host can see: - `timedNotificationCheckedAt` is written on an attempt (escalation step or reminder, including one with no eligible recipient), not on every tick. It still stops a second attempt in the same minute. `EscalationAndReminderTest.testAlertPages` asserted the old every-tick write on 101 alerts with nothing due; it now makes the 101 alerts due and asserts the same thing. `testNothingDueIsNotWritten` is new. - Updates of a type's `openCount` / `lastRaisedAt` are not record-audited (same rule as the other `RuntimeFields`). - Purge of old notifications is hourly per node. - A resolve queued behind other resolves is skipped when no alert was open for its key at the start of that run (it used to look once per resolve); order within the queue is preserved (`SinkAndScheduleTest.testBatchKeepsOrder`). ## Not done here - The bell still reads up to 1,000 recipient rows per poll, and the emailer still re-reads digest-waiting recipients on each tick (statements are constant; the rows are not). Reducing the rows changes when rows are marked SKIPPED, which needs a product decision. - The monitor claim is still read-then-update, as documented ("run on one scheduler node"). - A raise still costs 10 statements, 3 of them core's DML audit and old-record reads. ## Gates `./mvnw -B verify` (Checkstyle at validate, 246 tests, JaCoCo check) on this branch head: BUILD SUCCESS, no skip flags. ## Release Merge to `develop`, then `develop` into the release branch to cut the next RC; hosts pin that RC. No host changelog change is needed when the host includes `alerts.changelog.sql`. See `UPGRADE.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 2 commits 2026-11-10 21:40:45 +00:00

perf: scheduled jobs and the signal queue issue far fewer statements ... e26116e075

The alerts jobs run every 15 to 60 seconds, so their statement count is a
host's constant database load. StatementBudgetTest counts the statements of
each job and polled request and fails when they grow with the rows held.
Measured with the same scenarios before (0.1.0-RC.18) and after:

- alertsEscalateAndRemind, 50 open alerts, nothing due: 452 -> 1. It reads
  only alerts a reminder or an escalation policy can apply to, decides from
  the row and the reference tables, and takes the per-alert transaction only
  when something may be due. An alert with nothing due is not written, so
  timedNotificationCheckedAt is the last attempt, not the last tick.
- alertsEvaluateMonitors: 20 monitors not due 5 -> 2 (due monitors selected
  in SQL, upkeep in one read); 20 due clean monitors 185 -> 84; a per-record
  monitor refreshing 10 alerts 205 -> 120.
- alertsDeliverNotifications: idle 9 -> 7; 60 pending deliveries in 2 groups
  317 -> 22; the purge runs hourly instead of every tick.
- Queued producer signals are recorded in batches: 200 resolves for alerts
  that are not open plus 50 heartbeats 369 -> about 33. A repeated raise of
  an open alert 17 -> 10.

AlertsReferenceCache holds alert types, groups, escalation policies, steps
and enabled mutes in a core Memoization registered with core ClusterCaches.
A record-change listener clears a table on every node (ClusterBus) when it
is written through QQQ; entries expire after 60 seconds; a table over 2,000
rows is not held. A type's openCount and lastRaisedAt are engine counters:
they do not clear the cache and are no longer record-audited.

Changeset 55 adds ix_alerts_alert_flapping for the upkeep read.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs: upgrade notes for the statement reduction ... f0c6c77354

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes marked the pull request as ready for review 2026-11-10 22:11:41 +00:00

james.maes merged commit b6b849ff84 into develop 2026-11-10 22:11:43 +00:00

james.maes referenced this issue from a commit 2026-11-10 22:12:05 +00:00

Merge develop into release/0.3 for 0.3.0-RC.2 (#75 scheduled-job and signal-queue statement reduction)

james.maes referenced this pull request 2026-11-10 22:12:06 +00:00

Merge develop into release/0.3 for 0.3.0-RC.2 (#75 performance fix) #76

james.maes referenced this issue from a commit 2026-11-10 22:14:45 +00:00

Merge pull request 'Merge develop into release/0.3 for 0.3.0-RC.2 (#75 performance fix)' (#76) from merge/develop-into-release-0.3-rc2 into release/0.3

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

Architecture review: coupling, cohesion, UI API

bug

Something isn't working

dependencies

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

enhancement

New feature or request

github_actions

Pull requests that update GitHub Actions code

good first issue

Good for newcomers

help wanted

Extra attention is needed

invalid

This doesn't seem right

java

Pull requests that update java code

question

Further information is requested

release:rc

Merging this develop PR cuts the next release candidate

secuurity

ux

Admin UX / information architecture

wontfix

This will not be worked on

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

forge-recovery (Forge Recovery) james.maes (James Maes)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-alerts#75