james.maesandClaude Fable 5.2
e26116e075
The alerts jobs run every 15 to 60 seconds, so their statement count is a host's constant database load. StatementBudgetTest counts the statements of each job and polled request and fails when they grow with the rows held. Measured with the same scenarios before (0.1.0-RC.18) and after: - alertsEscalateAndRemind, 50 open alerts, nothing due: 452 -> 1. It reads only alerts a reminder or an escalation policy can apply to, decides from the row and the reference tables, and takes the per-alert transaction only when something may be due. An alert with nothing due is not written, so timedNotificationCheckedAt is the last attempt, not the last tick. - alertsEvaluateMonitors: 20 monitors not due 5 -> 2 (due monitors selected in SQL, upkeep in one read); 20 due clean monitors 185 -> 84; a per-record monitor refreshing 10 alerts 205 -> 120. - alertsDeliverNotifications: idle 9 -> 7; 60 pending deliveries in 2 groups 317 -> 22; the purge runs hourly instead of every tick. - Queued producer signals are recorded in batches: 200 resolves for alerts that are not open plus 50 heartbeats 369 -> about 33. A repeated raise of an open alert 17 -> 10. AlertsReferenceCache holds alert types, groups, escalation policies, steps and enabled mutes in a core Memoization registered with core ClusterCaches. A record-change listener clears a table on every node (ClusterBus) when it is written through QQQ; entries expire after 60 seconds; a table over 2,000 rows is not held. A type's openCount and lastRaisedAt are engine counters: they do not clear the cache and are no longer record-audited. Changeset 55 adds ix_alerts_alert_flapping for the upkeep read. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
6.0 KiB
6.0 KiB
QBit Alerts: design
The design for this qbit is tracked in QRun-IO/qqq#1090. This repository implements it. Phases P1 to P3 are built (see README.md); Quiet hours and subscription digests from P4 are also built; the remaining work is listed in README.md.
Deviations from the design, and why:
- Timed notifications (#27, part 2): reminders and escalation are implemented ahead of the remaining P4–P6 features. The minute job uses
stepOrderand typeduserId/teamId/channelIdfields instead of the design's generictargetId. Policy description and enabled fields make the standard admin records usable. Types override groups; only OPEN, unacknowledged alerts run, even if a step is configured for another status, because acknowledgement must stop escalation. - Escalation progress: existing ESCALATED events hold level transitions and final-step repeat attempts; repeats keep the same level. Reopen events define each cycle's start and repeat budget. One step runs per minute, including when several steps became overdue. Null/zero repeat settings disable repeats. Suppression consumes an escalation attempt, as the normal router records why no delivery was queued.
- Timed notification transactions:
AlertsDbpasses the current QQQ transaction to table actions. Each alert's timed fan-out, events and progress commit together. A read-onlytimedNotificationCheckedAtfield, written on each attempt, prevents a second attempt in the same minute, including attempts with no eligible recipients; alerts with nothing due are not read (when their type has no reminder and no policy) or not written. Alert and team scans use bounded pages. Runtime ticks are serialized locally; hosts must run the job on one scheduler node, matching the existing single-node scheduling guidance. - Reminders and EXPIRED: null reminder interval means 60 minutes for CRITICAL and off otherwise; explicit zero is off. Reminders use subscription event filters and NOTIFIED timeline entries, without introducing a REMINDER event. Record followers do not receive scheduled reminders without a matching subscription. EXPIRED remains reserved but is excluded from the resolution possible values; the design defines no expiration behavior. No ESCALATION feature switch is present in the current config.
- UI API and record following: the RC.6 alerts extension, collaboration ownership of follow, and machine webhook migration are specified in UI-API.md. Existing RECORD watch subscriptions remain readable without migration.
- Alerts and notifications use core's RC.6 producer API, including sink registration in core instance metadata (QRun-IO/qqq#1108 and #1125).
AlertsSinkAdapterdelegatesraise,resolve,heartbeatandnotifyto the runtime queue. No local SPI types remain; storage-specific audience and severity conversion belongs toNotificationService. - Visibility checks 1 to 3 (section 7.2) are enforced by record security locks on session keys (
alerts_permission,alerts_readable_table), not by a pre-query customizer. QQQ has no pre-query customizer, and locks also apply to counts, exports and the API. - Heartbeats are stored in an extra table,
alerts_heartbeat. - Claims are leases, not atomic conditional updates (see README.md, Known limits).
- Email threading headers wait for core messaging support.
- The delivery log is
alerts_delivery(the design calls italerts_notification), andalerts_inbox_itemis retired: user notifications took those roles (NOTIFICATIONS.md section 2).
User notifications (announcements, job failures, shares, security notices, releases and marketing), the recipient model and the Next UI notification center are designed in NOTIFICATIONS.md, with the owner decisions of 2026-10-13.
Quiet hours and digest delivery (#27, part 1)
- Quiet days identify the start day of the window. Missing/invalid time zones use UTC; equal endpoints disable quiet hours. Pause and quiet hours apply only to personal push channels (email, email lists and other non-IN_APP channels), never to the in-app bell.
- Existing delivery statuses are sufficient: pending
DIGESTitems are excluded from the ordinary worker. Changeset 21 adds window keys, a unique channel/window claim and the claim's last included item id to the standardalerts_deliverytable, mirrored in its entity and H2 DDL. - Digest keys extend the documented user/window prefix with cadence and channel. This avoids collisions between hourly and daily subscriptions with the same start, different channels, and general notification digests. Channel-only digest keys use the separate
alerts:channel-digestprefix and UTC windows; email adds a recipient hash for list channels. The unique claim is atomic even across schedulers; immediate-delivery leases retain their existing behavior. - Delivery has at-most-once dispatch semantics. A crash between claiming and recording a provider result cannot be resolved safely for every supported channel. Such claims fail after ten minutes and are not automatically resent. Transport errors are FAILED; late items after the frozen cutoff are SUPPRESSED/DUPLICATE. The shared email API also receives the durable key.
- Digests summarize alerts represented by events in their window, rather than resending the complete outstanding-alert inventory on every run. Rendering uses MINIMAL detail, checks recipient visibility again (failing closed for unverifiable protected content), and rechecks pause/quiet hours for push channels only. In-app summaries are snapshots of the content visible at dispatch.
- To bound memory, each message accepts 500 source events; overflow fails the window with an actionable error instead of silently dropping items or sending several messages. Each run scans at most 100 due candidates; result updates use pages of 200. This is an explicit v1 limit.
- There is no feature-switch collection or DIGESTS switch in the current config. Reminders, escalation and team member sources are unchanged.