feat: alerts tables, engine, monitors, in-app and email, Alerts home and bell API (P1-P3) #4

2026-10-12T21:58:11Z

KofTwentyTwo commented

2026-10-12 21:58:11 +00:00

(Migrated from github.com)

Summary

This PR implements phases P1 to P3 of the Alerts / manage-by-exception design (QRun-IO/qqq#1090). The result is a working vertical slice: alerts you can see and triage, with email.

Tables. There are 15 alerts_* tables with full record screens, child lists and relations, plus a bundled Liquibase changelog.
- The nav lists the main tables only. The timeline, alert records, team members, inbox items and heartbeats are relations on their parents.
- Security:
  - Personal rows (subscriptions, settings, inbox) are record-locked to their owner.
  - An alert is visible only when the user holds the type's visibility permission and can read its subject table. This is done with record-lock keys, so lists, counts, exports and the API are all filtered.
  - A write guard keeps alerts, the timeline and notifications engine-only.
Engine. Dedup on (type, dedupKey) with a unique openDedupKey and race handling, refresh, severity changes that unsnooze, snooze, resolve, a reopen window, flapping, quiet-period auto-resolve, and maintenance windows that suppress alerts.
Monitors. TABLE_QUERY (aggregate, or per record with a cap), AGGREGATE (bands and group-by), HEARTBEAT (on a key, a process or a table's newest row), PROCESS_FAILED (from the process lifecycle) and CODE (AlertCheckInterface).
- A clean check resolves the monitor's open alerts (full-state diff).
- forMinutes makes a condition hold before it fires, and every check has a timeout.
- Three failed checks in a row raise the alerts.monitor.failing meta-alert.
- Qbits can seed default monitors.
- "Run check now" previews in the Review step and applies in the Result step.
Routing. Subscriptions are matched by scope (all, group, type, monitor, table, record or attribute), with teams expanded to members.
- The bootstrap rule subscribes admins in-app to ERROR and worse.
- Each recipient is checked for visibility through an automated session.
- Suppressions are logged with their reason: type silent, muted, flapping, not visible, user paused, channel disabled, no address, rate limited or duplicate.
Delivery. In-app (the bell inbox), email and email lists go through SendMessageAction, and channels are pluggable through AlertChannelInterface.
- Notifications that arrive together are batched into one message.
- Retries back off 1 min, 5 min, 15 min, 1 h and 4 h, then the notification is DEAD.
- Channel health goes HEALTHY, PROBATION, UNHEALTHY. An UNHEALTHY channel raises alerts.channel.failing, which is sent in-app only so it can't loop.
- Every external message is redacted and capped by the channel's and the type's detail levels.
Actions. Each is a streamed ETL table or record action with Review and Result summary lines:
- alerts: Acknowledge, Snooze, Resolve, Reopen, Assign, Add comment and Mute
- types, groups and monitors: Subscribe me and Mute
- monitors: Run check now, Enable monitor and Disable monitor
- types: Create monitor
- channels: Send test notification
- notifications: Retry delivery
Two hidden scheduled processes run evaluation and delivery.
Alerts home. Open by severity, My alerts, Needs action, By group (with a 7-day sparkline), a 30-day trend with MTTA and MTTR, the noisiest types, delivery health (admins only) and recently resolved. Every widget is full width, uses real data and has an honest empty state. AlertSummaryLoader provides link-only summaries for other homes.
API. /qqq/v1/alerts routes serve the Next UI bell (inbox summary, mark read) and the record panel (open alerts, resolved in the last 30 days, watch).

Dependency on unreleased core. The producer SPI is QRun-IO/qqq#1108, which is not in a released RC yet. Its types are a stand-in in com.kingsrook.qbits.alerts.spi (see package-info.java), marked for replacement when the RC ships. That replacement is mechanical: change the imports and register the sink in core's AlertsInstanceMetaData.

Known limits (v1), also listed in the README:

Claims are leases, so delivery is at-least-once across several scheduler nodes.
Email threading headers need core messaging support.
P4 to P6 are not built: digests, quiet hours, reminders, escalation and the external channels.

Test plan

./mvnw verify: 47 tests, 0 failures, against H2 with the same DDL as the changelog. JaCoCo: 90% of instructions and 100% of classes covered (the gate is 70% and 90%).
The state machine, including a 4-thread race on one key.
Every monitor kind, the full-state diff, forMinutes, CHECK_FAILED and its meta-alert, previews and process failures.
Routing: scopes, teams, bootstrap, every suppression reason, and known versus unknown recipient visibility.
Delivery: batching, redaction, retries to DEAD, channel health, test sends and permanent failures.
Security: visibility by permission and subject-table read, personal rows, the write guard and process permissions.
Actions end to end through RunProcessAction.
The home widgets, empty and with data; the summary loader; the inbox and record-panel API.
Standards: naming, nav contents (no processes), full-width widgets, the changelog matching every column, and permissions.

🤖 Generated with Claude Code

## Summary This PR implements phases P1 to P3 of the Alerts / manage-by-exception design (QRun-IO/qqq#1090). The result is a working vertical slice: alerts you can see and triage, with email. - **Tables.** There are 15 `alerts_*` tables with full record screens, child lists and relations, plus a bundled Liquibase changelog. - The nav lists the main tables only. The timeline, alert records, team members, inbox items and heartbeats are relations on their parents. - Security: - Personal rows (subscriptions, settings, inbox) are record-locked to their owner. - An alert is visible only when the user holds the type's visibility permission and can read its subject table. This is done with record-lock keys, so lists, counts, exports and the API are all filtered. - A write guard keeps alerts, the timeline and notifications engine-only. - **Engine.** Dedup on `(type, dedupKey)` with a unique `openDedupKey` and race handling, refresh, severity changes that unsnooze, snooze, resolve, a reopen window, flapping, quiet-period auto-resolve, and maintenance windows that suppress alerts. - **Monitors.** `TABLE_QUERY` (aggregate, or per record with a cap), `AGGREGATE` (bands and group-by), `HEARTBEAT` (on a key, a process or a table's newest row), `PROCESS_FAILED` (from the process lifecycle) and `CODE` (`AlertCheckInterface`). - A clean check resolves the monitor's open alerts (full-state diff). - `forMinutes` makes a condition hold before it fires, and every check has a timeout. - Three failed checks in a row raise the `alerts.monitor.failing` meta-alert. - Qbits can seed default monitors. - "Run check now" previews in the Review step and applies in the Result step. - **Routing.** Subscriptions are matched by scope (all, group, type, monitor, table, record or attribute), with teams expanded to members. - The bootstrap rule subscribes admins in-app to ERROR and worse. - Each recipient is checked for visibility through an automated session. - Suppressions are logged with their reason: type silent, muted, flapping, not visible, user paused, channel disabled, no address, rate limited or duplicate. - **Delivery.** In-app (the bell inbox), email and email lists go through `SendMessageAction`, and channels are pluggable through `AlertChannelInterface`. - Notifications that arrive together are batched into one message. - Retries back off 1 min, 5 min, 15 min, 1 h and 4 h, then the notification is DEAD. - Channel health goes HEALTHY, PROBATION, UNHEALTHY. An UNHEALTHY channel raises `alerts.channel.failing`, which is sent in-app only so it can't loop. - Every external message is redacted and capped by the channel's and the type's detail levels. - **Actions.** Each is a streamed ETL table or record action with Review and Result summary lines: - alerts: Acknowledge, Snooze, Resolve, Reopen, Assign, Add comment and Mute - types, groups and monitors: Subscribe me and Mute - monitors: Run check now, Enable monitor and Disable monitor - types: Create monitor - channels: Send test notification - notifications: Retry delivery Two hidden scheduled processes run evaluation and delivery. - **Alerts home.** Open by severity, My alerts, Needs action, By group (with a 7-day sparkline), a 30-day trend with MTTA and MTTR, the noisiest types, delivery health (admins only) and recently resolved. Every widget is full width, uses real data and has an honest empty state. `AlertSummaryLoader` provides link-only summaries for other homes. - **API.** `/qqq/v1/alerts` routes serve the Next UI bell (inbox summary, mark read) and the record panel (open alerts, resolved in the last 30 days, watch). **Dependency on unreleased core.** The producer SPI is QRun-IO/qqq#1108, which is not in a released RC yet. Its types are a stand-in in `com.kingsrook.qbits.alerts.spi` (see `package-info.java`), marked for replacement when the RC ships. That replacement is mechanical: change the imports and register the sink in core's `AlertsInstanceMetaData`. **Known limits (v1)**, also listed in the README: - Claims are leases, so delivery is at-least-once across several scheduler nodes. - Email threading headers need core messaging support. - P4 to P6 are not built: digests, quiet hours, reminders, escalation and the external channels. ## Test plan - [x] `./mvnw verify`: 47 tests, 0 failures, against H2 with the same DDL as the changelog. JaCoCo: 90% of instructions and 100% of classes covered (the gate is 70% and 90%). - [x] The state machine, including a 4-thread race on one key. - [x] Every monitor kind, the full-state diff, `forMinutes`, `CHECK_FAILED` and its meta-alert, previews and process failures. - [x] Routing: scopes, teams, bootstrap, every suppression reason, and known versus unknown recipient visibility. - [x] Delivery: batching, redaction, retries to DEAD, channel health, test sends and permanent failures. - [x] Security: visibility by permission and subject-table read, personal rows, the write guard and process permissions. - [x] Actions end to end through `RunProcessAction`. - [x] The home widgets, empty and with data; the summary loader; the inbox and record-panel API. - [x] Standards: naming, nav contents (no processes), full-width widgets, the changelog matching every column, and permissions. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-alerts#4