feat: external channels and Alertmanager ingest (P4) #5

2026-10-12T22:23:13Z

KofTwentyTwo commented

2026-10-12 22:23:13 +00:00

(Migrated from github.com)

Summary

This PR adds the optional external channels from design phase P4 (QRun-IO/qqq#1090, section 5.6) and Alertmanager ingest (section 5.7). Slack stays optional: it's one channel among several, and nothing is pre-wired.

Channel	Secret (`secretRef`)	Sends
`SLACK`	Incoming Webhook URL	Block Kit with an "Open in Alerts" button
`TEAMS`	Workflows webhook URL	An Adaptive Card. The Office 365 connectors are retired.
`WEBHOOK`	Signing key (`whsec_...`)	`{type, subject, severity, link, alerts[]}`, signed with the Standard Webhooks headers
`PAGERDUTY`	Events API v2 routing key	One trigger, acknowledge or resolve event per alert, with `dedup_key` = `type:key`
`ALERTMANAGER`	Optional bearer token	`POST /api/v2/alerts` with the labels `qqq_type`, `qqq_group`, `qqq_key` and `qqq_origin="qbit-alerts"`, and `endsAt` on resolve

Signing. The webhook signature uses the shared format from the qbit-webhooks assessment (QRun-IO/qqq#1105, D2): webhook-signature: v1,<base64 HMAC-SHA256 of id.timestamp.body>. StandardWebhookSigner is a stand-in marked for replacement by core's WebhookSigner (QRun-IO/qqq#1109) once that release ships. The message id is stable across retries, so receivers can use it as an idempotency key.
HTTP rules.
- URLs must be https unless the channel's settings say allowInsecureHttp.
- The timeout is 10 s.
- 2xx is sent. 408, 425, 429, 5xx and network errors are retried. Any other 4xx fails permanently, which feeds the existing retry ladder and channel health.
- A URL never appears in an error message. For Slack and Teams the URL is the secret.
Ingest. POST /qqq/v1/alerts/ingest/alertmanager accepts the Alertmanager webhook payload (version 4).
- It authenticates with a bearer token, set through a secret reference (withAlertmanagerIngestTokenSecretRef). The comparison is constant-time, and ingest is off without a token.
- The alert's fingerprint becomes the dedup key, and firing and resolved become raise and resolve.
- The types are prometheus.<alertname>: EXTERNAL, in group "infrastructure", with notifications off.
- Alerts that carry qqq_origin="qbit-alerts" are ignored, which is the loop guard.
- The body limit is 1 MB.
AlertMessage now carries one structured, redacted entry per alert, capped by the detail level, for the channels that send one event per alert.

Still to do from P4 (listed in the README): digests, quiet hours, reminders, Micrometer metrics, and the ROLE and IdP-group team resolvers. Outbound requests block plain http but aren't fully SSRF-guarded until core's SafeHttpClient (#1109) ships. Channels are admin-only.

Test plan

ExternalChannelsTest runs against a local JDK HTTP receiver. It checks:
- the request shape of each channel
- the webhook signature, recomputed and compared, and its stable id
- the PagerDuty action mapping
- the Alertmanager labels, endsAt and the test message
- outcomes for 503, 429, 404 and no receiver
- https enforcement, and that the URL is kept out of errors
AlertmanagerIngestTest: token checks (valid, invalid, missing, disabled), firing and resolved alerts, the EXTERNAL type with notify off, the loop guard and a bad payload.
./mvnw verify: 54 tests, 0 failures. JaCoCo: 91% of instructions and 100% of classes covered.

🤖 Generated with Claude Code

## Summary This PR adds the optional external channels from design phase P4 (QRun-IO/qqq#1090, section 5.6) and Alertmanager ingest (section 5.7). Slack stays optional: it's one channel among several, and nothing is pre-wired. | Channel | Secret (`secretRef`) | Sends | |---|---|---| | `SLACK` | Incoming Webhook URL | Block Kit with an "Open in Alerts" button | | `TEAMS` | Workflows webhook URL | An Adaptive Card. The Office 365 connectors are retired. | | `WEBHOOK` | Signing key (`whsec_...`) | `{type, subject, severity, link, alerts[]}`, signed with the Standard Webhooks headers | | `PAGERDUTY` | Events API v2 routing key | One trigger, acknowledge or resolve event per alert, with `dedup_key` = `type:key` | | `ALERTMANAGER` | Optional bearer token | `POST /api/v2/alerts` with the labels `qqq_type`, `qqq_group`, `qqq_key` and `qqq_origin="qbit-alerts"`, and `endsAt` on resolve | - **Signing.** The webhook signature uses the shared format from the qbit-webhooks assessment (QRun-IO/qqq#1105, D2): `webhook-signature: v1,<base64 HMAC-SHA256 of id.timestamp.body>`. `StandardWebhookSigner` is a stand-in marked for replacement by core's `WebhookSigner` (QRun-IO/qqq#1109) once that release ships. The message id is stable across retries, so receivers can use it as an idempotency key. - **HTTP rules.** - URLs must be https unless the channel's settings say `allowInsecureHttp`. - The timeout is 10 s. - 2xx is sent. 408, 425, 429, 5xx and network errors are retried. Any other 4xx fails permanently, which feeds the existing retry ladder and channel health. - A URL never appears in an error message. For Slack and Teams the URL is the secret. - **Ingest.** `POST /qqq/v1/alerts/ingest/alertmanager` accepts the Alertmanager webhook payload (version 4). - It authenticates with a bearer token, set through a secret reference (`withAlertmanagerIngestTokenSecretRef`). The comparison is constant-time, and ingest is off without a token. - The alert's fingerprint becomes the dedup key, and `firing` and `resolved` become raise and resolve. - The types are `prometheus.<alertname>`: EXTERNAL, in group "infrastructure", with **notifications off**. - Alerts that carry `qqq_origin="qbit-alerts"` are ignored, which is the loop guard. - The body limit is 1 MB. - `AlertMessage` now carries one structured, redacted entry per alert, capped by the detail level, for the channels that send one event per alert. Still to do from P4 (listed in the README): digests, quiet hours, reminders, Micrometer metrics, and the ROLE and IdP-group team resolvers. Outbound requests block plain http but aren't fully SSRF-guarded until core's `SafeHttpClient` (#1109) ships. Channels are admin-only. ## Test plan - [x] `ExternalChannelsTest` runs against a local JDK HTTP receiver. It checks: - the request shape of each channel - the webhook signature, recomputed and compared, and its stable id - the PagerDuty action mapping - the Alertmanager labels, `endsAt` and the test message - outcomes for 503, 429, 404 and no receiver - https enforcement, and that the URL is kept out of errors - [x] `AlertmanagerIngestTest`: token checks (valid, invalid, missing, disabled), firing and resolved alerts, the EXTERNAL type with notify off, the loop guard and a bad payload. - [x] `./mvnw verify`: 54 tests, 0 failures. JaCoCo: 91% of instructions and 100% of classes covered. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-alerts#5