diff --git a/CHANGELOG.md b/CHANGELOG.md index 5be5535..83ada6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ All notable changes to this project are documented here. The format follows Keep ## [0.1.0] - Unreleased ### Added +- Optional external channels: + - Slack (an Incoming Webhook, Block Kit) + - Microsoft Teams (a Workflows webhook, an Adaptive Card) + - a generic webhook, signed with Standard Webhooks headers through a stand-in for core's `WebhookSigner` (QRun-IO/qqq#1109) + - PagerDuty (Events API v2: trigger, acknowledge and resolve) + - an Alertmanager sink (`/api/v2/alerts`, with the qqq labels and the origin guard) +- Prometheus Alertmanager ingest at `POST /qqq/v1/alerts/ingest/alertmanager`. It takes a bearer token, creates EXTERNAL types with notifications off, and keeps a loop guard. - Tables (`alerts_*`), with the Liquibase changelog `db/liquibase/qbit-alerts/alerts.changelog.sql`: - alerts, their timeline and records - types, groups and monitors diff --git a/README.md b/README.md index ec2a58d..f862c82 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,15 @@ Any qbit or app reports exceptional conditions ("a person should do something"). - `HEARTBEAT`, on a key, a process or a table's newest row - `PROCESS_FAILED` - `CODE`, through `AlertCheckInterface` -- **Notifies subscribers** (users, teams or channels) through in-app (the bell), email and email lists, and through any channel another qbit registers. Every suppressed notification is logged with its reason. +- **Notifies subscribers** (users, teams or channels) through in-app (the bell), email and email lists. These optional channels are also built in: + - Slack (an Incoming Webhook) + - Microsoft Teams (a Workflows webhook) + - a generic signed webhook + - PagerDuty (Events API v2) + - Prometheus Alertmanager, as a sink + + Any channel another qbit registers works too. Every suppressed notification is logged with its reason. +- **Ingests Prometheus Alertmanager** at `POST /qqq/v1/alerts/ingest/alertmanager`, with notifications off for those types, so infrastructure alerts are triaged in one place. - **Owns the Alerts home:** open alerts by severity, my alerts, needs action, by group, a 30-day trend, the noisiest types, delivery health and recently resolved. Other pages get link-only summaries from `AlertSummaryLoader`. All data is in standard QQQ tables named `alerts_*`. People change alerts through record actions, never by editing rows. @@ -78,13 +86,36 @@ Once the RC ships, producers call `QAlerts.raise(...)` from core and need no dep | POST | `/qqq/v1/alerts/inbox/read` | `{"alertId": n}` or `{"all": true}` | | GET | `/qqq/v1/alerts/record-panel?tableName=t&recordId=id` | The "Open alerts" panel for a record (`visible: false` when there is nothing to show) | | POST | `/qqq/v1/alerts/record-panel/watch` | `{"tableName": t, "recordId": id}`: a RECORD subscription for the user | +| POST | `/qqq/v1/alerts/ingest/alertmanager` | Alertmanager's webhook receiver (version 4). It takes a bearer token whose reference is set by `withAlertmanagerIngestTokenSecretRef`, and is off without one. | + +## External channels + +Each external channel is an `alerts_channel` row that an admin creates. Its secret is a reference (`secretRef`, an environment variable name or a `${...}` expression), never the value. The minimum detail level is MINIMAL by default. + +| Type | Secret | Settings | +|---|---|---| +| `SLACK` | The Incoming Webhook URL | none | +| `TEAMS` | The Workflows "when a webhook request is received" URL | none | +| `WEBHOOK` | The signing key (`whsec_...`) | `{"url": "https://receiver"}` | +| `PAGERDUTY` | The Events API v2 routing key | optional `url`, `source` | +| `ALERTMANAGER` | An optional bearer token | `{"url": "https://alertmanager:9093"}` | + +How each one sends: + +- **Webhook** bodies are `{type, subject, severity, link, alerts: [...]}`, signed with the Standard Webhooks headers (`webhook-id`, `webhook-timestamp`, `webhook-signature: v1,`). This is the shared format from the qbit-webhooks assessment (QRun-IO/qqq#1105). The signer is a stand-in until core's `WebhookSigner` (QRun-IO/qqq#1109) is released. +- **PagerDuty** gets one event per alert: `trigger`, `acknowledge` or `resolve`, with `dedup_key` set to the type and key. +- **Alertmanager** alerts carry the labels `qqq_type`, `qqq_group`, `qqq_key` and `qqq_origin="qbit-alerts"`. Ingest ignores alerts with that origin, so nothing loops. +- **Responses:** 2xx is sent. 408, 425, 429, 5xx and network errors are retried. Any other 4xx fails permanently. +- **URLs** must be https, unless the channel's settings say `"allowInsecureHttp": true`, which is meant for local receivers. ## Known limits (v1) - **Claims are best-effort.** Delivery and monitor claims use a lease, not an atomic conditional update. With several scheduler nodes, run the two scheduled processes on one node, or accept at-least-once delivery. - **Email threading headers.** `Message-ID` and `In-Reply-To` need core messaging support for custom headers, so they are not set. - **Recipient visibility.** It is checked through `createAutomatedSessionForUser`. If a host's authentication module can't build a session with permissions for a user, email goes out only for alerts that have no visibility permission and no subject table. In-app delivery is always filtered by the record locks. -- **Not in v1:** digests, quiet hours, reminders, escalation, external channels (Slack, Teams, webhook, PagerDuty, Alertmanager), mirrored lock columns and a configurable table namespace. These are design phases P4 to P6. +- **Outbound HTTP** blocks plain http, but has no full SSRF guard. Channels are admin-only. Core's `SafeHttpClient` (QRun-IO/qqq#1109) replaces the client when it is released. +- **Alertmanager sink:** open alerts are sent with an `endsAt` a week out, not re-posted. +- **Not built yet:** digests, quiet hours, reminders, Micrometer metrics, ROLE and IdP-group team resolvers, escalation, mirrored lock columns and a configurable table namespace. These are the rest of design phases P4 to P6. ## Build diff --git a/src/main/java/com/kingsrook/qbits/alerts/AlertsQBitConfig.java b/src/main/java/com/kingsrook/qbits/alerts/AlertsQBitConfig.java index fb15b0d..0ddd322 100644 --- a/src/main/java/com/kingsrook/qbits/alerts/AlertsQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/alerts/AlertsQBitConfig.java @@ -87,6 +87,8 @@ public class AlertsQBitConfig implements QBitConfig private List channelTypes = new ArrayList<>(); + private String alertmanagerIngestTokenSecretRef; + private Boolean registerRoutes = true; private Boolean synchronousSink = false; @@ -561,6 +563,30 @@ public class AlertsQBitConfig implements QBitConfig + /******************************************************************************* + ** Getter for alertmanagerIngestTokenSecretRef + *******************************************************************************/ + public String getAlertmanagerIngestTokenSecretRef() + { + return (this.alertmanagerIngestTokenSecretRef); + } + + + + /******************************************************************************* + ** Fluent setter for alertmanagerIngestTokenSecretRef: the reference (an + ** environment variable name or ${...} expression, never the value) of the + ** bearer token Alertmanager sends to POST /qqq/v1/alerts/ingest/alertmanager. + ** Null (the default) turns ingest off. + *******************************************************************************/ + public AlertsQBitConfig withAlertmanagerIngestTokenSecretRef(String alertmanagerIngestTokenSecretRef) + { + this.alertmanagerIngestTokenSecretRef = alertmanagerIngestTokenSecretRef; + return (this); + } + + + /******************************************************************************* ** Getter for runtime (built by the producer) *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/alerts/api/AlertsRouteProvider.java b/src/main/java/com/kingsrook/qbits/alerts/api/AlertsRouteProvider.java index e9dc887..f3887ea 100644 --- a/src/main/java/com/kingsrook/qbits/alerts/api/AlertsRouteProvider.java +++ b/src/main/java/com/kingsrook/qbits/alerts/api/AlertsRouteProvider.java @@ -42,6 +42,8 @@ import io.javalin.http.Context; ** - POST /inbox/read {"alertId": n} or {"all": true} ** - GET /record-panel?tableName=t&recordId=id the "Open alerts" panel ** - POST /record-panel/watch {"tableName": t, "recordId": id} + ** - POST /ingest/alertmanager Alertmanager's webhook (bearer token; off + ** unless alertmanagerIngestTokenSecretRef is configured) ** Each request gets a session from the instance's authentication module; ** AlertsApi checks alerts.view and works in that session. Errors map ** through QJavalinUtils.handleException (400, 401, 403, 404). POST bodies @@ -51,6 +53,8 @@ public class AlertsRouteProvider implements QJavalinRouteProviderInterface { public static final String BASE_PATH = "/qqq/v1/alerts"; + static final int MAX_INGEST_BYTES = 1024 * 1024; + private QInstance qInstance; @@ -92,6 +96,36 @@ public class AlertsRouteProvider implements QJavalinRouteProviderInterface config.routes.post(BASE_PATH + "/inbox/read", context -> respond(context, true, AlertsApi::markRead)); config.routes.get(BASE_PATH + "/record-panel", context -> respond(context, false, (api, body) -> api.recordPanel(context.queryParam("tableName"), context.queryParam("recordId")))); config.routes.post(BASE_PATH + "/record-panel/watch", context -> respond(context, true, AlertsApi::watchRecord)); + config.routes.post(BASE_PATH + "/ingest/alertmanager", this::ingestAlertmanager); + } + + + + /******************************************************************************* + ** Alertmanager's webhook (no QQQ session: a bearer token instead). + *******************************************************************************/ + private void ingestAlertmanager(Context context) + { + try + { + if(context.body().length() > MAX_INGEST_BYTES) + { + throw (new QBadRequestException("The payload is larger than 1 MB.")); + } + com.kingsrook.qbits.alerts.engine.AlertmanagerIngest ingest = new com.kingsrook.qbits.alerts.engine.AlertmanagerIngest(AlertsRuntime.get(qInstance)); + ingest.authenticate(context.header("Authorization")); + com.kingsrook.qbits.alerts.engine.AlertmanagerIngest.Result result = ingest.ingest(readJsonBody(context)); + context.contentType(ContentType.APPLICATION_JSON); + context.result(JsonUtils.toJson(Map.of("raised", result.raised(), "resolved", result.resolved(), "ignored", result.ignored()))); + } + catch(Exception e) + { + QJavalinUtils.handleException(null, context, e); + } + finally + { + QContext.clear(); + } } diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertChannels.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertChannels.java index c2accb0..a33bd71 100644 --- a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertChannels.java +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertChannels.java @@ -34,8 +34,8 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; /******************************************************************************* - ** The channel types (built-in in-app, email and email list, plus any the - ** config adds) and the two default channel rows every install has: "in-app" + ** The channel types (built-in in-app, email, email list, Slack, Teams, + ** webhook, PagerDuty and Alertmanager, plus any the config adds) and the two default channel rows every install has: "in-app" ** (the bell; always available) and "email" (a person's own address). *******************************************************************************/ public class AlertChannels @@ -60,6 +60,10 @@ public class AlertChannels register(new InAppChannel()); register(new EmailChannel(runtime.getConfig(), ChannelType.EMAIL)); register(new EmailChannel(runtime.getConfig(), ChannelType.EMAIL_LIST)); + for(AlertChannelInterface external : com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels.all()) + { + register(external); + } for(QCodeReference reference : runtime.getConfig().getChannelTypes()) { try diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessage.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessage.java index 7205e90..32f2465 100644 --- a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessage.java +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessage.java @@ -31,7 +31,48 @@ import java.util.List; ** @param kind RAISED, RESOLVED, REMINDER, ESCALATED, DIGEST or TEST ** @param alertIds the alerts in the message ** @param link a link to the alert (or the Alerts list) in the app + ** @param entries one structured entry per alert, for channels that send + ** one event per alert (PagerDuty, Alertmanager, webhooks); + ** already redacted and capped by the detail level *******************************************************************************/ -public record AlertMessage(String subject, String text, String html, String severity, String kind, List alertIds, String link) +public record AlertMessage(String subject, String text, String html, String severity, String kind, List alertIds, String link, List entries) { + /******************************************************************************* + ** A message without structured entries. + *******************************************************************************/ + public AlertMessage(String subject, String text, String html, String severity, String kind, List alertIds, String link) + { + this(subject, text, html, severity, kind, alertIds, link, List.of()); + } + + + + /******************************************************************************* + ** One alert, as structured channels send it. + ** + ** @param alertId the alert's id (null for a test message) + ** @param alertTypeName the type, e.g. cdn.purge.dead + ** @param typeLabel the type's label + ** @param dedupKey the producer's dedup key + ** @param severity INFO, WARNING, ERROR or CRITICAL + ** @param status OPEN, ACKNOWLEDGED, SNOOZED or RESOLVED + ** @param title the redacted title + ** @param summary the redacted summary (null at MINIMAL) + ** @param group the group's label + ** @param itemCount the record count (null at MINIMAL) + ** @param link the alert in the app + ** @param firstRaisedAt when it opened (ISO) + ** @param resolvedAt when it resolved (ISO), or null + *******************************************************************************/ + public record Entry(Integer alertId, String alertTypeName, String typeLabel, String dedupKey, String severity, String status, String title, String summary, + String group, Integer itemCount, String link, String firstRaisedAt, String resolvedAt) + { + /*************************************************************************** + ** The stable key for systems that dedup (PagerDuty, Alertmanager). + ***************************************************************************/ + public String stableKey() + { + return ((alertTypeName == null ? "alerts.test" : alertTypeName) + ":" + (dedupKey == null ? String.valueOf(alertId) : dedupKey)); + } + } } diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessageRenderer.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessageRenderer.java index 327f7ab..50f463c 100644 --- a/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessageRenderer.java +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/AlertMessageRenderer.java @@ -147,7 +147,22 @@ public final class AlertMessageRenderer html.append("

Change or stop these emails. ") .append("Personal data and secrets are removed from alert messages.

"); - return (new AlertMessage(subject, text.toString(), html.toString(), highest.name(), kind.name(), alertIds, link)); + List entries = new ArrayList<>(); + for(Item item : items) + { + QRecord alert = item.alert(); + DetailLevel level = item.level() == null ? DetailLevel.MINIMAL : item.level(); + boolean standard = level.compareTo(DetailLevel.STANDARD) >= 0; + Integer id = alert.getValueInteger("id"); + entries.add(new AlertMessage.Entry(id, alert.getValueString("alertTypeName"), safe(item.typeLabel()), alert.getValueString("dedupKey"), + AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING).name(), + kind == NotificationKind.RESOLVED ? "RESOLVED" : (alert.getValueString("status") == null ? "OPEN" : alert.getValueString("status")), + safe(alert.getValueString("title")), standard ? AlertRedactor.redact(alert.getValueString("summary")) : null, item.groupLabel(), + standard ? alert.getValueInteger("itemCount") : null, id == null ? base + "/app/" + AlertsAlert.TABLE_NAME : base + "/app/" + AlertsAlert.TABLE_NAME + "/" + id, + alert.getValueInstant("firstRaisedAt") == null ? null : alert.getValueInstant("firstRaisedAt").toString(), + alert.getValueInstant("resolvedAt") == null ? null : alert.getValueInstant("resolvedAt").toString())); + } + return (new AlertMessage(subject, text.toString(), html.toString(), highest.name(), kind.name(), alertIds, link, entries)); } diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/ExternalChannels.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/ExternalChannels.java new file mode 100644 index 0000000..307e7dd --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/ExternalChannels.java @@ -0,0 +1,448 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.delivery.channels; + + +import java.io.Serializable; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.HexFormat; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import com.kingsrook.qbits.alerts.delivery.AlertChannelInterface; +import com.kingsrook.qbits.alerts.delivery.AlertMessage; +import com.kingsrook.qbits.alerts.delivery.AlertSecretResolver; +import com.kingsrook.qbits.alerts.delivery.DeliveryResult; +import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.ChannelType; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.utils.StringUtils; + + +/******************************************************************************* + ** The optional external channels (design section 5.6). None is required; + ** each is an admin-configured alerts_channel row whose secret is a + ** reference (secretRef), resolved at send time: + ** - SLACK: an Incoming Webhook URL (the URL is the secret); Block Kit body. + ** - TEAMS: a Teams Workflows "when a webhook request is received" URL (the + ** secret); an Adaptive Card. (Office 365 connectors are retired.) + ** - WEBHOOK: settings {"url": ...}; the secret is the HMAC signing key; a + ** documented JSON body signed with the Standard Webhooks headers (the + ** shared format of QRun-IO/qqq#1105, see StandardWebhookSigner). + ** - PAGERDUTY: the Events API v2 routing key (the secret); one + ** trigger / acknowledge / resolve event per alert, dedup_key = type:key. + ** - ALERTMANAGER: settings {"url": "https://alertmanager:9093"} (optional + ** bearer secret); POST /api/v2/alerts with the qqq_* labels and + ** qqq_origin="qbit-alerts" (the ingest loop guard); endsAt on resolve. + ** No vendor SDKs: plain java.net.http and JSON. + *******************************************************************************/ +public final class ExternalChannels +{ + public static final String ORIGIN_LABEL = "qqq_origin"; + public static final String ORIGIN_VALUE = "qbit-alerts"; + + static final String PAGERDUTY_URL = "https://events.pagerduty.com/v2/enqueue"; + + + + /******************************************************************************* + ** Channels only. + *******************************************************************************/ + private ExternalChannels() + { + } + + + + /******************************************************************************* + ** All five, for registration. + *******************************************************************************/ + public static List all() + { + return (List.of(new Slack(), new Teams(), new Webhook(), new PagerDuty(), new Alertmanager())); + } + + + + /******************************************************************************* + ** Slack Incoming Webhook. + *******************************************************************************/ + public static class Slack implements AlertChannelInterface + { + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public String getType() + { + return (ChannelType.SLACK.name()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public List validate(QRecord channel, AlertSecretResolver secrets) + { + return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the Slack Incoming Webhook URL.") : List.of()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets) + { + Map body = new LinkedHashMap<>(); + body.put("text", message.subject()); + body.put("blocks", List.of( + Map.of("type", "section", "text", Map.of("type", "mrkdwn", "text", "*" + slackEscape(message.subject()) + "*\n" + slackEscape(truncate(message.text(), 2800)))), + Map.of("type", "actions", "elements", List.of(Map.of("type", "button", "text", Map.of("type", "plain_text", "text", "Open in Alerts"), "url", message.link()))))); + return (HttpChannelSupport.postJson(secrets.resolve(channel.getValueString("secretRef")), null, HttpChannelSupport.json(body), HttpChannelSupport.allowInsecure(channel))); + } + } + + + + /******************************************************************************* + ** Microsoft Teams through a Workflows webhook. + *******************************************************************************/ + public static class Teams implements AlertChannelInterface + { + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public String getType() + { + return (ChannelType.TEAMS.name()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public List validate(QRecord channel, AlertSecretResolver secrets) + { + return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the Teams Workflows webhook URL.") : List.of()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets) + { + Map card = new LinkedHashMap<>(); + card.put("$schema", "http://adaptivecards.io/schemas/adaptive-card.json"); + card.put("type", "AdaptiveCard"); + card.put("version", "1.4"); + card.put("body", List.of( + Map.of("type", "TextBlock", "text", message.subject(), "weight", "Bolder", "wrap", true), + Map.of("type", "TextBlock", "text", truncate(message.text(), 2800), "wrap", true))); + card.put("actions", List.of(Map.of("type", "Action.OpenUrl", "title", "Open in Alerts", "url", message.link()))); + Map body = Map.of("type", "message", "attachments", List.of(Map.of("contentType", "application/vnd.microsoft.card.adaptive", "content", card))); + return (HttpChannelSupport.postJson(secrets.resolve(channel.getValueString("secretRef")), null, HttpChannelSupport.json(body), HttpChannelSupport.allowInsecure(channel))); + } + } + + + + /******************************************************************************* + ** Generic signed webhook. + *******************************************************************************/ + public static class Webhook implements AlertChannelInterface + { + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public String getType() + { + return (ChannelType.WEBHOOK.name()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public List validate(QRecord channel, AlertSecretResolver secrets) + { + List problems = new ArrayList<>(); + if(!StringUtils.hasContent(Objects.toString(HttpChannelSupport.settings(channel).get("url"), null))) + { + problems.add("Add the receiver's URL to the channel's settings: {\"url\": \"https://...\"}."); + } + if(secrets.resolve(channel.getValueString("secretRef")) == null) + { + problems.add("Set the secret reference to the signing key (a whsec_ secret)."); + } + return (problems); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets) + { + List problems = validate(channel, secrets); + if(!problems.isEmpty()) + { + return (DeliveryResult.failed(problems.get(0))); + } + Map body = new LinkedHashMap<>(); + body.put("type", "qqq.alert." + message.kind().toLowerCase(Locale.ROOT)); + body.put("subject", message.subject()); + body.put("severity", message.severity()); + body.put("link", message.link()); + body.put("alerts", message.entries()); + byte[] bytes = HttpChannelSupport.json(body); + Map headers = StandardWebhookSigner.headers(messageId(message), Instant.now(), bytes, secrets.resolve(channel.getValueString("secretRef"))); + return (HttpChannelSupport.postJson(Objects.toString(HttpChannelSupport.settings(channel).get("url")), headers, bytes, HttpChannelSupport.allowInsecure(channel))); + } + } + + + + /******************************************************************************* + ** PagerDuty Events API v2. + *******************************************************************************/ + public static class PagerDuty implements AlertChannelInterface + { + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public String getType() + { + return (ChannelType.PAGERDUTY.name()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public List validate(QRecord channel, AlertSecretResolver secrets) + { + return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the PagerDuty Events API v2 routing key.") : List.of()); + } + + + + /*************************************************************************** + ** One event per alert; the first failure decides the outcome. + ***************************************************************************/ + @Override + public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets) + { + String routingKey = secrets.resolve(channel.getValueString("secretRef")); + if(routingKey == null) + { + return (DeliveryResult.failed(validate(channel, secrets).get(0))); + } + Map settings = HttpChannelSupport.settings(channel); + String url = Objects.toString(settings.get("url"), PAGERDUTY_URL); + String source = Objects.toString(settings.get("source"), ORIGIN_VALUE); + for(AlertMessage.Entry entry : entries(message)) + { + String action = "TEST".equals(message.kind()) ? "trigger" : switch(entry.status()) + { + case "RESOLVED" -> "resolve"; + case "ACKNOWLEDGED" -> "acknowledge"; + default -> "trigger"; + }; + Map event = new LinkedHashMap<>(); + event.put("routing_key", routingKey); + event.put("event_action", action); + event.put("dedup_key", entry.stableKey()); + if("trigger".equals(action)) + { + Map payload = new LinkedHashMap<>(); + payload.put("summary", truncate(entry.title(), 1024)); + payload.put("source", source); + payload.put("severity", entry.severity().toLowerCase(Locale.ROOT)); + payload.put("component", entry.typeLabel()); + payload.put("group", entry.group()); + Map details = new LinkedHashMap<>(); + details.put("summary", entry.summary()); + details.put("records", entry.itemCount()); + details.put("link", entry.link()); + details.values().removeIf(Objects::isNull); + payload.put("custom_details", details); + payload.values().removeIf(Objects::isNull); + event.put("payload", payload); + event.put("links", List.of(Map.of("href", entry.link(), "text", "Open in Alerts"))); + } + DeliveryResult result = HttpChannelSupport.postJson(url, null, HttpChannelSupport.json(event), HttpChannelSupport.allowInsecure(channel)); + if(result.outcome() != DeliveryResult.Outcome.SENT) + { + return (result); + } + } + return (DeliveryResult.sent(null)); + } + } + + + + /******************************************************************************* + ** Prometheus Alertmanager as a sink. + *******************************************************************************/ + public static class Alertmanager implements AlertChannelInterface + { + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public String getType() + { + return (ChannelType.ALERTMANAGER.name()); + } + + + + /*************************************************************************** + ** + ***************************************************************************/ + @Override + public List validate(QRecord channel, AlertSecretResolver secrets) + { + return (StringUtils.hasContent(Objects.toString(HttpChannelSupport.settings(channel).get("url"), null)) ? List.of() + : List.of("Add Alertmanager's base URL to the channel's settings: {\"url\": \"https://alertmanager:9093\"}.")); + } + + + + /*************************************************************************** + ** One POST with every alert. Open alerts get an endsAt a week out, so + ** they stay active without a re-post; a resolve sends endsAt = when. + ***************************************************************************/ + @Override + public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets) + { + List problems = validate(channel, secrets); + if(!problems.isEmpty()) + { + return (DeliveryResult.failed(problems.get(0))); + } + List> alerts = new ArrayList<>(); + for(AlertMessage.Entry entry : entries(message)) + { + Map labels = new LinkedHashMap<>(); + labels.put("alertname", entry.alertTypeName() == null ? "alerts.test" : entry.alertTypeName()); + labels.put("severity", entry.severity().toLowerCase(Locale.ROOT)); + labels.put("qqq_type", Objects.toString(entry.alertTypeName(), "alerts.test")); + labels.put("qqq_group", Objects.toString(entry.group(), "")); + labels.put("qqq_key", Objects.toString(entry.dedupKey(), "")); + labels.put(ORIGIN_LABEL, ORIGIN_VALUE); + Map alert = new LinkedHashMap<>(); + alert.put("labels", labels); + alert.put("annotations", Map.of("summary", entry.title(), "qqq_url", entry.link())); + alert.put("generatorURL", entry.link()); + if(entry.firstRaisedAt() != null) + { + alert.put("startsAt", entry.firstRaisedAt()); + } + boolean resolved = "RESOLVED".equals(entry.status()) || "TEST".equals(message.kind()); + alert.put("endsAt", resolved ? Objects.toString(entry.resolvedAt(), Instant.now().toString()) : Instant.now().plus(7, ChronoUnit.DAYS).toString()); + alerts.add(alert); + } + String bearer = secrets.resolve(channel.getValueString("secretRef")); + Map headers = bearer == null ? null : Map.of("Authorization", "Bearer " + bearer); + String base = Objects.toString(HttpChannelSupport.settings(channel).get("url")).replaceAll("/+$", ""); + return (HttpChannelSupport.postJson(base + "/api/v2/alerts", headers, HttpChannelSupport.json(alerts), HttpChannelSupport.allowInsecure(channel))); + } + } + + + + /******************************************************************************* + ** The message's entries (a TEST message gets one stand-in entry). + *******************************************************************************/ + static List entries(AlertMessage message) + { + if(message.entries() != null && !message.entries().isEmpty()) + { + return (message.entries()); + } + return (List.of(new AlertMessage.Entry(null, "alerts.test", "Test", "test", "INFO", "OPEN", message.subject(), null, null, null, message.link(), null, null))); + } + + + + /******************************************************************************* + ** A stable id for a webhook delivery (the same on every retry): a hash of + ** the kind and each alert's key and state. + *******************************************************************************/ + static String messageId(AlertMessage message) + { + StringBuilder content = new StringBuilder(message.kind()); + for(AlertMessage.Entry entry : entries(message)) + { + content.append('|').append(entry.alertId()).append(':').append(entry.status()).append(':').append(entry.firstRaisedAt()).append(':').append(entry.resolvedAt()); + } + try + { + byte[] digest = MessageDigest.getInstance("SHA-256").digest(content.toString().getBytes(StandardCharsets.UTF_8)); + return ("msg_" + HexFormat.of().formatHex(digest).substring(0, 32)); + } + catch(Exception e) + { + return ("msg_" + Integer.toHexString(content.toString().hashCode())); + } + } + + + + /******************************************************************************* + ** Slack mrkdwn escaping (&, <, >). + *******************************************************************************/ + static String slackEscape(String text) + { + return (text == null ? "" : text.replace("&", "&").replace("<", "<").replace(">", ">")); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + static String truncate(String text, int max) + { + return (text == null || text.length() <= max ? text : text.substring(0, max - 1) + "…"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/HttpChannelSupport.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/HttpChannelSupport.java new file mode 100644 index 0000000..6c7885b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/HttpChannelSupport.java @@ -0,0 +1,192 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.delivery.channels; + + +import java.io.Serializable; +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.LinkedHashMap; +import java.util.Locale; +import java.util.Map; +import com.kingsrook.qbits.alerts.delivery.AlertRedactor; +import com.kingsrook.qbits.alerts.delivery.DeliveryResult; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.utils.JsonUtils; +import com.kingsrook.qqq.backend.core.utils.StringUtils; + + +/******************************************************************************* + ** Shared HTTP plumbing for the external channels: one pooled client, a 10 + ** second timeout, JSON bodies, and the outcome rules: + ** - 2xx: SENT; + ** - 408, 425, 429 and 5xx, or a network error: RETRY; + ** - any other 4xx (a bad URL or key): FAILED, permanently. + ** URLs must be https unless the channel's settings say "allowInsecureHttp" + ** (for local receivers and tests). Error text is redacted and never includes + ** the URL, which for Slack and Teams is itself the secret. + *******************************************************************************/ +public final class HttpChannelSupport +{ + static final Duration TIMEOUT = Duration.ofSeconds(10); + + private static final HttpClient CLIENT = HttpClient.newBuilder() + .connectTimeout(TIMEOUT) + .followRedirects(HttpClient.Redirect.NEVER) + .build(); + + + + /******************************************************************************* + ** Helpers only. + *******************************************************************************/ + private HttpChannelSupport() + { + } + + + + /******************************************************************************* + ** POST a JSON body and classify the response. + *******************************************************************************/ + public static DeliveryResult postJson(String url, Map headers, byte[] body, boolean allowInsecure) + { + String problem = checkUrl(url, allowInsecure); + if(problem != null) + { + return (DeliveryResult.failed(problem)); + } + try + { + HttpRequest.Builder request = HttpRequest.newBuilder(URI.create(url)) + .timeout(TIMEOUT) + .header("Content-Type", "application/json") + .header("User-Agent", "qbit-alerts") + .POST(HttpRequest.BodyPublishers.ofByteArray(body)); + if(headers != null) + { + headers.forEach(request::header); + } + HttpResponse response = CLIENT.send(request.build(), HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)); + return (classify(response.statusCode(), response.body())); + } + catch(InterruptedException e) + { + Thread.currentThread().interrupt(); + return (DeliveryResult.retry("Interrupted while sending.")); + } + catch(Exception e) + { + return (DeliveryResult.retry(AlertRedactor.redact(e.getClass().getSimpleName() + (e.getMessage() == null ? "" : ": " + stripUrl(e.getMessage(), url))))); + } + } + + + + /******************************************************************************* + ** SENT, RETRY or FAILED for an HTTP status. + *******************************************************************************/ + static DeliveryResult classify(int status, String body) + { + if(status >= 200 && status < 300) + { + return (DeliveryResult.sent(null)); + } + String detail = "HTTP " + status + (StringUtils.hasContent(body) ? ": " + AlertRedactor.redact(StringUtils.safeTruncate(body.trim(), 200, "…")) : ""); + if(status == 408 || status == 425 || status == 429 || status >= 500) + { + return (DeliveryResult.retry(detail)); + } + return (DeliveryResult.failed(detail)); + } + + + + /******************************************************************************* + ** Why a URL can't be used (null when it can). + *******************************************************************************/ + static String checkUrl(String url, boolean allowInsecure) + { + if(!StringUtils.hasContent(url)) + { + return ("No URL is configured (check the channel's secret reference or settings)."); + } + String lower = url.trim().toLowerCase(Locale.ROOT); + if(lower.startsWith("https://") || (allowInsecure && lower.startsWith("http://"))) + { + return (null); + } + return ("The URL must use https."); + } + + + + /******************************************************************************* + ** The channel's settings JSON as a map (empty when blank or unreadable). + *******************************************************************************/ + @SuppressWarnings("unchecked") + public static Map settings(QRecord channel) + { + String json = channel == null ? null : channel.getValueString("settings"); + if(!StringUtils.hasContent(json)) + { + return (new LinkedHashMap<>()); + } + try + { + return (JsonUtils.toObject(json, LinkedHashMap.class)); + } + catch(Exception e) + { + return (new LinkedHashMap<>()); + } + } + + + + /******************************************************************************* + ** Whether the channel allows plain http (local receivers, tests). + *******************************************************************************/ + public static boolean allowInsecure(QRecord channel) + { + return (Boolean.TRUE.equals(settings(channel).get("allowInsecureHttp"))); + } + + + + /******************************************************************************* + ** JSON bytes. + *******************************************************************************/ + public static byte[] json(Object body) + { + return (JsonUtils.toJson(body).getBytes(StandardCharsets.UTF_8)); + } + + + + /******************************************************************************* + ** Remove a (secret) URL from an error message. + *******************************************************************************/ + private static String stripUrl(String message, String url) + { + return (url == null ? message : message.replace(url, "[url]")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/StandardWebhookSigner.java b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/StandardWebhookSigner.java new file mode 100644 index 0000000..9be6e1b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/alerts/delivery/channels/StandardWebhookSigner.java @@ -0,0 +1,111 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.delivery.channels; + + +import javax.crypto.Mac; +import javax.crypto.spec.SecretKeySpec; +import java.nio.charset.StandardCharsets; +import java.security.GeneralSecurityException; +import java.time.Instant; +import java.util.Base64; +import java.util.LinkedHashMap; +import java.util.Map; + + +/******************************************************************************* + ** STAND-IN for core's WebhookSigner (QRun-IO/qqq#1109, the shared signer of + ** the qbit-webhooks assessment, QRun-IO/qqq#1105 D2). Same format: the + ** Standard Webhooks headers webhook-id, webhook-timestamp and + ** webhook-signature = "v1," + base64(HMAC-SHA256(key, id + "." + timestamp + + ** "." + body)); a "whsec_" secret is base64 after the prefix, any other text + ** is the raw key. Replace with com.kingsrook.qqq.backend.core.utils.webhooks + ** .WebhookSigner when the QQQ RC with #1109 ships, so receivers verify Alerts + ** and qbit-webhooks deliveries the same way. + *******************************************************************************/ +public final class StandardWebhookSigner +{ + public static final String HEADER_ID = "webhook-id"; + public static final String HEADER_TIMESTAMP = "webhook-timestamp"; + public static final String HEADER_SIGNATURE = "webhook-signature"; + + private static final String PREFIX = "whsec_"; + + + + /******************************************************************************* + ** Signing only. + *******************************************************************************/ + private StandardWebhookSigner() + { + } + + + + /******************************************************************************* + ** The three signature headers. + *******************************************************************************/ + public static Map headers(String messageId, Instant timestamp, byte[] body, String secret) + { + long seconds = timestamp.getEpochSecond(); + Map headers = new LinkedHashMap<>(); + headers.put(HEADER_ID, messageId); + headers.put(HEADER_TIMESTAMP, Long.toString(seconds)); + headers.put(HEADER_SIGNATURE, "v1," + sign(messageId, seconds, body, secret)); + return (headers); + } + + + + /******************************************************************************* + ** base64 HMAC-SHA256 over id.timestamp.body. + *******************************************************************************/ + public static String sign(String messageId, long timestampSeconds, byte[] body, String secret) + { + if(messageId == null || messageId.isEmpty() || messageId.contains(".") || messageId.chars().anyMatch(Character::isWhitespace)) + { + throw (new IllegalArgumentException("A webhook message id is required and may not contain '.' or whitespace")); + } + try + { + Mac mac = Mac.getInstance("HmacSHA256"); + mac.init(new SecretKeySpec(keyBytes(secret), "HmacSHA256")); + mac.update((messageId + "." + timestampSeconds + ".").getBytes(StandardCharsets.UTF_8)); + mac.update(body == null ? new byte[0] : body); + return (Base64.getEncoder().encodeToString(mac.doFinal())); + } + catch(GeneralSecurityException e) + { + throw (new IllegalStateException("HMAC-SHA256 is unavailable", e)); + } + } + + + + /******************************************************************************* + ** The key bytes of a secret's text form. + *******************************************************************************/ + static byte[] keyBytes(String secret) + { + if(secret == null || secret.isBlank()) + { + throw (new IllegalArgumentException("A webhook signing secret is required")); + } + String trimmed = secret.trim(); + return (trimmed.startsWith(PREFIX) ? Base64.getDecoder().decode(trimmed.substring(PREFIX.length())) : trimmed.getBytes(StandardCharsets.UTF_8)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngest.java b/src/main/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngest.java new file mode 100644 index 0000000..52daf9e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngest.java @@ -0,0 +1,256 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.engine; + + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import com.kingsrook.qbits.alerts.AlertsRuntime; +import com.kingsrook.qbits.alerts.delivery.AlertRedactor; +import com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels; +import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.Resolution; +import com.kingsrook.qbits.alerts.model.AlertsType; +import com.kingsrook.qbits.alerts.spi.AlertSeverity; +import com.kingsrook.qbits.alerts.spi.AlertSignal; +import com.kingsrook.qbits.alerts.spi.DetailLevel; +import com.kingsrook.qbits.alerts.spi.QAlertTypeMetaData; +import com.kingsrook.qbits.alerts.spi.ResolveMode; +import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException; +import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.utils.StringUtils; + + +/******************************************************************************* + ** Prometheus Alertmanager as a source (design section 5.7): the webhook + ** receiver payload (version 4) becomes raises and resolves. + ** - the alert's fingerprint is the dedup key; firing / resolved are raise / + ** resolve; the severity label maps to a severity (unknown: WARNING); + ** summary / description become title / summary (redacted); labels become + ** attributes (bounded); + ** - the type is prometheus.(alertname), created on first sight as an + ** EXTERNAL type in group "infrastructure" with notifications OFF + ** (Alertmanager already notifies people; turning QQQ notifications on is a + ** per-type choice); + ** - alerts carrying qqq_origin="qbit-alerts" (sent by this qbit's + ** Alertmanager channel) are ignored, so nothing loops. + ** Authentication: a bearer token compared (in constant time) with the + ** configured secret reference. Ingest is off without one. + *******************************************************************************/ +public class AlertmanagerIngest +{ + public static final String TYPE_PREFIX = "prometheus."; + public static final String GROUP = "infrastructure"; + + private final AlertsRuntime runtime; + + + + /******************************************************************************* + ** What an ingest did. + ** + ** @param raised alerts raised (or refreshed) + ** @param resolved alerts resolved + ** @param ignored alerts ignored (loop guard, or no alertname) + *******************************************************************************/ + public record Result(int raised, int resolved, int ignored) + { + } + + + + /******************************************************************************* + ** Constructor + *******************************************************************************/ + public AlertmanagerIngest(AlertsRuntime runtime) + { + this.runtime = runtime; + } + + + + /******************************************************************************* + ** Check the bearer token: 404 when ingest is off, 401 when it doesn't match. + *******************************************************************************/ + public void authenticate(String authorizationHeader) throws QException + { + String ref = runtime.getConfig().getAlertmanagerIngestTokenSecretRef(); + if(!StringUtils.hasContent(ref)) + { + throw (new QNotFoundException("Alertmanager ingest is not enabled.")); + } + String expected = runtime.getChannels().getSecrets().resolve(ref); + String given = authorizationHeader != null && authorizationHeader.regionMatches(true, 0, "Bearer ", 0, 7) ? authorizationHeader.substring(7).trim() : null; + if(expected == null || given == null || !MessageDigest.isEqual(expected.getBytes(StandardCharsets.UTF_8), given.getBytes(StandardCharsets.UTF_8))) + { + throw (new QAuthenticationException("Invalid ingest token.")); + } + } + + + + /******************************************************************************* + ** Apply one webhook payload. + *******************************************************************************/ + @SuppressWarnings("unchecked") + public Result ingest(Map payload) throws QException + { + Object alerts = payload == null ? null : payload.get("alerts"); + if(!(alerts instanceof List list)) + { + throw (new QBadRequestException("Expected an Alertmanager webhook payload with an \"alerts\" list.")); + } + + int raised = 0; + int resolved = 0; + int ignored = 0; + for(Object item : list) + { + if(!(item instanceof Map alert)) + { + ignored++; + continue; + } + Map labels = alert.get("labels") instanceof Map map ? (Map) map : Map.of(); + Map annotations = alert.get("annotations") instanceof Map map ? (Map) map : Map.of(); + String alertname = labels.get("alertname") == null ? null : String.valueOf(labels.get("alertname")); + if(alertname == null || ExternalChannels.ORIGIN_VALUE.equals(String.valueOf(labels.get(ExternalChannels.ORIGIN_LABEL)))) + { + ignored++; + continue; + } + + String typeName = typeName(alertname); + String fingerprint = alert.get("fingerprint") == null ? fingerprint(labels) : String.valueOf(alert.get("fingerprint")); + ensureType(typeName, alertname); + + if("resolved".equalsIgnoreCase(String.valueOf(alert.get("status")))) + { + if(runtime.getEngine().resolve(typeName, fingerprint, Resolution.PRODUCER, null, "Resolved in Alertmanager") != null) + { + resolved++; + } + continue; + } + + String title = firstNonBlank(annotations.get("summary"), annotations.get("description"), alertname); + AlertSignal signal = new AlertSignal(typeName) + .withDedupKey(fingerprint) + .withSeverity(AlertSeverity.fromLabel(labels.get("severity") == null ? null : String.valueOf(labels.get("severity")), AlertSeverity.WARNING)) + .withTitle(AlertRedactor.redact(title)) + .withSummary(AlertRedactor.redact(annotations.get("description") == null ? null : String.valueOf(annotations.get("description")))) + .withOccurredAt(parseInstant(alert.get("startsAt"))); + for(Map.Entry label : new LinkedHashMap<>(labels).entrySet()) + { + signal.withAttribute(label.getKey(), String.valueOf(label.getValue())); + } + runtime.getEngine().raise(signal); + raised++; + } + return (new Result(raised, resolved, ignored)); + } + + + + /******************************************************************************* + ** Create the EXTERNAL type the first time (notifications off). + *******************************************************************************/ + private void ensureType(String typeName, String alertname) throws QException + { + runtime.asSystem(() -> + { + if(AlertsDb.first(AlertsType.TABLE_NAME, AlertsDb.eq("name", typeName)) == null) + { + QRecord type = runtime.getTypeSync().upsertType(new QAlertTypeMetaData() + .withName(typeName) + .withLabel(alertname) + .withSourceName("prometheus") + .withGroupName(GROUP) + .withDefaultSeverity(AlertSeverity.WARNING) + .withResolveMode(ResolveMode.PRODUCER) + .withExternalDetailCeiling(DetailLevel.STANDARD) + .withDescription("Ingested from Prometheus Alertmanager (" + alertname + ").") + .withHowToFix("See the alert's runbook in Prometheus. Alertmanager already notifies its receivers; turn on Notify here only to route it through Alerts too.")); + AlertsDb.update(AlertsType.TABLE_NAME, new QRecord().withValue("id", type.getValue("id")).withValue("notify", false)); + } + return (null); + }); + } + + + + /******************************************************************************* + ** "HighCPU Usage" to "prometheus.highcpu-usage". + *******************************************************************************/ + public static String typeName(String alertname) + { + String slug = alertname.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9-]+", "-").replaceAll("^-+|-+$", ""); + return (TYPE_PREFIX + (slug.isEmpty() ? "unnamed" : slug)); + } + + + + /******************************************************************************* + ** A fingerprint from the labels when Alertmanager didn't send one. + *******************************************************************************/ + static String fingerprint(Map labels) + { + return (Integer.toHexString(new java.util.TreeMap<>(labels).toString().hashCode())); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + private static String firstNonBlank(Object... values) + { + for(Object value : values) + { + if(value != null && StringUtils.hasContent(String.valueOf(value))) + { + return (String.valueOf(value)); + } + } + return (null); + } + + + + /******************************************************************************* + ** An ISO instant, or null (Alertmanager's zero time is null too). + *******************************************************************************/ + private static Instant parseInstant(Object value) + { + try + { + Instant instant = value == null ? null : Instant.parse(String.valueOf(value)); + return (instant == null || instant.getEpochSecond() <= 0 ? null : instant); + } + catch(Exception e) + { + return (null); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/alerts/delivery/ExternalChannelsTest.java b/src/test/java/com/kingsrook/qbits/alerts/delivery/ExternalChannelsTest.java new file mode 100644 index 0000000..0a40b55 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/alerts/delivery/ExternalChannelsTest.java @@ -0,0 +1,274 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.delivery; + + +import java.io.IOException; +import java.io.OutputStream; +import java.net.InetSocketAddress; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels; +import com.kingsrook.qbits.alerts.delivery.channels.StandardWebhookSigner; +import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.utils.JsonUtils; +import com.sun.net.httpserver.HttpServer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** The external channels against a local HTTP receiver: request shapes + ** (Slack Block Kit, Teams Adaptive Card, signed webhook, PagerDuty Events v2, + ** Alertmanager v2), outcome classification, https enforcement and secrets. + *******************************************************************************/ +class ExternalChannelsTest +{ + private HttpServer server; + private String base; + private final List received = new ArrayList<>(); + private final AtomicInteger status = new AtomicInteger(200); + private AlertSecretResolver secrets; + + + + /******************************************************************************* + ** One captured request. + *******************************************************************************/ + record Received(String path, Map> headers, String body) + { + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @BeforeEach + void start() throws IOException + { + server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/", exchange -> + { + String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8); + received.add(new Received(exchange.getRequestURI().getPath(), exchange.getRequestHeaders(), body)); + byte[] reply = "{\"ok\":true}".getBytes(StandardCharsets.UTF_8); + exchange.sendResponseHeaders(status.get(), reply.length); + try(OutputStream out = exchange.getResponseBody()) + { + out.write(reply); + } + }); + server.start(); + base = "http://127.0.0.1:" + server.getAddress().getPort(); + + QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter(); + interpreter.addValueMap("test", Map.of("URL", base + "/hook", "KEY", "whsec_c2VjcmV0LWtleS1mb3ItdGVzdHM=", "ROUTING", "R0UT1NG", "BEARER", "am-token")); + secrets = new AlertSecretResolver(interpreter); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @AfterEach + void stop() + { + server.stop(0); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + private static QRecord channel(String type, String secretRef, Map settings) + { + return (new QRecord().withValue("channelType", type).withValue("secretRef", secretRef).withValue("settings", JsonUtils.toJson(settings))); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + private static AlertMessage message(String kind, String status) + { + AlertMessage.Entry entry = new AlertMessage.Entry(7, "cdn.purge.dead", "CDN purge failed", "provider:cf", "ERROR", status, "3 CDN purges failed", "Retry later.", "CDN", 3, + "https://admin.example.com/app/alerts_alert/7", "2026-10-12T10:00:00Z", "RESOLVED".equals(status) ? "2026-10-12T11:00:00Z" : null); + return (new AlertMessage("[ERROR] 3 CDN purges failed", "[ERROR] CDN purge failed / CDN: 3 CDN purges failed (2 h)\nx", "

x

", "ERROR", kind, List.of(7), + "https://admin.example.com/app/alerts_alert/7", List.of(entry))); + } + + + + /******************************************************************************* + ** Slack and Teams post to the secret URL. + *******************************************************************************/ + @Test + void testSlackAndTeams() throws Exception + { + ExternalChannels.Slack slack = new ExternalChannels.Slack(); + assertThat(slack.validate(channel("SLACK", null, Map.of()), secrets)).hasSize(1); + QRecord slackChannel = channel("SLACK", "${test.URL}", Map.of("allowInsecureHttp", true)); + assertThat(slack.validate(slackChannel, secrets)).isEmpty(); + assertThat(slack.deliver(message("RAISED", "OPEN"), slackChannel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT); + Map slackBody = JsonUtils.toObject(received.get(0).body(), Map.class); + assertThat(slackBody.get("text")).isEqualTo("[ERROR] 3 CDN purges failed"); + assertThat(received.get(0).body()).contains("<b>x</b>").contains("Open in Alerts"); + + ExternalChannels.Teams teams = new ExternalChannels.Teams(); + QRecord teamsChannel = channel("TEAMS", "${test.URL}", Map.of("allowInsecureHttp", true)); + assertThat(teams.validate(channel("TEAMS", null, Map.of()), secrets)).hasSize(1); + assertThat(teams.deliver(message("RAISED", "OPEN"), teamsChannel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT); + assertThat(received.get(1).body()).contains("application/vnd.microsoft.card.adaptive").contains("AdaptiveCard").contains("Action.OpenUrl"); + + ////////////////////////////////////////////// + // https only, unless explicitly allowed // + ////////////////////////////////////////////// + DeliveryResult insecure = slack.deliver(message("RAISED", "OPEN"), channel("SLACK", "${test.URL}", Map.of()), null, secrets); + assertThat(insecure.outcome()).isEqualTo(DeliveryResult.Outcome.FAILED); + assertThat(insecure.error()).contains("https").doesNotContain(base); + } + + + + /******************************************************************************* + ** The webhook is signed with the shared Standard Webhooks format. + *******************************************************************************/ + @Test + void testSignedWebhook() throws Exception + { + ExternalChannels.Webhook webhook = new ExternalChannels.Webhook(); + assertThat(webhook.validate(channel("WEBHOOK", null, Map.of()), secrets)).hasSize(2); + assertThat(webhook.deliver(message("RAISED", "OPEN"), channel("WEBHOOK", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED); + + QRecord channel = channel("WEBHOOK", "${test.KEY}", Map.of("url", base + "/receiver", "allowInsecureHttp", true)); + assertThat(webhook.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT); + Received request = received.get(0); + assertThat(request.path()).isEqualTo("/receiver"); + String id = request.headers().get("Webhook-id").get(0); + String timestamp = request.headers().get("Webhook-timestamp").get(0); + String signature = request.headers().get("Webhook-signature").get(0); + assertThat(id).startsWith("msg_"); + String expected = StandardWebhookSigner.sign(id, Long.parseLong(timestamp), request.body().getBytes(StandardCharsets.UTF_8), "whsec_c2VjcmV0LWtleS1mb3ItdGVzdHM="); + assertThat(signature).isEqualTo("v1," + expected); + assertThat(request.body()).contains("\"type\":\"qqq.alert.resolved\"").contains("\"dedupKey\":\"provider:cf\""); + + //////////////////////////////////////////////////// + // the id is stable across retries of one message // + //////////////////////////////////////////////////// + webhook.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets); + assertThat(received.get(1).headers().get("Webhook-id").get(0)).isEqualTo(id); + } + + + + /******************************************************************************* + ** PagerDuty: trigger / acknowledge / resolve with a stable dedup key. + *******************************************************************************/ + @Test + void testPagerDuty() throws Exception + { + ExternalChannels.PagerDuty pagerDuty = new ExternalChannels.PagerDuty(); + assertThat(pagerDuty.validate(channel("PAGERDUTY", null, Map.of()), secrets)).hasSize(1); + assertThat(pagerDuty.deliver(message("RAISED", "OPEN"), channel("PAGERDUTY", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED); + QRecord channel = channel("PAGERDUTY", "${test.ROUTING}", Map.of("url", base + "/v2/enqueue", "allowInsecureHttp", true, "source", "kof22")); + + pagerDuty.deliver(message("RAISED", "OPEN"), channel, null, secrets); + pagerDuty.deliver(message("RAISED", "ACKNOWLEDGED"), channel, null, secrets); + pagerDuty.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets); + List> events = new ArrayList<>(); + for(Received request : received) + { + events.add(JsonUtils.toObject(request.body(), Map.class)); + } + assertThat(events.stream().map(event -> String.valueOf(event.get("event_action"))).toList()).containsExactly("trigger", "acknowledge", "resolve"); + assertThat(events).allMatch(event -> "cdn.purge.dead:provider:cf".equals(event.get("dedup_key")) && "R0UT1NG".equals(event.get("routing_key"))); + Map payload = (Map) events.get(0).get("payload"); + assertThat(payload.get("severity")).isEqualTo("error"); + assertThat(payload.get("source")).isEqualTo("kof22"); + } + + + + /******************************************************************************* + ** Alertmanager sink: v2 alerts with the qqq labels and the origin guard. + *******************************************************************************/ + @Test + void testAlertmanagerSink() throws Exception + { + ExternalChannels.Alertmanager alertmanager = new ExternalChannels.Alertmanager(); + assertThat(alertmanager.validate(channel("ALERTMANAGER", null, Map.of()), secrets)).hasSize(1); + assertThat(alertmanager.deliver(message("RAISED", "OPEN"), channel("ALERTMANAGER", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED); + QRecord channel = channel("ALERTMANAGER", "${test.BEARER}", Map.of("url", base + "/", "allowInsecureHttp", true)); + + alertmanager.deliver(message("RAISED", "OPEN"), channel, null, secrets); + alertmanager.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets); + assertThat(received.get(0).path()).isEqualTo("/api/v2/alerts"); + assertThat(received.get(0).headers().get("Authorization").get(0)).isEqualTo("Bearer am-token"); + List firing = JsonUtils.toObject(received.get(0).body(), List.class); + Map labels = (Map) ((Map) firing.get(0)).get("labels"); + assertThat(labels.get("qqq_origin")).isEqualTo("qbit-alerts"); + assertThat(labels.get("qqq_type")).isEqualTo("cdn.purge.dead"); + assertThat(labels.get("severity")).isEqualTo("error"); + List resolvedBody = JsonUtils.toObject(received.get(1).body(), List.class); + assertThat(((Map) resolvedBody.get(0)).get("endsAt")).isEqualTo("2026-10-12T11:00:00Z"); + + ////////////////////////////////////////////////////// + // a test message is one stand-in alert, already over // + ////////////////////////////////////////////////////// + alertmanager.deliver(new AlertMessage("TEST: hello", "t", "h", "INFO", "TEST", List.of(), "https://x/app/alerts_alert"), channel, null, secrets); + assertThat(received.get(2).body()).contains("alerts.test"); + } + + + + /******************************************************************************* + ** Outcomes: 5xx and 429 retry; other 4xx fail permanently; no receiver + ** retries. + *******************************************************************************/ + @Test + void testOutcomes() + { + ExternalChannels.Slack slack = new ExternalChannels.Slack(); + QRecord channel = channel("SLACK", "${test.URL}", Map.of("allowInsecureHttp", true)); + status.set(503); + assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY); + status.set(429); + assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY); + status.set(404); + DeliveryResult notFound = slack.deliver(message("RAISED", "OPEN"), channel, null, secrets); + assertThat(notFound.outcome()).isEqualTo(DeliveryResult.Outcome.FAILED); + assertThat(notFound.error()).startsWith("HTTP 404"); + + ExternalChannels.PagerDuty pagerDuty = new ExternalChannels.PagerDuty(); + assertThat(pagerDuty.deliver(message("RAISED", "OPEN"), channel("PAGERDUTY", "${test.ROUTING}", Map.of("url", base, "allowInsecureHttp", true)), null, secrets).outcome()) + .isEqualTo(DeliveryResult.Outcome.FAILED); + + server.stop(0); + assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY); + assertThat(ExternalChannels.all()).hasSize(5); + } +} diff --git a/src/test/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngestTest.java b/src/test/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngestTest.java new file mode 100644 index 0000000..b01a2ed --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/alerts/engine/AlertmanagerIngestTest.java @@ -0,0 +1,140 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.alerts.engine; + + +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.alerts.AlertsQBitConfig; +import com.kingsrook.qbits.alerts.BaseAlertsTest; +import com.kingsrook.qbits.alerts.model.AlertsAlert; +import com.kingsrook.qbits.alerts.model.AlertsType; +import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException; +import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Alertmanager as a source: authentication, firing and resolved alerts, the + ** EXTERNAL type with notifications off, and the loop guard. + *******************************************************************************/ +class AlertmanagerIngestTest extends BaseAlertsTest +{ + static final String TOKEN_PROPERTY = "alerts.test.ingest.token"; + + + + /******************************************************************************* + ** + *******************************************************************************/ + @BeforeAll + static void setToken() + { + System.setProperty(TOKEN_PROPERTY, "s3cret-ingest"); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @AfterAll + static void clearToken() + { + System.clearProperty(TOKEN_PROPERTY); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @Override + protected void customizeConfig(AlertsQBitConfig config) + { + config.withAlertmanagerIngestTokenSecretRef("${prop." + TOKEN_PROPERTY + "}"); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + private static Map alert(String status, String alertname, String fingerprint, Map extraLabels) + { + Map labels = new java.util.LinkedHashMap<>(Map.of("alertname", alertname, "severity", "critical", "instance", "node-1")); + labels.putAll(extraLabels); + return (Map.of("status", status, "labels", labels, "annotations", Map.of("summary", "CPU high on node-1", "description", "Over 90% for 10 minutes."), + "startsAt", "2026-10-12T11:00:00Z", "fingerprint", fingerprint)); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @Test + void testAuthentication() throws Exception + { + AlertmanagerIngest ingest = new AlertmanagerIngest(runtime); + ingest.authenticate("Bearer s3cret-ingest"); + assertThatThrownBy(() -> ingest.authenticate("Bearer nope")).isInstanceOf(QAuthenticationException.class); + assertThatThrownBy(() -> ingest.authenticate(null)).isInstanceOf(QAuthenticationException.class); + + config.withAlertmanagerIngestTokenSecretRef(null); + assertThatThrownBy(() -> ingest.authenticate("Bearer s3cret-ingest")).isInstanceOf(QNotFoundException.class); + } + + + + /******************************************************************************* + ** + *******************************************************************************/ + @Test + void testFiringResolvedAndLoopGuard() throws Exception + { + AlertmanagerIngest ingest = new AlertmanagerIngest(runtime); + AlertmanagerIngest.Result first = ingest.ingest(Map.of("version", "4", "alerts", List.of( + alert("firing", "HighCPU Usage", "abc123", Map.of()), + alert("firing", "AlertsFromQqq", "def456", Map.of("qqq_origin", "qbit-alerts")), + "not an alert"))); + assertThat(first).isEqualTo(new AlertmanagerIngest.Result(1, 0, 2)); + + QRecord type = runtime.asSystem(() -> AlertsDb.first(AlertsType.TABLE_NAME, AlertsDb.eq("name", "prometheus.highcpu-usage"))); + assertThat(type.getValueBoolean("notify")).isFalse(); + assertThat(type.getValueString("sourceName")).isEqualTo("prometheus"); + QRecord alert = all(AlertsAlert.TABLE_NAME).get(0); + assertThat(alert.getValueString("severity")).isEqualTo("CRITICAL"); + assertThat(alert.getValueString("title")).isEqualTo("CPU high on node-1"); + assertThat(alert.getValueString("dedupKey")).isEqualTo("abc123"); + assertThat(alert.getValueString("attributes")).contains("instance"); + + AlertmanagerIngest.Result second = ingest.ingest(Map.of("alerts", List.of(alert("resolved", "HighCPU Usage", "abc123", Map.of())))); + assertThat(second.resolved()).isEqualTo(1); + assertThat(all(AlertsAlert.TABLE_NAME).get(0).getValueString("status")).isEqualTo("RESOLVED"); + + assertThatThrownBy(() -> ingest.ingest(Map.of())).isInstanceOf(QBadRequestException.class); + assertThat(AlertmanagerIngest.typeName("!!")).isEqualTo("prometheus.unnamed"); + assertThat(AlertmanagerIngest.fingerprint(Map.of("a", "b"))).isNotBlank(); + } +}