feat: external channels and Alertmanager ingest (P4) #5
No files matched your search
@@ -5,6 +5,13 @@ All notable changes to this project are documented here. The format follows Keep
|
||||
## [0.1.0] - Unreleased
|
||||
|
||||
### Added
|
||||
- Optional external channels:
|
||||
- Slack (an Incoming Webhook, Block Kit)
|
||||
- Microsoft Teams (a Workflows webhook, an Adaptive Card)
|
||||
- a generic webhook, signed with Standard Webhooks headers through a stand-in for core's `WebhookSigner` (QRun-IO/qqq#1109)
|
||||
- PagerDuty (Events API v2: trigger, acknowledge and resolve)
|
||||
- an Alertmanager sink (`/api/v2/alerts`, with the qqq labels and the origin guard)
|
||||
- Prometheus Alertmanager ingest at `POST /qqq/v1/alerts/ingest/alertmanager`. It takes a bearer token, creates EXTERNAL types with notifications off, and keeps a loop guard.
|
||||
- Tables (`alerts_*`), with the Liquibase changelog `db/liquibase/qbit-alerts/alerts.changelog.sql`:
|
||||
- alerts, their timeline and records
|
||||
- types, groups and monitors
|
||||
|
||||
@@ -18,7 +18,15 @@ Any qbit or app reports exceptional conditions ("a person should do something").
|
||||
- `HEARTBEAT`, on a key, a process or a table's newest row
|
||||
- `PROCESS_FAILED`
|
||||
- `CODE`, through `AlertCheckInterface`
|
||||
- **Notifies subscribers** (users, teams or channels) through in-app (the bell), email and email lists, and through any channel another qbit registers. Every suppressed notification is logged with its reason.
|
||||
- **Notifies subscribers** (users, teams or channels) through in-app (the bell), email and email lists. These optional channels are also built in:
|
||||
- Slack (an Incoming Webhook)
|
||||
- Microsoft Teams (a Workflows webhook)
|
||||
- a generic signed webhook
|
||||
- PagerDuty (Events API v2)
|
||||
- Prometheus Alertmanager, as a sink
|
||||
|
||||
Any channel another qbit registers works too. Every suppressed notification is logged with its reason.
|
||||
- **Ingests Prometheus Alertmanager** at `POST /qqq/v1/alerts/ingest/alertmanager`, with notifications off for those types, so infrastructure alerts are triaged in one place.
|
||||
- **Owns the Alerts home:** open alerts by severity, my alerts, needs action, by group, a 30-day trend, the noisiest types, delivery health and recently resolved. Other pages get link-only summaries from `AlertSummaryLoader`.
|
||||
|
||||
All data is in standard QQQ tables named `alerts_*`. People change alerts through record actions, never by editing rows.
|
||||
@@ -78,13 +86,36 @@ Once the RC ships, producers call `QAlerts.raise(...)` from core and need no dep
|
||||
| POST | `/qqq/v1/alerts/inbox/read` | `{"alertId": n}` or `{"all": true}` |
|
||||
| GET | `/qqq/v1/alerts/record-panel?tableName=t&recordId=id` | The "Open alerts" panel for a record (`visible: false` when there is nothing to show) |
|
||||
| POST | `/qqq/v1/alerts/record-panel/watch` | `{"tableName": t, "recordId": id}`: a RECORD subscription for the user |
|
||||
| POST | `/qqq/v1/alerts/ingest/alertmanager` | Alertmanager's webhook receiver (version 4). It takes a bearer token whose reference is set by `withAlertmanagerIngestTokenSecretRef`, and is off without one. |
|
||||
|
||||
## External channels
|
||||
|
||||
Each external channel is an `alerts_channel` row that an admin creates. Its secret is a reference (`secretRef`, an environment variable name or a `${...}` expression), never the value. The minimum detail level is MINIMAL by default.
|
||||
|
||||
| Type | Secret | Settings |
|
||||
|---|---|---|
|
||||
| `SLACK` | The Incoming Webhook URL | none |
|
||||
| `TEAMS` | The Workflows "when a webhook request is received" URL | none |
|
||||
| `WEBHOOK` | The signing key (`whsec_...`) | `{"url": "https://receiver"}` |
|
||||
| `PAGERDUTY` | The Events API v2 routing key | optional `url`, `source` |
|
||||
| `ALERTMANAGER` | An optional bearer token | `{"url": "https://alertmanager:9093"}` |
|
||||
|
||||
How each one sends:
|
||||
|
||||
- **Webhook** bodies are `{type, subject, severity, link, alerts: [...]}`, signed with the Standard Webhooks headers (`webhook-id`, `webhook-timestamp`, `webhook-signature: v1,<base64 HMAC-SHA256 of id.timestamp.body>`). This is the shared format from the qbit-webhooks assessment (QRun-IO/qqq#1105). The signer is a stand-in until core's `WebhookSigner` (QRun-IO/qqq#1109) is released.
|
||||
- **PagerDuty** gets one event per alert: `trigger`, `acknowledge` or `resolve`, with `dedup_key` set to the type and key.
|
||||
- **Alertmanager** alerts carry the labels `qqq_type`, `qqq_group`, `qqq_key` and `qqq_origin="qbit-alerts"`. Ingest ignores alerts with that origin, so nothing loops.
|
||||
- **Responses:** 2xx is sent. 408, 425, 429, 5xx and network errors are retried. Any other 4xx fails permanently.
|
||||
- **URLs** must be https, unless the channel's settings say `"allowInsecureHttp": true`, which is meant for local receivers.
|
||||
|
||||
## Known limits (v1)
|
||||
|
||||
- **Claims are best-effort.** Delivery and monitor claims use a lease, not an atomic conditional update. With several scheduler nodes, run the two scheduled processes on one node, or accept at-least-once delivery.
|
||||
- **Email threading headers.** `Message-ID` and `In-Reply-To` need core messaging support for custom headers, so they are not set.
|
||||
- **Recipient visibility.** It is checked through `createAutomatedSessionForUser`. If a host's authentication module can't build a session with permissions for a user, email goes out only for alerts that have no visibility permission and no subject table. In-app delivery is always filtered by the record locks.
|
||||
- **Not in v1:** digests, quiet hours, reminders, escalation, external channels (Slack, Teams, webhook, PagerDuty, Alertmanager), mirrored lock columns and a configurable table namespace. These are design phases P4 to P6.
|
||||
- **Outbound HTTP** blocks plain http, but has no full SSRF guard. Channels are admin-only. Core's `SafeHttpClient` (QRun-IO/qqq#1109) replaces the client when it is released.
|
||||
- **Alertmanager sink:** open alerts are sent with an `endsAt` a week out, not re-posted.
|
||||
- **Not built yet:** digests, quiet hours, reminders, Micrometer metrics, ROLE and IdP-group team resolvers, escalation, mirrored lock columns and a configurable table namespace. These are the rest of design phases P4 to P6.
|
||||
|
||||
## Build
|
||||
|
||||
|
||||
@@ -87,6 +87,8 @@ public class AlertsQBitConfig implements QBitConfig
|
||||
|
||||
private List<QCodeReference> channelTypes = new ArrayList<>();
|
||||
|
||||
private String alertmanagerIngestTokenSecretRef;
|
||||
|
||||
private Boolean registerRoutes = true;
|
||||
private Boolean synchronousSink = false;
|
||||
|
||||
@@ -561,6 +563,30 @@ public class AlertsQBitConfig implements QBitConfig
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for alertmanagerIngestTokenSecretRef
|
||||
*******************************************************************************/
|
||||
public String getAlertmanagerIngestTokenSecretRef()
|
||||
{
|
||||
return (this.alertmanagerIngestTokenSecretRef);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fluent setter for alertmanagerIngestTokenSecretRef: the reference (an
|
||||
** environment variable name or ${...} expression, never the value) of the
|
||||
** bearer token Alertmanager sends to POST /qqq/v1/alerts/ingest/alertmanager.
|
||||
** Null (the default) turns ingest off.
|
||||
*******************************************************************************/
|
||||
public AlertsQBitConfig withAlertmanagerIngestTokenSecretRef(String alertmanagerIngestTokenSecretRef)
|
||||
{
|
||||
this.alertmanagerIngestTokenSecretRef = alertmanagerIngestTokenSecretRef;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Getter for runtime (built by the producer)
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -42,6 +42,8 @@ import io.javalin.http.Context;
|
||||
** - POST /inbox/read {"alertId": n} or {"all": true}
|
||||
** - GET /record-panel?tableName=t&recordId=id the "Open alerts" panel
|
||||
** - POST /record-panel/watch {"tableName": t, "recordId": id}
|
||||
** - POST /ingest/alertmanager Alertmanager's webhook (bearer token; off
|
||||
** unless alertmanagerIngestTokenSecretRef is configured)
|
||||
** Each request gets a session from the instance's authentication module;
|
||||
** AlertsApi checks alerts.view and works in that session. Errors map
|
||||
** through QJavalinUtils.handleException (400, 401, 403, 404). POST bodies
|
||||
@@ -51,6 +53,8 @@ public class AlertsRouteProvider implements QJavalinRouteProviderInterface
|
||||
{
|
||||
public static final String BASE_PATH = "/qqq/v1/alerts";
|
||||
|
||||
static final int MAX_INGEST_BYTES = 1024 * 1024;
|
||||
|
||||
private QInstance qInstance;
|
||||
|
||||
|
||||
@@ -92,6 +96,36 @@ public class AlertsRouteProvider implements QJavalinRouteProviderInterface
|
||||
config.routes.post(BASE_PATH + "/inbox/read", context -> respond(context, true, AlertsApi::markRead));
|
||||
config.routes.get(BASE_PATH + "/record-panel", context -> respond(context, false, (api, body) -> api.recordPanel(context.queryParam("tableName"), context.queryParam("recordId"))));
|
||||
config.routes.post(BASE_PATH + "/record-panel/watch", context -> respond(context, true, AlertsApi::watchRecord));
|
||||
config.routes.post(BASE_PATH + "/ingest/alertmanager", this::ingestAlertmanager);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Alertmanager's webhook (no QQQ session: a bearer token instead).
|
||||
*******************************************************************************/
|
||||
private void ingestAlertmanager(Context context)
|
||||
{
|
||||
try
|
||||
{
|
||||
if(context.body().length() > MAX_INGEST_BYTES)
|
||||
{
|
||||
throw (new QBadRequestException("The payload is larger than 1 MB."));
|
||||
}
|
||||
com.kingsrook.qbits.alerts.engine.AlertmanagerIngest ingest = new com.kingsrook.qbits.alerts.engine.AlertmanagerIngest(AlertsRuntime.get(qInstance));
|
||||
ingest.authenticate(context.header("Authorization"));
|
||||
com.kingsrook.qbits.alerts.engine.AlertmanagerIngest.Result result = ingest.ingest(readJsonBody(context));
|
||||
context.contentType(ContentType.APPLICATION_JSON);
|
||||
context.result(JsonUtils.toJson(Map.of("raised", result.raised(), "resolved", result.resolved(), "ignored", result.ignored())));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
QJavalinUtils.handleException(null, context, e);
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -34,8 +34,8 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The channel types (built-in in-app, email and email list, plus any the
|
||||
** config adds) and the two default channel rows every install has: "in-app"
|
||||
** The channel types (built-in in-app, email, email list, Slack, Teams,
|
||||
** webhook, PagerDuty and Alertmanager, plus any the config adds) and the two default channel rows every install has: "in-app"
|
||||
** (the bell; always available) and "email" (a person's own address).
|
||||
*******************************************************************************/
|
||||
public class AlertChannels
|
||||
@@ -60,6 +60,10 @@ public class AlertChannels
|
||||
register(new InAppChannel());
|
||||
register(new EmailChannel(runtime.getConfig(), ChannelType.EMAIL));
|
||||
register(new EmailChannel(runtime.getConfig(), ChannelType.EMAIL_LIST));
|
||||
for(AlertChannelInterface external : com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels.all())
|
||||
{
|
||||
register(external);
|
||||
}
|
||||
for(QCodeReference reference : runtime.getConfig().getChannelTypes())
|
||||
{
|
||||
try
|
||||
|
||||
@@ -31,7 +31,48 @@ import java.util.List;
|
||||
** @param kind RAISED, RESOLVED, REMINDER, ESCALATED, DIGEST or TEST
|
||||
** @param alertIds the alerts in the message
|
||||
** @param link a link to the alert (or the Alerts list) in the app
|
||||
** @param entries one structured entry per alert, for channels that send
|
||||
** one event per alert (PagerDuty, Alertmanager, webhooks);
|
||||
** already redacted and capped by the detail level
|
||||
*******************************************************************************/
|
||||
public record AlertMessage(String subject, String text, String html, String severity, String kind, List<Integer> alertIds, String link)
|
||||
public record AlertMessage(String subject, String text, String html, String severity, String kind, List<Integer> alertIds, String link, List<Entry> entries)
|
||||
{
|
||||
/*******************************************************************************
|
||||
** A message without structured entries.
|
||||
*******************************************************************************/
|
||||
public AlertMessage(String subject, String text, String html, String severity, String kind, List<Integer> alertIds, String link)
|
||||
{
|
||||
this(subject, text, html, severity, kind, alertIds, link, List.of());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One alert, as structured channels send it.
|
||||
**
|
||||
** @param alertId the alert's id (null for a test message)
|
||||
** @param alertTypeName the type, e.g. cdn.purge.dead
|
||||
** @param typeLabel the type's label
|
||||
** @param dedupKey the producer's dedup key
|
||||
** @param severity INFO, WARNING, ERROR or CRITICAL
|
||||
** @param status OPEN, ACKNOWLEDGED, SNOOZED or RESOLVED
|
||||
** @param title the redacted title
|
||||
** @param summary the redacted summary (null at MINIMAL)
|
||||
** @param group the group's label
|
||||
** @param itemCount the record count (null at MINIMAL)
|
||||
** @param link the alert in the app
|
||||
** @param firstRaisedAt when it opened (ISO)
|
||||
** @param resolvedAt when it resolved (ISO), or null
|
||||
*******************************************************************************/
|
||||
public record Entry(Integer alertId, String alertTypeName, String typeLabel, String dedupKey, String severity, String status, String title, String summary,
|
||||
String group, Integer itemCount, String link, String firstRaisedAt, String resolvedAt)
|
||||
{
|
||||
/***************************************************************************
|
||||
** The stable key for systems that dedup (PagerDuty, Alertmanager).
|
||||
***************************************************************************/
|
||||
public String stableKey()
|
||||
{
|
||||
return ((alertTypeName == null ? "alerts.test" : alertTypeName) + ":" + (dedupKey == null ? String.valueOf(alertId) : dedupKey));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -147,7 +147,22 @@ public final class AlertMessageRenderer
|
||||
html.append("<p style=\"font-size:12px;opacity:.75\"><a href=\"").append(HomeHtml.escape(base + "/app/alerts_subscription")).append("\">Change or stop these emails</a>. ")
|
||||
.append("Personal data and secrets are removed from alert messages.</p></div>");
|
||||
|
||||
return (new AlertMessage(subject, text.toString(), html.toString(), highest.name(), kind.name(), alertIds, link));
|
||||
List<AlertMessage.Entry> entries = new ArrayList<>();
|
||||
for(Item item : items)
|
||||
{
|
||||
QRecord alert = item.alert();
|
||||
DetailLevel level = item.level() == null ? DetailLevel.MINIMAL : item.level();
|
||||
boolean standard = level.compareTo(DetailLevel.STANDARD) >= 0;
|
||||
Integer id = alert.getValueInteger("id");
|
||||
entries.add(new AlertMessage.Entry(id, alert.getValueString("alertTypeName"), safe(item.typeLabel()), alert.getValueString("dedupKey"),
|
||||
AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING).name(),
|
||||
kind == NotificationKind.RESOLVED ? "RESOLVED" : (alert.getValueString("status") == null ? "OPEN" : alert.getValueString("status")),
|
||||
safe(alert.getValueString("title")), standard ? AlertRedactor.redact(alert.getValueString("summary")) : null, item.groupLabel(),
|
||||
standard ? alert.getValueInteger("itemCount") : null, id == null ? base + "/app/" + AlertsAlert.TABLE_NAME : base + "/app/" + AlertsAlert.TABLE_NAME + "/" + id,
|
||||
alert.getValueInstant("firstRaisedAt") == null ? null : alert.getValueInstant("firstRaisedAt").toString(),
|
||||
alert.getValueInstant("resolvedAt") == null ? null : alert.getValueInstant("resolvedAt").toString()));
|
||||
}
|
||||
return (new AlertMessage(subject, text.toString(), html.toString(), highest.name(), kind.name(), alertIds, link, entries));
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.delivery.channels;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Instant;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import com.kingsrook.qbits.alerts.delivery.AlertChannelInterface;
|
||||
import com.kingsrook.qbits.alerts.delivery.AlertMessage;
|
||||
import com.kingsrook.qbits.alerts.delivery.AlertSecretResolver;
|
||||
import com.kingsrook.qbits.alerts.delivery.DeliveryResult;
|
||||
import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.ChannelType;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.utils.StringUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The optional external channels (design section 5.6). None is required;
|
||||
** each is an admin-configured alerts_channel row whose secret is a
|
||||
** reference (secretRef), resolved at send time:
|
||||
** - SLACK: an Incoming Webhook URL (the URL is the secret); Block Kit body.
|
||||
** - TEAMS: a Teams Workflows "when a webhook request is received" URL (the
|
||||
** secret); an Adaptive Card. (Office 365 connectors are retired.)
|
||||
** - WEBHOOK: settings {"url": ...}; the secret is the HMAC signing key; a
|
||||
** documented JSON body signed with the Standard Webhooks headers (the
|
||||
** shared format of QRun-IO/qqq#1105, see StandardWebhookSigner).
|
||||
** - PAGERDUTY: the Events API v2 routing key (the secret); one
|
||||
** trigger / acknowledge / resolve event per alert, dedup_key = type:key.
|
||||
** - ALERTMANAGER: settings {"url": "https://alertmanager:9093"} (optional
|
||||
** bearer secret); POST /api/v2/alerts with the qqq_* labels and
|
||||
** qqq_origin="qbit-alerts" (the ingest loop guard); endsAt on resolve.
|
||||
** No vendor SDKs: plain java.net.http and JSON.
|
||||
*******************************************************************************/
|
||||
public final class ExternalChannels
|
||||
{
|
||||
public static final String ORIGIN_LABEL = "qqq_origin";
|
||||
public static final String ORIGIN_VALUE = "qbit-alerts";
|
||||
|
||||
static final String PAGERDUTY_URL = "https://events.pagerduty.com/v2/enqueue";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Channels only.
|
||||
*******************************************************************************/
|
||||
private ExternalChannels()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** All five, for registration.
|
||||
*******************************************************************************/
|
||||
public static List<AlertChannelInterface> all()
|
||||
{
|
||||
return (List.of(new Slack(), new Teams(), new Webhook(), new PagerDuty(), new Alertmanager()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Slack Incoming Webhook.
|
||||
*******************************************************************************/
|
||||
public static class Slack implements AlertChannelInterface
|
||||
{
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public String getType()
|
||||
{
|
||||
return (ChannelType.SLACK.name());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public List<String> validate(QRecord channel, AlertSecretResolver secrets)
|
||||
{
|
||||
return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the Slack Incoming Webhook URL.") : List.of());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets)
|
||||
{
|
||||
Map<String, Object> body = new LinkedHashMap<>();
|
||||
body.put("text", message.subject());
|
||||
body.put("blocks", List.of(
|
||||
Map.of("type", "section", "text", Map.of("type", "mrkdwn", "text", "*" + slackEscape(message.subject()) + "*\n" + slackEscape(truncate(message.text(), 2800)))),
|
||||
Map.of("type", "actions", "elements", List.of(Map.of("type", "button", "text", Map.of("type", "plain_text", "text", "Open in Alerts"), "url", message.link())))));
|
||||
return (HttpChannelSupport.postJson(secrets.resolve(channel.getValueString("secretRef")), null, HttpChannelSupport.json(body), HttpChannelSupport.allowInsecure(channel)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Microsoft Teams through a Workflows webhook.
|
||||
*******************************************************************************/
|
||||
public static class Teams implements AlertChannelInterface
|
||||
{
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public String getType()
|
||||
{
|
||||
return (ChannelType.TEAMS.name());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public List<String> validate(QRecord channel, AlertSecretResolver secrets)
|
||||
{
|
||||
return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the Teams Workflows webhook URL.") : List.of());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets)
|
||||
{
|
||||
Map<String, Object> card = new LinkedHashMap<>();
|
||||
card.put("$schema", "http://adaptivecards.io/schemas/adaptive-card.json");
|
||||
card.put("type", "AdaptiveCard");
|
||||
card.put("version", "1.4");
|
||||
card.put("body", List.of(
|
||||
Map.of("type", "TextBlock", "text", message.subject(), "weight", "Bolder", "wrap", true),
|
||||
Map.of("type", "TextBlock", "text", truncate(message.text(), 2800), "wrap", true)));
|
||||
card.put("actions", List.of(Map.of("type", "Action.OpenUrl", "title", "Open in Alerts", "url", message.link())));
|
||||
Map<String, Object> body = Map.of("type", "message", "attachments", List.of(Map.of("contentType", "application/vnd.microsoft.card.adaptive", "content", card)));
|
||||
return (HttpChannelSupport.postJson(secrets.resolve(channel.getValueString("secretRef")), null, HttpChannelSupport.json(body), HttpChannelSupport.allowInsecure(channel)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Generic signed webhook.
|
||||
*******************************************************************************/
|
||||
public static class Webhook implements AlertChannelInterface
|
||||
{
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public String getType()
|
||||
{
|
||||
return (ChannelType.WEBHOOK.name());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public List<String> validate(QRecord channel, AlertSecretResolver secrets)
|
||||
{
|
||||
List<String> problems = new ArrayList<>();
|
||||
if(!StringUtils.hasContent(Objects.toString(HttpChannelSupport.settings(channel).get("url"), null)))
|
||||
{
|
||||
problems.add("Add the receiver's URL to the channel's settings: {\"url\": \"https://...\"}.");
|
||||
}
|
||||
if(secrets.resolve(channel.getValueString("secretRef")) == null)
|
||||
{
|
||||
problems.add("Set the secret reference to the signing key (a whsec_ secret).");
|
||||
}
|
||||
return (problems);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets)
|
||||
{
|
||||
List<String> problems = validate(channel, secrets);
|
||||
if(!problems.isEmpty())
|
||||
{
|
||||
return (DeliveryResult.failed(problems.get(0)));
|
||||
}
|
||||
Map<String, Object> body = new LinkedHashMap<>();
|
||||
body.put("type", "qqq.alert." + message.kind().toLowerCase(Locale.ROOT));
|
||||
body.put("subject", message.subject());
|
||||
body.put("severity", message.severity());
|
||||
body.put("link", message.link());
|
||||
body.put("alerts", message.entries());
|
||||
byte[] bytes = HttpChannelSupport.json(body);
|
||||
Map<String, String> headers = StandardWebhookSigner.headers(messageId(message), Instant.now(), bytes, secrets.resolve(channel.getValueString("secretRef")));
|
||||
return (HttpChannelSupport.postJson(Objects.toString(HttpChannelSupport.settings(channel).get("url")), headers, bytes, HttpChannelSupport.allowInsecure(channel)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** PagerDuty Events API v2.
|
||||
*******************************************************************************/
|
||||
public static class PagerDuty implements AlertChannelInterface
|
||||
{
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public String getType()
|
||||
{
|
||||
return (ChannelType.PAGERDUTY.name());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public List<String> validate(QRecord channel, AlertSecretResolver secrets)
|
||||
{
|
||||
return (secrets.resolve(channel.getValueString("secretRef")) == null ? List.of("Set the secret reference to the PagerDuty Events API v2 routing key.") : List.of());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** One event per alert; the first failure decides the outcome.
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets)
|
||||
{
|
||||
String routingKey = secrets.resolve(channel.getValueString("secretRef"));
|
||||
if(routingKey == null)
|
||||
{
|
||||
return (DeliveryResult.failed(validate(channel, secrets).get(0)));
|
||||
}
|
||||
Map<String, Serializable> settings = HttpChannelSupport.settings(channel);
|
||||
String url = Objects.toString(settings.get("url"), PAGERDUTY_URL);
|
||||
String source = Objects.toString(settings.get("source"), ORIGIN_VALUE);
|
||||
for(AlertMessage.Entry entry : entries(message))
|
||||
{
|
||||
String action = "TEST".equals(message.kind()) ? "trigger" : switch(entry.status())
|
||||
{
|
||||
case "RESOLVED" -> "resolve";
|
||||
case "ACKNOWLEDGED" -> "acknowledge";
|
||||
default -> "trigger";
|
||||
};
|
||||
Map<String, Object> event = new LinkedHashMap<>();
|
||||
event.put("routing_key", routingKey);
|
||||
event.put("event_action", action);
|
||||
event.put("dedup_key", entry.stableKey());
|
||||
if("trigger".equals(action))
|
||||
{
|
||||
Map<String, Object> payload = new LinkedHashMap<>();
|
||||
payload.put("summary", truncate(entry.title(), 1024));
|
||||
payload.put("source", source);
|
||||
payload.put("severity", entry.severity().toLowerCase(Locale.ROOT));
|
||||
payload.put("component", entry.typeLabel());
|
||||
payload.put("group", entry.group());
|
||||
Map<String, Object> details = new LinkedHashMap<>();
|
||||
details.put("summary", entry.summary());
|
||||
details.put("records", entry.itemCount());
|
||||
details.put("link", entry.link());
|
||||
details.values().removeIf(Objects::isNull);
|
||||
payload.put("custom_details", details);
|
||||
payload.values().removeIf(Objects::isNull);
|
||||
event.put("payload", payload);
|
||||
event.put("links", List.of(Map.of("href", entry.link(), "text", "Open in Alerts")));
|
||||
}
|
||||
DeliveryResult result = HttpChannelSupport.postJson(url, null, HttpChannelSupport.json(event), HttpChannelSupport.allowInsecure(channel));
|
||||
if(result.outcome() != DeliveryResult.Outcome.SENT)
|
||||
{
|
||||
return (result);
|
||||
}
|
||||
}
|
||||
return (DeliveryResult.sent(null));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Prometheus Alertmanager as a sink.
|
||||
*******************************************************************************/
|
||||
public static class Alertmanager implements AlertChannelInterface
|
||||
{
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public String getType()
|
||||
{
|
||||
return (ChannelType.ALERTMANAGER.name());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
**
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public List<String> validate(QRecord channel, AlertSecretResolver secrets)
|
||||
{
|
||||
return (StringUtils.hasContent(Objects.toString(HttpChannelSupport.settings(channel).get("url"), null)) ? List.of()
|
||||
: List.of("Add Alertmanager's base URL to the channel's settings: {\"url\": \"https://alertmanager:9093\"}."));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** One POST with every alert. Open alerts get an endsAt a week out, so
|
||||
** they stay active without a re-post; a resolve sends endsAt = when.
|
||||
***************************************************************************/
|
||||
@Override
|
||||
public DeliveryResult deliver(AlertMessage message, QRecord channel, String recipientAddress, AlertSecretResolver secrets)
|
||||
{
|
||||
List<String> problems = validate(channel, secrets);
|
||||
if(!problems.isEmpty())
|
||||
{
|
||||
return (DeliveryResult.failed(problems.get(0)));
|
||||
}
|
||||
List<Map<String, Object>> alerts = new ArrayList<>();
|
||||
for(AlertMessage.Entry entry : entries(message))
|
||||
{
|
||||
Map<String, String> labels = new LinkedHashMap<>();
|
||||
labels.put("alertname", entry.alertTypeName() == null ? "alerts.test" : entry.alertTypeName());
|
||||
labels.put("severity", entry.severity().toLowerCase(Locale.ROOT));
|
||||
labels.put("qqq_type", Objects.toString(entry.alertTypeName(), "alerts.test"));
|
||||
labels.put("qqq_group", Objects.toString(entry.group(), ""));
|
||||
labels.put("qqq_key", Objects.toString(entry.dedupKey(), ""));
|
||||
labels.put(ORIGIN_LABEL, ORIGIN_VALUE);
|
||||
Map<String, Object> alert = new LinkedHashMap<>();
|
||||
alert.put("labels", labels);
|
||||
alert.put("annotations", Map.of("summary", entry.title(), "qqq_url", entry.link()));
|
||||
alert.put("generatorURL", entry.link());
|
||||
if(entry.firstRaisedAt() != null)
|
||||
{
|
||||
alert.put("startsAt", entry.firstRaisedAt());
|
||||
}
|
||||
boolean resolved = "RESOLVED".equals(entry.status()) || "TEST".equals(message.kind());
|
||||
alert.put("endsAt", resolved ? Objects.toString(entry.resolvedAt(), Instant.now().toString()) : Instant.now().plus(7, ChronoUnit.DAYS).toString());
|
||||
alerts.add(alert);
|
||||
}
|
||||
String bearer = secrets.resolve(channel.getValueString("secretRef"));
|
||||
Map<String, String> headers = bearer == null ? null : Map.of("Authorization", "Bearer " + bearer);
|
||||
String base = Objects.toString(HttpChannelSupport.settings(channel).get("url")).replaceAll("/+$", "");
|
||||
return (HttpChannelSupport.postJson(base + "/api/v2/alerts", headers, HttpChannelSupport.json(alerts), HttpChannelSupport.allowInsecure(channel)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The message's entries (a TEST message gets one stand-in entry).
|
||||
*******************************************************************************/
|
||||
static List<AlertMessage.Entry> entries(AlertMessage message)
|
||||
{
|
||||
if(message.entries() != null && !message.entries().isEmpty())
|
||||
{
|
||||
return (message.entries());
|
||||
}
|
||||
return (List.of(new AlertMessage.Entry(null, "alerts.test", "Test", "test", "INFO", "OPEN", message.subject(), null, null, null, message.link(), null, null)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A stable id for a webhook delivery (the same on every retry): a hash of
|
||||
** the kind and each alert's key and state.
|
||||
*******************************************************************************/
|
||||
static String messageId(AlertMessage message)
|
||||
{
|
||||
StringBuilder content = new StringBuilder(message.kind());
|
||||
for(AlertMessage.Entry entry : entries(message))
|
||||
{
|
||||
content.append('|').append(entry.alertId()).append(':').append(entry.status()).append(':').append(entry.firstRaisedAt()).append(':').append(entry.resolvedAt());
|
||||
}
|
||||
try
|
||||
{
|
||||
byte[] digest = MessageDigest.getInstance("SHA-256").digest(content.toString().getBytes(StandardCharsets.UTF_8));
|
||||
return ("msg_" + HexFormat.of().formatHex(digest).substring(0, 32));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return ("msg_" + Integer.toHexString(content.toString().hashCode()));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Slack mrkdwn escaping (&, <, >).
|
||||
*******************************************************************************/
|
||||
static String slackEscape(String text)
|
||||
{
|
||||
return (text == null ? "" : text.replace("&", "&").replace("<", "<").replace(">", ">"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
static String truncate(String text, int max)
|
||||
{
|
||||
return (text == null || text.length() <= max ? text : text.substring(0, max - 1) + "…");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.delivery.channels;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.alerts.delivery.AlertRedactor;
|
||||
import com.kingsrook.qbits.alerts.delivery.DeliveryResult;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
import com.kingsrook.qqq.backend.core.utils.StringUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Shared HTTP plumbing for the external channels: one pooled client, a 10
|
||||
** second timeout, JSON bodies, and the outcome rules:
|
||||
** - 2xx: SENT;
|
||||
** - 408, 425, 429 and 5xx, or a network error: RETRY;
|
||||
** - any other 4xx (a bad URL or key): FAILED, permanently.
|
||||
** URLs must be https unless the channel's settings say "allowInsecureHttp"
|
||||
** (for local receivers and tests). Error text is redacted and never includes
|
||||
** the URL, which for Slack and Teams is itself the secret.
|
||||
*******************************************************************************/
|
||||
public final class HttpChannelSupport
|
||||
{
|
||||
static final Duration TIMEOUT = Duration.ofSeconds(10);
|
||||
|
||||
private static final HttpClient CLIENT = HttpClient.newBuilder()
|
||||
.connectTimeout(TIMEOUT)
|
||||
.followRedirects(HttpClient.Redirect.NEVER)
|
||||
.build();
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Helpers only.
|
||||
*******************************************************************************/
|
||||
private HttpChannelSupport()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** POST a JSON body and classify the response.
|
||||
*******************************************************************************/
|
||||
public static DeliveryResult postJson(String url, Map<String, String> headers, byte[] body, boolean allowInsecure)
|
||||
{
|
||||
String problem = checkUrl(url, allowInsecure);
|
||||
if(problem != null)
|
||||
{
|
||||
return (DeliveryResult.failed(problem));
|
||||
}
|
||||
try
|
||||
{
|
||||
HttpRequest.Builder request = HttpRequest.newBuilder(URI.create(url))
|
||||
.timeout(TIMEOUT)
|
||||
.header("Content-Type", "application/json")
|
||||
.header("User-Agent", "qbit-alerts")
|
||||
.POST(HttpRequest.BodyPublishers.ofByteArray(body));
|
||||
if(headers != null)
|
||||
{
|
||||
headers.forEach(request::header);
|
||||
}
|
||||
HttpResponse<String> response = CLIENT.send(request.build(), HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
|
||||
return (classify(response.statusCode(), response.body()));
|
||||
}
|
||||
catch(InterruptedException e)
|
||||
{
|
||||
Thread.currentThread().interrupt();
|
||||
return (DeliveryResult.retry("Interrupted while sending."));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (DeliveryResult.retry(AlertRedactor.redact(e.getClass().getSimpleName() + (e.getMessage() == null ? "" : ": " + stripUrl(e.getMessage(), url)))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** SENT, RETRY or FAILED for an HTTP status.
|
||||
*******************************************************************************/
|
||||
static DeliveryResult classify(int status, String body)
|
||||
{
|
||||
if(status >= 200 && status < 300)
|
||||
{
|
||||
return (DeliveryResult.sent(null));
|
||||
}
|
||||
String detail = "HTTP " + status + (StringUtils.hasContent(body) ? ": " + AlertRedactor.redact(StringUtils.safeTruncate(body.trim(), 200, "…")) : "");
|
||||
if(status == 408 || status == 425 || status == 429 || status >= 500)
|
||||
{
|
||||
return (DeliveryResult.retry(detail));
|
||||
}
|
||||
return (DeliveryResult.failed(detail));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Why a URL can't be used (null when it can).
|
||||
*******************************************************************************/
|
||||
static String checkUrl(String url, boolean allowInsecure)
|
||||
{
|
||||
if(!StringUtils.hasContent(url))
|
||||
{
|
||||
return ("No URL is configured (check the channel's secret reference or settings).");
|
||||
}
|
||||
String lower = url.trim().toLowerCase(Locale.ROOT);
|
||||
if(lower.startsWith("https://") || (allowInsecure && lower.startsWith("http://")))
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
return ("The URL must use https.");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The channel's settings JSON as a map (empty when blank or unreadable).
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
public static Map<String, Serializable> settings(QRecord channel)
|
||||
{
|
||||
String json = channel == null ? null : channel.getValueString("settings");
|
||||
if(!StringUtils.hasContent(json))
|
||||
{
|
||||
return (new LinkedHashMap<>());
|
||||
}
|
||||
try
|
||||
{
|
||||
return (JsonUtils.toObject(json, LinkedHashMap.class));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (new LinkedHashMap<>());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Whether the channel allows plain http (local receivers, tests).
|
||||
*******************************************************************************/
|
||||
public static boolean allowInsecure(QRecord channel)
|
||||
{
|
||||
return (Boolean.TRUE.equals(settings(channel).get("allowInsecureHttp")));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** JSON bytes.
|
||||
*******************************************************************************/
|
||||
public static byte[] json(Object body)
|
||||
{
|
||||
return (JsonUtils.toJson(body).getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove a (secret) URL from an error message.
|
||||
*******************************************************************************/
|
||||
private static String stripUrl(String message, String url)
|
||||
{
|
||||
return (url == null ? message : message.replace(url, "[url]"));
|
||||
}
|
||||
}
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.delivery.channels;
|
||||
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.GeneralSecurityException;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** STAND-IN for core's WebhookSigner (QRun-IO/qqq#1109, the shared signer of
|
||||
** the qbit-webhooks assessment, QRun-IO/qqq#1105 D2). Same format: the
|
||||
** Standard Webhooks headers webhook-id, webhook-timestamp and
|
||||
** webhook-signature = "v1," + base64(HMAC-SHA256(key, id + "." + timestamp +
|
||||
** "." + body)); a "whsec_" secret is base64 after the prefix, any other text
|
||||
** is the raw key. Replace with com.kingsrook.qqq.backend.core.utils.webhooks
|
||||
** .WebhookSigner when the QQQ RC with #1109 ships, so receivers verify Alerts
|
||||
** and qbit-webhooks deliveries the same way.
|
||||
*******************************************************************************/
|
||||
public final class StandardWebhookSigner
|
||||
{
|
||||
public static final String HEADER_ID = "webhook-id";
|
||||
public static final String HEADER_TIMESTAMP = "webhook-timestamp";
|
||||
public static final String HEADER_SIGNATURE = "webhook-signature";
|
||||
|
||||
private static final String PREFIX = "whsec_";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Signing only.
|
||||
*******************************************************************************/
|
||||
private StandardWebhookSigner()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The three signature headers.
|
||||
*******************************************************************************/
|
||||
public static Map<String, String> headers(String messageId, Instant timestamp, byte[] body, String secret)
|
||||
{
|
||||
long seconds = timestamp.getEpochSecond();
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
headers.put(HEADER_ID, messageId);
|
||||
headers.put(HEADER_TIMESTAMP, Long.toString(seconds));
|
||||
headers.put(HEADER_SIGNATURE, "v1," + sign(messageId, seconds, body, secret));
|
||||
return (headers);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** base64 HMAC-SHA256 over id.timestamp.body.
|
||||
*******************************************************************************/
|
||||
public static String sign(String messageId, long timestampSeconds, byte[] body, String secret)
|
||||
{
|
||||
if(messageId == null || messageId.isEmpty() || messageId.contains(".") || messageId.chars().anyMatch(Character::isWhitespace))
|
||||
{
|
||||
throw (new IllegalArgumentException("A webhook message id is required and may not contain '.' or whitespace"));
|
||||
}
|
||||
try
|
||||
{
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(keyBytes(secret), "HmacSHA256"));
|
||||
mac.update((messageId + "." + timestampSeconds + ".").getBytes(StandardCharsets.UTF_8));
|
||||
mac.update(body == null ? new byte[0] : body);
|
||||
return (Base64.getEncoder().encodeToString(mac.doFinal()));
|
||||
}
|
||||
catch(GeneralSecurityException e)
|
||||
{
|
||||
throw (new IllegalStateException("HMAC-SHA256 is unavailable", e));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The key bytes of a secret's text form.
|
||||
*******************************************************************************/
|
||||
static byte[] keyBytes(String secret)
|
||||
{
|
||||
if(secret == null || secret.isBlank())
|
||||
{
|
||||
throw (new IllegalArgumentException("A webhook signing secret is required"));
|
||||
}
|
||||
String trimmed = secret.trim();
|
||||
return (trimmed.startsWith(PREFIX) ? Base64.getDecoder().decode(trimmed.substring(PREFIX.length())) : trimmed.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.engine;
|
||||
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Instant;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.alerts.AlertsRuntime;
|
||||
import com.kingsrook.qbits.alerts.delivery.AlertRedactor;
|
||||
import com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels;
|
||||
import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.Resolution;
|
||||
import com.kingsrook.qbits.alerts.model.AlertsType;
|
||||
import com.kingsrook.qbits.alerts.spi.AlertSeverity;
|
||||
import com.kingsrook.qbits.alerts.spi.AlertSignal;
|
||||
import com.kingsrook.qbits.alerts.spi.DetailLevel;
|
||||
import com.kingsrook.qbits.alerts.spi.QAlertTypeMetaData;
|
||||
import com.kingsrook.qbits.alerts.spi.ResolveMode;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.utils.StringUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Prometheus Alertmanager as a source (design section 5.7): the webhook
|
||||
** receiver payload (version 4) becomes raises and resolves.
|
||||
** - the alert's fingerprint is the dedup key; firing / resolved are raise /
|
||||
** resolve; the severity label maps to a severity (unknown: WARNING);
|
||||
** summary / description become title / summary (redacted); labels become
|
||||
** attributes (bounded);
|
||||
** - the type is prometheus.(alertname), created on first sight as an
|
||||
** EXTERNAL type in group "infrastructure" with notifications OFF
|
||||
** (Alertmanager already notifies people; turning QQQ notifications on is a
|
||||
** per-type choice);
|
||||
** - alerts carrying qqq_origin="qbit-alerts" (sent by this qbit's
|
||||
** Alertmanager channel) are ignored, so nothing loops.
|
||||
** Authentication: a bearer token compared (in constant time) with the
|
||||
** configured secret reference. Ingest is off without one.
|
||||
*******************************************************************************/
|
||||
public class AlertmanagerIngest
|
||||
{
|
||||
public static final String TYPE_PREFIX = "prometheus.";
|
||||
public static final String GROUP = "infrastructure";
|
||||
|
||||
private final AlertsRuntime runtime;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** What an ingest did.
|
||||
**
|
||||
** @param raised alerts raised (or refreshed)
|
||||
** @param resolved alerts resolved
|
||||
** @param ignored alerts ignored (loop guard, or no alertname)
|
||||
*******************************************************************************/
|
||||
public record Result(int raised, int resolved, int ignored)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Constructor
|
||||
*******************************************************************************/
|
||||
public AlertmanagerIngest(AlertsRuntime runtime)
|
||||
{
|
||||
this.runtime = runtime;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Check the bearer token: 404 when ingest is off, 401 when it doesn't match.
|
||||
*******************************************************************************/
|
||||
public void authenticate(String authorizationHeader) throws QException
|
||||
{
|
||||
String ref = runtime.getConfig().getAlertmanagerIngestTokenSecretRef();
|
||||
if(!StringUtils.hasContent(ref))
|
||||
{
|
||||
throw (new QNotFoundException("Alertmanager ingest is not enabled."));
|
||||
}
|
||||
String expected = runtime.getChannels().getSecrets().resolve(ref);
|
||||
String given = authorizationHeader != null && authorizationHeader.regionMatches(true, 0, "Bearer ", 0, 7) ? authorizationHeader.substring(7).trim() : null;
|
||||
if(expected == null || given == null || !MessageDigest.isEqual(expected.getBytes(StandardCharsets.UTF_8), given.getBytes(StandardCharsets.UTF_8)))
|
||||
{
|
||||
throw (new QAuthenticationException("Invalid ingest token."));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Apply one webhook payload.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
public Result ingest(Map<String, Object> payload) throws QException
|
||||
{
|
||||
Object alerts = payload == null ? null : payload.get("alerts");
|
||||
if(!(alerts instanceof List<?> list))
|
||||
{
|
||||
throw (new QBadRequestException("Expected an Alertmanager webhook payload with an \"alerts\" list."));
|
||||
}
|
||||
|
||||
int raised = 0;
|
||||
int resolved = 0;
|
||||
int ignored = 0;
|
||||
for(Object item : list)
|
||||
{
|
||||
if(!(item instanceof Map<?, ?> alert))
|
||||
{
|
||||
ignored++;
|
||||
continue;
|
||||
}
|
||||
Map<String, Object> labels = alert.get("labels") instanceof Map<?, ?> map ? (Map<String, Object>) map : Map.of();
|
||||
Map<String, Object> annotations = alert.get("annotations") instanceof Map<?, ?> map ? (Map<String, Object>) map : Map.of();
|
||||
String alertname = labels.get("alertname") == null ? null : String.valueOf(labels.get("alertname"));
|
||||
if(alertname == null || ExternalChannels.ORIGIN_VALUE.equals(String.valueOf(labels.get(ExternalChannels.ORIGIN_LABEL))))
|
||||
{
|
||||
ignored++;
|
||||
continue;
|
||||
}
|
||||
|
||||
String typeName = typeName(alertname);
|
||||
String fingerprint = alert.get("fingerprint") == null ? fingerprint(labels) : String.valueOf(alert.get("fingerprint"));
|
||||
ensureType(typeName, alertname);
|
||||
|
||||
if("resolved".equalsIgnoreCase(String.valueOf(alert.get("status"))))
|
||||
{
|
||||
if(runtime.getEngine().resolve(typeName, fingerprint, Resolution.PRODUCER, null, "Resolved in Alertmanager") != null)
|
||||
{
|
||||
resolved++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
String title = firstNonBlank(annotations.get("summary"), annotations.get("description"), alertname);
|
||||
AlertSignal signal = new AlertSignal(typeName)
|
||||
.withDedupKey(fingerprint)
|
||||
.withSeverity(AlertSeverity.fromLabel(labels.get("severity") == null ? null : String.valueOf(labels.get("severity")), AlertSeverity.WARNING))
|
||||
.withTitle(AlertRedactor.redact(title))
|
||||
.withSummary(AlertRedactor.redact(annotations.get("description") == null ? null : String.valueOf(annotations.get("description"))))
|
||||
.withOccurredAt(parseInstant(alert.get("startsAt")));
|
||||
for(Map.Entry<String, Object> label : new LinkedHashMap<>(labels).entrySet())
|
||||
{
|
||||
signal.withAttribute(label.getKey(), String.valueOf(label.getValue()));
|
||||
}
|
||||
runtime.getEngine().raise(signal);
|
||||
raised++;
|
||||
}
|
||||
return (new Result(raised, resolved, ignored));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create the EXTERNAL type the first time (notifications off).
|
||||
*******************************************************************************/
|
||||
private void ensureType(String typeName, String alertname) throws QException
|
||||
{
|
||||
runtime.asSystem(() ->
|
||||
{
|
||||
if(AlertsDb.first(AlertsType.TABLE_NAME, AlertsDb.eq("name", typeName)) == null)
|
||||
{
|
||||
QRecord type = runtime.getTypeSync().upsertType(new QAlertTypeMetaData()
|
||||
.withName(typeName)
|
||||
.withLabel(alertname)
|
||||
.withSourceName("prometheus")
|
||||
.withGroupName(GROUP)
|
||||
.withDefaultSeverity(AlertSeverity.WARNING)
|
||||
.withResolveMode(ResolveMode.PRODUCER)
|
||||
.withExternalDetailCeiling(DetailLevel.STANDARD)
|
||||
.withDescription("Ingested from Prometheus Alertmanager (" + alertname + ").")
|
||||
.withHowToFix("See the alert's runbook in Prometheus. Alertmanager already notifies its receivers; turn on Notify here only to route it through Alerts too."));
|
||||
AlertsDb.update(AlertsType.TABLE_NAME, new QRecord().withValue("id", type.getValue("id")).withValue("notify", false));
|
||||
}
|
||||
return (null);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** "HighCPU Usage" to "prometheus.highcpu-usage".
|
||||
*******************************************************************************/
|
||||
public static String typeName(String alertname)
|
||||
{
|
||||
String slug = alertname.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9-]+", "-").replaceAll("^-+|-+$", "");
|
||||
return (TYPE_PREFIX + (slug.isEmpty() ? "unnamed" : slug));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A fingerprint from the labels when Alertmanager didn't send one.
|
||||
*******************************************************************************/
|
||||
static String fingerprint(Map<String, Object> labels)
|
||||
{
|
||||
return (Integer.toHexString(new java.util.TreeMap<>(labels).toString().hashCode()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static String firstNonBlank(Object... values)
|
||||
{
|
||||
for(Object value : values)
|
||||
{
|
||||
if(value != null && StringUtils.hasContent(String.valueOf(value)))
|
||||
{
|
||||
return (String.valueOf(value));
|
||||
}
|
||||
}
|
||||
return (null);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An ISO instant, or null (Alertmanager's zero time is null too).
|
||||
*******************************************************************************/
|
||||
private static Instant parseInstant(Object value)
|
||||
{
|
||||
try
|
||||
{
|
||||
Instant instant = value == null ? null : Instant.parse(String.valueOf(value));
|
||||
return (instant == null || instant.getEpochSecond() <= 0 ? null : instant);
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (null);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.delivery;
|
||||
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.OutputStream;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import com.kingsrook.qbits.alerts.delivery.channels.ExternalChannels;
|
||||
import com.kingsrook.qbits.alerts.delivery.channels.StandardWebhookSigner;
|
||||
import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The external channels against a local HTTP receiver: request shapes
|
||||
** (Slack Block Kit, Teams Adaptive Card, signed webhook, PagerDuty Events v2,
|
||||
** Alertmanager v2), outcome classification, https enforcement and secrets.
|
||||
*******************************************************************************/
|
||||
class ExternalChannelsTest
|
||||
{
|
||||
private HttpServer server;
|
||||
private String base;
|
||||
private final List<Received> received = new ArrayList<>();
|
||||
private final AtomicInteger status = new AtomicInteger(200);
|
||||
private AlertSecretResolver secrets;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One captured request.
|
||||
*******************************************************************************/
|
||||
record Received(String path, Map<String, List<String>> headers, String body)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void start() throws IOException
|
||||
{
|
||||
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
|
||||
server.createContext("/", exchange ->
|
||||
{
|
||||
String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
|
||||
received.add(new Received(exchange.getRequestURI().getPath(), exchange.getRequestHeaders(), body));
|
||||
byte[] reply = "{\"ok\":true}".getBytes(StandardCharsets.UTF_8);
|
||||
exchange.sendResponseHeaders(status.get(), reply.length);
|
||||
try(OutputStream out = exchange.getResponseBody())
|
||||
{
|
||||
out.write(reply);
|
||||
}
|
||||
});
|
||||
server.start();
|
||||
base = "http://127.0.0.1:" + server.getAddress().getPort();
|
||||
|
||||
QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
|
||||
interpreter.addValueMap("test", Map.of("URL", base + "/hook", "KEY", "whsec_c2VjcmV0LWtleS1mb3ItdGVzdHM=", "ROUTING", "R0UT1NG", "BEARER", "am-token"));
|
||||
secrets = new AlertSecretResolver(interpreter);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void stop()
|
||||
{
|
||||
server.stop(0);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static QRecord channel(String type, String secretRef, Map<String, Object> settings)
|
||||
{
|
||||
return (new QRecord().withValue("channelType", type).withValue("secretRef", secretRef).withValue("settings", JsonUtils.toJson(settings)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static AlertMessage message(String kind, String status)
|
||||
{
|
||||
AlertMessage.Entry entry = new AlertMessage.Entry(7, "cdn.purge.dead", "CDN purge failed", "provider:cf", "ERROR", status, "3 CDN purges failed", "Retry later.", "CDN", 3,
|
||||
"https://admin.example.com/app/alerts_alert/7", "2026-10-12T10:00:00Z", "RESOLVED".equals(status) ? "2026-10-12T11:00:00Z" : null);
|
||||
return (new AlertMessage("[ERROR] 3 CDN purges failed", "[ERROR] CDN purge failed / CDN: 3 CDN purges failed (2 h)\n<b>x</b>", "<p>x</p>", "ERROR", kind, List.of(7),
|
||||
"https://admin.example.com/app/alerts_alert/7", List.of(entry)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Slack and Teams post to the secret URL.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testSlackAndTeams() throws Exception
|
||||
{
|
||||
ExternalChannels.Slack slack = new ExternalChannels.Slack();
|
||||
assertThat(slack.validate(channel("SLACK", null, Map.of()), secrets)).hasSize(1);
|
||||
QRecord slackChannel = channel("SLACK", "${test.URL}", Map.of("allowInsecureHttp", true));
|
||||
assertThat(slack.validate(slackChannel, secrets)).isEmpty();
|
||||
assertThat(slack.deliver(message("RAISED", "OPEN"), slackChannel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT);
|
||||
Map<?, ?> slackBody = JsonUtils.toObject(received.get(0).body(), Map.class);
|
||||
assertThat(slackBody.get("text")).isEqualTo("[ERROR] 3 CDN purges failed");
|
||||
assertThat(received.get(0).body()).contains("<b>x</b>").contains("Open in Alerts");
|
||||
|
||||
ExternalChannels.Teams teams = new ExternalChannels.Teams();
|
||||
QRecord teamsChannel = channel("TEAMS", "${test.URL}", Map.of("allowInsecureHttp", true));
|
||||
assertThat(teams.validate(channel("TEAMS", null, Map.of()), secrets)).hasSize(1);
|
||||
assertThat(teams.deliver(message("RAISED", "OPEN"), teamsChannel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT);
|
||||
assertThat(received.get(1).body()).contains("application/vnd.microsoft.card.adaptive").contains("AdaptiveCard").contains("Action.OpenUrl");
|
||||
|
||||
//////////////////////////////////////////////
|
||||
// https only, unless explicitly allowed //
|
||||
//////////////////////////////////////////////
|
||||
DeliveryResult insecure = slack.deliver(message("RAISED", "OPEN"), channel("SLACK", "${test.URL}", Map.of()), null, secrets);
|
||||
assertThat(insecure.outcome()).isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
assertThat(insecure.error()).contains("https").doesNotContain(base);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The webhook is signed with the shared Standard Webhooks format.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testSignedWebhook() throws Exception
|
||||
{
|
||||
ExternalChannels.Webhook webhook = new ExternalChannels.Webhook();
|
||||
assertThat(webhook.validate(channel("WEBHOOK", null, Map.of()), secrets)).hasSize(2);
|
||||
assertThat(webhook.deliver(message("RAISED", "OPEN"), channel("WEBHOOK", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
|
||||
QRecord channel = channel("WEBHOOK", "${test.KEY}", Map.of("url", base + "/receiver", "allowInsecureHttp", true));
|
||||
assertThat(webhook.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.SENT);
|
||||
Received request = received.get(0);
|
||||
assertThat(request.path()).isEqualTo("/receiver");
|
||||
String id = request.headers().get("Webhook-id").get(0);
|
||||
String timestamp = request.headers().get("Webhook-timestamp").get(0);
|
||||
String signature = request.headers().get("Webhook-signature").get(0);
|
||||
assertThat(id).startsWith("msg_");
|
||||
String expected = StandardWebhookSigner.sign(id, Long.parseLong(timestamp), request.body().getBytes(StandardCharsets.UTF_8), "whsec_c2VjcmV0LWtleS1mb3ItdGVzdHM=");
|
||||
assertThat(signature).isEqualTo("v1," + expected);
|
||||
assertThat(request.body()).contains("\"type\":\"qqq.alert.resolved\"").contains("\"dedupKey\":\"provider:cf\"");
|
||||
|
||||
////////////////////////////////////////////////////
|
||||
// the id is stable across retries of one message //
|
||||
////////////////////////////////////////////////////
|
||||
webhook.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets);
|
||||
assertThat(received.get(1).headers().get("Webhook-id").get(0)).isEqualTo(id);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** PagerDuty: trigger / acknowledge / resolve with a stable dedup key.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testPagerDuty() throws Exception
|
||||
{
|
||||
ExternalChannels.PagerDuty pagerDuty = new ExternalChannels.PagerDuty();
|
||||
assertThat(pagerDuty.validate(channel("PAGERDUTY", null, Map.of()), secrets)).hasSize(1);
|
||||
assertThat(pagerDuty.deliver(message("RAISED", "OPEN"), channel("PAGERDUTY", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
QRecord channel = channel("PAGERDUTY", "${test.ROUTING}", Map.of("url", base + "/v2/enqueue", "allowInsecureHttp", true, "source", "kof22"));
|
||||
|
||||
pagerDuty.deliver(message("RAISED", "OPEN"), channel, null, secrets);
|
||||
pagerDuty.deliver(message("RAISED", "ACKNOWLEDGED"), channel, null, secrets);
|
||||
pagerDuty.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets);
|
||||
List<Map<?, ?>> events = new ArrayList<>();
|
||||
for(Received request : received)
|
||||
{
|
||||
events.add(JsonUtils.toObject(request.body(), Map.class));
|
||||
}
|
||||
assertThat(events.stream().map(event -> String.valueOf(event.get("event_action"))).toList()).containsExactly("trigger", "acknowledge", "resolve");
|
||||
assertThat(events).allMatch(event -> "cdn.purge.dead:provider:cf".equals(event.get("dedup_key")) && "R0UT1NG".equals(event.get("routing_key")));
|
||||
Map<?, ?> payload = (Map<?, ?>) events.get(0).get("payload");
|
||||
assertThat(payload.get("severity")).isEqualTo("error");
|
||||
assertThat(payload.get("source")).isEqualTo("kof22");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Alertmanager sink: v2 alerts with the qqq labels and the origin guard.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testAlertmanagerSink() throws Exception
|
||||
{
|
||||
ExternalChannels.Alertmanager alertmanager = new ExternalChannels.Alertmanager();
|
||||
assertThat(alertmanager.validate(channel("ALERTMANAGER", null, Map.of()), secrets)).hasSize(1);
|
||||
assertThat(alertmanager.deliver(message("RAISED", "OPEN"), channel("ALERTMANAGER", null, Map.of()), null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
QRecord channel = channel("ALERTMANAGER", "${test.BEARER}", Map.of("url", base + "/", "allowInsecureHttp", true));
|
||||
|
||||
alertmanager.deliver(message("RAISED", "OPEN"), channel, null, secrets);
|
||||
alertmanager.deliver(message("RESOLVED", "RESOLVED"), channel, null, secrets);
|
||||
assertThat(received.get(0).path()).isEqualTo("/api/v2/alerts");
|
||||
assertThat(received.get(0).headers().get("Authorization").get(0)).isEqualTo("Bearer am-token");
|
||||
List<?> firing = JsonUtils.toObject(received.get(0).body(), List.class);
|
||||
Map<?, ?> labels = (Map<?, ?>) ((Map<?, ?>) firing.get(0)).get("labels");
|
||||
assertThat(labels.get("qqq_origin")).isEqualTo("qbit-alerts");
|
||||
assertThat(labels.get("qqq_type")).isEqualTo("cdn.purge.dead");
|
||||
assertThat(labels.get("severity")).isEqualTo("error");
|
||||
List<?> resolvedBody = JsonUtils.toObject(received.get(1).body(), List.class);
|
||||
assertThat(((Map<?, ?>) resolvedBody.get(0)).get("endsAt")).isEqualTo("2026-10-12T11:00:00Z");
|
||||
|
||||
//////////////////////////////////////////////////////
|
||||
// a test message is one stand-in alert, already over //
|
||||
//////////////////////////////////////////////////////
|
||||
alertmanager.deliver(new AlertMessage("TEST: hello", "t", "h", "INFO", "TEST", List.of(), "https://x/app/alerts_alert"), channel, null, secrets);
|
||||
assertThat(received.get(2).body()).contains("alerts.test");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Outcomes: 5xx and 429 retry; other 4xx fail permanently; no receiver
|
||||
** retries.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testOutcomes()
|
||||
{
|
||||
ExternalChannels.Slack slack = new ExternalChannels.Slack();
|
||||
QRecord channel = channel("SLACK", "${test.URL}", Map.of("allowInsecureHttp", true));
|
||||
status.set(503);
|
||||
assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY);
|
||||
status.set(429);
|
||||
assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY);
|
||||
status.set(404);
|
||||
DeliveryResult notFound = slack.deliver(message("RAISED", "OPEN"), channel, null, secrets);
|
||||
assertThat(notFound.outcome()).isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
assertThat(notFound.error()).startsWith("HTTP 404");
|
||||
|
||||
ExternalChannels.PagerDuty pagerDuty = new ExternalChannels.PagerDuty();
|
||||
assertThat(pagerDuty.deliver(message("RAISED", "OPEN"), channel("PAGERDUTY", "${test.ROUTING}", Map.of("url", base, "allowInsecureHttp", true)), null, secrets).outcome())
|
||||
.isEqualTo(DeliveryResult.Outcome.FAILED);
|
||||
|
||||
server.stop(0);
|
||||
assertThat(slack.deliver(message("RAISED", "OPEN"), channel, null, secrets).outcome()).isEqualTo(DeliveryResult.Outcome.RETRY);
|
||||
assertThat(ExternalChannels.all()).hasSize(5);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.alerts.engine;
|
||||
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.alerts.AlertsQBitConfig;
|
||||
import com.kingsrook.qbits.alerts.BaseAlertsTest;
|
||||
import com.kingsrook.qbits.alerts.model.AlertsAlert;
|
||||
import com.kingsrook.qbits.alerts.model.AlertsType;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QAuthenticationException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Alertmanager as a source: authentication, firing and resolved alerts, the
|
||||
** EXTERNAL type with notifications off, and the loop guard.
|
||||
*******************************************************************************/
|
||||
class AlertmanagerIngestTest extends BaseAlertsTest
|
||||
{
|
||||
static final String TOKEN_PROPERTY = "alerts.test.ingest.token";
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@BeforeAll
|
||||
static void setToken()
|
||||
{
|
||||
System.setProperty(TOKEN_PROPERTY, "s3cret-ingest");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@AfterAll
|
||||
static void clearToken()
|
||||
{
|
||||
System.clearProperty(TOKEN_PROPERTY);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
protected void customizeConfig(AlertsQBitConfig config)
|
||||
{
|
||||
config.withAlertmanagerIngestTokenSecretRef("${prop." + TOKEN_PROPERTY + "}");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
private static Map<String, Object> alert(String status, String alertname, String fingerprint, Map<String, Object> extraLabels)
|
||||
{
|
||||
Map<String, Object> labels = new java.util.LinkedHashMap<>(Map.of("alertname", alertname, "severity", "critical", "instance", "node-1"));
|
||||
labels.putAll(extraLabels);
|
||||
return (Map.of("status", status, "labels", labels, "annotations", Map.of("summary", "CPU high on node-1", "description", "Over 90% for 10 minutes."),
|
||||
"startsAt", "2026-10-12T11:00:00Z", "fingerprint", fingerprint));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testAuthentication() throws Exception
|
||||
{
|
||||
AlertmanagerIngest ingest = new AlertmanagerIngest(runtime);
|
||||
ingest.authenticate("Bearer s3cret-ingest");
|
||||
assertThatThrownBy(() -> ingest.authenticate("Bearer nope")).isInstanceOf(QAuthenticationException.class);
|
||||
assertThatThrownBy(() -> ingest.authenticate(null)).isInstanceOf(QAuthenticationException.class);
|
||||
|
||||
config.withAlertmanagerIngestTokenSecretRef(null);
|
||||
assertThatThrownBy(() -> ingest.authenticate("Bearer s3cret-ingest")).isInstanceOf(QNotFoundException.class);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
**
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFiringResolvedAndLoopGuard() throws Exception
|
||||
{
|
||||
AlertmanagerIngest ingest = new AlertmanagerIngest(runtime);
|
||||
AlertmanagerIngest.Result first = ingest.ingest(Map.of("version", "4", "alerts", List.of(
|
||||
alert("firing", "HighCPU Usage", "abc123", Map.of()),
|
||||
alert("firing", "AlertsFromQqq", "def456", Map.of("qqq_origin", "qbit-alerts")),
|
||||
"not an alert")));
|
||||
assertThat(first).isEqualTo(new AlertmanagerIngest.Result(1, 0, 2));
|
||||
|
||||
QRecord type = runtime.asSystem(() -> AlertsDb.first(AlertsType.TABLE_NAME, AlertsDb.eq("name", "prometheus.highcpu-usage")));
|
||||
assertThat(type.getValueBoolean("notify")).isFalse();
|
||||
assertThat(type.getValueString("sourceName")).isEqualTo("prometheus");
|
||||
QRecord alert = all(AlertsAlert.TABLE_NAME).get(0);
|
||||
assertThat(alert.getValueString("severity")).isEqualTo("CRITICAL");
|
||||
assertThat(alert.getValueString("title")).isEqualTo("CPU high on node-1");
|
||||
assertThat(alert.getValueString("dedupKey")).isEqualTo("abc123");
|
||||
assertThat(alert.getValueString("attributes")).contains("instance");
|
||||
|
||||
AlertmanagerIngest.Result second = ingest.ingest(Map.of("alerts", List.of(alert("resolved", "HighCPU Usage", "abc123", Map.of()))));
|
||||
assertThat(second.resolved()).isEqualTo(1);
|
||||
assertThat(all(AlertsAlert.TABLE_NAME).get(0).getValueString("status")).isEqualTo("RESOLVED");
|
||||
|
||||
assertThatThrownBy(() -> ingest.ingest(Map.of())).isInstanceOf(QBadRequestException.class);
|
||||
assertThat(AlertmanagerIngest.typeName("!!")).isEqualTo("prometheus.unnamed");
|
||||
assertThat(AlertmanagerIngest.fingerprint(Map.of("a", "b"))).isNotBlank();
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user