Merge remote-tracking branch 'origin/develop' into feature/reminders-escalation

# Conflicts:
#	src/main/resources/db/liquibase/qbit-alerts/alerts.changelog.sql
#	src/test/java/com/kingsrook/qbits/alerts/AlertsQBitProducerTest.java

CHANGELOG.md

@@ -5,6 +5,8 @@ All notable changes to this project are documented here. The format follows Keep
 ## [0.1.0] - Unreleased
 
 ### Changed
+- Team subscriptions now combine explicit members with ROLE and IDP_GROUP users from the optional public `AlertAudienceResolver` configured through `withAudienceResolver`. Recipients are deduplicated and retain per-user visibility checks. Missing resolvers resolve to nobody, log the reason, and show configuration-aware help on team/member screens.
+- Fix the alert Mute action to target its subject record, or its type and dedup key when no subject exists. Add the separate "Mute this alert type" action. RECORD mutes now compare table and record ID. Guarded changesets 21–22 add `scope_record_id` and allow full ALERT keys; existing type-wide mutes retain their scope.
 - Switch the optional qbit-email integration to its published `com.kingsrook.qbits.email.api` package in `0.1.0-RC.2`, preserving templates, categories, idempotency keys, linked records and outcome mapping. Narrow the architecture allowance to that API package while retaining all guardrails.
 - Email goes through qbit-email when it is produced in the instance (optional dependency, `qbit-email.version` 0.1.0-RC.2): alert channel emails (template `alerts_alert`), notification emails (`alerts_notification`) and digests (`alerts_digest`), all NOTIFICATION, one recipient per message, with idempotency keys and the alert or notification as the linked record. `messagingProviderName` stays as the fallback when qbit-email is absent, and is not required when it is present.
 - Target released QQQ `4.1.0-RC.6`; verify normally with `./mvnw -B verify` (no version override).
@@ -21,6 +23,9 @@ All notable changes to this project are documented here. The format follows Keep
 - Webhook signing delegates to core `WebhookSigner`/`WebhookSecret` (qbit-alerts#15, F11). Headers, `v1,` signatures and `whsec_` encoding are unchanged. A thin adapter preserves legacy keys shorter than core's 16-byte minimum using equivalent HMAC zero-padding.
 
 ### Added
+- Apply reminders and escalation (#27, part 2): a minute-based hidden job sends due reminders, advances ordered USER/TEAM/CHANNEL escalation steps, and caps final-step repeats. Acknowledgement, snooze, resolution and mutes stop both; reopening restarts escalation. Delivery, events and progress commit together, with a persisted same-minute checkpoint.
+- Add admin escalation policy and step tables, child step lists, and type/group policy links in guarded Liquibase changesets 23–26. Type policies override group policies. CRITICAL reminders default to 60 minutes; explicit zero turns them off.
+- Keep EXPIRED reserved and remove it from selectable resolution values because the design specifies no expiration rule.
 - Optional external channels:
   - Slack (an Incoming Webhook, Block Kit)
   - Microsoft Teams (a Workflows webhook, an Adaptive Card)

README.md

@@ -31,6 +31,40 @@ Any qbit or app reports exceptional conditions ("a person should do something").
 
 All data is in standard QQQ tables named `alerts_*`. People change alerts through record actions, never by editing rows.
 
+## Team audiences and mutes
+
+Teams keep their source in `alerts_team.memberSource` and `memberSourceValue`.
+Rows in `alerts_team_member` are explicit user memberships. A team's ROLE or
+IDP_GROUP source adds users to those explicit members at routing time. Register
+`com.kingsrook.qbits.alerts.AlertAudienceResolver` before producing the qbit:
+
+```java
+config.withAudienceResolver((sourceType, sourceValue) -> hostDirectory.resolveUserIds(sourceType, sourceValue));
+```
+
+The host implements that directory lookup through its role or identity-provider
+public API; this qbit has no dependency on either implementation. Return a
+`Set<String>` of user IDs (an empty set for an unknown source/value), bounded by
+`maxAudienceSize` (50,000 by default). Users are deduplicated across explicit
+members, resolved members and subscriptions, and each still passes the router's
+visibility checks. Missing or failing resolvers contribute no users and log a
+clear reason; explicit members still receive alerts. Team and member screens
+show the missing-resolver help text only when no resolver was configured.
+
+The alert's **Mute** record action silences its subject table and record ID when
+there is one. Without a subject record, it silences only that alert's type and
+dedup key, including later occurrences of the same problem. **Mute this alert
+type** is a separate record action on alerts and alert types. Both use the usual
+Review and Result summaries and require a reason and a duration. Mutes remain
+shared rules, not personal notification preferences.
+
+RECORD mutes store the table in `scopeValue` and its record ID in `scopeRecordId`.
+ALERT mutes store the engine's full `type|dedupKey` in `scopeValue`. Apply guarded
+Liquibase changesets 21–22 before running this version: they add the record ID
+column and widen mute scope values to 520 characters. Existing type-wide mutes
+remain type-wide; historical RECORD mutes need their intended record ID filled
+in because previous versions did not store it.
+
 ## User notifications
 
 The qbit also carries ordinary notifications for people (design: [docs/NOTIFICATIONS.md](docs/NOTIFICATIONS.md)):
@@ -167,14 +201,57 @@ When qbit-email is produced in the same instance (and on the classpath; it is an
 - **Text** in the variables is the existing redacted, external-safe text. No record labels are passed.
 - **Config.** `messagingProviderName` and `fromAddress` are not needed (the From address comes from the email provider). Without qbit-email they work as before.
 
+## Reminders and escalation
+
+The hidden `alertsEscalateAndRemind` process runs every minute on `schedulerName`.
+Without a configured scheduler, hosts can invoke the process themselves. Apply
+Liquibase changesets 23–26 before running this version.
+
+An OPEN, unacknowledged alert sends REMINDER deliveries to matching subscriptions
+whose events include REMINDER (an empty events list includes all kinds). Set the
+type's `reminderIntervalMinutes`: null defaults to 60 for CRITICAL alerts and off
+otherwise; zero disables reminders, including CRITICAL. The interval starts at
+the later of the latest raise and the latest successful notification. Delivery
+uses the normal permission, channel and suppression checks, writes NOTIFIED in
+the timeline when something is queued, and updates `lastNotifiedAt`.
+
+Admins can create **Escalation Policies** in the Alerts app and add **Steps** on a
+policy's record screen. Attach `escalationPolicyId` to a type or group; a type's
+explicit policy takes precedence, including when that policy is disabled. Steps
+have a positive, unique `stepOrder`, a nonnegative `afterMinutes`, and exactly
+one USER (`userId`), TEAM (`teamId`) or CHANNEL (`channelId`) target. User and team
+targets use the users' normal in-app/email preferences. Steps apply only to OPEN
+alerts matching `onlyIfStatus` (default OPEN). Timing starts at the first raise
+or the latest reopen; repeated raises do not postpone escalation.
+
+Each due step sends ESCALATED through the normal delivery checks, records an
+ESCALATED event with the previous and new levels, and advances `escalationLevel`.
+After the final step, positive `repeatAfterMinutes` and `maxRepeats` allow that
+many additional attempts at the final target. Null/zero disables repeats. Repeats
+leave the level unchanged and are counted in the timeline within the current
+open cycle. Reopening resets the level and repeat budget. One step runs per tick;
+a delayed scheduler catches up over subsequent minutes instead of sending a burst.
+Acknowledgement, snooze, resolution, mute, flapping, and a disabled or silent type
+stop timed notifications. Suppressed deliveries still count as escalation attempts.
+
+Each alert's delivery rows, timeline and progress commit together. The persisted
+`timedNotificationCheckedAt` prevents duplicate attempts in the same minute,
+including when no recipient is eligible. Run this job on **one scheduler node**;
+there is no distributed lock between scheduler nodes. No ESCALATION feature
+switch exists in the current config; enable or disable individual policies.
+
+`EXPIRED` remains a reserved resolution enum value. The design defines no alert
+expiration rule, so it is excluded from selectable resolution values and is never
+assigned automatically. The Resolve action continues to record MANUAL.
+
 ## Known limits (v1)
 
-- **Claims are best-effort.** Delivery and monitor claims use a lease, not an atomic conditional update. With several scheduler nodes, run the two scheduled processes on one node, or accept at-least-once delivery.
+- **Claims are best-effort.** Delivery and monitor claims use a lease, not an atomic conditional update. With several scheduler nodes, run the scheduled processes on one node, or accept at-least-once delivery.
 - **Email threading headers.** `Message-ID` and `In-Reply-To` need core messaging support for custom headers, so they are not set.
 - **Recipient visibility.** It is checked through `createAutomatedSessionForUser`. If a host's authentication module can't build a session with permissions for a user, email goes out only for alerts that have no visibility permission and no subject table. In-app delivery is always filtered by the record locks.
 - **Outbound HTTP** blocks plain http, but has no full SSRF guard. Channels are admin-only. Core's `SafeHttpClient` (QRun-IO/qqq#1109) replaces the client when it is released.
 - **Alertmanager sink:** open alerts are sent with an `endsAt` a week out, not re-posted.
-- **Not built yet:** digests, quiet hours, reminders, Micrometer metrics, ROLE and IdP-group team resolvers, escalation, mirrored lock columns and a configurable table namespace. These are the rest of design phases P4 to P6.
+- **Not built yet:** digests, quiet hours, Micrometer metrics, ROLE and IdP-group team resolvers, mirrored lock columns and a configurable table namespace. These are the rest of design phases P4 to P6.
 
 ## Build
 

docs/DESIGN.md

@@ -3,6 +3,10 @@
 The design for this qbit is tracked in [QRun-IO/qqq#1090](https://github.com/QRun-IO/qqq/pull/1090). This repository implements it. Phases P1 to P3 are built (see README.md); P4 to P6 remain.
 
 Deviations from the design, and why:
+- **Timed notifications (#27, part 2):** reminders and escalation are implemented ahead of the remaining P4–P6 features. The minute job uses `stepOrder` and typed `userId` / `teamId` / `channelId` fields instead of the design's generic `targetId`. Policy description and enabled fields make the standard admin records usable. Types override groups; only OPEN, unacknowledged alerts run, even if a step is configured for another status, because acknowledgement must stop escalation.
+- **Escalation progress:** existing ESCALATED events hold level transitions and final-step repeat attempts; repeats keep the same level. Reopen events define each cycle's start and repeat budget. One step runs per minute, including when several steps became overdue. Null/zero repeat settings disable repeats. Suppression consumes an escalation attempt, as the normal router records why no delivery was queued.
+- **Timed notification transactions:** `AlertsDb` passes the current QQQ transaction to table actions. Each alert's timed fan-out, events and progress commit together. A read-only `timedNotificationCheckedAt` field prevents repeated evaluation in the same minute, including attempts with no eligible recipients. Alert and team scans use bounded pages. Runtime ticks are serialized locally; hosts must run the job on one scheduler node, matching the existing single-node scheduling guidance.
+- **Reminders and EXPIRED:** null reminder interval means 60 minutes for CRITICAL and off otherwise; explicit zero is off. Reminders use subscription event filters and NOTIFIED timeline entries, without introducing a REMINDER event. Record followers do not receive scheduled reminders without a matching subscription. EXPIRED remains reserved but is excluded from the resolution possible values; the design defines no expiration behavior. No ESCALATION feature switch is present in the current config.
 - **UI API and record following:** the RC.6 alerts extension, collaboration ownership of follow, and machine webhook migration are specified in [UI-API.md](UI-API.md). Existing RECORD watch subscriptions remain readable without migration.
 - **Alerts and notifications use core's RC.6 producer API**, including sink registration in core instance metadata (QRun-IO/qqq#1108 and #1125). `AlertsSinkAdapter` delegates `raise`, `resolve`, `heartbeat` and `notify` to the runtime queue. No local SPI types remain; storage-specific audience and severity conversion belongs to `NotificationService`.
 - **Visibility checks 1 to 3 (section 7.2)** are enforced by record security locks on session keys (`alerts_permission`, `alerts_readable_table`), not by a pre-query customizer. QQQ has no pre-query customizer, and locks also apply to counts, exports and the API.

src/main/java/com/kingsrook/qbits/alerts/AlertAudienceResolver.java

@@ -0,0 +1,37 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.alerts;
+
+
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** Host SPI for ROLE and IDP_GROUP team membership. Implementations use the
+ ** host's public APIs and return user IDs, never display names. Return an empty
+ ** set for an unknown source or value. Results must be bounded by the configured
+ ** maxAudienceSize; routing still checks every user's alert visibility.
+ *******************************************************************************/
+@FunctionalInterface
+public interface AlertAudienceResolver
+{
+   /*******************************************************************************
+    ** Resolve the team's memberSource and memberSourceValue to user IDs.
+    *******************************************************************************/
+   Set<String> resolve(String sourceType, String sourceValue) throws QException;
+}

src/main/java/com/kingsrook/qbits/alerts/AlertsQBitConfig.java

@@ -22,6 +22,8 @@ import java.util.List;
 import com.kingsrook.qbits.alerts.model.AlertsAlert;
 import com.kingsrook.qbits.alerts.model.AlertsChannel;
 import com.kingsrook.qbits.alerts.model.AlertsDelivery;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationPolicy;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationStep;
 import com.kingsrook.qbits.alerts.model.AlertsEvent;
 import com.kingsrook.qbits.alerts.model.AlertsGroup;
 import com.kingsrook.qbits.alerts.model.AlertsHeartbeat;
@@ -106,10 +108,33 @@ public class AlertsQBitConfig implements QBitConfig
    private Boolean        notifyFailedJobs          = true;
    private String         jobRunLinkPattern;
 
+   private transient AlertAudienceResolver audienceResolver;
+
    private transient AlertsRuntime runtime;
 
 
 
+   /*******************************************************************************
+    ** Optional host resolver for ROLE and IDP_GROUP team members.
+    *******************************************************************************/
+   public AlertAudienceResolver getAudienceResolver()
+   {
+      return (audienceResolver);
+   }
+
+
+
+   /*******************************************************************************
+    ** Register the host resolver before producing the qbit's metadata.
+    *******************************************************************************/
+   public AlertsQBitConfig withAudienceResolver(AlertAudienceResolver audienceResolver)
+   {
+      this.audienceResolver = audienceResolver;
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** The backend must be named and exist; numbers must be positive.
     *******************************************************************************/
@@ -153,7 +178,7 @@ public class AlertsQBitConfig implements QBitConfig
     *******************************************************************************/
    public static List<String> allTableNames()
    {
-      return (List.of(AlertsAlert.TABLE_NAME, AlertsMonitor.TABLE_NAME, AlertsType.TABLE_NAME, AlertsGroup.TABLE_NAME, AlertsSubscription.TABLE_NAME,
+      return (List.of(AlertsEscalationPolicy.TABLE_NAME, AlertsEscalationStep.TABLE_NAME, AlertsAlert.TABLE_NAME, AlertsMonitor.TABLE_NAME, AlertsType.TABLE_NAME, AlertsGroup.TABLE_NAME, AlertsSubscription.TABLE_NAME,
          AlertsUserPreference.TABLE_NAME, AlertsTeam.TABLE_NAME, AlertsTeamMember.TABLE_NAME, AlertsChannel.TABLE_NAME, AlertsMute.TABLE_NAME,
          AlertsDelivery.TABLE_NAME, AlertsEvent.TABLE_NAME, AlertsRecordRef.TABLE_NAME, AlertsHeartbeat.TABLE_NAME, AlertsNotification.TABLE_NAME,
          AlertsNotificationRecipient.TABLE_NAME, AlertsNotificationPreference.TABLE_NAME));

src/main/java/com/kingsrook/qbits/alerts/engine/AlertRouter.java

@@ -22,9 +22,12 @@ import java.time.temporal.ChronoUnit;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qbits.alerts.AlertAudienceResolver;
 import com.kingsrook.qbits.alerts.AlertsRuntime;
 import com.kingsrook.qbits.alerts.AlertsSecurity;
 import com.kingsrook.qbits.alerts.engine.AlertsEngine.AlertTarget;
@@ -37,11 +40,13 @@ import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.ChannelType;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.DeliveryKind;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.DeliveryStatus;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.EventType;
+import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.MemberSource;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.MuteEffect;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.ScopeType;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.SubscriberType;
 import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.SuppressedReason;
 import com.kingsrook.qbits.alerts.model.AlertsSubscription;
+import com.kingsrook.qbits.alerts.model.AlertsTeam;
 import com.kingsrook.qbits.alerts.model.AlertsTeamMember;
 import com.kingsrook.qbits.alerts.model.AlertsType;
 import com.kingsrook.qbits.alerts.model.AlertsUserPreference;
@@ -142,129 +147,8 @@ public class AlertRouter
 
       try
       {
-         Instant       now      = runtime.now();
-         AlertTarget   target   = AlertTarget.of(alert);
-         QRecord       type     = AlertsDb.byId(AlertsType.TABLE_NAME, alert.getValue("alertTypeId"));
-         AlertSeverity severity = AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING);
-
-         List<Destination> destinations = destinations(target, severity, kind);
-         if(destinations.isEmpty())
-         {
-            return;
-         }
-
-         boolean silent    = type != null && Boolean.FALSE.equals(type.getValueBoolean("notify"));
-         boolean muted     = isMuted(target, now, MuteEffect.SILENCE_NOTIFICATIONS);
-         boolean flapping  = Boolean.TRUE.equals(alert.getValueBoolean("flapping")) && eventType != EventType.FLAPPING_STARTED;
-         if(muted && !Boolean.TRUE.equals(alert.getValueBoolean("muted")))
-         {
-            AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("muted", true));
-            runtime.getEngine().event(alert.getValueInteger("id"), EventType.MUTED, null, null, null, "A mute silenced notifications", now);
-         }
-
-         Map<String, Optional<Boolean>> visibleByUser = new HashMap<>();
-         int                  queued        = 0;
-         for(Destination destination : destinations)
-         {
-            SuppressedReason reason = null;
-            String           channelType = destination.channel().getValueString("channelType");
-            boolean          inApp       = ChannelType.IN_APP.name().equals(channelType);
-
-            if(severity == AlertSeverity.INFO && !inApp && !AlertSeverity.INFO.name().equals(destination.minSeverity()))
-            {
-               continue;
-            }
-            if(!inApp && "alerts.channel.failing".equals(alert.getValueString("alertTypeName")))
-            {
-               ///////////////////////////////////////////////////////////////
-               // a broken channel is only ever reported in-app (no loops) //
-               ///////////////////////////////////////////////////////////////
-               continue;
-            }
-            AlertSeverity channelMin = AlertSeverity.fromLabel(destination.channel().getValueString("minSeverity"), null);
-            if(channelMin != null && !severity.isAtLeast(channelMin) && !inApp)
-            {
-               continue;
-            }
-
-            if(silent)
-            {
-               reason = SuppressedReason.TYPE_SILENT;
-            }
-            else if(muted)
-            {
-               reason = SuppressedReason.MUTED;
-            }
-            else if(flapping)
-            {
-               reason = SuppressedReason.FLAPPING;
-            }
-            else if(destination.userId() != null && !visibleByUser.computeIfAbsent(destination.userId(), userId -> knownVisibility(userId, alert))
-               .orElse(inApp || (alert.getValue("visibilityPermission") == null && alert.getValue("subjectTableName") == null)))
-            {
-               reason = SuppressedReason.NOT_VISIBLE;
-            }
-            else if(destination.userId() != null && !inApp && isPaused(destination.preference(), now))
-            {
-               reason = SuppressedReason.USER_PAUSED;
-            }
-            else if(Boolean.FALSE.equals(destination.channel().getValueBoolean("enabled")))
-            {
-               reason = SuppressedReason.CHANNEL_DISABLED;
-            }
-            else if(ChannelType.EMAIL.name().equals(channelType) && !StringUtils.hasContent(emailAddress(destination)))
-            {
-               reason = SuppressedReason.NO_ADDRESS;
-            }
-            else if(!inApp && isRateLimited(destination.channel(), now))
-            {
-               reason = SuppressedReason.RATE_LIMITED;
-            }
-            else if(isDuplicate(alert, destination, kind, now))
-            {
-               reason = SuppressedReason.DUPLICATE;
-            }
-
-            QRecord notification = new QRecord()
-               .withValue("alertId", alert.getValue("id"))
-               .withValue("alertEventId", eventId)
-               .withValue("subscriptionId", destination.subscriptionId())
-               .withValue("channelId", destination.channel().getValue("id"))
-               .withValue("recipientUserId", destination.userId())
-               .withValue("recipientLabel", recipientLabel(destination))
-               .withValue("kind", kind.name())
-               .withValue("attempts", 0);
-
-            if(reason != null)
-            {
-               notification.setValue("status", DeliveryStatus.SUPPRESSED.name());
-               notification.setValue("suppressedReason", reason.name());
-               AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
-            }
-            else if(inApp)
-            {
-               notification.setValue("status", DeliveryStatus.SENT.name());
-               notification.setValue("sentAt", now);
-               QRecord inserted = AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
-               runtime.getNotifications().deliverAlert(destination.userId(), alert, now);
-               LOG.debug("In-app alert delivered", logPair("deliveryId", inserted.getValue("id")));
-               queued++;
-            }
-            else
-            {
-               notification.setValue("status", DeliveryStatus.PENDING.name());
-               notification.setValue("nextAttemptAt", now);
-               AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
-               queued++;
-            }
-         }
-
-         if(queued > 0)
-         {
-            AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("lastNotifiedAt", now));
-            runtime.getEngine().event(alert.getValueInteger("id"), EventType.NOTIFIED, null, null, String.valueOf(queued),
-               queued == 1 ? "1 notification" : queued + " notifications", now);
-         }
+         deliver(alert, eventType, eventId, kind, destinations(AlertTarget.of(alert),
+            AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING), kind));
       }
       catch(Exception e)
       {
@@ -274,6 +158,211 @@ public class AlertRouter
 
 
 
+   /*******************************************************************************
+    ** A due reminder uses subscription event filters and the normal delivery checks.
+    *******************************************************************************/
+   public int remind(QRecord alert) throws QException
+   {
+      return (deliver(alert, null, null, DeliveryKind.REMINDER, destinations(AlertTarget.of(alert),
+         AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING), DeliveryKind.REMINDER)));
+   }
+
+
+
+   /*******************************************************************************
+    ** A policy target uses the same channel, permission and suppression checks.
+    ** Team members are read in bounded pages; no membership source is changed.
+    *******************************************************************************/
+   public void escalate(QRecord alert, QRecord step, Integer eventId) throws QException
+   {
+      String targetType = step.getValueString("targetType");
+      if(SubscriberType.CHANNEL.name().equals(targetType))
+      {
+         QRecord channel = AlertsDb.byId(AlertsChannel.TABLE_NAME, step.getValue("channelId"));
+         if(channel != null)
+         {
+            deliver(alert, EventType.ESCALATED, eventId, DeliveryKind.ESCALATED,
+               List.of(new Destination(null, channel, null, null, null)));
+         }
+      }
+      else if(SubscriberType.USER.name().equals(targetType))
+      {
+         escalateToUser(alert, step.getValueString("userId"), eventId);
+      }
+      else if(SubscriberType.TEAM.name().equals(targetType))
+      {
+         int lastId = 0;
+         while(true)
+         {
+            List<QRecord> members = AlertsDb.query(AlertsTeamMember.TABLE_NAME, AlertsDb.eq("teamId", step.getValue("teamId"))
+               .withCriteria("id", QCriteriaOperator.GREATER_THAN, lastId)
+               .withOrderBy(new com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy("id", true)).withLimit(100));
+            if(members.isEmpty())
+            {
+               break;
+            }
+            for(QRecord member : members)
+            {
+               escalateToUser(alert, member.getValueString("userId"), eventId);
+               lastId = member.getValueInteger("id");
+            }
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** User and team targets use the user's normal destinations.
+    *******************************************************************************/
+   private void escalateToUser(QRecord alert, String userId, Integer eventId) throws QException
+   {
+      if(!StringUtils.hasContent(userId))
+      {
+         return;
+      }
+      QRecord preference = AlertsDb.first(AlertsUserPreference.TABLE_NAME, AlertsDb.eq("userId", userId));
+      for(QRecord channel : userChannels(null, preference))
+      {
+         deliver(alert, EventType.ESCALATED, eventId, DeliveryKind.ESCALATED,
+            List.of(new Destination(userId, channel, null, null, preference)));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Shared fan-out. Scheduled callers propagate failures so their transaction rolls back.
+    *******************************************************************************/
+   private int deliver(QRecord alert, EventType eventType, Integer eventId, DeliveryKind kind, List<Destination> destinations) throws QException
+   {
+      Instant       now      = runtime.now();
+      AlertTarget   target   = AlertTarget.of(alert);
+      QRecord       type     = AlertsDb.byId(AlertsType.TABLE_NAME, alert.getValue("alertTypeId"));
+      AlertSeverity severity = AlertSeverity.fromLabel(alert.getValueString("severity"), AlertSeverity.WARNING);
+
+      if(destinations.isEmpty())
+      {
+         return (0);
+      }
+
+      boolean silent    = type != null && Boolean.FALSE.equals(type.getValueBoolean("notify"));
+      boolean muted     = isMuted(target, now, MuteEffect.SILENCE_NOTIFICATIONS);
+      boolean flapping  = Boolean.TRUE.equals(alert.getValueBoolean("flapping")) && eventType != EventType.FLAPPING_STARTED;
+      if(muted && !Boolean.TRUE.equals(alert.getValueBoolean("muted")))
+      {
+         AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("muted", true));
+         runtime.getEngine().event(alert.getValueInteger("id"), EventType.MUTED, null, null, null, "A mute silenced notifications", now);
+      }
+
+      Map<String, Optional<Boolean>> visibleByUser = new HashMap<>();
+      int                  queued        = 0;
+      for(Destination destination : destinations)
+      {
+         SuppressedReason reason = null;
+         String           channelType = destination.channel().getValueString("channelType");
+         boolean          inApp       = ChannelType.IN_APP.name().equals(channelType);
+
+         if(severity == AlertSeverity.INFO && !inApp && !AlertSeverity.INFO.name().equals(destination.minSeverity()))
+         {
+            continue;
+         }
+         if(!inApp && "alerts.channel.failing".equals(alert.getValueString("alertTypeName")))
+         {
+            ///////////////////////////////////////////////////////////////
+            // a broken channel is only ever reported in-app (no loops) //
+            ///////////////////////////////////////////////////////////////
+            continue;
+         }
+         AlertSeverity channelMin = AlertSeverity.fromLabel(destination.channel().getValueString("minSeverity"), null);
+         if(channelMin != null && !severity.isAtLeast(channelMin) && !inApp)
+         {
+            continue;
+         }
+
+         if(silent)
+         {
+            reason = SuppressedReason.TYPE_SILENT;
+         }
+         else if(muted)
+         {
+            reason = SuppressedReason.MUTED;
+         }
+         else if(flapping)
+         {
+            reason = SuppressedReason.FLAPPING;
+         }
+         else if(destination.userId() != null && !visibleByUser.computeIfAbsent(destination.userId(), userId -> knownVisibility(userId, alert))
+            .orElse(inApp || (alert.getValue("visibilityPermission") == null && alert.getValue("subjectTableName") == null)))
+         {
+            reason = SuppressedReason.NOT_VISIBLE;
+         }
+         else if(destination.userId() != null && !inApp && isPaused(destination.preference(), now))
+         {
+            reason = SuppressedReason.USER_PAUSED;
+         }
+         else if(Boolean.FALSE.equals(destination.channel().getValueBoolean("enabled")))
+         {
+            reason = SuppressedReason.CHANNEL_DISABLED;
+         }
+         else if(ChannelType.EMAIL.name().equals(channelType) && !StringUtils.hasContent(emailAddress(destination)))
+         {
+            reason = SuppressedReason.NO_ADDRESS;
+         }
+         else if(!inApp && isRateLimited(destination.channel(), now))
+         {
+            reason = SuppressedReason.RATE_LIMITED;
+         }
+         else if(kind != DeliveryKind.REMINDER && isDuplicate(alert, destination, kind, eventId, now))
+         {
+            reason = SuppressedReason.DUPLICATE;
+         }
+
+         QRecord notification = new QRecord()
+            .withValue("alertId", alert.getValue("id"))
+            .withValue("alertEventId", eventId)
+            .withValue("subscriptionId", destination.subscriptionId())
+            .withValue("channelId", destination.channel().getValue("id"))
+            .withValue("recipientUserId", destination.userId())
+            .withValue("recipientLabel", recipientLabel(destination))
+            .withValue("kind", kind.name())
+            .withValue("attempts", 0);
+
+         if(reason != null)
+         {
+            notification.setValue("status", DeliveryStatus.SUPPRESSED.name());
+            notification.setValue("suppressedReason", reason.name());
+            AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
+         }
+         else if(inApp)
+         {
+            notification.setValue("status", DeliveryStatus.SENT.name());
+            notification.setValue("sentAt", now);
+            QRecord inserted = AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
+            runtime.getNotifications().deliverAlert(destination.userId(), alert, now);
+            LOG.debug("In-app alert delivered", logPair("deliveryId", inserted.getValue("id")));
+            queued++;
+         }
+         else
+         {
+            notification.setValue("status", DeliveryStatus.PENDING.name());
+            notification.setValue("nextAttemptAt", now);
+            AlertsDb.insert(AlertsDelivery.TABLE_NAME, notification);
+            queued++;
+         }
+      }
+
+      if(queued > 0)
+      {
+         AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("lastNotifiedAt", now));
+         runtime.getEngine().event(alert.getValueInteger("id"), EventType.NOTIFIED, null, null, String.valueOf(queued),
+            queued == 1 ? "1 notification" : queued + " notifications", now);
+      }
+      return (queued);
+   }
+
+
+
    /*******************************************************************************
     ** Whether an active SUPPRESS_ALERTS maintenance window covers the target.
     *******************************************************************************/
@@ -297,7 +386,7 @@ public class AlertRouter
       Map<Integer, List<String>> groupNames = new HashMap<>();
       for(QRecord mute : mutes)
       {
-         if(scopeMatches(mute.getValueString("scopeType"), mute.getValueString("scopeValue"), null, target, groupNames))
+         if(scopeMatches(mute.getValueString("scopeType"), mute.getValueString("scopeValue"), mute.getValueString("scopeRecordId"), target, groupNames))
          {
             return (true);
          }
@@ -338,13 +427,10 @@ public class AlertRouter
             continue;
          }
 
-         List<String> userIds = new ArrayList<>();
+         Set<String> userIds = new LinkedHashSet<>();
          if(SubscriberType.TEAM.name().equals(subscriberType))
          {
-            for(QRecord member : AlertsDb.query(AlertsTeamMember.TABLE_NAME, AlertsDb.eq("teamId", subscription.getValue("teamId"))))
-            {
-               userIds.add(member.getValueString("userId"));
-            }
+            userIds.addAll(teamMembers(subscription.getValueInteger("teamId")));
          }
          else if(StringUtils.hasContent(subscription.getValueString("userId")))
          {
@@ -353,6 +439,10 @@ public class AlertRouter
 
          for(String userId : userIds)
          {
+            if(!StringUtils.hasContent(userId))
+            {
+               continue;
+            }
             QRecord preference = AlertsDb.first(AlertsUserPreference.TABLE_NAME, AlertsDb.eq("userId", userId));
             for(QRecord channel : userChannels(subscription, preference))
             {
@@ -360,7 +450,7 @@ public class AlertRouter
             }
          }
       }
-      for(String recordKey : target.recordKeys())
+      for(String recordKey : kind == DeliveryKind.REMINDER ? java.util.Set.<String>of() : target.recordKeys())
       {
          String[] parts = recordKey.split(":", 2);
          RecordReference reference = new RecordReference(parts[0], parts[1]);
@@ -383,6 +473,61 @@ public class AlertRouter
 
 
 
+   /*******************************************************************************
+    ** Explicit members plus the team's optional host-managed source. Missing
+    ** or failed resolvers contribute nobody and leave an actionable log entry.
+    *******************************************************************************/
+   private Set<String> teamMembers(Integer teamId) throws QException
+   {
+      Set<String> userIds = new LinkedHashSet<>();
+      QRecord team = AlertsDb.byId(AlertsTeam.TABLE_NAME, teamId);
+      if(team == null)
+      {
+         return (userIds);
+      }
+      for(QRecord member : AlertsDb.query(AlertsTeamMember.TABLE_NAME, AlertsDb.eq("teamId", teamId).withLimit(Math.addExact(runtime.getConfig().getMaxAudienceSize(), 1))))
+      {
+         if(userIds.size() >= runtime.getConfig().getMaxAudienceSize())
+         {
+            LOG.warn("Team audience exceeds maxAudienceSize; extra members are not notified", logPair("teamId", teamId));
+            break;
+         }
+         userIds.add(member.getValueString("userId"));
+      }
+      String source = team.getValueString("memberSource");
+      if(MemberSource.ROLE.name().equals(source) || MemberSource.IDP_GROUP.name().equals(source))
+      {
+         AlertAudienceResolver resolver = runtime.getConfig().getAudienceResolver();
+         if(resolver == null)
+         {
+            LOG.warn("Team source resolves to nobody: no host audience resolver is configured", logPair("teamId", teamId), logPair("source", source));
+         }
+         else
+         {
+            try
+            {
+               Set<String> resolved = resolver.resolve(source, team.getValueString("memberSourceValue"));
+               if(resolved != null)
+               {
+                  if(resolved.size() > runtime.getConfig().getMaxAudienceSize()
+                     || resolved.stream().filter(userId -> !userIds.contains(userId)).count() > runtime.getConfig().getMaxAudienceSize() - userIds.size())
+                  {
+                     throw (new QException("Team audience exceeds maxAudienceSize"));
+                  }
+                  userIds.addAll(resolved);
+               }
+            }
+            catch(Exception e)
+            {
+               LOG.warn("Team source resolves to nobody: host audience resolver failed", logPair("teamId", teamId), logPair("source", source));
+            }
+         }
+      }
+      return (userIds);
+   }
+
+
+
    /*******************************************************************************
     ** A user's channels for a subscription: the ones it names, else in-app
     ** (unless turned off) and email (when enabled in their settings).
@@ -429,6 +574,7 @@ public class AlertRouter
       return switch(type)
       {
          case ALL -> true;
+         case ALERT -> scopeValue != null && scopeValue.equals(target.alertKey());
          case ALERT_TYPE -> scopeValue != null && scopeValue.equals(target.typeName());
          case MONITOR -> scopeValue != null && target.monitorId() != null && scopeValue.equals(String.valueOf(target.monitorId()));
          case TABLE -> scopeValue != null && (scopeValue.equals(target.subjectTableName()) || target.recordKeys().stream().anyMatch(key -> key.startsWith(scopeValue + ":")));
@@ -622,13 +768,20 @@ public class AlertRouter
    /*******************************************************************************
     ** The same alert, channel, recipient and kind within 10 minutes.
     *******************************************************************************/
-   private boolean isDuplicate(QRecord alert, Destination destination, DeliveryKind kind, Instant now) throws QException
+   private boolean isDuplicate(QRecord alert, Destination destination, DeliveryKind kind, Integer eventId, Instant now) throws QException
    {
       var filter = AlertsDb.eq("alertId", alert.getValue("id"))
          .withCriteria("channelId", QCriteriaOperator.EQUALS, destination.channel().getValue("id"))
          .withCriteria("kind", QCriteriaOperator.EQUALS, kind.name())
          .withCriteria("status", QCriteriaOperator.IN, List.of(DeliveryStatus.PENDING.name(), DeliveryStatus.SENDING.name(), DeliveryStatus.SENT.name(), DeliveryStatus.RETRYING.name()))
          .withCriteria("createDate", QCriteriaOperator.GREATER_THAN, now.minus(DUPLICATE_WINDOW_MINUTES, ChronoUnit.MINUTES));
+      if(kind == DeliveryKind.ESCALATED)
+      {
+         filter = AlertsDb.eq("alertId", alert.getValue("id"))
+            .withCriteria("channelId", QCriteriaOperator.EQUALS, destination.channel().getValue("id"))
+            .withCriteria("alertEventId", QCriteriaOperator.EQUALS, eventId)
+            .withCriteria("status", QCriteriaOperator.NOT_EQUALS, DeliveryStatus.SUPPRESSED.name());
+      }
       if(destination.userId() != null)
       {
          filter.withCriteria("recipientUserId", QCriteriaOperator.EQUALS, destination.userId());

src/main/java/com/kingsrook/qbits/alerts/engine/AlertsDb.java

@@ -23,11 +23,14 @@ import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
 import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.statusmessages.QStatusMessage;
@@ -56,7 +59,7 @@ public final class AlertsDb
     *******************************************************************************/
    public static List<QRecord> query(String tableName, QQueryFilter filter) throws QException
    {
-      return (QueryAction.execute(tableName, filter));
+      return (new QueryAction().execute(new QueryInput(tableName).withFilter(filter).withTransaction(QContext.getQBackendTransaction())).getRecords());
    }
 
 
@@ -66,7 +69,7 @@ public final class AlertsDb
     *******************************************************************************/
    public static QRecord first(String tableName, QQueryFilter filter) throws QException
    {
-      List<QRecord> records = QueryAction.execute(tableName, filter.withLimit(1));
+      List<QRecord> records = query(tableName, filter.withLimit(1));
       return (CollectionUtils.nullSafeIsEmpty(records) ? null : records.get(0));
    }
 
@@ -91,7 +94,7 @@ public final class AlertsDb
     *******************************************************************************/
    public static int count(String tableName, QQueryFilter filter) throws QException
    {
-      Integer count = CountAction.execute(tableName, filter);
+      Integer count = new CountAction().execute(new CountInput(tableName).withFilter(filter).withTransaction(QContext.getQBackendTransaction())).getCount();
       return (count == null ? 0 : count);
    }
 
@@ -102,7 +105,7 @@ public final class AlertsDb
     *******************************************************************************/
    public static QRecord insert(String tableName, QRecord record) throws QException
    {
-      List<QRecord> inserted = InsertAction.executeForRecords(new InsertInput(tableName).withRecord(record));
+      List<QRecord> inserted = InsertAction.executeForRecords(new InsertInput(tableName).withRecord(record).withTransaction(QContext.getQBackendTransaction()));
       QRecord       result   = inserted.get(0);
       throwIfErrors(tableName, result);
       return (result);
@@ -115,7 +118,7 @@ public final class AlertsDb
     *******************************************************************************/
    public static QRecord update(String tableName, QRecord record) throws QException
    {
-      List<QRecord> updated = UpdateAction.executeForRecords(new UpdateInput(tableName).withRecord(record));
+      List<QRecord> updated = UpdateAction.executeForRecords(new UpdateInput(tableName).withRecord(record).withTransaction(QContext.getQBackendTransaction()));
       QRecord       result  = updated.get(0);
       throwIfErrors(tableName, result);
       return (result);

src/main/java/com/kingsrook/qbits/alerts/engine/AlertsEngine.java

@@ -568,6 +568,8 @@ public class AlertsEngine
          .withValue("severity", severity.name())
          .withValue("lastRaisedAt", now)
          .withValue("occurrenceCount", (resolved.getValueInteger("occurrenceCount") == null ? 0 : resolved.getValueInteger("occurrenceCount")) + 1)
+         .withValue("escalationLevel", 0)
+         .withValue("timedNotificationCheckedAt", null)
          .withValue("reopenCount", (resolved.getValueInteger("reopenCount") == null ? 0 : resolved.getValueInteger("reopenCount")) + 1)
          .withValue("resolvedAt", null)
          .withValue("resolvedByUserId", null)
@@ -824,7 +826,7 @@ public class AlertsEngine
    /*******************************************************************************
     ** What a subscription or mute scope is matched against.
     *******************************************************************************/
-   public record AlertTarget(Integer typeId, String typeName, Integer groupId, Integer monitorId, String subjectTableName, Set<String> recordKeys, Map<String, String> attributes)
+   public record AlertTarget(Integer typeId, String typeName, Integer groupId, Integer monitorId, String subjectTableName, Set<String> recordKeys, Map<String, String> attributes, String alertKey)
    {
       /***************************************************************************
        ** From a type row and a signal (before the alert exists).
@@ -837,7 +839,7 @@ public class AlertsEngine
             keys.add(ref.getTableName() + ":" + ref.getPrimaryKey());
          }
          return (new AlertTarget(type.getValueInteger("id"), signal.getAlertTypeName(), type.getValueInteger("groupId"), monitorId,
-            type.getValueString("subjectTableName"), keys, signal.getAttributes()));
+            type.getValueString("subjectTableName"), keys, signal.getAttributes(), openKey(signal.getAlertTypeName(), signal.getEffectiveDedupKey())));
       }
 
 
@@ -852,6 +854,10 @@ public class AlertsEngine
          {
             keys.add(ref.getValueString("tableName") + ":" + ref.getValueString("recordId"));
          }
+         if(StringUtils.hasContent(alert.getValueString("subjectTableName")) && StringUtils.hasContent(alert.getValueString("subjectRecordId")))
+         {
+            keys.add(alert.getValueString("subjectTableName") + ":" + alert.getValueString("subjectRecordId"));
+         }
          Map<String, String> attributes = new HashMap<>();
          String              json       = alert.getValueString("attributes");
          if(StringUtils.hasContent(json))
@@ -867,7 +873,7 @@ public class AlertsEngine
             }
          }
          return (new AlertTarget(alert.getValueInteger("alertTypeId"), alert.getValueString("alertTypeName"), alert.getValueInteger("groupId"), alert.getValueInteger("monitorId"),
-            alert.getValueString("subjectTableName"), keys, attributes));
+            alert.getValueString("subjectTableName"), keys, attributes, openKey(alert.getValueString("alertTypeName"), alert.getValueString("dedupKey"))));
       }
    }
 }

src/main/java/com/kingsrook/qbits/alerts/engine/EscalationAndReminder.java

@@ -0,0 +1,259 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.alerts.engine;
+
+
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
+import java.util.List;
+import com.kingsrook.qbits.alerts.AlertsRuntime;
+import com.kingsrook.qbits.alerts.model.AlertsAlert;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationPolicy;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationStep;
+import com.kingsrook.qbits.alerts.model.AlertsEvent;
+import com.kingsrook.qbits.alerts.model.AlertsGroup;
+import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.EventType;
+import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.MuteEffect;
+import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.Status;
+import com.kingsrook.qbits.alerts.model.AlertsType;
+import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+
+
+/*******************************************************************************
+ ** Timed notifications for open, unacknowledged alerts. Progress and fan-out
+ ** commit together per alert. Run on one scheduler node, like the other jobs.
+ *******************************************************************************/
+public class EscalationAndReminder
+{
+   private static final int PAGE_SIZE = 100;
+
+   private final AlertsRuntime runtime;
+
+
+
+   /*******************************************************************************
+    ** Use the installed runtime, including its controllable clock.
+    *******************************************************************************/
+   public EscalationAndReminder(AlertsRuntime runtime)
+   {
+      this.runtime = runtime;
+   }
+
+
+
+   /*******************************************************************************
+    ** Read bounded pages. A persisted checkpoint makes repeated ticks harmless,
+    ** including attempts with no eligible recipients. Serialize this runtime's ticks.
+    *******************************************************************************/
+   public int runDue() throws QException
+   {
+      synchronized(runtime)
+      {
+         return (runtime.asSystem(() ->
+         {
+            int changed = 0;
+            int lastId = 0;
+            while(true)
+            {
+               List<QRecord> alerts = AlertsDb.query(AlertsAlert.TABLE_NAME, AlertsDb.eq("status", Status.OPEN.name())
+                  .withCriteria("id", QCriteriaOperator.GREATER_THAN, lastId)
+                  .withOrderBy(new QFilterOrderBy("id", true)).withLimit(PAGE_SIZE));
+               if(alerts.isEmpty())
+               {
+                  break;
+               }
+               for(QRecord alert : alerts)
+               {
+                  lastId = alert.getValueInteger("id");
+                  changed += process(lastId);
+               }
+            }
+            return (changed);
+         }));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** One alert is an atomic unit, including in-app recipients and delivery rows.
+    *******************************************************************************/
+   private int process(Integer alertId) throws QException
+   {
+      QBackendTransaction previous = QContext.getQBackendTransaction();
+      try(QBackendTransaction transaction = QBackendTransaction.openFor(new QueryInput(AlertsAlert.TABLE_NAME)))
+      {
+         QContext.setTransaction(transaction);
+         try
+         {
+            int changed = apply(AlertsDb.byId(AlertsAlert.TABLE_NAME, alertId), runtime.now());
+            transaction.commit();
+            return (changed);
+         }
+         catch(Exception e)
+         {
+            transaction.rollback();
+            throw (new QException("Could not apply timed notifications for alert " + alertId, e));
+         }
+      }
+      finally
+      {
+         QContext.setTransaction(previous);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Acknowledge, snooze, resolve, either mute effect, and disabled types stop work.
+    *******************************************************************************/
+   private int apply(QRecord alert, Instant now) throws QException
+   {
+      if(alert == null || !Status.OPEN.name().equals(alert.getValueString("status")) || alert.getValue("acknowledgedAt") != null)
+      {
+         return (0);
+      }
+      Instant checked = alert.getValueInstant("timedNotificationCheckedAt");
+      if(checked != null && !checked.truncatedTo(ChronoUnit.MINUTES).isBefore(now.truncatedTo(ChronoUnit.MINUTES)))
+      {
+         return (0);
+      }
+      QRecord type = AlertsDb.byId(AlertsType.TABLE_NAME, alert.getValue("alertTypeId"));
+      if(type == null || Boolean.FALSE.equals(type.getValueBoolean("enabled")) || Boolean.FALSE.equals(type.getValueBoolean("notify")))
+      {
+         return (0);
+      }
+      AlertsEngine.AlertTarget target = AlertsEngine.AlertTarget.of(alert);
+      if(Boolean.TRUE.equals(alert.getValueBoolean("muted")) || Boolean.TRUE.equals(alert.getValueBoolean("flapping"))
+         || runtime.getRouter().isMuted(target, now, MuteEffect.SILENCE_NOTIFICATIONS)
+         || runtime.getRouter().isSuppressedByMaintenance(target, now))
+      {
+         return (0);
+      }
+
+      int changed = escalate(alert, type, now) ? 1 : 0;
+      QRecord fresh = AlertsDb.byId(AlertsAlert.TABLE_NAME, alert.getValue("id"));
+      Integer interval = type.getValueInteger("reminderIntervalMinutes");
+      if(interval == null && "CRITICAL".equals(fresh.getValueString("severity")))
+      {
+         interval = 60;
+      }
+      Instant raised = latest(fresh.getValueInstant("lastRaisedAt"), fresh.getValueInstant("firstRaisedAt"));
+      Instant since = latest(fresh.getValueInstant("lastNotifiedAt"), raised);
+      if(interval != null && interval > 0 && since != null && !since.plus(interval, ChronoUnit.MINUTES).isAfter(now))
+      {
+         if(runtime.getRouter().remind(fresh) > 0)
+         {
+            changed = 1;
+         }
+      }
+      AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("timedNotificationCheckedAt", now));
+      return (changed);
+   }
+
+
+
+   /*******************************************************************************
+    ** Type policy wins; otherwise use the alert's group policy. Steps are ordered
+    ** by stepOrder and measured from the most recent reopen (else first raise).
+    ** At most one step is taken per minute; missed steps are not burst-delivered.
+    *******************************************************************************/
+   private boolean escalate(QRecord alert, QRecord type, Instant now) throws QException
+   {
+      Integer policyId = type.getValueInteger("escalationPolicyId");
+      if(policyId == null)
+      {
+         QRecord group = AlertsDb.byId(AlertsGroup.TABLE_NAME, alert.getValue("groupId"));
+         policyId = group == null ? null : group.getValueInteger("escalationPolicyId");
+      }
+      QRecord policy = AlertsDb.byId(AlertsEscalationPolicy.TABLE_NAME, policyId);
+      if(policy == null || !Boolean.TRUE.equals(policy.getValueBoolean("enabled")))
+      {
+         return (false);
+      }
+      QRecord reopened = AlertsDb.first(AlertsEvent.TABLE_NAME, AlertsDb.eq("alertId", alert.getValue("id"))
+         .withCriteria("eventType", QCriteriaOperator.EQUALS, EventType.REOPENED.name()).withOrderBy(new QFilterOrderBy("id", false)));
+      Instant start = reopened == null ? alert.getValueInstant("firstRaisedAt") : reopened.getValueInstant("at");
+      int level = alert.getValueInteger("escalationLevel") == null ? 0 : alert.getValueInteger("escalationLevel");
+      QRecord step = AlertsDb.first(AlertsEscalationStep.TABLE_NAME, AlertsDb.eq("policyId", policyId)
+         .withCriteria("stepOrder", QCriteriaOperator.GREATER_THAN, level).withOrderBy(new QFilterOrderBy("stepOrder", true)));
+      if(step == null)
+      {
+         step = repeatStep(alert, policy, reopened, level, now);
+      }
+      if(step == null || start == null || step.getValueInteger("afterMinutes") == null || step.getValueInteger("afterMinutes") < 0
+         || !Status.OPEN.name().equals(step.getValueString("onlyIfStatus"))
+         || start.plus(step.getValueInteger("afterMinutes"), ChronoUnit.MINUTES).isAfter(now))
+      {
+         return (false);
+      }
+      int nextLevel = step.getValueInteger("stepOrder");
+      Integer eventId = runtime.getEngine().event(alert.getValueInteger("id"), EventType.ESCALATED, null,
+         String.valueOf(level), String.valueOf(nextLevel), level == nextLevel ? "Repeated final escalation step" : "Escalation step " + nextLevel, now);
+      runtime.getRouter().escalate(alert, step, eventId);
+      AlertsDb.update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue("escalationLevel", nextLevel));
+      return (true);
+   }
+
+
+
+   /*******************************************************************************
+    ** Repeat only the final step, capped by the number of same-level events in
+    ** this open cycle. Repeat spacing uses escalation time, never reminder time.
+    *******************************************************************************/
+   private QRecord repeatStep(QRecord alert, QRecord policy, QRecord reopened, int level, Instant now) throws QException
+   {
+      Integer interval = policy.getValueInteger("repeatAfterMinutes");
+      Integer maxRepeats = policy.getValueInteger("maxRepeats");
+      if(level == 0 || interval == null || interval <= 0 || maxRepeats == null || maxRepeats <= 0)
+      {
+         return (null);
+      }
+      QQueryFilter cycle = AlertsDb.eq("alertId", alert.getValue("id"))
+         .withCriteria("eventType", QCriteriaOperator.EQUALS, EventType.ESCALATED.name())
+         .withCriteria("id", QCriteriaOperator.GREATER_THAN, reopened == null ? 0 : reopened.getValueInteger("id"));
+      QRecord last = AlertsDb.first(AlertsEvent.TABLE_NAME, cycle.withOrderBy(new QFilterOrderBy("id", false)));
+      if(last == null || last.getValueInstant("at").plus(interval, ChronoUnit.MINUTES).isAfter(now))
+      {
+         return (null);
+      }
+      int repeats = AlertsDb.count(AlertsEvent.TABLE_NAME, AlertsDb.eq("alertId", alert.getValue("id"))
+         .withCriteria("eventType", QCriteriaOperator.EQUALS, EventType.ESCALATED.name())
+         .withCriteria("id", QCriteriaOperator.GREATER_THAN, reopened == null ? 0 : reopened.getValueInteger("id"))
+         .withCriteria("fromValue", QCriteriaOperator.EQUALS, String.valueOf(level))
+         .withCriteria("toValue", QCriteriaOperator.EQUALS, String.valueOf(level)));
+      return (repeats >= maxRepeats ? null : AlertsDb.first(AlertsEscalationStep.TABLE_NAME, AlertsDb.eq("policyId", policy.getValue("id"))
+         .withCriteria("stepOrder", QCriteriaOperator.EQUALS, level)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Later of two optional timestamps.
+    *******************************************************************************/
+   private static Instant latest(Instant left, Instant right)
+   {
+      return (left == null ? right : right == null || left.isAfter(right) ? left : right);
+   }
+}

src/main/java/com/kingsrook/qbits/alerts/engine/MonitorEvaluator.java

@@ -216,7 +216,7 @@ public class MonitorEvaluator
          long    started = System.currentTimeMillis();
          String  typeName = type.getValueString("name");
 
-         AlertTarget target = new AlertTarget(type.getValueInteger("id"), typeName, type.getValueInteger("groupId"), monitorId, type.getValueString("subjectTableName"), Set.of(), Map.of());
+         AlertTarget target = new AlertTarget(type.getValueInteger("id"), typeName, type.getValueInteger("groupId"), monitorId, type.getValueString("subjectTableName"), Set.of(), Map.of(), null);
          if(runtime.getRouter().isSuppressedByMaintenance(target, now))
          {
             Plan plan = new Plan(MonitorStatus.SKIPPED_MUTED, "skipped: maintenance window", List.of(), List.of(), List.of(), List.of(), null);

src/main/java/com/kingsrook/qbits/alerts/metadata/AlertsAppMetaDataProducer.java

@@ -24,6 +24,7 @@ import com.kingsrook.qbits.alerts.AlertsQBitConfig;
 import com.kingsrook.qbits.alerts.model.AlertsAlert;
 import com.kingsrook.qbits.alerts.model.AlertsChannel;
 import com.kingsrook.qbits.alerts.model.AlertsDelivery;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationPolicy;
 import com.kingsrook.qbits.alerts.model.AlertsGroup;
 import com.kingsrook.qbits.alerts.model.AlertsMonitor;
 import com.kingsrook.qbits.alerts.model.AlertsMute;
@@ -55,7 +56,7 @@ public class AlertsAppMetaDataProducer extends QBitComponentMetaDataProducer<QAp
 {
    /** the main tables, in nav order. */
    public static final List<String> MAIN_TABLES = List.of(AlertsAlert.TABLE_NAME, AlertsNotification.TABLE_NAME, AlertsMonitor.TABLE_NAME, AlertsType.TABLE_NAME, AlertsGroup.TABLE_NAME,
-      AlertsSubscription.TABLE_NAME, AlertsUserPreference.TABLE_NAME, AlertsNotificationPreference.TABLE_NAME, AlertsTeam.TABLE_NAME, AlertsChannel.TABLE_NAME, AlertsMute.TABLE_NAME, AlertsDelivery.TABLE_NAME);
+      AlertsSubscription.TABLE_NAME, AlertsUserPreference.TABLE_NAME, AlertsNotificationPreference.TABLE_NAME, AlertsTeam.TABLE_NAME, AlertsChannel.TABLE_NAME, AlertsEscalationPolicy.TABLE_NAME, AlertsMute.TABLE_NAME, AlertsDelivery.TABLE_NAME);
 
 
 

src/main/java/com/kingsrook/qbits/alerts/metadata/AlertsProcessesMetaDataProducer.java

@@ -46,13 +46,15 @@ import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwith
  ** - the record and table actions (AlertsAction), each a standard streamed
  **   ETL process on its table: optional input, Review summary lines, Result
  **   summary lines;
- ** - two hidden scheduled processes: alertsEvaluateMonitors (monitors, plus
- **   snooze / quiet-period / flapping upkeep) and alertsDeliverNotifications.
+ ** - hidden scheduled processes: alertsEvaluateMonitors (monitors, plus
+ **   snooze / quiet-period / flapping upkeep), alertsDeliverNotifications,
+ **   and alertsEscalateAndRemind.
  **   They exist without a scheduler and run on one only when the config names
  **   it.
  *******************************************************************************/
 public class AlertsProcessesMetaDataProducer extends QBitComponentMetaDataProducer<MetaDataProducerMultiOutput, AlertsQBitConfig>
 {
+   public static final String ESCALATE_AND_REMIND = "alertsEscalateAndRemind";
    public static final String EVALUATE_MONITORS     = "alertsEvaluateMonitors";
    public static final String DELIVER_NOTIFICATIONS = "alertsDeliverNotifications";
 
@@ -87,6 +89,7 @@ public class AlertsProcessesMetaDataProducer extends QBitComponentMetaDataProduc
       }
       processes.add(scheduled(config, EVALUATE_MONITORS, "Evaluate alert monitors", AlertsScheduledSteps.EvaluateMonitors.class, config.getEvaluateEverySeconds()));
       processes.add(scheduled(config, DELIVER_NOTIFICATIONS, "Deliver alert notifications", AlertsScheduledSteps.DeliverNotifications.class, config.getDeliverEverySeconds()));
+      processes.add(scheduled(config, ESCALATE_AND_REMIND, "Escalate and remind", AlertsScheduledSteps.EscalateAndRemind.class, 60));
       return (processes);
    }
 

src/main/java/com/kingsrook/qbits/alerts/metadata/AlertsTableMetaDataHelper.java

@@ -21,6 +21,7 @@ import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
+import com.kingsrook.qbits.alerts.AlertsPermissionChecker;
 import com.kingsrook.qbits.alerts.AlertsQBitConfig;
 import com.kingsrook.qbits.alerts.AlertsSecurity;
 import com.kingsrook.qbits.alerts.customizers.AlertsNotificationCustomizer;
@@ -29,6 +30,8 @@ import com.kingsrook.qbits.alerts.customizers.AlertsWriteGuardCustomizer;
 import com.kingsrook.qbits.alerts.model.AlertsAlert;
 import com.kingsrook.qbits.alerts.model.AlertsChannel;
 import com.kingsrook.qbits.alerts.model.AlertsDelivery;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationPolicy;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationStep;
 import com.kingsrook.qbits.alerts.model.AlertsEvent;
 import com.kingsrook.qbits.alerts.model.AlertsGroup;
 import com.kingsrook.qbits.alerts.model.AlertsHeartbeat;
@@ -62,6 +65,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
@@ -94,6 +98,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
  *******************************************************************************/
 public final class AlertsTableMetaDataHelper
 {
+   public static final String JOIN_POLICY_STEP = "alerts_escalation_policy_join_alerts_escalation_step";
    public static final String JOIN_ALERT_EVENT        = "alerts_alert_join_alerts_event";
    public static final String JOIN_ALERT_RECORD_REF   = "alerts_alert_join_alerts_record_ref";
    public static final String JOIN_ALERT_DELIVERY = "alerts_alert_join_alerts_delivery";
@@ -134,7 +139,7 @@ public final class AlertsTableMetaDataHelper
          .withSection(widgetSection(JOIN_ALERT_RECORD_REF, "Records", "link"))
          .withSection(widgetSection(JOIN_ALERT_DELIVERY, "Notifications", "send"))
          .withSection(new QFieldSection("occurrence", "Occurrences", new QIcon("repeat"), Tier.T3, List.of("firstRaisedAt", "lastRaisedAt", "occurrenceCount", "reopenCount", "flapping",
-            "lastNotifiedAt", "monitorId", "dedupKey", "alertTypeName", "visibilityPermission", "openDedupKey", "escalationLevel", "links", "attributes", "createDate", "modifyDate")));
+            "lastNotifiedAt", "timedNotificationCheckedAt", "monitorId", "dedupKey", "alertTypeName", "visibilityPermission", "openDedupKey", "escalationLevel", "links", "attributes", "createDate", "modifyDate")));
       return (readOnly(table));
    }
 
@@ -153,7 +158,7 @@ public final class AlertsTableMetaDataHelper
          .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "label", "name", "groupId", "sourceName", "openCount")))
          .withSection(new QFieldSection("about", "About", new QIcon("info"), Tier.T2, List.of("description", "howToFix", "subjectTableName", "visibilityPermission")))
          .withSection(new QFieldSection("tuning", "Tuning", new QIcon("tune"), Tier.T2, List.of("enabled", "notify", "defaultSeverity", "severityOverride", "resolveMode",
-            "quietPeriodMinutes", "reminderIntervalMinutes", "externalDetailCeiling")))
+            "quietPeriodMinutes", "reminderIntervalMinutes", "escalationPolicyId", "externalDetailCeiling")))
          .withSection(widgetSection(JOIN_TYPE_MONITOR, "Monitors", "monitor_heart"))
          .withSection(widgetSection(JOIN_TYPE_ALERT, "Alerts", "notifications_active"))
          .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("installed", "lastRaisedAt", "createDate", "modifyDate")));
@@ -174,7 +179,7 @@ public final class AlertsTableMetaDataHelper
          .withUniqueKey(new UniqueKey("name"))
          .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
          .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "label", "name", "parentGroupId")))
-         .withSection(new QFieldSection("details", new QIcon("description"), Tier.T2, List.of("description", "ownerTeamId", "icon", "sortOrder")))
+         .withSection(new QFieldSection("details", new QIcon("description"), Tier.T2, List.of("description", "ownerTeamId", "escalationPolicyId", "icon", "sortOrder")))
          .withSection(widgetSection(JOIN_GROUP_TYPE, "Alert Types", "category"))
          .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
       return (table);
@@ -304,6 +309,10 @@ public final class AlertsTableMetaDataHelper
          .withSection(new QFieldSection("details", new QIcon("description"), Tier.T2, List.of("description")))
          .withSection(widgetSection(JOIN_TEAM_MEMBER, "Members", "person"))
          .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+      if(config.getAudienceResolver() == null)
+      {
+         table.getField("memberSource").withHelpContent(new QHelpContent("ROLE and IdP Group members need a host resolver; none is configured"));
+      }
       table.getField("memberSource").setDefaultValue(AlertsPossibleValues.MemberSource.MANUAL.name());
       return (table);
    }
@@ -322,6 +331,10 @@ public final class AlertsTableMetaDataHelper
          .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
          .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "teamId", "userId", "role")))
          .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+      if(config.getAudienceResolver() == null)
+      {
+         table.getField("teamId").withHelpContent(new QHelpContent("ROLE and IdP Group members need a host resolver; none is configured"));
+      }
       table.getField("role").setDefaultValue(AlertsPossibleValues.TeamRole.MEMBER.name());
       return (table);
    }
@@ -364,7 +377,7 @@ public final class AlertsTableMetaDataHelper
          .withRecordLabelFields("name")
          .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
          .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "name", "enabled", "effect")))
-         .withSection(new QFieldSection("scope", "What and When", new QIcon("filter_alt"), Tier.T2, List.of("scopeType", "scopeValue", "startsAt", "endsAt", "reason")))
+         .withSection(new QFieldSection("scope", "What and When", new QIcon("filter_alt"), Tier.T2, List.of("scopeType", "scopeValue", "scopeRecordId", "startsAt", "endsAt", "reason")))
          .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createdByUserId", "createDate", "modifyDate")));
       table.getField("effect").setDefaultValue(AlertsPossibleValues.MuteEffect.SILENCE_NOTIFICATIONS.name());
       table.getField("enabled").setDefaultValue(true);
@@ -497,12 +510,52 @@ public final class AlertsTableMetaDataHelper
 
 
 
+   /*******************************************************************************
+    ** Escalation policies, with ordered steps on the record screen.
+    *******************************************************************************/
+   public static QTableMetaData buildEscalationPolicyTable(AlertsQBitConfig config) throws QException
+   {
+      QTableMetaData table = text(base(config, AlertsEscalationPolicy.TABLE_NAME, "Escalation Policies", "trending_up", AlertsEscalationPolicy.class), "description")
+         .withPermissionRules(AlertsPermissionChecker.rules(AlertsQBitConfig.PERMISSION_ADMIN))
+         .withRecordLabelFormat("%s").withRecordLabelFields("name")
+         .withUniqueKey(new UniqueKey("name"))
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
+         .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "name", "enabled")))
+         .withSection(new QFieldSection("details", new QIcon("description"), Tier.T2, List.of("description", "repeatAfterMinutes", "maxRepeats")))
+         .withSection(widgetSection(JOIN_POLICY_STEP, "Steps", "trending_up"))
+         .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+      table.getField("enabled").setDefaultValue(true);
+      table.getField("maxRepeats").setDefaultValue(0);
+      return (table);
+   }
+
+
+
+   /*******************************************************************************
+    ** Escalation steps are standard admin tables, reached from their policy.
+    *******************************************************************************/
+   public static QTableMetaData buildEscalationStepTable(AlertsQBitConfig config) throws QException
+   {
+      QTableMetaData table = base(config, AlertsEscalationStep.TABLE_NAME, "Escalation Steps", "trending_up", AlertsEscalationStep.class)
+         .withPermissionRules(AlertsPermissionChecker.rules(AlertsQBitConfig.PERMISSION_ADMIN))
+         .withRecordLabelFormat("Step %s").withRecordLabelFields("stepOrder")
+         .withUniqueKey(new UniqueKey("policyId", "stepOrder"))
+         .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))
+         .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "policyId", "stepOrder", "afterMinutes")))
+         .withSection(new QFieldSection("target", new QIcon("person"), Tier.T2, List.of("targetType", "userId", "teamId", "channelId", "onlyIfStatus")))
+         .withSection(new QFieldSection("system", new QIcon("calendar_month"), Tier.T3, List.of("createDate", "modifyDate")));
+      table.getField("onlyIfStatus").setDefaultValue(AlertsPossibleValues.Status.OPEN.name());
+      return (table);
+   }
+
+
+
    /*******************************************************************************
     ** All the tables.
     *******************************************************************************/
    public static List<QTableMetaData> buildAllTables(AlertsQBitConfig config) throws QException
    {
-      return (List.of(buildAlertTable(config), buildMonitorTable(config), buildTypeTable(config), buildGroupTable(config), buildSubscriptionTable(config),
+      return (List.of(buildEscalationPolicyTable(config), buildEscalationStepTable(config), buildAlertTable(config), buildMonitorTable(config), buildTypeTable(config), buildGroupTable(config), buildSubscriptionTable(config),
          buildUserPreferenceTable(config), buildTeamTable(config), buildTeamMemberTable(config), buildChannelTable(config), buildMuteTable(config),
          buildDeliveryTable(config), buildEventTable(config), buildRecordRefTable(config), buildHeartbeatTable(config), buildNotificationTable(config),
          buildNotificationRecipientTable(config), buildNotificationPreferenceTable(config)));
@@ -516,7 +569,7 @@ public final class AlertsTableMetaDataHelper
    public static List<QPossibleValueSource> buildPossibleValueSources()
    {
       List<QPossibleValueSource> sources = new ArrayList<>(AlertsPossibleValues.buildEnumSources());
-      for(String table : List.of(AlertsAlert.TABLE_NAME, AlertsType.TABLE_NAME, AlertsGroup.TABLE_NAME, AlertsMonitor.TABLE_NAME, AlertsTeam.TABLE_NAME,
+      for(String table : List.of(AlertsEscalationPolicy.TABLE_NAME, AlertsAlert.TABLE_NAME, AlertsType.TABLE_NAME, AlertsGroup.TABLE_NAME, AlertsMonitor.TABLE_NAME, AlertsTeam.TABLE_NAME,
          AlertsChannel.TABLE_NAME, AlertsSubscription.TABLE_NAME, AlertsNotification.TABLE_NAME))
       {
          sources.add(QPossibleValueSource.newForTable(table));
@@ -532,6 +585,7 @@ public final class AlertsTableMetaDataHelper
    public static List<QJoinMetaData> buildJoins()
    {
       List<QJoinMetaData> joins = new ArrayList<>();
+      joins.add(oneToMany(JOIN_POLICY_STEP, AlertsEscalationPolicy.TABLE_NAME, AlertsEscalationStep.TABLE_NAME, "policyId"));
       joins.add(oneToMany(JOIN_ALERT_EVENT, AlertsAlert.TABLE_NAME, AlertsEvent.TABLE_NAME, "alertId"));
       joins.add(oneToMany(JOIN_ALERT_RECORD_REF, AlertsAlert.TABLE_NAME, AlertsRecordRef.TABLE_NAME, "alertId"));
       joins.add(oneToMany(JOIN_ALERT_DELIVERY, AlertsAlert.TABLE_NAME, AlertsDelivery.TABLE_NAME, "alertId"));
@@ -560,6 +614,9 @@ public final class AlertsTableMetaDataHelper
       }
 
       List<QWidgetMetaDataInterface> widgets = new ArrayList<>();
+      widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(joins.get(JOIN_POLICY_STEP)).withLabel("Steps").withCanAddChildRecord(true).withMaxRows(100)
+         .withOrderBys(List.of(new QFilterOrderBy("stepOrder", true)))
+         .withOnlyIncludeFieldNames(List.of("stepOrder", "afterMinutes", "targetType", "onlyIfStatus")).getWidgetMetaData());
       widgets.add(childList(joins.get(JOIN_ALERT_EVENT), "Timeline", 50, new QFilterOrderBy("at", false), List.of("at", "eventType", "userId", "toValue", "note")));
       widgets.add(childList(joins.get(JOIN_ALERT_RECORD_REF), "Records", 25, new QFilterOrderBy("id", true), List.of("tableName", "recordId", "label", "active")));
       widgets.add(childList(joins.get(JOIN_ALERT_DELIVERY), "Notifications", 25, new QFilterOrderBy("id", false), List.of("kind", "status", "recipientLabel", "sentAt")));

src/main/java/com/kingsrook/qbits/alerts/model/AlertsAlert.java

@@ -147,6 +147,9 @@ public class AlertsAlert extends QRecordEntity
    @QField(isEditable = false)
    private Integer escalationLevel;
 
+   @QField(isEditable = false)
+   private Instant timedNotificationCheckedAt;
+
    @QField(isEditable = false)
    private Instant createDate;
 
@@ -1380,4 +1383,35 @@ public class AlertsAlert extends QRecordEntity
       this.modifyDate = modifyDate;
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for timedNotificationCheckedAt
+    *******************************************************************************/
+   public Instant getTimedNotificationCheckedAt()
+   {
+      return (this.timedNotificationCheckedAt);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for timedNotificationCheckedAt
+    *******************************************************************************/
+   public void setTimedNotificationCheckedAt(Instant timedNotificationCheckedAt)
+   {
+      this.timedNotificationCheckedAt = timedNotificationCheckedAt;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for timedNotificationCheckedAt
+    *******************************************************************************/
+   public AlertsAlert withTimedNotificationCheckedAt(Instant timedNotificationCheckedAt)
+   {
+      this.timedNotificationCheckedAt = timedNotificationCheckedAt;
+      return (this);
+   }
 }

src/main/java/com/kingsrook/qbits/alerts/model/AlertsEscalationPolicy.java

@@ -0,0 +1,323 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.alerts.model;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/*******************************************************************************
+ ** Timed notification steps for an unacknowledged alert.
+ *******************************************************************************/
+public class AlertsEscalationPolicy extends QRecordEntity
+{
+   public static final String TABLE_NAME = "alerts_escalation_policy";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isRequired = true, maxLength = 100)
+   private String name;
+
+   @QField()
+   private String description;
+
+   @QField()
+   private Integer repeatAfterMinutes;
+
+   @QField()
+   private Integer maxRepeats;
+
+   @QField()
+   private Boolean enabled;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+
+
+   /*******************************************************************************
+    ** Default constructor.
+    *******************************************************************************/
+   public AlertsEscalationPolicy()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Populate from a QRecord.
+    *******************************************************************************/
+   public AlertsEscalationPolicy(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public AlertsEscalationPolicy withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for name
+    *******************************************************************************/
+   public String getName()
+   {
+      return (this.name);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for name
+    *******************************************************************************/
+   public void setName(String name)
+   {
+      this.name = name;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for name
+    *******************************************************************************/
+   public AlertsEscalationPolicy withName(String name)
+   {
+      this.name = name;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for description
+    *******************************************************************************/
+   public String getDescription()
+   {
+      return (this.description);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for description
+    *******************************************************************************/
+   public void setDescription(String description)
+   {
+      this.description = description;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for description
+    *******************************************************************************/
+   public AlertsEscalationPolicy withDescription(String description)
+   {
+      this.description = description;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for repeatAfterMinutes
+    *******************************************************************************/
+   public Integer getRepeatAfterMinutes()
+   {
+      return (this.repeatAfterMinutes);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for repeatAfterMinutes
+    *******************************************************************************/
+   public void setRepeatAfterMinutes(Integer repeatAfterMinutes)
+   {
+      this.repeatAfterMinutes = repeatAfterMinutes;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for repeatAfterMinutes
+    *******************************************************************************/
+   public AlertsEscalationPolicy withRepeatAfterMinutes(Integer repeatAfterMinutes)
+   {
+      this.repeatAfterMinutes = repeatAfterMinutes;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for maxRepeats
+    *******************************************************************************/
+   public Integer getMaxRepeats()
+   {
+      return (this.maxRepeats);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for maxRepeats
+    *******************************************************************************/
+   public void setMaxRepeats(Integer maxRepeats)
+   {
+      this.maxRepeats = maxRepeats;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for maxRepeats
+    *******************************************************************************/
+   public AlertsEscalationPolicy withMaxRepeats(Integer maxRepeats)
+   {
+      this.maxRepeats = maxRepeats;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for enabled
+    *******************************************************************************/
+   public Boolean getEnabled()
+   {
+      return (this.enabled);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for enabled
+    *******************************************************************************/
+   public void setEnabled(Boolean enabled)
+   {
+      this.enabled = enabled;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for enabled
+    *******************************************************************************/
+   public AlertsEscalationPolicy withEnabled(Boolean enabled)
+   {
+      this.enabled = enabled;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public AlertsEscalationPolicy withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (this.modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public AlertsEscalationPolicy withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+}

src/main/java/com/kingsrook/qbits/alerts/model/AlertsEscalationStep.java

@@ -0,0 +1,425 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.alerts.model;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/*******************************************************************************
+ ** One ordered destination in an escalation policy.
+ *******************************************************************************/
+public class AlertsEscalationStep extends QRecordEntity
+{
+   public static final String TABLE_NAME = "alerts_escalation_step";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isRequired = true, possibleValueSourceName = "alerts_escalation_policy")
+   private Integer policyId;
+
+   @QField(isRequired = true)
+   private Integer stepOrder;
+
+   @QField(isRequired = true)
+   private Integer afterMinutes;
+
+   @QField(isRequired = true, maxLength = 16, possibleValueSourceName = "alerts_subscriber_type")
+   private String targetType;
+
+   @QField(maxLength = 100)
+   private String userId;
+
+   @QField(possibleValueSourceName = "alerts_team")
+   private Integer teamId;
+
+   @QField(possibleValueSourceName = "alerts_channel")
+   private Integer channelId;
+
+   @QField(isRequired = true, maxLength = 16, possibleValueSourceName = "alerts_status")
+   private String onlyIfStatus;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+
+
+   /*******************************************************************************
+    ** Default constructor.
+    *******************************************************************************/
+   public AlertsEscalationStep()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Populate from a QRecord.
+    *******************************************************************************/
+   public AlertsEscalationStep(QRecord record)
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public AlertsEscalationStep withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for policyId
+    *******************************************************************************/
+   public Integer getPolicyId()
+   {
+      return (this.policyId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for policyId
+    *******************************************************************************/
+   public void setPolicyId(Integer policyId)
+   {
+      this.policyId = policyId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for policyId
+    *******************************************************************************/
+   public AlertsEscalationStep withPolicyId(Integer policyId)
+   {
+      this.policyId = policyId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for stepOrder
+    *******************************************************************************/
+   public Integer getStepOrder()
+   {
+      return (this.stepOrder);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for stepOrder
+    *******************************************************************************/
+   public void setStepOrder(Integer stepOrder)
+   {
+      this.stepOrder = stepOrder;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for stepOrder
+    *******************************************************************************/
+   public AlertsEscalationStep withStepOrder(Integer stepOrder)
+   {
+      this.stepOrder = stepOrder;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for afterMinutes
+    *******************************************************************************/
+   public Integer getAfterMinutes()
+   {
+      return (this.afterMinutes);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for afterMinutes
+    *******************************************************************************/
+   public void setAfterMinutes(Integer afterMinutes)
+   {
+      this.afterMinutes = afterMinutes;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for afterMinutes
+    *******************************************************************************/
+   public AlertsEscalationStep withAfterMinutes(Integer afterMinutes)
+   {
+      this.afterMinutes = afterMinutes;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for targetType
+    *******************************************************************************/
+   public String getTargetType()
+   {
+      return (this.targetType);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for targetType
+    *******************************************************************************/
+   public void setTargetType(String targetType)
+   {
+      this.targetType = targetType;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for targetType
+    *******************************************************************************/
+   public AlertsEscalationStep withTargetType(String targetType)
+   {
+      this.targetType = targetType;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for userId
+    *******************************************************************************/
+   public String getUserId()
+   {
+      return (this.userId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userId
+    *******************************************************************************/
+   public void setUserId(String userId)
+   {
+      this.userId = userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userId
+    *******************************************************************************/
+   public AlertsEscalationStep withUserId(String userId)
+   {
+      this.userId = userId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for teamId
+    *******************************************************************************/
+   public Integer getTeamId()
+   {
+      return (this.teamId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for teamId
+    *******************************************************************************/
+   public void setTeamId(Integer teamId)
+   {
+      this.teamId = teamId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for teamId
+    *******************************************************************************/
+   public AlertsEscalationStep withTeamId(Integer teamId)
+   {
+      this.teamId = teamId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for channelId
+    *******************************************************************************/
+   public Integer getChannelId()
+   {
+      return (this.channelId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for channelId
+    *******************************************************************************/
+   public void setChannelId(Integer channelId)
+   {
+      this.channelId = channelId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for channelId
+    *******************************************************************************/
+   public AlertsEscalationStep withChannelId(Integer channelId)
+   {
+      this.channelId = channelId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for onlyIfStatus
+    *******************************************************************************/
+   public String getOnlyIfStatus()
+   {
+      return (this.onlyIfStatus);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for onlyIfStatus
+    *******************************************************************************/
+   public void setOnlyIfStatus(String onlyIfStatus)
+   {
+      this.onlyIfStatus = onlyIfStatus;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for onlyIfStatus
+    *******************************************************************************/
+   public AlertsEscalationStep withOnlyIfStatus(String onlyIfStatus)
+   {
+      this.onlyIfStatus = onlyIfStatus;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public AlertsEscalationStep withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (this.modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public AlertsEscalationStep withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+}

src/main/java/com/kingsrook/qbits/alerts/model/AlertsGroup.java

@@ -60,6 +60,9 @@ public class AlertsGroup extends QRecordEntity
    @QField(possibleValueSourceName = "alerts_team", label = "Owner Team")
    private Integer ownerTeamId;
 
+   @QField(possibleValueSourceName = "alerts_escalation_policy")
+   private Integer escalationPolicyId;
+
    @QField(isEditable = false)
    private Instant createDate;
 
@@ -394,4 +397,35 @@ public class AlertsGroup extends QRecordEntity
       this.modifyDate = modifyDate;
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for escalationPolicyId
+    *******************************************************************************/
+   public Integer getEscalationPolicyId()
+   {
+      return (this.escalationPolicyId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for escalationPolicyId
+    *******************************************************************************/
+   public void setEscalationPolicyId(Integer escalationPolicyId)
+   {
+      this.escalationPolicyId = escalationPolicyId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for escalationPolicyId
+    *******************************************************************************/
+   public AlertsGroup withEscalationPolicyId(Integer escalationPolicyId)
+   {
+      this.escalationPolicyId = escalationPolicyId;
+      return (this);
+   }
 }

src/main/java/com/kingsrook/qbits/alerts/model/AlertsMute.java

@@ -47,9 +47,12 @@ public class AlertsMute extends QRecordEntity
    @QField(isRequired = true, maxLength = 20, valueTooLongBehavior = ValueTooLongBehavior.TRUNCATE_ELLIPSIS, possibleValueSourceName = "alerts_scope_type")
    private String scopeType;
 
-   @QField(maxLength = 300, valueTooLongBehavior = ValueTooLongBehavior.TRUNCATE_ELLIPSIS)
+   @QField(maxLength = 520, valueTooLongBehavior = ValueTooLongBehavior.TRUNCATE_ELLIPSIS)
    private String scopeValue;
 
+   @QField(maxLength = 100, valueTooLongBehavior = ValueTooLongBehavior.TRUNCATE_ELLIPSIS, label = "Scope Record")
+   private String scopeRecordId;
+
    @QField(isRequired = true)
    private Instant startsAt;
 
@@ -247,6 +250,37 @@ public class AlertsMute extends QRecordEntity
 
 
 
+   /*******************************************************************************
+    ** Getter for scopeRecordId
+    *******************************************************************************/
+   public String getScopeRecordId()
+   {
+      return (this.scopeRecordId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for scopeRecordId
+    *******************************************************************************/
+   public void setScopeRecordId(String scopeRecordId)
+   {
+      this.scopeRecordId = scopeRecordId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for scopeRecordId
+    *******************************************************************************/
+   public AlertsMute withScopeRecordId(String scopeRecordId)
+   {
+      this.scopeRecordId = scopeRecordId;
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for startsAt
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/alerts/model/AlertsPossibleValues.java

@@ -132,7 +132,7 @@ public final class AlertsPossibleValues
     *******************************************************************************/
    public enum ScopeType
    {
-      ALL, GROUP, ALERT_TYPE, MONITOR, TABLE, RECORD, ATTRIBUTE
+      ALL, GROUP, ALERT_TYPE, MONITOR, TABLE, RECORD, ATTRIBUTE, ALERT
    }
 
 
@@ -251,7 +251,7 @@ public final class AlertsPossibleValues
          enumSource(MONITOR_KIND, AlertMonitorKind.values()),
          enumSource(MONITOR_STATUS, MonitorStatus.values()),
          enumSource(EVENT_TYPE, EventType.values()),
-         enumSource(RESOLUTION, Resolution.values()),
+         enumSource(RESOLUTION, new Resolution[] { Resolution.PRODUCER, Resolution.MONITOR, Resolution.QUIET_PERIOD, Resolution.MANUAL }),
          enumSource(SUBSCRIBER_TYPE, SubscriberType.values()),
          enumSource(SCOPE_TYPE, ScopeType.values()),
          enumSource(DELIVERY_MODE, DeliveryMode.values()),

src/main/java/com/kingsrook/qbits/alerts/model/AlertsType.java

@@ -96,6 +96,9 @@ public class AlertsType extends QRecordEntity
    @QField(isEditable = false)
    private Integer openCount;
 
+   @QField(possibleValueSourceName = "alerts_escalation_policy")
+   private Integer escalationPolicyId;
+
    @QField(isEditable = false)
    private Instant createDate;
 
@@ -802,4 +805,35 @@ public class AlertsType extends QRecordEntity
       this.modifyDate = modifyDate;
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for escalationPolicyId
+    *******************************************************************************/
+   public Integer getEscalationPolicyId()
+   {
+      return (this.escalationPolicyId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for escalationPolicyId
+    *******************************************************************************/
+   public void setEscalationPolicyId(Integer escalationPolicyId)
+   {
+      this.escalationPolicyId = escalationPolicyId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for escalationPolicyId
+    *******************************************************************************/
+   public AlertsType withEscalationPolicyId(Integer escalationPolicyId)
+   {
+      this.escalationPolicyId = escalationPolicyId;
+      return (this);
+   }
 }

src/main/java/com/kingsrook/qbits/alerts/processes/AlertsAction.java

@@ -46,8 +46,9 @@ public enum AlertsAction
    ASSIGN("alertsAssign", "Assign", "assignment_ind", AlertsAlert.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, null),
    COMMENT("alertsAddComment", "Add comment", "comment", AlertsAlert.TABLE_NAME, AlertsQBitConfig.PERMISSION_VIEW, 1),
    MUTE_ALERT("alertsMuteAlert", "Mute", "notifications_off", AlertsAlert.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, 1),
+   MUTE_ALERT_TYPE("alertsMuteAlertType", "Mute this alert type", "notifications_off", AlertsAlert.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, 1),
    SUBSCRIBE_TYPE("alertsSubscribeToType", "Subscribe me", "notifications", AlertsType.TABLE_NAME, AlertsQBitConfig.PERMISSION_VIEW, 1),
-   MUTE_TYPE("alertsMuteType", "Mute", "notifications_off", AlertsType.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, 1),
+   MUTE_TYPE("alertsMuteType", "Mute this alert type", "notifications_off", AlertsType.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, 1),
    CREATE_MONITOR("alertsCreateMonitorFromType", "Create monitor", "add_alert", AlertsType.TABLE_NAME, AlertsQBitConfig.PERMISSION_ADMIN, 1),
    SUBSCRIBE_GROUP("alertsSubscribeToGroup", "Subscribe me", "notifications", AlertsGroup.TABLE_NAME, AlertsQBitConfig.PERMISSION_VIEW, 1),
    MUTE_GROUP("alertsMuteGroup", "Mute", "notifications_off", AlertsGroup.TABLE_NAME, AlertsQBitConfig.PERMISSION_MANAGE, 1),
@@ -131,7 +132,7 @@ public enum AlertsAction
                .withPossibleValueSourceName(AlertsPossibleValues.SEVERITY),
             new QFieldMetaData(FIELD_DELIVERY_MODE, QFieldType.STRING).withLabel("Delivery").withIsRequired(true).withDefaultValue("IMMEDIATE")
                .withPossibleValueSourceName(AlertsPossibleValues.DELIVERY_MODE));
-         case MUTE_ALERT, MUTE_TYPE, MUTE_GROUP, MUTE_MONITOR -> List.of(
+         case MUTE_ALERT, MUTE_ALERT_TYPE, MUTE_TYPE, MUTE_GROUP, MUTE_MONITOR -> List.of(
             new QFieldMetaData(FIELD_MUTE_MINUTES, QFieldType.INTEGER).withLabel("Mute for (minutes)").withIsRequired(true).withDefaultValue(60),
             new QFieldMetaData(FIELD_NOTE, QFieldType.TEXT).withLabel("Reason").withIsRequired(true));
          default -> List.of();

src/main/java/com/kingsrook/qbits/alerts/processes/AlertsActionSteps.java

@@ -26,6 +26,7 @@ import java.util.Objects;
 import com.kingsrook.qbits.alerts.AlertsRuntime;
 import com.kingsrook.qbits.alerts.AlertsSecurity;
 import com.kingsrook.qbits.alerts.engine.AlertsDb;
+import com.kingsrook.qbits.alerts.engine.AlertsEngine;
 import com.kingsrook.qbits.alerts.engine.MonitorEvaluator;
 import com.kingsrook.qbits.alerts.model.AlertsMonitor;
 import com.kingsrook.qbits.alerts.model.AlertsMute;
@@ -302,7 +303,8 @@ public final class AlertsActionSteps
             case MUTE_TYPE -> mute(ScopeType.ALERT_TYPE, record.getValueString("name"), "type " + record.getValueString("label"), input, actor);
             case MUTE_GROUP -> mute(ScopeType.GROUP, record.getValueString("name"), "group " + record.getValueString("label"), input, actor);
             case MUTE_MONITOR -> mute(ScopeType.MONITOR, String.valueOf(id), "monitor " + record.getValueString("name"), input, actor);
-            case MUTE_ALERT -> mute(ScopeType.ALERT_TYPE, record.getValueString("alertTypeName"), "alert type " + record.getValueString("alertTypeName"), input, actor);
+            case MUTE_ALERT -> muteAlert(record, input, actor);
+            case MUTE_ALERT_TYPE -> mute(ScopeType.ALERT_TYPE, record.getValueString("alertTypeName"), "alert type " + record.getValueString("alertTypeName"), input, actor);
             case RUN_CHECK ->
             {
                MonitorEvaluator.Plan plan = runtime.getEvaluator().runNow(id);
@@ -421,9 +423,34 @@ public final class AlertsActionSteps
 
 
    /*******************************************************************************
-    ** Create a silence-notifications mute from now (through the table).
+    ** Mute the subject record when present, otherwise this type and dedup key.
+    *******************************************************************************/
+   private static boolean muteAlert(QRecord alert, RunBackendStepInput input, String actor) throws QException
+   {
+      if(StringUtils.hasContent(alert.getValueString("subjectTableName")) && StringUtils.hasContent(alert.getValueString("subjectRecordId")))
+      {
+         return (mute(ScopeType.RECORD, alert.getValueString("subjectTableName"), alert.getValueString("subjectRecordId"), "alert " + alert.getValue("id"), input, actor));
+      }
+      return (mute(ScopeType.ALERT, AlertsEngine.openKey(alert.getValueString("alertTypeName"), alert.getValueString("dedupKey")), null,
+         "alert " + alert.getValue("id"), input, actor));
+   }
+
+
+
+   /*******************************************************************************
+    ** Create a mute without a subject record ID.
     *******************************************************************************/
    private static boolean mute(ScopeType scopeType, String scopeValue, String what, RunBackendStepInput input, String actor) throws QException
+   {
+      return (mute(scopeType, scopeValue, null, what, input, actor));
+   }
+
+
+
+   /*******************************************************************************
+    ** Create a silence-notifications mute from now (through the table).
+    *******************************************************************************/
+   private static boolean mute(ScopeType scopeType, String scopeValue, String scopeRecordId, String what, RunBackendStepInput input, String actor) throws QException
    {
       Integer minutes = input.getValueInteger(AlertsAction.FIELD_MUTE_MINUTES);
       String  reason  = input.getValueString(AlertsAction.FIELD_NOTE);
@@ -437,6 +464,7 @@ public final class AlertsActionSteps
          .withValue("reason", reason)
          .withValue("scopeType", scopeType.name())
          .withValue("scopeValue", scopeValue)
+         .withValue("scopeRecordId", scopeRecordId)
          .withValue("startsAt", now)
          .withValue("endsAt", now.plus(minutes == null || minutes < 1 ? 60 : minutes, ChronoUnit.MINUTES))
          .withValue("effect", MuteEffect.SILENCE_NOTIFICATIONS.name())
@@ -586,7 +614,7 @@ public final class AlertsActionSteps
          case ASSIGN -> "assigned";
          case COMMENT -> "commented on";
          case SUBSCRIBE_TYPE, SUBSCRIBE_GROUP, SUBSCRIBE_MONITOR -> "subscribed to";
-         case MUTE_ALERT, MUTE_TYPE, MUTE_GROUP, MUTE_MONITOR -> "muted";
+         case MUTE_ALERT, MUTE_ALERT_TYPE, MUTE_TYPE, MUTE_GROUP, MUTE_MONITOR -> "muted";
          case RUN_CHECK -> "checked";
          case ENABLE_MONITOR -> "enabled";
          case DISABLE_MONITOR -> "disabled";

src/main/java/com/kingsrook/qbits/alerts/processes/AlertsScheduledSteps.java

@@ -18,6 +18,7 @@ package com.kingsrook.qbits.alerts.processes;
 
 
 import com.kingsrook.qbits.alerts.AlertsRuntime;
+import com.kingsrook.qbits.alerts.engine.EscalationAndReminder;
 import com.kingsrook.qbits.alerts.notifications.NotificationEmailer;
 import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -80,4 +81,21 @@ public final class AlertsScheduledSteps
          output.addValue(OUTPUT_COUNT, runtime.getDeliveryWorker().deliverDue());
       }
    }
+
+
+
+   /*******************************************************************************
+    ** alertsEscalateAndRemind
+    *******************************************************************************/
+   public static class EscalateAndRemind implements BackendStep
+   {
+      /***************************************************************************
+       ** Run timed notifications using the installed runtime's clock.
+       ***************************************************************************/
+      @Override
+      public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
+      {
+         output.addValue(OUTPUT_COUNT, new EscalationAndReminder(AlertsRuntime.get()).runDue());
+      }
+   }
 }

src/main/resources/db/liquibase/qbit-alerts/alerts.changelog.sql

@@ -427,3 +427,69 @@ CREATE TABLE IF NOT EXISTS alerts_notification_preference (
    CONSTRAINT pk_alerts_notification_preference PRIMARY KEY (id),
    CONSTRAINT uq_alerts_notification_preference UNIQUE (user_id, category)
 );
+
+-- changeset qbit-alerts:21
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM INFORMATION_SCHEMA.COLUMNS WHERE LOWER(TABLE_NAME) = 'alerts_mute' AND LOWER(COLUMN_NAME) = 'scope_record_id'
+-- comment: Record mutes store the subject record ID separately from its table name.
+ALTER TABLE alerts_mute ADD COLUMN scope_record_id VARCHAR(100);
+
+-- changeset qbit-alerts:22
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:1 SELECT COUNT(*) FROM INFORMATION_SCHEMA.COLUMNS WHERE LOWER(TABLE_NAME) = 'alerts_mute' AND LOWER(COLUMN_NAME) = 'scope_value' AND CHARACTER_MAXIMUM_LENGTH < 520
+-- comment: ALERT mutes hold the complete type and dedup key, including long keys.
+ALTER TABLE alerts_mute ALTER COLUMN scope_value SET DATA TYPE VARCHAR(520);
+
+-- changeset qbit-alerts:23
+-- comment: Admin escalation policies and their ordered destinations.
+CREATE TABLE IF NOT EXISTS alerts_escalation_policy (
+   id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
+   name VARCHAR(100) NOT NULL,
+   description TEXT,
+   repeat_after_minutes INTEGER,
+   max_repeats INTEGER DEFAULT 0,
+   enabled BOOLEAN DEFAULT TRUE,
+   create_date TIMESTAMP,
+   modify_date TIMESTAMP,
+   CONSTRAINT pk_alerts_escalation_policy PRIMARY KEY (id),
+   CONSTRAINT uq_alerts_escalation_policy UNIQUE (name)
+);
+CREATE TABLE IF NOT EXISTS alerts_escalation_step (
+   id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
+   policy_id INTEGER NOT NULL,
+   step_order INTEGER NOT NULL,
+   after_minutes INTEGER NOT NULL,
+   target_type VARCHAR(16) NOT NULL,
+   user_id VARCHAR(100),
+   team_id INTEGER,
+   channel_id INTEGER,
+   only_if_status VARCHAR(16) DEFAULT 'OPEN' NOT NULL,
+   create_date TIMESTAMP,
+   modify_date TIMESTAMP,
+   CONSTRAINT pk_alerts_escalation_step PRIMARY KEY (id),
+   CONSTRAINT uq_alerts_escalation_step UNIQUE (policy_id, step_order),
+   CONSTRAINT ck_alerts_escalation_step_order CHECK (step_order > 0),
+   CONSTRAINT ck_alerts_escalation_step_minutes CHECK (after_minutes >= 0),
+   CONSTRAINT ck_alerts_escalation_step_target CHECK (
+      (target_type = 'USER' AND user_id IS NOT NULL AND team_id IS NULL AND channel_id IS NULL) OR
+      (target_type = 'TEAM' AND team_id IS NOT NULL AND user_id IS NULL AND channel_id IS NULL) OR
+      (target_type = 'CHANNEL' AND channel_id IS NOT NULL AND user_id IS NULL AND team_id IS NULL))
+);
+
+-- changeset qbit-alerts:24
+-- comment: Type policies override the group's policy.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_type' AND LOWER(column_name) = 'escalation_policy_id'
+ALTER TABLE alerts_type ADD COLUMN escalation_policy_id INTEGER;
+
+-- changeset qbit-alerts:25
+-- comment: A group policy is used when the type has none.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_group' AND LOWER(column_name) = 'escalation_policy_id'
+ALTER TABLE alerts_group ADD COLUMN escalation_policy_id INTEGER;
+
+-- changeset qbit-alerts:26
+-- comment: Persist the timed job checkpoint, including attempts without recipients.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_alert' AND LOWER(column_name) = 'timed_notification_checked_at'
+ALTER TABLE alerts_alert ADD COLUMN timed_notification_checked_at TIMESTAMP;

src/test/java/com/kingsrook/qbits/alerts/AlertsQBitProducerTest.java

@@ -26,6 +26,7 @@ import com.kingsrook.qbits.alerts.api.AlertsRouteProvider;
 import com.kingsrook.qbits.alerts.metadata.AlertsAppMetaDataProducer;
 import com.kingsrook.qbits.alerts.metadata.AlertsHomeWidgetsMetaDataProducer;
 import com.kingsrook.qbits.alerts.metadata.AlertsProcessesMetaDataProducer;
+import com.kingsrook.qbits.alerts.metadata.AlertsTableMetaDataHelper;
 import com.kingsrook.qbits.alerts.processes.AlertsAction;
 import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
 import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
@@ -69,6 +70,23 @@ class AlertsQBitProducerTest extends BaseAlertsTest
 
 
 
+   /*******************************************************************************
+    ** Team and member screens explain missing host setup only when needed.
+    *******************************************************************************/
+   @Test
+   void testTeamResolverHelp() throws Exception
+   {
+      assertThat(AlertsTableMetaDataHelper.buildTeamTable(config).getField("memberSource").getHelpContents())
+         .anyMatch(help -> help.getContent().contains("ROLE and IdP Group members need a host resolver; none is configured"));
+      assertThat(AlertsTableMetaDataHelper.buildTeamMemberTable(config).getField("teamId").getHelpContents())
+         .anyMatch(help -> help.getContent().contains("none is configured"));
+      config.withAudienceResolver((source, value) -> Set.of());
+      assertThat(AlertsTableMetaDataHelper.buildTeamTable(config).getField("memberSource").getHelpContents()).isNullOrEmpty();
+      assertThat(AlertsTableMetaDataHelper.buildTeamMemberTable(config).getField("teamId").getHelpContents()).isNullOrEmpty();
+   }
+
+
+
    /*******************************************************************************
     ** Without a config the producer is disabled; bad configs fail.
     *******************************************************************************/
@@ -182,7 +200,67 @@ class AlertsQBitProducerTest extends BaseAlertsTest
          QTableMetaData table = qInstance.getTable(tableName);
          for(QFieldMetaData field : table.getFields().values())
          {
-            assertThat(body).as(tableName + "." + field.getName()).contains("   " + QInstanceEnricher.inferBackendName(field.getName()) + " ");
+            String column = QInstanceEnricher.inferBackendName(field.getName());
+            assertThat(body.contains("   " + column + " ") || changelog.contains("ALTER TABLE " + created + " ADD COLUMN " + column + " "))
+               .as(tableName + "." + field.getName()).isTrue();
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Upgrade the old mute table using the shipped SQL and guards twice;
+    ** existing rules survive and the new fields accept full alert keys.
+    *******************************************************************************/
+   @Test
+   void testMuteSchemaUpgradeIsRepeatable() throws Exception
+   {
+      String changelog;
+      try(InputStream in = getClass().getClassLoader().getResourceAsStream("db/liquibase/qbit-alerts/alerts.changelog.sql"))
+      {
+         changelog = new String(in.readAllBytes(), StandardCharsets.UTF_8);
+      }
+      int start = changelog.indexOf("CREATE TABLE IF NOT EXISTS alerts_mute (");
+      try(java.sql.Connection connection = com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager.getConnection(defineH2Backend());
+         java.sql.Statement statement = connection.createStatement())
+      {
+         statement.execute("DROP TABLE alerts_mute");
+         statement.execute(changelog.substring(start, changelog.indexOf(");", start) + 2));
+         statement.execute("INSERT INTO alerts_mute (name, scope_type, scope_value, starts_at, ends_at) VALUES ('Existing', 'ALERT_TYPE', 'existing.type', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)");
+         for(int pass = 0; pass < 2; pass++)
+         {
+            for(String number : List.of("21", "22"))
+            {
+               String block = changelog.substring(changelog.indexOf("-- changeset qbit-alerts:" + number));
+               String guard = block.lines().filter(line -> line.startsWith("-- precondition-sql-check")).findFirst().orElseThrow();
+               String sql = block.lines().filter(line -> line.startsWith("ALTER TABLE")).findFirst().orElseThrow();
+               int expected = Integer.parseInt(guard.substring(guard.indexOf("expectedResult:") + 15, guard.indexOf(" SELECT")));
+               try(java.sql.ResultSet result = statement.executeQuery(guard.substring(guard.indexOf("SELECT"))))
+               {
+                  assertThat(result.next()).isTrue();
+                  if(result.getInt(1) != expected)
+                  {
+                     continue;
+                  }
+               }
+               statement.execute(sql);
+            }
+         }
+         try(java.sql.ResultSet result = statement.executeQuery("SELECT scope_type, scope_value, scope_record_id FROM alerts_mute"))
+         {
+            assertThat(result.next()).isTrue();
+            assertThat(result.getString(1)).isEqualTo("ALERT_TYPE");
+            assertThat(result.getString(2)).isEqualTo("existing.type");
+            assertThat(result.getString(3)).isNull();
+         }
+         String key = "k".repeat(520);
+         statement.execute("UPDATE alerts_mute SET scope_value = '" + key + "', scope_record_id = '123'");
+         try(java.sql.ResultSet result = statement.executeQuery("SELECT scope_value, scope_record_id FROM alerts_mute"))
+         {
+            assertThat(result.next()).isTrue();
+            assertThat(result.getString(1)).isEqualTo(key);
+            assertThat(result.getString(2)).isEqualTo("123");
          }
       }
    }
@@ -216,4 +294,37 @@ class AlertsQBitProducerTest extends BaseAlertsTest
       assertThat(AlertsIntegrations.esbTableNames()).containsExactly("alerts_alert", "alerts_monitor", "alerts_type");
       assertThat(AlertsIntegrations.quickSearchTables()).hasSize(3);
    }
+
+
+   /*******************************************************************************
+    ** Policy records are admin-only, steps are children, and the job is hidden.
+    *******************************************************************************/
+   @Test
+   void testEscalationMetadata() throws Exception
+   {
+      assertThat(AlertsAppMetaDataProducer.MAIN_TABLES).contains("alerts_escalation_policy").doesNotContain("alerts_escalation_step");
+      assertThat(qInstance.getTable("alerts_escalation_policy").getSections()).anySatisfy(section ->
+         assertThat(section.getWidgetName()).isEqualTo("alerts_escalation_policy_join_alerts_escalation_step"));
+      assertThat(qInstance.getTable("alerts_type").getField("escalationPolicyId").getPossibleValueSourceName()).isEqualTo("alerts_escalation_policy");
+      assertThat(qInstance.getTable("alerts_group").getField("escalationPolicyId").getPossibleValueSourceName()).isEqualTo("alerts_escalation_policy");
+      assertThat(qInstance.getTable("alerts_escalation_step").getField("onlyIfStatus").getDefaultValue()).isEqualTo("OPEN");
+      assertThat(qInstance.getPossibleValueSource("alerts_resolution").getEnumValues()).extracting(value -> String.valueOf(value.getId())).doesNotContain("EXPIRED");
+      QProcessMetaData process = qInstance.getProcess(AlertsProcessesMetaDataProducer.ESCALATE_AND_REMIND);
+      assertThat(process.getIsHidden()).isTrue();
+      assertThat(process.getSchedule()).isNull();
+      QProcessMetaData scheduled = AlertsProcessesMetaDataProducer.buildProcesses(new AlertsQBitConfig().withSchedulerName("scheduler"))
+         .stream().filter(candidate -> candidate.getName().equals(AlertsProcessesMetaDataProducer.ESCALATE_AND_REMIND)).findFirst().orElseThrow();
+      assertThat(scheduled.getSchedule().getSchedulerName()).isEqualTo("scheduler");
+      assertThat(scheduled.getSchedule().getRepeatSeconds()).isEqualTo(60);
+      loginAs(VIEWER);
+      for(String table : List.of("alerts_escalation_policy", "alerts_escalation_step"))
+      {
+         assertThat(PermissionsHelper.hasTablePermission(new com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput(), table,
+            com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType.READ)).isFalse();
+      }
+      loginAs(ADMIN);
+      assertThat(PermissionsHelper.hasTablePermission(new com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput(), "alerts_escalation_policy",
+         com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType.INSERT)).isTrue();
+   }
+
 }

src/test/java/com/kingsrook/qbits/alerts/SecurityAndActionsTest.java

@@ -271,6 +271,100 @@ class SecurityAndActionsTest extends BaseAlertsTest
 
 
 
+   /*******************************************************************************
+    ** The record action mutes its subject, leaving other records of the same
+    ** alert type deliverable. The separate type action silences those too.
+    *******************************************************************************/
+   @Test
+   void testRecordAndTypeMuteActions() throws Exception
+   {
+      List<Integer> widgets = insertWidgets("BROKEN", "BROKEN");
+      QRecord one = runtime.getEngine().raise(new AlertSignal(TYPE).withDedupKey("one").withRecordRef(AlertRecordRef.of(WIDGET, widgets.get(0))));
+      QRecord two = runtime.getEngine().raise(new AlertSignal(TYPE).withDedupKey("two").withRecordRef(AlertRecordRef.of(WIDGET, widgets.get(1))));
+      subscribeForMuteTest();
+      loginAs(MANAGER);
+      assertThat(run(AlertsAction.MUTE_ALERT, List.of(one.getValueInteger("id")), Map.of(AlertsAction.FIELD_NOTE, "Working on this record")))
+         .anyMatch(line -> line.startsWith("OK: 1"));
+      QRecord mute = all(AlertsMute.TABLE_NAME).get(0);
+      assertThat(mute.getValueString("scopeType")).isEqualTo("RECORD");
+      assertThat(mute.getValueString("scopeValue")).isEqualTo(WIDGET);
+      assertThat(mute.getValueString("scopeRecordId")).isEqualTo(String.valueOf(widgets.get(0)));
+      routeForMuteTest(one);
+      routeForMuteTest(two);
+      assertThat(deliveriesFor(one)).singleElement().matches(d -> "MUTED".equals(d.getValueString("suppressedReason")));
+      assertThat(deliveriesFor(two)).singleElement().matches(d -> "SENT".equals(d.getValueString("status")));
+
+      run(AlertsAction.MUTE_ALERT_TYPE, List.of(one.getValueInteger("id")), Map.of(AlertsAction.FIELD_NOTE, "Working on the type"));
+      routeForMuteTest(two);
+      assertThat(deliveriesFor(two)).anyMatch(d -> "MUTED".equals(d.getValueString("suppressedReason")));
+      assertThat(AlertsAction.MUTE_ALERT_TYPE.getLabel()).isEqualTo("Mute this alert type");
+      assertThat(AlertsAction.MUTE_TYPE.getLabel()).isEqualTo("Mute this alert type");
+   }
+
+
+
+   /*******************************************************************************
+    ** Subjectless alerts mute by full type and dedup key, never by type alone.
+    *******************************************************************************/
+   @Test
+   void testSubjectlessMuteAction() throws Exception
+   {
+      String type = "test.system." + "t".repeat(170);
+      String key = "k".repeat(300);
+      QRecord one = runtime.getEngine().raise(new AlertSignal(type).withDedupKey(key));
+      QRecord two = runtime.getEngine().raise(new AlertSignal(type).withDedupKey("other"));
+      QRecord otherType = runtime.getEngine().raise(new AlertSignal("test.system.other").withDedupKey(key));
+      subscribeForMuteTest();
+      loginAs(MANAGER);
+      run(AlertsAction.MUTE_ALERT, List.of(one.getValueInteger("id")), Map.of(AlertsAction.FIELD_NOTE, "Working on this alert"));
+      QRecord mute = all(AlertsMute.TABLE_NAME).get(0);
+      assertThat(mute.getValueString("scopeType")).isEqualTo("ALERT");
+      assertThat(mute.getValueString("scopeValue")).isEqualTo(type + "|" + key);
+      routeForMuteTest(one);
+      routeForMuteTest(two);
+      routeForMuteTest(otherType);
+      assertThat(deliveriesFor(one)).singleElement().matches(d -> "MUTED".equals(d.getValueString("suppressedReason")));
+      assertThat(deliveriesFor(two)).singleElement().matches(d -> "SENT".equals(d.getValueString("status")));
+      assertThat(deliveriesFor(otherType)).singleElement().matches(d -> "SENT".equals(d.getValueString("status")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Subscribe after raising so initial delivery does not mask mute assertions.
+    *******************************************************************************/
+   private void subscribeForMuteTest() throws QException
+   {
+      runtime.asSystem(() -> AlertsDb.insert(AlertsSubscription.TABLE_NAME, new QRecord().withValue("subscriberType", "USER")
+         .withValue("userId", VIEWER).withValue("scopeType", "ALL").withValue("minSeverity", "WARNING").withValue("enabled", true)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Route a fresh notification event against the current mute rules.
+    *******************************************************************************/
+   private void routeForMuteTest(QRecord alert) throws QException
+   {
+      runtime.asSystem(() ->
+      {
+         new com.kingsrook.qbits.alerts.engine.AlertRouter(runtime).route(alert, com.kingsrook.qbits.alerts.model.AlertsPossibleValues.EventType.RAISED, null);
+         return (null);
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Read delivery outcomes for one alert.
+    *******************************************************************************/
+   private List<QRecord> deliveriesFor(QRecord alert) throws QException
+   {
+      return (where(AlertsDelivery.TABLE_NAME, AlertsDb.eq("alertId", alert.getValue("id"))));
+   }
+
+
+
    /*******************************************************************************
     ** Type, group and monitor actions; channel and notification actions.
     *******************************************************************************/

src/test/java/com/kingsrook/qbits/alerts/engine/EscalationAndReminderTest.java

@@ -0,0 +1,534 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.alerts.engine;
+
+
+import java.time.Duration;
+import java.util.List;
+import com.kingsrook.qbits.alerts.BaseAlertsTest;
+import com.kingsrook.qbits.alerts.model.AlertsAlert;
+import com.kingsrook.qbits.alerts.model.AlertsChannel;
+import com.kingsrook.qbits.alerts.model.AlertsDelivery;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationPolicy;
+import com.kingsrook.qbits.alerts.model.AlertsEscalationStep;
+import com.kingsrook.qbits.alerts.model.AlertsEvent;
+import com.kingsrook.qbits.alerts.model.AlertsGroup;
+import com.kingsrook.qbits.alerts.model.AlertsMute;
+import com.kingsrook.qbits.alerts.model.AlertsPossibleValues.Resolution;
+import com.kingsrook.qbits.alerts.model.AlertsSubscription;
+import com.kingsrook.qbits.alerts.model.AlertsTeam;
+import com.kingsrook.qbits.alerts.model.AlertsTeamMember;
+import com.kingsrook.qbits.alerts.model.AlertsType;
+import com.kingsrook.qbits.alerts.processes.AlertsScheduledSteps;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
+import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
+import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
+import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
+import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Timed notifications against the real H2 tables and router, with a movable clock.
+ *******************************************************************************/
+class EscalationAndReminderTest extends BaseAlertsTest
+{
+   private static final String TYPE = "test.timed";
+
+
+
+   /*******************************************************************************
+    ** A normal declared alert type with a group.
+    *******************************************************************************/
+   @Override
+   protected void customizeInstance(QInstance instance)
+   {
+      AlertsInstanceMetaData.ofOrWithNew(instance).withAlertType(new QAlertTypeMetaData()
+         .withName(TYPE).withLabel("Timed alert").withGroupName("timed").withDefaultSeverity(AlertSeverity.ERROR));
+   }
+
+
+
+   /*******************************************************************************
+    ** Raise and tune the reminder interval without initial notifications.
+    *******************************************************************************/
+   private QRecord alert(Integer interval, AlertSeverity severity) throws QException
+   {
+      QRecord alert = runtime.getEngine().raise(new AlertSignal(TYPE).withSeverity(severity));
+      update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId")).withValue("reminderIntervalMinutes", interval));
+      return (alert);
+   }
+
+
+
+   /*******************************************************************************
+    ** Store changes as the system, like the engine.
+    *******************************************************************************/
+   private void update(String table, QRecord record) throws QException
+   {
+      runtime.asSystem(() -> AlertsDb.update(table, record));
+   }
+
+
+
+   /*******************************************************************************
+    ** Insert a test configuration row.
+    *******************************************************************************/
+   private QRecord insert(String table, QRecord record) throws QException
+   {
+      return (runtime.asSystem(() -> AlertsDb.insert(table, record)));
+   }
+
+
+
+   /*******************************************************************************
+    ** One subscription with an explicit event filter.
+    *******************************************************************************/
+   private void subscribe(String user, String events) throws QException
+   {
+      insert(AlertsSubscription.TABLE_NAME, new QRecord().withValue("subscriberType", "USER").withValue("userId", user)
+         .withValue("scopeType", "ALL").withValue("events", events).withValue("enabled", true));
+   }
+
+
+
+   /*******************************************************************************
+    ** A policy attached to this alert's type.
+    *******************************************************************************/
+   private QRecord policy(QRecord alert, int repeatMinutes, int maxRepeats) throws QException
+   {
+      QRecord policy = insert(AlertsEscalationPolicy.TABLE_NAME, new QRecord().withValue("name", "On call")
+         .withValue("enabled", true).withValue("repeatAfterMinutes", repeatMinutes).withValue("maxRepeats", maxRepeats));
+      update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId")).withValue("escalationPolicyId", policy.getValue("id")));
+      return (policy);
+   }
+
+
+
+   /*******************************************************************************
+    ** An explicit ordered target.
+    *******************************************************************************/
+   private QRecord step(QRecord policy, int order, int minutes, String targetType, String targetField, java.io.Serializable targetId) throws QException
+   {
+      return (insert(AlertsEscalationStep.TABLE_NAME, new QRecord().withValue("policyId", policy.getValue("id"))
+         .withValue("stepOrder", order).withValue("afterMinutes", minutes).withValue("targetType", targetType)
+         .withValue(targetField, targetId).withValue("onlyIfStatus", "OPEN")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Tick through the real hidden backend step.
+    *******************************************************************************/
+   private void tick() throws QException
+   {
+      new AlertsScheduledSteps.EscalateAndRemind().run(new RunBackendStepInput(), new RunBackendStepOutput());
+   }
+
+
+
+   /*******************************************************************************
+    ** Read a refreshed alert.
+    *******************************************************************************/
+   private QRecord fresh(QRecord alert) throws QException
+   {
+      return (runtime.asSystem(() -> AlertsDb.byId(AlertsAlert.TABLE_NAME, alert.getValue("id"))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Due boundary, event filter, NOTIFIED timeline and same-minute idempotency.
+    ** A one-minute interval also proves reminders bypass the ten-minute duplicate window.
+    *******************************************************************************/
+   @Test
+   void testReminderDueAndNotDue() throws Exception
+   {
+      QRecord alert = alert(1, AlertSeverity.ERROR);
+      subscribe("viewer-reminders", "REMINDER");
+      subscribe("viewer-raised", "RAISED");
+      clock.advance(Duration.ofSeconds(59));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+      clock.advance(Duration.ofSeconds(1));
+      tick();
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).singleElement().satisfies(row ->
+      {
+         assertThat(row.getValueString("kind")).isEqualTo("REMINDER");
+         assertThat(row.getValueString("recipientUserId")).isEqualTo("viewer-reminders");
+         assertThat(row.getValueString("status")).isEqualTo("SENT");
+      });
+      assertThat(fresh(alert).getValueInstant("lastNotifiedAt")).isEqualTo(clock.instant());
+      assertThat(all(AlertsEvent.TABLE_NAME)).extracting(row -> row.getValueString("eventType")).containsExactly("RAISED", "NOTIFIED");
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(2);
+   }
+
+
+
+   /*******************************************************************************
+    ** The newer of last raise and last successful notification sets the due time.
+    *******************************************************************************/
+   @Test
+   void testReminderUsesLatestRaiseOrNotification() throws Exception
+   {
+      QRecord alert = alert(10, AlertSeverity.ERROR);
+      subscribe("viewer-reminders", "REMINDER");
+      clock.advance(Duration.ofMinutes(8));
+      runtime.getEngine().raise(new AlertSignal(TYPE));
+      clock.advance(Duration.ofMinutes(2));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+      clock.advance(Duration.ofMinutes(8));
+      tick();
+      clock.advance(Duration.ofMinutes(9));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(2);
+      assertThat(fresh(alert).getValueInstant("lastNotifiedAt")).isEqualTo(clock.instant());
+   }
+
+
+
+   /*******************************************************************************
+    ** CRITICAL has a 60-minute default; explicit zero still disables reminders.
+    *******************************************************************************/
+   @Test
+   void testCriticalDefaultAndExplicitOff() throws Exception
+   {
+      QRecord alert = alert(null, AlertSeverity.CRITICAL);
+      subscribe("viewer", "REMINDER");
+      clock.advance(Duration.ofMinutes(59));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+      update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId")).withValue("reminderIntervalMinutes", 0));
+      clock.advance(Duration.ofHours(2));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Null is off for non-critical alerts.
+    *******************************************************************************/
+   @Test
+   void testReminderDefaultOff() throws Exception
+   {
+      alert(null, AlertSeverity.ERROR);
+      subscribe("viewer", "REMINDER");
+      clock.advance(Duration.ofDays(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Every stopping condition blocks both kinds of timed notification.
+    *******************************************************************************/
+   @ParameterizedTest
+   @ValueSource(strings = {"ACKNOWLEDGED", "SNOOZED", "RESOLVED", "muted", "disabled", "silent", "flapping", "SILENCE_NOTIFICATIONS", "SUPPRESS_ALERTS"})
+   void testStops(String condition) throws Exception
+   {
+      QRecord alert = alert(1, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 1, 2);
+      step(policy, 1, 1, "USER", "userId", "viewer");
+      subscribe("viewer", "REMINDER");
+      switch(condition)
+      {
+         case "ACKNOWLEDGED" -> runtime.getEngine().acknowledge(alert.getValueInteger("id"), ADMIN);
+         case "SNOOZED" -> runtime.getEngine().snooze(alert.getValueInteger("id"), clock.instant().plusSeconds(3600), ADMIN);
+         case "RESOLVED" -> runtime.getEngine().resolveById(alert.getValueInteger("id"), Resolution.MANUAL, ADMIN, "Done");
+         case "disabled", "silent" -> update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId"))
+            .withValue("disabled".equals(condition) ? "enabled" : "notify", false));
+         case "muted", "flapping" -> update(AlertsAlert.TABLE_NAME, new QRecord().withValue("id", alert.getValue("id")).withValue(condition, true));
+         default -> insert(AlertsMute.TABLE_NAME, new QRecord().withValue("name", "Maintenance").withValue("scopeType", "ALL")
+            .withValue("enabled", true).withValue("effect", condition).withValue("startsAt", clock.instant()).withValue("endsAt", clock.instant().plusSeconds(3600)));
+      }
+      clock.advance(Duration.ofMinutes(2));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** Sequential steps and capped repeats, with the same target inside ten minutes.
+    *******************************************************************************/
+   @Test
+   void testStepsRepeatsAndIdempotency() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 2, 2);
+      step(policy, 1, 1, "USER", "userId", "viewer");
+      step(policy, 2, 3, "USER", "userId", "viewer");
+      tick();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isZero();
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      tick();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isEqualTo(1);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isEqualTo(1);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isEqualTo(2);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(2);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      tick();
+      clock.advance(Duration.ofMinutes(2));
+      tick();
+      clock.advance(Duration.ofDays(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(4).allSatisfy(row ->
+      {
+         assertThat(row.getValueString("status")).isEqualTo("SENT");
+         assertThat(row.getValueString("kind")).isEqualTo("ESCALATED");
+      });
+      List<QRecord> events = where(AlertsEvent.TABLE_NAME, AlertsDb.eq("eventType", "ESCALATED"));
+      assertThat(events).extracting(row -> row.getValueString("fromValue")).containsExactly("0", "1", "2", "2");
+      assertThat(events).extracting(row -> row.getValueString("toValue")).containsExactly("1", "2", "2", "2");
+   }
+
+
+
+   /*******************************************************************************
+    ** Acknowledging after step one stops the next step and reminders.
+    *******************************************************************************/
+   @Test
+   void testAcknowledgeAfterFirstStep() throws Exception
+   {
+      QRecord alert = alert(1, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 1, 5);
+      step(policy, 1, 0, "USER", "userId", "viewer");
+      step(policy, 2, 1, "USER", "userId", "viewer");
+      subscribe("viewer", "REMINDER");
+      tick();
+      runtime.getEngine().acknowledge(alert.getValueInteger("id"), ADMIN);
+      clock.advance(Duration.ofHours(2));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isEqualTo(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Team expansion uses members; channel targets enqueue an external delivery.
+    *******************************************************************************/
+   @Test
+   void testTeamAndChannelTargets() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 0, 0);
+      QRecord team = insert(AlertsTeam.TABLE_NAME, new QRecord().withValue("name", "oncall").withValue("label", "On call"));
+      for(String user : List.of("viewer-one", "viewer-two"))
+      {
+         insert(AlertsTeamMember.TABLE_NAME, new QRecord().withValue("teamId", team.getValue("id")).withValue("userId", user));
+      }
+      QRecord channel = insert(AlertsChannel.TABLE_NAME, new QRecord().withValue("name", "chat").withValue("label", "Chat")
+         .withValue("channelType", "SLACK").withValue("enabled", true));
+      step(policy, 1, 0, "TEAM", "teamId", team.getValueInteger("id"));
+      step(policy, 2, 1, "CHANNEL", "channelId", channel.getValueInteger("id"));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).extracting(row -> row.getValueString("recipientUserId")).containsExactly("viewer-one", "viewer-two");
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(where(AlertsDelivery.TABLE_NAME, AlertsDb.eq("channelId", channel.getValue("id")))).singleElement().satisfies(row ->
+      {
+         assertThat(row.getValueString("kind")).isEqualTo("ESCALATED");
+         assertThat(row.getValueString("status")).isEqualTo("PENDING");
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** A group policy applies only when a type has no override; disabled overrides stop it.
+    *******************************************************************************/
+   @Test
+   void testGroupFallbackAndDisabledPolicy() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 0, 0);
+      step(policy, 1, 0, "USER", "userId", "viewer");
+      update(AlertsGroup.TABLE_NAME, new QRecord().withValue("id", alert.getValue("groupId")).withValue("escalationPolicyId", policy.getValue("id")));
+      QRecord disabled = insert(AlertsEscalationPolicy.TABLE_NAME, new QRecord().withValue("name", "Disabled").withValue("enabled", false));
+      update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId")).withValue("escalationPolicyId", disabled.getValue("id")));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+      update(AlertsType.TABLE_NAME, new QRecord().withValue("id", alert.getValue("alertTypeId")).withValue("escalationPolicyId", null));
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Repeated raises do not postpone escalation; reopening starts a fresh cycle.
+    *******************************************************************************/
+   @Test
+   void testReopenRestartsStepsAndRepeatBudget() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 1, 1);
+      step(policy, 1, 5, "USER", "userId", "viewer");
+      clock.advance(Duration.ofMinutes(4));
+      runtime.getEngine().raise(new AlertSignal(TYPE));
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      runtime.getEngine().resolveById(alert.getValueInteger("id"), Resolution.MANUAL, ADMIN, "Done");
+      clock.advance(Duration.ofMinutes(1));
+      runtime.getEngine().reopenById(alert.getValueInteger("id"), ADMIN, "Back");
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isZero();
+      tick();
+      clock.advance(Duration.ofMinutes(4));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(2);
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(4);
+   }
+
+
+
+   /*******************************************************************************
+    ** An incompatible status never escalates, even though the alert is open.
+    *******************************************************************************/
+   @Test
+   void testOnlyIfStatus() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 0, 0);
+      QRecord step = step(policy, 1, 0, "USER", "userId", "viewer");
+      update(AlertsEscalationStep.TABLE_NAME, new QRecord().withValue("id", step.getValue("id")).withValue("onlyIfStatus", "ACKNOWLEDGED"));
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Routing failures roll back the event, level, deliveries and minute checkpoint.
+    *******************************************************************************/
+   @Test
+   void testRollbackAndRetry() throws Exception
+   {
+      QRecord alert = alert(0, AlertSeverity.ERROR);
+      QRecord policy = policy(alert, 0, 0);
+      step(policy, 1, 0, "USER", "userId", "viewer");
+      QCodeReference original = qInstance.getTable(AlertsDelivery.TABLE_NAME).getCustomizers().put(TableCustomizers.PRE_INSERT_RECORD.getRole(), new QCodeReference(RejectDelivery.class));
+      assertThatThrownBy(this::tick).isInstanceOf(QException.class);
+      assertThat(where(AlertsEvent.TABLE_NAME, AlertsDb.eq("eventType", "ESCALATED"))).isEmpty();
+      assertThat(fresh(alert).getValueInteger("escalationLevel")).isZero();
+      assertThat(fresh(alert).getValueInstant("timedNotificationCheckedAt")).isNull();
+      qInstance.getTable(AlertsDelivery.TABLE_NAME).getCustomizers().put(TableCustomizers.PRE_INSERT_RECORD.getRole(), original);
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** Suppressed attempts do not repeat within a minute or claim a successful send.
+    *******************************************************************************/
+   @Test
+   void testSuppressedReminderCheckpoint() throws Exception
+   {
+      QRecord alert = alert(1, AlertSeverity.ERROR);
+      QRecord channel = insert(AlertsChannel.TABLE_NAME, new QRecord().withValue("name", "disabled-chat").withValue("label", "Chat")
+         .withValue("channelType", "SLACK").withValue("enabled", false));
+      insert(AlertsSubscription.TABLE_NAME, new QRecord().withValue("subscriberType", "CHANNEL").withValue("channelId", channel.getValue("id"))
+         .withValue("scopeType", "ALL").withValue("events", "REMINDER").withValue("enabled", true));
+      clock.advance(Duration.ofMinutes(1));
+      tick();
+      tick();
+      assertThat(all(AlertsDelivery.TABLE_NAME)).singleElement().satisfies(row ->
+      {
+         assertThat(row.getValueString("status")).isEqualTo("SUPPRESSED");
+         assertThat(row.getValueString("suppressedReason")).isEqualTo("CHANNEL_DISABLED");
+      });
+      assertThat(fresh(alert).getValueInstant("lastNotifiedAt")).isNull();
+      assertThat(where(AlertsEvent.TABLE_NAME, AlertsDb.eq("eventType", "NOTIFIED"))).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** The scan crosses page boundaries without missing alerts or retaining all rows.
+    *******************************************************************************/
+   @Test
+   void testAlertPages() throws Exception
+   {
+      QRecord first = alert(0, AlertSeverity.ERROR);
+      for(int i = 1; i <= 100; i++)
+      {
+         runtime.getEngine().raise(new AlertSignal(TYPE).withDedupKey("page-" + i));
+      }
+      tick();
+      assertThat(all(AlertsAlert.TABLE_NAME)).hasSize(101).allSatisfy(row ->
+         assertThat(row.getValueInstant("timedNotificationCheckedAt")).isEqualTo(clock.instant()));
+      assertThat(fresh(first).getValueInteger("escalationLevel")).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** A deliberate storage failure for transaction verification.
+    *******************************************************************************/
+   public static class RejectDelivery implements TableCustomizerInterface
+   {
+      /***************************************************************************
+       ** Reject before a delivery can be stored.
+       ***************************************************************************/
+      @Override
+      public List<QRecord> preInsert(InsertInput input, List<QRecord> records, boolean isPreview) throws QException
+      {
+         throw (new QException("Deliberate test failure"));
+      }
+   }
+}

src/test/java/com/kingsrook/qbits/alerts/engine/RoutingAndDeliveryTest.java

@@ -19,6 +19,7 @@ package com.kingsrook.qbits.alerts.engine;
 
 import java.time.Duration;
 import java.util.List;
+import java.util.Set;
 import com.kingsrook.qbits.alerts.AlertsPreferences;
 import com.kingsrook.qbits.alerts.AlertsQBitConfig;
 import com.kingsrook.qbits.alerts.BaseAlertsTest;
@@ -200,6 +201,124 @@ class RoutingAndDeliveryTest extends BaseAlertsTest
 
 
 
+   /*******************************************************************************
+    ** Both host sources combine with explicit members and other subscriptions;
+    ** duplicate users receive once and invisible users remain suppressed.
+    *******************************************************************************/
+   @Test
+   void testHostTeamSources() throws Exception
+   {
+      List<String> calls = new java.util.ArrayList<>();
+      config.withMaxAudienceSize(3).withAudienceResolver((source, value) ->
+      {
+         calls.add(source + ":" + value);
+         return (Set.of(VIEWER, MANAGER, OUTSIDER));
+      });
+      for(String source : List.of("ROLE", "IDP_GROUP"))
+      {
+         Integer teamId = sourcedTeam(source);
+         runtime.asSystem(() -> AlertsDb.insert(AlertsTeamMember.TABLE_NAME, new QRecord().withValue("teamId", teamId).withValue("userId", VIEWER)));
+      }
+      subscribe("USER", VIEWER, null, null, "ALL", null, null, "ERROR");
+      runtime.getEngine().raise(new AlertSignal(TYPE).withDedupKey("source-members"));
+
+      assertThat(calls).containsExactly("ROLE:source-ROLE", "IDP_GROUP:source-IDP_GROUP");
+      assertThat(notifications()).hasSize(3);
+      assertThat(notifications()).filteredOn(n -> "SENT".equals(n.getValueString("status")))
+         .extracting(n -> n.getValueString("recipientUserId")).containsExactlyInAnyOrder(VIEWER, MANAGER);
+      assertThat(notifications()).filteredOn(n -> OUTSIDER.equals(n.getValueString("recipientUserId")))
+         .singleElement().matches(n -> "NOT_VISIBLE".equals(n.getValueString("suppressedReason")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Unconfigured sources resolve to nobody; explicit members still work.
+    *******************************************************************************/
+   @Test
+   void testMissingTeamResolver() throws Exception
+   {
+      sourcedTeam("ROLE");
+      sourcedTeam("IDP_GROUP");
+      runtime.getEngine().raise(new AlertSignal(SYSTEM).withDedupKey("no-resolver"));
+      assertThat(notifications()).isEmpty();
+
+      Integer manualTeam = sourcedTeam("MANUAL");
+      runtime.asSystem(() -> AlertsDb.insert(AlertsTeamMember.TABLE_NAME, new QRecord().withValue("teamId", manualTeam).withValue("userId", VIEWER)));
+      config.withAudienceResolver((source, value) ->
+      {
+         throw (new QException("Unavailable"));
+      });
+      runtime.getEngine().raise(new AlertSignal(SYSTEM).withDedupKey("failed-resolver"));
+      assertThat(notifications()).singleElement().matches(n -> VIEWER.equals(n.getValueString("recipientUserId")) && "SENT".equals(n.getValueString("status")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Oversized resolver results fail closed without truncating the audience.
+    *******************************************************************************/
+   @Test
+   void testTeamAudienceLimit() throws Exception
+   {
+      config.withMaxAudienceSize(1).withAudienceResolver((source, value) -> Set.of(VIEWER, MANAGER));
+      sourcedTeam("ROLE");
+      runtime.getEngine().raise(new AlertSignal(SYSTEM).withDedupKey("too-many"));
+      assertThat(notifications()).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Create a team subscription using a host-managed or manual source.
+    *******************************************************************************/
+   private Integer sourcedTeam(String source) throws QException
+   {
+      Integer teamId = runtime.asSystem(() -> AlertsDb.insert(AlertsTeam.TABLE_NAME, new QRecord().withValue("name", source).withValue("label", source)
+         .withValue("memberSource", source).withValue("memberSourceValue", "source-" + source)).getValueInteger("id"));
+      subscribe("TEAM", null, teamId, null, "ALL", null, null, "ERROR");
+      return (teamId);
+   }
+
+
+
+   /*******************************************************************************
+    ** RECORD mutes require both table and ID, including an alert whose subject
+    ** is stored directly without an active record-reference row.
+    *******************************************************************************/
+   @Test
+   void testRecordMuteMatching() throws Exception
+   {
+      runtime.asSystem(() -> AlertsDb.insert(AlertsMute.TABLE_NAME, new QRecord().withValue("name", "Record mute")
+         .withValue("scopeType", "RECORD").withValue("scopeValue", WIDGET).withValue("scopeRecordId", "123")
+         .withValue("startsAt", clock.instant()).withValue("endsAt", clock.instant().plusSeconds(60))
+         .withValue("effect", "SILENCE_NOTIFICATIONS").withValue("enabled", true)));
+      QRecord alert = runtime.getEngine().raise(new AlertSignal(TYPE).withDedupKey("subject-only"));
+      alert.setValue("subjectRecordId", "123");
+      assertThat(recordMuted(alert)).isTrue();
+      alert.setValue("subjectRecordId", "124");
+      assertThat(recordMuted(alert)).isFalse();
+      alert.setValue("subjectRecordId", "123");
+      alert.setValue("subjectTableName", "another_table");
+      assertThat(recordMuted(alert)).isFalse();
+      alert.setValue("subjectTableName", WIDGET);
+      clock.advance(Duration.ofMinutes(1));
+      assertThat(recordMuted(alert)).isFalse();
+   }
+
+
+
+   /*******************************************************************************
+    ** Match a mute as the router does under the system session.
+    *******************************************************************************/
+   private boolean recordMuted(QRecord alert) throws QException
+   {
+      return (runtime.asSystem(() -> new AlertRouter(runtime).isMuted(AlertsEngine.AlertTarget.of(alert), clock.instant(),
+         com.kingsrook.qbits.alerts.model.AlertsPossibleValues.MuteEffect.SILENCE_NOTIFICATIONS)));
+   }
+
+
+
    /*******************************************************************************
     ** Suppressions are logged with their reason.
     *******************************************************************************/

src/test/java/com/kingsrook/qbits/alerts/model/EntitiesTest.java

@@ -34,7 +34,7 @@ import static org.assertj.core.api.Assertions.assertThat;
  *******************************************************************************/
 class EntitiesTest
 {
-   static final List<Class<? extends QRecordEntity>> ENTITIES = List.of(AlertsAlert.class, AlertsChannel.class, AlertsEvent.class, AlertsGroup.class, AlertsHeartbeat.class,
+   static final List<Class<? extends QRecordEntity>> ENTITIES = List.of(AlertsEscalationPolicy.class, AlertsEscalationStep.class, AlertsAlert.class, AlertsChannel.class, AlertsEvent.class, AlertsGroup.class, AlertsHeartbeat.class,
       AlertsMonitor.class, AlertsNotification.class, AlertsNotificationRecipient.class, AlertsNotificationPreference.class, AlertsMute.class, AlertsDelivery.class, AlertsRecordRef.class, AlertsSubscription.class, AlertsTeam.class,
       AlertsTeamMember.class, AlertsType.class, AlertsUserPreference.class);
 

src/test/resources/alerts-h2.sql

@@ -1,3 +1,5 @@
+DROP TABLE IF EXISTS alerts_escalation_step;
+DROP TABLE IF EXISTS alerts_escalation_policy;
 -- H2 version of src/main/resources/db/liquibase/qbit-alerts/alerts.changelog.sql, for unit tests (generated together).
 
 DROP TABLE IF EXISTS alerts_heartbeat;
@@ -265,7 +267,8 @@ CREATE TABLE IF NOT EXISTS alerts_mute (
    name VARCHAR(200) NOT NULL,
    reason VARCHAR(2000),
    scope_type VARCHAR(20) NOT NULL,
-   scope_value VARCHAR(300),
+   scope_value VARCHAR(520),
+   scope_record_id VARCHAR(100),
    starts_at TIMESTAMP NOT NULL,
    ends_at TIMESTAMP NOT NULL,
    effect VARCHAR(30),
@@ -373,3 +376,57 @@ CREATE TABLE IF NOT EXISTS alerts_notification_preference (
    CONSTRAINT pk_alerts_notification_preference PRIMARY KEY (id),
    CONSTRAINT uq_alerts_notification_preference UNIQUE (user_id, category)
 );
+
+-- changeset qbit-alerts:21
+-- comment: Admin escalation policies and their ordered destinations.
+CREATE TABLE IF NOT EXISTS alerts_escalation_policy (
+   id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
+   name VARCHAR(100) NOT NULL,
+   description TEXT,
+   repeat_after_minutes INTEGER,
+   max_repeats INTEGER DEFAULT 0,
+   enabled BOOLEAN DEFAULT TRUE,
+   create_date TIMESTAMP,
+   modify_date TIMESTAMP,
+   CONSTRAINT pk_alerts_escalation_policy PRIMARY KEY (id),
+   CONSTRAINT uq_alerts_escalation_policy UNIQUE (name)
+);
+CREATE TABLE IF NOT EXISTS alerts_escalation_step (
+   id INTEGER GENERATED BY DEFAULT AS IDENTITY NOT NULL,
+   policy_id INTEGER NOT NULL,
+   step_order INTEGER NOT NULL,
+   after_minutes INTEGER NOT NULL,
+   target_type VARCHAR(16) NOT NULL,
+   user_id VARCHAR(100),
+   team_id INTEGER,
+   channel_id INTEGER,
+   only_if_status VARCHAR(16) DEFAULT 'OPEN' NOT NULL,
+   create_date TIMESTAMP,
+   modify_date TIMESTAMP,
+   CONSTRAINT pk_alerts_escalation_step PRIMARY KEY (id),
+   CONSTRAINT uq_alerts_escalation_step UNIQUE (policy_id, step_order),
+   CONSTRAINT ck_alerts_escalation_step_order CHECK (step_order > 0),
+   CONSTRAINT ck_alerts_escalation_step_minutes CHECK (after_minutes >= 0),
+   CONSTRAINT ck_alerts_escalation_step_target CHECK (
+      (target_type = 'USER' AND user_id IS NOT NULL AND team_id IS NULL AND channel_id IS NULL) OR
+      (target_type = 'TEAM' AND team_id IS NOT NULL AND user_id IS NULL AND channel_id IS NULL) OR
+      (target_type = 'CHANNEL' AND channel_id IS NOT NULL AND user_id IS NULL AND team_id IS NULL))
+);
+
+-- changeset qbit-alerts:22
+-- comment: Type policies override the group's policy.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_type' AND LOWER(column_name) = 'escalation_policy_id'
+ALTER TABLE alerts_type ADD COLUMN escalation_policy_id INTEGER;
+
+-- changeset qbit-alerts:23
+-- comment: A group policy is used when the type has none.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_group' AND LOWER(column_name) = 'escalation_policy_id'
+ALTER TABLE alerts_group ADD COLUMN escalation_policy_id INTEGER;
+
+-- changeset qbit-alerts:24
+-- comment: Persist the timed job checkpoint, including attempts without recipients.
+-- preconditions onFail:MARK_RAN onError:HALT
+-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.columns WHERE LOWER(table_name) = 'alerts_alert' AND LOWER(column_name) = 'timed_notification_checked_at'
+ALTER TABLE alerts_alert ADD COLUMN timed_notification_checked_at TIMESTAMP;