docs(help): marketing_agent_run.heartbeatAt (from #18)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.circleci/config.yml

@@ -1,7 +1,7 @@
 version: 2.1
 
 orbs:
-   qqq-orb: kingsrook/qqq-orb@0.6.8
+   qqq-orb: kingsrook/qqq-orb@0.6.9
 
 commands:
    configure-package-install:

AGENTS.md

@@ -46,8 +46,8 @@ Services: `approval.ApprovalService`, `execution.ExecutionService`, `measurement
 ## Dependencies
 
 - Java 21.
-- Parent `com.kingsrook:qbit-build-parent:2.1.0` (qqq 4.0.0), with architecture enforcement and strict UI API routes enabled. The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
-- Assistant provider API and OpenAI-compatible provider `0.1.0-RC.11`; no full assistant dependency. For local verification before publication only, use `-Dqbit-assistant.version=0.1.0-ARCHLOCAL-SNAPSHOT -Dqbit.guardrails.enforce=false` and inspect guardrail findings.
+- Parent `com.kingsrook:qbit-build-parent:2.1.2` (qqq 4.0.0), with architecture enforcement and strict UI API routes enabled. The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.6` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
+- Assistant provider SPI `qbit-assistant-provider-api` `0.1.0-RC.12` (package `com.kingsrook.qbits.assistant.spi`); the OpenAI-compatible provider is test scope; no full assistant dependency. No per-repository guardrail allow-lists.
 - Tests: JUnit 5, AssertJ.
 
 ## Release

CHANGELOG.md

@@ -5,12 +5,15 @@ All notable changes to this project are documented here. The format follows Keep
 ## [0.1.0] - Unreleased
 
 ### Changed
+- Make the agent limits distributed (#4): `maxConcurrentAgentRuns` counts Running `marketing_agent_run` rows in the database (a new `heartbeatAt` column, updated on each model call, lets runs whose server stopped be marked Failed first), and `maxModelCallsPerMinute` counts model-call steps in the last 60 seconds and makes a run wait rather than fail. The in-memory semaphore and rate window are gone; the settings and Marketing Policy override keep their names and defaults (the policy `maxConcurrentAgentRuns` override now applies). Hosts add `heartbeat_at` to `marketing_agent_run`. The count-then-insert window is still racy; a conditional-update claim (qqq#1178) is a follow-up.
+- Move to the assistant provider SPI `0.1.0-RC.12` (package `com.kingsrook.qbits.assistant.spi`) and `qbit-build-parent` `2.1.2`, and drop the per-repository guardrail allow-lists. `qbit-assistant-provider-openai` becomes a test dependency; hosts that register it add it themselves.
 - Switch the optional qbit-email integration to the published `com.kingsrook.qbits.email.api` package in `0.1.0-RC.3` (fixes #13). Preserve MARKETING messages, idempotency keys, execution links and outcome mapping; use typed readiness failures so missing providers or brand postal addresses remain send-time rejections. Upgrade `qbit-build-parent` to `2.1.1` and remove all per-repository guardrail allow-lists.
 - Resolve architecture finding F5 (#9): depend on `qbit-assistant-provider-api` and `qbit-assistant-provider-openai` `0.1.0-RC.11`, with no full assistant dependency, including in tests.
 - Replace the local OpenAI-compatible implementation with the published provider and document `ProviderSettings`, environment-key resolution and provider-name migration. Agent tests use a local scripted double.
 - Upgrade QQQ to `4.1.0-RC.6`; enable architecture enforcement and strict UI API routes.
 
 ### Added
+- Help bundle `help/qbit-marketing.yaml` (346 entries) covering the app, tables, fields, actions, widgets and possible-value sources, with `HelpBundleCoverageTest` as the coverage gate until `HelpBundleAssertions` ships in core (qqq#1198). The producer registers it once that API exists.
 - Product-contingent possible values for editable campaign, claim, finding, metric and outcome-event references.
 - Repository scaffold: build, CI and the repository standard files.
 - **Model:** 18 standard `marketing_*` tables, 21 enum possible-value sources, joins and child lists. The PostgreSQL reference DDL (`docs/ddl/postgres.sql`) and an H2 twin are generated from one spec (`scripts/generate-model.py`).
@@ -51,5 +54,6 @@ All notable changes to this project are documented here. The format follows Keep
 - **Docs:** README, DESIGN, OPERATIONS and STARTER-DATA.
 
 ### Changed
+- Move to the assistant provider SPI `0.1.0-RC.12` (package `com.kingsrook.qbits.assistant.spi`) and `qbit-build-parent` `2.1.2`, and drop the per-repository guardrail allow-lists. `qbit-assistant-provider-openai` becomes a test dependency; hosts that register it add it themselves.
 - **Email:** new `QbitEmailTransport` sends campaign email through qbit-email (one MARKETING message per recipient, unsubscribe and suppression enforced there, idempotent, linked to the execution). It is selected automatically for the `qqq_email` channel when qbit-email is produced and the host registered no such transport; `QqqMessagingEmailTransport` remains the fallback. qbit-email is an optional dependency; QQQ moves to `4.1.0-RC.5`.
 

README.md

@@ -5,9 +5,9 @@
 
 > Marketing for QQQ applications: products and their verified evidence, campaigns, content with revision history, exact human approval, controlled execution, honest measurement, and a marketing agent that prepares work but never publishes.
 
-**Status:** 0.1.0, not yet published.
-- Built against QQQ `4.1.0-RC.6` with `qbit-build-parent` `2.1.1`.
-- Hosted in QRun-IO/Voyage next to qbit-crm, on a local branch.
+**Status:** 0.1.0 release candidates on Maven Central (`com.kingsrook.qbits:qbit-marketing`, latest `0.1.0-RC.2`); 0.1.0 GA is not yet released.
+- Built against QQQ `4.1.0-RC.6` with `qbit-build-parent` `2.1.2` and the assistant provider SPI `0.1.0-RC.12`.
+- Hosted in QRun-IO/Voyage next to qbit-crm (QRun-IO/Voyage#113).
 - The real integrations are implemented but unconfigured. See [what is configured](#what-is-implemented-what-is-not-configured).
 
 It lives inside the host QQQ application: its navigation, authentication, permissions, API, record screens and widgets. It is not a separate marketing app.
@@ -25,7 +25,7 @@ It lives inside the host QQQ application: its navigation, authentication, permis
 | **Measurement** | Delivery, awareness, engagement, activation, repeat use and qualified demand are reported separately. Each metric shows raw events and unique actors. "Not measured" is never shown as 0. Test data is kept apart. |
 | **CRM** | Uses the host CRM's own contacts, organizations, consent, suppression, lists, form submissions and activities through a narrow gateway. A qbit-crm adapter is included. Nothing is copied, and nobody becomes a lead without the campaign's explicit rule. |
 | **Agent** | Plans campaigns, drafts and revises content, flags unsupported claims, stale sources and readiness gaps, and analyzes results to recommend the next experiment. It uses narrow typed tools, runs as the requesting user, sees data only as untrusted input, and sees no CRM personal data. Every run is recorded with its steps, provider, model, tokens and cost. It can create drafts and findings; it cannot approve, schedule or send. |
-| **Models** | Interchangeable providers through `qbit-assistant-provider-api`'s `AssistantModelProvider`: OpenAI-compatible endpoints through `qbit-assistant-provider-openai` (included), or a host-supplied provider such as Claude. Tests use a local scripted double. With no provider configured, the agent says so and everything else works. Installing marketing does not install the assistant runtime, tables or routes. |
+| **Models** | Interchangeable providers through `qbit-assistant-provider-api`'s `AssistantModelProvider`: OpenAI-compatible endpoints through `qbit-assistant-provider-openai` (added by the host), or a host-supplied provider such as Claude. Tests use a local scripted double. With no provider configured, the agent says so and everything else works. Installing marketing does not install the assistant runtime, tables or routes. |
 
 ## Install
 
@@ -38,7 +38,7 @@ It lives inside the host QQQ application: its navigation, authentication, permis
 ```
 
 ```java
-import com.kingsrook.qbits.assistant.provider.ProviderSettings;
+import com.kingsrook.qbits.assistant.spi.ProviderSettings;
 import com.kingsrook.qbits.assistant.provider.openai.OpenAiCompatibleModelProvider;
 import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;
 
@@ -66,8 +66,8 @@ Then:
 
 [`docs/OPERATIONS.md`](docs/OPERATIONS.md) covers every setting, the run-books and the limits.
 
-The provider SPI, pricing and OpenAI-compatible implementation are pinned to `0.1.0-RC.11`.
-Replace imports of marketing's former `agent.provider.OpenAiCompatibleModelProvider` with the published class above and replace its builder with `ProviderSettings`.
+Marketing depends only on the provider SPI (`qbit-assistant-provider-api` `0.1.0-RC.12`, package `com.kingsrook.qbits.assistant.spi`). To use the OpenAI-compatible provider, the host adds `com.kingsrook.qbits:qbit-assistant-provider-openai:0.1.0-RC.12` itself.
+Replace imports of marketing's former `agent.provider.OpenAiCompatibleModelProvider` with the published class above and replace its builder with `ProviderSettings`. Hosts upgrading from RC.11 change `com.kingsrook.qbits.assistant.provider.*` and `…assistant.engine.ModelPricing` imports to `com.kingsrook.qbits.assistant.spi.*`.
 The host resolves the API key reference; `ProviderSettings.withApiKeyEnvName(...)` alone does not resolve it.
 Set any existing Marketing Policy provider override to `openai-compatible` as well. The base URL, key environment variable name and model remain host-configurable; omit the key for a local endpoint that needs none.
 
@@ -98,6 +98,12 @@ Set any existing Marketing Policy provider override to `openai-compatible` as we
   - Content: Proposed Publication.
 - **Actions:** every action is a streamed-ETL record action (record input, standard Review and Result summary lines) in the table's or record's Actions menu. None are nav items.
 
+## Help
+
+The help text ships with the code it describes, as a QQQ help bundle at `help/qbit-marketing.yaml` (a classpath resource, layer `QBIT`). It has 346 entries: the Marketing app, every `marketing_*` table and each of its non-trivial fields, every record and table action, every widget (the dashboard widgets and the child lists) and every possible-value source. Each entry has a one-line summary for the tooltip and a few sentences of plain help. Labels, maximum lengths and enum values are `{{...}}` references, so the help does not repeat what the metadata already says.
+
+The bundle is loaded with `HelpBundles.register(qInstance, "help/qbit-marketing.yaml")` once qqq#1198 ships. The producer will register it then; until that core API exists, nothing registers it. `HelpBundleCoverageTest` checks that the bundle is complete, that every key and reference resolves, and that summaries stay short and plain.
+
 ## Email through qbit-email
 
 When the optional [qbit-email](https://github.com/QRun-IO/qbit-email) qbit is produced in the same instance, email channels named `qqq_email` (the starter-data channel) send through it automatically, unless the host registered its own transport under that name. `QqqMessagingEmailTransport` stays the fallback; without qbit-email nothing changes.
@@ -121,7 +127,7 @@ When the optional [qbit-email](https://github.com/QRun-IO/qbit-email) qbit is pr
 | CRM gateway and qbit-crm adapter, explicit lead rules | |
 | Persistence across restart (H2 file, and PostgreSQL in the Voyage host) | |
 
-Scale gaps (single worker per JVM, in-memory rate limits) are listed in [`docs/OPERATIONS.md`](docs/OPERATIONS.md#scale-and-limits).
+Scale gaps (single execution worker per JVM; agent limits are database-counted but still racy across replicas) are listed in [`docs/OPERATIONS.md`](docs/OPERATIONS.md#scale-and-limits).
 
 ## Build
 
@@ -131,15 +137,7 @@ Scale gaps (single worker per JVM, in-memory rate limits) are listed in [`docs/O
 
 Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM. The entities, enums and DDL are generated from one spec: `python -I scripts/generate-model.py .` (commit the output).
 
-Architecture guardrails and strict UI API route checks are enabled. The artifact allow-list preserves the parent's defaults and adds only `qbit-assistant-provider-openai`; the package allow-list also permits the extracted provider SPI and pricing. Parent 2.1.0 matches package prefixes using `prefix + "."`, so the pricing allowance must be `com.kingsrook.qbits.assistant.engine`, not the class name `…engine.ModelPricing`. The API artifact contains only `ModelPricing` in that package; the full assistant artifact remains prohibited.
-
-Until provider `0.1.0-RC.11` is available on Central, verify against the locally installed assistant develop build only with:
-
-```bash
-./mvnw -Dqbit-assistant.version=0.1.0-ARCHLOCAL-SNAPSHOT -Dqbit.guardrails.enforce=false verify
-```
-
-This runs the guardrails in report mode so G7 can report the local SNAPSHOT dependencies. Keep the release version and enforcement enabled in the POM.
+Architecture guardrails and strict UI API route checks are enabled with the parent's default allow-lists only (no per-repository lists): `qbit-build-parent` 2.1.2 allows the `*-api` artifacts and the `com.kingsrook.qbits.assistant.spi` and `com.kingsrook.qbits.email.api` packages. `qbit-assistant-provider-openai` is a test dependency only, and the full assistant artifact is never a dependency.
 
 ## License
 

docs/DESIGN.md

@@ -18,7 +18,7 @@ The work is truthful and auditable: facts come from sources with dates, people a
 
 | From | Reused for |
 |---|---|
-| **qbit-assistant** (0.1.0-RC.2) | The model-provider SPI: `AssistantModelProvider`, `ModelRequest` / `ModelResponse` / `ModelPart`, `StreamHandle`, `ModelProviderException` with its retryable kinds, and `ModelUsage`. Also the Claude provider (`qbit-assistant-provider-anthropic`), `ScriptedModelProvider` for tests and demos, and `ModelPricing` for cost. Marketing adds an OpenAI-compatible provider that implements the same SPI. There is no second provider abstraction. |
+| **qbit-assistant** (0.1.0-RC.2) | The model-provider SPI: `AssistantModelProvider`, `ModelRequest` / `ModelResponse` / `ModelPart`, `StreamHandle`, `ModelProviderException` with its retryable kinds, and `ModelUsage`. Since RC.12 it lives in `com.kingsrook.qbits.assistant.spi` (`qbit-assistant-provider-api`), together with `ModelPricing` for cost; hosts register a provider such as `qbit-assistant-provider-anthropic` or `qbit-assistant-provider-openai`. There is no second provider abstraction. |
 | **QQQ AI tool kernel** (qqq#1078, core) | The tool contract: `AbstractAiTool` / `AiToolKind` (READ, WRITE_PROPOSAL), `AiToolOutcome`, the `qqq_data` untrusted-data envelope (`AiDataEnvelope`), secret redaction (`AiRedactor`), strict input keys, and running as the user's session. The RC.6 baseline includes `AiToolRegistry` (QRun-IO/qqq#1096); Marketing contributes its read tools there and runs its own agent tools with the same contract. |
 | **QQQ core permission enforcement** (qqq#1061) | Agent tools read with `PermissionEnforcement.ENFORCE`. Service entry points check their step's permission, then run as trusted internal work in an authorized scope, as QQQ does for calls nested in a core action. Record security locks always apply. |
 | **qbit-assistant / sql-workbench metadata builders** | Explicit builder-produced tables (not annotation scanning), snake_case `marketing_*` names, QQQ standard permission suffixes, `DenyBehavior.HIDDEN`, read-only system tables, record locks, reference DDL in `docs/ddl`, `tableMetaDataCustomizer` and `inferBackendNames`. |

docs/OPERATIONS.md

@@ -12,7 +12,7 @@
 | `providerName` / `model` / `effort` | none | Which provider and model the agent uses. The Marketing Policy can override them. |
 | `maxOutputTokens` | 8000 | Per model call. |
 | `maxToolCallsPerRun` / `maxModelCallsPerRun` / `maxRunSeconds` | 24 / 16 / 300 | Limits of one agent run. |
-| `maxConcurrentAgentRuns` / `maxModelCallsPerMinute` | 2 / 30 | Agent concurrency and rate (per JVM). |
+| `maxConcurrentAgentRuns` / `maxModelCallsPerMinute` | 2 / 30 | Agent concurrency and rate, counted in the database so they hold across replicas. |
 | `maxRowsToModel` | 50 | Rows per read tool. |
 | `pricing` | Anthropic list prices (qbit-assistant) | USD per million tokens, by model id. Models without a price show no cost. Prices are never invented. |
 | `registerKernelTools` | `true` | Contribute the read tools to the QQQ AI tool kernel (when QQQ has `AiToolRegistry`). |
@@ -95,7 +95,7 @@ Fix the cause. If the content changed, it needs a new approval. Then schedule ag
 These are real limitations of 0.1.0. Nothing here is claimed to be production-scale.
 
 - **Single execution worker per JVM.** The worker takes an in-process lock and claims a row by re-reading its status before marking it Running. Two replicas could both claim the same due execution in the instant between read and update, and the idempotency key is then the only protection (it fully protects the test transport, but not email). **Run Marketing's worker on one replica**, or add a database-level claim (`UPDATE ... WHERE status = SCHEDULED`, or `SELECT ... FOR UPDATE SKIP LOCKED`) before scaling out.
-- **Rate limits and agent concurrency are per JVM** (in memory). The channel hourly rate is computed from the database, so it holds across replicas, but only up to the race above.
+- **Agent concurrency and the model-call rate are counted in the database**, so they hold across replicas. Concurrency counts `marketing_agent_run` rows in Running (all products); a run records a heartbeat on each model call, and a Running run silent for `maxRunSeconds` plus 60 seconds is marked Failed ("Stopped responding; the server may have restarted.") before the count. The rate counts model-call steps in the last 60 seconds; at the limit a run waits (bounded by `maxRunSeconds`, and Cancel Run still works) instead of failing. The channel hourly rate is computed the same way. Counting and then inserting is still racy: two replicas can both see room, so a limit can be overshot briefly. A conditional-update claim (`ConditionalUpdateAction`, qqq#1178, QQQ 4.1.0-RC.7) will close that window. Hosts add the new `heartbeat_at` column to `marketing_agent_run` (see `docs/ddl/postgres.sql`).
 - **Agent runs are synchronous** inside the record action, bounded by `maxRunSeconds`. A restart interrupts them: they are marked Failed and can be run again. There is no queue or resume.
 - **The worker queries due executions page by page** (`workerBatchSize` per run) and scans channel attempts for the hourly rate. That suits hundreds of executions per hour, not bulk email. Bulk outreach is deliberately not supported.
 - **Measurement is computed in memory** from a campaign's events on every view. Add a rollup table before campaigns have hundreds of thousands of events.

docs/ddl/postgres.sql

@@ -321,6 +321,7 @@ CREATE TABLE IF NOT EXISTS marketing_agent_run (
    model VARCHAR(100),
    requested_by_user_id VARCHAR(100),
    started_at TIMESTAMP WITHOUT TIME ZONE,
+   heartbeat_at TIMESTAMP WITHOUT TIME ZONE,
    finished_at TIMESTAMP WITHOUT TIME ZONE,
    model_calls INTEGER,
    tool_calls INTEGER,

pom.xml

@@ -22,15 +22,10 @@
       <maven.compiler.source>21</maven.compiler.source>
       <maven.compiler.target>21</maven.compiler.target>
       <jacoco.version>0.8.15</jacoco.version>
-      <!-- Share the provider SPI and pricing without installing the assistant runtime. -->
-      <qbit-assistant.version>0.1.0-RC.11</qbit-assistant.version>
+      <!-- The provider SPI (com.kingsrook.qbits.assistant.spi) only: no assistant runtime. -->
+      <qbit-assistant.version>0.1.0-RC.12</qbit-assistant.version>
       <qbit.guardrails.enforce>true</qbit.guardrails.enforce>
       <qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
-      <!-- qbit-email from Maven Central -->
-      <!-- Assistant provider SPI only (not email): these replace the parent lists, so they repeat -->
-      <!-- the parent defaults (object storage, qbit-email's api, *-api) plus the assistant entries. -->
-      <qbit.guardrails.allowedApiPackages>com.kingsrook.qbits.objectstorage.api,com.kingsrook.qbits.email.api,com.kingsrook.qbits.assistant.provider,com.kingsrook.qbits.assistant.engine</qbit.guardrails.allowedApiPackages>
-      <qbit.guardrails.allowedQbitArtifacts>com.kingsrook.qbits:qbit-object-storage,com.kingsrook.qbits:qbit-email,com.kingsrook.qbits:*-api,com.kingsrook.qbits:qbit-assistant-provider-openai</qbit.guardrails.allowedQbitArtifacts>
       <qbit-email.version>0.1.0-RC.3</qbit-email.version>
       <h2.version>2.3.232</h2.version>
    </properties>
@@ -38,7 +33,7 @@
    <parent>
       <groupId>com.kingsrook</groupId>
       <artifactId>qbit-build-parent</artifactId>
-      <version>2.1.1</version>
+      <version>2.1.2</version>
    </parent>
 
    <groupId>com.kingsrook.qbits</groupId>
@@ -92,6 +87,8 @@
          <groupId>com.kingsrook.qbits</groupId>
          <artifactId>qbit-assistant-provider-openai</artifactId>
          <version>${qbit-assistant.version}</version>
+         <!-- tests only: hosts add the provider they register (README, Install) -->
+         <scope>test</scope>
       </dependency>
 
       <!-- optional: email through qbit-email when it is produced in the host instance (QbitEmailTransport) -->

scripts/generate-model.py

@@ -171,7 +171,7 @@ TABLES = [
    ("MarketingAgentRun", "marketing_agent_run", "A run of the marketing agent: task, status, provider, model usage and outputs.", [
       F("productId", "Fk:MarketingProduct"), F("campaignId", "Fk:MarketingCampaign"), F("contentId", "Fk:MarketingContent"),
       F("task", "Enum:AgentTask", required=True), F("status", "Enum:AgentRunStatus", required=True), F("instructions", "Text"),
-      F("providerName", "String:64"), F("model", "String:100"), F("requestedByUserId", "String:100"), F("startedAt", "Instant"),
+      F("providerName", "String:64"), F("model", "String:100"), F("requestedByUserId", "String:100"), F("startedAt", "Instant"), F("heartbeatAt", "Instant"),
       F("finishedAt", "Instant"), F("modelCalls", "Integer"), F("toolCalls", "Integer"), F("inputTokens", "Long"),
       F("outputTokens", "Long"), F("cacheReadTokens", "Long"), F("costMicros", "Long"), F("draftsCreated", "Integer"),
       F("findingsCreated", "Integer"), F("summary", "Text"), F("outputJson", "Text"), F("errorMessage", "Text")]),

src/main/java/com/kingsrook/qbits/marketing/MarketingQBitConfig.java

@@ -21,8 +21,8 @@ import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
-import com.kingsrook.qbits.assistant.engine.ModelPricing;
-import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.spi.ModelPricing;
+import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
 import com.kingsrook.qbits.marketing.crm.MarketingCrmGateway;
 import com.kingsrook.qbits.marketing.transport.MarketingTransport;
 import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;

src/main/java/com/kingsrook/qbits/marketing/MarketingRuntime.java

@@ -18,16 +18,13 @@ package com.kingsrook.qbits.marketing;
 
 
 import java.net.InetAddress;
-import java.util.ArrayDeque;
 import java.util.ArrayList;
-import java.util.Deque;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.UUID;
-import java.util.concurrent.Semaphore;
 import java.util.concurrent.locks.ReentrantLock;
-import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
 import com.kingsrook.qbits.marketing.crm.MarketingCrmGateway;
 import com.kingsrook.qbits.marketing.email.QbitEmailTransport;
 import com.kingsrook.qbits.marketing.model.MarketingPolicy;
@@ -50,14 +47,15 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
 
 /*******************************************************************************
  ** The live parts of the Marketing QBit for one instance: the transports, the
- ** model providers, the CRM gateway, and the in-process limits (agent
- ** concurrency, model-call rate, the single execution worker).
+ ** model providers, the CRM gateway, and the single execution worker's
+ ** in-process lock. (Agent concurrency and the model-call rate are counted in
+ ** the database by MarketingAgentService.)
  **
  ** Registered as supplemental instance meta-data (not published to
  ** frontends), so process steps and widgets find it from QContext.
  **
- ** Scale note: the limits here are per JVM. One replica is the supported shape
- ** today; see docs/OPERATIONS.md ("Scale and limits").
+ ** Scale note: the execution worker is per JVM. One worker replica is the
+ ** supported shape today; see docs/OPERATIONS.md ("Scale and limits").
  *******************************************************************************/
 public class MarketingRuntime implements QSupplementalInstanceMetaData
 {
@@ -69,9 +67,7 @@ public class MarketingRuntime implements QSupplementalInstanceMetaData
 
    private final MarketingQBitConfig             config;
    private final Map<String, MarketingTransport> transports = new LinkedHashMap<>();
-   private final Semaphore                       agentPermits;
    private final ReentrantLock                   workerLock = new ReentrantLock();
-   private final Deque<Long>                     modelCallTimes = new ArrayDeque<>();
    private final String                          workerName;
    private volatile MarketingTransport             qbitEmailTransport;
 
@@ -91,7 +87,6 @@ public class MarketingRuntime implements QSupplementalInstanceMetaData
       {
          transports.put(transport.getName(), transport);
       }
-      this.agentPermits = new Semaphore(config.getMaxConcurrentAgentRuns());
       this.workerName = hostName() + ":" + UUID.randomUUID().toString().substring(0, 8);
    }
 
@@ -328,47 +323,6 @@ public class MarketingRuntime implements QSupplementalInstanceMetaData
 
 
 
-   /*******************************************************************************
-    ** Try to start an agent run (bounded concurrency). Release with
-    ** releaseAgentPermit.
-    *******************************************************************************/
-   public boolean tryAcquireAgentPermit()
-   {
-      return (agentPermits.tryAcquire());
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   public void releaseAgentPermit()
-   {
-      agentPermits.release();
-   }
-
-
-
-   /*******************************************************************************
-    ** Reserve a model call within the per-minute rate (false = over the limit).
-    *******************************************************************************/
-   public synchronized boolean tryReserveModelCall()
-   {
-      long now = System.currentTimeMillis();
-      while(!modelCallTimes.isEmpty() && modelCallTimes.peekFirst() < now - 60_000)
-      {
-         modelCallTimes.pollFirst();
-      }
-      if(modelCallTimes.size() >= config.getMaxModelCallsPerMinute())
-      {
-         return (false);
-      }
-      modelCallTimes.addLast(now);
-      return (true);
-   }
-
-
-
    /*******************************************************************************
     ** The single execution worker's lock in this JVM.
     *******************************************************************************/

src/main/java/com/kingsrook/qbits/marketing/agent/MarketingAgentService.java

@@ -25,18 +25,18 @@ import java.util.ArrayList;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
-import com.kingsrook.qbits.assistant.engine.ModelPricing;
-import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
-import com.kingsrook.qbits.assistant.provider.CacheHint;
-import com.kingsrook.qbits.assistant.provider.ModelEventListener;
-import com.kingsrook.qbits.assistant.provider.ModelMessage;
-import com.kingsrook.qbits.assistant.provider.ModelPart;
-import com.kingsrook.qbits.assistant.provider.ModelProviderException;
-import com.kingsrook.qbits.assistant.provider.ModelRequest;
-import com.kingsrook.qbits.assistant.provider.ModelResponse;
-import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
-import com.kingsrook.qbits.assistant.provider.ModelUsage;
-import com.kingsrook.qbits.assistant.provider.SystemBlock;
+import com.kingsrook.qbits.assistant.spi.ModelPricing;
+import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.spi.CacheHint;
+import com.kingsrook.qbits.assistant.spi.ModelEventListener;
+import com.kingsrook.qbits.assistant.spi.ModelMessage;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelRequest;
+import com.kingsrook.qbits.assistant.spi.ModelResponse;
+import com.kingsrook.qbits.assistant.spi.ModelToolDefinition;
+import com.kingsrook.qbits.assistant.spi.ModelUsage;
+import com.kingsrook.qbits.assistant.spi.SystemBlock;
 import com.kingsrook.qbits.marketing.MarketingQBitConfig;
 import com.kingsrook.qbits.marketing.MarketingRuntime;
 import com.kingsrook.qbits.marketing.MarketingStore;
@@ -78,11 +78,15 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
  ** A run:
  ** 1. checks the policy (kill switch), the provider and model, the daily
  **    token and monthly cost budgets, and the concurrency limit - failing the
- **    run with a plain reason when any is missing (missing-provider behavior);
+ **    run with a plain reason when any is missing (missing-provider behavior).
+ **    The concurrency limit counts RUNNING runs in the database (after failing
+ **    those whose heartbeat stopped), so it holds across replicas;
  ** 2. runs the deterministic readiness checks (a CHECK step);
  ** 3. loops model call -> tool calls, bounded by model calls, tool calls,
- **    wall-clock time and the per-minute model-call rate; it re-reads its own
- **    status before every model call so Cancel Run stops it;
+ **    wall-clock time and the per-minute model-call rate (counted from the
+ **    database; at the limit the run waits, it does not fail); it re-reads its
+ **    own status before every model call so Cancel Run stops it, and records a
+ **    heartbeat on each model call;
  ** 4. records every model call (tokens, stop reason) and tool call (input,
  **    outcome, enveloped result), the usage and cost, the drafts and findings
  **    it created, and its final summary.
@@ -98,8 +102,29 @@ public class MarketingAgentService
 
    static final int MAX_STEP_JSON = 20_000;
 
+   /////////////////////////////////////////////////////////////////////////////
+   // a RUNNING run silent for maxRunSeconds plus this long is not alive      //
+   /////////////////////////////////////////////////////////////////////////////
+   public static final int STALE_GRACE_SECONDS = 60;
+
    private final AiRedactor redactor = new AiRedactor(List.of());
 
+   private Duration modelCallWindow = Duration.ofSeconds(60);
+   private Duration modelCallPoll   = Duration.ofSeconds(1);
+
+
+
+   /*******************************************************************************
+    ** Change the per-minute window and how often a waiting run re-checks it
+    ** (the defaults are 60 seconds and 1 second; tests use short ones).
+    *******************************************************************************/
+   public MarketingAgentService withModelCallWindow(Duration window, Duration poll)
+   {
+      this.modelCallWindow = window;
+      this.modelCallPoll = poll;
+      return (this);
+   }
+
 
 
    /*******************************************************************************
@@ -158,19 +183,7 @@ public class MarketingAgentService
          return (finish(run, AgentRunStatus.FAILED, null, refusal, null, new MarketingToolContext(run, 1), new ModelUsage(), 0, 0));
       }
 
-      if(!runtime.tryAcquireAgentPermit())
-      {
-         return (finish(run, AgentRunStatus.FAILED, null, "Too many agent runs at once (limit " + runtime.getConfig().getMaxConcurrentAgentRuns() + "); start it again shortly.",
-            null, new MarketingToolContext(run, 1), new ModelUsage(), 0, 0));
-      }
-      try
-      {
-         return (execute(runtime, policy, provider, model, run, task));
-      }
-      finally
-      {
-         runtime.releaseAgentPermit();
-      }
+      return (execute(runtime, policy, provider, model, run, task));
    }
 
 
@@ -213,6 +226,101 @@ public class MarketingAgentService
             return ("This month's agent cost budget is used up.");
          }
       }
+
+      /////////////////////////////////////////////////////////////////////////////
+      // concurrency is counted in the database, across every replica and        //
+      // product. NOTE: count-then-mark-RUNNING is still racy - two replicas can //
+      // both see room and both start. A conditional-update claim                //
+      // (ConditionalUpdateAction, qqq#1178, QQQ 4.1.0-RC.7) will close that     //
+      // window as a follow-up.                                                  //
+      /////////////////////////////////////////////////////////////////////////////
+      failStaleRuns(runtime.getConfig());
+      int limit = policy.getMaxConcurrentAgentRuns() != null ? policy.getMaxConcurrentAgentRuns() : runtime.getConfig().getMaxConcurrentAgentRuns();
+      if(runningRuns() >= limit)
+      {
+         return ("Too many agent runs at once (limit " + limit + "); start it again shortly.");
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    ** RUNNING runs (all products) whose heartbeat - or start, for a run that
+    ** never beat - is older than maxRunSeconds plus a grace period are failed:
+    ** their server stopped. Returns how many.
+    *******************************************************************************/
+   int failStaleRuns(MarketingQBitConfig config) throws QException
+   {
+      Instant       cutoff  = Instant.now().minusSeconds((long) config.getMaxRunSeconds() + STALE_GRACE_SECONDS);
+      List<QRecord> running = MarketingStore.asSystem(() -> MarketingStore.query(MarketingAgentRun.TABLE_NAME,
+         new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.EQUALS, AgentRunStatus.RUNNING.getId()))));
+
+      int count = 0;
+      for(QRecord record : running)
+      {
+         Instant last = record.getValueInstant("heartbeatAt");
+         last = last != null ? last : record.getValueInstant("startedAt");
+         last = last != null ? last : record.getValueInstant("createDate");
+         if(last != null && last.isBefore(cutoff))
+         {
+            MarketingStore.updateInternal(MarketingAgentRun.TABLE_NAME, new QRecord().withValue("id", record.getValue("id"))
+               .withValue("status", AgentRunStatus.FAILED.getId()).withValue("finishedAt", Instant.now())
+               .withValue("errorMessage", "Stopped responding; the server may have restarted."));
+            count++;
+         }
+      }
+      return (count);
+   }
+
+
+
+   /*******************************************************************************
+    ** RUNNING runs across all products, counted as the system user.
+    *******************************************************************************/
+   static int runningRuns() throws QException
+   {
+      return (MarketingStore.asSystem(() -> MarketingStore.count(MarketingAgentRun.TABLE_NAME,
+         new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.EQUALS, AgentRunStatus.RUNNING.getId())))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Model calls (steps of the model-call kind, all runs, system user) recorded
+    ** since the instant.
+    *******************************************************************************/
+   static int modelCallsSince(Instant since) throws QException
+   {
+      return (MarketingStore.asSystem(() -> MarketingStore.count(MarketingAgentStep.TABLE_NAME, new QQueryFilter(
+         new QFilterCriteria("kind", QCriteriaOperator.EQUALS, AgentStepKind.MODEL_CALL.getId()),
+         new QFilterCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, since)))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Wait until the per-minute model-call limit has room. Returns null to go
+    ** ahead, CANCELLED if a person cancelled the run while waiting, or FAILED
+    ** if the run's time ran out first.
+    **
+    ** NOTE: counting then calling is racy across replicas (see preflight); the
+    ** conditional-update claim follow-up (qqq#1178) will close it.
+    *******************************************************************************/
+   private AgentRunStatus awaitModelCallSlot(MarketingAgentRun run, int limit, Instant deadline) throws QException, InterruptedException
+   {
+      while(modelCallsSince(Instant.now().minus(modelCallWindow)) >= limit)
+      {
+         if(isCancelled(run.getId()))
+         {
+            return (AgentRunStatus.CANCELLED);
+         }
+         if(Instant.now().isAfter(deadline))
+         {
+            return (AgentRunStatus.FAILED);
+         }
+         Thread.sleep(modelCallPoll.toMillis());
+      }
       return (null);
    }
 
@@ -229,7 +337,7 @@ public class MarketingAgentService
       int                 maxTools = policy != null && policy.getMaxToolCallsPerRun() != null ? policy.getMaxToolCallsPerRun() : config.getMaxToolCallsPerRun();
 
       MarketingStore.updateInternal(MarketingAgentRun.TABLE_NAME, new QRecord().withValue("id", run.getId())
-         .withValue("status", AgentRunStatus.RUNNING.getId()).withValue("startedAt", Instant.now()));
+         .withValue("status", AgentRunStatus.RUNNING.getId()).withValue("startedAt", Instant.now()).withValue("heartbeatAt", Instant.now()));
 
       MarketingToolContext context = new MarketingToolContext(run, config.getMaxRowsToModel());
       ModelUsage           usage   = new ModelUsage();
@@ -278,11 +386,21 @@ public class MarketingAgentService
             {
                return (finish(run, AgentRunStatus.FAILED, null, "Stopped: the run reached its limit of " + config.getMaxModelCallsPerRun() + " model calls.", null, context, usage, modelCalls, toolCalls));
             }
-            if(!runtime.tryReserveModelCall())
+
+            /////////////////////////////////////////////////////////////////
+            // at the per-minute limit, wait (bounded by the run's time)   //
+            /////////////////////////////////////////////////////////////////
+            AgentRunStatus blocked = awaitModelCallSlot(run, config.getMaxModelCallsPerMinute(), deadline);
+            if(blocked == AgentRunStatus.CANCELLED)
             {
-               return (finish(run, AgentRunStatus.FAILED, null, "Stopped: the model-call rate limit (" + config.getMaxModelCallsPerMinute() + " per minute) was reached.", null, context, usage, modelCalls,
-                  toolCalls));
+               return (finish(run, AgentRunStatus.CANCELLED, null, "Cancelled by a person.", null, context, usage, modelCalls, toolCalls));
             }
+            if(blocked == AgentRunStatus.FAILED)
+            {
+               return (finish(run, AgentRunStatus.FAILED, null, "Stopped: the run exceeded " + config.getMaxRunSeconds() + " seconds, waiting for the model-call rate limit ("
+                  + config.getMaxModelCallsPerMinute() + " per minute).", null, context, usage, modelCalls, toolCalls));
+            }
+            MarketingStore.updateInternal(MarketingAgentRun.TABLE_NAME, new QRecord().withValue("id", run.getId()).withValue("heartbeatAt", Instant.now()));
 
             ModelRequest request = new ModelRequest()
                .withModel(model)

src/main/java/com/kingsrook/qbits/marketing/model/MarketingAgentRun.java

@@ -68,6 +68,9 @@ public class MarketingAgentRun extends QRecordEntity
    @QField()
    private Instant startedAt;
 
+   @QField()
+   private Instant heartbeatAt;
+
    @QField()
    private Instant finishedAt;
 
@@ -460,6 +463,36 @@ public class MarketingAgentRun extends QRecordEntity
    }
 
 
+   /*******************************************************************************
+    ** Getter for heartbeatAt
+    *******************************************************************************/
+   public Instant getHeartbeatAt()
+   {
+      return (this.heartbeatAt);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for heartbeatAt
+    *******************************************************************************/
+   public void setHeartbeatAt(Instant heartbeatAt)
+   {
+      this.heartbeatAt = heartbeatAt;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for heartbeatAt
+    *******************************************************************************/
+   public MarketingAgentRun withHeartbeatAt(Instant heartbeatAt)
+   {
+      this.heartbeatAt = heartbeatAt;
+      return (this);
+   }
+
+
    /*******************************************************************************
     ** Getter for finishedAt
     *******************************************************************************/

src/test/java/com/kingsrook/qbits/marketing/AgentAndSecurityTest.java

@@ -17,10 +17,15 @@
 package com.kingsrook.qbits.marketing;
 
 
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
-import com.kingsrook.qbits.assistant.provider.ModelProviderException;
-import com.kingsrook.qbits.assistant.provider.ModelUsage;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelResponse;
+import com.kingsrook.qbits.assistant.spi.ModelUsage;
 import com.kingsrook.qbits.marketing.agent.MarketingAgentService;
 import com.kingsrook.qbits.marketing.agent.tools.MarketingTools;
 import com.kingsrook.qbits.marketing.approval.ApprovalService;
@@ -44,6 +49,8 @@ import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement;
 import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory;
 import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.context.CapturedContext;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
 import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
@@ -280,11 +287,10 @@ class AgentAndSecurityTest extends BaseMarketingTest
 
 
    /*******************************************************************************
-    ** Runs stop at their model-call limit; a cancelled run stops; the
-    ** concurrency limit refuses extra runs.
+    ** Runs stop at their model-call limit; a cancelled run stops.
     *******************************************************************************/
    @Test
-   void testLimitsCancelAndConcurrency() throws QException
+   void testLimitsAndCancel() throws QException
    {
       config.withMaxModelCallsPerRun(2);
       try
@@ -315,25 +321,112 @@ class AgentAndSecurityTest extends BaseMarketingTest
          {
             throw (new RuntimeException(e));
          }
-         return (new com.kingsrook.qbits.assistant.provider.ModelResponse().withParts(new java.util.ArrayList<>(List.of(
-            com.kingsrook.qbits.assistant.provider.ModelPart.toolUse("t1", MarketingTools.LIST_CHANNELS, Map.of())))).withStopReason("tool_use"));
+         return (new com.kingsrook.qbits.assistant.spi.ModelResponse().withParts(new java.util.ArrayList<>(List.of(
+            com.kingsrook.qbits.assistant.spi.ModelPart.toolUse("t1", MarketingTools.LIST_CHANNELS, Map.of())))).withStopReason("tool_use"));
       });
       MarketingAgentRun cancelled = new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
       assertThat(AgentRunStatus.getById(cancelled.getStatus())).isEqualTo(AgentRunStatus.CANCELLED);
+   }
 
-      MarketingRuntime runtime = MarketingRuntime.current();
-      assertThat(runtime.tryAcquireAgentPermit()).isTrue();
-      assertThat(runtime.tryAcquireAgentPermit()).isTrue();
-      try
+
+   /*******************************************************************************
+    ** The concurrency limit is counted in the database: a second service
+    ** instance (another replica on the same backend) is refused while the
+    ** first one's run is RUNNING.
+    *******************************************************************************/
+   @Test
+   void testConcurrencyHoldsAcrossReplicas() throws QException
+   {
+      config.withMaxConcurrentAgentRuns(1);
+      CapturedContext         context = QContext.capture();
+      List<MarketingAgentRun> refused = new ArrayList<>();
+      model.thenRespond(request ->
       {
-         MarketingAgentRun busy = new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
-         assertThat(busy.getErrorMessage()).contains("Too many agent runs");
-      }
-      finally
-      {
-         runtime.releaseAgentPermit();
-         runtime.releaseAgentPermit();
-      }
+         try
+         {
+            QContext.init(context);
+            refused.add(new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null));
+         }
+         catch(QException e)
+         {
+            throw (new RuntimeException(e));
+         }
+         return (new ModelResponse().withParts(new ArrayList<>(List.of(ModelPart.text("Done.")))).withStopReason(ModelResponse.STOP_END_TURN));
+      });
+
+      MarketingAgentRun first = new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
+      assertThat(AgentRunStatus.getById(first.getStatus())).isEqualTo(AgentRunStatus.SUCCEEDED);
+      assertThat(first.getHeartbeatAt()).isNotNull();
+      assertThat(refused).hasSize(1);
+      assertThat(AgentRunStatus.getById(refused.get(0).getStatus())).isEqualTo(AgentRunStatus.FAILED);
+      assertThat(refused.get(0).getErrorMessage()).contains("Too many agent runs").contains("limit 1");
+
+      /////////////////////////////////////////////////////////////
+      // the first run finished, so the next one is let through  //
+      /////////////////////////////////////////////////////////////
+      model.thenText("Again.");
+      assertThat(AgentRunStatus.getById(new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null).getStatus())).isEqualTo(AgentRunStatus.SUCCEEDED);
+   }
+
+
+
+   /*******************************************************************************
+    ** A RUNNING run whose heartbeat is older than maxRunSeconds plus the grace
+    ** is failed with a plain reason (and no longer counts); a fresh one stays.
+    *******************************************************************************/
+   @Test
+   void testStaleRunningRunIsReaped() throws QException
+   {
+      config.withMaxConcurrentAgentRuns(1);
+      Instant stale = Instant.now().minusSeconds(config.getMaxRunSeconds() + MarketingAgentService.STALE_GRACE_SECONDS + 30);
+      Integer staleId = MarketingStore.insertInternal(new MarketingAgentRun().withTask(AgentTask.ANALYZE_RESULTS.getId()).withStatus(AgentRunStatus.RUNNING.getId())
+         .withStartedAt(stale).withHeartbeatAt(stale)).getValueInteger("id");
+
+      model.thenText("Ran.");
+      MarketingAgentRun run = new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
+      assertThat(AgentRunStatus.getById(run.getStatus())).isEqualTo(AgentRunStatus.SUCCEEDED);
+
+      QRecord reaped = MarketingStore.get(MarketingAgentRun.TABLE_NAME, staleId);
+      assertThat(reaped.getValueInteger("status")).isEqualTo(AgentRunStatus.FAILED.getId());
+      assertThat(reaped.getValueString("errorMessage")).isEqualTo("Stopped responding; the server may have restarted.");
+
+      ///////////////////////////////////////////////////////////////////
+      // a RUNNING run with a recent heartbeat is alive and counts     //
+      ///////////////////////////////////////////////////////////////////
+      MarketingStore.insertInternal(new MarketingAgentRun().withTask(AgentTask.ANALYZE_RESULTS.getId()).withStatus(AgentRunStatus.RUNNING.getId())
+         .withStartedAt(stale).withHeartbeatAt(Instant.now()));
+      MarketingAgentRun refused = new MarketingAgentService().run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
+      assertThat(refused.getErrorMessage()).contains("Too many agent runs");
+   }
+
+
+
+   /*******************************************************************************
+    ** At the per-minute model-call limit a run waits for the window to clear
+    ** rather than failing; the wait is bounded by maxRunSeconds.
+    *******************************************************************************/
+   @Test
+   void testModelCallRateWaitsInsteadOfFailing() throws QException
+   {
+      config.withMaxModelCallsPerMinute(1);
+      MarketingAgentService service = new MarketingAgentService().withModelCallWindow(Duration.ofMillis(1500), Duration.ofMillis(50));
+
+      model.thenText("One.").thenText("Two.");
+      assertThat(AgentRunStatus.getById(service.run(AgentTask.ANALYZE_RESULTS, campaignId, null, null).getStatus())).isEqualTo(AgentRunStatus.SUCCEEDED);
+
+      long              start  = System.currentTimeMillis();
+      MarketingAgentRun second = service.run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
+      assertThat(AgentRunStatus.getById(second.getStatus())).isEqualTo(AgentRunStatus.SUCCEEDED);
+      assertThat(System.currentTimeMillis() - start).isGreaterThanOrEqualTo(500);
+
+      ///////////////////////////////////////////////////////////////////
+      // a wait longer than the run's time ends the run, not the JVM   //
+      ///////////////////////////////////////////////////////////////////
+      config.withMaxRunSeconds(1);
+      MarketingAgentService slow     = new MarketingAgentService().withModelCallWindow(Duration.ofSeconds(60), Duration.ofMillis(50));
+      MarketingAgentRun     timedOut = slow.run(AgentTask.ANALYZE_RESULTS, campaignId, null, null);
+      assertThat(AgentRunStatus.getById(timedOut.getStatus())).isEqualTo(AgentRunStatus.FAILED);
+      assertThat(timedOut.getErrorMessage()).contains("exceeded 1 seconds").contains("rate limit");
    }
 
 }

src/test/java/com/kingsrook/qbits/marketing/EndToEndFlowTest.java

@@ -21,7 +21,7 @@ import java.io.Serializable;
 import java.time.Instant;
 import java.util.List;
 import java.util.Map;
-import com.kingsrook.qbits.assistant.provider.ModelUsage;
+import com.kingsrook.qbits.assistant.spi.ModelUsage;
 import com.kingsrook.qbits.marketing.agent.MarketingAgentService;
 import com.kingsrook.qbits.marketing.agent.tools.MarketingTools;
 import com.kingsrook.qbits.marketing.measurement.MeasurementReport;

src/test/java/com/kingsrook/qbits/marketing/HelpBundleCoverageTest.java

@@ -0,0 +1,421 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.marketing;
+
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import com.kingsrook.qbits.marketing.metadata.MarketingMetaDataBuilder;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with
+ ** HelpBundleAssertions.forInstance(qInstance).bundles("help/qbit-marketing.yaml").owning(...).assertComplete()
+ ** once qqq#1198 ships.
+ **
+ ** The help bundle (help/qbit-marketing.yaml) covers everything the qbit
+ ** produces, and every entry is sound:
+ ** - the app, every marketing_* table, every non-trivial field (all but the
+ **   primary key, createDate, modifyDate and hidden fields), every process,
+ **   every widget and every possible-value source has an entry;
+ ** - every entry key resolves to a real object;
+ ** - summaries are at most 120 characters, with no markup;
+ ** - every {{...}} reference resolves.
+ **
+ ** Each test fails with the full list of missing or broken keys.
+ *******************************************************************************/
+class HelpBundleCoverageTest extends BaseMarketingTest
+{
+   private static final String BUNDLE_RESOURCE = "help/qbit-marketing.yaml";
+
+   private static final int    MAX_SUMMARY_LENGTH = 120;
+   private static final String MARKUP_CHARACTERS  = "<>*_`[]#";
+
+   private static final Pattern REFERENCE = Pattern.compile("\\{\\{([A-Za-z]+):([^}]*)}}");
+
+
+
+   /*******************************************************************************
+    ** One bundle entry.
+    *******************************************************************************/
+   private record Entry(String key, String summary, String content)
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** The bundle header names this qbit, at its layer.
+    *******************************************************************************/
+   @Test
+   void testHeader() throws IOException
+   {
+      Map<String, Object> bundle = loadBundle();
+      assertThat(bundle.get("bundle")).isEqualTo("qbit-marketing");
+      assertThat(bundle.get("layer")).isEqualTo("QBIT");
+      assertThat(bundle.get("version")).isNotNull();
+   }
+
+
+
+   /*******************************************************************************
+    ** Every object the qbit owns has an entry.
+    *******************************************************************************/
+   @Test
+   void testEveryOwnedObjectHasAnEntry() throws IOException
+   {
+      Set<String>  keys     = new LinkedHashSet<>();
+      List<String> problems = new ArrayList<>();
+      for(Entry entry : loadEntries())
+      {
+         if(!keys.add(entry.key()))
+         {
+            problems.add("duplicate entry: " + entry.key());
+         }
+      }
+
+      for(String expected : expectedKeys())
+      {
+         if(!keys.contains(expected))
+         {
+            problems.add("missing entry: " + expected);
+         }
+      }
+      assertNoProblems(problems);
+   }
+
+
+
+   /*******************************************************************************
+    ** Every entry key names something that exists.
+    *******************************************************************************/
+   @Test
+   void testEveryKeyResolves() throws IOException
+   {
+      List<String> problems = new ArrayList<>();
+      for(Entry entry : loadEntries())
+      {
+         String problem = resolveKey(entry.key());
+         if(problem != null)
+         {
+            problems.add(entry.key() + ": " + problem);
+         }
+      }
+      assertNoProblems(problems);
+   }
+
+
+
+   /*******************************************************************************
+    ** Summaries are short, plain text; every entry has content.
+    *******************************************************************************/
+   @Test
+   void testSummariesAndContent() throws IOException
+   {
+      List<String> problems = new ArrayList<>();
+      for(Entry entry : loadEntries())
+      {
+         String summary = entry.summary();
+         if(summary == null || summary.isBlank())
+         {
+            problems.add(entry.key() + ": no summary");
+            continue;
+         }
+         if(summary.length() > MAX_SUMMARY_LENGTH)
+         {
+            problems.add(entry.key() + ": summary is " + summary.length() + " characters (limit " + MAX_SUMMARY_LENGTH + ")");
+         }
+         for(char c : MARKUP_CHARACTERS.toCharArray())
+         {
+            if(summary.indexOf(c) >= 0)
+            {
+               problems.add(entry.key() + ": summary contains the markup character " + c);
+            }
+         }
+         if(entry.content() == null || entry.content().isBlank())
+         {
+            problems.add(entry.key() + ": no content");
+         }
+      }
+      assertNoProblems(problems);
+   }
+
+
+
+   /*******************************************************************************
+    ** Every {{...}} reference resolves.
+    *******************************************************************************/
+   @Test
+   void testEveryReferenceResolves() throws IOException
+   {
+      List<String> problems = new ArrayList<>();
+      for(Entry entry : loadEntries())
+      {
+         for(String text : new String[] { entry.summary(), entry.content() })
+         {
+            if(text == null)
+            {
+               continue;
+            }
+            Matcher matcher = REFERENCE.matcher(text);
+            while(matcher.find())
+            {
+               String problem = resolveReference(matcher.group(1), matcher.group(2));
+               if(problem != null)
+               {
+                  problems.add(entry.key() + ": {{" + matcher.group(1) + ":" + matcher.group(2) + "}} " + problem);
+               }
+            }
+            if(text.replaceAll(REFERENCE.pattern(), "").contains("{{"))
+            {
+               problems.add(entry.key() + ": malformed {{ reference");
+            }
+         }
+      }
+      assertNoProblems(problems);
+   }
+
+
+
+   /*******************************************************************************
+    ** The keys the bundle must contain, from the produced instance.
+    *******************************************************************************/
+   private Set<String> expectedKeys()
+   {
+      Set<String> keys = new TreeSet<>();
+      keys.add("app:" + MarketingMetaDataBuilder.APP_NAME + ";slot:help");
+
+      for(QTableMetaData table : qInstance.getTables().values())
+      {
+         if(!table.getName().startsWith("marketing_"))
+         {
+            continue;
+         }
+         keys.add("table:" + table.getName() + ";slot:help");
+         for(QFieldMetaData field : table.getFields().values())
+         {
+            if(isNonTrivial(table, field))
+            {
+               keys.add("table:" + table.getName() + ";field:" + field.getName());
+            }
+         }
+      }
+
+      /////////////////////////////////////////////////////////////////////////
+      // the qbit's own processes (QQQ also generates bulk table processes)  //
+      /////////////////////////////////////////////////////////////////////////
+      for(QProcessMetaData process : qInstance.getProcesses().values())
+      {
+         if(process.getSourceQBitName() != null)
+         {
+            keys.add("process:" + process.getName() + ";slot:help");
+         }
+      }
+
+      qInstance.getWidgets().keySet().stream().filter(n -> n.startsWith("marketing_")).forEach(n -> keys.add("widget:" + n + ";slot:label"));
+      qInstance.getPossibleValueSources().keySet().stream().filter(n -> n.startsWith("marketing_")).forEach(n -> keys.add("possibleValueSource:" + n + ";slot:help"));
+      return (keys);
+   }
+
+
+
+   /*******************************************************************************
+    ** Every field except the primary key, createDate, modifyDate and hidden ones.
+    *******************************************************************************/
+   private static boolean isNonTrivial(QTableMetaData table, QFieldMetaData field)
+   {
+      String name = field.getName();
+      return (!name.equals(table.getPrimaryKeyField()) && !name.equals("createDate") && !name.equals("modifyDate") && !Boolean.TRUE.equals(field.getIsHidden()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Null if the key names a real object, else why not.
+    *******************************************************************************/
+   private String resolveKey(String key)
+   {
+      Map<String, String> parts = new LinkedHashMap<>();
+      for(String part : key.split(";"))
+      {
+         int colon = part.indexOf(':');
+         if(colon < 1)
+         {
+            return ("malformed key part [" + part + "]");
+         }
+         parts.put(part.substring(0, colon), part.substring(colon + 1));
+      }
+
+      if(parts.containsKey("app"))
+      {
+         return (qInstance.getApp(parts.get("app")) == null ? "no such app" : slotProblem(parts, "help"));
+      }
+      if(parts.containsKey("table"))
+      {
+         QTableMetaData table = qInstance.getTable(parts.get("table"));
+         if(table == null)
+         {
+            return ("no such table");
+         }
+         if(parts.containsKey("field"))
+         {
+            return (table.getFields().containsKey(parts.get("field")) ? null : "no such field");
+         }
+         if(parts.containsKey("section"))
+         {
+            return (table.getSections() != null && table.getSections().stream().map(QFieldSection::getName).anyMatch(parts.get("section")::equals) ? null : "no such section");
+         }
+         return (slotProblem(parts, "help"));
+      }
+      if(parts.containsKey("process"))
+      {
+         QProcessMetaData process = qInstance.getProcess(parts.get("process"));
+         if(process == null)
+         {
+            return ("no such process");
+         }
+         if(parts.containsKey("step"))
+         {
+            return (process.getStep(parts.get("step")) == null ? "no such step" : null);
+         }
+         return (parts.containsKey("field") ? "process fields are not covered by this bundle" : slotProblem(parts, "help"));
+      }
+      if(parts.containsKey("widget"))
+      {
+         return (qInstance.getWidget(parts.get("widget")) == null ? "no such widget" : slotProblem(parts, "label"));
+      }
+      if(parts.containsKey("possibleValueSource"))
+      {
+         return (qInstance.getPossibleValueSource(parts.get("possibleValueSource")) == null ? "no such possible-value source" : slotProblem(parts, "help"));
+      }
+      return ("unsupported key");
+   }
+
+
+
+   /*******************************************************************************
+    ** The slot must be the expected one.
+    *******************************************************************************/
+   private static String slotProblem(Map<String, String> parts, String expectedSlot)
+   {
+      return (expectedSlot.equals(parts.get("slot")) ? null : "expected slot:" + expectedSlot);
+   }
+
+
+
+   /*******************************************************************************
+    ** Null if the live reference resolves, else why not.
+    *******************************************************************************/
+   private String resolveReference(String kind, String target)
+   {
+      String[]       tableAndField = target.split("\\.", 2);
+      QTableMetaData table         = qInstance.getTable(tableAndField[0]);
+      QFieldMetaData field         = table == null || tableAndField.length < 2 ? null : table.getFields().get(tableAndField[1]);
+
+      switch(kind)
+      {
+         case "label":
+         {
+            if(tableAndField.length == 2)
+            {
+               return (field == null ? "does not resolve to a table field" : null);
+            }
+            boolean found = table != null || qInstance.getProcess(target) != null || qInstance.getApp(target) != null;
+            return (found ? null : "does not resolve to a table, process or app");
+         }
+         case "default":
+         {
+            return (field == null ? "does not resolve to a table field" : (field.getDefaultValue() == null ? "the field has no default value" : null));
+         }
+         case "maxLength":
+         {
+            return (field == null ? "does not resolve to a table field" : (field.getMaxLength() == null ? "the field has no maximum length" : null));
+         }
+         case "values":
+         {
+            QPossibleValueSource source = qInstance.getPossibleValueSource(target);
+            return (source == null ? "does not resolve to a possible-value source" : (source.getType() == QPossibleValueSourceType.ENUM ? null : "is not an enum source"));
+         }
+         default:
+         {
+            return ("unsupported reference kind");
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The bundle, as parsed YAML.
+    *******************************************************************************/
+   private static Map<String, Object> loadBundle() throws IOException
+   {
+      try(InputStream inputStream = HelpBundleCoverageTest.class.getClassLoader().getResourceAsStream(BUNDLE_RESOURCE))
+      {
+         assertThat(inputStream).as("classpath resource " + BUNDLE_RESOURCE).isNotNull();
+         @SuppressWarnings("unchecked")
+         Map<String, Object> bundle = new ObjectMapper(new YAMLFactory()).readValue(inputStream, Map.class);
+         return (bundle);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The bundle's entries.
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static List<Entry> loadEntries() throws IOException
+   {
+      List<Entry> entries = new ArrayList<>();
+      for(Map<String, Object> row : (List<Map<String, Object>>) loadBundle().get("entries"))
+      {
+         entries.add(new Entry(String.valueOf(row.get("key")), (String) row.get("summary"), (String) row.get("content")));
+      }
+      return (entries);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fail with every problem at once.
+    *******************************************************************************/
+   private static void assertNoProblems(List<String> problems)
+   {
+      assertThat(problems).as(problems.size() + " help bundle problem(s):\n" + String.join("\n", problems)).isEmpty();
+   }
+
+}

src/test/java/com/kingsrook/qbits/marketing/ScriptedModelProvider.java

@@ -22,16 +22,16 @@ import java.util.ArrayList;
 import java.util.Deque;
 import java.util.List;
 import java.util.Map;
-import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
-import com.kingsrook.qbits.assistant.provider.CompletableStreamHandle;
-import com.kingsrook.qbits.assistant.provider.ModelCapabilities;
-import com.kingsrook.qbits.assistant.provider.ModelEventListener;
-import com.kingsrook.qbits.assistant.provider.ModelPart;
-import com.kingsrook.qbits.assistant.provider.ModelProviderException;
-import com.kingsrook.qbits.assistant.provider.ModelRequest;
-import com.kingsrook.qbits.assistant.provider.ModelResponse;
-import com.kingsrook.qbits.assistant.provider.ModelUsage;
-import com.kingsrook.qbits.assistant.provider.StreamHandle;
+import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.spi.CompletableStreamHandle;
+import com.kingsrook.qbits.assistant.spi.ModelCapabilities;
+import com.kingsrook.qbits.assistant.spi.ModelEventListener;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelRequest;
+import com.kingsrook.qbits.assistant.spi.ModelResponse;
+import com.kingsrook.qbits.assistant.spi.ModelUsage;
+import com.kingsrook.qbits.assistant.spi.StreamHandle;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
 
 

src/test/java/com/kingsrook/qbits/marketing/agent/provider/OpenAiCompatibleModelProviderTest.java

@@ -25,14 +25,14 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.atomic.AtomicReference;
-import com.kingsrook.qbits.assistant.provider.ModelMessage;
-import com.kingsrook.qbits.assistant.provider.ModelPart;
-import com.kingsrook.qbits.assistant.provider.ModelProviderException;
-import com.kingsrook.qbits.assistant.provider.ModelRequest;
-import com.kingsrook.qbits.assistant.provider.ModelResponse;
-import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
-import com.kingsrook.qbits.assistant.provider.ProviderSettings;
-import com.kingsrook.qbits.assistant.provider.SystemBlock;
+import com.kingsrook.qbits.assistant.spi.ModelMessage;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelRequest;
+import com.kingsrook.qbits.assistant.spi.ModelResponse;
+import com.kingsrook.qbits.assistant.spi.ModelToolDefinition;
+import com.kingsrook.qbits.assistant.spi.ProviderSettings;
+import com.kingsrook.qbits.assistant.spi.SystemBlock;
 import com.kingsrook.qbits.assistant.provider.openai.OpenAiCompatibleModelProvider;
 import com.kingsrook.qbits.marketing.MarketingQBitConfig;
 import com.kingsrook.qqq.backend.core.instances.QMetaDataVariableInterpreter;

src/test/resources/marketing-h2.sql

@@ -314,6 +314,7 @@ CREATE TABLE IF NOT EXISTS marketing_agent_run (
    model VARCHAR(100),
    requested_by_user_id VARCHAR(100),
    started_at TIMESTAMP,
+   heartbeat_at TIMESTAMP,
    finished_at TIMESTAMP,
    model_calls INTEGER,
    tool_calls INTEGER,