feat(assistant): POST /assistant/client-errors with the assistant_client_error table and health check

UI API extension assistant 1.5.0 adds POST /qqq/v1/assistant/client-errors
(AssistantClientErrorReportV1 in, AssistantClientErrorResponseV1 out, 202).
Any signed-in user may report; scope and message are required (400
otherwise, also for a non-object body). Every field is bounded, message and
stacks are scrubbed (AiRedactor secrets, e-mails, digit runs over 6, mixed
tokens, query strings), the route keeps only its path, and conversation /
message / turn ids are kept only when they are the user's own. Over
clientErrorsPerUserPerMinute (default 30) the report is acknowledged with
accepted:false and dropped, never a 429.

Rows go to the new owner-locked assistant_client_error table (Assistant app
nav, record screens, bundle help, channel wall by pattern, SET NULL foreign
keys in the reference DDL) and are deleted by retention after
toolResultRetentionDays. The assistant.client-errors health check is
DEGRADED at clientErrorWarnPerHour (default 20) reports in the last hour, so
the Assistant health card shows it too. The deprecated AssistantErrorV1
description again says "since 1.3.0".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## Unreleased - Fail soft (UI API extension 1.5.0)
+
+- Every message part, tool render and presentation is validated before it is stored or streamed, and again when stored parts and events are read: all render types (records, record, count, widget, history, pending_action, bulk_load_dry_run) and presentation blocks always carry their lists. A render that cannot be repaired is replaced by an error notice ("This result could not be displayed."); unknown part and render types pass through. Problems are logged with the type and a fixed code only.
+- Tool calls have their own limit (`toolCallTimeoutSeconds`, default 60) besides the turn's (`maxTurnSeconds`, default 300). A timed-out or failing tool gets an ERROR (`is_error`) result for the model, an ERROR tool part and an error notice for the user; internal details go only to the log.
+- Every turn ends: terminal events are emitted independently, so a failing emit never skips `message.end` or the terminal turn row. Conversation detail repairs assistant messages left `STREAMING` (an orphaned run is reaped; a run that already ended gives its status), and the retention process finalizes runs still `RUNNING` past the hard stream limit (e.g. after a pod restart).
+- `POST /qqq/v1/assistant/client-errors` (extension 1.5.0): the panel reports errors it hit (`scope`, `message`, optional stacks, route, ids, part/render type, app version, `occurredAt`). Answers `202 {"accepted": true}`; over `clientErrorsPerUserPerMinute` (default 30) it answers `202 {"accepted": false}` and drops the report. Fields are bounded, message and stacks scrubbed (secrets, e-mails, long numbers, tokens, query strings), ids kept only when they are the user's own. Rows go to the new owner-locked `assistant_client_error` table (Assistant app nav, retention after `toolResultRetentionDays`), and the `assistant.client-errors` health check turns degraded at `clientErrorWarnPerHour` (default 20) reports in an hour. Apply the Fail soft DDL (UPGRADE.md).
+
+## Unreleased - Empty renders and stream context (UI API extension 1.4.0)
+
+- Tool result renders always carry their lists: a `records` render for a query that matched nothing now has `"rows": []` and `"columns": []` (QQQ's JSON omitted empty lists, and Next's Assistant crashed the page reading `rows.length`). Parts and events stored before the fix are completed when read. Extension 1.4.0 documents the shape as `AssistantRecordsRenderV1`.
+- Streamed text arrives on the provider's own (virtual) thread; the turn's progress updates now run in the turn's QContext, so job progress no longer fails with "Could not flush job progress" (NPE on the missing instance).
+
+## Unreleased - UI API 1.6.0
+
+- UI API extension assistant 1.3.0 (requires QQQ 4.1.0-RC.15, UI API 1.6.0): operationIds, standard error responses with BasicErrorResponseV1, extension tags and trace IDs. Preserve HTTP statuses and assistant-specific error codes.
+- Retain AssistantErrorV1 as a deprecated component for compatibility. Remove dangling `null` schema references from the two bodyless 204 responses (delete conversation and feedback), with an empty-content workaround for core schema traversal. Success payloads are unchanged; the contract test retains the release baseline with explicit exceptions for the error schema migration and these documentation corrections.
+
+## Unreleased - Central health
+
+### Added
+
+- Dynamic `assistant.provider.<id>` and `assistant.model.<id>` dependency checks in component `assistant`, category AI, reusing provider diagnostics and minimal connection calls with deadlines, cost-cap/kill-switch skips and TEST usage accounting.
+- LOCAL breaker reuse of central provider outcomes younger than 60 seconds, with its existing lazy probe as fallback.
+- Generic Assistant health status card linking to filtered System Health; registration, status, accounting, deadline and sentinel-secret tests.
+- QQQ 4.1 compatibility for the header shim: setting extra headers avoids duplicates now that core forwards them too.
+
+### Removed
+
+- The home's independent answering/last-test health indicators and per-provider health/down-until display. Central health owns status and history. Manual Test connection actions, `lastTest*` fields and the circuit breaker remain.
+
+## Unreleased - Jobs tray
+
+- Track assistant answers in the core Jobs tray through `JobTracking`, with thinking, tool and writing progress and Stop support.
+- Require QQQ 4.1.0-RC.8 and the host's durable jobs runtime and jobs tables; answers remain untracked when the runtime is absent.
+- Default to `JobTrackingRunTracker`; opt out with `withRunTracker(AssistantRunTracker.NOOP)`.
+
 ## Unreleased - In-app help
 
 - Ship the host-neutral Assistant help bundle through QQQ's native registry, replacing the temporary loader.

README.md

@@ -220,12 +220,41 @@ Admins set up providers on the **Providers** screen (`assistant_provider`), a st
 
 **Test connection** is a record action on a provider (select one or more). It is a standard streamed ETL: Review lists what will be called and anything wrong with the row (missing key, disallowed variable or URL) before any call. Result has one line per provider: `connected to claude-sonnet-5-5 in 812 ms`, or the error with its kind and HTTP status. It makes one short live call (a 1-sentence prompt, at most 32 output tokens, effort low), saves the result on the row, and records the call in `assistant_usage` with purpose `TEST`.
 
+## Health checks
+
+The producer registers dynamic central checks in component `assistant`, category `AI`:
+
+| Name | Kind | Interval | Timeout |
+| --- | --- | --- | --- |
+| `assistant.provider.<id>` | DEPENDENCY | 300s; 60s for LOCAL providers | 10s |
+| `assistant.model.<id>` | DEPENDENCY | 21600s (6 hours) | 30s |
+
+Suppliers include enabled provider/model rows and refresh with the central registry.
+Provider checks reuse `AssistantProviderType.checkHealth`. Unsupported probes use
+circuit-breaker state only when less than ten minutes old, otherwise return UNKNOWN.
+Model checks reuse `ConnectionTest` (embeddings for embedding models), respect the
+provider's monthly cost cap and the kill switch, and leave manual `lastTest*` fields
+unchanged. A capped check returns UNKNOWN, `skipped: cost cap`, even on demand.
+The automated runner records usage under its own owner key. Provider caps use the
+existing shared accounting projection without access to other users' usage records.
+Usage purpose is a string, so health diagnostics reuse `TEST` (not a new `HEALTH` enum).
+All checks honor their deadlines; messages and details omit provider error text,
+credentials, endpoints, prompts and completions.
+
+The LOCAL circuit breaker consults central results less than 60 seconds old before
+its lazy probe. UP/DEGRADED avoids a duplicate probe, DOWN rejects the call, and
+UNKNOWN or stale/missing results retain the existing probe fallback. The breaker
+and manual Test connection actions remain available.
+
+This integration uses the core health SPI released in QQQ 4.1.0-RC.8.
+The POM pins that release; verify with `./mvnw -o verify`.
+
 ## The app home
 
-The **Assistant** app's home is the one place for the assistant's health, usage and cost (no other dashboard repeats it). All of it comes from the tables below, read as the viewing admin; an empty table gives an empty state, never sample data.
+The **Assistant** app's home shows configuration, usage and cost read as the viewing admin. A generic status card summarizes the central runner's latest results and links to System Health filtered to `assistant`; it shows UNKNOWN before results are available. Health history and check lists live in System Health.
 
-- **Health:** is it answering (kill switch, default provider, key present, last test), the 7-day error rate of model calls, cost and requests this month against the instance's monthly limit.
-- **Providers:** one row per provider: type, model, masked key status, last test, cost this month against its cap, limits, and any problem with its settings.
+- **Configuration and usage:** default provider, key status, the 7-day error rate of model calls, cost and requests this month against the instance's monthly limit.
+- **Providers:** one row per provider: type, model, masked key status, cost this month against its cap, limits, and any problem with its settings.
 - **Usage, last 30 days:** bars of model requests per day, totals (conversations, requests, tokens in and out, cost) and the last 7 days in words.
 - **Costs this month:** month to date against the policy limit and each provider cap, then by model. Cost is tokens times the model's price from **Model prices**.
 - **By user, this month**, **Top tools** (calls, failures, average time) and **Recent failures** (model calls and tool calls, linked).
@@ -644,8 +673,8 @@ never automatically move to cloud. Provider state/signatures are removed when
 switching, preserving neutral tool history. Each attempted call is audited as usage.
 
 NETWORK/TIMEOUT marks a provider DOWN for 60 seconds; success marks it UP. Local
-health is probed when unknown/stale with a two-second timeout. Provider cards show
-health and down-until, failures show fallback totals, and turn records show the
+health uses recent central results or a lazy probe with a two-second timeout. The
+central health card links to System Health; failures show fallback totals, and turn records show the
 route and final destination. See [UPGRADE.md](UPGRADE.md) for the additive schema.
 
 ## Scheduled tasks

UPGRADE.md

@@ -1,5 +1,17 @@
 # Upgrading qbit-assistant
 
+## Fail soft (Unreleased, UI API extension assistant 1.5.0)
+
+Apply the **Fail soft** block at the end of `docs/schema-postgres.sql` before
+deploying. It creates `assistant_client_error` (errors the panel reports through
+`POST /qqq/v1/assistant/client-errors`) with its indexes and `SET NULL` foreign keys
+to the conversation, message and turn. It is `IF NOT EXISTS` and safe to re-run.
+
+New configuration, all optional: `toolCallTimeoutSeconds` (default 60),
+`clientErrorsPerUserPerMinute` (default 30) and `clientErrorWarnPerHour` (default
+20, when the `assistant.client-errors` health check turns degraded). The new table
+is in the Assistant app's nav for administrators.
+
 ## Transcript redaction (Unreleased)
 
 Apply the transcript-redaction column widening block in `docs/schema-postgres.sql`
@@ -69,12 +81,18 @@ reattach through the existing events route. Consumers should recognize
 `turn.phase` and `message.start.startedAt`. Starts during shutdown return
 503 `SHUTTING_DOWN`; clients may retry against a ready replica.
 
-Hosts may wire `AssistantRunTracker` for Jobs tray publication, progress, Stop,
-and terminal notification. It is a no-op by default and all callbacks are
-isolated from turn execution. Tracking ids are local to the owning runtime;
-reaping an already dead owner does not invent another tracking job. Hosts should
-associate their durable tracking row with the supplied turn id. The SPI is a
-stand-in for QQQ #1210.
+`JobTrackingRunTracker` is now the default `AssistantRunTracker`, using core
+`JobTracking` in QQQ **4.1.0-RC.8** to publish answers in the Jobs tray. The host
+must provide the jobs tables (through qbit-jobs / core `DurableJobsMetaDataProvider`)
+and run `DurableJobsRuntime`. Without that runtime, answers complete normally
+and are simply not tracked.
+
+Each job is owned by the assistant user, links to its `assistant_conversation`,
+and reports thinking, tool and writing phases. Tool progress contains only the
+tool label; terminal messages are fixed and never contain prompts or answers.
+The tray's Stop cancels the assistant turn. All tracker callbacks remain isolated
+from turn execution. Opt out with `withRunTracker(AssistantRunTracker.NOOP)`;
+hosts may still supply their own `AssistantRunTracker`.
 
 ## Starter defaults, setup card and table help
 

docs/schema-postgres.sql

@@ -713,3 +713,34 @@ ALTER TABLE assistant_data_audit ALTER COLUMN reason SET DATA TYPE TEXT;
 ALTER TABLE assistant_budget_override ALTER COLUMN reason SET DATA TYPE TEXT;
 ALTER TABLE assistant_scheduled_run ALTER COLUMN summary SET DATA TYPE TEXT;
 ALTER TABLE assistant_scheduled_run ALTER COLUMN error SET DATA TYPE TEXT;
+
+-- Fail soft (UI API extension assistant 1.5.0): errors the panel reports through
+-- POST /qqq/v1/assistant/client-errors. Owner-locked like the other user rows; the
+-- message and stacks are scrubbed and bounded before they are stored. The ids are
+-- kept only when they are the reporting user's own rows, and SET NULL when those go.
+-- Retention deletes rows older than toolResultRetentionDays. Safe to re-run.
+CREATE TABLE IF NOT EXISTS assistant_client_error
+(
+   id SERIAL PRIMARY KEY,
+   user_id VARCHAR(255),
+   scope VARCHAR(64),
+   message TEXT,
+   error_name VARCHAR(100),
+   stack TEXT,
+   component_stack TEXT,
+   request_id VARCHAR(128),
+   route VARCHAR(300),
+   conversation_id INTEGER CONSTRAINT fk_assistant_client_error_conversation_id REFERENCES assistant_conversation (id) ON DELETE SET NULL,
+   message_id INTEGER CONSTRAINT fk_assistant_client_error_message_id REFERENCES assistant_message (id) ON DELETE SET NULL,
+   turn_id INTEGER CONSTRAINT fk_assistant_client_error_turn_id REFERENCES assistant_turn (id) ON DELETE SET NULL,
+   part_type VARCHAR(64),
+   render_type VARCHAR(64),
+   app_version VARCHAR(64),
+   occurred_at TIMESTAMP WITH TIME ZONE,
+   create_date TIMESTAMP WITH TIME ZONE
+);
+CREATE INDEX IF NOT EXISTS assistant_client_error_user_id ON assistant_client_error (user_id);
+CREATE INDEX IF NOT EXISTS assistant_client_error_create_date ON assistant_client_error (create_date);
+CREATE INDEX IF NOT EXISTS assistant_client_error_conversation ON assistant_client_error (conversation_id);
+CREATE INDEX IF NOT EXISTS assistant_client_error_message ON assistant_client_error (message_id);
+CREATE INDEX IF NOT EXISTS assistant_client_error_turn ON assistant_client_error (turn_id);

pom.xml

@@ -215,8 +215,9 @@
             <activeByDefault>true</activeByDefault>
          </activation>
          <properties>
-            <!-- RC.7 provides classified-value tokens, log redaction and native help bundles. -->
-            <qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
+            <!-- RC.7 provides classified-value tokens, log redaction and native help bundles; -->
+            <!-- RC.8 provides core job tracking (JobTracking, the Jobs tray).                   -->
+            <qqq.snapshot.version>4.1.0-RC.15</qqq.snapshot.version>
          </properties>
          <dependencyManagement>
             <dependencies>

qbit-assistant/pom.xml

@@ -114,6 +114,7 @@
             <directory>src/main/resources</directory>
             <filtering>true</filtering>
             <includes>
+               <include>help/*.yaml</include>
                <include>qbit-assistant.properties</include>
             </includes>
          </resource>
@@ -121,6 +122,7 @@
             <directory>src/main/resources</directory>
             <filtering>false</filtering>
             <excludes>
+               <exclude>help/*.yaml</exclude>
                <exclude>qbit-assistant.properties</exclude>
             </excludes>
          </resource>

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantQBitConfig.java

@@ -33,6 +33,7 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.kingsrook.qbits.assistant.classification.ClassificationSource;
 import com.kingsrook.qbits.assistant.classification.ConfiguredClassificationSource;
 import com.kingsrook.qbits.assistant.jobs.AssistantRunTracker;
+import com.kingsrook.qbits.assistant.jobs.JobTrackingRunTracker;
 import com.kingsrook.qbits.assistant.rag.VectorStore;
 import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
 import com.kingsrook.qbits.assistant.spi.AssistantProviderType;
@@ -106,9 +107,12 @@ public class AssistantQBitConfig implements QBitConfig
    private Integer shutdownGraceSeconds = 8;
    private String defaultConcurrencyMode = "SUPERSEDE";
    private Boolean registerShutdownHook = true;
-   private AssistantRunTracker runTracker = AssistantRunTracker.NOOP;
+   private AssistantRunTracker runTracker = new JobTrackingRunTracker();
 
    private Integer               maxTurnSeconds             = 300;
+   private Integer               toolCallTimeoutSeconds     = 60;
+   private Integer               clientErrorsPerUserPerMinute = 30;
+   private Integer               clientErrorWarnPerHour     = 20;
    private Integer               maxUserMessageChars        = 8000;
    private Integer               historyMessageLimit        = 40;
    private AssistantBudgets      budgets                    = new AssistantBudgets();
@@ -232,6 +236,9 @@ public class AssistantQBitConfig implements QBitConfig
          errors.add("turnHeartbeatSeconds must be less than turnLeaseSeconds.");
       }
 
+      requirePositive(errors, "toolCallTimeoutSeconds", toolCallTimeoutSeconds);
+      requirePositive(errors, "clientErrorsPerUserPerMinute", clientErrorsPerUserPerMinute);
+      requirePositive(errors, "clientErrorWarnPerHour", clientErrorWarnPerHour);
       requirePositive(errors, "maxUserMessageChars", maxUserMessageChars);
       requirePositive(errors, "historyMessageLimit", historyMessageLimit);
       requirePositive(errors, "maxTurnsPerUserPerMinute", maxTurnsPerUserPerMinute);
@@ -746,6 +753,99 @@ public class AssistantQBitConfig implements QBitConfig
 
 
 
+   /*******************************************************************************
+    ** Getter for toolCallTimeoutSeconds (how long one tool call may run before it gets an error result; default 60)
+    *******************************************************************************/
+   public Integer getToolCallTimeoutSeconds()
+   {
+      return (this.toolCallTimeoutSeconds);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for toolCallTimeoutSeconds
+    *******************************************************************************/
+   public void setToolCallTimeoutSeconds(Integer toolCallTimeoutSeconds)
+   {
+      this.toolCallTimeoutSeconds = toolCallTimeoutSeconds;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for toolCallTimeoutSeconds
+    *******************************************************************************/
+   public AssistantQBitConfig withToolCallTimeoutSeconds(Integer toolCallTimeoutSeconds)
+   {
+      this.toolCallTimeoutSeconds = toolCallTimeoutSeconds;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for clientErrorsPerUserPerMinute (client error reports accepted per user and minute; more are acknowledged but dropped; default 30)
+    *******************************************************************************/
+   public Integer getClientErrorsPerUserPerMinute()
+   {
+      return (this.clientErrorsPerUserPerMinute);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for clientErrorsPerUserPerMinute
+    *******************************************************************************/
+   public void setClientErrorsPerUserPerMinute(Integer clientErrorsPerUserPerMinute)
+   {
+      this.clientErrorsPerUserPerMinute = clientErrorsPerUserPerMinute;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for clientErrorsPerUserPerMinute
+    *******************************************************************************/
+   public AssistantQBitConfig withClientErrorsPerUserPerMinute(Integer clientErrorsPerUserPerMinute)
+   {
+      this.clientErrorsPerUserPerMinute = clientErrorsPerUserPerMinute;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for clientErrorWarnPerHour (client error reports in the last hour at which the health check warns; default 20)
+    *******************************************************************************/
+   public Integer getClientErrorWarnPerHour()
+   {
+      return (this.clientErrorWarnPerHour);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for clientErrorWarnPerHour
+    *******************************************************************************/
+   public void setClientErrorWarnPerHour(Integer clientErrorWarnPerHour)
+   {
+      this.clientErrorWarnPerHour = clientErrorWarnPerHour;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for clientErrorWarnPerHour
+    *******************************************************************************/
+   public AssistantQBitConfig withClientErrorWarnPerHour(Integer clientErrorWarnPerHour)
+   {
+      this.clientErrorWarnPerHour = clientErrorWarnPerHour;
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for maxUserMessageChars
     *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantQBitProducer.java

@@ -23,6 +23,8 @@ import java.util.List;
 import java.util.Properties;
 import com.kingsrook.qbits.assistant.api.AssistantRoutes;
 import com.kingsrook.qbits.assistant.developer.AssistantDeveloperPanels;
+import com.kingsrook.qbits.assistant.health.AssistantHealthChecks;
+import com.kingsrook.qbits.assistant.health.AssistantHealthWidget;
 import com.kingsrook.qbits.assistant.home.AssistantAccessWidget;
 import com.kingsrook.qbits.assistant.home.AssistantHome;
 import com.kingsrook.qbits.assistant.metadata.AssistantChannelWall;
@@ -200,6 +202,7 @@ public class AssistantQBitProducer implements QBitMetaDataProducer<AssistantQBit
          qInstance.addProcess(process);
       }
       qInstance.addWidget(AssistantHome.widget());
+      qInstance.addWidget(AssistantHealthWidget.widget());
       qInstance.addWidget(AssistantAccessWidget.widget());
 
       QAppMetaData app = builder.buildAdminApp(tables);
@@ -217,6 +220,7 @@ public class AssistantQBitProducer implements QBitMetaDataProducer<AssistantQBit
          ClientChannelsMetaData.ofOrWithNew(qInstance).withDeniedPattern("assistant", "audit*");
       }
       postProduceActions(new MetaDataProducerMultiOutput(), qInstance);
+      AssistantHealthChecks.register(qInstance);
       return (new MetaDataProducerMultiOutput());
    }
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantRuntime.java

@@ -17,6 +17,7 @@
 package com.kingsrook.qbits.assistant;
 
 
+import java.lang.ref.WeakReference;
 import java.net.InetAddress;
 import java.time.Clock;
 import java.time.Duration;
@@ -29,6 +30,7 @@ import java.util.regex.Pattern;
 import com.kingsrook.qbits.assistant.access.AccessExplainer;
 import com.kingsrook.qbits.assistant.access.DataAccessService;
 import com.kingsrook.qbits.assistant.engine.BudgetService;
+import com.kingsrook.qbits.assistant.engine.ClientErrors;
 import com.kingsrook.qbits.assistant.engine.ContextBuilder;
 import com.kingsrook.qbits.assistant.engine.PolicyService;
 import com.kingsrook.qbits.assistant.engine.ProfileService;
@@ -85,6 +87,7 @@ public class AssistantRuntime
    private final ProviderService                  providerService;
    private final BudgetService                    budgetService;
    private final RateLimiter                      rateLimiter;
+   private final ClientErrors                     clientErrors;
    private final AiToolKernel                     kernel;
    private final ContextBuilder                   contextBuilder;
    private final DataAccessService                dataAccessService;
@@ -133,6 +136,7 @@ public class AssistantRuntime
       this.budgetService = new BudgetService(store, providerService, clock);
       this.policyService.withBudgetService(budgetService, providerService);
       this.rateLimiter = new RateLimiter(clock);
+      this.clientErrors = new ClientErrors(clock);
       AiToolKernelSettings settings = kernelSettings(config);
       this.kernel = new AiToolKernel(qInstance, settings);
       AiExposurePolicy exposurePolicy = new AiExposurePolicy(settings);
@@ -178,11 +182,22 @@ public class AssistantRuntime
       this.contextBuilder = new ContextBuilder(config, kernel, toolbox).withKnowledgeTool(knowledgeTool).withBulkLoadTools(bulkLoadTools);
       if(Boolean.TRUE.equals(config.getRegisterShutdownHook()))
       {
+         ///////////////////////////////////////////////////////////////////////
+         // the hook holds the runtime weakly: a runtime (and its QInstance)   //
+         // that is dropped without shutdown() - every test that builds an    //
+         // instance - must not stay reachable from the JVM's hooks           //
+         ///////////////////////////////////////////////////////////////////////
+         WeakReference<AssistantRuntime> self = new WeakReference<>(this);
+         Integer graceSeconds = config.getShutdownGraceSeconds();
          shutdownHook = new Thread(() ->
          {
             try
             {
-               shutdown(Duration.ofSeconds(config.getShutdownGraceSeconds()));
+               AssistantRuntime runtime = self.get();
+               if(runtime != null)
+               {
+                  runtime.shutdown(Duration.ofSeconds(graceSeconds));
+               }
             }
             catch(Throwable ignored)
             {
@@ -469,6 +484,17 @@ public class AssistantRuntime
 
 
 
+   /*******************************************************************************
+    ** Getter for clientErrors (bounds, scrubbing and rate limit of client
+    ** error reports)
+    *******************************************************************************/
+   public ClientErrors getClientErrors()
+   {
+      return (clientErrors);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for kernel
     *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/api/AssistantApi.java

@@ -32,6 +32,8 @@ import com.kingsrook.qbits.assistant.AssistantRuntime;
 import com.kingsrook.qbits.assistant.AssistantSecurity;
 import com.kingsrook.qbits.assistant.access.AccessExplainer;
 import com.kingsrook.qbits.assistant.access.DataAccessMode;
+import com.kingsrook.qbits.assistant.engine.AbandonedTurns;
+import com.kingsrook.qbits.assistant.engine.ClientErrors;
 import com.kingsrook.qbits.assistant.engine.ContextBuilder;
 import com.kingsrook.qbits.assistant.engine.MessageParts;
 import com.kingsrook.qbits.assistant.engine.PolicyService;
@@ -40,6 +42,7 @@ import com.kingsrook.qbits.assistant.engine.ProviderService;
 import com.kingsrook.qbits.assistant.engine.RunningTurn;
 import com.kingsrook.qbits.assistant.engine.TurnEngine;
 import com.kingsrook.qbits.assistant.engine.TurnEvent;
+import com.kingsrook.qbits.assistant.model.AssistantClientError;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
 import com.kingsrook.qbits.assistant.model.AssistantFeedback;
 import com.kingsrook.qbits.assistant.model.AssistantMessage;
@@ -206,7 +209,7 @@ public class AssistantApi
       List<Map<String, Object>> messages = new ArrayList<>();
       for(AssistantMessage message : store.listMessages(userId, id, beforeMessageId, pageSize))
       {
-         messages.add(toJson(message));
+         messages.add(toJson(repairIfAbandoned(message)));
       }
 
       if(conversation.getUnreadCount() != null && conversation.getUnreadCount() > 0)
@@ -223,6 +226,45 @@ public class AssistantApi
 
 
 
+   /*******************************************************************************
+    ** A message still STREAMING whose run is not live here: an orphaned run is
+    ** recovered; a run that already ended (or is gone) gives the message its
+    ** terminal status. A run still alive on another replica is left alone.
+    ** Never throws: the conversation is shown either way.
+    *******************************************************************************/
+   private AssistantMessage repairIfAbandoned(AssistantMessage message)
+   {
+      if(!"STREAMING".equals(message.getStatus()) || (message.getTurnId() != null && runtime.getRunningTurn(message.getTurnId()) != null))
+      {
+         return (message);
+      }
+      try
+      {
+         AssistantTurn turn = message.getTurnId() == null ? null : store.getTurn(userId, message.getTurnId());
+         if(isRunning(turn))
+         {
+            turn = reapIfOrphaned(turn);
+            if(isRunning(turn))
+            {
+               return (message);
+            }
+            AssistantMessage reread = store.getMessage(userId, message.getId());
+            if(reread != null && !"STREAMING".equals(reread.getStatus()))
+            {
+               return (reread);
+            }
+         }
+         return (AbandonedTurns.repairMessage(store, userId, message, turn));
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Could not check an unfinished assistant message", logPair("messageId", message.getId()));
+         return (message);
+      }
+   }
+
+
+
    /*******************************************************************************
     ** PATCH /conversations/{id}
     *******************************************************************************/
@@ -669,53 +711,7 @@ public class AssistantApi
       }
       // Recovery only appends fixed notices to already redacted stored parts (or
       // crash placeholders). It never reloads model/tool values or raw user text.
-      String status = cancelReason == null ? "ERROR" : "CANCELLED";
-      String text = cancelReason == null ? "This answer was interrupted (the server restarted). Ask again."
-         : ("SUPERSEDED".equals(cancelReason) ? TurnEngine.SUPERSEDED_TEXT : "Stopped by you.");
-      if(current.getAssistantMessageId() != null)
-      {
-         try
-         {
-            AssistantMessage message = store.getMessage(userId, current.getAssistantMessageId());
-            if(message != null)
-            {
-               store.updateFields(AssistantMessage.TABLE_NAME, new QRecord().withValue("id", message.getId()).withValue("status", status)
-                  .withValue("partsJson", MessageParts.fromJson(message.getPartsJson()).addNotice(cancelReason == null ? "error" : "info", text).toJson()));
-            }
-         }
-         catch(Exception e)
-         {
-            LOG.warn("Could not persist recovered assistant message");
-         }
-      }
-      try
-      {
-         int seq = store.lastEventSeq(userId, current.getId());
-         List<AssistantTurnEvent> last = store.listEvents(userId, current.getId(), Math.max(0, seq - 1), 1);
-         if(last.isEmpty() || !TurnEvent.MESSAGE_END.equals(last.getFirst().getEventType()))
-         {
-            if(cancelReason == null)
-            {
-               seq++;
-               store.insertEvent(current.getId(), userId, seq, TurnEvent.ERROR,
-                  new TurnEvent(seq, TurnEvent.ERROR, Map.of("code", "INTERNAL", "message", text)).toJson());
-            }
-            seq++;
-            Map<String, Object> end = new LinkedHashMap<>();
-            end.put("messageId", current.getAssistantMessageId());
-            end.put("status", status);
-            end.put("stopReason", cancelReason == null ? "interrupted" : "cancelled");
-            store.insertEvent(current.getId(), userId, seq, TurnEvent.MESSAGE_END, new TurnEvent(seq, TurnEvent.MESSAGE_END, end).toJson());
-         }
-      }
-      catch(Exception e)
-      {
-         LOG.warn("Could not persist recovered assistant events");
-      }
-      Instant now = runtime.getClock().instant();
-      store.updateRunningTurn(userId, new QRecord().withValue("id", current.getId()).withValue("status", status)
-         .withValue("errorCode", cancelReason == null ? "INTERNAL" : null).withValue("cancelReason", cancelReason)
-         .withValue("stopReason", cancelReason == null ? "interrupted" : "cancelled").withValue("phase", "done").withValue("endedAt", now));
+      AbandonedTurns.finalizeRun(store, runtime.getClock().instant(), userId, current, cancelReason);
       return (store.getTurn(userId, current.getId()));
    }
 
@@ -755,6 +751,71 @@ public class AssistantApi
 
 
 
+   /*******************************************************************************
+    ** POST /client-errors: an error the panel hit, reported by the user's own
+    ** browser. Any signed-in user may report (also when the assistant is off
+    ** for them); only bounded, scrubbed fields are stored (ClientErrors), and
+    ** conversation / message / turn ids only when they are the user's own.
+    ** Over the per-user rate limit the report is acknowledged but dropped
+    ** (accepted: false), never a 429, so a looping client is not spammed.
+    *******************************************************************************/
+   public Map<String, Object> reportClientError(Map<String, Object> body) throws QException, AssistantApiException
+   {
+      if(userId == null)
+      {
+         throw (new AssistantApiException(403, "NO_PERMISSION", "Sign in to report errors."));
+      }
+      if(body == null)
+      {
+         throw (new AssistantApiException(400, "A JSON object is required."));
+      }
+      String scope = ClientErrors.token(body.get("scope"), ClientErrors.SCOPE_MAX);
+      if(scope == null)
+      {
+         throw (new AssistantApiException(400, "scope is required."));
+      }
+      String message = ClientErrors.text(body.get("message"), ClientErrors.MESSAGE_MAX);
+      if(message == null)
+      {
+         throw (new AssistantApiException(400, "message is required."));
+      }
+      if(!runtime.getClientErrors().tryAcquire(userId, config.getClientErrorsPerUserPerMinute()))
+      {
+         return (Map.of("accepted", false));
+      }
+
+      Integer conversationId = ClientErrors.id(body.get("conversationId"));
+      Integer messageId      = ClientErrors.id(body.get("messageId"));
+      Integer turnId         = ClientErrors.id(body.get("turnId"));
+      AssistantClientError row = new AssistantClientError()
+         .withUserId(userId)
+         .withScope(scope)
+         .withMessage(message)
+         .withErrorName(ClientErrors.token(body.get("errorName"), ClientErrors.ERROR_NAME_MAX))
+         .withStack(ClientErrors.text(body.get("stack"), ClientErrors.STACK_MAX))
+         .withComponentStack(ClientErrors.text(body.get("componentStack"), ClientErrors.COMPONENT_STACK_MAX))
+         .withRequestId(ClientErrors.token(body.get("requestId"), ClientErrors.REQUEST_ID_MAX))
+         .withRoute(ClientErrors.route(body.get("route")))
+         .withConversationId(conversationId != null && store.getConversation(userId, conversationId) != null ? conversationId : null)
+         .withMessageId(messageId != null && store.getMessage(userId, messageId) != null ? messageId : null)
+         .withTurnId(turnId != null && store.getTurn(userId, turnId) != null ? turnId : null)
+         .withPartType(ClientErrors.token(body.get("partType"), ClientErrors.PART_TYPE_MAX))
+         .withRenderType(ClientErrors.token(body.get("renderType"), ClientErrors.RENDER_TYPE_MAX))
+         .withAppVersion(ClientErrors.token(body.get("appVersion"), ClientErrors.APP_VERSION_MAX))
+         .withOccurredAt(runtime.getClientErrors().occurredAt(body.get("occurredAt")));
+      AssistantClientError saved = store.insert(row);
+
+      //////////////////////////////////////////////////////////////////////
+      // structured, no reported text: the row holds the (scrubbed) text //
+      //////////////////////////////////////////////////////////////////////
+      LOG.info("Assistant client error reported", logPair("clientErrorId", saved.getId()), logPair("scope", row.getScope()),
+         logPair("partType", row.getPartType()), logPair("renderType", row.getRenderType()), logPair("conversationId", row.getConversationId()),
+         logPair("messageId", row.getMessageId()), logPair("turnId", row.getTurnId()));
+      return (Map.of("accepted", true));
+   }
+
+
+
    /*******************************************************************************
     ** GET /actions/{id}: one of the user's pending actions.
     *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/api/specs/AssistantSpecs.java

@@ -37,9 +37,11 @@ import com.kingsrook.qqq.middleware.javalin.specs.BasicResponse;
 import com.kingsrook.qqq.middleware.javalin.specs.extensions.AbstractJsonEndpointSpec;
 import com.kingsrook.qqq.middleware.javalin.specs.extensions.MiddlewareSpecExtension;
 import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials;
+import com.kingsrook.qqq.middleware.javalin.specs.v1.responses.BasicErrorResponseV1;
 import com.kingsrook.qqq.openapi.model.HttpMethod;
 import com.kingsrook.qqq.openapi.model.In;
 import com.kingsrook.qqq.openapi.model.Parameter;
+import com.kingsrook.qqq.openapi.model.Response;
 import com.kingsrook.qqq.openapi.model.Schema;
 import com.kingsrook.qqq.openapi.model.Type;
 import io.javalin.http.ContentType;
@@ -54,14 +56,14 @@ import io.javalin.http.HttpStatus;
  ** routes, with the same URLs and bodies, so they appear in the served
  ** /qqq/v1/openapi.yaml and a generated client covers them.
  **
- ** Errors keep the assistant's shape: HTTP status plus {"error", "code"}.
+ ** Errors use BasicErrorResponseV1 and retain assistant-specific codes.
  ** Turns and events are text/event-stream; each event type is documented as
  ** AssistantTurnEventV1.
  *******************************************************************************/
 public final class AssistantSpecs
 {
    public static final String EXTENSION_NAME    = "assistant";
-   public static final String EXTENSION_VERSION = "1.2.0";
+   public static final String EXTENSION_VERSION = "1.5.0";
    public static final String PREFIX            = "/assistant";
 
 
@@ -99,7 +101,8 @@ public final class AssistantSpecs
          .withSpec(GetActionSpec.class)
          .withSpec(ConfirmActionSpec.class)
          .withSpec(RejectActionSpec.class)
-         .withSpec(AccessSpec.class));
+         .withSpec(AccessSpec.class)
+         .withSpec(ClientErrorsSpec.class));
    }
 
 
@@ -208,9 +211,31 @@ public final class AssistantSpecs
       rs.put("AssistantMessageV1", object("A message: the user's text or the assistant's reply as parts (text, tool, record_refs, navigate, notice).",
          "id", integer(null), "role", string("USER or ASSISTANT."), "status", string(null), "turnId", integer(null).withNullable(true),
          "createDate", string(null).withNullable(true), "parts", array(freeObject("A message part."), null)));
+      rs.put("AssistantRecordsRenderV1", object("A tool part's render of type records (query results). columns and rows are always present - empty when the query matched nothing (since 1.4.0; older stored parts are completed when read).",
+         "type", string("records"), "tableName", string(null), "tableLabel", string(null),
+         "columns", array(object("A column.", "name", string(null), "label", string(null)), "Always present; empty when nothing matched."),
+         "rows", array(object("A row.", "id", string("The primary key."), "href", string("The record's /app link."), "values", freeObject("Display values by column name.")), "Always present; empty when nothing matched."),
+         "totalCount", integer("All matching rows.").withNullable(true), "truncated", bool("More rows match than are shown."), "openHref", string("The Query screen link.")));
       rs.put("AssistantConversationDetailResponseV1", object("A conversation and a page of its messages, oldest first.",
          "conversation", ref("AssistantConversationV1"), "messages", array(ref("AssistantMessageV1"), null)));
       rs.put("AssistantStatusResponseV1", object("An acknowledgement.", "status", string(null)));
+      rs.put("AssistantClientErrorReportV1", object("An error the client hit while showing the assistant (since 1.5.0). scope and message are required; "
+            + "longer values are truncated, message and stacks are scrubbed of e-mail addresses, long numbers, tokens and query strings, and ids "
+            + "are kept only when they are the user's own.",
+         "scope", string("Required. Where it happened: assistant-panel, assistant-message, assistant-part, assistant-render, assistant-stream, widget, record-section, page...").withMaxLength(64),
+         "message", string("Required. The error message.").withMaxLength(500),
+         "errorName", string("The error's name, e.g. TypeError.").withMaxLength(100).withNullable(true),
+         "stack", string("The JavaScript stack.").withMaxLength(4000).withNullable(true),
+         "componentStack", string("The React component stack.").withMaxLength(4000).withNullable(true),
+         "requestId", string("The request or trace id the client saw.").withMaxLength(128).withNullable(true),
+         "route", string("The page path (no host, query or fragment).").withMaxLength(300).withNullable(true),
+         "conversationId", integer(null).withNullable(true), "messageId", integer(null).withNullable(true), "turnId", integer(null).withNullable(true),
+         "partType", string("The message part type being shown.").withMaxLength(64).withNullable(true),
+         "renderType", string("The render type being shown.").withMaxLength(64).withNullable(true),
+         "appVersion", string("The client build.").withMaxLength(64).withNullable(true),
+         "occurredAt", string("ISO-8601 instant.").withFormat("date-time").withNullable(true)));
+      rs.put("AssistantClientErrorResponseV1", object("Whether the report was stored: false when the user's reports are over the per-minute limit (still 202).",
+         "accepted", bool(null)));
       rs.put("AssistantActionV1", object("A write the assistant proposed and its state.",
          "id", integer(null), "status", string("PENDING, RUNNING, EXECUTED, FAILED, REJECTED or EXPIRED."), "toolName", string(null), "tableName", string(null),
          "recordId", string(null).withNullable(true), "summary", string(null), "httpStatus", integer("The UI API status of the write.").withNullable(true),
@@ -227,7 +252,7 @@ public final class AssistantSpecs
       rs.put("AssistantTurnEventV1", object("One server-sent event of a turn (event: <type>, id: <seq>, data: this object). Types: message.start, turn.phase, text.delta, tool.start, tool.result, ui.navigate, notice, usage, message.end, error, ping.",
          "seq", integer("Sequence number; resume with ?after=<seq>."), "type", string("The event type (also the SSE event name)."),
          "data", freeObject("The event's fields, by type. message.start includes startedAt (ISO-8601). turn.phase includes phase (thinking, tool, writing), optional toolName and label.")));
-      rs.put("AssistantErrorV1", object("An error.", "error", string("A message for the user."), "code", string("BUDGET, RATE_LIMIT, NOT_CONFIGURED, KILL_SWITCH, DISABLED, NO_PERMISSION...").withNullable(true)));
+      rs.put("AssistantErrorV1", object("Deprecated since 1.3.0: errors use BasicErrorResponseV1.", "error", string("A message for the user."), "code", string("BUDGET, RATE_LIMIT, NOT_CONFIGURED, KILL_SWITCH, DISABLED, NO_PERMISSION...").withNullable(true)));
       return (rs);
    }
 
@@ -260,7 +285,7 @@ public final class AssistantSpecs
    /*******************************************************************************
     ** Base of the assistant's JSON specs: the user's own AssistantApi (with the
     ** request's credentials, forwarded to the UI API by the tools), errors in
-    ** the assistant's {"error","code"} shape with their status.
+    ** the standard error shape with their status and assistant-specific code.
     *******************************************************************************/
    public abstract static class AssistantJsonSpec extends AbstractJsonEndpointSpec
    {
@@ -336,6 +361,26 @@ public final class AssistantSpecs
 
 
 
+      /*******************************************************************************
+       ** A bodyless response has empty content. Core 1.6's schema traversal
+       ** requires a non-null map even when the response has no schema.
+       *******************************************************************************/
+      @Override
+      public Map<Integer, Response> defineResponses()
+      {
+         Map<Integer, Response> responses = super.defineResponses();
+         for(Response response : responses.values())
+         {
+            if(response.getContent() == null)
+            {
+               response.withContent(Map.of());
+            }
+         }
+         return (responses);
+      }
+
+
+
       /*******************************************************************************
        ** Keep the request (for its credentials and the server's local port).
        *******************************************************************************/
@@ -403,7 +448,7 @@ public final class AssistantSpecs
 
 
       /*******************************************************************************
-       ** Assistant errors keep their status and {"error","code"}; others go
+       ** Assistant errors keep their status and code in the standard shape; others go
        ** through QQQ's mapping (400 / 401 / 403 / 404 / 500).
        *******************************************************************************/
       @Override
@@ -413,15 +458,16 @@ public final class AssistantSpecs
          AssistantApiException apiException = ExceptionUtils.findClassInRootChain(e, AssistantApiException.class);
          if(apiException != null)
          {
-            Map<String, Object> body = new LinkedHashMap<>();
-            body.put("error", apiException.getMessage());
+            context.contentType(ContentType.APPLICATION_JSON);
+            QJavalinUtils.respondWithError(context, org.eclipse.jetty.http.HttpStatus.getCode(apiException.getStatus()), apiException.getMessage());
             if(apiException.getCode() != null)
             {
-               body.put("code", apiException.getCode());
+               context.result(JsonUtils.toJson(new BasicErrorResponseV1()
+                  .withError(apiException.getMessage())
+                  .withCode(apiException.getCode())
+                  .withTraceId(QJavalinUtils.getTraceId(context))
+                  .toMap()));
             }
-            context.status(apiException.getStatus());
-            context.contentType(ContentType.APPLICATION_JSON);
-            context.result(JsonUtils.toJson(body));
             return;
          }
          QJavalinUtils.handleException(null, context, e);
@@ -436,12 +482,12 @@ public final class AssistantSpecs
       public List<BasicResponse> defineAdditionalBasicResponses()
       {
          return (List.of(
-            new BasicResponse(HttpStatus.BAD_REQUEST, "Invalid input", "AssistantErrorV1"),
-            new BasicResponse(HttpStatus.FORBIDDEN, "No access to the assistant", "AssistantErrorV1"),
-            new BasicResponse(HttpStatus.NOT_FOUND, "Not found (or not the user's own)", "AssistantErrorV1"),
-            new BasicResponse(HttpStatus.CONFLICT, "Already finished or decided", "AssistantErrorV1"),
-            new BasicResponse(HttpStatus.TOO_MANY_REQUESTS, "Budget or rate limit", "AssistantErrorV1"),
-            new BasicResponse(HttpStatus.SERVICE_UNAVAILABLE, "Disabled, not configured, or the kill switch is on", "AssistantErrorV1")));
+            new BasicResponse(HttpStatus.BAD_REQUEST, "Invalid input", BasicErrorResponseV1.class.getSimpleName()),
+            new BasicResponse(HttpStatus.FORBIDDEN, "No access to the assistant", BasicErrorResponseV1.class.getSimpleName()),
+            new BasicResponse(HttpStatus.NOT_FOUND, "Not found (or not the user's own)", BasicErrorResponseV1.class.getSimpleName()),
+            new BasicResponse(HttpStatus.CONFLICT, "Already finished or decided", BasicErrorResponseV1.class.getSimpleName()),
+            new BasicResponse(HttpStatus.TOO_MANY_REQUESTS, "Budget or rate limit", BasicErrorResponseV1.class.getSimpleName()),
+            new BasicResponse(HttpStatus.SERVICE_UNAVAILABLE, "Disabled, not configured, or the kill switch is on", BasicErrorResponseV1.class.getSimpleName())));
       }
 
 
@@ -1361,6 +1407,89 @@ public final class AssistantSpecs
 
 
 
+   /*******************************************************************************
+    ** POST /assistant/client-errors (since 1.5.0)
+    *******************************************************************************/
+   public static class ClientErrorsSpec extends AssistantJsonSpec
+   {
+      /*******************************************************************************
+       ** HTTP method.
+       *******************************************************************************/
+      @Override
+      protected HttpMethod method()
+      {
+         return (HttpMethod.POST);
+      }
+
+
+
+      /*******************************************************************************
+       ** Route path.
+       *******************************************************************************/
+      @Override
+      protected String path()
+      {
+         return (PREFIX + "/client-errors");
+      }
+
+
+
+      /*******************************************************************************
+       ** Operation summary.
+       *******************************************************************************/
+      @Override
+      protected String summary()
+      {
+         return ("Report an error the client hit while showing the assistant (stored for administrators, rate limited per user)");
+      }
+
+
+
+      /*******************************************************************************
+       ** Request body.
+       *******************************************************************************/
+      @Override
+      protected Schema defineRequestBodySchema()
+      {
+         return (ref("AssistantClientErrorReportV1"));
+      }
+
+
+
+      /*******************************************************************************
+       ** Response schema.
+       *******************************************************************************/
+      @Override
+      protected String getResponseSchemaName()
+      {
+         return ("AssistantClientErrorResponseV1");
+      }
+
+
+
+      /*******************************************************************************
+       ** 202: accepted (or over the limit and dropped).
+       *******************************************************************************/
+      @Override
+      protected HttpStatus successStatus()
+      {
+         return (HttpStatus.ACCEPTED);
+      }
+
+
+
+      /*******************************************************************************
+       ** Runs in the invoker's authenticated user session.
+       *******************************************************************************/
+      @Override
+      protected Object handle(AssistantApi api, JsonInput input) throws Exception
+      {
+         return (api.reportClientError(input.getBody()));
+      }
+   }
+
+
+
    /*******************************************************************************
     ** GET /assistant/access
     *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/AbandonedTurns.java

@@ -0,0 +1,194 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.engine;
+
+
+import java.time.Instant;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.model.AssistantMessage;
+import com.kingsrook.qbits.assistant.model.AssistantTurn;
+import com.kingsrook.qbits.assistant.model.AssistantTurnEvent;
+import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** Terminal state for runs whose engine is gone (a pod restart, a crash, a
+ ** finalizer that could not write): the turn row, its assistant message and a
+ ** terminal replay (error + message.end) are written, so no reader ever sees
+ ** an answer "in progress" forever and the "answer already in progress"
+ ** lockout cannot persist.
+ **
+ ** Used by the API when it reads an orphaned run (active turns, events,
+ ** conversation detail, the next question) and by the retention process,
+ ** which sweeps every run older than the hard stream limit. Only fixed
+ ** notices are appended to already stored (redacted) parts; nothing is
+ ** reloaded from the model, tools or the user's text.
+ *******************************************************************************/
+public final class AbandonedTurns
+{
+   private static final QLogger LOG = QLogger.getLogger(AbandonedTurns.class);
+
+   public static final String INTERRUPTED_TEXT = "This answer was interrupted (the server restarted). Ask again.";
+   public static final String UNFINISHED_TEXT  = "This answer did not finish. Ask again.";
+
+   private static final int BATCH = 200;
+
+
+
+   /*******************************************************************************
+    ** not instantiable
+    *******************************************************************************/
+   private AbandonedTurns()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Finalize one abandoned run as ERROR (no cancel reason: interrupted) or
+    ** CANCELLED (a stop nobody acknowledged). Each step is independent; the
+    ** turn row is written last and only while it still says it runs.
+    *******************************************************************************/
+   public static void finalizeRun(AssistantStore store, Instant now, String userId, AssistantTurn current, String cancelReason) throws QException
+   {
+      String status = cancelReason == null ? "ERROR" : "CANCELLED";
+      String text = cancelReason == null ? INTERRUPTED_TEXT
+         : ("SUPERSEDED".equals(cancelReason) ? TurnEngine.SUPERSEDED_TEXT : "Stopped by you.");
+      if(current.getAssistantMessageId() != null)
+      {
+         try
+         {
+            AssistantMessage message = store.getMessage(userId, current.getAssistantMessageId());
+            if(message != null)
+            {
+               store.updateFields(AssistantMessage.TABLE_NAME, new QRecord().withValue("id", message.getId()).withValue("status", status)
+                  .withValue("partsJson", MessageParts.fromJson(message.getPartsJson()).addNotice(cancelReason == null ? "error" : "info", text).toJson()));
+            }
+         }
+         catch(Exception e)
+         {
+            LOG.warn("Could not persist recovered assistant message", logPair("turnId", current.getId()));
+         }
+      }
+      try
+      {
+         int seq = store.lastEventSeq(userId, current.getId());
+         List<AssistantTurnEvent> last = store.listEvents(userId, current.getId(), Math.max(0, seq - 1), 1);
+         if(last.isEmpty() || !TurnEvent.MESSAGE_END.equals(last.getFirst().getEventType()))
+         {
+            if(cancelReason == null)
+            {
+               seq++;
+               store.insertEvent(current.getId(), userId, seq, TurnEvent.ERROR,
+                  new TurnEvent(seq, TurnEvent.ERROR, Map.of("code", "INTERNAL", "message", text)).toJson());
+            }
+            seq++;
+            Map<String, Object> end = new LinkedHashMap<>();
+            end.put("messageId", current.getAssistantMessageId());
+            end.put("status", status);
+            end.put("stopReason", cancelReason == null ? "interrupted" : "cancelled");
+            store.insertEvent(current.getId(), userId, seq, TurnEvent.MESSAGE_END, new TurnEvent(seq, TurnEvent.MESSAGE_END, end).toJson());
+         }
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Could not persist recovered assistant events", logPair("turnId", current.getId()));
+      }
+      store.updateRunningTurn(userId, new QRecord().withValue("id", current.getId()).withValue("status", status)
+         .withValue("errorCode", cancelReason == null ? "INTERNAL" : null).withValue("cancelReason", cancelReason)
+         .withValue("stopReason", cancelReason == null ? "interrupted" : "cancelled").withValue("phase", "done").withValue("endedAt", now));
+   }
+
+
+
+   /*******************************************************************************
+    ** An assistant message still STREAMING whose run is over (or gone): give it
+    ** the run's terminal status (ERROR when the run is unknown) and, unless the
+    ** run completed, an error notice. Returns the repaired message (or the
+    ** given one when nothing changed or the repair could not be written).
+    *******************************************************************************/
+   public static AssistantMessage repairMessage(AssistantStore store, String userId, AssistantMessage message, AssistantTurn turn)
+   {
+      if(message == null || !"STREAMING".equals(message.getStatus()))
+      {
+         return (message);
+      }
+      String status = turn == null || turn.getStatus() == null || "RUNNING".equals(turn.getStatus()) || "CANCELLING".equals(turn.getStatus()) ? "ERROR" : turn.getStatus();
+      try
+      {
+         MessageParts parts = MessageParts.fromJson(message.getPartsJson());
+         if(!"COMPLETE".equals(status))
+         {
+            parts.addNotice("ERROR".equals(status) ? "error" : "info", "ERROR".equals(status) ? UNFINISHED_TEXT : "Stopped.");
+         }
+         String partsJson = parts.toJson();
+         store.updateFields(AssistantMessage.TABLE_NAME, new QRecord().withValue("id", message.getId()).withValue("status", status).withValue("partsJson", partsJson));
+         message.setStatus(status);
+         message.setPartsJson(partsJson);
+         LOG.info("Repaired an unfinished assistant message", logPair("messageId", message.getId()), logPair("turnId", message.getTurnId()), logPair("status", status));
+      }
+      catch(Exception e)
+      {
+         LOG.warn("Could not repair an unfinished assistant message", logPair("messageId", message.getId()), logPair("turnId", message.getTurnId()));
+      }
+      return (message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Retention sweep: every run (of any user the running session can see)
+    ** still RUNNING or CANCELLING although it started before the hard stream
+    ** limit (maxTurnSeconds + force-finalize grace + lease) is finalized as
+    ** interrupted. Returns how many.
+    *******************************************************************************/
+   public static int sweep(AssistantQBitConfig config, AssistantStore store, Instant now) throws QException
+   {
+      Instant cutoff = now.minusSeconds(config.getMaxTurnSeconds() + (long) config.getTurnForceFinalizeGraceSeconds() + config.getTurnLeaseSeconds());
+      int     count  = 0;
+      Integer afterId = 0;
+      while(true)
+      {
+         List<AssistantTurn> batch = store.listRunningTurnsStartedBefore(cutoff, afterId, BATCH);
+         for(AssistantTurn turn : batch)
+         {
+            afterId = turn.getId();
+            try
+            {
+               finalizeRun(store, now, turn.getUserId(), turn, null);
+               count++;
+            }
+            catch(Exception e)
+            {
+               LOG.warn("Could not finalize an abandoned assistant turn", logPair("turnId", turn.getId()));
+            }
+         }
+         if(batch.size() < BATCH)
+         {
+            break;
+         }
+      }
+      return (count);
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ClientErrors.java

@@ -0,0 +1,226 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.engine;
+
+
+import java.time.Clock;
+import java.time.Instant;
+import java.time.OffsetDateTime;
+import java.util.ArrayDeque;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.regex.Pattern;
+import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
+
+
+/*******************************************************************************
+ ** Client error reports (POST /assistant/client-errors): the bounds of every
+ ** field, the scrubbing of free text, and the per-user rate limit.
+ **
+ ** Reported text may contain personal data (an error message can quote a
+ ** value), so message and stacks are scrubbed before they are stored: secrets
+ ** through the kernel's AiRedactor, then e-mail addresses, long digit runs
+ ** (more than 6 digits: phone, account and card numbers, ids), long opaque
+ ** tokens (32 or more letters and digits, mixed) and query strings. Short identifiers keep only printable characters.
+ ** A route keeps only its path.
+ *******************************************************************************/
+public class ClientErrors
+{
+   public static final int SCOPE_MAX           = 64;
+   public static final int MESSAGE_MAX         = 500;
+   public static final int ERROR_NAME_MAX      = 100;
+   public static final int STACK_MAX           = 4000;
+   public static final int COMPONENT_STACK_MAX = 4000;
+   public static final int REQUEST_ID_MAX      = 128;
+   public static final int ROUTE_MAX           = 300;
+   public static final int PART_TYPE_MAX       = 64;
+   public static final int RENDER_TYPE_MAX     = 64;
+   public static final int APP_VERSION_MAX     = 64;
+
+   private static final AiRedactor SECRETS = new AiRedactor(List.of());
+
+   private static final Pattern EMAIL        = Pattern.compile("[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}");
+   private static final Pattern LONG_DIGITS  = Pattern.compile("\\d(?:[\\s.-]?\\d){6,}");
+   private static final Pattern LONG_TOKEN   = Pattern.compile("(?=[A-Za-z0-9_+=-]*\\d)(?=[A-Za-z0-9_+=-]*[A-Za-z])[A-Za-z0-9_+=-]{32,}");
+   private static final Pattern QUERY_STRING = Pattern.compile("\\?[^\\s)\"']*");
+   private static final Pattern CONTROL      = Pattern.compile("[\\p{Cntrl}&&[^\\n\\t]]");
+   private static final Pattern NOT_TOKEN    = Pattern.compile("[^\\p{Graph} ]");
+
+   private final Clock                       clock;
+   private final Map<String, Deque<Instant>> windows = new ConcurrentHashMap<>();
+
+
+
+   /*******************************************************************************
+    ** Constructor.
+    *******************************************************************************/
+   public ClientErrors(Clock clock)
+   {
+      this.clock = clock;
+   }
+
+
+
+   /*******************************************************************************
+    ** Whether one more report from this user fits in the last minute (and
+    ** count it if so).
+    *******************************************************************************/
+   public boolean tryAcquire(String userId, int perMinute)
+   {
+      Instant        now    = clock.instant();
+      Deque<Instant> window = windows.computeIfAbsent(userId, key -> new ArrayDeque<>());
+      synchronized(window)
+      {
+         while(!window.isEmpty() && !window.peekFirst().isAfter(now.minusSeconds(60)))
+         {
+            window.pollFirst();
+         }
+         if(window.size() >= perMinute)
+         {
+            return (false);
+         }
+         window.addLast(now);
+         return (true);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Free text (message, stacks): secrets, e-mails, long numbers, tokens and
+    ** query strings removed, control characters dropped, then truncated.
+    ** Null for a non-string or blank value.
+    *******************************************************************************/
+   public static String text(Object value, int max)
+   {
+      if(!(value instanceof String text) || text.isBlank())
+      {
+         return (null);
+      }
+      String scrubbed = text.length() > max * 4 ? text.substring(0, max * 4) : text;
+      scrubbed = SECRETS.redact(scrubbed, new HashSet<>());
+      scrubbed = EMAIL.matcher(scrubbed).replaceAll("[email]");
+      scrubbed = QUERY_STRING.matcher(scrubbed).replaceAll("?[query]");
+      scrubbed = LONG_TOKEN.matcher(scrubbed).replaceAll("[token]");
+      scrubbed = LONG_DIGITS.matcher(scrubbed).replaceAll("[number]");
+      scrubbed = CONTROL.matcher(scrubbed).replaceAll("").strip();
+      return (truncate(scrubbed, max));
+   }
+
+
+
+   /*******************************************************************************
+    ** A short identifier (scope, error name, request id, types, version):
+    ** printable characters only, truncated. Null for a non-string or blank value.
+    *******************************************************************************/
+   public static String token(Object value, int max)
+   {
+      if(!(value instanceof String text))
+      {
+         return (null);
+      }
+      String cleaned = NOT_TOKEN.matcher(text).replaceAll("").strip();
+      cleaned = EMAIL.matcher(cleaned).replaceAll("[email]");
+      return (cleaned.isEmpty() ? null : truncate(cleaned, max));
+   }
+
+
+
+   /*******************************************************************************
+    ** The path of a route: scheme, host, query and fragment removed; long
+    ** numbers and e-mails scrubbed. Null unless it is a path ("/...").
+    *******************************************************************************/
+   public static String route(Object value)
+   {
+      if(!(value instanceof String text))
+      {
+         return (null);
+      }
+      String path = text.strip().replaceFirst("^[A-Za-z][A-Za-z0-9+.-]*://[^/]*", "");
+      int    cut  = path.indexOf('?');
+      path = cut < 0 ? path : path.substring(0, cut);
+      cut = path.indexOf('#');
+      path = cut < 0 ? path : path.substring(0, cut);
+      if(!path.startsWith("/") || path.startsWith("//"))
+      {
+         return (null);
+      }
+      path = NOT_TOKEN.matcher(path).replaceAll("").replace(" ", "");
+      path = EMAIL.matcher(path).replaceAll("[email]");
+      path = LONG_DIGITS.matcher(path).replaceAll("[number]");
+      return (truncate(path, ROUTE_MAX));
+   }
+
+
+
+   /*******************************************************************************
+    ** A positive integer id, or null.
+    *******************************************************************************/
+   public static Integer id(Object value)
+   {
+      if(value instanceof Number number && number.doubleValue() == number.longValue() && number.longValue() > 0 && number.longValue() <= Integer.MAX_VALUE)
+      {
+         return (number.intValue());
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    ** An ISO-8601 instant (with Z or an offset), or null; times in the future
+    ** become now.
+    *******************************************************************************/
+   public Instant occurredAt(Object value)
+   {
+      if(!(value instanceof String text) || text.length() > 64)
+      {
+         return (null);
+      }
+      Instant instant;
+      try
+      {
+         instant = OffsetDateTime.parse(text.strip()).toInstant();
+      }
+      catch(RuntimeException e)
+      {
+         try
+         {
+            instant = Instant.parse(text.strip());
+         }
+         catch(RuntimeException e2)
+         {
+            return (null);
+         }
+      }
+      Instant now = clock.instant();
+      return (instant.isAfter(now) ? now : instant);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static String truncate(String value, int max)
+   {
+      return (value.length() <= max ? value : value.substring(0, max));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ConnectionTest.java

@@ -18,9 +18,11 @@ package com.kingsrook.qbits.assistant.engine;
 
 
 import java.time.Duration;
+import java.time.Instant;
 import java.util.ArrayList;
 import java.util.List;
 import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.AssistantSecurity;
 import com.kingsrook.qbits.assistant.model.AssistantModel;
 import com.kingsrook.qbits.assistant.model.AssistantProvider;
 import com.kingsrook.qbits.assistant.provider.RegistryModelProvider;
@@ -34,6 +36,7 @@ import com.kingsrook.qbits.assistant.spi.ModelRequest;
 import com.kingsrook.qbits.assistant.spi.ModelResponse;
 import com.kingsrook.qbits.assistant.spi.StreamHandle;
 import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 
@@ -124,6 +127,16 @@ public final class ConnectionTest
     ** Probe health, then make and account for a minimal completion.
     *******************************************************************************/
    private static Result call(AssistantRuntime runtime, ProviderService.Active active, String userId)
+   {
+      return (call(runtime, active, userId, Instant.now().plusSeconds(runtime.getConfig().getProviderTestTimeoutSeconds()), true));
+   }
+
+
+
+   /*******************************************************************************
+    ** Share the minimal call and TEST accounting with central model health.
+    *******************************************************************************/
+   private static Result call(AssistantRuntime runtime, ProviderService.Active active, String userId, Instant deadline, Boolean probe)
    {
       List<ModelMessage> messages = new ArrayList<>();
       messages.add(ModelMessage.user(PING_PROMPT));
@@ -136,19 +149,27 @@ public final class ConnectionTest
       UsageRecorder.Call usageCall = new UsageRecorder.Call(userId, null, null, active, UsageRecorder.PURPOSE_TEST);
       ModelPricing       pricing   = runtime.getProviderService().pricing(active, active.model());
       long               started   = System.nanoTime();
+      StreamHandle handle = null;
       try
       {
-         AssistantProviderType type = runtime.getConfig().getProviderTypes().get(active.providerType());
-         HealthResult health = type.checkHealth(runtime.getProviderService().settings(active.row()));
-         if(Boolean.FALSE.equals(health.healthy()))
+         if(probe)
          {
-            throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, health.message()));
+            AssistantProviderType type = runtime.getConfig().getProviderTypes().get(active.providerType());
+            HealthResult health = type.checkHealth(runtime.getProviderService().settings(active.row()));
+            if(Boolean.FALSE.equals(health.healthy()))
+            {
+               throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, health.message()));
+            }
          }
-         StreamHandle  handle   = active.provider().stream(request, text ->
+         if(!Instant.now().isBefore(deadline) || Thread.currentThread().isInterrupted())
+         {
+            return (new Result(STATUS_ERROR, null, null, "The test deadline passed."));
+         }
+         handle = active.provider().stream(request, text ->
          {
             // the reply text is not used
          });
-         ModelResponse response = handle.await(Duration.ofSeconds(runtime.getConfig().getProviderTestTimeoutSeconds()));
+         ModelResponse response = handle.await(Duration.between(Instant.now(), deadline));
          long          latency  = (System.nanoTime() - started) / 1_000_000;
          UsageRecorder.save(runtime.getStore(), UsageRecorder.row(usageCall, response, null, latency, pricing));
          if(ModelResponse.STOP_CANCELLED.equals(response.getStopReason()))
@@ -170,8 +191,45 @@ public final class ConnectionTest
       catch(InterruptedException e)
       {
          Thread.currentThread().interrupt();
+         UsageRecorder.save(runtime.getStore(), UsageRecorder.row(usageCall, null,
+            new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "The test was interrupted."),
+            (System.nanoTime() - started) / 1_000_000, pricing));
          return (new Result(STATUS_ERROR, null, null, "The test was interrupted."));
       }
+      finally
+      {
+         if(handle != null)
+         {
+            handle.cancel();
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Central health uses the same probe without changing manual lastTest fields.
+    ** Purpose is a string, so health calls reuse TEST rather than adding an enum.
+    *******************************************************************************/
+   public static Result checkModel(AssistantRuntime runtime, AssistantModel model, Instant deadline) throws QException
+   {
+      if(!Instant.now().isBefore(deadline) || Thread.currentThread().isInterrupted())
+      {
+         return (new Result(STATUS_ERROR, null, null, "The test deadline passed."));
+      }
+      if(!model.isChat())
+      {
+         return (testEmbedding(runtime, model));
+      }
+      AssistantProvider row = runtime.getStore().getProvider(model.getProviderId());
+      var type = runtime.getConfig().getProviderTypes().get(row.getProviderType());
+      var settings = runtime.getProviderService().settings(row);
+      int seconds = (int) Math.max(1, Math.min(30, Duration.between(Instant.now(), deadline).toSeconds()));
+      settings.withConnectTimeoutSeconds(Math.min(settings.getConnectTimeoutSeconds(), seconds))
+         .withRequestTimeoutSeconds(Math.min(settings.getRequestTimeoutSeconds(), seconds));
+      var provider = new RegistryModelProvider(type.create(settings), model);
+      return (call(runtime, new ProviderService.Active(row, row.getProviderType(), model.getModelId(), provider),
+         AssistantSecurity.userId(QContext.getQSession()), deadline, false));
    }
 
 
@@ -246,7 +304,7 @@ public final class ConnectionTest
       try
       {
          new EmbeddingService(runtime).embed(model.getId(), List.of("Connection test."),
-            DataClassification.PUBLIC, "", null, null);
+            DataClassification.PUBLIC, "", null, null, UsageRecorder.PURPOSE_TEST);
          return (new Result(STATUS_OK, (int) ((System.nanoTime() - started) / 1_000_000), model.getModelId(), null));
       }
       catch(QException e)

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/MessageParts.java

@@ -50,11 +50,12 @@ public class MessageParts
       {
          try
          {
-            List<Map<String, Object>> list = JsonUtils.toObject(json, new TypeReference<List<Map<String, Object>>>() {});
-            if(list != null)
-            {
-               result.parts.addAll(list);
-            }
+            List<Object> list = JsonUtils.toObject(json, new TypeReference<List<Object>>() {});
+            //////////////////////////////////////////////////////////////////////
+            // stored parts are repaired on read: older rows may lack empty     //
+            // render lists, and a part that cannot be shown becomes a notice   //
+            //////////////////////////////////////////////////////////////////////
+            result.parts.addAll(RenderShapes.parts(list));
          }
          catch(Exception e)
          {
@@ -240,7 +241,7 @@ public class MessageParts
     *******************************************************************************/
    public synchronized String toJson()
    {
-      return (JsonUtils.toJson(parts));
+      return (RenderShapes.toJson(RenderShapes.parts(parts)));
    }
 
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ProviderHealth.java

@@ -27,6 +27,8 @@ import com.kingsrook.qbits.assistant.spi.HealthResult;
 import com.kingsrook.qbits.assistant.spi.ModelProviderException;
 import com.kingsrook.qbits.assistant.spi.ProviderSettings;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthRunner;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 
 
@@ -72,6 +74,21 @@ public class ProviderHealth
       {
          return;
       }
+      HealthRunner runner = HealthRunner.getForInstance(runtime.getQInstance());
+      String checkName = "assistant.provider." + row.getId();
+      Instant checked = runner == null ? null : runner.getLastOutcomeAt(checkName);
+      var outcome = runner == null ? null : runner.getLastOutcome(checkName);
+      if(checked != null && !checked.isAfter(now) && checked.plusSeconds(60).isAfter(now) && outcome != null)
+      {
+         if(outcome.getStatus() == HealthStatus.DOWN)
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, "Provider health check failed."));
+         }
+         if(outcome.getStatus() == HealthStatus.UP || outcome.getStatus() == HealthStatus.DEGRADED)
+         {
+            return;
+         }
+      }
       ProviderSettings settings = runtime.getProviderService().settings(row).withConnectTimeoutSeconds(2).withRequestTimeoutSeconds(2);
       FutureTask<HealthResult> probe = new FutureTask<>(() -> runtime.getConfig().getProviderTypes().get(row.getProviderType()).checkHealth(settings));
       Thread.ofVirtual().name("assistant-health").start(probe);

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ProviderService.java

@@ -481,13 +481,23 @@ public class ProviderService
     ** micro-dollars, from the usage rows the session can see.
     *******************************************************************************/
    public long monthToDateCostMicros(Integer providerId) throws QException
+   {
+      return (monthToDateCostMicros(providerId, false));
+   }
+
+
+
+   /*******************************************************************************
+    ** Shared accounting permits health-runner caps without granting owner access.
+    *******************************************************************************/
+   public long monthToDateCostMicros(Integer providerId, Boolean sharedAccounting) throws QException
    {
       if(providerId == null)
       {
          return (0);
       }
       Instant monthStart = LocalDate.now(clock.withZone(ZoneOffset.UTC)).withDayOfMonth(1).atStartOfDay().toInstant(ZoneOffset.UTC);
-      return (store.sumUsageCostSince(providerId, monthStart));
+      return (store.sumUsageCostSince(providerId, monthStart, sharedAccounting));
    }
 
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/RenderShapes.java

@@ -0,0 +1,502 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.engine;
+
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.kingsrook.qqq.backend.core.logging.QLogger;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
+
+
+/*******************************************************************************
+ ** The shapes clients rely on for message parts, tool result renders and
+ ** presentations (extension "assistant", AssistantMessageV1 parts and the
+ ** tool.result / presentation events). Every part, render and presentation
+ ** is checked before it is persisted or streamed, and again when stored
+ ** parts and events are read back (older rows are repaired on read):
+ **
+ ** - required lists are always present, empty when there is nothing (a query
+ **   that matched no rows has "rows": [] and "columns": []);
+ ** - list entries that are not objects are dropped, required strings are
+ **   filled with safe defaults;
+ ** - a render that cannot be repaired (no type, a pending action without an
+ **   action id, a presentation with nothing to show) is dropped and a notice
+ **   part ({@link #UNDISPLAYABLE_TEXT}) takes its place;
+ ** - unknown part and render types pass through untouched.
+ **
+ ** Problems are logged with the render type and a fixed problem code only;
+ ** never with values. QQQ's default JSON omits empty lists, so parts and
+ ** stream events are written with {@link #toJson(Object)}.
+ *******************************************************************************/
+public final class RenderShapes
+{
+   private static final QLogger LOG = QLogger.getLogger(RenderShapes.class);
+
+   /** what the user sees instead of a result that cannot be shown */
+   public static final String UNDISPLAYABLE_TEXT = "This result could not be displayed.";
+
+   private static final Pattern    SAFE_TYPE     = Pattern.compile("[a-z][a-z0-9_.-]{0,63}");
+   private static final Set<String> NOTICE_LEVELS = Set.of("info", "warning", "error");
+   private static final Set<String> ITEM_BLOCKS   = Set.of("metrics", "filterLinks", "recordLinks", "actions");
+
+
+
+   /*******************************************************************************
+    ** not instantiable
+    *******************************************************************************/
+   private RenderShapes()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** JSON that keeps empty lists (nulls are still left out).
+    *******************************************************************************/
+   public static String toJson(Object value)
+   {
+      return (JsonUtils.toJsonCustomized(value, builder -> builder.serializationInclusion(JsonInclude.Include.NON_NULL)));
+   }
+
+
+
+   /*******************************************************************************
+    ** The notice part shown in place of a part or render that cannot be shown.
+    *******************************************************************************/
+   public static Map<String, Object> undisplayableNotice()
+   {
+      Map<String, Object> notice = new LinkedHashMap<>();
+      notice.put("type", "notice");
+      notice.put("level", "error");
+      notice.put("text", UNDISPLAYABLE_TEXT);
+      return (notice);
+   }
+
+
+
+   /*******************************************************************************
+    ** Normalized copies of message parts (never throws; never null). A part
+    ** that is not an object, or has no type, becomes an error notice; a tool
+    ** part whose render cannot be repaired loses its render and is followed by
+    ** an error notice; unknown part types pass through.
+    *******************************************************************************/
+   public static List<Map<String, Object>> parts(Object parts)
+   {
+      List<Map<String, Object>> result = new ArrayList<>();
+      if(!(parts instanceof List<?> list))
+      {
+         return (result);
+      }
+      for(Object entry : list)
+      {
+         try
+         {
+            addPart(result, entry);
+         }
+         catch(RuntimeException e)
+         {
+            problem(entry instanceof Map<?, ?> map ? map.get("type") : null, "part-exception");
+            result.add(undisplayableNotice());
+         }
+      }
+      return (result);
+   }
+
+
+
+   /*******************************************************************************
+    ** Normalize one part into the result.
+    *******************************************************************************/
+   private static void addPart(List<Map<String, Object>> result, Object entry)
+   {
+      if(!(entry instanceof Map<?, ?> map) || !(map.get("type") instanceof String type))
+      {
+         problem(null, "part-without-type");
+         result.add(undisplayableNotice());
+         return;
+      }
+      Map<String, Object> part = copy(map);
+      switch(type)
+      {
+         case "text" -> part.put("text", string(part.get("text"), ""));
+         case "notice" ->
+         {
+            part.put("level", part.get("level") instanceof String level && NOTICE_LEVELS.contains(level) ? level : "info");
+            part.put("text", string(part.get("text"), ""));
+         }
+         case "navigate" ->
+         {
+            if(!(part.get("href") instanceof String href) || !TurnEngine.isAppHref(href))
+            {
+               problem(type, "navigate-without-app-href");
+               return;
+            }
+         }
+         case "record_refs" -> part.put("refs", objects(part.get("refs")));
+         case "tool" ->
+         {
+            part.put("name", string(part.get("name"), "tool"));
+            part.put("summary", string(part.get("summary"), ""));
+            part.put("outcome", string(part.get("outcome"), "ERROR"));
+            if(part.containsKey("render"))
+            {
+               Map<String, Object> render = render(part.get("render"));
+               if(render == null)
+               {
+                  part.remove("render");
+                  result.add(part);
+                  result.add(undisplayableNotice());
+                  return;
+               }
+               part.put("render", render);
+            }
+         }
+         case "presentation" ->
+         {
+            Map<String, Object> presentation = presentation(part);
+            result.add(presentation == null ? undisplayableNotice() : presentation);
+            return;
+         }
+         default ->
+         {
+            //////////////////////////////////////////////////
+            // unknown part types pass through untouched    //
+            //////////////////////////////////////////////////
+         }
+      }
+      result.add(part);
+   }
+
+
+
+   /*******************************************************************************
+    ** A normalized copy of a tool result render, or null when it cannot be
+    ** repaired (the problem is logged without values). Unknown render types
+    ** pass through as a shallow copy.
+    *******************************************************************************/
+   public static Map<String, Object> render(Object value)
+   {
+      try
+      {
+         if(!(value instanceof Map<?, ?> map) || !(map.get("type") instanceof String type))
+         {
+            problem(null, "render-without-type");
+            return (null);
+         }
+         Map<String, Object> render = copy(map);
+         switch(type)
+         {
+            case "records" ->
+            {
+               tableLabel(render);
+               List<Map<String, Object>> columns = new ArrayList<>();
+               for(Map<String, Object> column : objects(render.get("columns")))
+               {
+                  if(column.get("name") instanceof String name)
+                  {
+                     column.put("label", string(column.get("label"), name));
+                     columns.add(column);
+                  }
+               }
+               render.put("columns", columns);
+               List<Map<String, Object>> rows = objects(render.get("rows"));
+               for(Map<String, Object> row : rows)
+               {
+                  row.put("values", row.get("values") instanceof Map<?, ?> values ? copy(values) : new LinkedHashMap<>());
+               }
+               render.put("rows", rows);
+               render.put("totalCount", number(render.get("totalCount")));
+               render.put("truncated", Boolean.TRUE.equals(render.get("truncated")));
+            }
+            case "record" ->
+            {
+               tableLabel(render);
+               List<Map<String, Object>> fields = new ArrayList<>();
+               for(Map<String, Object> field : objects(render.get("fields")))
+               {
+                  if(field.get("name") instanceof String name)
+                  {
+                     field.put("label", string(field.get("label"), name));
+                     Object fieldValue = field.get("value");
+                     field.put("value", fieldValue == null || fieldValue instanceof String ? fieldValue : String.valueOf(fieldValue));
+                     fields.add(field);
+                  }
+               }
+               render.put("fields", fields);
+            }
+            case "count" ->
+            {
+               tableLabel(render);
+               render.put("count", number(render.get("count")));
+            }
+            case "widget" ->
+            {
+               render.put("label", string(render.get("label"), string(render.get("widgetName"), "Widget")));
+               render.put("data", render.get("data") instanceof Map<?, ?> data ? copy(data) : new LinkedHashMap<>());
+            }
+            case "history" ->
+            {
+               List<Map<String, Object>> entries = objects(render.get("entries"));
+               for(Map<String, Object> entry : entries)
+               {
+                  entry.put("details", objects(entry.get("details")));
+               }
+               render.put("entries", entries);
+            }
+            case "pending_action" ->
+            {
+               Object actionId = number(render.get("actionId"));
+               if(actionId == null)
+               {
+                  problem(type, "pending-action-without-id");
+                  return (null);
+               }
+               render.put("actionId", actionId);
+               render.put("summary", string(render.get("summary"), ""));
+               render.put("changes", objects(render.get("changes")));
+            }
+            case "bulk_load_dry_run" -> render.put("errors", objects(render.get("errors")));
+            default ->
+            {
+               //////////////////////////////////////////////////
+               // unknown render types pass through untouched  //
+               //////////////////////////////////////////////////
+            }
+         }
+         return (render);
+      }
+      catch(RuntimeException e)
+      {
+         problem(value instanceof Map<?, ?> map ? map.get("type") : null, "render-exception");
+         return (null);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A normalized copy of a presentation part, or null when it has nothing to
+    ** show. Blocks that are not objects or have no kind are dropped; table,
+    ** chart and item blocks always carry their lists.
+    *******************************************************************************/
+   public static Map<String, Object> presentation(Object value)
+   {
+      try
+      {
+         if(!(value instanceof Map<?, ?> map))
+         {
+            problem("presentation", "presentation-not-object");
+            return (null);
+         }
+         Map<String, Object> part = copy(map);
+         part.put("type", "presentation");
+         part.put("summary", string(part.get("summary"), ""));
+         List<Map<String, Object>> blocks = new ArrayList<>();
+         for(Map<String, Object> block : objects(part.get("blocks")))
+         {
+            if(!(block.get("kind") instanceof String kind))
+            {
+               problem("presentation", "block-without-kind");
+               continue;
+            }
+            if("table".equals(kind))
+            {
+               block.put("title", string(block.get("title"), ""));
+               List<Map<String, Object>> columns = new ArrayList<>();
+               for(Map<String, Object> column : objects(block.get("columns")))
+               {
+                  if(column.get("key") instanceof String key)
+                  {
+                     column.put("label", string(column.get("label"), key));
+                     columns.add(column);
+                  }
+               }
+               block.put("columns", columns);
+               List<Map<String, Object>> rows = objects(block.get("rows"));
+               for(Map<String, Object> row : rows)
+               {
+                  row.put("cells", row.get("cells") instanceof Map<?, ?> cells ? copy(cells) : new LinkedHashMap<>());
+                  row.put("display", row.get("display") instanceof Map<?, ?> display ? copy(display) : new LinkedHashMap<>());
+               }
+               block.put("rows", rows);
+            }
+            else if("chart".equals(kind))
+            {
+               block.put("title", string(block.get("title"), ""));
+               block.put("chartType", string(block.get("chartType"), "bar"));
+               block.put("points", objects(block.get("points")));
+            }
+            else if(ITEM_BLOCKS.contains(kind))
+            {
+               block.put("items", objects(block.get("items")));
+            }
+            blocks.add(block);
+         }
+         part.put("blocks", blocks);
+         part.put("markdown", string(part.get("markdown"), (String) part.get("summary")));
+         if(blocks.isEmpty() && ((String) part.get("summary")).isBlank() && ((String) part.get("markdown")).isBlank())
+         {
+            problem("presentation", "presentation-empty");
+            return (null);
+         }
+         return (part);
+      }
+      catch(RuntimeException e)
+      {
+         problem("presentation", "presentation-exception");
+         return (null);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Normalize an event's data in place: a tool.result render that cannot be
+    ** repaired is removed; a presentation event's part that cannot be shown is
+    ** replaced by the error notice part.
+    *******************************************************************************/
+   public static void normalizeEvent(String type, Map<String, Object> data)
+   {
+      if(data == null)
+      {
+         return;
+      }
+      if(data.containsKey("render"))
+      {
+         Map<String, Object> render = render(data.get("render"));
+         if(render == null)
+         {
+            data.remove("render");
+         }
+         else
+         {
+            data.put("render", render);
+         }
+      }
+      if(TurnEvent.PRESENTATION.equals(type) && data.containsKey("part"))
+      {
+         Map<String, Object> part = presentation(data.get("part"));
+         data.put("part", part == null ? undisplayableNotice() : part);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Fill in a single part (in place) - kept for callers of the 1.4.0 helper.
+    ** A render that cannot be repaired is removed.
+    *******************************************************************************/
+   public static Map<String, Object> complete(Map<String, Object> part)
+   {
+      if(part != null)
+      {
+         normalizeEvent(null, part);
+      }
+      return (part);
+   }
+
+
+
+   /*******************************************************************************
+    ** tableLabel defaults to the table name.
+    *******************************************************************************/
+   private static void tableLabel(Map<String, Object> render)
+   {
+      render.put("tableLabel", string(render.get("tableLabel"), string(render.get("tableName"), "")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Copies of the object entries of a list (others dropped); empty when the
+    ** value is not a list.
+    *******************************************************************************/
+   private static List<Map<String, Object>> objects(Object value)
+   {
+      List<Map<String, Object>> result = new ArrayList<>();
+      if(value instanceof List<?> list)
+      {
+         for(Object entry : list)
+         {
+            if(entry instanceof Map<?, ?> map)
+            {
+               result.add(copy(map));
+            }
+         }
+      }
+      return (result);
+   }
+
+
+
+   /*******************************************************************************
+    ** A mutable copy with string keys.
+    *******************************************************************************/
+   private static Map<String, Object> copy(Map<?, ?> map)
+   {
+      Map<String, Object> copy = new LinkedHashMap<>();
+      map.forEach((key, value) -> copy.put(String.valueOf(key), value));
+      return (copy);
+   }
+
+
+
+   /*******************************************************************************
+    ** The value when it is a string; otherwise the default.
+    *******************************************************************************/
+   private static String string(Object value, String defaultValue)
+   {
+      return (value instanceof String s ? s : defaultValue);
+   }
+
+
+
+   /*******************************************************************************
+    ** A number (numeric strings are converted), or null.
+    *******************************************************************************/
+   private static Number number(Object value)
+   {
+      if(value instanceof Number n)
+      {
+         return (n);
+      }
+      if(value instanceof String s && s.matches("-?\\d{1,18}"))
+      {
+         return (Long.parseLong(s));
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    ** Log a shape problem: the type (when it looks like a type) and a fixed
+    ** code, never values.
+    *******************************************************************************/
+   private static void problem(Object type, String code)
+   {
+      String safeType = type instanceof String s && SAFE_TYPE.matcher(s).matches() ? s : null;
+      LOG.warn("Assistant part or render was not displayable", logPair("renderType", safeType), logPair("problem", code));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/TurnEngine.java

@@ -29,12 +29,14 @@ import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
+import java.util.concurrent.CancellationException;
 import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.Future;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicLongArray;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
 import com.kingsrook.qbits.assistant.AssistantRuntime;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
@@ -67,6 +69,7 @@ import com.kingsrook.qqq.backend.core.actions.ai.AiToolCall;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
 import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
+import com.kingsrook.qqq.backend.core.context.CapturedContext;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
@@ -770,13 +773,28 @@ public class TurnEngine implements Runnable
       {
          throw (new BudgetService.Blocked("The assistant has reached this provider's monthly cost cap. Ask an administrator."));
       }
+      ///////////////////////////////////////////////////////////////////////
+      // providers stream on their own (virtual) threads, which have no     //
+      // QContext: run the delta handler in the turn's context, or progress //
+      // writes (JobLogWriter) fail with no instance                        //
+      ///////////////////////////////////////////////////////////////////////
+      CapturedContext turnContext = QContext.capture();
       ModelEventListener listener = new ModelEventListener()
       {
          @Override
          public void onTextDelta(String text)
          {
-            phase("writing", null);
-            bufferText(text);
+            if(QContext.getQInstance() != null)
+            {
+               phase("writing", null);
+               bufferText(text);
+               return;
+            }
+            QContext.withTemporaryContext(turnContext, () ->
+            {
+               phase("writing", null);
+               bufferText(text);
+            });
          }
       };
 
@@ -855,6 +873,10 @@ public class TurnEngine implements Runnable
       AiToolResult[]   results      = new AiToolResult[toolUses.size()];
       UiApiToolbox.Outcome[] outcomes = new UiApiToolbox.Outcome[toolUses.size()];
       List<Future<?>>  futures      = new ArrayList<>();
+      List<Integer>    futureIndexes = new ArrayList<>();
+      AtomicLongArray  startedNanos = new AtomicLongArray(toolUses.size());
+      boolean[]        failed       = new boolean[toolUses.size()];
+      boolean[]        timedOut     = new boolean[toolUses.size()];
       Semaphore        permits      = new Semaphore(Math.max(1, config.getMaxParallelToolCalls()));
 
       for(int i = 0; i < toolUses.size(); i++)
@@ -879,6 +901,7 @@ public class TurnEngine implements Runnable
 
          final int    index = i;
          AiToolCall   call  = new AiToolCall().withId(use.getToolUseId()).withName(use.getToolName()).withInput(use.getInput());
+         futureIndexes.add(index);
          futures.add(runtime.getExecutor().submit(() ->
          {
             if(!transcript.retainWorker())
@@ -890,6 +913,7 @@ public class TurnEngine implements Runnable
             {
                QContext.init(runtime.getQInstance(), session);
                permits.acquire();
+               startedNanos.set(index, System.nanoTime());
                try
                {
                   if(!running.isCancelled())
@@ -944,11 +968,15 @@ public class TurnEngine implements Runnable
             {
                Thread.currentThread().interrupt();
             }
-            catch(RuntimeException e)
+            catch(RuntimeException | Error e)
             {
-               LOG.warn("Tool kernel threw", hipaaSafe(e), logPair("tool", call.getName()));
-               results[index] = new AiToolResult().withToolCallId(call.getId()).withToolName(call.getName()).withOutcome(AiToolOutcome.ERROR)
-                  .withSummary("Failed").withModelContent("The tool failed unexpectedly.");
+               ///////////////////////////////////////////////////////////////////
+               // internal details go to the log only; the model and the user  //
+               // get a fixed, safe text                                       //
+               ///////////////////////////////////////////////////////////////////
+               LOG.warn("Tool kernel threw", hipaaSafe(e), logPair("turnId", turn.getId()), logPair("tool", call.getName()), logPair("errorClass", e.getClass().getSimpleName()));
+               failed[index] = true;
+               results[index] = failedResult(call.getId(), call.getName());
             }
             finally
             {
@@ -960,20 +988,39 @@ public class TurnEngine implements Runnable
       }
       persistPendingEvents();
 
-      for(Future<?> future : futures)
+      long toolTimeoutNanos = TimeUnit.SECONDS.toNanos(config.getToolCallTimeoutSeconds() == null ? 60 : config.getToolCallTimeoutSeconds());
+      for(int f = 0; f < futures.size(); f++)
       {
-         long waitMillis = Math.max(1, Duration.between(runtime.getClock().instant(), deadline).toMillis());
+         Future<?> future = futures.get(f);
+         int       index  = futureIndexes.get(f);
+         ///////////////////////////////////////////////////////////////////////
+         // each call has its own limit (from when it got a permit), and no  //
+         // call outlives the turn's deadline                                 //
+         ///////////////////////////////////////////////////////////////////////
+         long now        = System.nanoTime();
+         long started    = startedNanos.get(index);
+         long toolMillis = TimeUnit.NANOSECONDS.toMillis((started == 0 ? now : started) + toolTimeoutNanos - now);
+         long turnMillis = Duration.between(runtime.getClock().instant(), deadline).toMillis();
+         long waitMillis = Math.max(1, Math.min(toolMillis, turnMillis));
          try
          {
             future.get(waitMillis, TimeUnit.MILLISECONDS);
          }
          catch(TimeoutException e)
          {
+            timedOut[index] = !future.isDone();
             future.cancel(true);
+            LOG.warn("Assistant tool call timed out", logPair("turnId", turn.getId()), logPair("tool", toolUses.get(index).getToolName()),
+               logPair("limitSeconds", config.getToolCallTimeoutSeconds()), logPair("turnDeadline", toolMillis > turnMillis));
          }
          catch(ExecutionException e)
          {
-            LOG.debug("tool future failed", hipaaSafe(e));
+            failed[index] = true;
+            LOG.warn("Assistant tool call failed", hipaaSafe(e), logPair("turnId", turn.getId()), logPair("tool", toolUses.get(index).getToolName()));
+         }
+         catch(CancellationException e)
+         {
+            LOG.debug("tool future cancelled", logPair("turnId", turn.getId()));
          }
       }
 
@@ -987,15 +1034,30 @@ public class TurnEngine implements Runnable
             continue;
          }
 
-         AiToolResult result = results[i];
-         if(result == null)
+         AiToolResult         result = timedOut[i] ? null : results[i];
+         UiApiToolbox.Outcome extras = timedOut[i] ? null : outcomes[i];
+         if(result == null && failed[i])
          {
+            result = failedResult(use.getToolUseId(), use.getToolName());
+         }
+         else if(result == null)
+         {
+            boolean stopped = running.isCancelled() && !timedOut[i];
             result = new AiToolResult().withToolCallId(use.getToolUseId()).withToolName(use.getToolName()).withOutcome(AiToolOutcome.ERROR)
-               .withSummary(running.isCancelled() ? "Not run (stopped)" : "Timed out")
-               .withModelContent(running.isCancelled() ? "Not run: the user stopped the answer." : "The tool did not finish in time.");
+               .withSummary(stopped ? "Not run (stopped)" : "Timed out")
+               .withModelContent(stopped ? "Not run: the user stopped the answer." : "The tool did not finish in time. Answer with what you have, or try a narrower request.");
+            timedOut[i] = !stopped;
          }
          classification.observe(use, result);
-         recordToolResult(use, result, outcomes[i]);
+         recordToolResult(use, result, extras);
+         if(timedOut[i])
+         {
+            notice("error", "A step took too long and was stopped (" + ToolLabels.running(use.getToolName()) + ").");
+         }
+         else if(failed[i] || (extras != null && extras.isFailed()))
+         {
+            notice("error", "A step failed (" + ToolLabels.running(use.getToolName()) + ").");
+         }
 
          String content = result.getModelContent() == null ? "" : result.getModelContent();
          modelResults.add(ModelPart.toolResult(use.getToolUseId(), content, result.getOutcome() != null && ERROR_OUTCOMES.contains(result.getOutcome())));
@@ -1008,7 +1070,21 @@ public class TurnEngine implements Runnable
 
 
    /*******************************************************************************
-    ** Audit row, events and message parts of one tool result.
+    ** The result of a tool call that threw: an ERROR (is_error) result with a
+    ** fixed text; what went wrong is only in the log.
+    *******************************************************************************/
+   private static AiToolResult failedResult(String toolCallId, String toolName)
+   {
+      return (new AiToolResult().withToolCallId(toolCallId).withToolName(toolName).withOutcome(AiToolOutcome.ERROR)
+         .withSummary("Failed").withModelContent("The tool failed unexpectedly. Answer with what you have, or try another way."));
+   }
+
+
+
+   /*******************************************************************************
+    ** Audit row, events and message parts of one tool result. The render is
+    ** checked first (RenderShapes): one that cannot be shown is left out and an
+    ** error notice says so.
     *******************************************************************************/
    private void recordToolResult(ModelPart use, AiToolResult result, UiApiToolbox.Outcome extras)
    {
@@ -1026,9 +1102,23 @@ public class TurnEngine implements Runnable
 
       Map<String, Object> data = map("messageId", assistantMessage.getId(), "toolCallId", use.getToolUseId(), "name", use.getToolName(), "outcome", outcome, "summary", summary);
       Boolean presentation = PresentResultsTool.NAME.equals(use.getToolName()) && AiToolOutcome.OK.equals(result.getOutcome()) && result.getRender() != null;
-      if(result.getRender() != null && !presentation)
+      Map<String, Object> render           = null;
+      Map<String, Object> presentationPart = null;
+      if(result.getRender() != null)
       {
-         data.put("render", result.getRender());
+         if(presentation)
+         {
+            presentationPart = RenderShapes.presentation(result.getRender());
+         }
+         else
+         {
+            render = RenderShapes.render(result.getRender());
+         }
+      }
+      boolean undisplayable = result.getRender() != null && render == null && presentationPart == null;
+      if(render != null)
+      {
+         data.put("render", render);
       }
       if(!refs.isEmpty())
       {
@@ -1036,11 +1126,16 @@ public class TurnEngine implements Runnable
       }
       emit(TurnEvent.TOOL_RESULT, data);
 
-      parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, presentation ? null : result.getRender());
-      if(presentation)
+      parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, render);
+      if(presentationPart != null)
       {
-         parts.addPresentation(result.getRender());
-         emit(TurnEvent.PRESENTATION, map("messageId", assistantMessage.getId(), "part", result.getRender()));
+         parts.addPresentation(presentationPart);
+         emit(TurnEvent.PRESENTATION, map("messageId", assistantMessage.getId(), "part", presentationPart));
+      }
+      if(undisplayable)
+      {
+         LOG.warn("Assistant tool result could not be displayed", logPair("turnId", turn.getId()), logPair("tool", use.getToolName()));
+         notice("error", RenderShapes.UNDISPLAYABLE_TEXT);
       }
       parts.addRecordRefs(refs);
 
@@ -1215,13 +1310,18 @@ public class TurnEngine implements Runnable
                .withValue("model", model).withValue("partsJson", parts.toJson())));
          }
 
-         emit(TurnEvent.USAGE, map("turnId", turn.getId(), "inputTokens", usage.getInputTokens(), "outputTokens", usage.getOutputTokens(),
-            "cacheReadTokens", usage.getCacheReadTokens(), "cacheWriteTokens", usage.getCacheWriteTokens(), "costMicros", costMicros));
+         //////////////////////////////////////////////////////////////////////
+         // each terminal event is independent: a failing emit never skips  //
+         // message.end or the terminal turn row below                      //
+         //////////////////////////////////////////////////////////////////////
+         safely(() -> emit(TurnEvent.USAGE, map("turnId", turn.getId(), "inputTokens", usage.getInputTokens(), "outputTokens", usage.getOutputTokens(),
+            "cacheReadTokens", usage.getCacheReadTokens(), "cacheWriteTokens", usage.getCacheWriteTokens(), "costMicros", costMicros)));
          if(errorCode != null)
          {
-            emit(TurnEvent.ERROR, map("code", errorCode, "message", errorText));
+            String finalErrorText = errorText;
+            safely(() -> emit(TurnEvent.ERROR, map("code", finalErrorCode, "message", finalErrorText)));
          }
-         emit(TurnEvent.MESSAGE_END, map("messageId", assistantMessage.getId(), "status", status, "stopReason", stopReason));
+         safely(() -> emit(TurnEvent.MESSAGE_END, map("messageId", assistantMessage.getId(), "status", finalStatus, "stopReason", finalStopReason)));
          if(!alreadyTerminal)
          {
             safely(this::persistPendingEvents);

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/TurnEvent.java

@@ -67,6 +67,11 @@ public class TurnEvent
             }
          });
       }
+      ////////////////////////////////////////////////////////////////////
+      // renders and presentations are checked before they are streamed //
+      // or stored, and again when stored events are replayed           //
+      ////////////////////////////////////////////////////////////////////
+      RenderShapes.normalizeEvent(type, this.data);
    }
 
 
@@ -110,7 +115,7 @@ public class TurnEvent
     *******************************************************************************/
    public String toJson()
    {
-      return (JsonUtils.toJson(data));
+      return (RenderShapes.toJson(data));
    }
 
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/health/AssistantHealthChecks.java

@@ -0,0 +1,308 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.health;
+
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.FutureTask;
+import java.util.concurrent.TimeUnit;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.AssistantSecurity;
+import com.kingsrook.qbits.assistant.engine.ConnectionTest;
+import com.kingsrook.qbits.assistant.engine.ProviderService;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.spi.HealthResult;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplier;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplierMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthDetails;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+
+
+/*******************************************************************************
+ ** Central health adapters reuse the existing probes without exposing their text.
+ *******************************************************************************/
+public final class AssistantHealthChecks
+{
+   public static final String CLIENT_ERRORS = "assistant.client-errors";
+
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private AssistantHealthChecks()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Register dynamic namespaces once, even if production is repeated.
+    *******************************************************************************/
+   public static void register(QInstance instance)
+   {
+      HealthChecksMetaData registry = HealthChecksMetaData.ofOrWithNew(instance);
+      register(registry, "assistant.provider", Providers.class);
+      register(registry, "assistant.model", Models.class);
+      if(registry.getChecks().stream().noneMatch(existing -> CLIENT_ERRORS.equals(existing.getName())))
+      {
+         registry.withCheck(new HealthCheckMetaData().withName(CLIENT_ERRORS).withLabel("Assistant client errors").withComponent("assistant")
+            .withKind(HealthCheckKind.DEPENDENCY).withCategory(HealthCategory.AI)
+            .withIntervalSeconds(300).withTimeoutMs(10000)
+            .withCheckInstance(input -> bounded(input, () -> clientErrors(AssistantRuntime.of(input.getQInstance())))));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Errors the panel reported in the last hour (POST /assistant/client-errors):
+    ** DEGRADED at clientErrorWarnPerHour or more, so administrators see in
+    ** System Health that users are hitting broken screens. Only the count.
+    *******************************************************************************/
+   public static HealthCheckOutcome clientErrors(AssistantRuntime runtime) throws QException
+   {
+      if(runtime == null)
+      {
+         return (HealthCheckOutcome.unknown("Assistant not installed"));
+      }
+      int count = runtime.getStore().countClientErrorsSince(runtime.getClock().instant().minusSeconds(3600));
+      Integer threshold = runtime.getConfig().getClientErrorWarnPerHour();
+      String  text      = count + " client error" + (count == 1 ? "" : "s") + " reported in the last hour";
+      HealthCheckOutcome outcome = threshold != null && count >= threshold ? HealthCheckOutcome.degraded(text) : HealthCheckOutcome.up(text);
+      return (outcome.withDetails(new HealthDetails().withValue("lastHour", count).withValue("warnAt", threshold)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Suppliers carry the component; individual definitions also declare it.
+    *******************************************************************************/
+   private static void register(HealthChecksMetaData registry, String prefix, Class<? extends HealthCheckSupplier> supplier)
+   {
+      if(registry.getSuppliers().stream().noneMatch(existing -> prefix.equals(existing.getNamePrefix())))
+      {
+         registry.withSupplier(new HealthCheckSupplierMetaData().withNamePrefix(prefix).withComponent("assistant").withSupplier(new QCodeReference(supplier)));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Provider and model ids are integer primary keys, hence already lowercase.
+    *******************************************************************************/
+   private static HealthCheckMetaData definition(String name, Integer interval, Integer timeout, HealthCheck check)
+   {
+      return (new HealthCheckMetaData().withName(name).withLabel(name).withComponent("assistant")
+         .withKind(HealthCheckKind.DEPENDENCY).withCategory(HealthCategory.AI)
+         .withIntervalSeconds(interval).withTimeoutMs(timeout).withCheckInstance(check));
+   }
+
+
+
+   /*******************************************************************************
+    ** One definition per enabled provider; no I/O to providers during resolution.
+    *******************************************************************************/
+   public static class Providers implements HealthCheckSupplier
+   {
+      /***************************************************************************
+       ** Re-resolve rows when the central registry refreshes its suppliers.
+       ***************************************************************************/
+      @Override
+      public List<HealthCheckMetaData> supply(QInstance instance) throws QException
+      {
+         AssistantRuntime runtime = AssistantRuntime.of(instance);
+         List<HealthCheckMetaData> checks = new ArrayList<>();
+         for(AssistantProvider row : runtime.getStore().listProviders())
+         {
+            if(Boolean.TRUE.equals(row.getEnabled()))
+            {
+               checks.add(definition("assistant.provider." + row.getId(), "LOCAL".equals(runtime.getProviderService().location(row)) ? 60 : 300, 10000,
+                  input -> bounded(input, () -> provider(runtime, row.getId(), input))));
+            }
+         }
+         return (checks);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** One definition per enabled model, including models of disabled providers.
+    *******************************************************************************/
+   public static class Models implements HealthCheckSupplier
+   {
+      /***************************************************************************
+       ** Eligibility is checked again when a scheduled check actually executes.
+       ***************************************************************************/
+      @Override
+      public List<HealthCheckMetaData> supply(QInstance instance) throws QException
+      {
+         AssistantRuntime runtime = AssistantRuntime.of(instance);
+         List<HealthCheckMetaData> checks = new ArrayList<>();
+         for(AssistantModel row : runtime.getStore().listModels())
+         {
+            if(Boolean.TRUE.equals(row.getEnabled()))
+            {
+               checks.add(definition("assistant.model." + row.getId(), 21600, 30000,
+                  input -> bounded(input, () -> model(runtime, row.getId(), input))));
+            }
+         }
+         return (checks);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Clamp transport timeouts and bound even an adapter that ignores them.
+    ** Keep the caller's session; never create or elevate a health-check session.
+    *******************************************************************************/
+   private static HealthCheckOutcome bounded(HealthCheckInput input, Callable<HealthCheckOutcome> call)
+   {
+      long remaining = Duration.between(Instant.now(), input.getDeadline()).toMillis();
+      if(remaining <= 0)
+      {
+         return (HealthCheckOutcome.down("Health check timed out"));
+      }
+      var session = QContext.getQSession();
+      FutureTask<HealthCheckOutcome> task = new FutureTask<>(() ->
+      {
+         QContext.init(input.getQInstance(), session);
+         try
+         {
+            return (call.call());
+         }
+         finally
+         {
+            QContext.clear();
+         }
+      });
+      Thread.ofVirtual().name("assistant-central-health").start(task);
+      try
+      {
+         return (task.get(remaining, TimeUnit.MILLISECONDS));
+      }
+      catch(InterruptedException e)
+      {
+         Thread.currentThread().interrupt();
+         return (HealthCheckOutcome.down("Health check interrupted"));
+      }
+      catch(java.util.concurrent.TimeoutException e)
+      {
+         return (HealthCheckOutcome.down("Health check timed out"));
+      }
+      catch(java.util.concurrent.ExecutionException e)
+      {
+         return (HealthCheckOutcome.down("Health check failed"));
+      }
+      finally
+      {
+         task.cancel(true);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Unsupported probes may read recent traffic-produced breaker state only.
+    *******************************************************************************/
+   private static HealthCheckOutcome provider(AssistantRuntime runtime, Integer id, HealthCheckInput input) throws QException
+   {
+      AssistantProvider row = runtime.getStore().getProvider(id);
+      if(row == null || !Boolean.TRUE.equals(row.getEnabled()))
+      {
+         return (HealthCheckOutcome.unknown("Provider disabled or unavailable"));
+      }
+      var type = runtime.getConfig().getProviderTypes().get(row.getProviderType());
+      if(type == null)
+      {
+         return (HealthCheckOutcome.unknown("Provider type unavailable"));
+      }
+      var settings = runtime.getProviderService().settings(row);
+      int seconds = (int) Math.max(1, Math.min(10, Duration.between(Instant.now(), input.getDeadline()).toSeconds()));
+      settings.withConnectTimeoutSeconds(Math.min(settings.getConnectTimeoutSeconds(), seconds))
+         .withRequestTimeoutSeconds(Math.min(settings.getRequestTimeoutSeconds(), seconds));
+      HealthResult result = type.checkHealth(settings);
+      HealthCheckOutcome outcome;
+      if(result.healthy() == null)
+      {
+         Instant checked = row.getHealthCheckedAt();
+         Instant now = runtime.getClock().instant();
+         if(checked != null && !checked.isAfter(now) && checked.plusSeconds(600).isAfter(now) && List.of("UP", "DOWN", "DEGRADED").contains(row.getHealthState() == null ? "" : row.getHealthState()))
+         {
+            outcome = new HealthCheckOutcome().withStatus(HealthStatus.valueOf(row.getHealthState())).withMessage("Recent circuit-breaker state");
+         }
+         else
+         {
+            outcome = HealthCheckOutcome.unknown("health probe not supported");
+         }
+      }
+      else
+      {
+         outcome = result.healthy() ? HealthCheckOutcome.up("Provider reachable") : HealthCheckOutcome.down("Provider health probe failed");
+      }
+      return (outcome.withLatencyMs(result.latencyMs()).withDetails(new HealthDetails().withValue("providerId", id)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Reuse the minimal connection call and its accounting, without lastTest writes.
+    *******************************************************************************/
+   private static HealthCheckOutcome model(AssistantRuntime runtime, Integer id, HealthCheckInput input) throws QException
+   {
+      AssistantModel row = runtime.getStore().listModels().stream().filter(model -> id.equals(model.getId())).findFirst().orElse(null);
+      if(row == null || !Boolean.TRUE.equals(row.getEnabled()))
+      {
+         return (HealthCheckOutcome.unknown("Model disabled or unavailable"));
+      }
+      AssistantProvider provider = runtime.getStore().getProvider(row.getProviderId());
+      if(provider == null || !Boolean.TRUE.equals(provider.getEnabled()))
+      {
+         return (HealthCheckOutcome.unknown("Provider disabled or unavailable"));
+      }
+      Long cap = ProviderService.capMicros(provider);
+      if(cap != null && runtime.getProviderService().monthToDateCostMicros(provider.getId(), true) >= cap)
+      {
+         return (HealthCheckOutcome.unknown("skipped: cost cap"));
+      }
+      if(runtime.getPolicyService().current(true).killSwitch())
+      {
+         return (HealthCheckOutcome.unknown("skipped: kill switch"));
+      }
+      AssistantSecurity.ownerKey(QContext.getQSession(), runtime.getConfig());
+      ConnectionTest.Result result = ConnectionTest.checkModel(runtime, row, input.getDeadline());
+      return ((result.ok() ? HealthCheckOutcome.up("Model connection succeeded") : HealthCheckOutcome.down("Model connection failed"))
+         .withDetails(new HealthDetails().withValue("modelId", id).withValue("providerId", provider.getId()))
+         .withLatencyMs(result.latencyMs() == null ? null : result.latencyMs().longValue()));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/health/AssistantHealthWidget.java

@@ -0,0 +1,79 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.health;
+
+
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
+import com.kingsrook.qbits.assistant.tools.FilterLinkBuilder;
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthRunner;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
+
+
+/*******************************************************************************
+ ** A single generic status card links to the central checks, without probing.
+ *******************************************************************************/
+public class AssistantHealthWidget extends AbstractWidgetRenderer
+{
+   public static final String NAME = "assistantHealth";
+
+
+
+   /*******************************************************************************
+    ** No HTML, history or per-provider check list on the Assistant home.
+    *******************************************************************************/
+   public static QWidgetMetaData widget()
+   {
+      return (new QWidgetMetaData().withName(NAME).withLabel("Assistant health")
+         .withType(WidgetType.STATISTICS.getType()).withCodeReference(new QCodeReference(AssistantHealthWidget.class))
+         .withGridColumns(12).withPermissionRules(AssistantMetaDataBuilder.adminRules()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Summarize the runner's latest results; full history lives in System Health.
+    *******************************************************************************/
+   @Override
+   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
+   {
+      var instance = QContext.getQInstance();
+      HealthRunner runner = HealthRunner.getForInstance(instance);
+      HealthStatus status = HealthStatus.UNKNOWN;
+      if(runner != null)
+      {
+         status = HealthStatus.aggregate(HealthChecksMetaData.ofOrWithNew(instance).resolveAll(instance).stream()
+            .filter(check -> "assistant".equals(check.getComponent()))
+            .map(check -> runner.getLastOutcome(check.getName()))
+            .map(outcome -> outcome == null ? HealthStatus.UNKNOWN : outcome.getStatus()).toArray(HealthStatus[]::new));
+      }
+      return (new RenderWidgetOutput(new StatisticsData().withCount(status.name())
+         .withCountContext("System Health")
+         .withCountURL(FilterLinkBuilder.href("health_check", List.of(Map.of("fieldName", "component", "operator", "EQUALS", "values", List.of("assistant")))))));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/home/AssistantHome.java

@@ -32,7 +32,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 
 /*******************************************************************************
  ** The Assistant app home: one full-width HTML widget, the single owner of the
- ** assistant's health, usage and cost data (no other dashboard repeats it).
+ ** assistant's configuration, usage and cost data. Health lives in System Health.
  *******************************************************************************/
 public final class AssistantHome
 {

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/home/AssistantHomeView.java

@@ -17,16 +17,6 @@
 package com.kingsrook.qbits.assistant.home;
 
 
-import com.kingsrook.qbits.assistant.engine.BudgetService;
-import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
-import com.kingsrook.qbits.assistant.model.AssistantBudget;
-import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
-import com.kingsrook.qbits.assistant.model.AssistantConversation;
-import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
-import com.kingsrook.qbits.assistant.model.AssistantProvider;
-import com.kingsrook.qbits.assistant.model.AssistantToolCall;
-import com.kingsrook.qbits.assistant.model.AssistantUsage;
-import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import java.math.BigDecimal;
 import java.math.RoundingMode;
 import java.text.NumberFormat;
@@ -38,6 +28,16 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import com.kingsrook.qbits.assistant.engine.BudgetService;
+import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
+import com.kingsrook.qbits.assistant.model.AssistantBudget;
+import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
+import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.model.AssistantToolCall;
+import com.kingsrook.qbits.assistant.model.AssistantUsage;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
 
 
 /*******************************************************************************
@@ -159,44 +159,17 @@ public final class AssistantHomeView
 
 
    /*******************************************************************************
-    ** The status row: is the assistant answering, with what, at what cost.
+    ** Configuration and usage; health status belongs to the central card.
     *******************************************************************************/
    private static void status(StringBuilder html, AssistantHomeData data, String prefix)
    {
-      open(html, "Health");
+      open(html, "Configuration and usage");
       AssistantHomeData.ProviderHealth active = data.getProviders().stream().filter(AssistantHomeData.ProviderHealth::active).findFirst().orElse(null);
-
-      html.append("<div style=\"display:flex;flex-wrap:wrap;gap:.5rem .75rem;align-items:center;\">");
-      if(data.isKillSwitch())
-      {
-         html.append(chip(RED, "Turned off by the kill switch"));
-      }
-      else if(data.getProviders().isEmpty())
-      {
-         html.append(chip(GREY, "No provider configured here (the application's built-in provider is used, if any)"));
-      }
-      else if(active == null)
-      {
-         html.append(chip(RED, "No provider is turned on"));
-      }
-      else if(!active.keyPresent() || !active.problems().isEmpty())
-      {
-         html.append(chip(RED, "Default provider cannot be used"));
-      }
-      else if("ERROR".equals(active.row().getLastTestStatus()))
-      {
-         html.append(chip(AMBER, "Last connection test failed"));
-      }
-      else
-      {
-         html.append(chip(GREEN, "Answering"));
-      }
-      html.append("</div><div style=\"").append(CELLS).append("\">");
+      html.append("<div style=\"").append(CELLS).append("\">");
       cell(html, "Provider", active == null ? "None" : active.row().getName() + " (" + nvl(active.row().getModel(), "default model") + ")");
       cell(html, "HIPAA mode", Boolean.TRUE.equals(data.getHipaaMode()) ? "On" : "Off");
       cell(html, "PHI retention", data.getHipaaRetentionDays() == null ? "Normal retention" : data.getHipaaRetentionDays() + " days");
       cell(html, "API key", active == null ? "Not applicable" : active.keyStatus());
-      cell(html, "Last test", active == null ? "Not applicable" : lastTest(active.row()));
       Integer rate = data.errorRatePercent7d();
       cell(html, "Error rate, 7 days", rate == null ? "No requests yet" : rate + "% of " + number(data.getCalls7d()) + " requests");
       cell(html, "Cost this month", money(data.getMonthCostMicros()) + (data.getInstanceMonthlyCapMicros() == null ? "" : " of " + money(data.getInstanceMonthlyCapMicros()) + " limit"));
@@ -234,17 +207,8 @@ public final class AssistantHomeView
          {
             html.append(chip(GREY, "Off"));
          }
-         if(row.getLastTestStatus() != null)
-         {
-            html.append(chip("OK".equals(row.getLastTestStatus()) ? GREEN : RED, "OK".equals(row.getLastTestStatus()) ? "Last test passed" : "Last test failed"));
-         }
          html.append("</div><div style=\"").append(CELLS).append("\">");
          cell(html, "Type", nvl(p.typeLabel(), "Not set"));
-         cell(html, "Health", nvl(row.getHealthState(), "Unknown"));
-         if(row.getDownUntil() != null)
-         {
-            cell(html, "Down until", when(row.getDownUntil()));
-         }
          cell(html, "Kind", AssistantRecordScreens.providerKindLabel(p.kind()));
          cell(html, "Location", "LOCAL".equals(p.location()) ? "Local" : "Cloud");
          if(Boolean.TRUE.equals(row.getHipaaEligible()))
@@ -254,7 +218,6 @@ public final class AssistantHomeView
          cell(html, "Enabled models", String.valueOf(p.enabledModels()));
          cell(html, "Model", nvl(row.getModel(), "Default model"));
          cell(html, "API key", p.keyStatus());
-         cell(html, "Last test", lastTest(row));
          cell(html, "Cost this month", money(p.monthCostMicros()) + (p.capMicros() == null ? ", no cap" : " of " + money(p.capMicros()) + " cap"));
          List<String> limits = new ArrayList<>();
          if(row.getMaxOutputTokensPerRequest() != null)
@@ -502,24 +465,6 @@ public final class AssistantHomeView
 
 
 
-   /*******************************************************************************
-    ** A provider's last test in words.
-    *******************************************************************************/
-   static String lastTest(AssistantProvider row)
-   {
-      if(row.getLastTestAt() == null)
-      {
-         return ("Not tested yet: run Test connection");
-      }
-      if("OK".equals(row.getLastTestStatus()))
-      {
-         return ("Passed " + WHEN.format(row.getLastTestAt()) + ", " + nvl(row.getLastTestModel(), "model") + " in " + row.getLastTestLatencyMs() + " ms");
-      }
-      return ("Failed " + WHEN.format(row.getLastTestAt()) + ": " + nvl(row.getLastTestError(), "no detail"));
-   }
-
-
-
    /*******************************************************************************
     ** Micro-dollars as dollars ("$0.00", "<$0.01" for a little, "$12.34").
     *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/jobs/AssistantRunTracker.java

@@ -17,9 +17,14 @@
 package com.kingsrook.qbits.assistant.jobs;
 
 
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
 /*******************************************************************************
- ** STAND-IN(core=QRun-IO/qqq#1210, removeBy=RC.9): core job tracking
- ** (jobs_job tracking rows) will implement this contract; hosts may wire their own.
+ ** Publishes assistant answers as jobs. The default, JobTrackingRunTracker,
+ ** uses QQQ core job tracking (the Jobs tray); NOOP turns it off, and hosts
+ ** may wire their own. Every call is isolated from the turn: a failure here
+ ** never affects the answer.
  *******************************************************************************/
 public interface AssistantRunTracker
 {
@@ -54,7 +59,7 @@ public interface AssistantRunTracker
    /*******************************************************************************
     ** Publish a run and return its tracking id.
     *******************************************************************************/
-   String start(String userId, Integer turnId, Integer conversationId, String label);
+   String start(String userId, Integer turnId, Integer conversationId, String label) throws QException;
 
    /*******************************************************************************
     ** Publish a changed phase.
@@ -64,12 +69,12 @@ public interface AssistantRunTracker
    /*******************************************************************************
     ** Publish the terminal outcome.
     *******************************************************************************/
-   void finish(String trackingId, String status, String message);
+   void finish(String trackingId, String status, String message) throws QException;
 
    /*******************************************************************************
     ** Wire the host's Stop action to the owner's cancellation path.
     *******************************************************************************/
-   default void onCancel(String trackingId, Runnable cancel)
+   default void onCancel(String trackingId, Runnable cancel) throws QException
    {
    }
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/jobs/JobTrackingRunTracker.java

@@ -0,0 +1,170 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.jobs;
+
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.jobs.model.JobStatus;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgress;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobTracking;
+import com.kingsrook.qqq.backend.core.jobs.progress.TrackedJobRequest;
+import com.kingsrook.qqq.backend.core.jobs.runtime.DurableJobsRuntime;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+
+
+/*******************************************************************************
+ ** Publishes each assistant answer to the Jobs tray as a QQQ tracking job
+ ** (core JobTracking, qqq#1210): processName "assistant.turn", phases
+ ** thinking / tool / writing, the tool's label as the message, and the tray's
+ ** Stop wired to the turn's own Stop. Never prompt or answer text. When the
+ ** durable jobs runtime is not running on this pod, answers are simply not
+ ** tracked.
+ *******************************************************************************/
+public class JobTrackingRunTracker implements AssistantRunTracker
+{
+   public static final String PROCESS_NAME = "assistant.turn";
+   public static final String LABEL        = "Assistant answer";
+
+   private final Map<String, JobProgress> reporters = new ConcurrentHashMap<>();
+
+
+
+   /*******************************************************************************
+    ** Start a tracking job linked to the conversation, or null when the jobs
+    ** runtime is not running here.
+    *******************************************************************************/
+   @Override
+   public String start(String userId, Integer turnId, Integer conversationId, String label) throws QException
+   {
+      QInstance qInstance = QContext.getQInstance();
+      if(qInstance == null || DurableJobsRuntime.get(qInstance) == null)
+      {
+         return (null);
+      }
+
+      Integer jobId = JobTracking.start(new TrackedJobRequest()
+         .withUserId(userId)
+         .withLabel(LABEL)
+         .withProcessName(PROCESS_NAME)
+         .withTableName(AssistantConversation.TABLE_NAME)
+         .withRecordId(conversationId));
+      String trackingId = String.valueOf(jobId);
+      reporters.put(trackingId, JobTracking.progress(jobId).phases(
+         JobPhase.of("thinking", "Thinking", 1),
+         JobPhase.of("tool", "Using a tool", 1),
+         JobPhase.of("writing", "Writing", 1)));
+      return (trackingId);
+   }
+
+
+
+   /*******************************************************************************
+    ** The current phase, with the tool's label as the message.
+    *******************************************************************************/
+   @Override
+   public void progress(String trackingId, String phase, String message)
+   {
+      JobProgress reporter = trackingId == null ? null : reporters.get(trackingId);
+      if(reporter == null || !("thinking".equals(phase) || "tool".equals(phase) || "writing".equals(phase)))
+      {
+         return;
+      }
+      reporter.phase(phase);
+      if("tool".equals(phase) && message != null)
+      {
+         reporter.message(message);
+         reporter.flush();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** COMPLETE -> COMPLETED, CANCELLED -> CANCELLED, anything else -> FAILED.
+    *******************************************************************************/
+   @Override
+   public void finish(String trackingId, String status, String message) throws QException
+   {
+      if(trackingId == null)
+      {
+         return;
+      }
+      JobProgress reporter = reporters.remove(trackingId);
+      if(reporter != null)
+      {
+         reporter.flush();
+      }
+      JobTracking.finish(Integer.valueOf(trackingId), jobStatus(status), finishMessage(status));
+   }
+
+
+
+   /*******************************************************************************
+    ** The tray's Stop runs on this pod and stops the turn; the turn's own
+    ** finaliser then finishes the job as CANCELLED.
+    *******************************************************************************/
+   @Override
+   public void onCancel(String trackingId, Runnable cancel) throws QException
+   {
+      if(trackingId != null)
+      {
+         JobTracking.onCancel(Integer.valueOf(trackingId), cancel);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The jobs status for an assistant turn status.
+    *******************************************************************************/
+   static JobStatus jobStatus(String status)
+   {
+      if("COMPLETE".equals(status))
+      {
+         return (JobStatus.COMPLETED);
+      }
+      if("CANCELLED".equals(status))
+      {
+         return (JobStatus.CANCELLED);
+      }
+      return (JobStatus.FAILED);
+   }
+
+
+
+   /*******************************************************************************
+    ** A fixed message (never the answer, the prompt or a provider error).
+    *******************************************************************************/
+   static String finishMessage(String status)
+   {
+      if("COMPLETE".equals(status))
+      {
+         return ("Answered.");
+      }
+      if("CANCELLED".equals(status))
+      {
+         return ("Stopped.");
+      }
+      return ("The answer did not finish.");
+   }
+
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantMetaDataBuilder.java

@@ -20,12 +20,14 @@ package com.kingsrook.qbits.assistant.metadata;
 import java.util.ArrayList;
 import java.util.List;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.health.AssistantHealthWidget;
 import com.kingsrook.qbits.assistant.home.AssistantAccessWidget;
 import com.kingsrook.qbits.assistant.home.AssistantHome;
 import com.kingsrook.qbits.assistant.model.AssistantBudget;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
+import com.kingsrook.qbits.assistant.model.AssistantClientError;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
 import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
 import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
@@ -113,7 +115,7 @@ public class AssistantMetaDataBuilder
     ** calls on a conversation; model calls on a provider) or reached from the
     ** home. Processes are table and record actions, never nav entries.
     *******************************************************************************/
-   public static final List<String> NAV_TABLES = List.of(AssistantScheduledTask.TABLE_NAME, AssistantRagSource.TABLE_NAME, AssistantBudget.TABLE_NAME, AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
+   public static final List<String> NAV_TABLES = List.of(AssistantScheduledTask.TABLE_NAME, AssistantRagSource.TABLE_NAME, AssistantBudget.TABLE_NAME, AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantClientError.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
 
    private final AssistantQBitConfig config;
 
@@ -178,6 +180,9 @@ public class AssistantMetaDataBuilder
 
       tables.add(text(readOnly(owned(base(AssistantFeedback.TABLE_NAME, "Assistant Feedback", "thumbs_up_down", AssistantFeedback.class), false)), "comment"));
 
+      tables.add(text(readOnly(owned(base(AssistantClientError.TABLE_NAME, "Assistant Client Error", "bug_report", AssistantClientError.class), false)), "message", "stack", "componentStack")
+         .withRecordLabelFormat("%s #%s").withRecordLabelFields("scope", "id"));
+
       tables.add(base(AssistantPolicy.TABLE_NAME, "Assistant Policy", "policy", AssistantPolicy.class)
          .withRecordLabelFormat("%s").withRecordLabelFields("name"));
 
@@ -383,7 +388,7 @@ public class AssistantMetaDataBuilder
          .withLabel("Assistant")
          .withIcon(new QIcon("smart_toy"))
          .withPermissionRules(adminRules())
-         .withWidgets(List.of(AssistantHome.WIDGET_NAME, AssistantAccessWidget.WIDGET_NAME));
+         .withWidgets(List.of(AssistantHealthWidget.NAME, AssistantHome.WIDGET_NAME, AssistantAccessWidget.WIDGET_NAME));
       for(String name : NAV_TABLES)
       {
          tables.stream().filter(t -> t.getName().equals(name)).findFirst().ifPresent(app::withChild);

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantRecordScreens.java

@@ -30,6 +30,7 @@ import com.kingsrook.qbits.assistant.engine.ProviderService;
 import com.kingsrook.qbits.assistant.model.AssistantBudget;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
+import com.kingsrook.qbits.assistant.model.AssistantClientError;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
 import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
 import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
@@ -215,6 +216,7 @@ public final class AssistantRecordScreens
          case AssistantTurnEvent.TABLE_NAME -> turnEvent(table);
          case AssistantToolCall.TABLE_NAME -> toolCall(table);
          case AssistantFeedback.TABLE_NAME -> feedback(table);
+         case AssistantClientError.TABLE_NAME -> clientError(table);
          case AssistantPolicy.TABLE_NAME -> policy(table);
          case AssistantUsageDaily.TABLE_NAME -> usageDaily(table);
          case AssistantProvider.TABLE_NAME -> provider(table);
@@ -695,6 +697,21 @@ public final class AssistantRecordScreens
 
 
 
+   /*******************************************************************************
+    ** Errors the panel reported: what and where first, the stacks last.
+    *******************************************************************************/
+   private static void clientError(QTableMetaData table)
+   {
+      link(table, "messageId", AssistantMessage.TABLE_NAME, "Message");
+      link(table, "turnId", AssistantTurn.TABLE_NAME, "Turn");
+      sections(table,
+         section("error", "Error", "bug_report", Tier.T1, "id", "scope", "errorName", "message", "occurredAt"),
+         section("where", "Where", "place", Tier.T2, "route", "partType", "renderType", "appVersion", "requestId", "userId", "conversationId", "messageId", "turnId"),
+         section("stacks", "Stacks", "code", Tier.T2, "stack", "componentStack"));
+   }
+
+
+
    /*******************************************************************************
     ** The one table people edit: limits are in tokens or micro-dollars, soft
     ** limits warn and hard limits stop.

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/model/AssistantClientError.java

@@ -0,0 +1,634 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.model;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/*******************************************************************************
+ ** An error the panel (Next UI) hit while showing the assistant, reported by
+ ** the user's browser through POST /assistant/client-errors. Only bounded,
+ ** scrubbed fields are kept (no free text beyond them); the ids are kept only
+ ** when they are the reporting user's own rows.
+ *******************************************************************************/
+public class AssistantClientError extends QRecordEntity
+{
+   public static final String TABLE_NAME = "assistant_client_error";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isEditable = false, maxLength = 255)
+   private String userId;
+
+   @QField(isEditable = false, maxLength = 64)
+   private String scope;
+
+   @QField(isEditable = false)
+   private String message;
+
+   @QField(isEditable = false, maxLength = 100)
+   private String errorName;
+
+   @QField(isEditable = false)
+   private String stack;
+
+   @QField(isEditable = false)
+   private String componentStack;
+
+   @QField(isEditable = false, maxLength = 128)
+   private String requestId;
+
+   @QField(isEditable = false, maxLength = 300)
+   private String route;
+
+   @QField(isEditable = false, possibleValueSourceName = AssistantConversation.TABLE_NAME)
+   private Integer conversationId;
+
+   @QField(isEditable = false)
+   private Integer messageId;
+
+   @QField(isEditable = false)
+   private Integer turnId;
+
+   @QField(isEditable = false, maxLength = 64)
+   private String partType;
+
+   @QField(isEditable = false, maxLength = 64)
+   private String renderType;
+
+   @QField(isEditable = false, maxLength = 64)
+   private String appVersion;
+
+   @QField(isEditable = false)
+   private Instant occurredAt;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+
+
+   /*******************************************************************************
+    ** Default constructor.
+    *******************************************************************************/
+   public AssistantClientError()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Constructor that populates from a QRecord.
+    *******************************************************************************/
+   public AssistantClientError(QRecord record) throws QException
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (this.id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public AssistantClientError withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for userId
+    *******************************************************************************/
+   public String getUserId()
+   {
+      return (this.userId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for userId
+    *******************************************************************************/
+   public void setUserId(String userId)
+   {
+      this.userId = userId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for userId
+    *******************************************************************************/
+   public AssistantClientError withUserId(String userId)
+   {
+      this.userId = userId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for scope
+    *******************************************************************************/
+   public String getScope()
+   {
+      return (this.scope);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for scope
+    *******************************************************************************/
+   public void setScope(String scope)
+   {
+      this.scope = scope;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for scope
+    *******************************************************************************/
+   public AssistantClientError withScope(String scope)
+   {
+      this.scope = scope;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for message
+    *******************************************************************************/
+   public String getMessage()
+   {
+      return (this.message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for message
+    *******************************************************************************/
+   public void setMessage(String message)
+   {
+      this.message = message;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for message
+    *******************************************************************************/
+   public AssistantClientError withMessage(String message)
+   {
+      this.message = message;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for errorName
+    *******************************************************************************/
+   public String getErrorName()
+   {
+      return (this.errorName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for errorName
+    *******************************************************************************/
+   public void setErrorName(String errorName)
+   {
+      this.errorName = errorName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for errorName
+    *******************************************************************************/
+   public AssistantClientError withErrorName(String errorName)
+   {
+      this.errorName = errorName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for stack
+    *******************************************************************************/
+   public String getStack()
+   {
+      return (this.stack);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for stack
+    *******************************************************************************/
+   public void setStack(String stack)
+   {
+      this.stack = stack;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for stack
+    *******************************************************************************/
+   public AssistantClientError withStack(String stack)
+   {
+      this.stack = stack;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for componentStack
+    *******************************************************************************/
+   public String getComponentStack()
+   {
+      return (this.componentStack);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for componentStack
+    *******************************************************************************/
+   public void setComponentStack(String componentStack)
+   {
+      this.componentStack = componentStack;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for componentStack
+    *******************************************************************************/
+   public AssistantClientError withComponentStack(String componentStack)
+   {
+      this.componentStack = componentStack;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for requestId
+    *******************************************************************************/
+   public String getRequestId()
+   {
+      return (this.requestId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for requestId
+    *******************************************************************************/
+   public void setRequestId(String requestId)
+   {
+      this.requestId = requestId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for requestId
+    *******************************************************************************/
+   public AssistantClientError withRequestId(String requestId)
+   {
+      this.requestId = requestId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for route
+    *******************************************************************************/
+   public String getRoute()
+   {
+      return (this.route);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for route
+    *******************************************************************************/
+   public void setRoute(String route)
+   {
+      this.route = route;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for route
+    *******************************************************************************/
+   public AssistantClientError withRoute(String route)
+   {
+      this.route = route;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for conversationId
+    *******************************************************************************/
+   public Integer getConversationId()
+   {
+      return (this.conversationId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for conversationId
+    *******************************************************************************/
+   public void setConversationId(Integer conversationId)
+   {
+      this.conversationId = conversationId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for conversationId
+    *******************************************************************************/
+   public AssistantClientError withConversationId(Integer conversationId)
+   {
+      this.conversationId = conversationId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for messageId
+    *******************************************************************************/
+   public Integer getMessageId()
+   {
+      return (this.messageId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for messageId
+    *******************************************************************************/
+   public void setMessageId(Integer messageId)
+   {
+      this.messageId = messageId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for messageId
+    *******************************************************************************/
+   public AssistantClientError withMessageId(Integer messageId)
+   {
+      this.messageId = messageId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for turnId
+    *******************************************************************************/
+   public Integer getTurnId()
+   {
+      return (this.turnId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for turnId
+    *******************************************************************************/
+   public void setTurnId(Integer turnId)
+   {
+      this.turnId = turnId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for turnId
+    *******************************************************************************/
+   public AssistantClientError withTurnId(Integer turnId)
+   {
+      this.turnId = turnId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for partType
+    *******************************************************************************/
+   public String getPartType()
+   {
+      return (this.partType);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for partType
+    *******************************************************************************/
+   public void setPartType(String partType)
+   {
+      this.partType = partType;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for partType
+    *******************************************************************************/
+   public AssistantClientError withPartType(String partType)
+   {
+      this.partType = partType;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for renderType
+    *******************************************************************************/
+   public String getRenderType()
+   {
+      return (this.renderType);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for renderType
+    *******************************************************************************/
+   public void setRenderType(String renderType)
+   {
+      this.renderType = renderType;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for renderType
+    *******************************************************************************/
+   public AssistantClientError withRenderType(String renderType)
+   {
+      this.renderType = renderType;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for appVersion
+    *******************************************************************************/
+   public String getAppVersion()
+   {
+      return (this.appVersion);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for appVersion
+    *******************************************************************************/
+   public void setAppVersion(String appVersion)
+   {
+      this.appVersion = appVersion;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for appVersion
+    *******************************************************************************/
+   public AssistantClientError withAppVersion(String appVersion)
+   {
+      this.appVersion = appVersion;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for occurredAt
+    *******************************************************************************/
+   public Instant getOccurredAt()
+   {
+      return (this.occurredAt);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for occurredAt
+    *******************************************************************************/
+   public void setOccurredAt(Instant occurredAt)
+   {
+      this.occurredAt = occurredAt;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for occurredAt
+    *******************************************************************************/
+   public AssistantClientError withOccurredAt(Instant occurredAt)
+   {
+      this.occurredAt = occurredAt;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (this.createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public AssistantClientError withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/persistence/AssistantStore.java

@@ -36,6 +36,7 @@ import com.kingsrook.qbits.assistant.model.AssistantBudget;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
 import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
+import com.kingsrook.qbits.assistant.model.AssistantClientError;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
 import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
 import com.kingsrook.qbits.assistant.model.AssistantFeedback;
@@ -54,6 +55,7 @@ import com.kingsrook.qbits.assistant.model.AssistantUsage;
 import com.kingsrook.qbits.assistant.model.AssistantUsageDaily;
 import com.kingsrook.qbits.assistant.scheduled.ScheduledTasks;
 import com.kingsrook.qbits.assistant.spi.ModelUsage;
+import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
 import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
 import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
@@ -61,6 +63,7 @@ import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.ActionFlag;
+import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertOutput;
@@ -437,6 +440,23 @@ public class AssistantStore
 
 
 
+   /*******************************************************************************
+    ** Runs of any user (that the running session can see) still RUNNING or
+    ** CANCELLING although they started before a time, by id after afterId
+    ** (the retention sweep of abandoned runs).
+    *******************************************************************************/
+   public List<AssistantTurn> listRunningTurnsStartedBefore(Instant cutoff, Integer afterId, int limit) throws QException
+   {
+      return (query(AssistantTurn.TABLE_NAME, new QQueryFilter()
+         .withCriteria("status", QCriteriaOperator.IN, List.of("RUNNING", "CANCELLING"))
+         .withCriteria("startedAt", QCriteriaOperator.LESS_THAN, cutoff)
+         .withCriteria("id", QCriteriaOperator.GREATER_THAN, afterId == null ? 0 : afterId)
+         .withOrderBy(new QFilterOrderBy("id", true))
+         .withLimit(limit), AssistantTurn.class));
+   }
+
+
+
    /*******************************************************************************
     ** Persist one streamed event.
     *******************************************************************************/
@@ -726,6 +746,39 @@ public class AssistantStore
 
 
 
+   /*******************************************************************************
+    ** How many client error reports (that the running session can see) were
+    ** stored since a time.
+    *******************************************************************************/
+   public int countClientErrorsSince(Instant since) throws QException
+   {
+      Integer count = new CountAction().execute(new CountInput(AssistantClientError.TABLE_NAME)
+         .withFilter(new QQueryFilter().withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, since))).getCount();
+      return (count == null ? 0 : count);
+   }
+
+
+
+   /*******************************************************************************
+    ** Delete client error reports stored before a time; returns how many.
+    *******************************************************************************/
+   public int deleteClientErrorsBefore(Instant cutoff, int limit) throws QException
+   {
+      List<AssistantClientError> errors = query(AssistantClientError.TABLE_NAME, new QQueryFilter()
+         .withCriteria("createDate", QCriteriaOperator.LESS_THAN, cutoff)
+         .withLimit(limit), AssistantClientError.class);
+      if(errors.isEmpty())
+      {
+         return (0);
+      }
+      List<Serializable> ids = new ArrayList<>();
+      errors.forEach(e -> ids.add(e.getId()));
+      new DeleteAction().execute(new DeleteInput(AssistantClientError.TABLE_NAME).withPrimaryKeys(ids));
+      return (errors.size());
+   }
+
+
+
    /*******************************************************************************
     ** A page of assistant messages created before a time, by id after afterId.
     *******************************************************************************/
@@ -814,11 +867,22 @@ public class AssistantStore
     ** The cost (micro-dollars) of a provider's calls since a time.
     *******************************************************************************/
    public long sumUsageCostSince(Integer providerId, Instant since) throws QException
+   {
+      return (sumUsageCostSince(providerId, since, false));
+   }
+
+
+
+   /*******************************************************************************
+    ** Automated health uses the existing private accounting projection so a cap
+    ** includes all users, without reading their owner-locked usage records.
+    *******************************************************************************/
+   public long sumUsageCostSince(Integer providerId, Instant since, Boolean sharedAccounting) throws QException
    {
       long total = 0;
-      for(QRecord record : queryRecords(AssistantUsage.TABLE_NAME, new QQueryFilter()
+      for(QRecord record : queryRecords(sharedAccounting ? AssistantBudgetUsage.TABLE_NAME : AssistantUsage.TABLE_NAME, new QQueryFilter()
          .withCriteria("providerId", QCriteriaOperator.EQUALS, providerId)
-         .withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, since), MAX_SUM_ROWS))
+         .withCriteria(sharedAccounting ? "occurredAt" : "createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, since), MAX_SUM_ROWS))
       {
          Long cost = record.getValueLong("costMicros");
          total += cost == null ? 0 : cost;

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/processes/AssistantRetentionStep.java

@@ -22,6 +22,7 @@ import java.time.Instant;
 import java.util.List;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
 import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.engine.AbandonedTurns;
 import com.kingsrook.qbits.assistant.engine.MessageParts;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
 import com.kingsrook.qbits.assistant.model.AssistantMessage;
@@ -41,7 +42,10 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
  ** - deletes conversations not active for conversationRetentionDays, with
  **   their messages, turns, events, tool calls and feedback;
  ** - nulls assistantToolCall.resultJson older than toolResultRetentionDays
- **   (the audit row stays).
+ **   (the audit row stays);
+ ** - finalizes abandoned runs (still RUNNING past the hard stream limit, e.g.
+ **   after a pod restart nobody looked at) as interrupted;
+ ** - deletes client error reports older than toolResultRetentionDays.
  ** It works on the rows the running session can see: run on a schedule (the
  ** scheduler's session) or by an admin holding the owner all-access key.
  *******************************************************************************/
@@ -68,6 +72,8 @@ public class AssistantRetentionStep implements BackendStep
       output.addValue("toolResultsCleared", result.toolResultsCleared());
       output.addValue("turnEventsDeleted", result.turnEventsDeleted());
       output.addValue("messagesStripped", result.messagesStripped());
+      output.addValue("turnsFinalized", result.turnsFinalized());
+      output.addValue("clientErrorsDeleted", result.clientErrorsDeleted());
    }
 
 
@@ -75,7 +81,7 @@ public class AssistantRetentionStep implements BackendStep
    /*******************************************************************************
     ** What was purged.
     *******************************************************************************/
-   public record Result(int conversationsDeleted, int toolResultsCleared, int turnEventsDeleted, int messagesStripped)
+   public record Result(int conversationsDeleted, int toolResultsCleared, int turnEventsDeleted, int messagesStripped, int turnsFinalized, int clientErrorsDeleted)
    {
    }
 
@@ -160,9 +166,23 @@ public class AssistantRetentionStep implements BackendStep
          }
       }
 
+      int turnsFinalized = AbandonedTurns.sweep(config, store, now);
+
+      int clientErrorsDeleted = 0;
+      while(true)
+      {
+         int count = store.deleteClientErrorsBefore(toolResultCutoff, BATCH);
+         clientErrorsDeleted += count;
+         if(count < BATCH)
+         {
+            break;
+         }
+      }
+
       LOG.info("Assistant retention ran", logPair("conversationsDeleted", deleted), logPair("toolResultsCleared", cleared),
-         logPair("turnEventsDeleted", eventsDeleted), logPair("messagesStripped", messagesStripped));
-      return (new Result(deleted, cleared, eventsDeleted, messagesStripped));
+         logPair("turnEventsDeleted", eventsDeleted), logPair("messagesStripped", messagesStripped), logPair("turnsFinalized", turnsFinalized),
+         logPair("clientErrorsDeleted", clientErrorsDeleted));
+      return (new Result(deleted, cleared, eventsDeleted, messagesStripped, turnsFinalized, clientErrorsDeleted));
    }
 
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/EmbeddingService.java

@@ -64,11 +64,21 @@ public class EmbeddingService
     ** including failures, and updates the existing daily budget rollups.
     *******************************************************************************/
    public List<float[]> embed(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId) throws QException
+   {
+      return (embed(modelId, texts, classification, tables, conversationId, turnId, UsageRecorder.PURPOSE_EMBED));
+   }
+
+
+
+   /*******************************************************************************
+    ** Connection diagnostics use TEST accounting with the same embedding guards.
+    *******************************************************************************/
+   public List<float[]> embed(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId, String purpose) throws QException
    {
       List<float[]> vectors = new ArrayList<>();
       for(Integer offset = 0; offset < texts.size(); offset += 64)
       {
-         vectors.addAll(batch(modelId, texts.subList(offset, Math.min(offset + 64, texts.size())), classification, tables, conversationId, turnId));
+         vectors.addAll(batch(modelId, texts.subList(offset, Math.min(offset + 64, texts.size())), classification, tables, conversationId, turnId, purpose));
       }
       return (vectors);
    }
@@ -104,7 +114,7 @@ public class EmbeddingService
    /*******************************************************************************
     ** Never downgrade an embedding call into a different vector space.
     *******************************************************************************/
-   private List<float[]> batch(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId) throws QException
+   private List<float[]> batch(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId, String purpose) throws QException
    {
       String userId = AssistantSecurity.userId(QContext.getQSession());
       ScheduledTurn.checkCost(runtime, conversationId, turnId);
@@ -140,6 +150,10 @@ public class EmbeddingService
       ModelProviderException error = null;
       try
       {
+         if(Thread.currentThread().isInterrupted())
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "Embedding request interrupted."));
+         }
          EmbeddingResponse embedded = active.provider().embed(new EmbeddingRequest(active.model(), texts));
          response = new ModelResponse().withModel(active.model()).withRequestId(embedded.requestId()).withUsage(embedded.usage());
          if(embedded.vectors().size() != texts.size())
@@ -160,7 +174,7 @@ public class EmbeddingService
             gate.release();
          }
          var pricing = runtime.getProviderService().pricing(active, active.model());
-         runtime.getStore().insert(UsageRecorder.row(new UsageRecorder.Call(userId, conversationId, turnId, active, UsageRecorder.PURPOSE_EMBED),
+         runtime.getStore().insert(UsageRecorder.row(new UsageRecorder.Call(userId, conversationId, turnId, active, purpose),
             response, error, (System.nanoTime() - started) / 1_000_000, pricing));
          var usage = response == null ? (error == null ? null : error.getPartialUsage()) : response.getUsage();
          if(usage != null)

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/UiApiToolbox.java

@@ -223,6 +223,7 @@ public class UiApiToolbox
       private Integer            httpStatus;
       private String             auditHref;
       private Integer            pendingActionId;
+      private boolean            failed;
 
 
 
@@ -251,6 +252,17 @@ public class UiApiToolbox
 
 
 
+      /*******************************************************************************
+       ** Whether the tool failed unexpectedly (an exception, not a refusal or a
+       ** bad input): the user is told a step failed.
+       *******************************************************************************/
+      public boolean isFailed()
+      {
+         return (failed);
+      }
+
+
+
       /*******************************************************************************
        ** Getter for result
        *******************************************************************************/
@@ -483,6 +495,7 @@ public class UiApiToolbox
       }
       catch(QException | RuntimeException e)
       {
+         outcome.failed = true;
          result.withOutcome(AiToolOutcome.ERROR).withSummary("Failed").withModelContent("The tool failed: " + shortText(e.getMessage()));
       }
       finally

qbit-assistant/src/main/resources/help/qbit-assistant.yaml

@@ -1,5 +1,5 @@
 bundle: qbit-assistant
-version: 0.1.0-SNAPSHOT
+version: ${project.version}
 layer: QBIT
 entries:
 - key: app:assistantAdmin;slot:help
@@ -358,6 +358,52 @@ entries:
   summary: The conversation the rated answer belongs to.
 - key: table:assistant_feedback;field:rating
   summary: 1 for thumbs up, -1 for thumbs down.
+- key: table:assistant_client_error;slot:help
+  summary: Errors the assistant panel hit in users' browsers, such as a result it could not display.
+  content: |
+    When the assistant panel or one of its results fails to show, the user's browser reports the error here instead of breaking the page. Each row says where it happened (the scope, page and part or render type), the error and its stacks, and links to the conversation, message and turn when they are the user's own.
+
+    Reports are scrubbed before they are stored: e-mail addresses, long numbers, tokens and query strings are removed and every field is cut to a fixed length. Each user can report a limited number per minute; more are dropped. The **Assistant client errors** check in System Health turns degraded when many arrive in an hour.
+
+    Records here are read-only and are deleted after the tool result retention period by **{{label:assistantRetention}}**.
+- key: table:assistant_client_error;section:error
+  summary: What went wrong and when the browser saw it.
+- key: table:assistant_client_error;section:where
+  summary: Where it happened (page, part and render type, build) and the conversation, message and turn it was about.
+- key: table:assistant_client_error;section:stacks
+  summary: The JavaScript and component stacks, scrubbed and shortened.
+- key: table:assistant_client_error;section:dates
+  summary: When the report was stored.
+- key: table:assistant_client_error;field:scope
+  summary: The part of the screen that failed, such as the panel, a message, a part, a render, the stream or a widget.
+- key: table:assistant_client_error;field:message
+  summary: The error message, scrubbed and cut to 500 characters.
+- key: table:assistant_client_error;field:errorName
+  summary: The kind of error the browser raised, such as TypeError.
+- key: table:assistant_client_error;field:stack
+  summary: Where in the client code the error happened, scrubbed and cut to 4,000 characters.
+- key: table:assistant_client_error;field:componentStack
+  summary: Which screen components were showing when it happened, scrubbed and cut to 4,000 characters.
+- key: table:assistant_client_error;field:requestId
+  summary: The request or trace id the client saw, to find the matching server log.
+- key: table:assistant_client_error;field:route
+  summary: The page the user was on (its path only).
+- key: table:assistant_client_error;field:conversationId
+  summary: The conversation being shown, when it is the reporting user's own.
+- key: table:assistant_client_error;field:messageId
+  summary: The message being shown, when it is the reporting user's own.
+- key: table:assistant_client_error;field:turnId
+  summary: The question-and-answer cycle being shown, when it is the reporting user's own.
+- key: table:assistant_client_error;field:partType
+  summary: The type of message part being shown, such as tool or presentation.
+- key: table:assistant_client_error;field:renderType
+  summary: The type of result being shown, such as records or widget.
+- key: table:assistant_client_error;field:appVersion
+  summary: The client build that reported the error.
+- key: table:assistant_client_error;field:occurredAt
+  summary: When the browser says the error happened.
+- key: table:assistant_client_error;field:userId
+  summary: The user whose browser reported the error.
 - key: table:assistant_policy;slot:help
   summary: The one place to set the assistant's spending limits, the kill switch, and which tools and data it may use.
   content: |
@@ -1287,7 +1333,9 @@ entries:
 - key: widget:assistant_providerJoinAssistant_usage;slot:label
   summary: Recent requests sent to this provider, with model, tokens, cost and result.
 - key: widget:assistantHome;slot:label
-  summary: 'The assistant''s health at a glance: providers, cost this month, use over the last 30 days, denials and failures.'
+  summary: 'The assistant''s setup, providers, cost this month, use over the last 30 days, denials and failures.'
+- key: widget:assistantHealth;slot:label
+  summary: Central health status for assistant providers and models, with a link to filtered System Health.
 - key: widget:assistantAccess;slot:label
   summary: Shows, for your own login, what the assistant may do on each table, and why an action is not allowed.
 - key: possibleValueSource:assistant_budget_scopeType;slot:help

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/AssistantQBitProducerTest.java

@@ -90,8 +90,8 @@ class AssistantQBitProducerTest extends BaseAssistantTest
       // the nav: the home plus the main tables; processes are actions    //
       //////////////////////////////////////////////////////////////////////
       assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getChildren()).extracting(c -> c.getName())
-         .containsExactly("assistant_scheduled_task", "assistant_rag_source", "assistant_budget", "assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_model_price", "assistant_policy");
-      assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getWidgets()).containsExactly("assistantHome", "assistantAccess");
+         .containsExactly("assistant_scheduled_task", "assistant_rag_source", "assistant_budget", "assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_client_error", "assistant_model_price", "assistant_policy");
+      assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getWidgets()).containsExactly("assistantHealth", "assistantHome", "assistantAccess");
       assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME).getTableName()).isEqualTo("assistant_conversation");
       assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME)).isNotNull();
       assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME).getSchedule()).isNull();

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/AssistantTablesGuardrailTest.java

@@ -71,6 +71,7 @@ class AssistantTablesGuardrailTest extends BaseAssistantTest
       Map.entry("assistant_tool_call", "runtime: every tool call"),
       Map.entry("assistant_pending_action", "runtime: a proposed change"),
       Map.entry("assistant_feedback", "user content: a rating"),
+      Map.entry("assistant_client_error", "runtime: the panel reports an error it hit"),
       Map.entry("assistant_usage", "runtime: every model call"),
       Map.entry("assistant_usage_daily", "runtime: after every answer"),
       Map.entry("assistant_data_audit", "runtime: every model call carrying data"));

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/api/AssistantRouteProviderTest.java

@@ -179,7 +179,11 @@ class AssistantRouteProviderTest extends BaseAssistantTest
       assertThat((Map<String, Object>) config.get("features")).containsEntry("writes", false);
 
       assertThat(json(send("GET", "/config", NO_ACCESS, null))).containsEntry("enabled", false).containsEntry("reason", "NO_PERMISSION");
-      assertThat(send("GET", "/conversations", NO_ACCESS, null).statusCode()).isEqualTo(403);
+      HttpResponse<String> denied = send("GET", "/conversations", NO_ACCESS, null);
+      assertThat(denied.statusCode()).isEqualTo(403);
+      assertThat(json(denied)).containsEntry("code", "NO_PERMISSION")
+         .containsEntry("message", json(denied).get("error"))
+         .containsEntry("traceId", denied.headers().firstValue("X-QQQ-Trace-Id").orElseThrow());
 
       setKillSwitch(true);
       assertThat(json(send("GET", "/config", EDITOR, null))).containsEntry("enabled", false).containsEntry("reason", "KILL_SWITCH");
@@ -217,6 +221,9 @@ class AssistantRouteProviderTest extends BaseAssistantTest
       HttpResponse<String> foreign = send("GET", "/conversations/" + id, OTHER_EDITOR, null);
       assertThat(foreign.statusCode()).isEqualTo(404);
       assertThat(foreign.body()).doesNotContain("Posts");
+      assertThat(json(foreign)).containsEntry("code", "NOT_FOUND")
+         .containsEntry("message", json(foreign).get("error"))
+         .containsEntry("traceId", foreign.headers().firstValue("X-QQQ-Trace-Id").orElseThrow());
       assertThat(send("PATCH", "/conversations/" + id, OTHER_EDITOR, "{\"title\":\"x\"}").statusCode()).isEqualTo(404);
       assertThat(send("DELETE", "/conversations/" + id, OTHER_EDITOR, null).statusCode()).isEqualTo(404);
       assertThat(send("POST", "/conversations/" + id + "/turns", OTHER_EDITOR, "{\"text\":\"hi\"}").statusCode()).isEqualTo(404);
@@ -238,7 +245,11 @@ class AssistantRouteProviderTest extends BaseAssistantTest
       HttpRequest form = HttpRequest.newBuilder(URI.create(base + "/conversations")).header("Cookie", "sessionId=" + EDITOR)
          .header("Content-Type", "application/x-www-form-urlencoded").POST(HttpRequest.BodyPublishers.ofString("title=x")).build();
       assertThat(client.send(form, HttpResponse.BodyHandlers.ofString()).statusCode()).isEqualTo(400);
-      assertThat(send("GET", "/conversations/abc", EDITOR, null).statusCode()).isEqualTo(400);
+      HttpResponse<String> invalid = send("GET", "/conversations/abc", EDITOR, null);
+      assertThat(invalid.statusCode()).isEqualTo(400);
+      assertThat(json(invalid)).containsEntry("code", "BAD_REQUEST")
+         .containsEntry("message", json(invalid).get("error"))
+         .containsEntry("traceId", invalid.headers().firstValue("X-QQQ-Trace-Id").orElseThrow());
       assertThat(send("POST", "/conversations/" + id + "/turns", EDITOR, "{}").statusCode()).isEqualTo(400);
 
       assertThat(send("DELETE", "/conversations/" + id, EDITOR, null).statusCode()).isEqualTo(204);

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/api/ClientErrorsTest.java

@@ -0,0 +1,261 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.api;
+
+
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.engine.ClientErrors;
+import com.kingsrook.qbits.assistant.health.AssistantHealthChecks;
+import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
+import com.kingsrook.qbits.assistant.model.AssistantClientError;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qbits.assistant.processes.AssistantRetentionStep;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qbits.assistant.testing.UiApiTestServer;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import org.junit.jupiter.api.Test;
+import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.EDITOR;
+import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.OTHER_EDITOR;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** POST /assistant/client-errors: bounds and scrubbing, owned ids only, the
+ ** per-user rate limit (202 accepted:false), owner security, the HTTP
+ ** contract, the health check and retention.
+ *******************************************************************************/
+class ClientErrorsTest extends BaseAssistantTest
+{
+   /*******************************************************************************
+    ** A small limit and warn threshold keep the tests short.
+    *******************************************************************************/
+   @Override
+   protected void customizeConfig(AssistantQBitConfig config)
+   {
+      config.withClientErrorsPerUserPerMinute(3).withClientErrorWarnPerHour(2);
+   }
+
+
+
+   /*******************************************************************************
+    ** The API bound to a user's own session (in QContext).
+    *******************************************************************************/
+   private AssistantApi as(String userId)
+   {
+      QSession session = AssistantTestFixture.ownerSession(userId, config);
+      QContext.init(qInstance, session);
+      return (new AssistantApi(config.getRuntime(), session));
+   }
+
+
+
+   /*******************************************************************************
+    ** The user's stored reports.
+    *******************************************************************************/
+   private List<AssistantClientError> rows(String userId) throws Exception
+   {
+      as(userId);
+      return (new AssistantStore().query(AssistantClientError.TABLE_NAME, new QQueryFilter(), AssistantClientError.class));
+   }
+
+
+
+   /*******************************************************************************
+    ** Every field is bounded and the free text scrubbed; ids are kept only when
+    ** they are the user's own.
+    *******************************************************************************/
+   @Test
+   void storesBoundedScrubbedReports() throws Exception
+   {
+      AssistantApi          api   = as(EDITOR);
+      AssistantConversation mine  = config.getRuntime().getStore().createConversation(EDITOR, "mine", null);
+      as(OTHER_EDITOR);
+      AssistantConversation other = config.getRuntime().getStore().createConversation(OTHER_EDITOR, "other", null);
+      api = as(EDITOR);
+
+      Map<String, Object> body = new LinkedHashMap<>();
+      body.put("scope", "assistant-render" + "x".repeat(100));
+      body.put("message", "Cannot read properties of undefined (reading 'length') for jane.doe@example.com card 4111 1111 1111 1111 " + "y".repeat(800));
+      body.put("errorName", "TypeError");
+      body.put("stack", "at render (https://app.example.com/_next/chunk.js?token=abc:1:2)\n" + "z".repeat(5000));
+      body.put("componentStack", "in RecordsRender\n in AssistantPart key 9f8e7d6c5b4a3928170655e4d3c2b1a0AbCdEfQx");
+      body.put("requestId", "req-123");
+      body.put("route", "https://app.example.com/app/orders/12345678?filter=secret#top");
+      body.put("conversationId", mine.getId());
+      body.put("messageId", 999_999);
+      body.put("turnId", "12");
+      body.put("partType", "tool");
+      body.put("renderType", "records");
+      body.put("appVersion", "1.2.3");
+      body.put("occurredAt", "2026-10-15T12:00:00Z");
+      body.put("unknownField", "ignored");
+      assertThat(api.reportClientError(body)).isEqualTo(Map.of("accepted", true));
+
+      body.put("conversationId", other.getId());
+      body.put("route", "javascript:alert(1)");
+      body.put("occurredAt", "not a date");
+      assertThat(as(EDITOR).reportClientError(body)).containsEntry("accepted", true);
+
+      List<AssistantClientError> rows = rows(EDITOR);
+      assertThat(rows).hasSize(2);
+      AssistantClientError row = rows.get(0);
+      assertThat(row.getUserId()).isEqualTo(EDITOR);
+      assertThat(row.getScope()).hasSize(ClientErrors.SCOPE_MAX).startsWith("assistant-render");
+      assertThat(row.getMessage()).hasSize(ClientErrors.MESSAGE_MAX).startsWith("Cannot read properties of undefined (reading 'length') for [email] card [")
+         .doesNotContain("jane.doe").doesNotContain("4111");
+      assertThat(ClientErrors.text("order 12345678 for 555 123 4567, version 1.2.3", 100)).isEqualTo("order [number] for [number], version 1.2.3");
+      assertThat(row.getStack()).hasSize(ClientErrors.STACK_MAX).contains("chunk.js?[query]").doesNotContain("token=abc");
+      assertThat(row.getComponentStack()).contains("[token]").doesNotContain("9f8e7d6c");
+      assertThat(row.getRoute()).isEqualTo("/app/orders/[number]");
+      assertThat(row.getConversationId()).isEqualTo(mine.getId());
+      assertThat(row.getMessageId()).isNull();
+      assertThat(row.getTurnId()).isNull();
+      assertThat(row.getPartType()).isEqualTo("tool");
+      assertThat(row.getRenderType()).isEqualTo("records");
+      assertThat(row.getAppVersion()).isEqualTo("1.2.3");
+      assertThat(row.getOccurredAt()).isEqualTo(Instant.parse("2026-10-15T12:00:00Z"));
+
+      AssistantClientError second = rows.get(1);
+      assertThat(second.getConversationId()).as("another user's conversation is not linked").isNull();
+      assertThat(second.getRoute()).isNull();
+      assertThat(second.getOccurredAt()).isNull();
+
+      ///////////////////////////////////////////////////////////
+      // owner-locked: another user never sees these reports   //
+      ///////////////////////////////////////////////////////////
+      assertThat(rows(OTHER_EDITOR)).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** scope and message are required; everything else is optional.
+    *******************************************************************************/
+   @Test
+   void requiresScopeAndMessage()
+   {
+      AssistantApi api = as(EDITOR);
+      assertThatThrownBy(() -> api.reportClientError(null)).isInstanceOf(AssistantApiException.class).extracting("status").isEqualTo(400);
+      assertThatThrownBy(() -> api.reportClientError(Map.of("message", "x"))).extracting("status").isEqualTo(400);
+      assertThatThrownBy(() -> api.reportClientError(Map.of("scope", "page", "message", "   "))).extracting("status").isEqualTo(400);
+      assertThatThrownBy(() -> api.reportClientError(Map.of("scope", 7, "message", "x"))).extracting("status").isEqualTo(400);
+   }
+
+
+
+   /*******************************************************************************
+    ** Over the per-user limit: still acknowledged (accepted:false), nothing
+    ** stored; other users have their own limit.
+    *******************************************************************************/
+   @Test
+   void rateLimitedPerUser() throws Exception
+   {
+      Map<String, Object> body = Map.of("scope", "page", "message", "boom");
+      for(int i = 0; i < 3; i++)
+      {
+         assertThat(as(EDITOR).reportClientError(body)).containsEntry("accepted", true);
+      }
+      assertThat(as(EDITOR).reportClientError(body)).isEqualTo(Map.of("accepted", false));
+      assertThat(as(OTHER_EDITOR).reportClientError(body)).containsEntry("accepted", true);
+      assertThat(rows(EDITOR)).hasSize(3);
+   }
+
+
+
+   /*******************************************************************************
+    ** The health check is UP below the threshold and DEGRADED at it.
+    *******************************************************************************/
+   @Test
+   void healthCheckWarnsOnManyReports() throws Exception
+   {
+      assertThat(HealthChecksMetaData.ofOrWithNew(qInstance).getChecks()).anySatisfy(check -> assertThat(check.getName()).isEqualTo(AssistantHealthChecks.CLIENT_ERRORS));
+      as(EDITOR);
+      assertThat(AssistantHealthChecks.clientErrors(config.getRuntime()).getStatus()).isEqualTo(HealthStatus.UP);
+      as(EDITOR).reportClientError(Map.of("scope", "page", "message", "one"));
+      as(EDITOR).reportClientError(Map.of("scope", "page", "message", "two"));
+      as(EDITOR);
+      assertThat(AssistantHealthChecks.clientErrors(config.getRuntime()).getStatus()).isEqualTo(HealthStatus.DEGRADED);
+      assertThat(AssistantHealthChecks.clientErrors(null).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Retention deletes reports older than the tool result retention.
+    *******************************************************************************/
+   @Test
+   void retentionDeletesOldReports() throws Exception
+   {
+      as(EDITOR).reportClientError(Map.of("scope", "page", "message", "old"));
+      as(EDITOR);
+      AssistantRetentionStep.Result kept = AssistantRetentionStep.purge(config, config.getRuntime().getStore(), Instant.now());
+      assertThat(kept.clientErrorsDeleted()).isZero();
+      AssistantRetentionStep.Result result = AssistantRetentionStep.purge(config, config.getRuntime().getStore(), Instant.now().plus(Duration.ofDays(config.getToolResultRetentionDays() + 1)));
+      assertThat(result.clientErrorsDeleted()).isEqualTo(1);
+      assertThat(rows(EDITOR)).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Over HTTP: 202 {accepted}, a non-object body or a missing field is a 400;
+    ** the table is in the Assistant nav. (The session comes from the standard
+    ** UI API authentication; the fixture's MOCK module signs anyone in.)
+    *******************************************************************************/
+   @Test
+   void httpContract() throws Exception
+   {
+      QContext.clear();
+      String     base   = UiApiTestServer.serve(qInstance, config) + AssistantRouteProvider.BASE_PATH;
+      HttpClient client = HttpClient.newHttpClient();
+
+      HttpResponse<String> accepted = client.send(HttpRequest.newBuilder(URI.create(base + "/client-errors")).header("Cookie", "sessionId=" + EDITOR)
+         .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString("{\"scope\":\"assistant-part\",\"message\":\"Cannot read properties of undefined\"}")).build(),
+         HttpResponse.BodyHandlers.ofString());
+      assertThat(accepted.statusCode()).isEqualTo(202);
+      assertThat(JsonUtils.toObject(accepted.body(), new TypeReference<Map<String, Object>>() {})).isEqualTo(Map.of("accepted", true));
+
+      HttpResponse<String> notObject = client.send(HttpRequest.newBuilder(URI.create(base + "/client-errors")).header("Cookie", "sessionId=" + EDITOR)
+         .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString("[1,2]")).build(), HttpResponse.BodyHandlers.ofString());
+      assertThat(notObject.statusCode()).isEqualTo(400);
+
+      HttpResponse<String> missing = client.send(HttpRequest.newBuilder(URI.create(base + "/client-errors")).header("Cookie", "sessionId=" + EDITOR)
+         .header("Content-Type", "application/json").POST(HttpRequest.BodyPublishers.ofString("{\"scope\":\"page\"}")).build(), HttpResponse.BodyHandlers.ofString());
+      assertThat(missing.statusCode()).isEqualTo(400);
+
+      assertThat(rows(EDITOR)).hasSize(1);
+      assertThat(AssistantMetaDataBuilder.NAV_TABLES).contains(AssistantClientError.TABLE_NAME);
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/api/specs/AssistantSpecsContractTest.java

@@ -65,8 +65,17 @@ class AssistantSpecsContractTest extends BaseAssistantTest
 
       assertThat(generated).contains("/qqq/v1/assistant/conversations/{id}/turns").contains("text/event-stream").contains("/qqq/v1/assistant/actions/{id}/confirm")
          .contains("AssistantTurnEventV1").contains("/qqq/v1/assistant/access");
+      ///////////////////////////////////////////////////////////////////////////////////////
+      // The standard error schema retains error/code and adds message/traceId/fieldErrors. //
+      // The two 204 responses never had bodies; their old refs pointed to schema "null".   //
+      // Keep the release baseline and accept only these documented migration corrections. //
+      ///////////////////////////////////////////////////////////////////////////////////////
+      List<String> acceptedChanges = List.of(
+         "$ref changed from #/components/schemas/AssistantErrorV1 to #/components/schemas/BasicErrorResponseV1",
+         "DELETE /qqq/v1/assistant/conversations/{id} response 204: content type removed: application/json",
+         "POST /qqq/v1/assistant/messages/{id}/feedback response 204: content type removed: application/json");
       List<String> problems = UiApiContract.check(generated, Files.readString(COMMITTED, StandardCharsets.UTF_8),
-         Files.exists(BASELINE) ? Files.readString(BASELINE, StandardCharsets.UTF_8) : null, 1, "./mvnw test -pl qbit-assistant -Dtest=AssistantSpecsContractTest -Dassistant.regenerateOpenApi=true");
+         Files.exists(BASELINE) ? Files.readString(BASELINE, StandardCharsets.UTF_8) : null, 1, "./mvnw test -pl qbit-assistant -Dtest=AssistantSpecsContractTest -Dassistant.regenerateOpenApi=true", acceptedChanges);
       assertThat(problems).isEmpty();
    }
 

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/DurableTurnsTest.java

@@ -208,6 +208,42 @@ class DurableTurnsTest extends BaseAssistantTest
 
 
 
+   /*******************************************************************************
+    ** Streamed text arrives on the provider's own thread; the "writing" progress
+    ** it triggers still runs in the turn's QContext (the job tracker writes
+    ** through QQQ, which needs the instance).
+    *******************************************************************************/
+   @Test
+   void streamedProgressRunsInTheTurnContext() throws Exception
+   {
+      List<String> contexts = new CopyOnWriteArrayList<>();
+      config.setRunTracker(new AssistantRunTracker()
+      {
+         @Override
+         public String start(String userId, Integer turnId, Integer conversationId, String label)
+         {
+            return ("job-" + turnId);
+         }
+
+         @Override
+         public void progress(String trackingId, String phase, String message)
+         {
+            contexts.add(phase + ":" + (QContext.getQInstance() != null));
+         }
+
+         @Override
+         public void finish(String trackingId, String status, String message)
+         {
+         }
+      });
+      provider.thenText("A streamed answer on the provider's thread");
+      AssistantTurn row = completed(api.startTurn(conversation(), Map.of("text", "hello")));
+      assertThat(row.getStatus()).isEqualTo("COMPLETE");
+      assertThat(contexts).contains("writing:true").doesNotContain("writing:false");
+   }
+
+
+
    /*******************************************************************************
     ** A throwing post-processing collaborator cannot strand a streamed answer.
     *******************************************************************************/

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/FailSoftTurnsTest.java

@@ -0,0 +1,347 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.engine;
+
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.api.AssistantApi;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qbits.assistant.model.AssistantMessage;
+import com.kingsrook.qbits.assistant.model.AssistantTurn;
+import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qbits.assistant.processes.AssistantRetentionStep;
+import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
+import com.kingsrook.qbits.assistant.spi.ModelEventListener;
+import com.kingsrook.qbits.assistant.spi.ModelMessage;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelRequest;
+import com.kingsrook.qbits.assistant.spi.StreamHandle;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qbits.assistant.testing.TurnRunner;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import org.junit.jupiter.api.Test;
+import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.EDITOR;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Fail soft: tool failures and timeouts become ERROR (is_error) results plus
+ ** an error notice; any exception in the loop still ends the turn (ERROR,
+ ** message.end, terminal row, unregistered); abandoned runs and stuck
+ ** messages are finalized when read or swept.
+ *******************************************************************************/
+class FailSoftTurnsTest extends BaseAssistantTest
+{
+   private static final String INTERNAL_DETAIL = "INTERNAL-DETAIL-7731";
+
+   /** what the UI API gateway does: THROW (an unexpected exception) or SLEEP (hang) */
+   private final AtomicReference<String> gatewayMode = new AtomicReference<>("THROW");
+
+   /** whether the provider's stream() throws */
+   private volatile boolean providerThrows = false;
+
+
+
+   /*******************************************************************************
+    ** A gateway that fails or hangs, a short tool limit, and a provider whose
+    ** stream() can throw outright (not a ModelProviderException).
+    *******************************************************************************/
+   @Override
+   protected void customizeConfig(AssistantQBitConfig config)
+   {
+      config.withToolCallTimeoutSeconds(1);
+      config.withUiApiGateway(new UiApiGateway()
+      {
+         @Override
+         public boolean isInvocable(String method, String pathTemplate)
+         {
+            return (true);
+         }
+
+
+
+         @Override
+         public UiApiResult invoke(UiApiCall call) throws QException
+         {
+            if("SLEEP".equals(gatewayMode.get()))
+            {
+               try
+               {
+                  Thread.sleep(30_000);
+               }
+               catch(InterruptedException e)
+               {
+                  Thread.currentThread().interrupt();
+                  throw (new QException("interrupted"));
+               }
+            }
+            throw (new IllegalStateException(INTERNAL_DETAIL));
+         }
+      });
+      provider = new ScriptedModelProvider()
+      {
+         @Override
+         public StreamHandle stream(ModelRequest request, ModelEventListener listener)
+         {
+            if(providerThrows)
+            {
+               throw (new IllegalStateException(INTERNAL_DETAIL));
+            }
+            return (super.stream(request, listener));
+         }
+      };
+      config.withModelProvider(provider);
+   }
+
+
+
+   /*******************************************************************************
+    ** The tool results the model got back in its second request.
+    *******************************************************************************/
+   private List<ModelPart> toolResultsSentBack()
+   {
+      ModelRequest second = provider.getRequests().get(1);
+      ModelMessage results = second.getMessages().get(second.getMessages().size() - 1);
+      return (results.getParts());
+   }
+
+
+
+   /*******************************************************************************
+    ** A tool that throws: an ERROR is_error result for the model, an ERROR tool
+    ** part and an error notice for the user, no internal detail in either.
+    *******************************************************************************/
+   @Test
+   void toolExceptionBecomesErrorPart() throws Exception
+   {
+      provider.thenToolUse("count_records", input("table", "blog_post", "filter", filter("AND"))).thenText("Sorry, I could not count them.");
+      TurnRunner.Outcome outcome = runner.run(EDITOR, "how many posts?");
+
+      assertThat(outcome.end()).containsEntry("status", "COMPLETE");
+      assertThat(outcome.events(TurnEvent.TOOL_RESULT)).singleElement().satisfies(result -> assertThat(result).containsEntry("outcome", "ERROR").containsEntry("summary", "Failed"));
+      assertThat(outcome.events(TurnEvent.NOTICE)).anySatisfy(notice -> assertThat(notice).containsEntry("level", "error"));
+      assertThat(toolResultsSentBack()).singleElement().satisfies(part -> assertThat(part.getIsError()).isTrue());
+
+      List<Map<String, Object>> parts = assistantParts(outcome.turnId());
+      assertThat(parts).anySatisfy(part -> assertThat(part).containsEntry("type", "tool").containsEntry("outcome", "ERROR"));
+      assertThat(parts).anySatisfy(part -> assertThat(part).containsEntry("type", "notice").containsEntry("level", "error"));
+      assertThat(String.valueOf(parts)).doesNotContain(INTERNAL_DETAIL);
+      outcome.events().forEach(event -> assertThat(event.toJson()).doesNotContain(INTERNAL_DETAIL));
+   }
+
+
+
+   /*******************************************************************************
+    ** A tool that hangs past toolCallTimeoutSeconds gets an ERROR (is_error)
+    ** result; the turn goes on and completes.
+    *******************************************************************************/
+   @Test
+   void toolTimeoutBecomesErrorResult() throws Exception
+   {
+      gatewayMode.set("SLEEP");
+      provider.thenToolUse("count_records", input("table", "blog_post", "filter", filter("AND"))).thenText("That took too long.");
+      long started = System.currentTimeMillis();
+      TurnRunner.Outcome outcome = runner.run(EDITOR, "how many posts?");
+
+      assertThat(System.currentTimeMillis() - started).isLessThan(20_000);
+      assertThat(outcome.end()).containsEntry("status", "COMPLETE");
+      assertThat(outcome.events(TurnEvent.TOOL_RESULT)).singleElement().satisfies(result -> assertThat(result).containsEntry("outcome", "ERROR").containsEntry("summary", "Timed out"));
+      assertThat(outcome.events(TurnEvent.NOTICE)).anySatisfy(notice -> assertThat(notice).containsEntry("level", "error"));
+      assertThat(toolResultsSentBack()).singleElement().satisfies(part ->
+      {
+         assertThat(part.getIsError()).isTrue();
+         assertThat(part.getText()).contains("did not finish in time");
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** The turn's own limit: status ERROR, an error event with code TIMEOUT,
+    ** then message.end.
+    *******************************************************************************/
+   @Test
+   void turnTimeoutEndsWithTimeoutError() throws Exception
+   {
+      config.setMaxTurnSeconds(1);
+      provider.withDelay(Duration.ofSeconds(3)).thenText("too late");
+      TurnRunner.Outcome outcome = runner.run(EDITOR, "slow");
+      assertThat(outcome.end()).containsEntry("status", "ERROR");
+      assertThat(outcome.events(TurnEvent.ERROR)).singleElement().satisfies(error -> assertThat(error).containsEntry("code", "TIMEOUT"));
+      List<TurnEvent> events = outcome.events();
+      assertThat(events.getLast().getType()).isEqualTo(TurnEvent.MESSAGE_END);
+      assertThat(turnRow(outcome.turnId()).getStatus()).isEqualTo("ERROR");
+   }
+
+
+
+   /*******************************************************************************
+    ** An unexpected exception anywhere in the loop: ERROR, message.end, the turn
+    ** row and the message finalized, the run unregistered, and the next
+    ** question is not locked out.
+    *******************************************************************************/
+   @Test
+   void loopExceptionStillEndsTheTurn() throws Exception
+   {
+      providerThrows = true;
+      TurnRunner.Outcome failed = runner.run(EDITOR, "hello");
+      assertThat(failed.end()).containsEntry("status", "ERROR");
+      assertThat(failed.errorCode()).isEqualTo("INTERNAL");
+      assertThat(failed.events().getLast().getType()).isEqualTo(TurnEvent.MESSAGE_END);
+      assertThat(String.valueOf(failed.events())).doesNotContain(INTERNAL_DETAIL);
+      assertThat(runner.runtime().getRunningTurn(failed.turnId())).isNull();
+
+      AssistantTurn row = turnRow(failed.turnId());
+      assertThat(row.getStatus()).isEqualTo("ERROR");
+      assertThat(row.getErrorCode()).isEqualTo("INTERNAL");
+      assertThat(row.getEndedAt()).isNotNull();
+      assertThat(message(row.getAssistantMessageId()).getStatus()).isEqualTo("ERROR");
+      assertThat(assistantParts(failed.turnId())).anySatisfy(part -> assertThat(part).containsEntry("type", "notice").containsEntry("level", "error"));
+
+      providerThrows = false;
+      provider.thenText("Fine now.");
+      assertThat(runner.run(EDITOR, failed.conversationId(), "again", null).end()).containsEntry("status", "COMPLETE");
+   }
+
+
+
+   /*******************************************************************************
+    ** Conversation detail repairs a message left STREAMING: by a run that
+    ** already ended (its status), and by an abandoned run (reaped, ERROR).
+    ** Legacy parts come back normalized.
+    *******************************************************************************/
+   @Test
+   @SuppressWarnings("unchecked")
+   void conversationDetailRepairsStuckMessages() throws Exception
+   {
+      QSession session = AssistantTestFixture.ownerSession(EDITOR, config);
+      QContext.init(qInstance, session);
+      AssistantStore        store        = config.getRuntime().getStore();
+      AssistantApi          api          = new AssistantApi(config.getRuntime(), session);
+      AssistantConversation conversation = store.createConversation(EDITOR, "stuck", null);
+
+      AssistantTurn ended = store.insert(new AssistantTurn().withConversationId(conversation.getId()).withUserId(EDITOR).withStatus("COMPLETE").withStartedAt(Instant.now()));
+      store.insert(new AssistantMessage().withConversationId(conversation.getId()).withUserId(EDITOR).withRole("ASSISTANT").withStatus("STREAMING").withSeq(2)
+         .withTurnId(ended.getId()).withPartsJson("[{\"type\":\"tool\",\"name\":\"query_records\",\"render\":{\"type\":\"records\",\"tableName\":\"t\"}}]"));
+
+      AssistantTurn abandoned = store.insert(new AssistantTurn().withConversationId(conversation.getId()).withUserId(EDITOR).withStatus("RUNNING")
+         .withStartedAt(Instant.now().minus(Duration.ofHours(1))).withHeartbeatAt(Instant.now().minus(Duration.ofHours(1))));
+      AssistantMessage stuck = store.insert(new AssistantMessage().withConversationId(conversation.getId()).withUserId(EDITOR).withRole("ASSISTANT").withStatus("STREAMING").withSeq(4)
+         .withTurnId(abandoned.getId()).withPartsJson("[{\"type\":\"text\",\"text\":\"Partial\"}]"));
+      store.updateFields(AssistantTurn.TABLE_NAME, new QRecord().withValue("id", abandoned.getId()).withValue("assistantMessageId", stuck.getId()));
+
+      AssistantMessage orphan = store.insert(new AssistantMessage().withConversationId(conversation.getId()).withUserId(EDITOR).withRole("ASSISTANT").withStatus("STREAMING").withSeq(6)
+         .withPartsJson("[]"));
+
+      List<Map<String, Object>> messages = (List<Map<String, Object>>) api.getConversation(conversation.getId(), null, null).get("messages");
+      assertThat(messages).extracting(m -> m.get("status")).containsExactly("COMPLETE", "ERROR", "ERROR");
+      Map<String, Object> render = (Map<String, Object>) ((List<Map<String, Object>>) messages.get(0).get("parts")).get(0).get("render");
+      assertThat(render).containsEntry("rows", List.of()).containsEntry("columns", List.of());
+      assertThat(String.valueOf(messages.get(1).get("parts"))).contains(AbandonedTurns.INTERRUPTED_TEXT);
+      assertThat(String.valueOf(messages.get(2).get("parts"))).contains(AbandonedTurns.UNFINISHED_TEXT);
+
+      assertThat(store.getTurn(EDITOR, abandoned.getId()).getStatus()).isEqualTo("ERROR");
+      assertThat(store.getMessage(EDITOR, orphan.getId()).getStatus()).isEqualTo("ERROR");
+      assertThat(api.activeTurns().get("turns")).isEqualTo(List.of());
+   }
+
+
+
+   /*******************************************************************************
+    ** The retention process finalizes runs nobody read again (e.g. after a pod
+    ** restart), with their messages and a terminal replay.
+    *******************************************************************************/
+   @Test
+   void retentionSweepsAbandonedRuns() throws Exception
+   {
+      QSession session = AssistantTestFixture.ownerSession(EDITOR, config);
+      QContext.init(qInstance, session);
+      AssistantStore        store        = config.getRuntime().getStore();
+      AssistantConversation conversation = store.createConversation(EDITOR, "restarted", null);
+      AssistantMessage message = store.insert(new AssistantMessage().withConversationId(conversation.getId()).withUserId(EDITOR).withRole("ASSISTANT").withStatus("STREAMING").withSeq(2)
+         .withPartsJson("[]"));
+      AssistantTurn old = store.insert(new AssistantTurn().withConversationId(conversation.getId()).withUserId(EDITOR).withStatus("RUNNING")
+         .withAssistantMessageId(message.getId()).withStartedAt(Instant.now().minus(Duration.ofHours(2))));
+      AssistantTurn fresh = store.insert(new AssistantTurn().withConversationId(conversation.getId()).withUserId(EDITOR).withStatus("RUNNING")
+         .withStartedAt(Instant.now()).withHeartbeatAt(Instant.now()));
+
+      AssistantRetentionStep.Result result = AssistantRetentionStep.purge(config, store, Instant.now());
+      assertThat(result.turnsFinalized()).isEqualTo(1);
+      assertThat(store.getTurn(EDITOR, old.getId()).getStatus()).isEqualTo("ERROR");
+      assertThat(store.getTurn(EDITOR, fresh.getId()).getStatus()).isEqualTo("RUNNING");
+      assertThat(store.getMessage(EDITOR, message.getId()).getStatus()).isEqualTo("ERROR");
+      assertThat(store.listEvents(EDITOR, old.getId(), 0, 10)).extracting(e -> e.getEventType()).containsExactly(TurnEvent.ERROR, TurnEvent.MESSAGE_END);
+   }
+
+
+
+   /*******************************************************************************
+    ** The turn row, read as its owner.
+    *******************************************************************************/
+   private AssistantTurn turnRow(Integer turnId) throws Exception
+   {
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(EDITOR, config));
+      try
+      {
+         return (config.getRuntime().getStore().getTurn(EDITOR, turnId));
+      }
+      finally
+      {
+         QContext.clear();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A message, read as its owner.
+    *******************************************************************************/
+   private AssistantMessage message(Integer messageId) throws Exception
+   {
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(EDITOR, config));
+      try
+      {
+         return (config.getRuntime().getStore().getMessage(EDITOR, messageId));
+      }
+      finally
+      {
+         QContext.clear();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The stored parts of a turn's assistant message.
+    *******************************************************************************/
+   private List<Map<String, Object>> assistantParts(Integer turnId) throws Exception
+   {
+      return (MessageParts.fromJson(message(turnRow(turnId).getAssistantMessageId()).getPartsJson()).list());
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/RenderShapesTest.java

@@ -0,0 +1,208 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.engine;
+
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Records renders keep their (empty) lists in stored parts and stream events,
+ ** and parts stored without them are completed when read.
+ *******************************************************************************/
+class RenderShapesTest
+{
+
+   /*******************************************************************************
+    ** A query that matched nothing.
+    *******************************************************************************/
+   private static Map<String, Object> emptyRecordsPart()
+   {
+      Map<String, Object> render = new LinkedHashMap<>();
+      render.put("type", "records");
+      render.put("tableName", "db_perf_finding");
+      render.put("tableLabel", "Findings");
+      render.put("columns", new ArrayList<>());
+      render.put("rows", new ArrayList<>());
+      render.put("totalCount", 0);
+      render.put("truncated", false);
+      render.put("openHref", "/app/db_perf_finding");
+      Map<String, Object> part = new LinkedHashMap<>();
+      part.put("type", "tool");
+      part.put("name", "query_records");
+      part.put("render", render);
+      return (part);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void emptyListsSurviveSerialization()
+   {
+      String json = RenderShapes.toJson(List.of(emptyRecordsPart()));
+      assertThat(json).contains("\"columns\":[]").contains("\"rows\":[]");
+      assertThat(new TurnEvent(1, "tool.result", emptyRecordsPart()).toJson()).contains("\"rows\":[]").contains("\"columns\":[]");
+   }
+
+
+
+   /*******************************************************************************
+    ** The exact shape stored on kof22 before the fix (lists omitted).
+    *******************************************************************************/
+   @Test
+   void partsStoredWithoutListsAreCompletedWhenRead()
+   {
+      String stored = "[{\"type\":\"tool\",\"name\":\"query_records\",\"outcome\":\"OK\",\"render\":{\"type\":\"records\",\"tableName\":\"db_perf_finding\",\"tableLabel\":\"Findings\",\"openHref\":\"/app/db_perf_finding\",\"totalCount\":0,\"truncated\":false}},"
+         + "{\"type\":\"tool\",\"name\":\"history\",\"render\":{\"type\":\"history\",\"entries\":[{\"message\":\"Edited\"}]}}]";
+      List<Map<String, Object>> parts = MessageParts.fromJson(stored).list();
+      @SuppressWarnings("unchecked")
+      Map<String, Object> records = (Map<String, Object>) parts.get(0).get("render");
+      assertThat(records.get("rows")).isEqualTo(List.of());
+      assertThat(records.get("columns")).isEqualTo(List.of());
+      @SuppressWarnings("unchecked")
+      Map<String, Object> history = (Map<String, Object>) parts.get(1).get("render");
+      @SuppressWarnings("unchecked")
+      Map<String, Object> entry = ((List<Map<String, Object>>) history.get("entries")).get(0);
+      assertThat(entry.get("details")).isEqualTo(List.of());
+
+      Map<String, Object> event = TurnEvent.fromJson(3, "tool.result", "{\"render\":{\"type\":\"records\",\"tableLabel\":\"Findings\"}}").getData();
+      @SuppressWarnings("unchecked")
+      Map<String, Object> eventRender = (Map<String, Object>) event.get("render");
+      assertThat(eventRender.get("rows")).isEqualTo(List.of());
+   }
+
+
+
+   /*******************************************************************************
+    ** One render through the normalizer, as a stored part would be read.
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static Map<String, Object> renderOf(String renderJson)
+   {
+      List<Map<String, Object>> parts = MessageParts.fromJson("[{\"type\":\"tool\",\"name\":\"t\",\"render\":" + renderJson + "}]").list();
+      assertThat(parts).hasSize(1);
+      return ((Map<String, Object>) parts.get(0).get("render"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Every known render type gets its required lists and safe defaults.
+    *******************************************************************************/
+   @Test
+   @SuppressWarnings("unchecked")
+   void everyRenderTypeIsNormalized()
+   {
+      Map<String, Object> records = renderOf("{\"type\":\"records\",\"tableName\":\"t\",\"columns\":[{\"name\":\"a\"},\"junk\",{\"label\":\"no name\"}],\"rows\":[{\"id\":1},7],\"totalCount\":\"3\",\"truncated\":\"yes\"}");
+      assertThat(records).containsEntry("tableLabel", "t").containsEntry("totalCount", 3L).containsEntry("truncated", false);
+      assertThat((List<Map<String, Object>>) records.get("columns")).containsExactly(Map.of("name", "a", "label", "a"));
+      assertThat((List<Map<String, Object>>) records.get("rows")).hasSize(1).first().satisfies(row -> assertThat(row.get("values")).isEqualTo(Map.of()));
+
+      Map<String, Object> record = renderOf("{\"type\":\"record\",\"tableName\":\"t\",\"fields\":[{\"name\":\"n\",\"value\":5},{\"value\":\"orphan\"}]}");
+      assertThat((List<Map<String, Object>>) record.get("fields")).containsExactly(Map.of("name", "n", "label", "n", "value", "5"));
+
+      Map<String, Object> count = renderOf("{\"type\":\"count\",\"tableName\":\"t\",\"count\":\"many\"}");
+      assertThat(count).containsEntry("tableLabel", "t").containsEntry("count", null);
+
+      Map<String, Object> widget = renderOf("{\"type\":\"widget\",\"widgetName\":\"w\",\"data\":\"bad\"}");
+      assertThat(widget).containsEntry("label", "w").containsEntry("data", Map.of());
+
+      Map<String, Object> history = renderOf("{\"type\":\"history\",\"entries\":[{\"message\":\"x\",\"details\":\"bad\"},3]}");
+      assertThat((List<Map<String, Object>>) history.get("entries")).containsExactly(Map.of("message", "x", "details", List.of()));
+
+      Map<String, Object> pending = renderOf("{\"type\":\"pending_action\",\"actionId\":\"12\"}");
+      assertThat(pending).containsEntry("actionId", 12L).containsEntry("summary", "").containsEntry("changes", List.of());
+
+      assertThat(renderOf("{\"type\":\"bulk_load_dry_run\",\"okCount\":1}")).containsEntry("errors", List.of());
+      assertThat(renderOf("{\"type\":\"records\"}")).containsEntry("rows", List.of()).containsEntry("columns", List.of()).containsEntry("tableLabel", "");
+   }
+
+
+
+   /*******************************************************************************
+    ** Unknown render and part types pass through untouched.
+    *******************************************************************************/
+   @Test
+   void unknownTypesPassThrough()
+   {
+      assertThat(renderOf("{\"type\":\"sparkline\",\"points\":\"whatever\"}")).isEqualTo(Map.of("type", "sparkline", "points", "whatever"));
+      List<Map<String, Object>> parts = MessageParts.fromJson("[{\"type\":\"future_part\",\"x\":1}]").list();
+      assertThat(parts).containsExactly(Map.of("type", "future_part", "x", 1));
+   }
+
+
+
+   /*******************************************************************************
+    ** What cannot be repaired becomes the error notice; nothing throws.
+    *******************************************************************************/
+   @Test
+   void unrepairableRendersBecomeNotices()
+   {
+      List<Map<String, Object>> parts = MessageParts.fromJson("[{\"type\":\"tool\",\"name\":\"t\",\"outcome\":\"OK\",\"render\":{\"type\":\"pending_action\"}},"
+         + "{\"type\":\"tool\",\"name\":\"u\",\"render\":\"not an object\"},"
+         + "42, {\"no\":\"type\"}, {\"type\":\"presentation\"},"
+         + "{\"type\":\"navigate\",\"href\":\"https://evil.example\"}, {\"type\":\"notice\",\"level\":\"panic\"}]").list();
+      Map<String, Object> notice = RenderShapes.undisplayableNotice();
+      assertThat(parts).containsExactly(
+         Map.of("type", "tool", "name", "t", "outcome", "OK", "summary", ""), notice,
+         Map.of("type", "tool", "name", "u", "outcome", "ERROR", "summary", ""), notice,
+         notice, notice, notice,
+         Map.of("type", "notice", "level", "info", "text", ""));
+      assertThat(notice).containsEntry("level", "error").containsEntry("text", RenderShapes.UNDISPLAYABLE_TEXT);
+      assertThat(MessageParts.fromJson("{\"not\":\"a list\"}").list()).isEmpty();
+      assertThat(RenderShapes.parts(null)).isEmpty();
+      assertThat(RenderShapes.render(null)).isNull();
+   }
+
+
+
+   /*******************************************************************************
+    ** Presentations: blocks keep their lists; a presentation with nothing to
+    ** show is replaced (in parts and in the presentation event).
+    *******************************************************************************/
+   @Test
+   @SuppressWarnings("unchecked")
+   void presentationsAreNormalized()
+   {
+      Map<String, Object> presentation = RenderShapes.presentation(Map.of("summary", "Sales", "blocks", List.of(
+         Map.of("kind", "table", "columns", List.of(Map.of("key", "c0"), "junk"), "rows", List.of(Map.of("cells", "bad"))),
+         Map.of("kind", "chart"), Map.of("kind", "metrics"), Map.of("no", "kind"), Map.of("kind", "future"))));
+      assertThat(presentation).containsEntry("type", "presentation").containsEntry("markdown", "Sales");
+      List<Map<String, Object>> blocks = (List<Map<String, Object>>) presentation.get("blocks");
+      assertThat(blocks).hasSize(4);
+      assertThat(blocks.get(0)).containsEntry("title", "").containsEntry("columns", List.of(Map.of("key", "c0", "label", "c0")))
+         .containsEntry("rows", List.of(Map.of("cells", Map.of(), "display", Map.of())));
+      assertThat(blocks.get(1)).containsEntry("chartType", "bar").containsEntry("points", List.of());
+      assertThat(blocks.get(2)).containsEntry("items", List.of());
+      assertThat(blocks.get(3)).isEqualTo(Map.of("kind", "future"));
+
+      assertThat(RenderShapes.presentation(Map.of("blocks", List.of()))).isNull();
+      TurnEvent event = new TurnEvent(4, TurnEvent.PRESENTATION, Map.of("part", Map.of("type", "presentation")));
+      assertThat(event.getData().get("part")).isEqualTo(RenderShapes.undisplayableNotice());
+      TurnEvent toolResult = TurnEvent.fromJson(5, TurnEvent.TOOL_RESULT, "{\"name\":\"t\",\"render\":{\"type\":\"pending_action\"}}");
+      assertThat(toolResult.getData()).doesNotContainKey("render").containsEntry("name", "t");
+      assertThat(TurnEvent.fromJson(6, TurnEvent.TOOL_RESULT, "not json").getData()).containsOnlyKeys("seq");
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/RoutingFallbackTest.java

@@ -676,7 +676,7 @@ class RoutingFallbackTest extends BaseAssistantTest
       assertThat(script("a").getRequests()).isEmpty();
       assertThat(store().getProvider(a.getProviderId()).getHealthState()).isEqualTo("DOWN");
       String html = AssistantHomeView.render(AssistantHomeData.load(config.getRuntime()), "");
-      assertThat(html).contains("DOWN", "Down until", "0 fallbacks");
+      assertThat(html).contains("0 fallbacks").doesNotContain("Down until", "Last test failed");
    }
 
 

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/health/AssistantHealthChecksTest.java

@@ -0,0 +1,627 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.health;
+
+
+import java.math.BigDecimal;
+import java.time.Clock;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.engine.ProviderHealth;
+import com.kingsrook.qbits.assistant.engine.ProviderService;
+import com.kingsrook.qbits.assistant.engine.UsageRecorder;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantPolicy;
+import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.model.AssistantUsage;
+import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
+import com.kingsrook.qbits.assistant.spi.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.spi.AssistantProviderType;
+import com.kingsrook.qbits.assistant.spi.EmbeddingRequest;
+import com.kingsrook.qbits.assistant.spi.EmbeddingResponse;
+import com.kingsrook.qbits.assistant.spi.HealthResult;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelUsage;
+import com.kingsrook.qbits.assistant.spi.ProviderSettings;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthRunner;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.model.session.QUser;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
+
+
+/*******************************************************************************
+ ** Registration, deadlines, billing and secret safety through the memory backend.
+ *******************************************************************************/
+class AssistantHealthChecksTest extends BaseAssistantTest
+{
+   private static final String SECRET = "S3NT1NEL-secret";
+   private final FakeType type = new FakeType();
+   private Boolean healthy = true;
+   private Boolean failProbe = false;
+   private Boolean failCreate = false;
+   private Boolean failEmbedding = false;
+   private Integer probes = 0;
+   private Integer embeddings = 0;
+   private ProviderSettings lastSettings;
+   private CountDownLatch probeWait;
+   private final CountDownLatch interrupted = new CountDownLatch(1);
+
+
+
+   /*******************************************************************************
+    ** Resolve a poisoned secret through the normal key-reference mechanism.
+    *******************************************************************************/
+   @Override
+   protected void customizeConfig(AssistantQBitConfig config)
+   {
+      provider = new EmbeddingProvider();
+      config.withModelProvider(provider).withProviderType(type).withAllowedApiKeyEnvNames(Set.of("HEALTH_TEST_KEY"))
+         .withSecretResolver(name -> SECRET);
+   }
+
+
+
+   /*******************************************************************************
+    ** An adapter with no network and controllable provider failures.
+    *******************************************************************************/
+   class FakeType implements AssistantProviderType
+   {
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public String getType()
+      {
+         return ("health-fake");
+      }
+
+
+
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public String getLabel()
+      {
+         return ("Health fake");
+      }
+
+
+
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public List<String> getModels()
+      {
+         return (List.of("model"));
+      }
+
+
+
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public String getDefaultApiKeyEnvName()
+      {
+         return ("HEALTH_TEST_KEY");
+      }
+
+
+
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public String getDefaultBaseUrl()
+      {
+         return ("https://health.invalid");
+      }
+
+
+
+      /***************************************************************************
+       ** Fake provider contract.
+       ***************************************************************************/
+      @Override
+      public String getDefaultLocation(String kind)
+      {
+         return ("LOCAL");
+      }
+
+
+
+      /***************************************************************************
+       ** Reuse the existing scripted completion fake.
+       ***************************************************************************/
+      @Override
+      public AssistantModelProvider create(String key, String url)
+      {
+         assertThat(key).isEqualTo(SECRET);
+         if(failCreate)
+         {
+            throw (new IllegalStateException(SECRET));
+         }
+         return (provider);
+      }
+
+
+
+      /***************************************************************************
+       ** Verify bounded settings and exercise hostile exception messages.
+       ***************************************************************************/
+      @Override
+      public HealthResult checkHealth(ProviderSettings settings)
+      {
+         probes++;
+         lastSettings = settings;
+         assertThat(settings.getApiKey()).isEqualTo(SECRET);
+         if(failProbe)
+         {
+            throw (new IllegalStateException(SECRET));
+         }
+         if(probeWait != null)
+         {
+            try
+            {
+               probeWait.await();
+            }
+            catch(InterruptedException e)
+            {
+               interrupted.countDown();
+               Thread.currentThread().interrupt();
+            }
+         }
+         return (new HealthResult(healthy, 7L, SECRET));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Keep embedding tests on the existing scripted-provider fixture as well.
+    *******************************************************************************/
+   class EmbeddingProvider extends ScriptedModelProvider
+   {
+      /***************************************************************************
+       ** Minimal embedding result; no external service is contacted.
+       ***************************************************************************/
+      @Override
+      public EmbeddingResponse embed(EmbeddingRequest request) throws ModelProviderException
+      {
+         embeddings++;
+         if(failEmbedding)
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, SECRET));
+         }
+         return (new EmbeddingResponse(List.of(new float[] { 1, 0 }), new ModelUsage().withInputTokens(2L), "fake"));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Set up real settings records in the existing admin fixture session.
+    *******************************************************************************/
+   private AssistantProvider row() throws Exception
+   {
+      QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+      return (config.getRuntime().getStore().insert(new AssistantProvider().withName("Health test")
+         .withProviderType(type.getType()).withEnabled(true)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Enabled chat model with a stable integer id.
+    *******************************************************************************/
+   private AssistantModel model(AssistantProvider row) throws Exception
+   {
+      return (config.getRuntime().getStore().insert(new AssistantModel().withProviderId(row.getId()).withModelId("model").withEnabled(true)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Exercise the producer-installed suppliers, not a separately constructed check.
+    *******************************************************************************/
+   private HealthCheckMetaData check(String name) throws Exception
+   {
+      return (HealthChecksMetaData.ofOrWithNew(qInstance).resolveAll(qInstance).stream()
+         .filter(check -> name.equals(check.getName())).findFirst().orElseThrow());
+   }
+
+
+
+   /*******************************************************************************
+    ** Raw outcomes must be clean; core's sanitize method is package-private.
+    *******************************************************************************/
+   private HealthCheckOutcome run(HealthCheckMetaData check) throws Exception
+   {
+      HealthCheckOutcome outcome = check.getCheckInstance().run(new HealthCheckInput().withQInstance(qInstance).withDeadline(Instant.now().plusSeconds(5)));
+      assertThat(outcome.getMessage()).doesNotContain(SECRET, "health.invalid", "Connection test.", "\n");
+      assertThat(outcome.getDetails().toJson()).doesNotContain(SECRET, "health.invalid", "Connection test.");
+      return (outcome);
+   }
+
+
+
+   /*******************************************************************************
+    ** Both suppliers are installed by production, refresh enabled sets and use AI.
+    *******************************************************************************/
+   @Test
+   void registration() throws Exception
+   {
+      AssistantProvider local = row();
+      AssistantProvider cloud = config.getRuntime().getStore().insert(new AssistantProvider().withName("Cloud")
+         .withProviderType(type.getType()).withEnabled(true).withLocation("CLOUD"));
+      AssistantModel model = model(local);
+      model(cloud);
+      config.getRuntime().getStore().insert(new AssistantProvider().withName("Disabled").withProviderType(type.getType()).withEnabled(false));
+      config.getRuntime().getStore().insert(new AssistantModel().withProviderId(local.getId()).withModelId("disabled").withEnabled(false));
+      var registry = HealthChecksMetaData.ofOrWithNew(qInstance);
+      AssistantHealthChecks.register(qInstance);
+      assertThat(registry.getSuppliers()).hasSize(2).allSatisfy(supplier -> assertThat(supplier.getComponent()).isEqualTo("assistant"));
+      assertThat(registry.resolveAll(qInstance)).hasSize(5).allSatisfy(check ->
+      {
+         assertThat(check.getKind()).isEqualTo(HealthCheckKind.DEPENDENCY);
+         assertThat(check.getCategory()).isEqualTo(HealthCategory.AI);
+         assertThat(check.getComponent()).isEqualTo("assistant");
+         assertThat(check.getName()).matches("[a-z0-9]+([._-][a-z0-9]+)*");
+      });
+      assertThat(check("assistant.provider." + local.getId()).getIntervalSeconds()).isEqualTo(60);
+      assertThat(check("assistant.provider." + cloud.getId()).getIntervalSeconds()).isEqualTo(300);
+      var modelCheck = check("assistant.model." + model.getId());
+      assertThat(modelCheck.getIntervalSeconds()).isEqualTo(21600);
+      assertThat(modelCheck.getTimeoutMs()).isEqualTo(30000);
+      assertThat(check("assistant.provider." + local.getId()).getTimeoutMs()).isEqualTo(10000);
+      config.getRuntime().getStore().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", model.getId()).withValue("enabled", false));
+      assertThat(registry.resolveAll(qInstance)).hasSize(4);
+      assertThat(run(modelCheck).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** HealthResult messages and thrown adapter messages never enter outcomes.
+    *******************************************************************************/
+   @Test
+   void providerStatesAndNoSecrets() throws Exception
+   {
+      AssistantProvider row = row();
+      var check = check("assistant.provider." + row.getId());
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UP);
+      assertThat(lastSettings.getRequestTimeoutSeconds()).isLessThanOrEqualTo(5);
+      healthy = false;
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+      failProbe = true;
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+      failProbe = false;
+      healthy = null;
+      assertThat(run(check).getMessage()).isEqualTo("health probe not supported");
+      for(String state : List.of("UP", "DOWN", "DEGRADED"))
+      {
+         config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId())
+            .withValue("healthState", state).withValue("healthCheckedAt", Instant.now().minusSeconds(30)));
+         assertThat(run(check).getStatus()).isEqualTo(HealthStatus.valueOf(state));
+      }
+      for(Instant time : List.of(Instant.now().minusSeconds(600), Instant.now().plusSeconds(30)))
+      {
+         config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("healthCheckedAt", time));
+         assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      }
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("enabled", false));
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Missing adapter/configuration is safe and does not dispatch a completion.
+    *******************************************************************************/
+   @Test
+   void missingAdapterAndSecret() throws Exception
+   {
+      AssistantProvider row = row();
+      var check = check("assistant.provider." + row.getId());
+      config.withSecretResolver(name -> null);
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+      config.getProviderTypes().remove(type.getType());
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Successful and failed completions use TEST and preserve manual test history.
+    *******************************************************************************/
+   @Test
+   void modelAccountingAndNoSecrets() throws Exception
+   {
+      AssistantModel model = model(row());
+      Instant previous = Instant.now().minusSeconds(1000);
+      config.getRuntime().getStore().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", model.getId())
+         .withValue("lastTestAt", previous).withValue("lastTestStatus", "OK"));
+      var check = check("assistant.model." + model.getId());
+      provider.thenText(SECRET);
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UP);
+      assertThat(provider.getRequests().getFirst().getMaxTokens()).isEqualTo(32);
+      assertThat(probes).isZero();
+      provider.thenError(ModelProviderException.Kind.NETWORK, SECRET);
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+      var usage = config.getRuntime().getStore().queryRecords(AssistantUsage.TABLE_NAME, new QQueryFilter(), 10);
+      assertThat(usage).hasSize(2).allSatisfy(record -> assertThat(record.getValueString("purpose")).isEqualTo(UsageRecorder.PURPOSE_TEST));
+      assertThat(config.getRuntime().getStore().listModels().getFirst().getLastTestAt()).isEqualTo(previous);
+      failCreate = true;
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Monthly cap skips even on-demand checks, before any billable request.
+    *******************************************************************************/
+   @Test
+   void modelCostCapAndDisabledProvider() throws Exception
+   {
+      AssistantProvider row = row();
+      AssistantModel model = model(row);
+      var check = check("assistant.model." + model.getId());
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("monthlyCostCapUsd", BigDecimal.ZERO));
+      var outcome = check.getCheckInstance().run(new HealthCheckInput().withQInstance(qInstance).withDeadline(Instant.now().plusSeconds(5)).withOnDemand(true));
+      assertThat(outcome.getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      assertThat(outcome.getMessage()).isEqualTo("skipped: cost cap");
+      assertThat(provider.getRequests()).isEmpty();
+      assertThat(config.getRuntime().getStore().queryRecords(AssistantUsage.TABLE_NAME, new QQueryFilter(), 10)).isEmpty();
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("enabled", false));
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** The real runner identity records only its own usage and sees shared caps.
+    *******************************************************************************/
+   @Test
+   void automatedSessionAccountingAndSharedCap() throws Exception
+   {
+      AssistantProvider row = row();
+      AssistantModel model = model(row);
+      var check = check("assistant.model." + model.getId());
+      QContext.init(qInstance, new QSession().withUser(new QUser().withIdReference("health_runner"))
+         .withSecurityKeyValues(Map.of()).withPermissions("health.read", "health.write"));
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UP);
+      assertThat(QContext.getQSession().getSecurityKeyValues()).containsOnlyKeys(config.getOwnerSecurityKeyType());
+      assertThat(QContext.getQSession().hasPermission("assistant.admin")).isFalse();
+      var usage = config.getRuntime().getStore().queryRecords(AssistantUsage.TABLE_NAME, new QQueryFilter(), 10);
+      assertThat(usage).hasSize(1);
+      assertThat(usage.getFirst().getValueString("userId")).isEqualTo("health_runner");
+      var healthSession = QContext.getQSession();
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(AssistantTestFixture.ADMIN, config));
+      config.getRuntime().getStore().insert(new AssistantUsage().withUserId(AssistantTestFixture.EDITOR).withProviderId(row.getId())
+         .withCostMicros(1_000_000L).withCreateDate(Instant.now()).withPurpose("TURN").withStatus("OK"));
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("monthlyCostCapUsd", BigDecimal.ONE));
+      QContext.init(qInstance, healthSession);
+      assertThat(config.getRuntime().getProviderService().monthToDateCostMicros(row.getId())).isZero();
+      assertThat(run(check).getMessage()).isEqualTo("skipped: cost cap");
+      assertThat(provider.getRequests()).hasSize(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** The kill switch is read fresh before every scheduled model call.
+    *******************************************************************************/
+   @Test
+   void modelKillSwitch() throws Exception
+   {
+      var check = check("assistant.model." + model(row()).getId());
+      config.getRuntime().getStore().insert(new AssistantPolicy().withName("Stop").withKillSwitch(true));
+      assertThat(run(check).getMessage()).isEqualTo("skipped: kill switch");
+      assertThat(provider.getRequests()).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Embedding models reuse ConnectionTest's embedding path, never chat.
+    *******************************************************************************/
+   @Test
+   void embeddingAccountingAndNoSecrets() throws Exception
+   {
+      AssistantProvider row = row();
+      AssistantModel model = config.getRuntime().getStore().insert(new AssistantModel().withProviderId(row.getId()).withModelId("embedding")
+         .withModelKind("EMBEDDING").withEnabled(true));
+      var check = check("assistant.model." + model.getId());
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.UP);
+      failEmbedding = true;
+      assertThat(run(check).getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(embeddings).isEqualTo(2);
+      assertThat(provider.getRequests()).isEmpty();
+      assertThat(config.getRuntime().getStore().queryRecords(AssistantUsage.TABLE_NAME, new QQueryFilter(), 10))
+         .hasSize(2).allSatisfy(record -> assertThat(record.getValueString("purpose")).isEqualTo("TEST"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Expired input performs no work; a stuck existing probe is interrupted.
+    *******************************************************************************/
+   @Test
+   void providerDeadline() throws Exception
+   {
+      var check = check("assistant.provider." + row().getId());
+      assertThat(check.getCheckInstance().run(new HealthCheckInput().withQInstance(qInstance).withDeadline(Instant.now().minusSeconds(1))).getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(probes).isZero();
+      probeWait = new CountDownLatch(1);
+      assertTimeoutPreemptively(Duration.ofSeconds(3), () ->
+      {
+         QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+         try
+         {
+            var outcome = check.getCheckInstance().run(new HealthCheckInput().withQInstance(qInstance).withDeadline(Instant.now().plusMillis(200)));
+            assertThat(outcome.getMessage()).isEqualTo("Health check timed out");
+            assertThat(interrupted.await(1, TimeUnit.SECONDS)).isTrue();
+         }
+         finally
+         {
+            QContext.clear();
+         }
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** A slow completion cannot outlive the caller's deadline.
+    *******************************************************************************/
+   @Test
+   void modelDeadline() throws Exception
+   {
+      var check = check("assistant.model." + model(row()).getId());
+      provider.withDelay(Duration.ofSeconds(10)).thenText("OK");
+      assertTimeoutPreemptively(Duration.ofSeconds(3), () ->
+      {
+         QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+         try
+         {
+            var outcome = check.getCheckInstance().run(new HealthCheckInput().withQInstance(qInstance).withDeadline(Instant.now().plusMillis(300)));
+            assertThat(outcome.getStatus()).isEqualTo(HealthStatus.DOWN);
+         }
+         finally
+         {
+            QContext.clear();
+         }
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Drive the real runner cache without its scheduler or health-table writes.
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private HealthRunner cached(String name, HealthCheckOutcome outcome, Instant at) throws Exception
+   {
+      HealthRunner runner = new HealthRunner(qInstance, "test", Clock.fixed(at, ZoneOffset.UTC));
+      var cache = HealthRunner.class.getDeclaredMethod("cacheOutcome", String.class, HealthCheckOutcome.class);
+      cache.setAccessible(true);
+      cache.invoke(runner, name, outcome);
+      var runners = HealthRunner.class.getDeclaredField("RUNNERS");
+      runners.setAccessible(true);
+      ((Map<QInstance, HealthRunner>) runners.get(null)).put(qInstance, runner);
+      return (runner);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fresh central results avoid duplicate local probes; stale/unknown fall back.
+    *******************************************************************************/
+   @Test
+   void breakerUsesCentralResults() throws Exception
+   {
+      AssistantProvider row = row();
+      var active = new ProviderService.Active(row, type.getType(), "model", provider);
+      ProviderHealth breaker = new ProviderHealth(config.getRuntime());
+      String name = "assistant.provider." + row.getId();
+      for(HealthStatus status : List.of(HealthStatus.UP, HealthStatus.DEGRADED, HealthStatus.DOWN))
+      {
+         try(var central = cached(name, new HealthCheckOutcome().withStatus(status), Instant.now().minusSeconds(1)))
+         {
+            if(status == HealthStatus.DOWN)
+            {
+               assertThatThrownBy(() -> breaker.beforeCall(active)).isInstanceOf(ModelProviderException.class);
+            }
+            else
+            {
+               breaker.beforeCall(active);
+            }
+            assertThat(probes).isZero();
+         }
+      }
+      try(var central = cached(name, HealthCheckOutcome.down("Down"), Instant.now().minusSeconds(60)))
+      {
+         breaker.beforeCall(active);
+         assertThat(probes).isEqualTo(1);
+      }
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("healthCheckedAt", null));
+      try(var central = cached(name, HealthCheckOutcome.unknown("Unsupported"), Instant.now()))
+      {
+         breaker.beforeCall(active);
+         assertThat(probes).isEqualTo(2);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The replacement card is generic and links to filtered central health.
+    *******************************************************************************/
+   @Test
+   void centralHealthCard() throws Exception
+   {
+      AssistantProvider row = row();
+      assertThat(qInstance.getWidget(AssistantHealthWidget.NAME).getType()).isEqualTo(WidgetType.STATISTICS.getType());
+      var renderer = new AssistantHealthWidget();
+      StatisticsData data = (StatisticsData) renderer.render(new RenderWidgetInput()).getWidgetData();
+      assertThat(data.getCount()).isEqualTo("UNKNOWN");
+      assertThat(data.getCountURL()).startsWith("/app/").contains("health_check", "assistant");
+      try(var central = cached("assistant.provider." + row.getId(), HealthCheckOutcome.up("Reachable"), Instant.now()))
+      {
+         var cache = HealthRunner.class.getDeclaredMethod("cacheOutcome", String.class, HealthCheckOutcome.class);
+         cache.setAccessible(true);
+         cache.invoke(central, AssistantHealthChecks.CLIENT_ERRORS, HealthCheckOutcome.up("0 client errors reported in the last hour"));
+         data = (StatisticsData) renderer.render(new RenderWidgetInput()).getWidgetData();
+         assertThat(data.getCount()).isEqualTo("UP");
+
+         //////////////////////////////////////////////////////////////////
+         // many client errors in the last hour show on the card as well //
+         //////////////////////////////////////////////////////////////////
+         cache.invoke(central, AssistantHealthChecks.CLIENT_ERRORS, HealthCheckOutcome.degraded("25 client errors reported in the last hour"));
+         data = (StatisticsData) renderer.render(new RenderWidgetInput()).getWidgetData();
+         assertThat(data.getCount()).isEqualTo("DEGRADED");
+      }
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/jobs/JobTrackingRunTrackerTest.java

@@ -0,0 +1,513 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.jobs;
+
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.api.AssistantApi;
+import com.kingsrook.qbits.assistant.engine.RunningTurn;
+import com.kingsrook.qbits.assistant.engine.TurnEvent;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qbits.assistant.model.AssistantTurn;
+import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
+import com.kingsrook.qbits.assistant.spi.ModelEventListener;
+import com.kingsrook.qbits.assistant.spi.ModelPart;
+import com.kingsrook.qbits.assistant.spi.ModelProviderException;
+import com.kingsrook.qbits.assistant.spi.ModelRequest;
+import com.kingsrook.qbits.assistant.spi.ModelResponse;
+import com.kingsrook.qbits.assistant.spi.StreamHandle;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qbits.assistant.uiapi.CoreUiApiGateway;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
+import com.kingsrook.qqq.backend.core.jobs.DurableJobsMetaData;
+import com.kingsrook.qqq.backend.core.jobs.DurableJobsMetaDataProvider;
+import com.kingsrook.qqq.backend.core.jobs.model.JobStatus;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobPhase;
+import com.kingsrook.qqq.backend.core.jobs.progress.JobProgressSnapshot;
+import com.kingsrook.qqq.backend.core.jobs.runtime.DurableJobsRuntime;
+import com.kingsrook.qqq.backend.core.jobs.store.DurableJobStore;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.EDITOR;
+import static com.kingsrook.qqq.backend.core.jobs.DurableJobsTables.TABLE_JOB;
+import static com.kingsrook.qqq.backend.core.jobs.DurableJobsTables.TABLE_LOG;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Real Jobs tray rows, progress and cancellation for assistant answers.
+ *******************************************************************************/
+class JobTrackingRunTrackerTest extends BaseAssistantTest
+{
+   private static final String PROMPT = "PRIVATE-PROMPT-JOBS-CANARY";
+   private static final String ANSWER = "PRIVATE-ANSWER-JOBS-CANARY";
+
+   private final CountDownLatch toolEntered = new CountDownLatch(1);
+   private final CountDownLatch toolRelease = new CountDownLatch(1);
+   private DurableJobStore jobs;
+   private AssistantApi api;
+   private ControlledProvider controlled;
+
+
+
+   /*******************************************************************************
+    ** Add jobs only to this fixture, leaving runtime startup explicit per test.
+    *******************************************************************************/
+   @BeforeEach
+   void initializeJobs() throws Exception
+   {
+      new DurableJobsMetaDataProvider().defineAll(qInstance, AssistantTestFixture.BACKEND, null);
+      DurableJobsMetaData settings = new DurableJobsMetaData().withWorkerThreads(1).withLeaseTtlSeconds(3)
+         .withReaperIntervalSeconds(1).withPollIntervalMillis(50).withPodId("test-pod");
+      qInstance.withSupplementalMetaData(settings);
+      new QInstanceEnricher(qInstance).enrich();
+      jobs = new DurableJobStore(settings);
+
+      config.getRuntime().close();
+      config.withUiApiGateway(new CoreUiApiGateway(config::getUiApiBaseUrl, qInstance)
+      {
+         /*******************************************************************************
+          ** Hold the real count call so the tool phase can be inspected in storage.
+          *******************************************************************************/
+         @Override
+         public UiApiResult invoke(UiApiCall call) throws QException
+         {
+            if("/table/{tableName}/count".equals(call.getPathTemplate()))
+            {
+               toolEntered.countDown();
+               try
+               {
+                  assertThat(toolRelease.await(15, TimeUnit.SECONDS)).as("tool released").isTrue();
+               }
+               catch(InterruptedException e)
+               {
+                  Thread.currentThread().interrupt();
+                  throw (new QException("Test tool interrupted", e));
+               }
+            }
+            return (super.invoke(call));
+         }
+      });
+      config.setRuntime(new AssistantRuntime(qInstance, config));
+      var session = AssistantTestFixture.ownerSession(EDITOR, config);
+      QContext.init(qInstance, session);
+      api = new AssistantApi(config.getRuntime(), session);
+      assertThat(config.getRunTracker()).isExactlyInstanceOf(JobTrackingRunTracker.class);
+   }
+
+
+
+   /*******************************************************************************
+    ** Release every gate before draining both runtimes, even after an assertion.
+    *******************************************************************************/
+   @AfterEach
+   void stopJobs()
+   {
+      toolRelease.countDown();
+      if(controlled != null)
+      {
+         controlled.handles.forEach(handle -> handle.release.countDown());
+      }
+      config.getRuntime().close();
+      DurableJobsRuntime.stop(qInstance, Duration.ofSeconds(5));
+   }
+
+
+
+   /*******************************************************************************
+    ** A single linked owner job progresses through thinking, tool and writing.
+    *******************************************************************************/
+   @Test
+   void answerTracksProgressAndCompletes() throws Exception
+   {
+      assertThat(DurableJobsRuntime.start(qInstance)).isNotNull();
+      ControlledHandle thinking = new ControlledHandle(new ModelResponse().withParts(List.of(
+         ModelPart.toolUse("count-1", "count_records", input("table", "blog_post", "filter", filter("AND")))))
+         .withStopReason(ModelResponse.STOP_TOOL_USE), null, false);
+      ControlledHandle writing = answer(false);
+      useProvider(thinking, writing);
+      Integer conversationId = (Integer) api.createConversation(Map.of()).get("id");
+      RunningTurn run = api.startTurn(conversationId, Map.of("text", PROMPT));
+      entered(thinking);
+      QRecord job = onlyJob();
+      Integer jobId = job.getValueInteger("id");
+      assertThat(job.getValueString("processName")).isEqualTo("assistant.turn");
+      assertThat(job.getValueString("label")).isEqualTo("Assistant answer");
+      assertThat(job.getValueString("userId")).isEqualTo(EDITOR);
+      assertThat(job.getValueString("tableName")).isEqualTo(AssistantConversation.TABLE_NAME);
+      assertThat(JsonUtils.toObject(job.getValueString("sourceJson"), Map.class)).containsEntry("recordId", conversationId);
+      assertPhase(jobId, "thinking", null);
+
+      thinking.release.countDown();
+      assertThat(toolEntered.await(15, TimeUnit.SECONDS)).as("tool entered").isTrue();
+      assertPhase(jobId, "tool", "Counting records");
+      toolRelease.countDown();
+      entered(writing);
+      assertPhase(jobId, "writing", null);
+      writing.release.countDown();
+
+      assertThat(completed(run).getStatus()).isEqualTo("COMPLETE");
+      job = onlyJob();
+      assertThat(job.getValueInteger("id")).isEqualTo(jobId);
+      assertThat(job.getValueString("status")).isEqualTo("COMPLETED");
+      assertThat(job.getValueString("progressMessage")).isEqualTo("Answered.");
+      assertThat(job.getValueString("statusReason")).isEqualTo("Answered.");
+      assertThat(run.getLog().after(0, 1)).filteredOn(event -> TurnEvent.TOOL_RESULT.equals(event.getType()))
+         .singleElement().satisfies(event -> assertThat(event.getData()).containsEntry("outcome", "OK"));
+      assertPrivateTextAbsent();
+   }
+
+
+
+   /*******************************************************************************
+    ** The runtime's periodic tick observes the same row change as Jobs tray Stop.
+    *******************************************************************************/
+   @Test
+   void jobsTrayStopCancelsAnswer() throws Exception
+   {
+      DurableJobsRuntime.start(qInstance);
+      ControlledHandle writing = answer(false);
+      useProvider(writing);
+      RunningTurn run = startTurn();
+      entered(writing);
+      Integer jobId = onlyJob().getValueInteger("id");
+      jobs.updateJob(jobId, Map.of("status", "CANCELLING"));
+
+      assertThat(completed(run).getStatus()).isEqualTo("CANCELLED");
+      assertThat(writing.isCancelled()).isTrue();
+      assertThat(onlyJob().getValueString("status")).isEqualTo("CANCELLED");
+      assertThat(onlyJob().getValueString("progressMessage")).isEqualTo("Stopped.");
+      assertPrivateTextAbsent();
+   }
+
+
+
+   /*******************************************************************************
+    ** Even partial output and provider errors containing private text stay out.
+    *******************************************************************************/
+   @Test
+   void failedAnswerHasOnlyFixedJobMessages() throws Exception
+   {
+      DurableJobsRuntime.start(qInstance);
+      ControlledHandle writing = answer(true);
+      useProvider(writing);
+      RunningTurn run = startTurn();
+      entered(writing);
+      writing.release.countDown();
+
+      assertThat(completed(run).getStatus()).isEqualTo("ERROR");
+      QRecord job = onlyJob();
+      assertThat(job.getValueString("status")).isEqualTo("FAILED");
+      assertThat(job.getValueString("progressMessage")).isEqualTo("The answer did not finish.");
+      assertThat(job.getValueString("statusReason")).isEqualTo("The answer did not finish.");
+      assertThat(jobs.query(TABLE_LOG, new QQueryFilter())).isNotEmpty();
+      assertPrivateTextAbsent();
+   }
+
+
+
+   /*******************************************************************************
+    ** Hosts without jobs keep ordinary answers and safe null tracker callbacks.
+    *******************************************************************************/
+   @Test
+   void absentRuntimeDoesNotTrackOrDisruptAnswer() throws Exception
+   {
+      assertThat(DurableJobsRuntime.get(qInstance)).isNull();
+      AssistantRunTracker tracker = config.getRunTracker();
+      assertThat(tracker.start(EDITOR, 1, 1, "ignored")).isNull();
+      tracker.progress(null, "thinking", PROMPT);
+      tracker.onCancel(null, () ->
+      {
+         throw (new AssertionError("No job to cancel"));
+      });
+      tracker.finish(null, "COMPLETE", ANSWER);
+      provider.thenText(ANSWER);
+      RunningTurn run = startTurn();
+      assertThat(completed(run).getStatus()).isEqualTo("COMPLETE");
+      assertThat(jobs.query(TABLE_JOB, new QQueryFilter())).isEmpty();
+      assertThat(jobs.query(TABLE_LOG, new QQueryFilter())).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Core mutates phase objects; every new job must start with fresh phases.
+    *******************************************************************************/
+   @Test
+   void jobsHaveIndependentPhaseState() throws Exception
+   {
+      DurableJobsRuntime.start(qInstance);
+      AssistantRunTracker tracker = config.getRunTracker();
+      Integer conversationId = (Integer) api.createConversation(Map.of()).get("id");
+      String first = tracker.start(EDITOR, 1, conversationId, PROMPT);
+      tracker.progress(first, "writing", ANSWER);
+      String second = tracker.start(EDITOR, 2, conversationId, PROMPT);
+      QRecord secondJob = jobs.getJob(Integer.valueOf(second));
+      JobProgressSnapshot snapshot = JsonUtils.toObject(secondJob.getValueString("progressJson"), JobProgressSnapshot.class);
+      assertThat(snapshot.getPhases()).extracting(JobPhase::getStatus)
+         .containsExactly(JobPhase.Status.PENDING, JobPhase.Status.PENDING, JobPhase.Status.PENDING);
+      tracker.progress(second, "thinking", PROMPT);
+      tracker.progress(first, "unknown", PROMPT);
+      tracker.progress("missing", "tool", PROMPT);
+      assertPhase(Integer.valueOf(first), "writing", null);
+      assertPhase(Integer.valueOf(second), "thinking", null);
+      tracker.finish(first, "COMPLETE", ANSWER);
+      tracker.finish(second, "COMPLETE", ANSWER);
+      assertPrivateTextAbsent();
+   }
+
+
+
+   /*******************************************************************************
+    ** Terminal mapping is fixed, including unknown and absent assistant statuses.
+    *******************************************************************************/
+   @ParameterizedTest
+   @CsvSource({ "COMPLETE,COMPLETED,Answered.", "CANCELLED,CANCELLED,Stopped.",
+      "ERROR,FAILED,The answer did not finish.", "unknown,FAILED,The answer did not finish.",
+      ",FAILED,The answer did not finish." })
+   void terminalMapping(String status, JobStatus expected, String message)
+   {
+      assertThat(JobTrackingRunTracker.jobStatus(status)).isEqualTo(expected);
+      assertThat(JobTrackingRunTracker.finishMessage(status)).isEqualTo(message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Use independently controlled model calls without changing the tracker.
+    *******************************************************************************/
+   private void useProvider(ControlledHandle... handles)
+   {
+      controlled = new ControlledProvider(List.of(handles));
+      config.withModelProvider(controlled);
+   }
+
+
+
+   /*******************************************************************************
+    ** Partial text is emitted before this response is released or fails.
+    *******************************************************************************/
+   private ControlledHandle answer(boolean fail)
+   {
+      return (new ControlledHandle(new ModelResponse().withParts(List.of(ModelPart.text(ANSWER)))
+         .withStopReason(ModelResponse.STOP_END_TURN), ANSWER, fail));
+   }
+
+
+
+   /*******************************************************************************
+    ** Create a conversation and answer as the fixture's editor.
+    *******************************************************************************/
+   private RunningTurn startTurn() throws Exception
+   {
+      Integer conversationId = (Integer) api.createConversation(Map.of()).get("id");
+      return (api.startTurn(conversationId, Map.of("text", PROMPT)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Wait until a model call has emitted its optional text and is blocked.
+    *******************************************************************************/
+   private void entered(ControlledHandle handle) throws InterruptedException
+   {
+      assertThat(handle.entered.await(15, TimeUnit.SECONDS)).as("model call entered").isTrue();
+   }
+
+
+
+   /*******************************************************************************
+    ** Check persisted progress while the answer is still running.
+    *******************************************************************************/
+   private void assertPhase(Integer jobId, String phase, String message) throws QException
+   {
+      QRecord job = jobs.getJob(jobId);
+      assertThat(job.getValueString("status")).isEqualTo("RUNNING");
+      assertThat(job.getValueString("currentPhase")).isEqualTo(phase);
+      assertThat(job.getValueString("progressMessage")).isEqualTo(message);
+   }
+
+
+
+   /*******************************************************************************
+    ** Assert uniqueness across all jobs, not just those matching this turn.
+    *******************************************************************************/
+   private QRecord onlyJob() throws QException
+   {
+      List<QRecord> rows = jobs.query(TABLE_JOB, new QQueryFilter());
+      assertThat(rows).hasSize(1);
+      return (rows.getFirst());
+   }
+
+
+
+   /*******************************************************************************
+    ** Inspect every persisted job and log field for both private canaries.
+    *******************************************************************************/
+   private void assertPrivateTextAbsent() throws QException
+   {
+      for(String table : List.of(TABLE_JOB, TABLE_LOG))
+      {
+         assertThat(jobs.query(table, new QQueryFilter())).allSatisfy(row ->
+            assertThat(JsonUtils.toJson(row)).doesNotContain(PROMPT, ANSWER));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Wait for finalization and cleanup before reading the assistant turn.
+    *******************************************************************************/
+   private AssistantTurn completed(RunningTurn run) throws Exception
+   {
+      await(() -> config.getRuntime().getRunningTurn(run.getTurnId()) == null);
+      assertThat(run.getLog().isClosed()).isTrue();
+      return (config.getRuntime().getStore().getTurn(EDITOR, run.getTurnId()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Bound polling by observable completion rather than fixed timing guesses.
+    *******************************************************************************/
+   private static void await(Callable<Boolean> condition) throws Exception
+   {
+      long deadline = System.nanoTime() + Duration.ofSeconds(15).toNanos();
+      while(!condition.call() && System.nanoTime() < deadline)
+      {
+         Thread.sleep(10);
+      }
+      assertThat(condition.call()).as("condition reached within fifteen seconds").isTrue();
+   }
+
+
+
+   /*******************************************************************************
+    ** Script model responses with separate gates for thinking and writing.
+    *******************************************************************************/
+   private static class ControlledProvider extends ScriptedModelProvider
+   {
+      private final List<ControlledHandle> handles;
+      private final AtomicInteger next = new AtomicInteger();
+
+      /*******************************************************************************
+       ** Retain the scripted handles for teardown and assertions.
+       *******************************************************************************/
+      ControlledProvider(List<ControlledHandle> handles)
+      {
+         this.handles = handles;
+      }
+
+      /*******************************************************************************
+       ** Publish partial text, then let the engine wait on the selected handle.
+       *******************************************************************************/
+      @Override
+      public StreamHandle stream(ModelRequest request, ModelEventListener listener)
+      {
+         ControlledHandle handle = handles.get(next.getAndIncrement());
+         if(handle.text != null)
+         {
+            listener.onTextDelta(handle.text);
+         }
+         return (handle);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A bounded, cancellable model call that can fail after streaming text.
+    *******************************************************************************/
+   private static class ControlledHandle implements StreamHandle
+   {
+      private final ModelResponse response;
+      private final String text;
+      private final boolean fail;
+      private final CountDownLatch entered = new CountDownLatch(1);
+      private final CountDownLatch release = new CountDownLatch(1);
+      private volatile boolean cancelled;
+
+      /*******************************************************************************
+       ** Set the eventual response and optional partial-output failure.
+       *******************************************************************************/
+      ControlledHandle(ModelResponse response, String text, boolean fail)
+      {
+         this.response = response;
+         this.text = text;
+         this.fail = fail;
+      }
+
+      /*******************************************************************************
+       ** Stop unblocks the engine without test intervention.
+       *******************************************************************************/
+      @Override
+      public void cancel()
+      {
+         cancelled = true;
+         release.countDown();
+      }
+
+      /*******************************************************************************
+       ** Report whether the real cancellation callback reached the provider.
+       *******************************************************************************/
+      @Override
+      public boolean isCancelled()
+      {
+         return (cancelled);
+      }
+
+      /*******************************************************************************
+       ** Wait until the test or the Jobs tray releases this model call.
+       *******************************************************************************/
+      @Override
+      public ModelResponse await(Duration timeout) throws ModelProviderException, InterruptedException
+      {
+         entered.countDown();
+         if(!release.await(timeout.toMillis(), TimeUnit.MILLISECONDS))
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "Test provider timed out"));
+         }
+         if(cancelled)
+         {
+            return (new ModelResponse().withStopReason(ModelResponse.STOP_CANCELLED));
+         }
+         if(fail)
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.OTHER, null, PROMPT + " " + ANSWER));
+         }
+         return (response);
+      }
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/persistence/SchemaForeignKeyPostgresTest.java

@@ -94,6 +94,9 @@ class SchemaForeignKeyPostgresTest extends BaseAssistantTest
       rule("assistant_tool_call", "pending_action_id", "assistant_pending_action", "n");
       rule("assistant_feedback", "conversation_id", "assistant_conversation", "c");
       rule("assistant_feedback", "message_id", "assistant_message", "c");
+      rule("assistant_client_error", "conversation_id", "assistant_conversation", "n");
+      rule("assistant_client_error", "message_id", "assistant_message", "n");
+      rule("assistant_client_error", "turn_id", "assistant_turn", "n");
       rule("assistant_pending_action", "conversation_id", "assistant_conversation", "c");
       rule("assistant_pending_action", "turn_id", "assistant_turn", "n");
       rule("assistant_data_audit", "conversation_id", "assistant_conversation", "n");

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/settings/AssistantSettingsTest.java

@@ -561,7 +561,7 @@ class AssistantSettingsTest extends BaseAssistantTest
       assertThat(data.getMonthCostMicros()).isZero();
 
       String html = render();
-      assertThat(html).contains("Answering").contains("Fake").contains("In use").contains("Last test passed").contains("…wxyz")
+      assertThat(html).contains("Configuration and usage").contains("Fake").contains("In use").doesNotContain("Answering", "Last test passed").contains("…wxyz")
          .contains("of $50.00 cap").contains("list_tables").contains("Model call #").contains("25% of 4 requests").doesNotContain(KEY).doesNotContain("<table");
    }