Compare commits

..
Author SHA1 Message Date
james.maes 57673131ff Merge remote-tracking branch 'origin/develop' into feature/answer-presentation 2026-10-13 19:54:07 -05:00
james.maesandClaude Opus 5.5 c27994c432 feat: answer presentation: present_results tool, presentation parts and the default presentation skill
Owner priority: answers must be easy to read and act on.

- present_results tool: summary, metrics, table (computed percent
  columns, per-row filtered-list links), bar/pie/line chart from the
  table, filter and record links, follow-up actions. Every table and
  field is checked against what the user can see through the UI API;
  hrefs (/app/<table>?filter=<QQQ filter JSON>, records) and number
  formats are built server-side from metadata; the model never writes
  URLs.
- presentation message part (persisted, SSE event) with a server-built
  multi-line GFM markdown fallback for clients without blocks.
- Static system-prompt Presentation section (built-in default skill):
  answer first, no restating, tables for 3+ items, charts for shares and
  trends, filterLinks whenever a list is offered.
- Golden eval of the owner's example: summary, 2-row table with 80.2% /
  19.8%, a 2-point pie and two mimeType filter links; negative cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:54:05 -05:00
james.maes 14bd4e377c Merge pull request #31 from QRun-IO/fix/core-alerts-rc5
refactor: budget alerts through core QAlerts on QQQ RC.5 (#28)
2026-10-13 19:34:21 -05:00
james.maesandClaude Opus 5.5 cb7d5d4d5a refactor: budget alerts through core QAlerts (QQQ 4.1.0-RC.5), alert types declared
Closes #28 (first half; the DataClassification and loopback gateway
stand-ins go with RC.6).

- Default QQQ is 4.1.0-RC.5.
- BudgetAlerts calls QAlerts.raise(AlertSignal...) directly instead of
  binding by reflection.
- The producer declares assistant.budget.warning, .downgraded and .stopped
  as QAlertTypeMetaData (assistant.admin, subject assistant_budget).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:30:01 -05:00
james.maes 51e0692f46 Merge pull request #27 from QRun-IO/feature/p5-routing-fallback
feat: simple/complex model routing and fallback chains (P-e)
2026-10-13 19:13:26 -05:00
james.maes 6f80dcaff4 Merge remote-tracking branch 'origin/develop' into feature/p5-routing-fallback 2026-10-13 19:09:28 -05:00
james.maesandClaude Opus 5.5 55804d9059 feat: simple/complex model routing and fallback chains across providers (P-e)
- Profiles gain routing (OFF, HEURISTIC, CLASSIFIER) with simple, complex
  and classifier models. The heuristic is a small documented rule set; the
  classifier is one 4-token call on the question text only (ROUTE usage,
  3 s timeout, heuristic on any failure, skipped for PHI on a
  non-eligible provider).
- assistant_model.fallback_model_id chains (cycle-checked, max 4 hops)
  across providers; retryable failures before any output fall back.
  Candidates that are disabled, down, over a STOP budget or fail the PHI
  rule are skipped; without a chain, LOCAL falls back only to LOCAL.
- Circuit breaker (provider health UP/DOWN, down_until) and a cached
  health probe for stale LOCAL providers.
- A mid-turn provider switch rebuilds history from neutral parts and drops
  adapter state (Anthropic thinking signatures).
- Turn route decision, source and fallback count; home shows health and
  fallbacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:09:26 -05:00
james.maes 73159e536e Merge pull request #26 from QRun-IO/feature/p4-budgets
feat: budgets with warn, downgrade, stop and admin overrides (P-d)
2026-10-13 18:46:19 -05:00
james.maes c05c606745 Merge remote-tracking branch 'origin/develop' into feature/p4-budgets 2026-10-13 18:43:47 -05:00
james.maesandClaude Opus 5.5 51fcfa6a33 feat: data-driven budgets with warn, downgrade, stop and admin overrides (P-d)
- assistant_budget (scope INSTANCE/USER/ROLE/PROVIDER/MODEL, DAY/MONTH,
  cost or tokens, warn %, DOWNGRADE or STOP at 100%, hard ceiling),
  assistant_budget_override (reason, grantor, expiry) and
  assistant_budget_event (deduplicated WARNED/DOWNGRADED/STOPPED).
- BudgetService checks before every model call; strictest outcome wins.
  Warnings raise a panel notice and an optional Alerts signal; downgrade
  prefers a cheaper LOCAL model and never crosses the PHI rule.
- assistant_budget_usage: an owner-lock-free accounting projection of
  usage counters so shared-scope sums run under the caller's own session.
- App home: budgets card with state and overrides, 30-day cost and token
  bars with user, model, provider and tool breakdowns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 18:43:45 -05:00
63 changed files with 7595 additions and 166 deletions

No files matched your search

+31 -4
View File
@@ -1,6 +1,33 @@
# Changelog
## Unreleased — P-c
## Unreleased - Answer presentation
- Add `present_results`: server-formatted metrics, tables, computed percentages,
charts and validated in-app filter/record/action links.
- Persist and stream presentation parts with a multi-line GFM fallback and
record-data retention; add a static default presentation skill.
- Add the owner's 81 JPEG / 20 PNG golden example, shared live eval coverage,
permission/formatting/input tests and strict-schema budget coverage.
## Unreleased - P-e
- Add per-profile heuristic/classifier model routing, with budgeted ROUTE usage.
- Add bounded cross-provider fallback chains, PHI-safe selection, neutral history
rebuilding, and a 60-second connection circuit breaker with cached local probes.
- Show provider health, fallback totals and turn routing; include additive schema
migration and admin record-screen controls.
## Unreleased - P-d
- Require QQQ RC.5 and send budget alerts directly through core QAlerts, with admin-visible alert type metadata and core sink tests.
- Admin budgets, expiring overrides, deduplicated warning/downgrade/stop events.
- Per-dispatch limits preserving PHI routing and existing caps.
- Home budget cards, 30-day cost/token bars and provider breakdowns.
## Unreleased — P-c
- Add configurable table/field classification, persistent turn/conversation maxima,
and PHI routing checks before every model call, with a safe blocked-turn reason.
@@ -30,8 +57,8 @@ Phase 0/1 of the QQQ Assistant: read-only Q&A and navigation. Requires QQQ 4.1 w
- Tools: `list_tables`, `describe_table`, `query_records`, `count_records`, `get_record`, `search_records`, `render_widget`, `get_record_history`, and new **`create_record`, `update_record`, `delete_record`**.
### Added (0.1.0-RC.6)
- **Profiles** (`assistant_profile`, nav "Profiles"): display name, icon, personality (≤ 2,000 chars, appended to the per-user system block), greeting, starter prompts; one default, optional app / permission scope. `/assistant/config` returns `profile`.
- **Skills** (`assistant_skill`, nav "Skills", linked through `assistant_profile_skill`, a child list on profiles): admin-written Markdown instructions read only when relevant (keywords / apps), ≤ 6,000 chars each and 12,000 per question. Instructions only: they never grant tools, endpoints or access (ArchUnit-enforced).
- **Profiles** (`assistant_profile`, nav "Profiles"): display name, icon, personality (≤ 2,000 chars, appended to the per-user system block), greeting, starter prompts; one default, optional app / permission scope. `/assistant/config` returns `profile`.
- **Skills** (`assistant_skill`, nav "Skills", linked through `assistant_profile_skill`, a child list on profiles): admin-written Markdown instructions read only when relevant (keywords / apps), ≤ 6,000 chars each and 12,000 per question. Instructions only: they never grant tools, endpoints or access (ArchUnit-enforced).
- **Explain access**: `GET /qqq/v1/assistant/access?table=&prompt=` and the `assistantAccess` home widget: per table and tool, allowed or the reason.
- **Recent denials** card on the Assistant home.
- `AssistantStore` refuses any table that is not `assistant_*`; ArchUnit forbids the tool / UI API / access packages from using QQQ actions, the AI kernel or the profile and skill classes.
@@ -54,7 +81,7 @@ Phase 0/1 of the QQQ Assistant: read-only Q&A and navigation. Requires QQQ 4.1 w
### Added (0.1.0-RC.4)
- **Provider settings** (`assistant_provider`): type (registered `AssistantProviderType`s; `AnthropicProviderType` in the Anthropic module), model, enabled, default, base URL override, monthly cost cap, max output tokens per request, max concurrent requests, and the last connection test. The default enabled row is the active provider; with no rows the provider registered in code is used, so existing hosts keep working.
- **Key handling:** the key is never stored. A row names the environment variable that holds it (`apiKeyEnvName`, allow-listed); `apiKeyStatus` is computed on read (`Set • …abcd`) and dropped on save. A pasted key, an unrelated variable or an unlisted base URL is refused. New config: `withProviderType`, `allowedApiKeyEnvNames`, `allowedBaseUrls`, `withSecretResolver`, `providerTestTimeoutSeconds`.
- **Key handling:** the key is never stored. A row names the environment variable that holds it (`apiKeyEnvName`, allow-listed); `apiKeyStatus` is computed on read (`Set • …abcd`) and dropped on save. A pasted key, an unrelated variable or an unlisted base URL is refused. New config: `withProviderType`, `allowedApiKeyEnvNames`, `allowedBaseUrls`, `withSecretResolver`, `providerTestTimeoutSeconds`.
- **Test connection**, a record action on providers (standard streamed ETL Review and Result lines): one short live call, result saved on the row and recorded as a `TEST` usage row.
- **Per-call usage** (`assistant_usage`): one row per model call (turn steps and tests) with provider, model, status, error kind, HTTP status, latency, tokens and cost. Child list on conversations and providers.
- **Model prices** (`assistant_model_price`), editable, used for every call's cost; the reference DDL seeds the Anthropic list prices of 2026-10. The built-in `pricing` defaults now cover every current Claude model.
+2
View File
@@ -69,6 +69,8 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def
- **Links:** only `/app` or `/app/...` hrefs are emitted (`TurnEngine.isAppHref`).
- **PHI routing:** PHI only to `hipaaEligible` providers; checked before every model call. Conversation classification never decreases.
- **HIPAA storage:** Conversation storage encrypted in HIPAA mode, only in `AssistantStore`; encryption keys are environment references and never logged or stored in tables.
- **Fallback:** fallback never crosses the PHI rule; never local-to-cloud by default. No retry after partial output from that call; provider switches rebuild neutral history without providerState.
- **Budgets:** budgets are checked before every model call; downgrade never crosses the PHI rule.
- **Kill switch:** read fresh before every model call and every tool step.
## Tests
+64 -2
View File
@@ -9,7 +9,7 @@ Design spec: `QRun-IO/qqq` `docs/specs/2026-10-11-qqq-assistant-qbit-design.md`.
| `com.kingsrook.qbits:qbit-assistant` | The QBit: conversation tables, turn engine, `/qqq/v1/assistant` routes (JSON + Server-Sent Events), budgets, kill switch, the provider SPI, and a scripted provider for tests and demos. |
| `com.kingsrook.qbits:qbit-assistant-provider-anthropic` | The Claude provider (`com.anthropic:anthropic-java`). Add it to use Claude. The core QBit has no LLM SDK dependency. |
The data access itself is done by the AI tool kernel in QQQ core (`com.kingsrook.qqq.backend.core.actions.ai.AiToolKernel`). The kernel checks table permissions and exposure rules, applies record security, and caps and redacts what the model sees. Every tool result reaches the model wrapped as untrusted data.
Data tools call the QQQ UI API through `UiApiToolbox` and `UiApiGateway` with the signed-in user's session. The UI enforces permissions and record security; the assistant further narrows access and caps and redacts returned values. Data tool results reach the model wrapped as untrusted data. Only record-free navigation and help use the core AI tool kernel.
## Requirements
@@ -203,6 +203,7 @@ Base path `/qqq/v1/assistant`. Every request is authenticated through the app's
The stream sends these events, each with a `seq`:
- `message.start`, `text.delta`
- `tool.start`, `tool.result` (with `render` and `recordRefs`)
- `presentation` (`messageId`, `part`: summary, blocks and server-built GFM markdown)
- `ui.navigate`, `notice`
- `usage`, `message.end`
- `error`
@@ -385,7 +386,7 @@ Apache 2.0.
## Data classification and HIPAA mode
On QQQ 4.1.0-RC.4, classify application tables/fields in the host configuration:
On QQQ 4.1.0-RC.5, classify application tables/fields in the host configuration:
```java
import com.kingsrook.qbits.assistant.classification.DataClassification;
@@ -439,3 +440,64 @@ Both modes audit each PII/PHI model call and every routing block in
PHI conversations use `hipaaRetentionDays` even with HIPAA mode off. A null value
uses normal retention; tool-result/replay cleanup remains independently configured.
See [UPGRADE.md](UPGRADE.md) for schema changes, plaintext migration, and key handling.
## Data-driven budgets
Admins configure **Assistant > Budgets** for UTC DAY or MONTH allowances in USD or
TOKENS, scoped to INSTANCE, USER, ROLE (a permission), PROVIDER or MODEL. Provider
and model scope values are registry record ids. Warnings default to 80%; at 100%
the default action is DOWNGRADE, and the hard ceiling defaults to 120% (always
STOP). Multiple budgets apply together and STOP wins.
An explicit downgrade model is optional. Otherwise the cheapest enabled usable
model is selected, preferring LOCAL providers. PHI never goes to a provider that
is not HIPAA-eligible. No eligible target means STOP. Administrators can grant
extra allowance or temporary unlimited use from the budget's Overrides child list,
with a required reason and expiry; the session sets the grantor.
Budget state changes appear once per scope/period/state in the Events child list,
as panel notices and, through core QAlerts, bell alerts when an alerts sink is installed (logging otherwise).
The Assistant home shows budgets, overrides, daily cost and tokens, and breakdowns
by user, model, provider and tool. Existing `AssistantBudgets`/policy limits and
provider monthly caps continue to apply independently of these rows and overrides.
See [UPGRADE.md](UPGRADE.md) for the required additive schema.
### Profile routing and provider fallback
On an Assistant Profile, **Model routing** defaults to OFF. HEURISTIC selects the
complex slot for long questions (>600 characters), two mentioned app tables,
analysis/comparison/explanation/trend/planning requests, pending writes, more than
eight previous turns, or a previous tool limit; otherwise it selects the simple
slot. CLASSIFIER uses an optional registry model with just the question and a
fixed instruction, four output tokens and a three-second timeout. An invalid reply,
error, unavailable classifier or PHI-ineligible classifier uses HEURISTIC. ROUTE
usage counts toward budgets. Empty model slots keep the provider/policy default;
budgets may downgrade either choice. Without a profile, enabled code-level
`AssistantModelRouting.escalation*` supplies the complex default.
A model's **Fallback model** points to its next candidate, including another
provider. Chains are cycle-free and bounded to four hops. Only retryable failures
before any text or tool start from that call permit fallback. Disabled, down,
PHI-ineligible or budget-blocked candidates are skipped. Without explicit links,
only another LOCAL model in the same adapter family is eligible; local requests
never automatically move to cloud. Provider state/signatures are removed when
switching, preserving neutral tool history. Each attempted call is audited as usage.
NETWORK/TIMEOUT marks a provider DOWN for 60 seconds; success marks it UP. Local
health is probed when unknown/stale with a two-second timeout. Provider cards show
health and down-until, failures show fallback totals, and turn records show the
route and final destination. See [UPGRADE.md](UPGRADE.md) for the additive schema.
### Answer presentation
The always-available `present_results` tool turns retrieved results into metrics,
tables, charts and actionable in-app links. The server validates the user's UI API
visibility, formats field values, computes shares and builds all links. It emits
and stores a `presentation` part with a complete GFM Markdown fallback. Profile
skills can extend the built-in presentation instructions. See the
[part, stream and input contract](docs/answer-presentation.md).
The image-breakdown golden task also runs in the existing opt-in live provider suite:
`./mvnw verify -Dassistant.live=true -Dtest.excludedGroups= -Dgroups=live -Djacoco.skip=true -pl qbit-assistant-provider-anthropic -am`
(with `ANTHROPIC_API_KEY` set). Live calls are excluded from normal verification.
+59
View File
@@ -1,5 +1,64 @@
# Upgrading qbit-assistant
## P-e: profile routing and provider fallback
Apply the additive **P-e** ALTER TABLE statements at the end of
[docs/schema-postgres.sql](docs/schema-postgres.sql) before deploying. Existing
profiles default to OFF; optional model references and health columns start null.
No routing or fallback configuration is seeded. Existing turns retain unknown
routing/final-provider values; fallback_count defaults to zero.
Profiles now have a **Model routing** section. Select HEURISTIC or CLASSIFIER,
then optional simple/complex registry models and a classifier model. Empty route
slots retain provider/policy selection. CLASSIFIER with no usable classifier
uses HEURISTIC. OFF preserves prior provider selection. With no effective profile,
code-configured escalation supplies the complex model/effort when enabled.
Set a model's **Fallback model** to opt into a cross-provider chain. Saves reject
cycles, more than four edges, or disabled/missing targets. Provider health is
read-only; NETWORK/TIMEOUT opens a 60-second circuit, and successful calls close
it. Local endpoints with stale health get a bounded two-second probe. Automatic
fallback without an explicit link is confined to local models of the same adapter
family. PHI/HIPAA eligibility and budgets apply to every destination.
Classifier usage is recorded with purpose ROUTE and included in budgets. Reports
that filter usage purpose should include ROUTE. Turn screens show route decision,
source, fallback count and final provider; the home shows health and fallback totals.
All configuration remains assistant.admin-only, with AI exposure NONE.
## P-d: data-driven budgets
Requires QQQ 4.1.0-RC.5 for the core Alerts API and alert type metadata.
Create `assistant_budget`, `assistant_budget_override`, `assistant_budget_event`
and `assistant_budget_usage`
using the P-d DDL in [schema-postgres.sql](docs/schema-postgres.sql) before deploying.
The event unique key is required: it deduplicates each state per budget, scope and
UTC period across replicas. Do not remove it or truncate events during a period.
Budgets appear in Assistant navigation. Overrides and immutable events are child
lists. All four tables require `assistant.admin` and have AI exposure NONE.
Overrides record the actual session grantor and require a reason and expiry.
The standard host QQQ audit configuration applies to budget and override edits.
`assistant_budget_usage` is a private accounting projection populated from persisted
usage writes. It copies only counters, time, user, provider and model, with no owner
lock, so shared INSTANCE/PROVIDER/MODEL sums can run under the caller's own session.
The source usage table retains its owner locks. The projection is read-only, has no
nav entry and no AI exposure. **Pause usage writes during migration and run the DDL's
backfill before restarting**; otherwise shared totals would omit pre-upgrade usage.
Keep accounting rows through their budget periods even if detail rows are removed.
ROLE budgets give each permission holder a separate allowance, checked with their
current session. Override scopeValue, when set, narrows the allowance to one user id.
Existing policy/config budgets and provider monthly caps remain additional limits;
an override of a budget row does not waive those existing caps.
Provider and MODEL scope values are numeric provider/model registry row ids;
USER is a user id and ROLE is a permission name. INSTANCE has no scope value.
Periods reset at midnight UTC or the first of the month UTC. Token usage includes
input, output, cache-read and cache-write tokens; USD uses the recorded micro-dollar
cost, not today's model price. No default budget rows are seeded.
## P-c: classification, PHI routing and encrypted storage
Apply these additive schema changes before deploying P-c (adjust column mappings
+79
View File
@@ -376,6 +376,52 @@ AI exposure NONE is not enough, because the assistant acts as the user and an ad
to 2190 days in either mode; shorter tool-result/replay cleanup is unchanged.
The app home shows HIPAA mode and PHI retention.
### P-d as built
- ROLE allowances are per permission holder. Shared scopes sum a private immutable
accounting projection of usage counters; owner-locked source rows stay locked.
Migration backfills existing usage before writes resume. No sessions are elevated.
- Standard admin budget rows add daily/monthly cost or token allowances above the
existing config/policy budgets and provider monthly caps. Overrides carry a
reason, session grantor and expiry; they never waive the other limit layers.
- Start-turn checks use the existing 429 BUDGET path through PolicyService.
Every dispatch, including strict retries, evaluates current usage and overrides.
Events deduplicate WARNED, DOWNGRADED and STOPPED.
- QQQ RC.5 core QAlerts sends budget signals alongside panel notices, with three
admin-visible alert types and a default logging sink. Resolution awaits active
alert/recurrence tracking so overrides cannot clear shared alerts incorrectly.
- Explicit registry model selection leaves provider defaults untouched. Automatic
downgrades prefer LOCAL, then price. PHI eligibility and destination provider caps
still apply; a missing eligible target stops the turn.
- The home owns the budget cards and 30-day cost/token bars, with user, model,
provider and tool breakdowns, inherited colors and empty states.
### P-e as built
- Profiles select OFF (the migration default), HEURISTIC or CLASSIFIER and optional
simple/complex/classifier registry models. Selection uses ProfileService's existing
specificity. Without a profile, enabled escalation config supplies the complex
model and effort. Empty slots retain provider row/default/policy selection.
- The documented heuristic is a pure rule table: >600 characters, two distinct app
table names, complexity keywords, pending write context, >8 previous turns or a
previous tool limit. This branch has no conversation write-mode field.
- Classifier transport is non-streaming, bounded to four output tokens/three seconds,
and contains only the fixed instruction and user question. PHI guards apply before
dispatch. Errors/unexpected replies use the heuristic. ROUTE usage enters budget
accounting; normal dispatch budgets may override the selected route.
- FallbackChain follows at most four edges on retryable failures, only before any
text/tool start of that call. Candidates are checked for enablement, health,
PHI/HIPAA eligibility and budgets. Without explicit links only the same LOCAL
adapter family is considered. Provider switches rebuild neutral messages and
discard providerState; failed attempts remain visible in usage.
- ProviderHealth persists UP/DOWN, checked-at and a 60-second connection-failure
circuit. Local stale/unknown health uses a bounded two-second probe. Successful
completions close the breaker. Turn rows persist route/source, fallback count and
final provider/model; notices name only the unavailable and answering providers.
- Admin record screens, additive Postgres DDL and the flow-layout home expose these
settings and counters without granting AI access to assistant configuration.
### Phases (each a green PR)
| | Scope | Depends on |
|---|---|---|
@@ -400,3 +446,36 @@ AI exposure NONE is not enough, because the assistant acts as the user and an ad
- an "Explain access" action (for a user, and a table or a prompt, which operations the assistant could call and which it would be denied, with the reason);
- a recent-denials widget on the app home.
- **(d) Spec'd assistant endpoints:** move `/qqq/v1/assistant/...` (conversations, turns, events, feedback, cancel, config, actions) onto `AbstractEndpointSpec`s through the qbit spec extension point (`MiddlewareSpecExtension`, established prefix `/assistant`, QRun-IO/qqq#1128). The SSE turn and events streams stay raw routes until the spec layer can describe a stream.
## Answer presentation (owner priority 2026-10-13)
`present_results` is always offered and reads no records. It takes a direct answer,
metrics, a table, an optional table-derived chart, filtered list links, record links
and next-step actions. The server validates metadata and the user's UI API table
visibility once per turn, formats values using QQQ field types/display formats,
computes percentages and builds every `/app` href. Unknown or denied items are
dropped with an explanation. The assistant must still retrieve every fact through
data tools before presenting it.
The [presentation part and SSE contract](answer-presentation.md) defines `summary`,
`blocks` and the complete server-built GFM `markdown` fallback. Each successful
call emits `presentation` with `{messageId, part}` and persists that part; the tool
response tells the model not to repeat its tables or links. Audit rows contain
only accepted presentation data. Normal record-data retention removes these parts.
The built-in Presentation skill is compact static prompt text in the cached rules
block. It asks for a one-sentence answer first, metrics for single numbers, tables
for breakdowns/comparisons, charts for useful shares/trends and actual filter links
whenever offering a list. Administrator profile skills append after it as before
and extend it; there is no replacement boolean or extra permission surface.
The owner's one-line table was a frontend parsing defect: Next's `AssistantMarkdown`
used react-markdown without `remark-gfm`, so table lines became a paragraph.
The renderer fix belongs to `qqq-frontend-next`; this worktree preserves real
newlines and provides structured blocks plus a proper multi-line GFM fallback.
The frontend change is maintained separately and is not verified by this build.
The shared golden task asks "How many images do I have by type?": five UI count
calls establish 81 JPEG, 20 PNG and zero other configured types, then a presentation
with two rows, 80.2%/19.8% shares, two pie points and working MIME filter links.
The same task is included in the opt-in `live` Anthropic eval suite.
+56
View File
@@ -0,0 +1,56 @@
# Assistant answer presentation contract (v1)
Owner priority (2026-10-13): answers must ALWAYS be easy to read and act on. Bad example the owner saw, rendered on ONE line:
"| Type | Count | |---|---| | JPEG (image/jpeg) | 81 | | PNG (image/png) | 20 | | WebP, GIF, SVG | 0 |" followed by "You can open the lists filtered by JPEG or PNG" with NO links.
Root cause found in Next: `AssistantMarkdown` uses react-markdown WITHOUT `remark-gfm`, so GFM tables are not parsed and the table's lines collapse into one paragraph (soft line breaks). Backend streaming preserves `\n` (events are JSON).
## New message part: `presentation`
An assistant message's `parts` (persisted in `assistant_message.parts_json`, streamed as an SSE event, returned by the conversation API) may contain, besides `text`, `notice`, `tool`/`render` parts etc., ONE part per `present_results` call:
```json
{
"type": "presentation",
"summary": "101 media files are images: 81 JPEG and 20 PNG.",
"markdown": "<server-built GFM fallback of the whole presentation, for clients that do not render blocks (CLI/TUI)>",
"blocks": [
{ "kind": "metrics", "items": [ { "label": "Images", "value": 101, "display": "101", "href": "/app/media?filter=..." } ] },
{ "kind": "table", "title": "Media by type",
"columns": [ { "key": "type", "label": "Type", "align": "left" },
{ "key": "count", "label": "Count", "align": "right" },
{ "key": "share", "label": "Share", "align": "right" } ],
"rows": [ { "cells": { "type": "JPEG (image/jpeg)", "count": 81, "share": 0.802 },
"display": { "type": "JPEG (image/jpeg)", "count": "81", "share": "80.2%" },
"href": "/app/media?filter=..." } ] },
{ "kind": "chart", "chartType": "pie", "title": "Share by type",
"points": [ { "label": "JPEG", "value": 81, "display": "81", "href": "/app/media?filter=..." },
{ "label": "PNG", "value": 20, "display": "20", "href": "/app/media?filter=..." } ] },
{ "kind": "filterLinks", "items": [ { "label": "Open JPEG media (81)", "tableName": "media", "href": "/app/media?filter=..." } ] },
{ "kind": "recordLinks", "items": [ { "label": "Sunset.jpg", "tableName": "media", "recordId": "12", "href": "/app/media/12" } ] },
{ "kind": "actions", "items": [ { "label": "Show the PNG files", "prompt": "Show me the PNG media files" },
{ "label": "Open media", "href": "/app/media" } ] }
]
}
```
Rules:
- `summary` is required, one sentence, the direct answer first.
- `chartType` ∈ `bar`, `pie`, `line`; ≤ 12 points; values are numbers.
- Every `href` is built SERVER-SIDE from metadata and is an `/app` path (same rules as `TurnEngine.isAppHref`). Filtered list links are `/app/<tableName>?filter=<URL-encoded JSON {"criteria":[{"fieldName":..,"operator":..,"values":[..]}],"booleanOperator":"AND"}>`, which Next's `deserializeFilter` accepts (JSON starting with `{`). The model never writes URLs.
- `display` strings are formatted server-side from field metadata (`displayFormat`, e.g. `%,d`, `$%,.2f`; percent columns as `80.2%`); clients show `display` when present, else the raw value.
- `actions` items have either `prompt` (send as the user's next message) or `href` (navigate), never both.
- Unknown `kind`s must be ignored by clients (forward compatible); clients that do not render blocks render `markdown`.
- The SSE event is `presentation` with `{ "messageId": <id>, "part": <the part above> }`; Next's reducer appends it to the assistant message's parts.
## Tool input and implementation
The input schema is [present-results.schema.json](../qbit-assistant/src/main/resources/present-results.schema.json). `table.rowKeys` supplies one raw filter value per row; `table.rowFilter` names the table, field and optional operator (EQUALS by default). Columns receive stable server keys `c0`, `c1`, etc. Input cells remain raw scalars; each row has a parallel display map. The example keys above are illustrative.
`PresentResultsTool` is instantiated per turn. It caches `GET /metaData` with that turn's session, checks table read visibility and instance fields, validates QQQ criteria operators and builds paths using RC.5 `AiLinkBuilder`. Filter links use URL-encoded JSON accepted by Next. Record ids are encoded as opaque path segments. Metadata denial or invalid item drops that item (an invalid table drops its table/chart), with a short explanation to the model; other items survive. No data query or permission elevation happens here.
Limits: summary 300 characters, display text 2,000 characters, table 20 columns/200 rows, chart 12 points, item lists 100 entries, filters 20 criteria/200 values. Chart truncation is reported to the model. Percent columns reference a raw numeric column; a zero total yields 0.0%. Numbers still must originate in earlier data tool results; presentation is formatting, not a new source of facts.
The schema uses small named, nonrecursive definitions to avoid deeply inlined structures. Each object component is at most two levels of inline containers; the resolved table/criteria structure necessarily has the nesting shown in the owner contract. Strict is off by default. Opting `present_results` into strict mode requires all properties (empty arrays/strings, empty table and -1 for unused percentage indexes), removes provider-unsupported length constraints, and retains server validation. This has zero optional parameters and one shared scalar union in the existing strict-budget test.
Presentation parts are persisted with the assistant message and replayed as `presentation` events with `{messageId, part}` (plus the standard event sequence). They are removed by record-data retention along with tool renders. Model history and text-only consumers can use the server-built Markdown. Unknown block kinds should be ignored. Plain labels must be rendered as text, never parsed as links or HTML; fallback Markdown escapes them. Tool audit input records the accepted presentation only, excluding dropped values.
+89
View File
@@ -388,3 +388,92 @@ CREATE TABLE assistant_data_audit
CREATE INDEX assistant_data_audit_user_id ON assistant_data_audit (user_id);
CREATE INDEX assistant_data_audit_conversation_id ON assistant_data_audit (conversation_id);
CREATE INDEX assistant_data_audit_turn_id ON assistant_data_audit (turn_id);
-- P-d: data-driven budgets (UTC periods, USD or whole-token allowances).
CREATE TABLE assistant_budget
(
id SERIAL PRIMARY KEY,
name VARCHAR(200) NOT NULL,
enabled BOOLEAN DEFAULT TRUE,
scope_type VARCHAR(16) NOT NULL,
scope_value VARCHAR(200),
period VARCHAR(16) NOT NULL,
metric VARCHAR(16) NOT NULL,
limit_value NUMERIC NOT NULL CHECK (limit_value > 0),
warn_percent INTEGER DEFAULT 80 CHECK (warn_percent >= 0 AND warn_percent < 100),
at_limit_action VARCHAR(16) DEFAULT 'DOWNGRADE',
downgrade_model_id INTEGER REFERENCES assistant_model(id),
hard_ceiling_percent INTEGER DEFAULT 120 CHECK (hard_ceiling_percent >= 100),
notes VARCHAR(2000),
create_date TIMESTAMP WITH TIME ZONE,
modify_date TIMESTAMP WITH TIME ZONE
);
CREATE TABLE assistant_budget_override
(
id SERIAL PRIMARY KEY,
budget_id INTEGER NOT NULL REFERENCES assistant_budget(id),
scope_value VARCHAR(200),
extra_value NUMERIC,
unlimited_until TIMESTAMP WITH TIME ZONE,
reason VARCHAR(1000) NOT NULL,
granted_by VARCHAR(200),
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
create_date TIMESTAMP WITH TIME ZONE,
modify_date TIMESTAMP WITH TIME ZONE,
CHECK ((extra_value > 0 AND unlimited_until IS NULL) OR (extra_value IS NULL AND unlimited_until IS NOT NULL))
);
CREATE INDEX assistant_budget_override_budget_id ON assistant_budget_override(budget_id);
CREATE TABLE assistant_budget_event
(
id SERIAL PRIMARY KEY,
budget_id INTEGER NOT NULL REFERENCES assistant_budget(id),
scope_key VARCHAR(400) NOT NULL,
period_start TIMESTAMP WITH TIME ZONE NOT NULL,
state VARCHAR(16) NOT NULL,
used_value NUMERIC,
limit_value NUMERIC,
user_id VARCHAR(200),
create_date TIMESTAMP WITH TIME ZONE,
UNIQUE (budget_id, scope_key, period_start, state)
);
CREATE INDEX assistant_usage_budget_period ON assistant_usage(create_date, user_id, provider_id, model);
-- Private projection: standard admin permissions, AI NONE, no owner lock.
-- No conversation identifiers, text, request ids or credentials are copied.
CREATE TABLE assistant_budget_usage
(
id SERIAL PRIMARY KEY,
usage_id INTEGER NOT NULL UNIQUE,
user_id VARCHAR(255),
provider_id INTEGER,
model VARCHAR(200),
occurred_at TIMESTAMP WITH TIME ZONE NOT NULL,
cost_micros BIGINT,
input_tokens BIGINT,
output_tokens BIGINT,
cache_read_tokens BIGINT,
cache_write_tokens BIGINT
);
CREATE INDEX assistant_budget_usage_period ON assistant_budget_usage(occurred_at, provider_id, model);
-- Run once before enabling P-d, with writes paused, to include existing usage.
INSERT INTO assistant_budget_usage
(usage_id, user_id, provider_id, model, occurred_at, cost_micros, input_tokens, output_tokens, cache_read_tokens, cache_write_tokens)
SELECT id, user_id, provider_id, model, create_date, cost_micros, input_tokens, output_tokens, cache_read_tokens, cache_write_tokens
FROM assistant_usage
ON CONFLICT (usage_id) DO NOTHING;
-- P-e: profile routing, bounded fallback chains, provider health and turn outcome.
ALTER TABLE assistant_profile ADD COLUMN routing_mode VARCHAR(16) DEFAULT 'OFF'
CHECK (routing_mode IN ('OFF', 'HEURISTIC', 'CLASSIFIER'));
ALTER TABLE assistant_profile ADD COLUMN simple_model_id INTEGER REFERENCES assistant_model(id);
ALTER TABLE assistant_profile ADD COLUMN complex_model_id INTEGER REFERENCES assistant_model(id);
ALTER TABLE assistant_profile ADD COLUMN classifier_model_id INTEGER REFERENCES assistant_model(id);
ALTER TABLE assistant_model ADD COLUMN fallback_model_id INTEGER REFERENCES assistant_model(id);
ALTER TABLE assistant_provider ADD COLUMN health_state VARCHAR(8) CHECK (health_state IN ('UP', 'DOWN'));
ALTER TABLE assistant_provider ADD COLUMN health_checked_at TIMESTAMP WITH TIME ZONE;
ALTER TABLE assistant_provider ADD COLUMN down_until TIMESTAMP WITH TIME ZONE;
ALTER TABLE assistant_turn ADD COLUMN route_decision VARCHAR(16);
ALTER TABLE assistant_turn ADD COLUMN route_source VARCHAR(16);
ALTER TABLE assistant_turn ADD COLUMN fallback_count INTEGER DEFAULT 0;
ALTER TABLE assistant_turn ADD COLUMN provider_id INTEGER REFERENCES assistant_provider(id);
+3 -3
View File
@@ -20,7 +20,7 @@
<!-- Multi-module build. The qqq-orb sets the revision here; the modules -->
<!-- inherit it, and the flatten plugin resolves it in the published POMs. -->
<properties>
<revision>0.1.0-RC.7</revision>
<revision>0.1.0-SNAPSHOT</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -193,7 +193,7 @@
</build>
<profiles>
<!-- The assistant needs QQQ 4.1.0-RC.4: the AI tool kernel (package -->
<!-- The assistant needs QQQ 4.1.0-RC.5: the AI tool kernel (package -->
<!-- actions.ai, qqq#1078) and the developer panels (qqq#1076). Until -->
<!-- qbit-build-parent imports the 4.1 BOM, this -->
<!-- profile is active by default and imports qqq-bom-pom ahead of the -->
@@ -205,7 +205,7 @@
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.5</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -278,6 +278,26 @@ public class AnthropicModelProvider implements AssistantModelProvider
return (handle);
}
if(Boolean.FALSE.equals(request.getStreaming()))
{
java.util.concurrent.Future<?> completion = executor.submit(() ->
{
try
{
handle.complete(AnthropicMapping.toResponse(client.messages().create(params)));
}
catch(Throwable error)
{
if(!handle.isCancelled())
{
handle.fail(AnthropicMapping.toProviderException(error));
}
}
});
handle.setOnCancel(() -> completion.cancel(true));
return (handle);
}
AtomicReference<ModelUsage> seen = new AtomicReference<>();
handle.setPartialUsage(seen::get);
@@ -51,6 +51,39 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
*******************************************************************************/
class AnthropicModelProviderTest
{
/*******************************************************************************
** Routing requests use a single JSON reply without text deltas.
*******************************************************************************/
@Test
void nonStreamingClassifier() throws Exception
{
server.thenError(200, "classifier-response.json");
ModelRequest request = new ModelRequest().withModel("claude-sonnet-5-5").withMaxTokens(4).withStreaming(false)
.withMessages(List.of(ModelMessage.user("hello")));
List<String> deltas = new ArrayList<>();
ModelResponse response = provider.stream(request, deltas::add).await(Duration.ofSeconds(3));
assertThat(response.getText()).isEqualTo("SIMPLE");
assertThat(response.getUsage().getOutputTokens()).isEqualTo(1L);
assertThat(deltas).isEmpty();
assertThat(sentBody(0).get("stream")).isIn(null, false);
assertThat(sentBody(0).get("max_tokens")).isEqualTo(4);
}
/*******************************************************************************
** Non-streaming errors retain the normal exception mapping.
*******************************************************************************/
@Test
void nonStreamingError() throws Exception
{
server.thenError(401, "unauthorized.json");
assertThatThrownBy(() -> provider.stream(request().withStreaming(false), new StringBuilder()::append).await(Duration.ofSeconds(3)))
.isInstanceOfSatisfying(ModelProviderException.class, error -> assertThat(error.getKind()).isEqualTo(ModelProviderException.Kind.AUTHENTICATION));
}
private static final String TEST_KEY = "sk-ant-test-0000000000000000000000";
private RecordedAnthropicServer server;
@@ -108,6 +108,24 @@ class AnthropicRequestContractTest
/*******************************************************************************
** Presentation remains inside the actual serialized strict request budget.
*******************************************************************************/
@Test
void testPresentationStrictOptIn() throws Exception
{
Map<String, Object> body = requestBody(new AssistantQBitConfig().withModelProvider(new ScriptedModelProvider()).withStrictToolNames(Set.of("present_results")));
assertThat(violations(body)).isEmpty();
assertThat(strictBudgetViolations(body)).isEmpty();
assertThat(tools(body)).anySatisfy(tool ->
{
assertThat(tool.get("name")).isEqualTo("present_results");
assertThat(tool.get("strict")).isEqualTo(true);
});
}
/*******************************************************************************
** The RC.3 shape (every tool strict) is recognized as over budget, and a
** host can still opt a few tools in.
@@ -0,0 +1 @@
{"id":"msg_classify","type":"message","role":"assistant","model":"claude-sonnet-5-5","content":[{"type":"text","text":"SIMPLE"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":18,"output_tokens":1}}
@@ -157,7 +157,7 @@ public class OpenAiCompatibleModelProvider implements AssistantModelProvider
{
throw (http.error(response.statusCode(), new String(response.body().readNBytes(64 * 1024), StandardCharsets.UTF_8), result.requestId, strict));
}
if(Boolean.TRUE.equals(caps.getStreaming()))
if(!Boolean.FALSE.equals(request.getStreaming()) && Boolean.TRUE.equals(caps.getStreaming()))
{
readEvents(response.body(), lastRead, result, strict, handle);
}
@@ -52,7 +52,7 @@ final class OpenAiMapping
static ObjectNode request(ModelRequest request, String kind, ModelCapabilities capabilities) throws JsonProcessingException
{
ObjectNode body = OpenAiHttp.JSON.createObjectNode();
boolean streaming = Boolean.TRUE.equals(capabilities.getStreaming());
boolean streaming = !Boolean.FALSE.equals(request.getStreaming()) && Boolean.TRUE.equals(capabilities.getStreaming());
boolean nativeOpenAi = List.of("OPENAI", "AZURE_OPENAI").contains(kind);
String model = request.getModel();
body.put("model", "VERTEX".equals(kind) && model != null && !model.contains("/") ? "google/" + model : model);
@@ -45,6 +45,27 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
*******************************************************************************/
class OpenAiContractTest
{
/*******************************************************************************
** Classifiers opt out of streaming even when the model supports it.
*******************************************************************************/
@Test
void nonStreamingClassifier() throws Exception
{
try(MockOpenAiServer server = new MockOpenAiServer())
{
server.enqueue(200, "{\"model\":\"test\",\"choices\":[{\"message\":{\"content\":\"SIMPLE\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":18,\"completion_tokens\":1}}");
var provider = new OpenAiCompatibleModelProvider(settings(server, "OPENAI"));
ModelResponse response = provider.stream(request().withStreaming(false).withMaxTokens(4), new StringBuilder()::append).await(Duration.ofSeconds(3));
assertThat(response.getText()).isEqualTo("SIMPLE");
JsonNode body = OpenAiHttp.JSON.readTree(server.requests.getFirst().body());
assertThat(body.path("stream").asBoolean()).isFalse();
assertThat(body.path("max_completion_tokens").asInt()).isEqualTo(4);
assertThat(body.has("stream_options")).isFalse();
}
}
static final String TEXT = "data: {\"model\":\"test\",\"choices\":[{\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}]}\r\n\r\ndata: [DONE]\r\n\r\n";
@@ -28,9 +28,9 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
** from the provider (no date suffix). Effort is sent explicitly on every
** request, because the models' defaults differ (Sonnet 5.5 defaults to high).
**
** Phase 1 uses only the main slot. Escalation (Opus) and the summary model
** (Haiku: titles, compaction) arrive in phase 2; they are configurable now
** so the admin policy shape is stable.
** With no effective profile, enabled escalation supplies the complex route's
** model and effort. A profile's routing mode takes precedence. The summary
** slot remains available for titles and compaction.
*******************************************************************************/
public class AssistantModelRouting implements Serializable
{
@@ -28,6 +28,9 @@ import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
import com.kingsrook.qbits.assistant.processes.DiscoverModelsProcess;
import com.kingsrook.qbits.assistant.processes.TestConnectionProcess;
import com.kingsrook.qbits.assistant.processes.TestModelConnectionProcess;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
@@ -155,6 +158,9 @@ public class AssistantQBitProducer implements QBitMetaDataProducer<AssistantQBit
qInstance.addTable(customized);
tables.add(customized);
}
qInstance.add(budgetAlertType("warning", "Assistant budget warning", "Assistant usage has reached the budget warning threshold.", AlertSeverity.WARNING));
qInstance.add(budgetAlertType("downgraded", "Assistant budget downgraded", "Assistant usage has reached the budget limit and switched to a cheaper model.", AlertSeverity.WARNING));
qInstance.add(budgetAlertType("stopped", "Assistant budget stopped", "Assistant dispatch has stopped because a budget limit was reached.", AlertSeverity.ERROR));
qInstance.addPossibleValueSource(builder.buildConversationPossibleValueSource());
AssistantRecordScreens.buildPossibleValueSources().forEach(qInstance::addPossibleValueSource);
AssistantRecordScreens.buildSettingsPossibleValueSources(qBitConfig).forEach(qInstance::addPossibleValueSource);
@@ -185,6 +191,18 @@ public class AssistantQBitProducer implements QBitMetaDataProducer<AssistantQBit
/*******************************************************************************
** Declare budget alerts as top-level core metadata for admin visibility.
*******************************************************************************/
private QAlertTypeMetaData budgetAlertType(String state, String label, String description, AlertSeverity severity)
{
return (new QAlertTypeMetaData().withName("assistant.budget." + state).withLabel(label).withDescription(description)
.withDefaultSeverity(severity).withHowToFix("Raise the budget or grant an override in Assistant → Budgets")
.withSourceQBitName(ARTIFACT_ID).withVisibilityPermission("assistant.admin").withSubjectTableName(AssistantBudget.TABLE_NAME));
}
/*******************************************************************************
** Build the runtime and register the routes.
*******************************************************************************/
@@ -17,14 +17,9 @@
package com.kingsrook.qbits.assistant;
import java.time.Clock;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.regex.Pattern;
import com.kingsrook.qbits.assistant.access.AccessExplainer;
import com.kingsrook.qbits.assistant.access.DataAccessService;
import com.kingsrook.qbits.assistant.engine.BudgetService;
import com.kingsrook.qbits.assistant.engine.ContextBuilder;
import com.kingsrook.qbits.assistant.engine.PolicyService;
import com.kingsrook.qbits.assistant.engine.ProfileService;
@@ -42,6 +37,12 @@ import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import java.time.Clock;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.regex.Pattern;
/*******************************************************************************
@@ -57,6 +58,7 @@ public class AssistantRuntime
private final AssistantStore store;
private final PolicyService policyService;
private final ProviderService providerService;
private final BudgetService budgetService;
private final RateLimiter rateLimiter;
private final AiToolKernel kernel;
private final ContextBuilder contextBuilder;
@@ -92,6 +94,8 @@ public class AssistantRuntime
this.store = new AssistantStore(config);
this.policyService = new PolicyService(config, store, clock);
this.providerService = new ProviderService(config, store, clock);
this.budgetService = new BudgetService(store, providerService, clock);
this.policyService.withBudgetService(budgetService, providerService);
this.rateLimiter = new RateLimiter(clock);
AiToolKernelSettings settings = kernelSettings(config);
this.kernel = new AiToolKernel(qInstance, settings);
@@ -405,4 +409,12 @@ public class AssistantRuntime
return (providerService);
}
/*******************************************************************************
** Data-driven budget enforcement.
*******************************************************************************/
public BudgetService getBudgetService()
{
return (budgetService);
}
}
@@ -0,0 +1,46 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
/*******************************************************************************
** Budget signals through core's non-throwing Alerts API and default logging sink.
*******************************************************************************/
public final class BudgetAlerts
{
/*******************************************************************************
** Utility class.
*******************************************************************************/
private BudgetAlerts()
{
}
/*******************************************************************************
** Raise one deduplicated budget transition through the configured core sink.
*******************************************************************************/
public static void raise(String type, String dedupKey, AlertSeverity severity, String title, String summary)
{
QAlerts.raise(new AlertSignal(type).withDedupKey(dedupKey).withSeverity(severity).withTitle(title).withSummary(summary));
}
}
@@ -0,0 +1,329 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.Clock;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
/*******************************************************************************
** Data-driven budgets layered on the existing policy and provider caps.
** Evaluation occurs at dispatch boundaries, never during streaming tokens.
*******************************************************************************/
public class BudgetService
{
private final AssistantStore store;
private final ProviderService providers;
private final Clock clock;
/*******************************************************************************
** Read-only budget snapshot, also used by the app home.
*******************************************************************************/
public record Snapshot(AssistantBudget budget, String scopeKey, Instant periodStart, BigDecimal used, BigDecimal limit, BigDecimal percent, String state, List<AssistantBudgetOverride> overrides)
{
}
/*******************************************************************************
** Strictest decision and newly raised notices.
*******************************************************************************/
public record Decision(boolean stopped, String reason, ProviderService.Active active, List<String> notices)
{
}
/*******************************************************************************
** A safe terminal budget error, distinct from provider failures.
*******************************************************************************/
public static class Blocked extends QException
{
/*******************************************************************************
** Name the budget that refused this dispatch.
*******************************************************************************/
public Blocked(String reason)
{
super(reason);
}
}
/*******************************************************************************
** Production constructor.
*******************************************************************************/
public BudgetService(AssistantStore store, ProviderService providers, Clock clock)
{
this.store = store;
this.providers = providers;
this.clock = clock;
}
/*******************************************************************************
** Evaluate all applicable budgets. STOP wins; a downgrade is checked against
** the original and target scopes so switching cannot evade another budget.
*******************************************************************************/
public Decision evaluate(String userId, ProviderService.Active active, boolean requiresHipaa, boolean preflight) throws QException
{
List<String> notices = new ArrayList<>();
ProviderService.Active selected = active;
List<AssistantBudget> budgets = store.budgets();
List<Integer> applied = new ArrayList<>();
List<Snapshot> pending = new ArrayList<>();
List<String> messages = new ArrayList<>();
for(int pass = 0; pass <= budgets.size(); pass++)
{
boolean changed = false;
for(AssistantBudget budget : budgets)
{
if(applied.contains(budget.getId()) || (!applies(budget, userId, active) && !applies(budget, userId, selected)))
{
continue;
}
applied.add(budget.getId());
Snapshot snapshot = snapshot(budget, userId);
String state = snapshot.state();
String message = null;
ProviderService.Active target = null;
if("DOWNGRADED".equals(state))
{
target = providers.budgetDowngrade(budget.getDowngradeModelId(), requiresHipaa, active);
if(target == null)
{
state = "STOPPED";
}
else
{
message = "Budget reached: switched to " + target.model() + " for the rest of the period.";
}
}
if("STOPPED".equals(state))
{
message = "Assistant budget '" + budget.getName() + "' has been reached. Ask an administrator for an override.";
}
if("WARNED".equals(state))
{
message = "You have used " + snapshot.percent().setScale(0, RoundingMode.DOWN) + "% of the "
+ ("DAY".equals(budget.getPeriod()) ? "daily" : "monthly") + " assistant budget '" + budget.getName() + "'.";
}
if("STOPPED".equals(state))
{
if(transition(snapshot, state, userId, message))
{
notices.add(message);
}
return (new Decision(true, message, selected, notices));
}
if(message != null && (!preflight || !"DOWNGRADED".equals(state)))
{
pending.add(snapshot);
messages.add(message);
}
if(target != null)
{
selected = target;
changed = true;
}
}
if(!changed)
{
break;
}
}
for(int i = 0; i < pending.size(); i++)
{
Snapshot snapshot = pending.get(i);
String message = "DOWNGRADED".equals(snapshot.state()) ? "Budget reached: switched to " + selected.model() + " for the rest of the period." : messages.get(i);
if(transition(snapshot, snapshot.state(), userId, message))
{
notices.add(message);
}
}
return (new Decision(false, null, selected, notices));
}
/*******************************************************************************
** Whether the caller and selected provider/model are within this scope.
*******************************************************************************/
private boolean applies(AssistantBudget budget, String userId, ProviderService.Active active) throws QException
{
return (switch(budget.getScopeType())
{
case "INSTANCE" -> true;
case "USER" -> Objects.equals(budget.getScopeValue(), userId);
case "ROLE" -> QContext.getQSession().hasPermission(budget.getScopeValue());
case "PROVIDER" -> active != null && Objects.equals(budget.getScopeValue(), String.valueOf(active.providerId()));
case "MODEL" -> modelMatches(budget, active);
default -> false;
});
}
/*******************************************************************************
** Model scope is a registry row id, never an ambiguous cross-provider name.
*******************************************************************************/
private boolean modelMatches(AssistantBudget budget, ProviderService.Active active) throws QException
{
AssistantModel model = scopedModel(budget);
return (model != null && active != null && Objects.equals(model.getProviderId(), active.providerId()) && Objects.equals(model.getModelId(), active.model()));
}
/*******************************************************************************
** Resolve the scoped registry model.
*******************************************************************************/
private AssistantModel scopedModel(AssistantBudget budget) throws QException
{
return (store.listModels().stream().filter(m -> String.valueOf(m.getId()).equals(budget.getScopeValue())).findFirst().orElse(null));
}
/*******************************************************************************
** UTC period, usage, active overrides and effective state, without events.
** ROLE allowances are per permission holder, evaluated with their session.
*******************************************************************************/
public Snapshot snapshot(AssistantBudget budget, String userId) throws QException
{
userId = "USER".equals(budget.getScopeType()) ? budget.getScopeValue() : userId;
Instant now = clock.instant();
LocalDate day = LocalDate.ofInstant(now, ZoneOffset.UTC);
LocalDate start = "MONTH".equals(budget.getPeriod()) ? day.withDayOfMonth(1) : day;
Instant from = start.atStartOfDay().toInstant(ZoneOffset.UTC);
Instant until = ("MONTH".equals(budget.getPeriod()) ? start.plusMonths(1) : start.plusDays(1)).atStartOfDay().toInstant(ZoneOffset.UTC);
String scopeKey = budget.getScopeType() + ":" + ("ROLE".equals(budget.getScopeType()) ? budget.getScopeValue() + ":" + userId : Objects.toString(budget.getScopeValue(), ""));
boolean shared = List.of("INSTANCE", "PROVIDER", "MODEL").contains(budget.getScopeType());
String dateField = shared ? "occurredAt" : "createDate";
QQueryFilter filter = new QQueryFilter().withCriteria(dateField, QCriteriaOperator.GREATER_THAN_OR_EQUALS, from)
.withCriteria(dateField, QCriteriaOperator.LESS_THAN, until);
switch(budget.getScopeType())
{
case "USER" -> filter.withCriteria("userId", QCriteriaOperator.EQUALS, budget.getScopeValue());
case "ROLE" -> filter.withCriteria("userId", QCriteriaOperator.EQUALS, userId);
case "PROVIDER" -> filter.withCriteria("providerId", QCriteriaOperator.EQUALS, Integer.valueOf(budget.getScopeValue()));
case "MODEL" ->
{
AssistantModel model = scopedModel(budget);
filter.withCriteria("providerId", QCriteriaOperator.EQUALS, model == null ? -1 : model.getProviderId());
filter.withCriteria("model", QCriteriaOperator.EQUALS, model == null ? "" : model.getModelId());
}
default ->
{
// Shared accounting contains no conversation or request data.
}
}
BigDecimal used = BigDecimal.ZERO;
for(AssistantUsage usage : store.query(shared ? AssistantBudgetUsage.TABLE_NAME : AssistantUsage.TABLE_NAME, filter, AssistantUsage.class))
{
used = used.add("COST_USD".equals(budget.getMetric()) ? BigDecimal.valueOf(nz(usage.getCostMicros()), 6)
: BigDecimal.valueOf(nz(usage.getInputTokens())).add(BigDecimal.valueOf(nz(usage.getOutputTokens())))
.add(BigDecimal.valueOf(nz(usage.getCacheReadTokens()))).add(BigDecimal.valueOf(nz(usage.getCacheWriteTokens()))));
}
BigDecimal limit = budget.getLimitValue();
boolean unlimited = false;
List<AssistantBudgetOverride> overrides = new ArrayList<>();
for(AssistantBudgetOverride override : store.budgetOverrides(budget.getId()))
{
if(override.getExpiresAt() == null || !override.getExpiresAt().isAfter(now)
|| (override.getScopeValue() != null && !override.getScopeValue().isBlank() && !override.getScopeValue().equals(userId)))
{
continue;
}
overrides.add(override);
if(override.getUnlimitedUntil() != null && override.getUnlimitedUntil().isAfter(now))
{
unlimited = true;
}
if(override.getExtraValue() != null)
{
limit = limit.add(override.getExtraValue());
}
}
BigDecimal percent = unlimited ? BigDecimal.ZERO : used.multiply(BigDecimal.valueOf(100)).divide(limit, 2, RoundingMode.DOWN);
String state = unlimited ? "UNLIMITED" : percent.compareTo(BigDecimal.valueOf(budget.getHardCeilingPercent())) >= 0 ? "STOPPED"
: percent.compareTo(BigDecimal.valueOf(100)) >= 0 ? ("STOP".equals(budget.getAtLimitAction()) ? "STOPPED" : "DOWNGRADED")
: percent.compareTo(BigDecimal.valueOf(budget.getWarnPercent())) >= 0 ? "WARNED" : "OK";
return (new Snapshot(budget, scopeKey, from, used, unlimited ? null : limit, percent, state, overrides));
}
/*******************************************************************************
** The persisted unique key is also the warning/alert deduplication record.
** Resolution on period rollover or override recovery needs an active-alert
** lifecycle: these immutable events cannot re-raise a resolved state within
** the same period, and user-specific overrides must not resolve shared alerts.
** Track active keys and recurrence before calling QAlerts.resolve here.
*******************************************************************************/
private boolean transition(Snapshot snapshot, String state, String userId, String message) throws QException
{
if(!store.budgetEvent(new AssistantBudgetEvent().withBudgetId(snapshot.budget().getId()).withScopeKey(snapshot.scopeKey())
.withPeriodStart(snapshot.periodStart()).withState(state).withUsedValue(snapshot.used()).withLimitValue(snapshot.limit())
.withUserId(userId).withCreateDate(clock.instant())))
{
return (false);
}
String type = "assistant.budget." + switch(state)
{
case "WARNED" -> "warning";
case "DOWNGRADED" -> "downgraded";
default -> "stopped";
};
BudgetAlerts.raise(type, snapshot.budget().getId() + ":" + snapshot.scopeKey() + ":" + snapshot.periodStart(),
"STOPPED".equals(state) ? AlertSeverity.ERROR : AlertSeverity.WARNING, "Assistant budget " + snapshot.budget().getName(), message);
return (true);
}
/*******************************************************************************
** Null usage counters are zero.
*******************************************************************************/
private static long nz(Long value)
{
return (value == null ? 0 : value);
}
}
@@ -34,6 +34,7 @@ import com.kingsrook.qbits.assistant.provider.ModelPart;
import com.kingsrook.qbits.assistant.provider.ModelRequest;
import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
import com.kingsrook.qbits.assistant.provider.SystemBlock;
import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolDefinition;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolKernel;
@@ -116,6 +117,7 @@ public class ContextBuilder
.withStrict(isStrict(definition.getName(), !Boolean.FALSE.equals(definition.getStrict()))));
}
}
byName.put(PresentResultsTool.NAME, PresentResultsTool.definition(isStrict(PresentResultsTool.NAME, true)));
return (new ArrayList<>(byName.values()));
}
@@ -0,0 +1,362 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Consumer;
import java.util.function.Supplier;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.classification.DataClassification;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantTurn;
import com.kingsrook.qbits.assistant.provider.ModelEventListener;
import com.kingsrook.qbits.assistant.provider.ModelMessage;
import com.kingsrook.qbits.assistant.provider.ModelProviderException;
import com.kingsrook.qbits.assistant.provider.ModelRequest;
import com.kingsrook.qbits.assistant.provider.ModelResponse;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** At most four fallback edges per call, never after that call emitted output.
** Explicit chains may cross providers. Implicit alternatives stay local and in
** the same adapter family; PHI, enablement, health and budgets always narrow them.
*******************************************************************************/
public class FallbackChain
{
public static final Integer MAX_HOPS = 4;
private final AssistantRuntime runtime;
private final TurnClassification classification;
private final AssistantTurn turn;
private final Consumer<String> notice;
private final AtomicBoolean output = new AtomicBoolean();
private final ProviderHealth health;
/*******************************************************************************
** The engine owns dispatch, usage, cancellation and strict-schema retries.
*******************************************************************************/
@FunctionalInterface
public interface Dispatch
{
/*******************************************************************************
** Dispatch one selected model through the existing guards.
*******************************************************************************/
ModelResponse call(ProviderService.Active selected) throws QException, ModelProviderException, InterruptedException;
}
/*******************************************************************************
** One chain tracker per turn.
*******************************************************************************/
public FallbackChain(AssistantRuntime runtime, TurnClassification classification, AssistantTurn turn, Consumer<String> notice)
{
this.runtime = runtime;
this.classification = classification;
this.turn = turn;
this.notice = notice;
this.health = new ProviderHealth(runtime);
turn.setFallbackCount(0);
}
/*******************************************************************************
** Start a distinct attempt, including a strict-schema retry.
*******************************************************************************/
public void beforeCall(ProviderService.Active active) throws QException, ModelProviderException, InterruptedException
{
output.set(false);
health.beforeCall(active);
}
/*******************************************************************************
** Count text and tool starts conservatively, even when text is still buffered.
*******************************************************************************/
public ModelEventListener listener(ModelEventListener delegate)
{
return (new ModelEventListener()
{
/*******************************************************************************
** Visible text prevents another provider from replacing this attempt.
*******************************************************************************/
@Override
public void onTextDelta(String text)
{
if(text != null && !text.isEmpty())
{
output.set(true);
}
delegate.onTextDelta(text);
}
/*******************************************************************************
** A partial tool call also prevents retry.
*******************************************************************************/
@Override
public void onToolUseStart(String id, String name)
{
output.set(true);
delegate.onToolUseStart(id, name);
}
});
}
/*******************************************************************************
** Used by strict retries as well as provider fallback.
*******************************************************************************/
public Boolean hasOutput()
{
return (output.get());
}
/*******************************************************************************
** Retain the original route's budget scopes across downgrades and retries.
** A fallback may change the destination, never evade an original STOP.
*******************************************************************************/
public BudgetService.Decision budget(ProviderService.Active origin, ProviderService.Active candidate) throws QException
{
Boolean phi = classification.getClassification() == DataClassification.PHI;
BudgetService.Decision original = runtime.getBudgetService().evaluate(turn.getUserId(), origin, phi, false);
if(original.stopped() || (Objects.equals(origin.providerId(), candidate.providerId()) && Objects.equals(origin.model(), candidate.model())))
{
return (original);
}
BudgetService.Decision destination = runtime.getBudgetService().evaluate(turn.getUserId(), candidate, phi, false);
List<String> notices = new ArrayList<>(original.notices());
notices.addAll(destination.notices());
return (new BudgetService.Decision(destination.stopped(), destination.reason(), destination.active(), notices));
}
/*******************************************************************************
** Resolve candidates lazily after a failure; refresh rows before every attempt.
*******************************************************************************/
public ModelResponse call(ModelRequest request, Supplier<ProviderService.Active> current, Dispatch dispatch) throws QException, ModelProviderException, InterruptedException
{
ProviderService.Active selected = current.get();
String unavailable = null;
Set<Integer> visited = new HashSet<>();
Integer hops = 0;
Boolean explicitChain = null;
while(true)
{
try
{
ModelResponse response = dispatch.call(selected);
if(!ModelResponse.STOP_CANCELLED.equals(response.getStopReason()))
{
health.succeeded(current.get());
if(unavailable != null)
{
notice.accept("Provider " + unavailable + " is unavailable; answered with " + name(current.get()) + ".");
}
}
return (response);
}
catch(ModelProviderException error)
{
ProviderService.Active failed = current.get();
health.failed(failed, error);
if(!error.isRetryable() || output.get() || hops >= MAX_HOPS)
{
throw (error);
}
List<AssistantModel> models = runtime.getStore().listModels();
AssistantModel origin = models.stream().filter(m -> Objects.equals(m.getProviderId(), failed.providerId()) && Objects.equals(m.getModelId(), failed.model())).findFirst().orElse(null);
if(origin != null)
{
visited.add(origin.getId());
}
if(explicitChain == null)
{
explicitChain = origin != null && origin.getFallbackModelId() != null;
}
selected = null;
List<AssistantModel> candidates = explicitChain && (origin == null || origin.getFallbackModelId() == null) ? List.of() : candidates(origin, failed, models);
for(AssistantModel candidate : candidates)
{
if(hops >= MAX_HOPS)
{
break;
}
if(!visited.add(candidate.getId()))
{
continue;
}
hops++;
ProviderService.Active resolved = runtime.getProviderService().resolveModel(candidate.getId(), classification.getClassification() == DataClassification.PHI);
if(resolved == null || health.isDown(resolved.row()) || PhiRoutingGuard.check(classification, resolved.row(), runtime.getConfig().getHipaaMode()).isPresent())
{
continue;
}
BudgetService.Decision budget = runtime.getBudgetService().evaluate(turn.getUserId(), resolved, classification.getClassification() == DataClassification.PHI, false);
if(budget.stopped())
{
continue;
}
if(!explicitChain && !"LOCAL".equals(runtime.getProviderService().location(budget.active().row())))
{
continue;
}
Long cap = ProviderService.capMicros(budget.active().row());
if(health.isDown(budget.active().row()) || (cap != null && runtime.getProviderService().monthToDateCostMicros(budget.active().providerId()) >= cap))
{
continue;
}
selected = budget.active();
budget.notices().forEach(notice);
break;
}
if(selected == null)
{
throw (error);
}
if(unavailable == null)
{
unavailable = name(failed);
}
turn.setFallbackCount(turn.getFallbackCount() + 1);
neutralHistory(request);
}
}
}
/*******************************************************************************
** Explicit edges win. The automatic pool is LOCAL only, same adapter family.
*******************************************************************************/
private List<AssistantModel> candidates(AssistantModel origin, ProviderService.Active failed, List<AssistantModel> models) throws QException
{
List<AssistantModel> result = new ArrayList<>();
if(origin != null && origin.getFallbackModelId() != null)
{
Integer next = origin.getFallbackModelId();
Set<Integer> seen = new HashSet<>();
while(next != null && seen.add(next) && result.size() < MAX_HOPS)
{
Integer id = next;
AssistantModel row = models.stream().filter(m -> id.equals(m.getId())).findFirst().orElse(null);
if(row == null)
{
break;
}
result.add(row);
next = row.getFallbackModelId();
}
}
else if(failed.row() != null && "LOCAL".equals(runtime.getProviderService().location(failed.row())))
{
for(AssistantProvider provider : runtime.getStore().listProviders())
{
if("LOCAL".equals(runtime.getProviderService().location(provider)) && Objects.equals(provider.getProviderType(), failed.providerType()))
{
models.stream().filter(m -> Objects.equals(m.getProviderId(), provider.getId())).forEach(result::add);
}
}
}
return (result);
}
/*******************************************************************************
** Rebuild messages from neutral parts; signatures never cross provider boundaries.
*******************************************************************************/
public static void neutralHistory(ModelRequest request)
{
List<ModelMessage> messages = new ArrayList<>();
for(ModelMessage message : request.getMessages())
{
messages.add(new ModelMessage().withRole(message.getRole()).withParts(new ArrayList<>(message.getParts())).withCacheHint(message.getCacheHint()));
}
request.setMessages(messages);
request.getTools().forEach(tool -> tool.setStrict(false));
}
/*******************************************************************************
** User-visible names only.
*******************************************************************************/
private static String name(ProviderService.Active active)
{
return (active.row() == null ? active.providerType() : active.row().getName());
}
/*******************************************************************************
** Validate the proposed graph (including every row of a batch update).
*******************************************************************************/
public static String validate(AssistantModel origin, Map<Integer, AssistantModel> models)
{
return (validate(origin, models, true));
}
/*******************************************************************************
** Recheck ancestor lengths without preventing an admin from disabling a
** destination. Runtime selection skips disabled models and follows the chain.
*******************************************************************************/
public static String validate(AssistantModel origin, Map<Integer, AssistantModel> models, Boolean requireEnabled)
{
Set<Integer> seen = new HashSet<>();
seen.add(origin.getId());
Integer next = origin.getFallbackModelId();
Integer hops = 0;
while(next != null)
{
if(!seen.add(next))
{
return ("Fallback chain must not contain a cycle.");
}
if(++hops > MAX_HOPS)
{
return ("Fallback chain must have at most four hops.");
}
AssistantModel candidate = models.get(next);
if(candidate == null || (requireEnabled && !Boolean.TRUE.equals(candidate.getEnabled())))
{
return ("Fallback model must exist and be enabled.");
}
next = candidate.getFallbackModelId();
}
return (null);
}
}
@@ -31,7 +31,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
** assistantMessage.partsJson and returned by the API:
** {type:"text", text} | {type:"tool", toolCallId, name, summary, outcome, render?}
** | {type:"record_refs", refs:[...]} | {type:"navigate", href}
** | {type:"notice", level, text}
** | {type:"notice", level, text} | {type:"presentation", summary, blocks, markdown}
** Consecutive text is merged into one part. Thread-safe.
*******************************************************************************/
public class MessageParts
@@ -110,6 +110,17 @@ public class MessageParts
/*******************************************************************************
** Append a server-built presentation, one part per successful tool call.
*******************************************************************************/
public synchronized MessageParts addPresentation(Map<String, Object> presentation)
{
parts.add(new LinkedHashMap<>(presentation));
return (this);
}
/*******************************************************************************
** Append record references.
*******************************************************************************/
@@ -150,12 +161,12 @@ public class MessageParts
/*******************************************************************************
** Remove record data (retention): the render payloads of tool parts and
** the record_refs parts. Tool summaries, text, navigation and notices
** the record_refs and presentation parts. Tool summaries, text, navigation and notices
** stay. Returns whether anything was removed.
*******************************************************************************/
public synchronized boolean stripRecordData()
{
boolean changed = parts.removeIf(p -> "record_refs".equals(p.get("type")));
boolean changed = parts.removeIf(p -> ("record_refs".equals(p.get("type")) || "presentation".equals(p.get("type"))));
for(Map<String, Object> part : parts)
{
if("tool".equals(part.get("type")) && part.remove("render") != null)
@@ -169,7 +180,7 @@ public class MessageParts
/*******************************************************************************
** All text parts joined (for the model's history and titles).
** Text and presentation fallbacks joined (for model history and titles).
*******************************************************************************/
public synchronized String text()
{
@@ -180,6 +191,10 @@ public class MessageParts
{
builder.append(part.get("text"));
}
else if("presentation".equals(part.get("type")))
{
builder.append("\n").append(part.get("markdown")).append("\n");
}
}
return (builder.toString());
}
@@ -0,0 +1,281 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import java.util.stream.Collectors;
import java.util.Objects;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.Semaphore;
import java.util.function.Consumer;
import java.util.regex.Pattern;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.classification.DataClassification;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantPendingAction;
import com.kingsrook.qbits.assistant.model.AssistantProfile;
import com.kingsrook.qbits.assistant.model.AssistantTurn;
import com.kingsrook.qbits.assistant.provider.CacheHint;
import com.kingsrook.qbits.assistant.provider.ModelMessage;
import com.kingsrook.qbits.assistant.provider.ModelProviderException;
import com.kingsrook.qbits.assistant.provider.ModelRequest;
import com.kingsrook.qbits.assistant.provider.ModelResponse;
import com.kingsrook.qbits.assistant.provider.ModelUsage;
import com.kingsrook.qbits.assistant.provider.StreamHandle;
import com.kingsrook.qbits.assistant.provider.SystemBlock;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Selects the initial model from the effective profile. Classifier content is
** strictly the fixed instruction plus the question; no context is attached.
*******************************************************************************/
public class ModelRouter
{
public static final String CLASSIFIER_PROMPT = "Classify the user's request as SIMPLE or COMPLEX. Reply with one word.";
private static final Pattern COMPLEX_WORDS = Pattern.compile("\\b(analyze|compare|why|trend|explain|summarize\\s+across|plan)\\b", Pattern.CASE_INSENSITIVE);
private final AssistantRuntime runtime;
private final ModelUsage usage = new ModelUsage();
private Long costMicros = 0L;
private Integer calls = 0;
/*******************************************************************************
** Routing returns accounting as well as the selected model.
*******************************************************************************/
public record Route(ProviderService.Active active, String effort, ModelUsage usage, Long costMicros, Integer calls)
{
}
/*******************************************************************************
** One router per turn; accounting is never shared between users.
*******************************************************************************/
public ModelRouter(AssistantRuntime runtime)
{
this.runtime = runtime;
}
/*******************************************************************************
** Preserve classifier accounting even when later route resolution fails.
*******************************************************************************/
public Route accounting()
{
return (new Route(null, null, usage, costMicros, calls));
}
/*******************************************************************************
** Each rule is independently sufficient. Boundaries are >600 characters,
** at least two distinct table names, and >8 previous turns. Match words,
** not substrings (for example, planet does not imply plan).
*******************************************************************************/
public static String heuristic(String question, Set<String> tableNames, Boolean writeContext, Integer priorTurns, Boolean previousToolLimit)
{
String text = question == null ? "" : question;
Long mentioned = tableNames.stream().filter(name -> Pattern.compile("(?<![\\p{L}\\p{N}_])" + Pattern.quote(name) + "(?![\\p{L}\\p{N}_])", Pattern.CASE_INSENSITIVE).matcher(text).find()).count();
return (Boolean.TRUE.equals(writeContext) || text.length() > 600 || mentioned >= 2 || COMPLEX_WORDS.matcher(text).find()
|| priorTurns > 8 || Boolean.TRUE.equals(previousToolLimit) ? "COMPLEX" : "SIMPLE");
}
/*******************************************************************************
** OFF preserves row/default/policy selection. With no profile, configured
** escalation supplies the complex slot and its effort when enabled.
*******************************************************************************/
public Route route(QSession session, AssistantConversation conversation, AssistantTurn turn, String question, Map<String, Object> pageContext,
PolicyService.EffectivePolicy policy, TurnClassification classification, RunningTurn running, Instant deadline, Consumer<String> notice) throws QException, InterruptedException
{
AssistantProfile profile = runtime.getProfileService().pick(session, runtime.appNameFor(pageContext));
String mode = profile == null || profile.getRoutingMode() == null ? "OFF" : profile.getRoutingMode();
Boolean escalation = profile == null && Boolean.TRUE.equals(runtime.getConfig().getModels().getEscalationEnabled());
String decision = "OFF";
String source = null;
Integer selectedId = null;
String effort = policy.mainEffort();
if(!"OFF".equals(mode) || escalation)
{
List<AssistantTurn> previous = runtime.getStore().query(AssistantTurn.TABLE_NAME, new QQueryFilter()
.withCriteria("userId", QCriteriaOperator.EQUALS, turn.getUserId()).withCriteria("conversationId", QCriteriaOperator.EQUALS, conversation.getId())
.withCriteria("id", QCriteriaOperator.LESS_THAN, turn.getId()).withOrderBy(new QFilterOrderBy("id", false)).withLimit(9), AssistantTurn.class);
Boolean pending = !runtime.getStore().query(AssistantPendingAction.TABLE_NAME, new QQueryFilter()
.withCriteria("userId", QCriteriaOperator.EQUALS, turn.getUserId()).withCriteria("conversationId", QCriteriaOperator.EQUALS, conversation.getId())
.withCriteria("status", QCriteriaOperator.EQUALS, AssistantPendingAction.STATUS_PENDING).withLimit(1), AssistantPendingAction.class).isEmpty();
String digest = runtime.getContextBuilder().appDigest();
Set<String> tables = runtime.getQInstance().getTables().keySet().stream().filter(digest::contains).collect(Collectors.toSet());
decision = heuristic(question, tables, pending, previous.size(), !previous.isEmpty() && "tool_limit".equals(previous.get(0).getStopReason()));
source = "HEURISTIC";
if("CLASSIFIER".equals(mode) && profile.getClassifierModelId() != null)
{
String classified = classify(profile.getClassifierModelId(), question, turn, policy, classification, running, deadline, notice);
if(classified != null)
{
decision = classified;
source = "CLASSIFIER";
}
}
selectedId = profile == null ? null : ("COMPLEX".equals(decision) ? profile.getComplexModelId() : profile.getSimpleModelId());
}
turn.withRouteDecision(decision).withRouteSource(source);
ProviderService.Active active = selectedId == null ? null : runtime.getProviderService().resolveModel(selectedId, classification.getClassification() == DataClassification.PHI);
if(selectedId != null && active == null)
{
throw (new ProviderService.ProviderSetupException("The routed model is disabled, unavailable or not eligible for this conversation."));
}
if(active == null)
{
TurnDataRouting.beforeResolve(runtime, classification, turn, policy.mainModel());
active = runtime.getProviderService().resolve(policy.mainModel());
if(active != null && escalation && "COMPLEX".equals(decision))
{
String model = runtime.getConfig().getModels().getEscalationModel();
if(active.row() == null)
{
active = new ProviderService.Active(null, active.providerType(), model, active.provider());
}
else
{
active = runtime.getProviderService().forRow(new AssistantProvider(active.row().toQRecord()).withModel(model), model);
}
effort = runtime.getConfig().getModels().getEscalationEffort();
}
}
return (new Route(active, effort, usage, costMicros, calls));
}
/*******************************************************************************
** Buffer a classifier reply, with a three-second deadline and four output
** tokens. Errors and non-exact answers leave the heuristic decision intact.
*******************************************************************************/
private String classify(Integer modelId, String question, AssistantTurn turn, PolicyService.EffectivePolicy policy, TurnClassification classification,
RunningTurn running, Instant deadline, Consumer<String> notice) throws QException, InterruptedException
{
classification.inherit(runtime.getStore().getConversation(turn.getUserId(), turn.getConversationId()));
ProviderService.Active active = runtime.getProviderService().resolveModel(modelId, classification.getClassification() == DataClassification.PHI, false);
if(active == null || PhiRoutingGuard.check(classification, active.row(), runtime.getConfig().getHipaaMode()).isPresent())
{
return (null);
}
BudgetService.Decision budget = runtime.getBudgetService().evaluate(turn.getUserId(), active, classification.getClassification() == DataClassification.PHI, false);
budget.notices().forEach(notice);
PolicyService.BudgetStatus configured = runtime.getPolicyService().checkConfiguredBudget(policy, turn.getUserId(), 0, 0);
if(budget.stopped() || configured.hardExceeded() || !Objects.equals(budget.active().model(), active.model()) || !Objects.equals(budget.active().providerId(), active.providerId()))
{
return (null);
}
Long cap = ProviderService.capMicros(active.row());
if((cap != null && runtime.getProviderService().monthToDateCostMicros(active.providerId()) >= cap) || running.isCancelled() || runtime.getPolicyService().isKillSwitchOn() || !runtime.getClock().instant().isBefore(deadline))
{
return (null);
}
try
{
TurnDataRouting.beforeModelCall(runtime, classification, turn, active);
}
catch(TurnDataRouting.Blocked e)
{
return (null);
}
ModelRequest request = new ModelRequest().withModel(active.model()).withMaxTokens(4).withStreaming(false)
.withSystem(List.of(new SystemBlock(CLASSIFIER_PROMPT, CacheHint.NONE))).withMessages(List.of(ModelMessage.user(question)));
ModelResponse response = null;
ModelProviderException error = null;
StreamHandle handle = null;
Long started = System.nanoTime();
ProviderHealth health = new ProviderHealth(runtime);
Semaphore gate = runtime.getProviderService().gate(active.row());
Boolean acquired = false;
try
{
health.beforeCall(active);
if(gate != null)
{
acquired = gate.tryAcquire();
if(!acquired)
{
throw (new ModelProviderException(ModelProviderException.Kind.RATE_LIMITED, null, "Classifier concurrency limit reached."));
}
}
calls++;
handle = active.provider().stream(request, text ->
{
});
running.setCurrentHandle(handle);
Duration remaining = Duration.between(runtime.getClock().instant(), deadline);
response = handle.await(remaining.compareTo(Duration.ofSeconds(3)) < 0 ? remaining : Duration.ofSeconds(3));
if(!ModelResponse.STOP_CANCELLED.equals(response.getStopReason()))
{
health.succeeded(active);
String answer = response.getText().strip().toUpperCase(Locale.ROOT);
if(List.of("SIMPLE", "COMPLEX").contains(answer) && response.getToolUses().isEmpty())
{
return (answer);
}
}
return (null);
}
catch(ModelProviderException e)
{
error = e;
health.failed(active, e);
return (null);
}
catch(RuntimeException e)
{
error = new ModelProviderException(ModelProviderException.Kind.OTHER, null, "Classifier did not return a usable answer.");
return (null);
}
finally
{
if(acquired)
{
gate.release();
}
if(handle != null)
{
handle.cancel();
}
running.setCurrentHandle(null);
ModelUsage measured = response == null ? (error == null ? null : error.getPartialUsage()) : response.getUsage();
if(measured != null)
{
usage.add(measured);
costMicros += runtime.getProviderService().pricing(active, active.model()).costMicros(measured);
}
UsageRecorder.save(runtime.getStore(), UsageRecorder.row(new UsageRecorder.Call(turn.getUserId(), turn.getConversationId(), turn.getId(), active, UsageRecorder.PURPOSE_ROUTE),
response, error, (System.nanoTime() - started) / 1_000_000, runtime.getProviderService().pricing(active, active.model())));
}
}
}
@@ -17,12 +17,6 @@
package com.kingsrook.qbits.assistant.engine;
import java.time.Clock;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
import java.util.LinkedHashSet;
import java.util.Set;
import com.kingsrook.qbits.assistant.AssistantBudgets;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.model.AssistantPolicy;
@@ -31,6 +25,12 @@ import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import java.time.Clock;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
import java.util.LinkedHashSet;
import java.util.Set;
/*******************************************************************************
@@ -47,6 +47,9 @@ public class PolicyService
private final AssistantStore store;
private final Clock clock;
private BudgetService budgetService;
private ProviderService providerService;
private volatile AssistantPolicy cached;
private volatile Instant cachedAt;
@@ -210,6 +213,26 @@ public class PolicyService
** `pendingCostMicros` of a running turn not yet recorded.
*******************************************************************************/
public BudgetStatus checkBudget(EffectivePolicy policy, String userId, long pendingTokens, long pendingCostMicros) throws QException
{
if(budgetService != null && pendingTokens == 0 && pendingCostMicros == 0)
{
BudgetService.Decision decision = budgetService.evaluate(userId, providerService.resolve(policy.mainModel()), false, true);
if(decision.stopped() || !decision.notices().isEmpty())
{
BudgetStatus configured = checkConfiguredBudget(policy, userId, pendingTokens, pendingCostMicros);
return (configured.hardExceeded() ? configured : new BudgetStatus(decision.stopped(), !decision.notices().isEmpty(),
decision.stopped() ? decision.reason() : String.join(" ", decision.notices())));
}
}
return (checkConfiguredBudget(policy, userId, pendingTokens, pendingCostMicros));
}
/*******************************************************************************
** Existing config budgets remain additive and unchanged.
*******************************************************************************/
public BudgetStatus checkConfiguredBudget(EffectivePolicy policy, String userId, long pendingTokens, long pendingCostMicros) throws QException
{
AssistantBudgets b = policy.budgets();
LocalDate today = LocalDate.now(clock.withZone(ZoneOffset.UTC));
@@ -320,4 +343,14 @@ public class PolicyService
return (override != null ? override : base);
}
/*******************************************************************************
** Wire the data-driven start-turn check into the existing 429 BUDGET path.
*******************************************************************************/
public PolicyService withBudgetService(BudgetService budgets, ProviderService providers)
{
this.budgetService = budgets;
this.providerService = providers;
return (this);
}
}
@@ -215,7 +215,7 @@ public class ProfileService
/*******************************************************************************
**
*******************************************************************************/
private AssistantProfile pick(QSession session, String appName)
public AssistantProfile pick(QSession session, String appName)
{
List<AssistantProfile> enabled = load().profiles().stream().filter(p -> !Boolean.FALSE.equals(p.getEnabled())).toList();
return (enabled.stream()
@@ -0,0 +1,144 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.FutureTask;
import java.util.concurrent.TimeUnit;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.provider.HealthResult;
import com.kingsrook.qbits.assistant.provider.ModelProviderException;
import com.kingsrook.qbits.assistant.provider.ProviderSettings;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Persisted sixty-second circuit breaker; only local endpoints are probed.
*******************************************************************************/
public class ProviderHealth
{
private final AssistantRuntime runtime;
/*******************************************************************************
** Use the runtime clock for all cache and breaker boundaries.
*******************************************************************************/
public ProviderHealth(AssistantRuntime runtime)
{
this.runtime = runtime;
}
/*******************************************************************************
** Refuse an open circuit; probe unknown or stale local endpoints within two seconds.
*******************************************************************************/
public void beforeCall(ProviderService.Active active) throws QException, ModelProviderException, InterruptedException
{
if(active.providerId() == null)
{
return;
}
AssistantProvider row = runtime.getStore().getProvider(active.providerId());
if(row == null || !Boolean.TRUE.equals(row.getEnabled()))
{
throw (new ModelProviderException(ModelProviderException.Kind.AUTHENTICATION, null, "Provider is disabled."));
}
if(isDown(row))
{
throw (new ModelProviderException(ModelProviderException.Kind.OVERLOADED, null, "Provider circuit is open."));
}
Instant now = runtime.getClock().instant();
if(!"LOCAL".equals(runtime.getProviderService().location(row)) || (row.getHealthCheckedAt() != null && row.getHealthCheckedAt().plusSeconds(60).isAfter(now)))
{
return;
}
ProviderSettings settings = runtime.getProviderService().settings(row).withConnectTimeoutSeconds(2).withRequestTimeoutSeconds(2);
FutureTask<HealthResult> probe = new FutureTask<>(() -> runtime.getConfig().getProviderTypes().get(row.getProviderType()).checkHealth(settings));
Thread.ofVirtual().name("assistant-health").start(probe);
try
{
HealthResult result = probe.get(Duration.ofSeconds(2).toMillis(), TimeUnit.MILLISECONDS);
if(Boolean.FALSE.equals(result.healthy()))
{
throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, "Provider health check failed."));
}
save(row.getId(), Boolean.TRUE.equals(result.healthy()) ? "UP" : null, null);
}
catch(java.util.concurrent.ExecutionException | java.util.concurrent.TimeoutException e)
{
throw (new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "Provider health check did not complete."));
}
finally
{
probe.cancel(true);
}
}
/*******************************************************************************
** A future downUntil alone keeps the circuit open; expiry permits recovery.
*******************************************************************************/
public Boolean isDown(AssistantProvider row)
{
return (row != null && row.getDownUntil() != null && row.getDownUntil().isAfter(runtime.getClock().instant()));
}
/*******************************************************************************
** Connection errors open the breaker; model errors do not poison the endpoint.
*******************************************************************************/
public void failed(ProviderService.Active active, ModelProviderException error) throws QException
{
if(active.providerId() != null && (error.getKind() == ModelProviderException.Kind.NETWORK || error.getKind() == ModelProviderException.Kind.TIMEOUT))
{
save(active.providerId(), "DOWN", runtime.getClock().instant().plusSeconds(60));
}
}
/*******************************************************************************
** A successful completion closes the breaker.
*******************************************************************************/
public void succeeded(ProviderService.Active active) throws QException
{
if(active.providerId() != null)
{
save(active.providerId(), "UP", null);
}
}
/*******************************************************************************
** Only health columns are changed, under the requesting session.
*******************************************************************************/
private void save(Integer providerId, String state, Instant downUntil) throws QException
{
runtime.getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", providerId)
.withValue("healthState", state).withValue("healthCheckedAt", runtime.getClock().instant()).withValue("downUntil", downUntil));
runtime.getProviderService().invalidate();
}
}
@@ -17,6 +17,18 @@
package com.kingsrook.qbits.assistant.engine;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qbits.assistant.provider.ProviderSettings;
import com.kingsrook.qbits.assistant.provider.RegistryModelProvider;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.nio.charset.StandardCharsets;
@@ -31,18 +43,6 @@ import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Semaphore;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qbits.assistant.provider.ProviderSettings;
import com.kingsrook.qbits.assistant.provider.RegistryModelProvider;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -732,4 +732,88 @@ public class ProviderService
{
return (java.util.Optional.empty());
}
/*******************************************************************************
** Resolve a specific enabled registry model and provider without changing
** the default. Copy the row so the shared provider cache is never mutated.
*******************************************************************************/
public Active resolveModel(Integer modelId, boolean requiresHipaa) throws QException
{
return (resolveModel(modelId, requiresHipaa, true));
}
/*******************************************************************************
** Classifiers may use models without tool support.
*******************************************************************************/
public Active resolveModel(Integer modelId, Boolean requiresHipaa, Boolean requireTools) throws QException
{
AssistantModel model = store.listModels().stream().filter(m -> modelId.equals(m.getId()) && Boolean.TRUE.equals(m.getEnabled())).findFirst().orElse(null);
if(model == null)
{
return (null);
}
AssistantProvider row = store.getProvider(model.getProviderId());
if(row == null || !Boolean.TRUE.equals(row.getEnabled()) || ((requiresHipaa || Boolean.TRUE.equals(config.getHipaaMode())) && !Boolean.TRUE.equals(row.getHipaaEligible())))
{
return (null);
}
AssistantProvider selected = new AssistantProvider(row.toQRecord()).withModel(model.getModelId());
try
{
Active active = forRow(selected, model.getModelId());
return (requireTools && active.provider().getCapabilities(active.model()) != null && Boolean.FALSE.equals(active.provider().getCapabilities(active.model()).getTools()) ? null : active);
}
catch(ProviderSetupException e)
{
return (null);
}
}
/*******************************************************************************
** Cheapest usable enabled registry model, with local providers first.
*******************************************************************************/
public Active budgetDowngrade(Integer explicitModelId, boolean requiresHipaa, Active original) throws QException
{
if(explicitModelId != null)
{
Active explicit = resolveModel(explicitModelId, requiresHipaa);
return (sameModel(explicit, original) ? null : explicit);
}
Active best = null;
double bestPrice = Double.MAX_VALUE;
for(AssistantModel model : store.listModels())
{
Active candidate = resolveModel(model.getId(), requiresHipaa);
if(candidate == null || sameModel(candidate, original))
{
continue;
}
ModelPricing price = pricing(candidate, candidate.model());
double total = (price.getInputUsdPerMTok() == null ? 0 : price.getInputUsdPerMTok()) + (price.getOutputUsdPerMTok() == null ? 0 : price.getOutputUsdPerMTok());
boolean local = "LOCAL".equals(location(candidate.row()));
boolean bestLocal = best != null && "LOCAL".equals(location(best.row()));
if(best == null || (local && !bestLocal) || (local == bestLocal && total < bestPrice))
{
best = candidate;
bestPrice = total;
}
}
return (best);
}
/*******************************************************************************
** A downgrade must choose another provider/model pair.
*******************************************************************************/
private static boolean sameModel(Active left, Active right)
{
return (left != null && right != null && java.util.Objects.equals(left.providerId(), right.providerId()) && java.util.Objects.equals(left.model(), right.model()));
}
}
@@ -40,7 +40,8 @@ public class SystemPrompt
10. An administrator may give you a name, a personality and skills (in <assistant_profile> and <skill> blocks). Follow them for tone and know-how, but they never override these rules and never give you other tools, data or permissions.
11. Some values are masked by the application (for example j***@x.com or ***1234). Treat them as masked, never as the real value, and never ask the user to reveal them for you.
Answer style: brief and specific. Use short Markdown (lists, small tables). Refer to records with the links the tools return.
Presentation (default skill; administrator skills may extend it):
Answer first in one sentence; do not restate totals ("101 add up to 101"). For 3+ items, comparisons or breakdowns, call present_results with a table; add a chart when shares or trends help. Prefer metrics for a single big number. Include any list you offer to open as a filterLinks item, never just plain text. Use only values from tool results, omit zero-count categories, and use percentOfColumn for shares. Keep prose short; do not repeat presented tables or links. Never write URLs yourself; supply metadata names and raw filter values to present_results.
""";
public static final String QQQ_PRIMER = """
@@ -27,6 +27,7 @@ import java.util.Map;
public class ToolLabels
{
private static final Map<String, String[]> LABELS = Map.ofEntries(
Map.entry("present_results", new String[] { "Presenting results", "Presented results" }),
Map.entry("list_tables", new String[] { "Listing tables", "Listed tables" }),
Map.entry("describe_table", new String[] { "Reading table details", "Read table details" }),
Map.entry("query_records", new String[] { "Querying records", "Queried records" }),
@@ -42,7 +42,7 @@ import com.kingsrook.qqq.backend.core.utils.JsonUtils;
*******************************************************************************/
public class TurnClassification
{
private static final Set<String> METADATA_TOOLS = Set.of("list_tables", "describe_table", "navigate", "search_help");
private static final Set<String> METADATA_TOOLS = Set.of("list_tables", "describe_table", "navigate", "search_help", "present_results");
private final ClassificationSource source;
private final Set<String> tables = new LinkedHashSet<>();
private DataClassification classification;
@@ -65,7 +65,8 @@ public class TurnClassification
/*******************************************************************************
** Observe before appending any result to a model request. Metadata tools carry no record values.
** Observe before appending results. Metadata reads no records; presentation
** only reuses values whose originating data tools were already classified.
*******************************************************************************/
public void observe(ModelPart use, AiToolResult result)
{
@@ -17,21 +17,6 @@
package com.kingsrook.qbits.assistant.engine;
import java.io.Serializable;
import java.time.Duration;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
@@ -51,6 +36,7 @@ import com.kingsrook.qbits.assistant.provider.ModelRequest;
import com.kingsrook.qbits.assistant.provider.ModelResponse;
import com.kingsrook.qbits.assistant.provider.ModelUsage;
import com.kingsrook.qbits.assistant.provider.StreamHandle;
import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
import com.kingsrook.qqq.backend.core.actions.ai.AiRecordRef;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolCall;
@@ -64,6 +50,21 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import java.io.Serializable;
import java.time.Duration;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.Future;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
@@ -101,6 +102,7 @@ public class TurnEngine implements Runnable
private static final Set<AiToolOutcome> ERROR_OUTCOMES = Set.of(AiToolOutcome.DENIED, AiToolOutcome.NOT_FOUND, AiToolOutcome.INVALID_INPUT, AiToolOutcome.ERROR);
private final PresentResultsTool presenter;
private final AssistantRuntime runtime;
private final AssistantQBitConfig config;
private final AssistantStore store;
@@ -130,10 +132,10 @@ public class TurnEngine implements Runnable
private long costMicros = 0;
private TurnClassification classification;
private FallbackChain fallback;
private ProviderService.Active active;
private String model;
private Long capMicros;
private long monthCostAtStart = 0;
private ProviderService.Active budgetOrigin;
private volatile boolean finished = false;
@@ -146,6 +148,7 @@ public class TurnEngine implements Runnable
AssistantTurn turn, String userText, Map<String, Object> pageContext, PolicyService.EffectivePolicy policy, RunningTurn running, String softBudgetNotice)
{
this.runtime = runtime;
this.presenter = new PresentResultsTool(runtime.getQInstance());
this.config = runtime.getConfig();
this.store = runtime.getStore();
this.session = session;
@@ -192,18 +195,26 @@ public class TurnEngine implements Runnable
persistPendingEvents();
model = policy.mainModel();
TurnDataRouting.beforeResolve(runtime, classification, turn, policy.mainModel());
active = runtime.getProviderService().resolve(policy.mainModel());
ModelRouter router = new ModelRouter(runtime);
ModelRouter.Route route;
try
{
route = router.route(session, conversation, turn, userText, pageContext, policy, classification, running, deadline, text -> notice("warning", text));
}
finally
{
usage.add(router.accounting().usage());
costMicros += router.accounting().costMicros();
modelCallCount += router.accounting().calls();
}
active = route.active();
fallback = new FallbackChain(runtime, classification, turn, text -> notice("warning", text));
if(active == null)
{
throw (new ModelProviderException(ModelProviderException.Kind.AUTHENTICATION, null, "No usable assistant provider is configured."));
}
model = active.model();
capMicros = ProviderService.capMicros(active.row());
if(capMicros != null)
{
monthCostAtStart = runtime.getProviderService().monthToDateCostMicros(active.providerId());
}
budgetOrigin = active;
AssistantModelProvider provider = active.provider();
ModelCapabilities caps = provider.getCapabilities(model);
@@ -219,7 +230,7 @@ public class TurnEngine implements Runnable
String appName = runtime.appNameFor(pageContext);
ProfileService.Persona persona = runtime.getProfileService().persona(session, appName);
String profileText = ProfileService.instructions(persona, runtime.getProfileService().relevantSkills(persona.profileId(), userText, appName));
ModelRequest request = runtime.getContextBuilder().build(session, model, policy.mainEffort(), history, userText, pageContext, policy.tools(), profileText);
ModelRequest request = runtime.getContextBuilder().build(session, model, route.effort(), history, userText, pageContext, policy.tools(), profileText);
Integer maxPerRequest = active.row() == null ? null : active.row().getMaxOutputTokensPerRequest();
if(maxPerRequest != null && (request.getMaxTokens() == null || request.getMaxTokens() > maxPerRequest))
{
@@ -244,15 +255,7 @@ public class TurnEngine implements Runnable
break;
}
if(capMicros != null && monthCostAtStart + costMicros >= capMicros)
{
status = "ERROR";
errorCode = "BUDGET";
errorText = "The assistant has reached this provider's monthly cost cap. Ask an administrator.";
break;
}
ModelResponse response = callModelWithStrictFallback(provider, request);
ModelResponse response = fallback.call(request, () -> active, selected -> callModelWithStrictFallback(selected, request));
stopReason = response.getStopReason();
if(ModelResponse.STOP_CANCELLED.equals(stopReason))
@@ -264,7 +267,7 @@ public class TurnEngine implements Runnable
break;
}
PolicyService.BudgetStatus budget = runtime.getPolicyService().checkBudget(policy, userId, usage.billableTokens(), costMicros);
PolicyService.BudgetStatus budget = runtime.getPolicyService().checkConfiguredBudget(policy, userId, usage.billableTokens(), costMicros);
if(budget.hardExceeded())
{
status = "ERROR";
@@ -335,6 +338,12 @@ public class TurnEngine implements Runnable
}
}
}
catch(BudgetService.Blocked e)
{
status = "ERROR";
errorCode = "BUDGET";
errorText = e.getMessage();
}
catch(TurnDataRouting.Blocked e)
{
status = "ERROR";
@@ -343,11 +352,6 @@ public class TurnEngine implements Runnable
}
catch(ModelProviderException e)
{
if(e.getPartialUsage() != null)
{
usage.add(e.getPartialUsage());
costMicros += pricing().costMicros(e.getPartialUsage());
}
status = "ERROR";
errorCode = e.getKind() == ModelProviderException.Kind.TIMEOUT ? "TIMEOUT" : "PROVIDER";
errorText = e.getKind() == ModelProviderException.Kind.TIMEOUT
@@ -361,7 +365,7 @@ public class TurnEngine implements Runnable
errorText += " The provider said: " + UsageRecorder.errorText(e);
}
LOG.warn("Assistant model call failed", logPair("turnId", turn.getId()), logPair("kind", e.getKind()), logPair("httpStatus", e.getHttpStatus()),
logPair("errorType", hipaaSafe(e).getProviderErrorType()), logPair("errorMessage", hipaaSafe(e).getProviderMessage()), logPair("requestId", hipaaSafe(e).getRequestId()));
logPair("errorType", hipaaSafe(e.getProviderErrorType())), logPair("errorMessage", hipaaSafe(e.getProviderMessage())), logPair("requestId", hipaaSafe(e.getRequestId())));
}
catch(InterruptedException e)
{
@@ -506,23 +510,23 @@ public class TurnEngine implements Runnable
** tool schemas, retry once (and for the rest of the turn) with no strict
** tools. The kernel validates every tool input either way.
*******************************************************************************/
private ModelResponse callModelWithStrictFallback(AssistantModelProvider provider, ModelRequest request) throws QException, ModelProviderException, InterruptedException
private ModelResponse callModelWithStrictFallback(ProviderService.Active candidate, ModelRequest request) throws QException, ModelProviderException, InterruptedException
{
try
{
return (callModel(provider, request));
return (callModel(candidate, request));
}
catch(ModelProviderException e)
{
boolean anyStrict = request.getTools().stream().anyMatch(tool -> Boolean.TRUE.equals(tool.getStrict()));
if(!e.isStrictSchemaRejected() || !anyStrict)
if(!e.isStrictSchemaRejected() || !anyStrict || fallback.hasOutput())
{
throw (e);
}
LOG.warn("Assistant provider refused the strict tool schemas; retrying without strict tools", logPair("turnId", turn.getId()), logPair("model", model),
logPair("errorMessage", hipaaSafe(e).getProviderMessage()), logPair("requestId", hipaaSafe(e).getRequestId()));
logPair("errorMessage", hipaaSafe(e.getProviderMessage())), logPair("requestId", hipaaSafe(e.getRequestId())));
request.getTools().forEach(tool -> tool.setStrict(false));
return (callModel(provider, request));
return (callModel(candidate, request));
}
}
@@ -531,8 +535,49 @@ public class TurnEngine implements Runnable
/*******************************************************************************
** One streamed model call.
*******************************************************************************/
private ModelResponse callModel(AssistantModelProvider provider, ModelRequest request) throws QException, ModelProviderException, InterruptedException
private ModelResponse callModel(ProviderService.Active candidate, ModelRequest request) throws QException, ModelProviderException, InterruptedException
{
if(checkStop() != null)
{
return (new ModelResponse().withStopReason(ModelResponse.STOP_CANCELLED));
}
PolicyService.BudgetStatus configuredBudget = runtime.getPolicyService().checkConfiguredBudget(policy, userId, usage.billableTokens(), costMicros);
if(configuredBudget.hardExceeded())
{
throw (new BudgetService.Blocked(configuredBudget.message()));
}
classification.inherit(store.getConversation(userId, conversation.getId()));
BudgetService.Decision budgetDecision = fallback.budget(budgetOrigin, candidate);
budgetDecision.notices().forEach(text -> notice("warning", text));
if(budgetDecision.stopped())
{
throw (new BudgetService.Blocked(budgetDecision.reason()));
}
ProviderService.Active selected = budgetDecision.active();
if(!java.util.Objects.equals(active.providerId(), selected.providerId()) || !java.util.Objects.equals(active.model(), selected.model()))
{
active = selected;
model = active.model();
request.setModel(model);
FallbackChain.neutralHistory(request);
request.getTools().forEach(tool -> tool.setStrict(false));
ModelCapabilities capabilities = active.provider().getCapabilities(model);
if(capabilities != null && Boolean.FALSE.equals(capabilities.getParallelToolCalls()))
{
request.setParallelToolCalls(false);
}
Integer maxTokens = active.row() == null ? null : active.row().getMaxOutputTokensPerRequest();
if(maxTokens != null && (request.getMaxTokens() == null || request.getMaxTokens() > maxTokens))
{
request.setMaxTokens(maxTokens);
}
}
AssistantModelProvider provider = active.provider();
Long selectedCap = ProviderService.capMicros(active.row());
if(selectedCap != null && runtime.getProviderService().monthToDateCostMicros(active.providerId()) >= selectedCap)
{
throw (new BudgetService.Blocked("The assistant has reached this provider's monthly cost cap. Ask an administrator."));
}
ModelEventListener listener = new ModelEventListener()
{
@Override
@@ -557,8 +602,9 @@ public class TurnEngine implements Runnable
try
{
TurnDataRouting.beforeModelCall(runtime, classification, turn, active);
fallback.beforeCall(active);
modelCallCount++;
StreamHandle handle = provider.stream(request.copy(), listener);
StreamHandle handle = provider.stream(request.copy(), fallback.listener(listener));
running.setCurrentHandle(handle);
try
{
@@ -574,6 +620,11 @@ public class TurnEngine implements Runnable
catch(ModelProviderException e)
{
recordUsage(null, e, started);
if(e.getPartialUsage() != null)
{
usage.add(e.getPartialUsage());
costMicros += pricing().costMicros(e.getPartialUsage());
}
throw (e);
}
finally
@@ -641,7 +692,11 @@ public class TurnEngine implements Runnable
{
if(!running.isCancelled())
{
if(ContextBuilder.KERNEL_TOOLS.contains(call.getName()))
if(PresentResultsTool.NAME.equals(call.getName()))
{
results[index] = presenter.execute(toolContext(), call.getId(), call.getInput());
}
else if(ContextBuilder.KERNEL_TOOLS.contains(call.getName()))
{
///////////////////////////////////////////////////
// navigate / search_help: no records are read //
@@ -750,7 +805,8 @@ public class TurnEngine implements Runnable
}
Map<String, Object> data = map("messageId", assistantMessage.getId(), "toolCallId", use.getToolUseId(), "name", use.getToolName(), "outcome", outcome, "summary", summary);
if(result.getRender() != null)
Boolean presentation = PresentResultsTool.NAME.equals(use.getToolName()) && AiToolOutcome.OK.equals(result.getOutcome()) && result.getRender() != null;
if(result.getRender() != null && !presentation)
{
data.put("render", result.getRender());
}
@@ -760,7 +816,12 @@ public class TurnEngine implements Runnable
}
emit(TurnEvent.TOOL_RESULT, data);
parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, result.getRender());
parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, presentation ? null : result.getRender());
if(presentation)
{
parts.addPresentation(result.getRender());
emit(TurnEvent.PRESENTATION, map("messageId", assistantMessage.getId(), "part", result.getRender()));
}
parts.addRecordRefs(refs);
String href = result.getNavigateHref();
@@ -778,7 +839,7 @@ public class TurnEngine implements Runnable
.withUserId(userId)
.withToolCallId(use.getToolUseId())
.withToolName(use.getToolName())
.withInputJson(truncate(redactInput(use.getInput()), MAX_INPUT_JSON_CHARS))
.withInputJson(truncate(redactInput(PresentResultsTool.NAME.equals(use.getToolName()) ? (presentation ? result.getRender() : Map.of()) : use.getInput()), MAX_INPUT_JSON_CHARS))
.withOutcome(outcome)
.withPermissionCheck(result.getPermissionCheck())
.withTableName(result.getTableName())
@@ -940,6 +1001,10 @@ public class TurnEngine implements Runnable
.withValue("errorCode", errorCode)
.withValue("cancelReason", running.getCancelReason())
.withValue("model", model)
.withValue("providerId", active == null ? null : active.providerId())
.withValue("routeDecision", turn.getRouteDecision())
.withValue("routeSource", turn.getRouteSource())
.withValue("fallbackCount", turn.getFallbackCount())
.withValue("endedAt", now)
.withValue("toolCallCount", toolCallCount)
.withValue("modelCallCount", modelCallCount)
@@ -33,6 +33,7 @@ public class TurnEvent
public static final String TEXT_DELTA = "text.delta";
public static final String TOOL_START = "tool.start";
public static final String TOOL_RESULT = "tool.result";
public static final String PRESENTATION = "presentation";
public static final String UI_NAVIGATE = "ui.navigate";
public static final String NOTICE = "notice";
public static final String USAGE = "usage";
@@ -39,6 +39,7 @@ public final class UsageRecorder
public static final String PURPOSE_TURN = "TURN";
public static final String PURPOSE_TEST = "TEST";
public static final String PURPOSE_ROUTE = "ROUTE";
public static final String STATUS_OK = "OK";
public static final String STATUS_ERROR = "ERROR";
@@ -17,6 +17,26 @@
package com.kingsrook.qbits.assistant.home;
import com.kingsrook.qbits.assistant.model.AssistantTurn;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.AssistantSecurity;
import com.kingsrook.qbits.assistant.engine.BudgetService;
import com.kingsrook.qbits.assistant.engine.PolicyService;
import com.kingsrook.qbits.assistant.engine.ProviderService;
import com.kingsrook.qbits.assistant.engine.UsageRecorder;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
@@ -26,20 +46,6 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.engine.PolicyService;
import com.kingsrook.qbits.assistant.engine.ProviderService;
import com.kingsrook.qbits.assistant.engine.UsageRecorder;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
@@ -65,6 +71,10 @@ public class AssistantHomeData
private final List<QRecord> failedTools = new ArrayList<>();
private final List<QRecord> deniedTools = new ArrayList<>();
private final List<BudgetService.Snapshot> budgets = new ArrayList<>();
private final List<Totals> byProvider = new ArrayList<>();
private Long fallbackCount = 0L;
private Boolean hipaaMode;
private Integer hipaaRetentionDays;
private LocalDate today;
@@ -137,6 +147,14 @@ public class AssistantHomeData
AssistantHomeData data = new AssistantHomeData();
data.hipaaMode = runtime.getConfig().getHipaaMode();
data.hipaaRetentionDays = runtime.getConfig().getHipaaRetentionDays();
for(AssistantBudget budget : runtime.getStore().budgets())
{
BudgetService.Snapshot snapshot = runtime.getBudgetService().snapshot(budget, AssistantSecurity.userId(QContext.getQSession()));
List<AssistantBudgetOverride> overrides = runtime.getStore().budgetOverrides(budget.getId()).stream()
.filter(o -> o.getExpiresAt() != null && o.getExpiresAt().isAfter(runtime.getClock().instant()))
.filter(o -> o.getUnlimitedUntil() == null || o.getUnlimitedUntil().isAfter(runtime.getClock().instant())).toList();
data.budgets.add(new BudgetService.Snapshot(budget, snapshot.scopeKey(), snapshot.periodStart(), snapshot.used(), snapshot.limit(), snapshot.percent(), snapshot.state(), overrides));
}
data.today = LocalDate.now(runtime.getClock().withZone(ZoneOffset.UTC));
LocalDate firstDay = data.today.minusDays(DAYS - 1);
LocalDate monthStart = data.today.withDayOfMonth(1);
@@ -163,6 +181,7 @@ public class AssistantHomeData
.withOrderBy(new QFilterOrderBy("id", false)), max);
data.truncated |= usage.size() >= max;
Map<String, Totals> providerTotals = new LinkedHashMap<>();
Map<String, Totals> users = new LinkedHashMap<>();
Map<String, Totals> models = new LinkedHashMap<>();
Map<Integer, Long> providerMonth = new LinkedHashMap<>();
@@ -187,6 +206,9 @@ public class AssistantHomeData
day.tokensIn += in;
day.tokensOut += out;
day.costMicros += cost;
add(users, row.getValueString("userId") == null ? "(none)" : row.getValueString("userId"), in, out, cost);
add(models, row.getValueString("model") == null ? "(unknown)" : row.getValueString("model"), in, out, cost);
add(providerTotals, row.getValueString("providerId") == null ? "Code-registered provider" : "Provider #" + row.getValueString("providerId"), in, out, cost);
}
if(!at.isBefore(weekAt) && turn)
@@ -204,8 +226,6 @@ public class AssistantHomeData
{
data.monthCostMicros += cost;
data.monthRequests++;
add(users, row.getValueString("userId") == null ? "(none)" : row.getValueString("userId"), in, out, cost);
add(models, row.getValueString("model") == null ? "(unknown)" : row.getValueString("model"), in, out, cost);
Integer providerId = row.getValueInteger("providerId");
if(providerId != null)
{
@@ -213,6 +233,7 @@ public class AssistantHomeData
}
}
}
data.byProvider.addAll(top(providerTotals));
data.byUser.addAll(top(users));
data.byModel.addAll(top(models));
@@ -220,6 +241,16 @@ public class AssistantHomeData
// conversations started per day //
////////////////////////////////////////
Instant dayOneAt = firstDay.atStartOfDay().toInstant(ZoneOffset.UTC);
List<QRecord> routedTurns = runtime.getStore().queryRecords(AssistantTurn.TABLE_NAME, new QQueryFilter()
.withCriteria("startedAt", QCriteriaOperator.GREATER_THAN_OR_EQUALS, dayOneAt), max);
data.truncated |= routedTurns.size() >= max;
for(QRecord row : routedTurns)
{
if(row.getValueInteger("fallbackCount") != null)
{
data.fallbackCount += row.getValueInteger("fallbackCount");
}
}
List<QRecord> conversations = runtime.getStore().queryRecords(AssistantConversation.TABLE_NAME, new QQueryFilter()
.withCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, dayOneAt), max);
data.truncated |= conversations.size() >= max;
@@ -517,4 +548,34 @@ public class AssistantHomeData
{
return (hipaaRetentionDays);
}
/*******************************************************************************
** Current enabled budgets and active overrides.
*******************************************************************************/
public List<BudgetService.Snapshot> getBudgets()
{
return (budgets);
}
/*******************************************************************************
** Provider usage over the same 30-day window as users and models.
*******************************************************************************/
public List<Totals> getByProvider()
{
return (byProvider);
}
/*******************************************************************************
** Provider switches in the displayed period, including recovered failures.
*******************************************************************************/
public Long getFallbackCount()
{
return (fallbackCount);
}
}
@@ -17,6 +17,16 @@
package com.kingsrook.qbits.assistant.home;
import com.kingsrook.qbits.assistant.engine.BudgetService;
import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.text.NumberFormat;
@@ -28,13 +38,6 @@ import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
@@ -90,7 +93,9 @@ public final class AssistantHomeView
status(html, data, prefix);
providers(html, data, prefix);
html.append("<div style=\"").append(GRID).append("\">");
budgets(html, data, prefix);
usageByDay(html, data, prefix);
byProvider(html, data);
costs(html, data, prefix);
byUser(html, data);
tools(html, data, prefix);
@@ -188,6 +193,11 @@ public final class AssistantHomeView
}
html.append("</div><div style=\"").append(CELLS).append("\">");
cell(html, "Type", nvl(p.typeLabel(), "Not set"));
cell(html, "Health", nvl(row.getHealthState(), "Unknown"));
if(row.getDownUntil() != null)
{
cell(html, "Down until", when(row.getDownUntil()));
}
cell(html, "Kind", AssistantRecordScreens.providerKindLabel(p.kind()));
cell(html, "Location", "LOCAL".equals(p.location()) ? "Local" : "Cloud");
if(Boolean.TRUE.equals(row.getHipaaEligible()))
@@ -232,14 +242,13 @@ public final class AssistantHomeView
*******************************************************************************/
private static void usageByDay(StringBuilder html, AssistantHomeData data, String prefix)
{
open(html, "Usage, last " + AssistantHomeData.DAYS + " days");
open(html, "Usage over time, last " + AssistantHomeData.DAYS + " days");
Map<LocalDate, AssistantHomeData.DayUsage> days = data.getDays();
long conversations = 0;
long requests = 0;
long tokensIn = 0;
long tokensOut = 0;
long cost = 0;
long max = 1;
for(AssistantHomeData.DayUsage day : days.values())
{
conversations += day.conversations;
@@ -247,7 +256,6 @@ public final class AssistantHomeView
tokensIn += day.tokensIn;
tokensOut += day.tokensOut;
cost += day.costMicros;
max = Math.max(max, day.requests);
}
if(requests == 0 && conversations == 0)
@@ -256,16 +264,16 @@ public final class AssistantHomeView
}
else
{
html.append("<div role=\"img\" aria-label=\"Model requests per day, last ").append(days.size()).append(" days\" style=\"display:flex;align-items:flex-end;gap:2px;height:3rem;margin:0 0 .25rem;\">");
long maxCost = days.values().stream().mapToLong(d -> d.costMicros).max().orElse(1);
long maxTokens = days.values().stream().mapToLong(d -> d.tokensIn + d.tokensOut).max().orElse(1);
for(Map.Entry<LocalDate, AssistantHomeData.DayUsage> entry : days.entrySet())
{
AssistantHomeData.DayUsage day = entry.getValue();
int pct = (int) Math.max(day.requests == 0 ? 0 : 4, Math.round(day.requests * 100.0 / max));
html.append("<span title=\"").append(esc(DAY.format(entry.getKey()) + ": " + day.requests + " requests")).append("\" style=\"flex:1 1 0;min-width:0;height:").append(pct)
.append("%;background:").append(day.errors > 0 ? AMBER : BAR).append(";border-radius:2px 2px 0 0;\"></span>");
html.append("<div style=\"").append(LINE).append("\"><span>").append(esc(DAY.format(entry.getKey()))).append("</span><span>")
.append(esc(money(day.costMicros) + ", " + number(day.tokensIn + day.tokensOut) + " tokens")).append("</span></div>");
bar(html, maxCost == 0 ? 0 : 100.0 * day.costMicros / maxCost, BAR, "Daily cost");
bar(html, maxTokens == 0 ? 0 : 100.0 * (day.tokensIn + day.tokensOut) / maxTokens, GREY, "Daily tokens");
}
html.append("</div><div style=\"display:flex;justify-content:space-between;font-size:.75rem;margin:0 0 .75rem;").append(MUTED).append("\"><span>")
.append(esc(DAY.format(days.keySet().iterator().next()))).append("</span><span>").append(esc(DAY.format(data.getToday()))).append("</span></div>");
html.append("<dl style=\"").append(LIST).append("\">");
row(html, "Conversations", number(conversations));
@@ -316,9 +324,10 @@ public final class AssistantHomeView
}
}
html.append("</dl>");
html.append("<h4>By model, last 30 days</h4>");
if(data.getByModel().isEmpty())
{
empty(html, "No model calls this month.");
empty(html, "No model calls in the last 30 days.");
}
else
{
@@ -342,10 +351,10 @@ public final class AssistantHomeView
*******************************************************************************/
private static void byUser(StringBuilder html, AssistantHomeData data)
{
open(html, "By user, this month");
open(html, "By user, last 30 days");
if(data.getByUser().isEmpty())
{
empty(html, "No one has used the assistant this month.");
empty(html, "No one has used the assistant in the last 30 days.");
}
for(AssistantHomeData.Totals t : data.getByUser())
{
@@ -419,6 +428,7 @@ public final class AssistantHomeView
private static void failures(StringBuilder html, AssistantHomeData data, String prefix)
{
open(html, "Recent failures");
html.append("<p>").append(esc(number(data.getFallbackCount()) + " fallbacks in the last " + AssistantHomeData.DAYS + " days.")).append("</p>");
if(data.getFailedCalls().isEmpty() && data.getFailedTools().isEmpty())
{
empty(html, "No failed model calls or tool calls in the last " + AssistantHomeData.DAYS + " days.");
@@ -668,4 +678,65 @@ public final class AssistantHomeView
return (out.toString());
}
/*******************************************************************************
** Enabled budgets with their effective allowance and active overrides.
*******************************************************************************/
private static void budgets(StringBuilder html, AssistantHomeData data, String prefix)
{
open(html, "Budgets");
if(data.getBudgets().isEmpty())
{
empty(html, "No enabled data-driven budgets. Configured policy and provider limits still apply.");
}
for(BudgetService.Snapshot snapshot : data.getBudgets())
{
AssistantBudget budget = snapshot.budget();
html.append("<div style=\"").append(ROW).append("\"><strong>")
.append(recordLink(prefix, AssistantBudget.TABLE_NAME, budget.getId(), budget.getName())).append("</strong>");
html.append("<p>").append(esc(snapshot.scopeKey() + " · " + budget.getPeriod() + " (UTC) · " + budget.getMetric())).append("</p>");
html.append("<p>").append(esc(snapshot.used().stripTrailingZeros().toPlainString() + " / "
+ (snapshot.limit() == null ? "Unlimited" : snapshot.limit().stripTrailingZeros().toPlainString()) + " · " + snapshot.state())).append("</p>");
bar(html, snapshot.percent().doubleValue(), snapshot.percent().intValue() >= 100 ? RED
: snapshot.percent().intValue() >= budget.getWarnPercent() ? AMBER : "currentColor", "Budget used " + snapshot.percent() + "%");
for(AssistantBudgetOverride override : snapshot.overrides())
{
html.append("<p>").append(esc("Override: " + (override.getExtraValue() == null ? "Unlimited until " + override.getUnlimitedUntil() : "+" + override.getExtraValue())
+ " · " + nvl(override.getScopeValue(), "Whole scope") + " · " + override.getReason() + " · expires " + override.getExpiresAt())).append("</p>");
}
html.append("</div>");
}
close(html);
}
/*******************************************************************************
** A fluid semantic bar; values are clamped and text remains readable.
*******************************************************************************/
private static void bar(StringBuilder html, double percent, String color, String label)
{
html.append("<div aria-label=\"").append(esc(label)).append("\" style=\"background:rgba(127,127,127,.15);margin:.2rem 0;height:.35rem;\"><div style=\"height:100%;width:")
.append(Math.max(0, Math.min(100, percent))).append("%;background:").append(color).append(";\"></div></div>");
}
/*******************************************************************************
** Provider breakdown over the same window as users, models and tools.
*******************************************************************************/
private static void byProvider(StringBuilder html, AssistantHomeData data)
{
open(html, "By provider, last 30 days");
if(data.getByProvider().isEmpty())
{
empty(html, "No provider usage in the last 30 days.");
}
for(AssistantHomeData.Totals total : data.getByProvider())
{
html.append("<div style=\"").append(LINE).append("\"><span>").append(esc(total.key)).append("</span><span>")
.append(esc(money(total.costMicros) + ", " + number(total.tokensIn + total.tokensOut) + " tokens")).append("</span></div>");
}
close(html);
}
}
@@ -0,0 +1,208 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.metadata;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.AssistantSecurity;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
import java.io.Serializable;
import java.math.BigDecimal;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
/*******************************************************************************
** Validates budgets and attributes admin overrides to the current session.
*******************************************************************************/
public class AssistantBudgetCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Validate the merged row so partial updates cannot bypass invariants.
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean isPreview, Optional<List<QRecord>> oldRecordList) throws QException
{
Optional<Map<Serializable, QRecord>> old = oldRecordListToMap("id", oldRecordList);
for(QRecord record : records)
{
QRecord merged = new QRecord();
old.map(m -> m.get(record.getValue("id"))).ifPresent(o -> merged.setValues(new LinkedHashMap<>(o.getValues())));
record.getValues().forEach(merged::setValue);
if(merged.getValue("budgetId") != null)
{
checkOverride(record, merged, input instanceof InsertInput);
}
else
{
checkBudget(record, merged);
}
}
return (records);
}
/*******************************************************************************
** Defaults and scope-specific validation.
*******************************************************************************/
private static void checkBudget(QRecord record, QRecord merged) throws QException
{
defaults(record, merged, "enabled", true);
defaults(record, merged, "warnPercent", 80);
defaults(record, merged, "hardCeilingPercent", 120);
defaults(record, merged, "atLimitAction", "DOWNGRADE");
String scope = merged.getValueString("scopeType");
String value = merged.getValueString("scopeValue");
choice(record, merged, "scopeType", List.of("INSTANCE", "USER", "ROLE", "PROVIDER", "MODEL"));
choice(record, merged, "period", List.of("DAY", "MONTH"));
choice(record, merged, "metric", List.of("COST_USD", "TOKENS"));
choice(record, merged, "atLimitAction", List.of("DOWNGRADE", "STOP"));
if("INSTANCE".equals(scope) ? value != null && !value.isBlank() : value == null || value.isBlank())
{
error(record, "Scope value must be empty for INSTANCE and present for every other scope.");
}
AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
if("PROVIDER".equals(scope) || "MODEL".equals(scope))
{
try
{
Integer id = Integer.valueOf(value);
boolean exists = "PROVIDER".equals(scope) ? runtime.getStore().getProvider(id) != null : runtime.getStore().listModels().stream().anyMatch(m -> id.equals(m.getId()));
if(!exists)
{
error(record, "Scope value must reference an existing provider or model registry row.");
}
}
catch(NumberFormatException e)
{
error(record, "Scope value must be a numeric record id.");
}
}
Integer modelId = merged.getValueInteger("downgradeModelId");
if(modelId != null && runtime.getStore().listModels().stream().noneMatch(m -> modelId.equals(m.getId())))
{
error(record, "Downgrade model must reference an existing model registry row.");
}
BigDecimal limit = merged.getValueBigDecimal("limitValue");
if(limit == null || limit.signum() <= 0 || ("TOKENS".equals(merged.getValueString("metric")) && limit.stripTrailingZeros().scale() > 0))
{
error(record, "Limit must be positive; token limits must be whole numbers.");
}
int warn = merged.getValueInteger("warnPercent");
if(warn < 0 || warn >= 100 || merged.getValueInteger("hardCeilingPercent") < 100)
{
error(record, "Warning percent must be 0 through 99 and hard ceiling must be at least 100.");
}
}
/*******************************************************************************
** Only admins grant overrides; never accept a supplied grantor.
*******************************************************************************/
private static void checkOverride(QRecord record, QRecord merged, boolean insert) throws QException
{
if(!QContext.getQSession().hasPermission(AssistantQBitConfig.ADMIN_PERMISSION))
{
error(record, "Only assistant.admin may grant budget overrides.");
}
if(insert)
{
record.setValue("grantedBy", AssistantSecurity.userId(QContext.getQSession()));
}
else
{
record.getValues().remove("grantedBy");
}
AssistantBudgetOverride override = new AssistantBudgetOverride(merged);
AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
AssistantBudget budget = runtime.getStore().budget(override.getBudgetId());
if(budget == null)
{
error(record, "Budget must reference an existing budget.");
}
if(override.getReason() == null || override.getReason().isBlank())
{
error(record, "A reason is required.");
}
if(override.getExpiresAt() == null)
{
error(record, "An expiry is required.");
}
if((override.getExtraValue() == null) == (override.getUnlimitedUntil() == null))
{
error(record, "Set exactly one of extra value or unlimited until.");
}
if(override.getExtraValue() != null && (override.getExtraValue().signum() <= 0 || (budget != null && "TOKENS".equals(budget.getMetric()) && override.getExtraValue().stripTrailingZeros().scale() > 0)))
{
error(record, "Extra value must be positive; token values must be whole numbers.");
}
if(override.getUnlimitedUntil() != null && override.getExpiresAt() != null && override.getUnlimitedUntil().isAfter(override.getExpiresAt()))
{
error(record, "Unlimited until cannot be after expiry.");
}
}
/*******************************************************************************
** Fill an omitted setting.
*******************************************************************************/
private static void defaults(QRecord record, QRecord merged, String field, Serializable value)
{
if(merged.getValue(field) == null)
{
merged.setValue(field, value);
record.setValue(field, value);
}
}
/*******************************************************************************
** Validate a possible value even for non-UI callers.
*******************************************************************************/
private static void choice(QRecord record, QRecord merged, String field, List<String> choices)
{
if(merged.getValueString(field) == null || !choices.contains(merged.getValueString(field)))
{
error(record, "Invalid " + field + ".");
}
}
/*******************************************************************************
** Safe validation message.
*******************************************************************************/
private static void error(QRecord record, String message)
{
record.addError(new BadInputStatusMessage(message));
}
}
@@ -17,11 +17,13 @@
package com.kingsrook.qbits.assistant.metadata;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.home.AssistantAccessWidget;
import com.kingsrook.qbits.assistant.home.AssistantHome;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
@@ -65,6 +67,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import java.util.ArrayList;
import java.util.List;
/*******************************************************************************
@@ -106,7 +110,7 @@ public class AssistantMetaDataBuilder
** calls on a conversation; model calls on a provider) or reached from the
** home. Processes are table and record actions, never nav entries.
*******************************************************************************/
public static final List<String> NAV_TABLES = List.of(AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
public static final List<String> NAV_TABLES = List.of(AssistantBudget.TABLE_NAME, AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
private final AssistantQBitConfig config;
@@ -241,6 +245,19 @@ public class AssistantMetaDataBuilder
table.getField("resultSummary").withBehavior(ValueTooLongBehavior.PASS_THROUGH);
}
}
tables.add(readOnly(base(AssistantBudgetUsage.TABLE_NAME, "Budget Accounting", "receipt_long", AssistantBudgetUsage.class))
.withoutCapabilities(Capability.TABLE_DELETE).withUniqueKey(new UniqueKey("usageId")));
QCodeReference budgetCustomizer = new QCodeReference(AssistantBudgetCustomizer.class);
tables.add(base(AssistantBudget.TABLE_NAME, "Budgets", "savings", AssistantBudget.class)
.withRecordLabelFormat("%s").withRecordLabelFields("name")
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, budgetCustomizer)
.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, budgetCustomizer));
tables.add(base(AssistantBudgetOverride.TABLE_NAME, "Budget Override", "approval", AssistantBudgetOverride.class)
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, budgetCustomizer)
.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, budgetCustomizer));
tables.add(readOnly(base(AssistantBudgetEvent.TABLE_NAME, "Budget Event", "notifications", AssistantBudgetEvent.class))
.withoutCapabilities(Capability.TABLE_DELETE)
.withUniqueKey(new UniqueKey("budgetId", "scopeKey", "periodStart", "state")));
tables.forEach(AssistantRecordScreens::apply);
return (tables);
}
@@ -92,6 +92,15 @@ public class AssistantProfileCustomizer implements TableCustomizerInterface
*******************************************************************************/
private static void checkProfile(QRecord record, QRecord merged)
{
String routingMode = merged.getValueString("routingMode");
if(routingMode == null)
{
record.setValue("routingMode", "OFF");
}
else if(!List.of("OFF", "HEURISTIC", "CLASSIFIER").contains(routingMode))
{
record.addError(new BadInputStatusMessage("Routing mode must be OFF, HEURISTIC or CLASSIFIER."));
}
String personality = merged.getValueString("personality");
if(personality != null && personality.length() > ProfileService.MAX_PERSONALITY_CHARS)
{
@@ -17,15 +17,14 @@
package com.kingsrook.qbits.assistant.metadata;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
import com.kingsrook.qbits.assistant.AssistantModelRouting;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.access.DataAccessMode;
import com.kingsrook.qbits.assistant.engine.ProfileService;
import com.kingsrook.qbits.assistant.engine.ProviderService;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
@@ -65,6 +64,10 @@ import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleVal
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
/*******************************************************************************
@@ -124,6 +127,13 @@ public final class AssistantRecordScreens
}
return (List.of(
QPossibleValueSource.newForTable(AssistantBudget.TABLE_NAME),
QPossibleValueSource.newForTable(AssistantModel.TABLE_NAME).withName("assistant_registry_model"),
QPossibleValueSource.newForTable(AssistantModel.TABLE_NAME).withName("assistant_budget_model"),
budgetEnum("scopeType", "INSTANCE", "USER", "ROLE", "PROVIDER", "MODEL"),
budgetEnum("period", "DAY", "MONTH"),
budgetEnum("metric", "COST_USD", "TOKENS"),
budgetEnum("atLimitAction", "DOWNGRADE", "STOP"),
new QPossibleValueSource().withName(EFFORT_PVS_NAME).withType(QPossibleValueSourceType.ENUM)
.withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY).withEnumValues(efforts),
QPossibleValueSource.newForTable(AssistantTurn.TABLE_NAME),
@@ -158,6 +168,12 @@ public final class AssistantRecordScreens
{
switch(table.getName())
{
case AssistantBudget.TABLE_NAME -> budget(table);
case AssistantBudgetOverride.TABLE_NAME, AssistantBudgetEvent.TABLE_NAME ->
{
link(table, "budgetId", AssistantBudget.TABLE_NAME, "Budget");
sections(table);
}
case AssistantConversation.TABLE_NAME -> conversation(table);
case AssistantMessage.TABLE_NAME -> message(table);
case AssistantTurn.TABLE_NAME -> turn(table);
@@ -219,6 +235,12 @@ public final class AssistantRecordScreens
}
}
QTableMetaData budget = qInstance.getTable(AssistantBudget.TABLE_NAME);
if(budget != null)
{
childList(qInstance, budget, AssistantBudgetOverride.TABLE_NAME, "Overrides", "approval", List.of("scopeValue", "extraValue", "unlimitedUntil", "reason", "grantedBy", "expiresAt"), "budgetId", "budget");
childList(qInstance, budget, AssistantBudgetEvent.TABLE_NAME, "Events", "notifications", List.of("scopeKey", "periodStart", "state", "usedValue", "limitValue", "userId", "createDate"), "budgetId", "budget");
}
QTableMetaData provider = qInstance.getTable(AssistantProvider.TABLE_NAME);
if(provider != null)
{
@@ -268,6 +290,7 @@ public final class AssistantRecordScreens
.withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY).withEnumValues(modelValues),
enumSource("assistant_provider_kind", kinds),
enumSource("assistant_provider_location", List.of(new QPossibleValue<>("LOCAL", "Local"), new QPossibleValue<>("CLOUD", "Cloud"))),
enumSource("assistant_routing_mode", List.of(new QPossibleValue<>("OFF", "Off"), new QPossibleValue<>("HEURISTIC", "Heuristic"), new QPossibleValue<>("CLASSIFIER", "Classifier"))),
enumSource("assistant_model_source", List.of(new QPossibleValue<>("DISCOVERED", "Discovered"), new QPossibleValue<>("MANUAL", "Manual"))),
QPossibleValueSource.newForTable(AssistantProvider.TABLE_NAME)));
}
@@ -344,6 +367,7 @@ public final class AssistantRecordScreens
section("connection", "Connection", "key", Tier.T2, "apiKeyEnvName", ProviderService.KEY_STATUS_FIELD, "baseUrl", "tlsCaBundle", "cloudProject"),
section("limits", "Limits", "speed", Tier.T2, "monthlyCostCapUsd", "maxOutputTokensPerRequest", "maxConcurrentRequests"),
section("registry", "Provider settings", "settings", Tier.T2, "kind", "location", "hipaaEligible", "region", "apiVersion", "connectTimeoutSeconds", "requestTimeoutSeconds"),
section("health", "Health", "network_check", Tier.T2, "healthState", "healthCheckedAt", "downUntil"),
section("discovery", "Last discovery", "memory", Tier.T2, "lastDiscoveryAt", "lastDiscoveryStatus", "lastDiscoveryError"),
section("lastTest", "Last connection test", "network_check", Tier.T2, "lastTestAt", "lastTestStatus", "lastTestLatencyMs", "lastTestModel", "lastTestError"));
}
@@ -385,7 +409,7 @@ public final class AssistantRecordScreens
micros(table, "costMicros");
counts(table, "inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "latencyMs");
table.getField("latencyMs").withLabel("Latency (ms)");
help(table, "purpose", "TURN: a step of an answer. TEST: a Test connection call.");
help(table, "purpose", "TURN: a step of an answer. ROUTE: a routing classifier call. TEST: a Test connection call.");
sections(table,
section("call", "Model Call", "receipt_long", Tier.T1, "id", "purpose", "status", "model", "providerId", "providerType", "userId", "conversationId", "turnId"),
section("result", "Result", "fact_check", Tier.T2, "latencyMs", "errorKind", "httpStatus", "requestId"),
@@ -446,11 +470,13 @@ public final class AssistantRecordScreens
*******************************************************************************/
private static void turn(QTableMetaData table)
{
link(table, "providerId", AssistantProvider.TABLE_NAME, "Final provider");
link(table, "userMessageId", AssistantMessage.TABLE_NAME, "User Message");
link(table, "assistantMessageId", AssistantMessage.TABLE_NAME, "Assistant Message");
micros(table, "costMicros");
counts(table, "inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "toolCallCount", "modelCallCount");
sections(table,
section("routing", "Routing", "route", Tier.T2, "routeDecision", "routeSource", "fallbackCount", "providerId"),
section("turn", "Turn", "sync_alt", Tier.T1, "id", "conversationId", "status", "model", "startedAt", "endedAt", "userMessageId", "assistantMessageId", "userId"),
section("outcome", "Outcome", "flag", Tier.T2, "stopReason", "cancelReason", "errorCode"),
section("usage", "Usage", "query_stats", Tier.T2, "inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens", "costMicros", "toolCallCount", "modelCallCount"),
@@ -499,6 +525,10 @@ public final class AssistantRecordScreens
help(table, "name", "An internal name for this profile.");
help(table, "displayName", "The assistant's name in the panel header and its messages, e.g. Ada.");
help(table, "icon", "A Material icon name for the panel, e.g. auto_awesome, support_agent, school.");
pvs(table, "routingMode", "assistant_routing_mode", "Routing mode");
pvs(table, "simpleModelId", "assistant_registry_model", "Simple model");
pvs(table, "complexModelId", "assistant_registry_model", "Complex model");
pvs(table, "classifierModelId", "assistant_registry_model", "Classifier model");
help(table, "personality", "Tone and style instructions, at most " + ProfileService.MAX_PERSONALITY_CHARS + " characters, e.g. \"Friendly and brief. Use the user's first name.\" Added to the assistant's instructions; it never changes its rules, tools or what the user may do.");
help(table, "greeting", "Shown when a new conversation opens.");
help(table, "starterPrompts", "Suggested first questions, one per line (at most " + ProfileService.MAX_STARTER_PROMPTS + ").");
@@ -507,6 +537,7 @@ public final class AssistantRecordScreens
help(table, "permissionName", "Optional: use this profile only for users who hold this permission (a role).");
sections(table,
section("profile", "Profile", "face", Tier.T1, "id", "name", "displayName", "icon", "enabled", "isDefault"),
section("routing", "Model routing", "route", Tier.T2, "routingMode", "simpleModelId", "complexModelId", "classifierModelId"),
section("voice", "Personality", "record_voice_over", Tier.T2, "personality", "greeting", "starterPrompts"),
section("scope", "Where it applies", "filter_alt", Tier.T2, "appName", "permissionName"));
}
@@ -840,7 +871,7 @@ public final class AssistantRecordScreens
qInstance.addWidget(ChildRecordListRenderer.widgetMetaDataBuilder(join)
.withLabel(label)
.withMaxRows(50)
.withCanAddChildRecord(AssistantModel.TABLE_NAME.equals(childTableName))
.withCanAddChildRecord(AssistantModel.TABLE_NAME.equals(childTableName) || AssistantBudgetOverride.TABLE_NAME.equals(childTableName))
.withOnlyIncludeFieldNames(shown)
.getWidgetMetaData());
@@ -857,6 +888,7 @@ public final class AssistantRecordScreens
{
link(table, "providerId", AssistantProvider.TABLE_NAME, "Provider");
pvs(table, "source", "assistant_model_source", "Source");
pvs(table, "fallbackModelId", "assistant_registry_model", "Fallback model");
table.getField("contextWindowTokens").withLabel("Context window (tokens)");
table.getField("maxOutputTokens").withLabel("Max output tokens");
table.getField("supportsReasoningEffort").withLabel("Supports reasoning effort");
@@ -864,10 +896,44 @@ public final class AssistantRecordScreens
table.getField("outputUsdPerMTok").withLabel("Output (USD per million tokens)");
table.getField("lastTestLatencyMs").withLabel("Last test latency (ms)");
sections(table,
section("model", "Model", "memory", Tier.T1, "id", "providerId", "modelId", "displayName", "enabled", "source", "discoveredAt"),
section("model", "Model", "memory", Tier.T1, "id", "providerId", "modelId", "displayName", "enabled", "source", "discoveredAt", "fallbackModelId"),
section("capabilities", "Capabilities", "settings", Tier.T2, "supportsTools", "supportsVision", "supportsStreaming", "supportsReasoningEffort", "contextWindowTokens", "maxOutputTokens"),
section("prices", "Prices", "payments", Tier.T2, "inputUsdPerMTok", "outputUsdPerMTok"),
section("lastTest", "Last connection test", "network_check", Tier.T2, "lastTestAt", "lastTestStatus", "lastTestLatencyMs", "lastTestError"));
}
/*******************************************************************************
** Budget choices use standard possible-value sources.
*******************************************************************************/
private static QPossibleValueSource budgetEnum(String field, String... values)
{
List<QPossibleValue<?>> choices = new ArrayList<>();
for(String value : values)
{
choices.add(new QPossibleValue<>(value, value));
}
return (enumSource("assistant_budget_" + field, choices));
}
/*******************************************************************************
** Budget record screen.
*******************************************************************************/
private static void budget(QTableMetaData table)
{
for(String field : List.of("scopeType", "period", "metric", "atLimitAction"))
{
table.getField(field).withPossibleValueSourceName("assistant_budget_" + field);
}
table.getField("downgradeModelId").withPossibleValueSourceName("assistant_budget_model");
help(table, "scopeValue", "Empty for INSTANCE; user id, permission name, provider row id or model registry row id for the selected scope.");
help(table, "limitValue", "USD for COST_USD, or whole tokens for TOKENS. Periods use UTC.");
help(table, "downgradeModelId", "Empty chooses the cheapest enabled eligible model, preferring LOCAL providers.");
sections(table,
section("budget", "Budget", "savings", Tier.T1, "id", "name", "enabled", "scopeType", "scopeValue", "period", "metric", "limitValue"),
section("thresholds", "Thresholds", "notifications", Tier.T2, "warnPercent", "atLimitAction", "downgradeModelId", "hardCeilingPercent", "notes"));
}
}
@@ -17,6 +17,8 @@
package com.kingsrook.qbits.assistant.metadata;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.engine.FallbackChain;
import java.io.Serializable;
import java.util.List;
import java.util.Map;
@@ -77,7 +79,7 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean isPreview, Optional<List<QRecord>> oldRecordList) throws QException
{
ProviderService service = service();
if(com.kingsrook.qbits.assistant.model.AssistantModel.TABLE_NAME.equals(tableName(input)))
if(AssistantModel.TABLE_NAME.equals(tableName(input)))
{
for(QRecord record : records)
{
@@ -93,6 +95,7 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
}
}
}
validateFallbacks(records, oldRecordList);
return (records);
}
if(service == null || !AssistantProvider.TABLE_NAME.equals(tableName(input)))
@@ -204,6 +207,7 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
{
String name = entry.getKey();
boolean system = name.equals("id") || name.equals("modifyDate") || name.startsWith("lastTest") || name.startsWith("lastDiscovery")
|| List.of("healthState", "healthCheckedAt", "downUntil").contains(name)
|| (name.equals("isDefault") && Boolean.FALSE.equals(entry.getValue()));
if(!system)
{
@@ -242,4 +246,54 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
return (runtime == null ? null : runtime.getProviderService());
}
/*******************************************************************************
** Validate the resulting graph, including all changes in this save batch.
*******************************************************************************/
private void validateFallbacks(List<QRecord> records, Optional<List<QRecord>> oldRecords) throws QException
{
AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
if(runtime == null)
{
return;
}
Map<Integer, AssistantModel> models = new java.util.LinkedHashMap<>();
for(AssistantModel model : runtime.getStore().listModels())
{
models.put(model.getId(), model);
}
Optional<Map<Serializable, QRecord>> old = oldRecordListToMap("id", oldRecords);
List<AssistantModel> proposed = new java.util.ArrayList<>();
for(QRecord record : records)
{
QRecord merged = new QRecord();
old.map(m -> m.get(record.getValue("id"))).ifPresent(o -> merged.setValues(new java.util.LinkedHashMap<>(o.getValues())));
record.getValues().forEach(merged::setValue);
AssistantModel model = new AssistantModel(merged);
proposed.add(model);
if(model.getId() != null)
{
models.put(model.getId(), model);
}
}
for(Integer i = 0; i < records.size(); i++)
{
String error = FallbackChain.validate(proposed.get(i), models);
if(error != null)
{
records.get(i).addError(new BadInputStatusMessage(error));
}
}
for(AssistantModel model : models.values())
{
String error = FallbackChain.validate(model, models, false);
if(error != null)
{
records.forEach(record -> record.addError(new BadInputStatusMessage(error)));
break;
}
}
}
}
@@ -0,0 +1,563 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.model;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import java.math.BigDecimal;
import java.time.Instant;
/*******************************************************************************
** Persisted assistant budget budget.
*******************************************************************************/
public class AssistantBudget extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_budget";
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@QField(isRequired = true, maxLength = 200)
private String name;
@QField()
private Boolean enabled;
@QField(isRequired = true, maxLength = 16)
private String scopeType;
@QField(maxLength = 200)
private String scopeValue;
@QField(isRequired = true, maxLength = 16)
private String period;
@QField(isRequired = true, maxLength = 16)
private String metric;
@QField(isRequired = true)
private BigDecimal limitValue;
@QField()
private Integer warnPercent;
@QField(maxLength = 16)
private String atLimitAction;
@QField()
private Integer downgradeModelId;
@QField()
private Integer hardCeilingPercent;
@QField(maxLength = 2000)
private String notes;
@QField(isEditable = false)
private Instant createDate;
@QField(isEditable = false)
private Instant modifyDate;
/*******************************************************************************
** Default constructor.
*******************************************************************************/
public AssistantBudget()
{
}
/*******************************************************************************
** Populate from a record.
*******************************************************************************/
public AssistantBudget(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id
*******************************************************************************/
public Integer getId()
{
return (this.id);
}
/*******************************************************************************
** Setter for id
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Fluent setter for id
*******************************************************************************/
public AssistantBudget withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for name
*******************************************************************************/
public String getName()
{
return (this.name);
}
/*******************************************************************************
** Setter for name
*******************************************************************************/
public void setName(String name)
{
this.name = name;
}
/*******************************************************************************
** Fluent setter for name
*******************************************************************************/
public AssistantBudget withName(String name)
{
this.name = name;
return (this);
}
/*******************************************************************************
** Getter for enabled
*******************************************************************************/
public Boolean getEnabled()
{
return (this.enabled);
}
/*******************************************************************************
** Setter for enabled
*******************************************************************************/
public void setEnabled(Boolean enabled)
{
this.enabled = enabled;
}
/*******************************************************************************
** Fluent setter for enabled
*******************************************************************************/
public AssistantBudget withEnabled(Boolean enabled)
{
this.enabled = enabled;
return (this);
}
/*******************************************************************************
** Getter for scopeType
*******************************************************************************/
public String getScopeType()
{
return (this.scopeType);
}
/*******************************************************************************
** Setter for scopeType
*******************************************************************************/
public void setScopeType(String scopeType)
{
this.scopeType = scopeType;
}
/*******************************************************************************
** Fluent setter for scopeType
*******************************************************************************/
public AssistantBudget withScopeType(String scopeType)
{
this.scopeType = scopeType;
return (this);
}
/*******************************************************************************
** Getter for scopeValue
*******************************************************************************/
public String getScopeValue()
{
return (this.scopeValue);
}
/*******************************************************************************
** Setter for scopeValue
*******************************************************************************/
public void setScopeValue(String scopeValue)
{
this.scopeValue = scopeValue;
}
/*******************************************************************************
** Fluent setter for scopeValue
*******************************************************************************/
public AssistantBudget withScopeValue(String scopeValue)
{
this.scopeValue = scopeValue;
return (this);
}
/*******************************************************************************
** Getter for period
*******************************************************************************/
public String getPeriod()
{
return (this.period);
}
/*******************************************************************************
** Setter for period
*******************************************************************************/
public void setPeriod(String period)
{
this.period = period;
}
/*******************************************************************************
** Fluent setter for period
*******************************************************************************/
public AssistantBudget withPeriod(String period)
{
this.period = period;
return (this);
}
/*******************************************************************************
** Getter for metric
*******************************************************************************/
public String getMetric()
{
return (this.metric);
}
/*******************************************************************************
** Setter for metric
*******************************************************************************/
public void setMetric(String metric)
{
this.metric = metric;
}
/*******************************************************************************
** Fluent setter for metric
*******************************************************************************/
public AssistantBudget withMetric(String metric)
{
this.metric = metric;
return (this);
}
/*******************************************************************************
** Getter for limitValue
*******************************************************************************/
public BigDecimal getLimitValue()
{
return (this.limitValue);
}
/*******************************************************************************
** Setter for limitValue
*******************************************************************************/
public void setLimitValue(BigDecimal limitValue)
{
this.limitValue = limitValue;
}
/*******************************************************************************
** Fluent setter for limitValue
*******************************************************************************/
public AssistantBudget withLimitValue(BigDecimal limitValue)
{
this.limitValue = limitValue;
return (this);
}
/*******************************************************************************
** Getter for warnPercent
*******************************************************************************/
public Integer getWarnPercent()
{
return (this.warnPercent);
}
/*******************************************************************************
** Setter for warnPercent
*******************************************************************************/
public void setWarnPercent(Integer warnPercent)
{
this.warnPercent = warnPercent;
}
/*******************************************************************************
** Fluent setter for warnPercent
*******************************************************************************/
public AssistantBudget withWarnPercent(Integer warnPercent)
{
this.warnPercent = warnPercent;
return (this);
}
/*******************************************************************************
** Getter for atLimitAction
*******************************************************************************/
public String getAtLimitAction()
{
return (this.atLimitAction);
}
/*******************************************************************************
** Setter for atLimitAction
*******************************************************************************/
public void setAtLimitAction(String atLimitAction)
{
this.atLimitAction = atLimitAction;
}
/*******************************************************************************
** Fluent setter for atLimitAction
*******************************************************************************/
public AssistantBudget withAtLimitAction(String atLimitAction)
{
this.atLimitAction = atLimitAction;
return (this);
}
/*******************************************************************************
** Getter for downgradeModelId
*******************************************************************************/
public Integer getDowngradeModelId()
{
return (this.downgradeModelId);
}
/*******************************************************************************
** Setter for downgradeModelId
*******************************************************************************/
public void setDowngradeModelId(Integer downgradeModelId)
{
this.downgradeModelId = downgradeModelId;
}
/*******************************************************************************
** Fluent setter for downgradeModelId
*******************************************************************************/
public AssistantBudget withDowngradeModelId(Integer downgradeModelId)
{
this.downgradeModelId = downgradeModelId;
return (this);
}
/*******************************************************************************
** Getter for hardCeilingPercent
*******************************************************************************/
public Integer getHardCeilingPercent()
{
return (this.hardCeilingPercent);
}
/*******************************************************************************
** Setter for hardCeilingPercent
*******************************************************************************/
public void setHardCeilingPercent(Integer hardCeilingPercent)
{
this.hardCeilingPercent = hardCeilingPercent;
}
/*******************************************************************************
** Fluent setter for hardCeilingPercent
*******************************************************************************/
public AssistantBudget withHardCeilingPercent(Integer hardCeilingPercent)
{
this.hardCeilingPercent = hardCeilingPercent;
return (this);
}
/*******************************************************************************
** Getter for notes
*******************************************************************************/
public String getNotes()
{
return (this.notes);
}
/*******************************************************************************
** Setter for notes
*******************************************************************************/
public void setNotes(String notes)
{
this.notes = notes;
}
/*******************************************************************************
** Fluent setter for notes
*******************************************************************************/
public AssistantBudget withNotes(String notes)
{
this.notes = notes;
return (this);
}
/*******************************************************************************
** Getter for createDate
*******************************************************************************/
public Instant getCreateDate()
{
return (this.createDate);
}
/*******************************************************************************
** Setter for createDate
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Fluent setter for createDate
*******************************************************************************/
public AssistantBudget withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate
*******************************************************************************/
public Instant getModifyDate()
{
return (this.modifyDate);
}
/*******************************************************************************
** Setter for modifyDate
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Fluent setter for modifyDate
*******************************************************************************/
public AssistantBudget withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
}
@@ -0,0 +1,359 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.model;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import java.math.BigDecimal;
import java.time.Instant;
/*******************************************************************************
** Persisted assistant budget budget event.
*******************************************************************************/
public class AssistantBudgetEvent extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_budget_event";
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@QField(isRequired = true)
private Integer budgetId;
@QField(isRequired = true, maxLength = 400)
private String scopeKey;
@QField(isRequired = true)
private Instant periodStart;
@QField(isRequired = true, maxLength = 16)
private String state;
@QField()
private BigDecimal usedValue;
@QField()
private BigDecimal limitValue;
@QField(maxLength = 200)
private String userId;
@QField(isEditable = false)
private Instant createDate;
/*******************************************************************************
** Default constructor.
*******************************************************************************/
public AssistantBudgetEvent()
{
}
/*******************************************************************************
** Populate from a record.
*******************************************************************************/
public AssistantBudgetEvent(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id
*******************************************************************************/
public Integer getId()
{
return (this.id);
}
/*******************************************************************************
** Setter for id
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Fluent setter for id
*******************************************************************************/
public AssistantBudgetEvent withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for budgetId
*******************************************************************************/
public Integer getBudgetId()
{
return (this.budgetId);
}
/*******************************************************************************
** Setter for budgetId
*******************************************************************************/
public void setBudgetId(Integer budgetId)
{
this.budgetId = budgetId;
}
/*******************************************************************************
** Fluent setter for budgetId
*******************************************************************************/
public AssistantBudgetEvent withBudgetId(Integer budgetId)
{
this.budgetId = budgetId;
return (this);
}
/*******************************************************************************
** Getter for scopeKey
*******************************************************************************/
public String getScopeKey()
{
return (this.scopeKey);
}
/*******************************************************************************
** Setter for scopeKey
*******************************************************************************/
public void setScopeKey(String scopeKey)
{
this.scopeKey = scopeKey;
}
/*******************************************************************************
** Fluent setter for scopeKey
*******************************************************************************/
public AssistantBudgetEvent withScopeKey(String scopeKey)
{
this.scopeKey = scopeKey;
return (this);
}
/*******************************************************************************
** Getter for periodStart
*******************************************************************************/
public Instant getPeriodStart()
{
return (this.periodStart);
}
/*******************************************************************************
** Setter for periodStart
*******************************************************************************/
public void setPeriodStart(Instant periodStart)
{
this.periodStart = periodStart;
}
/*******************************************************************************
** Fluent setter for periodStart
*******************************************************************************/
public AssistantBudgetEvent withPeriodStart(Instant periodStart)
{
this.periodStart = periodStart;
return (this);
}
/*******************************************************************************
** Getter for state
*******************************************************************************/
public String getState()
{
return (this.state);
}
/*******************************************************************************
** Setter for state
*******************************************************************************/
public void setState(String state)
{
this.state = state;
}
/*******************************************************************************
** Fluent setter for state
*******************************************************************************/
public AssistantBudgetEvent withState(String state)
{
this.state = state;
return (this);
}
/*******************************************************************************
** Getter for usedValue
*******************************************************************************/
public BigDecimal getUsedValue()
{
return (this.usedValue);
}
/*******************************************************************************
** Setter for usedValue
*******************************************************************************/
public void setUsedValue(BigDecimal usedValue)
{
this.usedValue = usedValue;
}
/*******************************************************************************
** Fluent setter for usedValue
*******************************************************************************/
public AssistantBudgetEvent withUsedValue(BigDecimal usedValue)
{
this.usedValue = usedValue;
return (this);
}
/*******************************************************************************
** Getter for limitValue
*******************************************************************************/
public BigDecimal getLimitValue()
{
return (this.limitValue);
}
/*******************************************************************************
** Setter for limitValue
*******************************************************************************/
public void setLimitValue(BigDecimal limitValue)
{
this.limitValue = limitValue;
}
/*******************************************************************************
** Fluent setter for limitValue
*******************************************************************************/
public AssistantBudgetEvent withLimitValue(BigDecimal limitValue)
{
this.limitValue = limitValue;
return (this);
}
/*******************************************************************************
** Getter for userId
*******************************************************************************/
public String getUserId()
{
return (this.userId);
}
/*******************************************************************************
** Setter for userId
*******************************************************************************/
public void setUserId(String userId)
{
this.userId = userId;
}
/*******************************************************************************
** Fluent setter for userId
*******************************************************************************/
public AssistantBudgetEvent withUserId(String userId)
{
this.userId = userId;
return (this);
}
/*******************************************************************************
** Getter for createDate
*******************************************************************************/
public Instant getCreateDate()
{
return (this.createDate);
}
/*******************************************************************************
** Setter for createDate
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Fluent setter for createDate
*******************************************************************************/
public AssistantBudgetEvent withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
}
@@ -0,0 +1,393 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.model;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import java.math.BigDecimal;
import java.time.Instant;
/*******************************************************************************
** Persisted assistant budget budget override.
*******************************************************************************/
public class AssistantBudgetOverride extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_budget_override";
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@QField(isRequired = true)
private Integer budgetId;
@QField(maxLength = 200)
private String scopeValue;
@QField()
private BigDecimal extraValue;
@QField()
private Instant unlimitedUntil;
@QField(isRequired = true, maxLength = 1000)
private String reason;
@QField(isEditable = false, maxLength = 200)
private String grantedBy;
@QField(isRequired = true)
private Instant expiresAt;
@QField(isEditable = false)
private Instant createDate;
@QField(isEditable = false)
private Instant modifyDate;
/*******************************************************************************
** Default constructor.
*******************************************************************************/
public AssistantBudgetOverride()
{
}
/*******************************************************************************
** Populate from a record.
*******************************************************************************/
public AssistantBudgetOverride(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id
*******************************************************************************/
public Integer getId()
{
return (this.id);
}
/*******************************************************************************
** Setter for id
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Fluent setter for id
*******************************************************************************/
public AssistantBudgetOverride withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for budgetId
*******************************************************************************/
public Integer getBudgetId()
{
return (this.budgetId);
}
/*******************************************************************************
** Setter for budgetId
*******************************************************************************/
public void setBudgetId(Integer budgetId)
{
this.budgetId = budgetId;
}
/*******************************************************************************
** Fluent setter for budgetId
*******************************************************************************/
public AssistantBudgetOverride withBudgetId(Integer budgetId)
{
this.budgetId = budgetId;
return (this);
}
/*******************************************************************************
** Getter for scopeValue
*******************************************************************************/
public String getScopeValue()
{
return (this.scopeValue);
}
/*******************************************************************************
** Setter for scopeValue
*******************************************************************************/
public void setScopeValue(String scopeValue)
{
this.scopeValue = scopeValue;
}
/*******************************************************************************
** Fluent setter for scopeValue
*******************************************************************************/
public AssistantBudgetOverride withScopeValue(String scopeValue)
{
this.scopeValue = scopeValue;
return (this);
}
/*******************************************************************************
** Getter for extraValue
*******************************************************************************/
public BigDecimal getExtraValue()
{
return (this.extraValue);
}
/*******************************************************************************
** Setter for extraValue
*******************************************************************************/
public void setExtraValue(BigDecimal extraValue)
{
this.extraValue = extraValue;
}
/*******************************************************************************
** Fluent setter for extraValue
*******************************************************************************/
public AssistantBudgetOverride withExtraValue(BigDecimal extraValue)
{
this.extraValue = extraValue;
return (this);
}
/*******************************************************************************
** Getter for unlimitedUntil
*******************************************************************************/
public Instant getUnlimitedUntil()
{
return (this.unlimitedUntil);
}
/*******************************************************************************
** Setter for unlimitedUntil
*******************************************************************************/
public void setUnlimitedUntil(Instant unlimitedUntil)
{
this.unlimitedUntil = unlimitedUntil;
}
/*******************************************************************************
** Fluent setter for unlimitedUntil
*******************************************************************************/
public AssistantBudgetOverride withUnlimitedUntil(Instant unlimitedUntil)
{
this.unlimitedUntil = unlimitedUntil;
return (this);
}
/*******************************************************************************
** Getter for reason
*******************************************************************************/
public String getReason()
{
return (this.reason);
}
/*******************************************************************************
** Setter for reason
*******************************************************************************/
public void setReason(String reason)
{
this.reason = reason;
}
/*******************************************************************************
** Fluent setter for reason
*******************************************************************************/
public AssistantBudgetOverride withReason(String reason)
{
this.reason = reason;
return (this);
}
/*******************************************************************************
** Getter for grantedBy
*******************************************************************************/
public String getGrantedBy()
{
return (this.grantedBy);
}
/*******************************************************************************
** Setter for grantedBy
*******************************************************************************/
public void setGrantedBy(String grantedBy)
{
this.grantedBy = grantedBy;
}
/*******************************************************************************
** Fluent setter for grantedBy
*******************************************************************************/
public AssistantBudgetOverride withGrantedBy(String grantedBy)
{
this.grantedBy = grantedBy;
return (this);
}
/*******************************************************************************
** Getter for expiresAt
*******************************************************************************/
public Instant getExpiresAt()
{
return (this.expiresAt);
}
/*******************************************************************************
** Setter for expiresAt
*******************************************************************************/
public void setExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
}
/*******************************************************************************
** Fluent setter for expiresAt
*******************************************************************************/
public AssistantBudgetOverride withExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
return (this);
}
/*******************************************************************************
** Getter for createDate
*******************************************************************************/
public Instant getCreateDate()
{
return (this.createDate);
}
/*******************************************************************************
** Setter for createDate
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Fluent setter for createDate
*******************************************************************************/
public AssistantBudgetOverride withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate
*******************************************************************************/
public Instant getModifyDate()
{
return (this.modifyDate);
}
/*******************************************************************************
** Setter for modifyDate
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Fluent setter for modifyDate
*******************************************************************************/
public AssistantBudgetOverride withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
}
@@ -0,0 +1,428 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.model;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import java.time.Instant;
/*******************************************************************************
** Private accounting projection of usage, without conversation or request data.
** Admin-only in the UI; no owner lock so internal shared-budget sums can run
** under the current session. The original usage record remains owner-locked.
*******************************************************************************/
public class AssistantBudgetUsage extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_budget_usage";
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@QField(isRequired = true)
private Integer usageId;
@QField(maxLength = 255)
private String userId;
@QField()
private Integer providerId;
@QField(maxLength = 200)
private String model;
@QField(isRequired = true)
private Instant occurredAt;
@QField()
private Long costMicros;
@QField()
private Long inputTokens;
@QField()
private Long outputTokens;
@QField()
private Long cacheReadTokens;
@QField()
private Long cacheWriteTokens;
/*******************************************************************************
** Default constructor.
*******************************************************************************/
public AssistantBudgetUsage()
{
}
/*******************************************************************************
** Populate from a record.
*******************************************************************************/
public AssistantBudgetUsage(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id
*******************************************************************************/
public Integer getId()
{
return (this.id);
}
/*******************************************************************************
** Setter for id
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Fluent setter for id
*******************************************************************************/
public AssistantBudgetUsage withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for usageId
*******************************************************************************/
public Integer getUsageId()
{
return (this.usageId);
}
/*******************************************************************************
** Setter for usageId
*******************************************************************************/
public void setUsageId(Integer usageId)
{
this.usageId = usageId;
}
/*******************************************************************************
** Fluent setter for usageId
*******************************************************************************/
public AssistantBudgetUsage withUsageId(Integer usageId)
{
this.usageId = usageId;
return (this);
}
/*******************************************************************************
** Getter for userId
*******************************************************************************/
public String getUserId()
{
return (this.userId);
}
/*******************************************************************************
** Setter for userId
*******************************************************************************/
public void setUserId(String userId)
{
this.userId = userId;
}
/*******************************************************************************
** Fluent setter for userId
*******************************************************************************/
public AssistantBudgetUsage withUserId(String userId)
{
this.userId = userId;
return (this);
}
/*******************************************************************************
** Getter for providerId
*******************************************************************************/
public Integer getProviderId()
{
return (this.providerId);
}
/*******************************************************************************
** Setter for providerId
*******************************************************************************/
public void setProviderId(Integer providerId)
{
this.providerId = providerId;
}
/*******************************************************************************
** Fluent setter for providerId
*******************************************************************************/
public AssistantBudgetUsage withProviderId(Integer providerId)
{
this.providerId = providerId;
return (this);
}
/*******************************************************************************
** Getter for model
*******************************************************************************/
public String getModel()
{
return (this.model);
}
/*******************************************************************************
** Setter for model
*******************************************************************************/
public void setModel(String model)
{
this.model = model;
}
/*******************************************************************************
** Fluent setter for model
*******************************************************************************/
public AssistantBudgetUsage withModel(String model)
{
this.model = model;
return (this);
}
/*******************************************************************************
** Getter for occurredAt
*******************************************************************************/
public Instant getOccurredAt()
{
return (this.occurredAt);
}
/*******************************************************************************
** Setter for occurredAt
*******************************************************************************/
public void setOccurredAt(Instant occurredAt)
{
this.occurredAt = occurredAt;
}
/*******************************************************************************
** Fluent setter for occurredAt
*******************************************************************************/
public AssistantBudgetUsage withOccurredAt(Instant occurredAt)
{
this.occurredAt = occurredAt;
return (this);
}
/*******************************************************************************
** Getter for costMicros
*******************************************************************************/
public Long getCostMicros()
{
return (this.costMicros);
}
/*******************************************************************************
** Setter for costMicros
*******************************************************************************/
public void setCostMicros(Long costMicros)
{
this.costMicros = costMicros;
}
/*******************************************************************************
** Fluent setter for costMicros
*******************************************************************************/
public AssistantBudgetUsage withCostMicros(Long costMicros)
{
this.costMicros = costMicros;
return (this);
}
/*******************************************************************************
** Getter for inputTokens
*******************************************************************************/
public Long getInputTokens()
{
return (this.inputTokens);
}
/*******************************************************************************
** Setter for inputTokens
*******************************************************************************/
public void setInputTokens(Long inputTokens)
{
this.inputTokens = inputTokens;
}
/*******************************************************************************
** Fluent setter for inputTokens
*******************************************************************************/
public AssistantBudgetUsage withInputTokens(Long inputTokens)
{
this.inputTokens = inputTokens;
return (this);
}
/*******************************************************************************
** Getter for outputTokens
*******************************************************************************/
public Long getOutputTokens()
{
return (this.outputTokens);
}
/*******************************************************************************
** Setter for outputTokens
*******************************************************************************/
public void setOutputTokens(Long outputTokens)
{
this.outputTokens = outputTokens;
}
/*******************************************************************************
** Fluent setter for outputTokens
*******************************************************************************/
public AssistantBudgetUsage withOutputTokens(Long outputTokens)
{
this.outputTokens = outputTokens;
return (this);
}
/*******************************************************************************
** Getter for cacheReadTokens
*******************************************************************************/
public Long getCacheReadTokens()
{
return (this.cacheReadTokens);
}
/*******************************************************************************
** Setter for cacheReadTokens
*******************************************************************************/
public void setCacheReadTokens(Long cacheReadTokens)
{
this.cacheReadTokens = cacheReadTokens;
}
/*******************************************************************************
** Fluent setter for cacheReadTokens
*******************************************************************************/
public AssistantBudgetUsage withCacheReadTokens(Long cacheReadTokens)
{
this.cacheReadTokens = cacheReadTokens;
return (this);
}
/*******************************************************************************
** Getter for cacheWriteTokens
*******************************************************************************/
public Long getCacheWriteTokens()
{
return (this.cacheWriteTokens);
}
/*******************************************************************************
** Setter for cacheWriteTokens
*******************************************************************************/
public void setCacheWriteTokens(Long cacheWriteTokens)
{
this.cacheWriteTokens = cacheWriteTokens;
}
/*******************************************************************************
** Fluent setter for cacheWriteTokens
*******************************************************************************/
public AssistantBudgetUsage withCacheWriteTokens(Long cacheWriteTokens)
{
this.cacheWriteTokens = cacheWriteTokens;
return (this);
}
}
@@ -32,6 +32,9 @@ public class AssistantModel extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_model";
@QField()
private Integer fallbackModelId;
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@@ -764,4 +767,35 @@ public class AssistantModel extends QRecordEntity
return (this);
}
/*******************************************************************************
** Get fallbackModelId.
*******************************************************************************/
public Integer getFallbackModelId()
{
return (this.fallbackModelId);
}
/*******************************************************************************
** Set fallbackModelId.
*******************************************************************************/
public void setFallbackModelId(Integer fallbackModelId)
{
this.fallbackModelId = fallbackModelId;
}
/*******************************************************************************
** With fallbackModelId.
*******************************************************************************/
public AssistantModel withFallbackModelId(Integer fallbackModelId)
{
this.fallbackModelId = fallbackModelId;
return (this);
}
}
@@ -32,6 +32,18 @@ public class AssistantProfile extends QRecordEntity
{
public static final String TABLE_NAME = "assistant_profile";
@QField(maxLength = 16)
private String routingMode = "OFF";
@QField()
private Integer simpleModelId;
@QField()
private Integer complexModelId;
@QField()
private Integer classifierModelId;
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@@ -481,4 +493,128 @@ public class AssistantProfile extends QRecordEntity
return (this);
}
/*******************************************************************************
** Get routingMode.
*******************************************************************************/
public String getRoutingMode()
{
return (this.routingMode);
}
/*******************************************************************************
** Set routingMode.
*******************************************************************************/
public void setRoutingMode(String routingMode)
{
this.routingMode = routingMode;
}
/*******************************************************************************
** With routingMode.
*******************************************************************************/
public AssistantProfile withRoutingMode(String routingMode)
{
this.routingMode = routingMode;
return (this);
}
/*******************************************************************************
** Get simpleModelId.
*******************************************************************************/
public Integer getSimpleModelId()
{
return (this.simpleModelId);
}
/*******************************************************************************
** Set simpleModelId.
*******************************************************************************/
public void setSimpleModelId(Integer simpleModelId)
{
this.simpleModelId = simpleModelId;
}
/*******************************************************************************
** With simpleModelId.
*******************************************************************************/
public AssistantProfile withSimpleModelId(Integer simpleModelId)
{
this.simpleModelId = simpleModelId;
return (this);
}
/*******************************************************************************
** Get complexModelId.
*******************************************************************************/
public Integer getComplexModelId()
{
return (this.complexModelId);
}
/*******************************************************************************
** Set complexModelId.
*******************************************************************************/
public void setComplexModelId(Integer complexModelId)
{
this.complexModelId = complexModelId;
}
/*******************************************************************************
** With complexModelId.
*******************************************************************************/
public AssistantProfile withComplexModelId(Integer complexModelId)
{
this.complexModelId = complexModelId;
return (this);
}
/*******************************************************************************
** Get classifierModelId.
*******************************************************************************/
public Integer getClassifierModelId()
{
return (this.classifierModelId);
}
/*******************************************************************************
** Set classifierModelId.
*******************************************************************************/
public void setClassifierModelId(Integer classifierModelId)
{
this.classifierModelId = classifierModelId;
}
/*******************************************************************************
** With classifierModelId.
*******************************************************************************/
public AssistantProfile withClassifierModelId(Integer classifierModelId)
{
this.classifierModelId = classifierModelId;
return (this);
}
}
@@ -71,6 +71,15 @@ public class AssistantProvider extends QRecordEntity
public static final String TABLE_NAME = "assistant_provider";
@QField(isEditable = false, maxLength = 8)
private String healthState;
@QField(isEditable = false)
private Instant healthCheckedAt;
@QField(isEditable = false)
private Instant downUntil;
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@@ -1116,4 +1125,97 @@ public class AssistantProvider extends QRecordEntity
this.cloudProject = cloudProject;
return (this);
}
/*******************************************************************************
** Get healthState.
*******************************************************************************/
public String getHealthState()
{
return (this.healthState);
}
/*******************************************************************************
** Set healthState.
*******************************************************************************/
public void setHealthState(String healthState)
{
this.healthState = healthState;
}
/*******************************************************************************
** With healthState.
*******************************************************************************/
public AssistantProvider withHealthState(String healthState)
{
this.healthState = healthState;
return (this);
}
/*******************************************************************************
** Get healthCheckedAt.
*******************************************************************************/
public Instant getHealthCheckedAt()
{
return (this.healthCheckedAt);
}
/*******************************************************************************
** Set healthCheckedAt.
*******************************************************************************/
public void setHealthCheckedAt(Instant healthCheckedAt)
{
this.healthCheckedAt = healthCheckedAt;
}
/*******************************************************************************
** With healthCheckedAt.
*******************************************************************************/
public AssistantProvider withHealthCheckedAt(Instant healthCheckedAt)
{
this.healthCheckedAt = healthCheckedAt;
return (this);
}
/*******************************************************************************
** Get downUntil.
*******************************************************************************/
public Instant getDownUntil()
{
return (this.downUntil);
}
/*******************************************************************************
** Set downUntil.
*******************************************************************************/
public void setDownUntil(Instant downUntil)
{
this.downUntil = downUntil;
}
/*******************************************************************************
** With downUntil.
*******************************************************************************/
public AssistantProvider withDownUntil(Instant downUntil)
{
this.downUntil = downUntil;
return (this);
}
}
@@ -44,6 +44,18 @@ public class AssistantTurn extends QRecordEntity
@QField(isEditable = false, maxLength = 500)
private String classificationTables;
@QField(isEditable = false, maxLength = 16)
private String routeDecision;
@QField(isEditable = false, maxLength = 16)
private String routeSource;
@QField(isEditable = false)
private Integer fallbackCount;
@QField(isEditable = false)
private Integer providerId;
@QField(isEditable = false, isPrimaryKey = true)
private Integer id;
@@ -875,4 +887,128 @@ public class AssistantTurn extends QRecordEntity
/*******************************************************************************
** Get routeDecision.
*******************************************************************************/
public String getRouteDecision()
{
return (this.routeDecision);
}
/*******************************************************************************
** Set routeDecision.
*******************************************************************************/
public void setRouteDecision(String routeDecision)
{
this.routeDecision = routeDecision;
}
/*******************************************************************************
** With routeDecision.
*******************************************************************************/
public AssistantTurn withRouteDecision(String routeDecision)
{
this.routeDecision = routeDecision;
return (this);
}
/*******************************************************************************
** Get routeSource.
*******************************************************************************/
public String getRouteSource()
{
return (this.routeSource);
}
/*******************************************************************************
** Set routeSource.
*******************************************************************************/
public void setRouteSource(String routeSource)
{
this.routeSource = routeSource;
}
/*******************************************************************************
** With routeSource.
*******************************************************************************/
public AssistantTurn withRouteSource(String routeSource)
{
this.routeSource = routeSource;
return (this);
}
/*******************************************************************************
** Get fallbackCount.
*******************************************************************************/
public Integer getFallbackCount()
{
return (this.fallbackCount);
}
/*******************************************************************************
** Set fallbackCount.
*******************************************************************************/
public void setFallbackCount(Integer fallbackCount)
{
this.fallbackCount = fallbackCount;
}
/*******************************************************************************
** With fallbackCount.
*******************************************************************************/
public AssistantTurn withFallbackCount(Integer fallbackCount)
{
this.fallbackCount = fallbackCount;
return (this);
}
/*******************************************************************************
** Get providerId.
*******************************************************************************/
public Integer getProviderId()
{
return (this.providerId);
}
/*******************************************************************************
** Set providerId.
*******************************************************************************/
public void setProviderId(Integer providerId)
{
this.providerId = providerId;
}
/*******************************************************************************
** With providerId.
*******************************************************************************/
public AssistantTurn withProviderId(Integer providerId)
{
this.providerId = providerId;
return (this);
}
}
@@ -17,17 +17,12 @@
package com.kingsrook.qbits.assistant.persistence;
import java.io.Serializable;
import java.time.Instant;
import java.time.LocalDate;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
import com.kingsrook.qbits.assistant.model.AssistantFeedback;
@@ -61,6 +56,15 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateOutput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import com.kingsrook.qqq.backend.core.utils.CollectionUtils;
import java.io.Serializable;
import java.time.Instant;
import java.time.LocalDate;
import java.util.ArrayList;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/*******************************************************************************
@@ -146,6 +150,10 @@ public class AssistantStore
{
throw (new QException("Could not save to " + entity.tableName() + ": " + record.getErrors().get(0).getMessage()));
}
if(entity instanceof AssistantUsage)
{
projectBudgetUsage(record);
}
return ((T) QRecordEntity.fromQRecord(entity.getClass(), cryptRecord(entity.tableName(), record, false)));
}
@@ -179,6 +187,11 @@ public class AssistantStore
{
throw (new QException("Could not update " + tableName + ": " + result.getErrors().get(0).getMessage()));
}
if(AssistantUsage.TABLE_NAME.equals(tableName))
{
AssistantUsage usage = first(query(tableName, new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, record.getValue("id")), AssistantUsage.class));
projectBudgetUsage(usage.toQRecord());
}
}
@@ -834,4 +847,91 @@ public class AssistantStore
.withCriteria("id", QCriteriaOperator.GREATER_THAN, afterId)
.withOrderBy(new QFilterOrderBy("id", true)).withLimit(limit), AssistantConversation.class));
}
/*******************************************************************************
** Enabled data-driven budgets, read once per evaluation (never per token).
*******************************************************************************/
public List<AssistantBudget> budgets() throws QException
{
return (query(AssistantBudget.TABLE_NAME, new QQueryFilter().withCriteria("enabled", QCriteriaOperator.EQUALS, true)
.withOrderBy(new QFilterOrderBy("id", true)), AssistantBudget.class));
}
/*******************************************************************************
** One budget, including disabled rows for override validation.
*******************************************************************************/
public AssistantBudget budget(Integer id) throws QException
{
return (first(query(AssistantBudget.TABLE_NAME, new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, id), AssistantBudget.class)));
}
/*******************************************************************************
** Overrides of one budget; expiry and narrower scope are evaluated together.
*******************************************************************************/
public List<AssistantBudgetOverride> budgetOverrides(Integer id) throws QException
{
return (query(AssistantBudgetOverride.TABLE_NAME, new QQueryFilter().withCriteria("budgetId", QCriteriaOperator.EQUALS, id), AssistantBudgetOverride.class));
}
/*******************************************************************************
** Persist a transition once. The unique key arbitrates concurrent replicas;
** an insert failure is ignored only when the winning row can be read.
*******************************************************************************/
public synchronized boolean budgetEvent(AssistantBudgetEvent event) throws QException
{
QQueryFilter filter = new QQueryFilter().withCriteria("budgetId", QCriteriaOperator.EQUALS, event.getBudgetId())
.withCriteria("scopeKey", QCriteriaOperator.EQUALS, event.getScopeKey())
.withCriteria("periodStart", QCriteriaOperator.EQUALS, event.getPeriodStart())
.withCriteria("state", QCriteriaOperator.EQUALS, event.getState());
if(!query(AssistantBudgetEvent.TABLE_NAME, filter, AssistantBudgetEvent.class).isEmpty())
{
return (false);
}
try
{
insert(event);
return (true);
}
catch(QException e)
{
if(query(AssistantBudgetEvent.TABLE_NAME, filter, AssistantBudgetEvent.class).isEmpty())
{
throw (e);
}
return (false);
}
}
/*******************************************************************************
** Accounting derives only from persisted usage. No conversation or request
** data is copied, and no session or security key is changed.
*******************************************************************************/
private void projectBudgetUsage(QRecord usage) throws QException
{
List<AssistantBudgetUsage> existing = query(AssistantBudgetUsage.TABLE_NAME,
new QQueryFilter().withCriteria("usageId", QCriteriaOperator.EQUALS, usage.getValueInteger("id")), AssistantBudgetUsage.class);
QRecord projection = new QRecord().withValue("usageId", usage.getValueInteger("id")).withValue("occurredAt", usage.getValueInstant("createDate"));
for(String field : List.of("userId", "providerId", "model", "costMicros", "inputTokens", "outputTokens", "cacheReadTokens", "cacheWriteTokens"))
{
projection.setValue(field, usage.getValue(field));
}
if(existing.isEmpty())
{
insert(new AssistantBudgetUsage(projection));
}
else
{
updateFields(AssistantBudgetUsage.TABLE_NAME, projection.withValue("id", existing.get(0).getId()));
}
}
}
@@ -36,6 +36,7 @@ public class ModelRequest
private ModelCapabilities capabilities;
private Boolean streaming = true;
private String model;
private String effort;
private Integer maxTokens;
@@ -52,6 +53,7 @@ public class ModelRequest
public ModelRequest copy()
{
return (new ModelRequest()
.withStreaming(streaming)
.withModel(model)
.withCapabilities(capabilities)
.withEffort(effort)
@@ -311,4 +313,25 @@ public class ModelRequest
return (this);
}
/*******************************************************************************
** Whether to request streaming transport (classifiers use one JSON reply).
*******************************************************************************/
public Boolean getStreaming()
{
return (streaming);
}
/*******************************************************************************
** Select streaming transport independently of model capability.
*******************************************************************************/
public ModelRequest withStreaming(Boolean streaming)
{
this.streaming = streaming;
return (this);
}
}
@@ -0,0 +1,727 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.tools;
import java.io.InputStream;
import java.math.BigDecimal;
import java.math.MathContext;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.function.Function;
import com.fasterxml.jackson.core.type.TypeReference;
import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
import com.kingsrook.qqq.backend.core.actions.ai.output.AiLinkBuilder;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
/*******************************************************************************
** Builds presentations from previously retrieved values, never reading records.
** One instance belongs to one turn: UI metadata, including denials, is cached
** only for that turn and session. All output links are built by the server.
*******************************************************************************/
public class PresentResultsTool
{
public static final String NAME = "present_results";
private final QInstance qInstance;
private UiApiResult metadata;
/*******************************************************************************
** Create a turn-scoped presenter.
*******************************************************************************/
public PresentResultsTool(QInstance qInstance)
{
this.qInstance = qInstance;
}
/*******************************************************************************
** Stable schema. Strict opt-in requires empty placeholders instead of many
** optional productions, keeping the existing strict grammar budget small.
*******************************************************************************/
public static ModelToolDefinition definition(Boolean strict)
{
try(InputStream stream = PresentResultsTool.class.getResourceAsStream("/present-results.schema.json"))
{
Map<String, Object> schema = JsonUtils.toObject(new String(stream.readAllBytes(), StandardCharsets.UTF_8), new TypeReference<Map<String, Object>>() {});
if(strict)
{
requireProperties(schema);
}
return (new ModelToolDefinition().withName(NAME).withDescription("Present an answer as metrics, a table, a chart and actionable links. Use only data from tool results; never supply URLs.")
.withInputSchema(schema).withStrict(strict));
}
catch(Exception e)
{
throw (new IllegalStateException("Cannot load presentation schema", e));
}
}
/*******************************************************************************
** Eliminate optional productions for strict mode without adding unions.
*******************************************************************************/
private static void requireProperties(Object node)
{
if(node instanceof Map<?, ?>)
{
Map<String, Object> schema = object(node);
schema.remove("maxLength");
if(schema.get("properties") instanceof Map<?, ?> properties)
{
schema.put("required", new ArrayList<>(properties.keySet()));
}
schema.values().forEach(PresentResultsTool::requireProperties);
}
else if(node instanceof List<?> list)
{
list.forEach(PresentResultsTool::requireProperties);
}
}
/*******************************************************************************
** Preserve valid items when others are dropped; never echo rejected values.
*******************************************************************************/
public synchronized AiToolResult execute(UiApiToolbox.Context context, String callId, Map<String, Object> input)
{
AiToolResult result = new AiToolResult().withToolCallId(callId).withToolName(NAME);
try
{
require(input != null && input.get("summary") instanceof String, "summary required");
String summary = text(input.get("summary"));
require(!summary.isBlank() && summary.length() <= 300, "summary must contain 1 to 300 characters");
List<String> notes = new ArrayList<>();
List<Map<String, Object>> blocks = new ArrayList<>();
addItems(blocks, "metrics", input.get("metrics"), notes, item -> metric(context, item));
Map<String, Object> table = null;
if(input.get("table") != null)
{
try
{
table = table(context, object(input.get("table")));
if(!list(table.get("columns")).isEmpty())
{
blocks.add(table);
}
}
catch(IllegalArgumentException e)
{
notes.add("Dropped table: " + e.getMessage());
}
}
if(input.get("chart") != null && table != null && !list(table.get("rows")).isEmpty())
{
try
{
blocks.add(chart(object(input.get("chart")), table, notes));
}
catch(IllegalArgumentException e)
{
notes.add("Dropped chart: " + e.getMessage());
}
}
if(input.get("chart") != null && (table == null || list(table.get("rows")).isEmpty()))
{
notes.add("Dropped chart: requires a valid nonempty table.");
}
addItems(blocks, "filterLinks", input.get("filterLinks"), notes, item -> filterLink(context, item));
addItems(blocks, "recordLinks", input.get("recordLinks"), notes, item -> recordLink(context, item));
addItems(blocks, "actions", input.get("actions"), notes, item -> action(context, item));
Map<String, Object> part = map("type", "presentation", "summary", summary, "blocks", blocks, "markdown", markdown(summary, blocks));
return (result.withOutcome(AiToolOutcome.OK).withSummary("Presented results").withRender(part)
.withPermissionCheck(metadata == null ? "presentation:no table references" : "uiApi:GET /metaData:" + metadata.status())
.withModelContent("Presented to the user: " + summary + (summary.endsWith(".") ? "" : ".") + " Do not repeat the table or links in text." + (notes.isEmpty() ? "" : "\n" + String.join("\n", notes))));
}
catch(IllegalArgumentException e)
{
return (result.withOutcome(AiToolOutcome.INVALID_INPUT).withSummary("Invalid presentation").withModelContent(e.getMessage()));
}
}
/*******************************************************************************
** Resolve tables through the same metadata operation and session as the UI.
*******************************************************************************/
private QTableMetaData visibleTable(UiApiToolbox.Context context, String name)
{
QTableMetaData table = qInstance.getTable(name);
require(table != null, "unknown table " + name);
if(metadata == null)
{
try
{
metadata = context.gateway().invoke(new UiApiCall("GET", "/metaData").withSessionUUID(context.sessionUUID()).withUserTimezone(context.userTimezone()));
}
catch(QException e)
{
metadata = new UiApiResult(503, "{}");
}
}
require(metadata.isSuccess(), "GET /metaData answered HTTP " + metadata.status());
Map<String, Object> tables = object(object(metadata.json()).get("tables"));
require(tables.containsKey(name) && Boolean.TRUE.equals(object(tables.get(name)).get("readPermission")), "table not visible through UI API: " + name);
return (table);
}
/*******************************************************************************
** A field reference must be paired with a visible table reference.
*******************************************************************************/
private QFieldMetaData field(UiApiToolbox.Context context, Map<String, Object> item)
{
String tableName = text(item.get("tableName"));
String fieldName = text(item.get("fieldName"));
require(fieldName.isEmpty() || !tableName.isEmpty(), "fieldName requires tableName");
QTableMetaData table = tableName.isEmpty() ? null : visibleTable(context, tableName);
QFieldMetaData field = fieldName.isEmpty() ? null : table.getFields().get(fieldName);
require(fieldName.isEmpty() || field != null, "unknown field " + fieldName);
return (field);
}
/*******************************************************************************
** A metric preserves its numeric value and adds the metadata display.
*******************************************************************************/
private Map<String, Object> metric(UiApiToolbox.Context context, Map<String, Object> item)
{
QFieldMetaData field = field(context, item);
number(item.get("value"));
return (map("label", label(item), "value", item.get("value"), "display", display(item.get("value"), field)));
}
/*******************************************************************************
** Build table cells and percentages from raw values, never model displays.
*******************************************************************************/
private Map<String, Object> table(UiApiToolbox.Context context, Map<String, Object> input)
{
List<?> columns = list(input.get("columns"));
List<?> rows = list(input.get("rows"));
require(columns.size() <= 20 && rows.size() <= 200, "table exceeds 20 columns or 200 rows");
require(rows.isEmpty() || !columns.isEmpty(), "rows require columns");
List<QFieldMetaData> fields = new ArrayList<>();
List<Map<String, Object>> outputColumns = new ArrayList<>();
List<BigDecimal> totals = new ArrayList<>();
for(Integer c = 0; c < columns.size(); c++)
{
Map<String, Object> column = object(columns.get(c));
fields.add(field(context, column));
BigDecimal total = BigDecimal.ZERO;
Boolean numeric = !rows.isEmpty();
for(Object row : rows)
{
require(list(row).size() == columns.size(), "row width must match columns");
Object value = list(row).get(c);
scalar(value);
if(value instanceof Number)
{
total = total.add(number(value));
}
else
{
numeric = false;
}
}
totals.add(numeric ? total : null);
outputColumns.add(map("key", "c" + c, "label", label(column), "align", numeric || percentColumn(column) >= 0 ? "right" : "left"));
}
Map<String, Object> rowFilter = input.get("rowFilter") == null ? Map.of() : object(input.get("rowFilter"));
List<?> rowKeys = input.get("rowKeys") == null ? List.of() : list(input.get("rowKeys"));
Boolean hasFilter = !text(rowFilter.get("tableName")).isEmpty();
require(!hasFilter || rowKeys.size() == rows.size(), "rowKeys must match rows");
QFieldMetaData filterField = field(context, rowFilter);
require(!hasFilter || filterField != null, "rowFilter requires a field");
List<Map<String, Object>> outputRows = new ArrayList<>();
for(Integer r = 0; r < rows.size(); r++)
{
Map<String, Object> cells = new LinkedHashMap<>();
Map<String, Object> displays = new LinkedHashMap<>();
for(Integer c = 0; c < columns.size(); c++)
{
Integer source = percentColumn(object(columns.get(c)));
Object value = list(rows.get(r)).get(c);
String formatted;
if(source >= 0)
{
require(source < columns.size() && totals.get(source) != null && percentColumn(object(columns.get(source))) < 0, "percentOfColumn must reference a numeric raw column");
BigDecimal share = totals.get(source).signum() == 0 ? BigDecimal.ZERO : number(list(rows.get(r)).get(source)).divide(totals.get(source), MathContext.DECIMAL64);
value = share;
formatted = String.format(Locale.US, "%.1f%%", share.multiply(BigDecimal.valueOf(100)));
}
else
{
formatted = display(value, fields.get(c));
}
cells.put("c" + c, value);
displays.put("c" + c, formatted);
}
Map<String, Object> row = map("cells", cells, "display", displays);
if(hasFilter)
{
Object key = rowKeys.get(r);
scalar(key);
Map<String, Object> criterion = map("fieldName", rowFilter.get("fieldName"), "operator", rowFilter.getOrDefault("operator", "EQUALS"), "values", java.util.Collections.singletonList(key));
row.put("href", filterHref(context, text(rowFilter.get("tableName")), List.of(criterion)));
}
outputRows.add(row);
}
return (map("kind", "table", "title", text(input.get("title")), "columns", outputColumns, "rows", outputRows));
}
/*******************************************************************************
** A chart uses the table's raw values, formatted values and row links.
*******************************************************************************/
private Map<String, Object> chart(Map<String, Object> input, Map<String, Object> table, List<String> notes)
{
String type = text(input.get("chartType"));
require(List.of("bar", "pie", "line").contains(type), "invalid chartType");
Integer labelColumn = index(input.get("labelColumn"));
Integer valueColumn = index(input.get("valueColumn"));
Integer width = list(table.get("columns")).size();
require(labelColumn >= 0 && valueColumn >= 0 && labelColumn < width && valueColumn < width, "chart column index out of range");
List<Map<String, Object>> points = new ArrayList<>();
List<?> rows = list(table.get("rows"));
for(Object entry : rows)
{
Map<String, Object> row = object(entry);
Object value = object(row.get("cells")).get("c" + valueColumn);
number(value);
if(points.size() < 12)
{
Map<String, Object> point = map("label", object(row.get("display")).get("c" + labelColumn), "value", value, "display", object(row.get("display")).get("c" + valueColumn));
if(row.containsKey("href"))
{
point.put("href", row.get("href"));
}
points.add(point);
}
}
if(rows.size() > 12)
{
notes.add("Chart truncated to 12 points; the table retains all rows.");
}
return (map("kind", "chart", "chartType", type, "title", text(input.get("title")), "points", points));
}
/*******************************************************************************
** A filtered list link, including only validated metadata and raw values.
*******************************************************************************/
private Map<String, Object> filterLink(UiApiToolbox.Context context, Map<String, Object> item)
{
String tableName = text(item.get("tableName"));
return (map("label", label(item), "tableName", tableName, "href", filterHref(context, tableName, list(item.get("criteria")))));
}
/*******************************************************************************
** Serialize the UI's JSON filter format. The table path uses core navigate's
** helper; filter values stay raw inside the URL-encoded JSON.
*******************************************************************************/
private String filterHref(UiApiToolbox.Context context, String tableName, List<?> criteria)
{
visibleTable(context, tableName);
require(!criteria.isEmpty() && criteria.size() <= 20, "filter requires 1 to 20 criteria");
List<Map<String, Object>> output = new ArrayList<>();
for(Object entry : criteria)
{
Map<String, Object> criterion = object(entry);
String fieldName = text(criterion.get("fieldName"));
require(!fieldName.isBlank(), "filter requires fieldName");
field(context, map("tableName", tableName, "fieldName", fieldName));
String operator = text(criterion.get("operator"));
operator = operator.isEmpty() ? "EQUALS" : operator;
try
{
QCriteriaOperator.valueOf(operator);
}
catch(IllegalArgumentException e)
{
throw (new IllegalArgumentException("invalid criteria operator"));
}
List<?> values = list(criterion.get("values"));
require(values.size() <= 200, "too many filter values");
values.forEach(PresentResultsTool::scalar);
output.add(map("fieldName", fieldName, "operator", operator, "values", values));
}
return (safeHref(AiLinkBuilder.objectHref(tableName) + "?filter=" + URLEncoder.encode(JsonUtils.toJson(map("criteria", output, "booleanOperator", "AND")), StandardCharsets.UTF_8)));
}
/*******************************************************************************
** Record ids are opaque path segments, encoded by core's link helper.
*******************************************************************************/
private Map<String, Object> recordLink(UiApiToolbox.Context context, Map<String, Object> item)
{
String tableName = text(item.get("tableName"));
visibleTable(context, tableName);
Object id = item.get("recordId");
require(id instanceof String || id instanceof Number, "recordId must be a string or number");
String recordId = text(id);
require(!recordId.isBlank() && !List.of(".", "..").contains(recordId), "invalid recordId");
return (map("label", label(item), "tableName", tableName, "recordId", recordId, "href", safeHref(AiLinkBuilder.recordHref(tableName, recordId))));
}
/*******************************************************************************
** Actions are either a next-message prompt or a visible table destination.
*******************************************************************************/
private Map<String, Object> action(UiApiToolbox.Context context, Map<String, Object> item)
{
String prompt = text(item.get("prompt"));
String destination = text(item.get("navigateTo"));
require(prompt.isBlank() != destination.isBlank(), "action requires exactly one of prompt or navigateTo");
if(!destination.isBlank())
{
visibleTable(context, destination);
return (map("label", label(item), "href", safeHref(AiLinkBuilder.objectHref(destination))));
}
return (map("label", label(item), "prompt", prompt));
}
/*******************************************************************************
** Preserve independently valid items and report dropped positions.
*******************************************************************************/
private static void addItems(List<Map<String, Object>> blocks, String kind, Object input, List<String> notes, Function<Map<String, Object>, Map<String, Object>> build)
{
if(input == null)
{
return;
}
List<?> entries = list(input);
require(entries.size() <= 100, "too many " + kind);
List<Map<String, Object>> items = new ArrayList<>();
for(Integer i = 0; i < entries.size(); i++)
{
try
{
items.add(build.apply(object(entries.get(i))));
}
catch(IllegalArgumentException e)
{
notes.add("Dropped " + kind + "[" + i + "]: " + e.getMessage());
}
}
if(!items.isEmpty())
{
blocks.add(map("kind", kind, "items", items));
}
}
/*******************************************************************************
** Metadata formats use QQQ field coercion and Java display format strings.
*******************************************************************************/
private static String display(Object value, QFieldMetaData field)
{
if(value == null)
{
return ("");
}
if(field != null)
{
try
{
Object typed = ValueUtils.getValueAsFieldType(field.getType(), value);
if(field.getDisplayFormat() != null)
{
return (String.format(Locale.US, field.getDisplayFormat(), typed));
}
value = typed;
}
catch(Exception e)
{
throw (new IllegalArgumentException("value does not match field format"));
}
}
if(value instanceof Number)
{
BigDecimal decimal = number(value);
return (decimal.stripTrailingZeros().scale() <= 0 ? String.format(Locale.US, "%,d", decimal.toBigIntegerExact()) : decimal.toPlainString());
}
return (text(value));
}
/*******************************************************************************
** Complete GFM fallback with real newlines; chart data is already in table.
*******************************************************************************/
private static String markdown(String summary, List<Map<String, Object>> blocks)
{
StringBuilder markdown = new StringBuilder(escape(summary)).append("\n");
for(Map<String, Object> block : blocks)
{
if("table".equals(block.get("kind")))
{
markdown.append("\n").append(escape(text(block.get("title")))).append("\n\n|");
List<?> columns = list(block.get("columns"));
for(Object column : columns)
{
markdown.append(' ').append(escape(text(object(column).get("label")))).append(" |");
}
markdown.append("\n|");
for(Object column : columns)
{
markdown.append("right".equals(object(column).get("align")) ? " ---: |" : " --- |");
}
for(Object entry : list(block.get("rows")))
{
Map<String, Object> row = object(entry);
markdown.append("\n|");
for(Integer c = 0; c < columns.size(); c++)
{
String value = escape(text(object(row.get("display")).get("c" + c)));
markdown.append(' ').append(c == 0 && row.containsKey("href") ? "[" + value + "](" + row.get("href") + ")" : value).append(" |");
}
}
markdown.append("\n");
}
else if(block.get("items") instanceof List<?> items)
{
markdown.append("\n");
for(Object entry : items)
{
Map<String, Object> item = object(entry);
String label = escape(text(item.get("label")));
markdown.append("- ");
if(item.containsKey("href"))
{
markdown.append('[').append(label).append("](").append(item.get("href")).append(')');
}
else
{
markdown.append(label);
if(item.containsKey("display") || item.containsKey("prompt"))
{
markdown.append(": ").append(escape(text(item.getOrDefault("display", item.get("prompt")))));
}
}
markdown.append("\n");
}
}
}
return (markdown.toString());
}
/*******************************************************************************
** Literal text only: no Markdown links, HTML or GFM URL autolinks from labels.
*******************************************************************************/
private static String escape(String value)
{
StringBuilder output = new StringBuilder();
for(char character : value.toCharArray())
{
if("\\`*_{}[]()<>|!&#:.@~".indexOf(character) >= 0)
{
output.append("&#").append((int) character).append(';');
}
else
{
output.append(character);
}
}
return (output.toString());
}
/*******************************************************************************
** Reject unsafe paths even if metadata contains unexpected names.
*******************************************************************************/
private static String safeHref(String href)
{
require(AiLinkBuilder.isSafeAppPath(href), "unsafe application path");
return (href);
}
/*******************************************************************************
** A column with no percentage uses -1 in strict mode.
*******************************************************************************/
private static Integer percentColumn(Map<String, Object> column)
{
return (column.get("percentOfColumn") == null ? -1 : index(column.get("percentOfColumn")));
}
/*******************************************************************************
** Indexes must be integral; never silently truncate model input.
*******************************************************************************/
private static Integer index(Object value)
{
try
{
return (number(value).intValueExact());
}
catch(ArithmeticException e)
{
throw (new IllegalArgumentException("index must be an integer"));
}
}
/*******************************************************************************
** Finite numeric input only.
*******************************************************************************/
private static BigDecimal number(Object value)
{
require(value instanceof Number, "numeric value required");
try
{
return (new BigDecimal(value.toString()));
}
catch(NumberFormatException e)
{
throw (new IllegalArgumentException("finite numeric value required"));
}
}
/*******************************************************************************
** Cells and filter values remain raw JSON scalars.
*******************************************************************************/
private static void scalar(Object value)
{
require(value == null || value instanceof String || value instanceof Number || value instanceof Boolean, "scalar value required");
if(value instanceof Number)
{
number(value);
}
if(value instanceof String)
{
text(value);
}
}
/*******************************************************************************
** Bounded single-line display text.
*******************************************************************************/
private static String text(Object value)
{
require(value == null || value instanceof String || value instanceof Number || value instanceof Boolean, "text or scalar required");
String text = value == null ? "" : String.valueOf(value);
require(text.length() <= 2000, "text exceeds 2000 characters");
return (text.replaceAll("[\\p{Cntrl}\\u2028\\u2029]", " "));
}
/*******************************************************************************
** Labels are mandatory plain text.
*******************************************************************************/
private static String label(Map<String, Object> item)
{
String label = text(item.get("label"));
require(!label.isBlank(), "label required");
return (label);
}
/*******************************************************************************
** Validate a JSON object without converting arbitrary values to strings.
*******************************************************************************/
@SuppressWarnings("unchecked")
private static Map<String, Object> object(Object value)
{
require(value instanceof Map<?, ?>, "object required");
return ((Map<String, Object>) value);
}
/*******************************************************************************
** Validate a JSON array.
*******************************************************************************/
private static List<?> list(Object value)
{
require(value instanceof List<?>, "array required");
return ((List<?>) value);
}
/*******************************************************************************
** Ordered output maps allow null raw cells.
*******************************************************************************/
private static Map<String, Object> map(Object... pairs)
{
Map<String, Object> map = new LinkedHashMap<>();
for(Integer i = 0; i < pairs.length; i += 2)
{
map.put((String) pairs[i], pairs[i + 1]);
}
return (map);
}
/*******************************************************************************
** Validation failures contain only server-authored explanations.
*******************************************************************************/
private static void require(Boolean condition, String message)
{
if(!condition)
{
throw (new IllegalArgumentException(message));
}
}
}
@@ -0,0 +1,280 @@
{
"type": "object",
"properties": {
"summary": {
"type": "string",
"maxLength": 300
},
"metrics": {
"type": "array",
"items": {
"$ref": "#/$defs/metric"
}
},
"table": {
"$ref": "#/$defs/table"
},
"chart": {
"$ref": "#/$defs/chart"
},
"filterLinks": {
"type": "array",
"items": {
"$ref": "#/$defs/filterLink"
}
},
"recordLinks": {
"type": "array",
"items": {
"$ref": "#/$defs/recordLink"
}
},
"actions": {
"type": "array",
"items": {
"$ref": "#/$defs/action"
}
}
},
"required": [
"summary"
],
"additionalProperties": false,
"$defs": {
"cell": {
"type": [
"string",
"number",
"boolean",
"null"
]
},
"metric": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"value": {
"type": "number"
},
"tableName": {
"type": "string"
},
"fieldName": {
"type": "string"
}
},
"required": [
"label",
"value"
],
"additionalProperties": false
},
"column": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"tableName": {
"type": "string"
},
"fieldName": {
"type": "string"
},
"percentOfColumn": {
"type": "integer"
}
},
"required": [
"label"
],
"additionalProperties": false
},
"rowFilter": {
"type": "object",
"properties": {
"tableName": {
"type": "string"
},
"fieldName": {
"type": "string"
},
"operator": {
"type": "string"
}
},
"required": [
"tableName",
"fieldName"
],
"additionalProperties": false
},
"criterion": {
"type": "object",
"properties": {
"fieldName": {
"type": "string"
},
"operator": {
"type": "string"
},
"values": {
"type": "array",
"items": {
"$ref": "#/$defs/cell"
}
}
},
"required": [
"fieldName",
"operator",
"values"
],
"additionalProperties": false
},
"filterLink": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"tableName": {
"type": "string"
},
"criteria": {
"$ref": "#/$defs/criteria"
}
},
"required": [
"label",
"tableName",
"criteria"
],
"additionalProperties": false
},
"recordLink": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"tableName": {
"type": "string"
},
"recordId": {
"$ref": "#/$defs/cell"
}
},
"required": [
"label",
"tableName",
"recordId"
],
"additionalProperties": false
},
"action": {
"type": "object",
"properties": {
"label": {
"type": "string"
},
"prompt": {
"type": "string"
},
"navigateTo": {
"type": "string"
}
},
"required": [
"label"
],
"additionalProperties": false
},
"chart": {
"type": "object",
"properties": {
"chartType": {
"type": "string",
"enum": [
"bar",
"pie",
"line"
]
},
"title": {
"type": "string"
},
"labelColumn": {
"type": "integer"
},
"valueColumn": {
"type": "integer"
}
},
"required": [
"chartType",
"labelColumn",
"valueColumn"
],
"additionalProperties": false
},
"row": {
"type": "array",
"items": {
"$ref": "#/$defs/cell"
}
},
"columns": {
"type": "array",
"items": {
"$ref": "#/$defs/column"
}
},
"rows": {
"type": "array",
"items": {
"$ref": "#/$defs/row"
}
},
"rowKeys": {
"type": "array",
"items": {
"$ref": "#/$defs/cell"
}
},
"criteria": {
"type": "array",
"items": {
"$ref": "#/$defs/criterion"
}
},
"table": {
"type": "object",
"properties": {
"title": {
"type": "string"
},
"columns": {
"$ref": "#/$defs/columns"
},
"rows": {
"$ref": "#/$defs/rows"
},
"rowFilter": {
"$ref": "#/$defs/rowFilter"
},
"rowKeys": {
"$ref": "#/$defs/rowKeys"
}
},
"required": [
"columns",
"rows"
],
"additionalProperties": false
}
},
"description": "Present only facts from previous tool results. Optional sections may be omitted. In strict mode use empty arrays, empty strings, empty table rows/columns and -1 for unused indexes; every property must be supplied. rowKeys belongs to table, one raw key per row. Percent columns may use null cells; the server computes them."
}
@@ -17,8 +17,6 @@
package com.kingsrook.qbits.assistant;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.assistant.metadata.AssistantAdminPermissionChecker;
import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
@@ -28,6 +26,9 @@ import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionCheckResult;
import com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
@@ -44,6 +45,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule;
import java.util.ArrayList;
import java.util.List;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
@@ -85,7 +88,7 @@ class AssistantQBitProducerTest extends BaseAssistantTest
// the nav: the home plus the main tables; processes are actions //
//////////////////////////////////////////////////////////////////////
assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getChildren()).extracting(c -> c.getName())
.containsExactly("assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_model_price", "assistant_policy");
.containsExactly("assistant_budget", "assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_model_price", "assistant_policy");
assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getWidgets()).containsExactly("assistantHome", "assistantAccess");
assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME).getTableName()).isEqualTo("assistant_conversation");
assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME)).isNotNull();
@@ -103,6 +106,28 @@ class AssistantQBitProducerTest extends BaseAssistantTest
/*******************************************************************************
** Budget types are discoverable through core with admin-only visibility.
*******************************************************************************/
@Test
void budgetAlertTypes()
{
var types = AlertsInstanceMetaData.of(qInstance).getAlertTypes();
assertThat(types).containsOnlyKeys("assistant.budget.warning", "assistant.budget.downgraded", "assistant.budget.stopped");
for(QAlertTypeMetaData type : types.values())
{
assertThat(type.getLabel()).isNotBlank();
assertThat(type.getDescription()).isNotBlank();
assertThat(type.getDefaultSeverity()).isEqualTo(type.getName().endsWith("stopped") ? AlertSeverity.ERROR : AlertSeverity.WARNING);
assertThat(type.getHowToFix()).isEqualTo("Raise the budget or grant an override in Assistant → Budgets");
assertThat(type.getSourceQBitName()).isEqualTo("assistant");
assertThat(type.getVisibilityPermission()).isEqualTo("assistant.admin");
assertThat(type.getSubjectTableName()).isEqualTo("assistant_budget");
}
}
/*******************************************************************************
** The admin app and tables: visible only with assistant.admin.
*******************************************************************************/
@@ -93,6 +93,10 @@ class ClassificationTest extends BaseAssistantTest
assertThat(record.getClassification()).isEqualTo(DataClassification.PHI);
page.observe(ModelPart.toolUse("1", "describe_table", Map.of("table", "page")), new AiToolResult());
assertThat(page.getClassification()).isEqualTo(DataClassification.PUBLIC);
page.observe(ModelPart.toolUse("present", "present_results", Map.of("summary", "Available results")), new AiToolResult());
assertThat(page.getClassification()).isEqualTo(DataClassification.PUBLIC);
record.observe(ModelPart.toolUse("present", "present_results", Map.of("summary", "Available results")), new AiToolResult());
assertThat(record.getClassification()).isEqualTo(DataClassification.PHI);
page.observe(ModelPart.toolUse("2", "search_records", Map.of()), new AiToolResult().withModelContent("<qqq_data>{\"results\":[{\"table\":\"media\"}]}</qqq_data>"));
assertThat(page.getClassification()).isEqualTo(DataClassification.PII);
page.observe(ModelPart.toolUse("3", "search_records", Map.of()), new AiToolResult());
@@ -0,0 +1,534 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.api.AssistantApiException;
import com.kingsrook.qbits.assistant.home.AssistantHomeData;
import com.kingsrook.qbits.assistant.home.AssistantHomeView;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantBudgetEvent;
import com.kingsrook.qbits.assistant.model.AssistantBudgetOverride;
import com.kingsrook.qbits.assistant.model.AssistantBudgetUsage;
import com.kingsrook.qbits.assistant.model.AssistantConversation;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qbits.assistant.provider.ModelUsage;
import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.LoggingAlertSink;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceWithProperties;
import com.kingsrook.qqq.backend.core.model.metadata.code.InitializableViaCodeReference;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import java.math.BigDecimal;
import java.time.Clock;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Budget boundaries through the fixture store and scripted model dispatch.
*******************************************************************************/
class BudgetServiceTest extends BaseAssistantTest
{
private static final String USER = "editor-budget";
private static final Instant NOW = Instant.parse("2026-10-13T12:00:00Z");
/*******************************************************************************
** Keyless test adapter.
*******************************************************************************/
@Override
protected void customizeConfig(AssistantQBitConfig config)
{
config.withProviderType(new BudgetProviderType());
}
/*******************************************************************************
** Minimal adapter sharing the fixture scripted provider.
*******************************************************************************/
class BudgetProviderType implements AssistantProviderType
{
/*******************************************************************************
** Provider type.
*******************************************************************************/
public String getType()
{
return ("budget-test");
}
/*******************************************************************************
** Provider label.
*******************************************************************************/
public String getLabel()
{
return ("Budget test");
}
/*******************************************************************************
** Models.
*******************************************************************************/
public List<String> getModels()
{
return (List.of("main", "cheap"));
}
/*******************************************************************************
** Keyless.
*******************************************************************************/
public String getDefaultApiKeyEnvName()
{
return (null);
}
/*******************************************************************************
** No network.
*******************************************************************************/
public String getDefaultBaseUrl()
{
return ("https://budget.invalid");
}
/*******************************************************************************
** Scripted adapter.
*******************************************************************************/
public AssistantModelProvider create(String key, String url)
{
return (provider);
}
}
/*******************************************************************************
** Admin store context.
*******************************************************************************/
private AssistantStore store()
{
QContext.init(qInstance, AssistantTestFixture.ownerSession("admin-budget", config));
return (config.getRuntime().getStore());
}
/*******************************************************************************
** Default per-user daily USD budget.
*******************************************************************************/
private AssistantBudget budget() throws Exception
{
return (store().insert(new AssistantBudget().withName("Per user daily").withScopeType("USER").withScopeValue(USER)
.withPeriod("DAY").withMetric("COST_USD").withLimitValue(BigDecimal.ONE)));
}
/*******************************************************************************
** Normal usage insertion, with a deterministic date.
*******************************************************************************/
private void usage(long micros, Instant at) throws Exception
{
AssistantUsage row = store().insert(new AssistantUsage().withUserId(USER).withPurpose("TURN").withStatus("OK").withCostMicros(micros)
.withInputTokens(micros).withOutputTokens(0L).withCreateDate(at));
store().updateFields(AssistantUsage.TABLE_NAME, new QRecord().withValue("id", row.getId()).withValue("createDate", at));
}
/*******************************************************************************
** Fixed-clock evaluator with the production core sink configuration.
*******************************************************************************/
private BudgetService service(List<AlertSignal> alerts)
{
AlertsInstanceMetaData.ofOrWithNew(qInstance).withSink(new QCodeReferenceWithProperties(TestAlertSink.class, Map.of("signals", (ArrayList<AlertSignal>) alerts)));
return (new BudgetService(store(), config.getRuntime().getProviderService(), Clock.fixed(NOW, ZoneOffset.UTC)));
}
/*******************************************************************************
** Core loads a separate sink per instance; captured signals stay test-local.
*******************************************************************************/
public static class TestAlertSink extends LoggingAlertSink implements InitializableViaCodeReference
{
private List<AlertSignal> signals;
/*******************************************************************************
** Receive this test's capture list from the configured code reference.
*******************************************************************************/
@Override
@SuppressWarnings("unchecked")
public void initialize(QCodeReference codeReference)
{
signals = (List<AlertSignal>) ((QCodeReferenceWithProperties) codeReference).getProperties().get("signals");
}
/*******************************************************************************
** Capture the signal delivered through core.
*******************************************************************************/
@Override
public void raise(AlertSignal signal)
{
signals.add(signal);
}
}
/*******************************************************************************
** 79 percent is quiet; 80 warns and deduplicates event, notice and alert.
*******************************************************************************/
@Test
void warningBoundaryAndDedup() throws Exception
{
AssistantBudget budget = budget();
usage(790_000, NOW);
List<AlertSignal> alerts = new ArrayList<>();
BudgetService service = service(alerts);
assertThat(service.evaluate(USER, null, false, false).notices()).isEmpty();
usage(10_000, NOW);
assertThat(service.evaluate(USER, null, false, false).notices()).singleElement().asString().contains("80%", "Per user daily");
assertThat(service.evaluate(USER, null, false, false).notices()).isEmpty();
assertThat(alerts).singleElement().satisfies(signal ->
{
assertThat(signal.getAlertTypeName()).isEqualTo("assistant.budget.warning");
assertThat(signal.getDedupKey()).isEqualTo(budget.getId() + ":USER:" + USER + ":2026-10-13T00:00:00Z");
assertThat(signal.getSeverity()).isEqualTo(AlertSeverity.WARNING);
assertThat(signal.getTitle()).isEqualTo("Assistant budget Per user daily");
assertThat(signal.getSummary()).contains("80%", "Per user daily");
});
assertThat(store().query(AssistantBudgetEvent.TABLE_NAME, new QQueryFilter(), AssistantBudgetEvent.class)).singleElement()
.extracting(AssistantBudgetEvent::getState).isEqualTo("WARNED");
}
/*******************************************************************************
** Explicit and automatic downgrade selection, including PHI eligibility.
*******************************************************************************/
@Test
void downgradeAndPhi() throws Exception
{
AssistantBudget budget = budget();
AssistantProvider cloud = store().insert(new AssistantProvider().withName("Cloud").withProviderType("budget-test").withModel("main").withEnabled(true).withIsDefault(true));
AssistantProvider local = store().insert(new AssistantProvider().withName("Local").withProviderType("budget-test").withLocation("LOCAL").withModel("cheap").withEnabled(true));
AssistantModel cheap = store().insert(new AssistantModel().withProviderId(local.getId()).withModelId("cheap").withEnabled(true).withInputUsdPerMTok(BigDecimal.ONE));
store().insert(new AssistantModel().withProviderId(cloud.getId()).withModelId("main").withEnabled(true).withInputUsdPerMTok(BigDecimal.ZERO));
usage(1_000_000, NOW);
List<AlertSignal> alerts = new ArrayList<>();
BudgetService service = service(alerts);
ProviderService.Active active = config.getRuntime().getProviderService().resolve("main");
assertThat(service.evaluate(USER, active, false, false).active().providerId()).isEqualTo(local.getId());
assertThat(service.evaluate(USER, active, true, false).stopped()).isTrue();
assertThat(alerts).extracting(AlertSignal::getAlertTypeName).containsExactly("assistant.budget.downgraded", "assistant.budget.stopped");
assertThat(alerts).extracting(AlertSignal::getSeverity).containsExactly(AlertSeverity.WARNING, AlertSeverity.ERROR);
assertThat(alerts).extracting(AlertSignal::getDedupKey).containsOnly(budget.getId() + ":USER:" + USER + ":2026-10-13T00:00:00Z");
store().updateFields(AssistantBudget.TABLE_NAME, new QRecord().withValue("id", budget.getId()).withValue("downgradeModelId", cheap.getId()));
assertThat(service.evaluate(USER, active, false, false).active().model()).isEqualTo("cheap");
}
/*******************************************************************************
** Hard ceiling always stops, and the API uses its existing 429 path.
*******************************************************************************/
@Test
void ceilingAndStartTurn429() throws Exception
{
budget();
usage(1_200_000, NOW);
assertThat(service(new ArrayList<>()).evaluate(USER, null, false, false).stopped()).isTrue();
usage(1_200_000, Instant.now());
assertThatThrownBy(() -> runner.run(USER, "Hello")).isInstanceOf(AssistantApiException.class)
.hasMessageContaining("Per user daily").satisfies(e ->
{
assertThat(((AssistantApiException) e).getStatus()).isEqualTo(429);
assertThat(((AssistantApiException) e).getCode()).isEqualTo("BUDGET");
});
}
/*******************************************************************************
** Additive, unlimited and expired overrides; grantor cannot be supplied.
*******************************************************************************/
@Test
void overridesAndPermission() throws Exception
{
AssistantBudget budget = budget();
usage(1_100_000, NOW);
AssistantBudgetOverride extra = store().insert(new AssistantBudgetOverride().withBudgetId(budget.getId()).withExtraValue(BigDecimal.ONE)
.withReason("Project launch").withExpiresAt(NOW.plusSeconds(3600)).withGrantedBy("forged"));
assertThat(extra.getGrantedBy()).isEqualTo("admin-budget");
BudgetService service = service(new ArrayList<>());
assertThat(service.snapshot(budget, USER).limit()).isEqualByComparingTo("2");
store().insert(new AssistantBudgetOverride().withBudgetId(budget.getId()).withUnlimitedUntil(NOW.plusSeconds(60))
.withExpiresAt(NOW.plusSeconds(60)).withReason("Investigation"));
assertThat(service.snapshot(budget, USER).state()).isEqualTo("UNLIMITED");
BudgetService later = new BudgetService(store(), config.getRuntime().getProviderService(), Clock.fixed(NOW.plusSeconds(3601), ZoneOffset.UTC));
assertThat(later.snapshot(budget, USER).limit()).isEqualByComparingTo("1");
AssistantStore store = store();
QContext.init(qInstance, AssistantTestFixture.ownerSession(USER, config));
assertThatThrownBy(() -> store.insert(new AssistantBudgetOverride().withBudgetId(budget.getId()).withExtraValue(BigDecimal.ONE)
.withExpiresAt(NOW.plusSeconds(60)).withReason("No permission"))).hasMessageContaining("assistant.admin");
}
/*******************************************************************************
** Monthly UTC boundaries and token accounting include all token counters.
*******************************************************************************/
@Test
void monthAndTokens() throws Exception
{
AssistantBudget budget = store().insert(new AssistantBudget().withName("Month").withScopeType("INSTANCE").withPeriod("MONTH")
.withMetric("TOKENS").withLimitValue(BigDecimal.valueOf(100)).withAtLimitAction("STOP"));
usage(500, Instant.parse("2026-09-30T23:59:59Z"));
usage(50, Instant.parse("2026-10-01T00:00:00Z"));
usage(500, Instant.parse("2026-11-01T00:00:00Z"));
assertThat(service(new ArrayList<>()).snapshot(budget, USER).used()).isEqualByComparingTo("50");
}
/*******************************************************************************
** A stop dominates a warning or downgrade in another applicable budget.
*******************************************************************************/
@Test
void strictestWinsAndRoleMembership() throws Exception
{
budget();
AssistantBudget role = store().insert(new AssistantBudget().withName("Role").withScopeType("ROLE").withScopeValue("assistant.admin")
.withPeriod("DAY").withMetric("TOKENS").withLimitValue(BigDecimal.TEN).withAtLimitAction("STOP"));
usage(800_000, NOW);
BudgetService service = service(new ArrayList<>());
assertThat(service.evaluate(USER, null, false, false).stopped()).isTrue();
QContext.init(qInstance, AssistantTestFixture.ownerSession(USER, config));
assertThat(service.evaluate(USER, null, false, false).stopped()).isFalse();
store().updateFields(AssistantBudget.TABLE_NAME, new QRecord().withValue("id", role.getId()).withValue("scopeType", "INSTANCE").withValue("scopeValue", ""));
assertThat(service.evaluate(USER, null, false, false).stopped()).isTrue();
}
/*******************************************************************************
** App home has budget and time-series data and honest empty states.
*******************************************************************************/
@Test
void homeDataAndEmptyStates() throws Exception
{
store();
String empty = AssistantHomeView.render(AssistantHomeData.load(config.getRuntime()), "/app/assistantAdmin");
assertThat(empty).contains("No enabled data-driven budgets", "No assistant use", "No provider usage");
budget();
usage(800_000, Instant.now());
String html = AssistantHomeView.render(AssistantHomeData.load(config.getRuntime()), "/app/assistantAdmin");
assertThat(html).contains("Budgets", "Per user daily", "Usage over time", "Daily tokens", "Daily cost", "By provider");
assertThat(html).doesNotContain("<table", "overflow:auto");
}
/*******************************************************************************
** Next actual model dispatch uses the explicit downgrade model.
*******************************************************************************/
@Test
void turnDispatchUsesDowngradeModel() throws Exception
{
AssistantBudget budget = budget();
AssistantProvider row = store().insert(new AssistantProvider().withName("Local").withProviderType("budget-test").withModel("main").withLocation("LOCAL").withEnabled(true));
AssistantModel cheap = store().insert(new AssistantModel().withProviderId(row.getId()).withModelId("cheap").withEnabled(true));
store().updateFields(AssistantBudget.TABLE_NAME, new QRecord().withValue("id", budget.getId()).withValue("downgradeModelId", cheap.getId()));
usage(1_000_000, Instant.now());
provider.thenText("Done");
assertThat(runner.run(USER, "Hello").errorCode()).isNull();
assertThat(provider.getRequests()).singleElement().extracting(r -> r.getModel()).isEqualTo("cheap");
}
/*******************************************************************************
** Provider and model scopes isolate the other provider's same-named model.
*******************************************************************************/
@Test
void providerAndModelScopes() throws Exception
{
AssistantProvider row = store().insert(new AssistantProvider().withName("Scope provider").withProviderType("budget-test").withModel("main").withEnabled(true));
AssistantModel model = store().insert(new AssistantModel().withProviderId(row.getId()).withModelId("main").withEnabled(true));
AssistantBudget budget = store().insert(new AssistantBudget().withName("Provider").withScopeType("PROVIDER").withScopeValue(row.getId().toString())
.withPeriod("DAY").withMetric("COST_USD").withLimitValue(BigDecimal.ONE).withAtLimitAction("STOP"));
AssistantUsage usage = store().insert(new AssistantUsage().withUserId(USER).withProviderId(row.getId()).withModel("main").withCostMicros(1_000_000L));
store().updateFields(AssistantUsage.TABLE_NAME, new QRecord().withValue("id", usage.getId()).withValue("createDate", NOW));
BudgetService service = service(new ArrayList<>());
ProviderService.Active active = config.getRuntime().getProviderService().resolve("main");
assertThat(service.evaluate(USER, active, false, false).stopped()).isTrue();
assertThat(service.evaluate(USER, null, false, false).stopped()).isFalse();
store().updateFields(AssistantBudget.TABLE_NAME, new QRecord().withValue("id", budget.getId()).withValue("scopeType", "MODEL").withValue("scopeValue", model.getId().toString()));
assertThat(service.evaluate(USER, active, false, false).stopped()).isTrue();
assertThat(service.evaluate(USER, new ProviderService.Active(row, "budget-test", "another-model", provider), false, false).stopped()).isFalse();
}
/*******************************************************************************
** Usage from the first step stops the next dispatch with BUDGET.
*******************************************************************************/
@Test
void midTurnStop() throws Exception
{
store().insert(new AssistantBudget().withName("Token steps").withScopeType("USER").withScopeValue(USER)
.withPeriod("DAY").withMetric("TOKENS").withLimitValue(BigDecimal.valueOf(100)).withAtLimitAction("STOP"));
provider.withUsagePerCall(new ModelUsage().withInputTokens(100L).withOutputTokens(0L));
provider.thenToolUse("count_records", Map.of("table", "page")).thenText("Should not dispatch");
assertThat(runner.run(USER, "Count pages").errorCode()).isEqualTo("BUDGET");
assertThat(provider.getRequests()).hasSize(1);
}
/*******************************************************************************
** The start-turn warning reaches the panel once and is not replayed.
*******************************************************************************/
@Test
void panelWarningOnce() throws Exception
{
budget();
usage(800_000, Instant.now());
provider.thenText("Hello").thenText("Hello again");
assertThat(runner.run(USER, "Hello").events(TurnEvent.NOTICE)).hasSize(1);
assertThat(runner.run(USER, "Hello again").events(TurnEvent.NOTICE)).isEmpty();
}
/*******************************************************************************
** Another user's cost counts against shared budgets without unlocking detail.
*******************************************************************************/
@Test
void sharedBudgetKeepsUsageOwnerLocks() throws Exception
{
store().insert(new AssistantBudget().withName("Whole instance").withScopeType("INSTANCE").withPeriod("DAY")
.withMetric("COST_USD").withLimitValue(BigDecimal.ONE).withAtLimitAction("STOP"));
usage(1_000_000, NOW);
BudgetService service = service(new ArrayList<>());
AssistantStore store = store();
QContext.init(qInstance, AssistantTestFixture.ownerSession("editor-other", config));
assertThat(store.query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class)).isEmpty();
assertThat(service.evaluate("editor-other", null, false, false).stopped()).isTrue();
assertThat(qInstance.getTable(AssistantBudgetUsage.TABLE_NAME).getRecordSecurityLocks()).isNullOrEmpty();
assertThat(qInstance.getTable(AssistantUsage.TABLE_NAME).getRecordSecurityLocks()).isNotEmpty();
}
/*******************************************************************************
** A provider STOP is not skipped after a user budget selects another provider.
*******************************************************************************/
@Test
void originalProviderStopWinsOverDowngrade() throws Exception
{
AssistantBudget user = budget();
AssistantProvider original = store().insert(new AssistantProvider().withName("Original").withProviderType("budget-test").withModel("main").withEnabled(true).withIsDefault(true));
AssistantProvider local = store().insert(new AssistantProvider().withName("Target").withProviderType("budget-test").withModel("cheap").withEnabled(true).withLocation("LOCAL"));
AssistantModel model = store().insert(new AssistantModel().withProviderId(local.getId()).withModelId("cheap").withEnabled(true));
store().updateFields(AssistantBudget.TABLE_NAME, new QRecord().withValue("id", user.getId()).withValue("downgradeModelId", model.getId()));
store().insert(new AssistantBudget().withName("Provider stop").withScopeType("PROVIDER").withScopeValue(original.getId().toString())
.withPeriod("DAY").withMetric("COST_USD").withLimitValue(BigDecimal.ONE).withAtLimitAction("STOP"));
AssistantUsage usage = store().insert(new AssistantUsage().withUserId(USER).withProviderId(original.getId()).withModel("main").withCostMicros(1_000_000L));
store().updateFields(AssistantUsage.TABLE_NAME, new QRecord().withValue("id", usage.getId()).withValue("createDate", NOW));
BudgetService service = service(new ArrayList<>());
assertThat(service.evaluate(USER, config.getRuntime().getProviderService().resolve("main"), false, false).reason()).contains("Provider stop");
assertThat(store().query(AssistantBudgetEvent.TABLE_NAME, new QQueryFilter(), AssistantBudgetEvent.class)).extracting(AssistantBudgetEvent::getState).containsExactly("STOPPED");
}
/*******************************************************************************
** A PHI conversation cannot downgrade back to its original model or to an
** ineligible candidate; no model receives the request.
*******************************************************************************/
@Test
void phiConversationWithoutEligibleDowngradeStops() throws Exception
{
budget();
AssistantProvider original = store().insert(new AssistantProvider().withName("Eligible original").withProviderType("budget-test")
.withModel("main").withEnabled(true).withIsDefault(true).withHipaaEligible(true));
AssistantProvider target = store().insert(new AssistantProvider().withName("Ineligible target").withProviderType("budget-test")
.withModel("cheap").withEnabled(true).withLocation("LOCAL").withHipaaEligible(false));
store().insert(new AssistantModel().withProviderId(original.getId()).withModelId("main").withEnabled(true));
store().insert(new AssistantModel().withProviderId(target.getId()).withModelId("cheap").withEnabled(true));
AssistantConversation conversation = store().insert(new AssistantConversation().withUserId(USER).withTitle("Classified")
.withMaxClassification("PHI").withStatus("ACTIVE"));
usage(1_000_000, Instant.now());
assertThat(runner.run(USER, conversation.getId(), "Continue", Map.of()).errorCode()).isEqualTo("BUDGET");
assertThat(provider.getRequests()).isEmpty();
}
/*******************************************************************************
** Partial edits validate the full row; narrow overrides affect only their user.
*******************************************************************************/
@Test
void validationAndNarrowOverrides() throws Exception
{
AssistantBudget budget = store().insert(new AssistantBudget().withName("Shared").withScopeType("INSTANCE").withPeriod("DAY")
.withMetric("COST_USD").withLimitValue(BigDecimal.ONE));
assertThatThrownBy(() -> store().updateFields(AssistantBudget.TABLE_NAME,
new QRecord().withValue("id", budget.getId()).withValue("hardCeilingPercent", 99))).hasMessageContaining("hard ceiling");
store().insert(new AssistantBudgetOverride().withBudgetId(budget.getId()).withScopeValue(USER).withExtraValue(BigDecimal.ONE)
.withExpiresAt(NOW.plusSeconds(3600)).withReason("One user"));
BudgetService service = service(new ArrayList<>());
assertThat(service.snapshot(budget, USER).limit()).isEqualByComparingTo("2");
assertThat(service.snapshot(budget, "editor-other").limit()).isEqualByComparingTo("1");
assertThatThrownBy(() -> store().insert(new AssistantBudget().withName("Invalid provider").withScopeType("PROVIDER")
.withScopeValue("missing").withPeriod("DAY").withMetric("TOKENS").withLimitValue(BigDecimal.TEN))).hasMessageContaining("numeric");
}
}
@@ -0,0 +1,784 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import com.kingsrook.qbits.assistant.model.AssistantPendingAction;
import java.math.BigDecimal;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.AssistantRuntime;
import com.kingsrook.qbits.assistant.classification.DataClassification;
import com.kingsrook.qbits.assistant.home.AssistantHomeData;
import com.kingsrook.qbits.assistant.home.AssistantHomeView;
import com.kingsrook.qbits.assistant.model.AssistantBudget;
import com.kingsrook.qbits.assistant.model.AssistantModel;
import com.kingsrook.qbits.assistant.model.AssistantProfile;
import com.kingsrook.qbits.assistant.model.AssistantProvider;
import com.kingsrook.qbits.assistant.model.AssistantTurn;
import com.kingsrook.qbits.assistant.model.AssistantUsage;
import com.kingsrook.qbits.assistant.persistence.AssistantStore;
import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
import com.kingsrook.qbits.assistant.provider.CompletableStreamHandle;
import com.kingsrook.qbits.assistant.provider.HealthResult;
import com.kingsrook.qbits.assistant.provider.ModelEventListener;
import com.kingsrook.qbits.assistant.provider.ModelPart;
import com.kingsrook.qbits.assistant.provider.ModelProviderException;
import com.kingsrook.qbits.assistant.provider.ModelRequest;
import com.kingsrook.qbits.assistant.provider.ModelResponse;
import com.kingsrook.qbits.assistant.provider.ProviderSettings;
import com.kingsrook.qbits.assistant.provider.StreamHandle;
import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qbits.assistant.testing.TurnRunner;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import org.junit.jupiter.params.provider.EnumSource;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Multi-provider dispatch through the fixture with persisted rows and fixed time.
*******************************************************************************/
class RoutingFallbackTest extends BaseAssistantTest
{
private static final String USER = "editor-routing";
private static final Instant NOW = Instant.now();
private final Map<String, ScriptedModelProvider> scripts = new LinkedHashMap<>();
private Integer probes = 0;
private Boolean healthy = true;
/*******************************************************************************
** Keyless adapter with distinct scripts for each endpoint.
*******************************************************************************/
@Override
protected void customizeConfig(AssistantQBitConfig value)
{
value.withProviderType(new RoutingType()).withAllowedBaseUrls(Set.of("https://a.invalid", "https://b.invalid", "https://c.invalid"));
}
/*******************************************************************************
** Fix time for budgets, routing and health.
*******************************************************************************/
@BeforeEach
void fixedClock()
{
config.getRuntime().close();
config.setRuntime(new AssistantRuntime(qInstance, config, Clock.fixed(NOW, ZoneOffset.UTC)));
}
/*******************************************************************************
** One adapter family with separate recording instances.
*******************************************************************************/
class RoutingType implements AssistantProviderType
{
/*******************************************************************************
** Type identifier.
*******************************************************************************/
public String getType()
{
return ("routing-test");
}
/*******************************************************************************
** Display label.
*******************************************************************************/
public String getLabel()
{
return ("Routing test");
}
/*******************************************************************************
** Registry supplies actual models.
*******************************************************************************/
public List<String> getModels()
{
return (List.of("a", "b", "c"));
}
/*******************************************************************************
** No secret needed.
*******************************************************************************/
public String getDefaultApiKeyEnvName()
{
return (null);
}
/*******************************************************************************
** No network traffic.
*******************************************************************************/
public String getDefaultBaseUrl()
{
return ("https://a.invalid");
}
/*******************************************************************************
** Resolve the endpoint's own script.
*******************************************************************************/
public AssistantModelProvider create(String key, String url)
{
return (scripts.get(url == null ? "https://a.invalid" : url));
}
/*******************************************************************************
** Count probes and assert the transport timeout.
*******************************************************************************/
public HealthResult checkHealth(ProviderSettings settings)
{
probes++;
assertThat(settings.getRequestTimeoutSeconds()).isEqualTo(2);
return (new HealthResult(healthy, 1L, "Test probe"));
}
}
/*******************************************************************************
** Admin setup; actual turns use the editor session.
*******************************************************************************/
private AssistantStore store()
{
QContext.init(qInstance, AssistantTestFixture.ownerSession("admin-routing", config));
return (config.getRuntime().getStore());
}
/*******************************************************************************
** One enabled registry model and endpoint.
*******************************************************************************/
private AssistantModel model(String name, String location, Boolean eligible) throws Exception
{
scripts.put("https://" + name + ".invalid", new ScriptedModelProvider(name));
AssistantProvider row = store().insert(new AssistantProvider().withName(name.toUpperCase()).withProviderType("routing-test")
.withBaseUrl("https://" + name + ".invalid").withModel(name).withEnabled(true).withIsDefault("a".equals(name)).withLocation(location).withHipaaEligible(eligible));
return (store().insert(new AssistantModel().withProviderId(row.getId()).withModelId(name).withEnabled(true)));
}
/*******************************************************************************
** Retrieve a recording endpoint.
*******************************************************************************/
private ScriptedModelProvider script(String name)
{
return (scripts.get("https://" + name + ".invalid"));
}
/*******************************************************************************
** Save through the real validation path.
*******************************************************************************/
private void link(AssistantModel from, AssistantModel to) throws Exception
{
store().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", from.getId()).withValue("fallbackModelId", to.getId()));
}
/*******************************************************************************
** Default profile for this fixture.
*******************************************************************************/
private void profile(String mode, AssistantModel simple, AssistantModel complex, AssistantModel classifier) throws Exception
{
store().insert(new AssistantProfile().withName("routing").withDisplayName("Assistant").withIsDefault(true).withEnabled(true).withRoutingMode(mode)
.withSimpleModelId(simple.getId()).withComplexModelId(complex.getId()).withClassifierModelId(classifier == null ? null : classifier.getId()));
}
/*******************************************************************************
** Read all terminal fields.
*******************************************************************************/
private AssistantTurn turn(TurnRunner.Outcome outcome) throws Exception
{
return (store().getTurn(USER, outcome.turnId()));
}
/*******************************************************************************
** Each keyword and negative word-boundary case.
*******************************************************************************/
@ParameterizedTest
@CsvSource({"analyze sales,COMPLEX", "compare months,COMPLEX", "why,COMPLEX", "trend,COMPLEX", "explain this,COMPLEX", "summarize across stores,COMPLEX", "plan work,COMPLEX", "planet,SIMPLE", "hello,SIMPLE", "blog_post page,COMPLEX", "blog_post blog_post,SIMPLE", "blog_posts,SIMPLE"})
void heuristicWords(String question, String expected)
{
assertThat(ModelRouter.heuristic(question, Set.of("blog_post", "page"), false, 8, false)).isEqualTo(expected);
}
/*******************************************************************************
** Exact length/turn boundaries, pending writes, and tool exhaustion.
*******************************************************************************/
@Test
void heuristicBoundaries()
{
assertThat(ModelRouter.heuristic("x".repeat(600), Set.of(), false, 8, false)).isEqualTo("SIMPLE");
assertThat(ModelRouter.heuristic("x".repeat(601), Set.of(), false, 8, false)).isEqualTo("COMPLEX");
assertThat(ModelRouter.heuristic("hi", Set.of(), true, 0, false)).isEqualTo("COMPLEX");
assertThat(ModelRouter.heuristic("hi", Set.of(), false, 9, false)).isEqualTo("COMPLEX");
assertThat(ModelRouter.heuristic("hi", Set.of(), false, 0, true)).isEqualTo("COMPLEX");
assertThat(ModelRouter.heuristic(null, Set.of(), false, 0, false)).isEqualTo("SIMPLE");
}
/*******************************************************************************
** Exact classifier replies select a route; garbage uses the heuristic.
*******************************************************************************/
@ParameterizedTest
@CsvSource({"SIMPLE,a,CLASSIFIER", "COMPLEX,b,CLASSIFIER", "garbage,b,HEURISTIC"})
void classifierAnswers(String answer, String selected, String source) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
store().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", c.getId()).withValue("supportsTools", false));
profile("CLASSIFIER", a, b, c);
script("c").thenText(answer);
script(selected).thenText("Answer");
TurnRunner.Outcome result = runner.run(USER, "explain this");
assertThat(result.text()).isEqualTo("Answer");
assertThat(turn(result).getModel()).isEqualTo(selected);
assertThat(turn(result).getRouteSource()).isEqualTo(source);
ModelRequest request = script("c").getRequests().get(0);
assertThat(request.getMaxTokens()).isEqualTo(4);
assertThat(request.getStreaming()).isFalse();
assertThat(request.getTools()).isEmpty();
assertThat(request.getSystem()).hasSize(1);
assertThat(request.getSystem().get(0).getText()).isEqualTo(ModelRouter.CLASSIFIER_PROMPT);
assertThat(request.getMessages()).hasSize(1);
assertThat(request.getMessages().get(0).getParts()).hasSize(1);
assertThat(request.getMessages().get(0).getParts().get(0).getText()).isEqualTo("explain this");
List<AssistantUsage> rows = store().query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class);
assertThat(rows).extracting(AssistantUsage::getPurpose).containsExactly("ROUTE", "TURN");
assertThat(turn(result).getInputTokens()).isEqualTo(2400L);
}
/*******************************************************************************
** Error categories leave the heuristic in charge and record ROUTE failures.
*******************************************************************************/
@ParameterizedTest
@EnumSource(value = ModelProviderException.Kind.class, names = {"TIMEOUT", "SERVER"})
void classifierErrors(ModelProviderException.Kind kind) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
profile("CLASSIFIER", a, b, c);
script("c").thenError(kind, "unavailable");
script("b").thenText("Answer");
TurnRunner.Outcome result = runner.run(USER, "why");
assertThat(turn(result).getRouteSource()).isEqualTo("HEURISTIC");
assertThat(result.text()).isEqualTo("Answer");
assertThat(store().query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class)).anyMatch(row -> "ROUTE".equals(row.getPurpose()) && "ERROR".equals(row.getStatus()));
}
/*******************************************************************************
** Real wait timeout cancels the classifier and opens its circuit.
*******************************************************************************/
@Test
void classifierDeadline() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
profile("CLASSIFIER", a, b, c);
script("c").withDelay(Duration.ofSeconds(10)).thenText("SIMPLE");
script("b").thenText("Answer");
Long start = System.nanoTime();
TurnRunner.Outcome result = runner.run(USER, "why");
assertThat(Duration.ofNanos(System.nanoTime() - start)).isLessThan(Duration.ofSeconds(8));
assertThat(turn(result).getRouteSource()).isEqualTo("HEURISTIC");
assertThat(store().getProvider(c.getProviderId()).getHealthState()).isEqualTo("DOWN");
}
/*******************************************************************************
** PHI cannot reach an ineligible classifier.
*******************************************************************************/
@Test
void phiSkipsClassifier() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", false);
profile("CLASSIFIER", a, b, c);
config.withDataClassification("blog_post", DataClassification.PHI);
script("b").thenText("Answer");
TurnRunner.Outcome result = runner.run(USER, null, "why", Map.of("tableName", "blog_post"));
assertThat(script("c").getRequests()).isEmpty();
assertThat(turn(result).getRouteSource()).isEqualTo("HEURISTIC");
}
/*******************************************************************************
** OFF preserves row selection even for a complex question.
*******************************************************************************/
@Test
void offKeepsDefault() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
profile("OFF", b, b, null);
script("a").thenText("Default");
TurnRunner.Outcome result = runner.run(USER, "why");
assertThat(result.text()).isEqualTo("Default");
assertThat(turn(result).getRouteDecision()).isEqualTo("OFF");
assertThat(turn(result).getRouteSource()).isNull();
}
/*******************************************************************************
** Every retryable category records both attempts and the final destination.
*******************************************************************************/
@ParameterizedTest
@EnumSource(value = ModelProviderException.Kind.class, names = {"NETWORK", "TIMEOUT", "OVERLOADED", "SERVER", "RATE_LIMITED"})
void fallbackBeforeOutput(ModelProviderException.Kind kind) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
link(a, b);
script("a").thenError(kind, "unavailable");
script("b").thenText("Recovered");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.text()).isEqualTo("Recovered");
assertThat(turn(result).getFallbackCount()).isEqualTo(1);
assertThat(turn(result).getProviderId()).isEqualTo(b.getProviderId());
assertThat(turn(result).getModel()).isEqualTo("b");
assertThat(result.events(TurnEvent.NOTICE).toString()).contains("Provider A is unavailable; answered with B.");
List<AssistantUsage> rows = store().query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class);
assertThat(rows).extracting(AssistantUsage::getStatus).containsExactly("ERROR", "OK");
assertThat(rows.get(0).getErrorKind()).isEqualTo(kind.name());
assertThat(turn(result).getInputTokens()).isEqualTo(2400L);
}
/*******************************************************************************
** Permanent errors cannot use another provider.
*******************************************************************************/
@ParameterizedTest
@EnumSource(value = ModelProviderException.Kind.class, names = {"INVALID_REQUEST", "AUTHENTICATION", "OTHER"})
void noFallbackForPermanentError(ModelProviderException.Kind kind) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
link(a, b);
script("a").thenError(kind, "bad request");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.end()).containsEntry("status", "ERROR");
assertThat(script("b").getRequests()).isEmpty();
}
/*******************************************************************************
** Three models, two failures, one final answer.
*******************************************************************************/
@Test
void threeModelChain() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
link(a, b);
link(b, c);
script("a").thenError(ModelProviderException.Kind.OVERLOADED, "busy");
script("b").thenError(ModelProviderException.Kind.SERVER, "busy");
script("c").thenText("Recovered");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.text()).isEqualTo("Recovered");
assertThat(turn(result).getFallbackCount()).isEqualTo(2);
}
/*******************************************************************************
** Real save validation rejects cycles and disabled destinations.
*******************************************************************************/
@Test
void rejectsInvalidChains() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
link(a, b);
assertThatThrownBy(() -> link(b, a)).hasMessageContaining("cycle");
assertThatThrownBy(() -> link(a, a)).hasMessageContaining("cycle");
AssistantModel c = model("c", "CLOUD", true);
store().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", c.getId()).withValue("enabled", false));
assertThatThrownBy(() -> link(a, c)).hasMessageContaining("enabled");
Map<Integer, AssistantModel> graph = new LinkedHashMap<>();
for(Integer i = 1; i <= 6; i++)
{
graph.put(i, new AssistantModel().withId(i).withEnabled(true).withFallbackModelId(i == 6 ? null : i + 1));
}
assertThat(FallbackChain.validate(graph.get(1), graph)).contains("four hops");
assertThat(FallbackChain.validate(graph.get(2), graph)).isNull();
}
/*******************************************************************************
** Skip unavailable intermediate endpoints but follow their next edge.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"disabled", "down", "disabledModel"})
void skipsUnavailableCandidate(String state) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
link(a, b);
link(b, c);
if("disabledModel".equals(state))
{
store().updateFields(AssistantModel.TABLE_NAME, new QRecord().withValue("id", b.getId()).withValue("enabled", false));
}
else
{
store().updateFields(AssistantProvider.TABLE_NAME, "disabled".equals(state)
? new QRecord().withValue("id", b.getProviderId()).withValue("enabled", false)
: new QRecord().withValue("id", b.getProviderId()).withValue("healthState", "DOWN").withValue("downUntil", NOW.plusSeconds(60)));
}
script("a").thenError(ModelProviderException.Kind.SERVER, "busy");
script("c").thenText("Recovered");
assertThat(runner.run(USER, "hello").text()).isEqualTo("Recovered");
assertThat(script("b").getRequests()).isEmpty();
}
/*******************************************************************************
** Explicit links never override PHI eligibility.
*******************************************************************************/
@Test
void phiNeverFallsToIneligible() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", false);
link(a, b);
config.withDataClassification("blog_post", DataClassification.PHI);
script("a").thenError(ModelProviderException.Kind.OVERLOADED, "busy");
TurnRunner.Outcome result = runner.run(USER, null, "hello", Map.of("tableName", "blog_post"));
assertThat(result.end()).containsEntry("status", "ERROR");
assertThat(script("b").getRequests()).isEmpty();
}
/*******************************************************************************
** Automatic alternatives stay local, even when cloud is available.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"LOCAL", "CLOUD"})
void implicitLocalOnly(String destination) throws Exception
{
model("a", "LOCAL", true);
model("b", destination, true);
script("a").thenError(ModelProviderException.Kind.OVERLOADED, "busy");
script("b").thenText("Recovered");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.end()).containsEntry("status", "LOCAL".equals(destination) ? "COMPLETE" : "ERROR");
assertThat(script("b").getRequests()).hasSize("LOCAL".equals(destination) ? 1 : 0);
}
/*******************************************************************************
** Destination STOP skips a candidate and continues to its successor.
*******************************************************************************/
@Test
void budgetSkipsCandidate() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
link(a, b);
link(b, c);
store().insert(new AssistantBudget().withName("Stop B").withScopeType("MODEL").withScopeValue(String.valueOf(b.getId()))
.withPeriod("DAY").withMetric("TOKENS").withLimitValue(BigDecimal.ONE).withAtLimitAction("STOP"));
store().insert(new AssistantUsage().withUserId(USER).withProviderId(b.getProviderId()).withModel("b").withPurpose("TURN").withStatus("OK").withInputTokens(100L).withOutputTokens(0L).withCreateDate(NOW));
script("a").thenError(ModelProviderException.Kind.SERVER, "busy");
script("c").thenText("Recovered");
assertThat(runner.run(USER, "hello").text()).isEqualTo("Recovered");
assertThat(script("b").getRequests()).isEmpty();
}
/*******************************************************************************
** Circuit expiry, cached probes and successful recovery.
*******************************************************************************/
@Test
void healthCircuitAndProbeCache() throws Exception
{
AssistantModel a = model("a", "LOCAL", true);
ProviderService.Active active = config.getRuntime().getProviderService().resolveModel(a.getId(), false);
ProviderHealth health = new ProviderHealth(config.getRuntime());
health.beforeCall(active);
health.beforeCall(active);
assertThat(probes).isEqualTo(1);
health.failed(active, new ModelProviderException(ModelProviderException.Kind.NETWORK, null, "offline"));
assertThat(store().getProvider(a.getProviderId()).getDownUntil()).isEqualTo(NOW.plusSeconds(60));
assertThatThrownBy(() -> health.beforeCall(active)).isInstanceOf(ModelProviderException.class);
store().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", a.getProviderId()).withValue("downUntil", NOW).withValue("healthCheckedAt", NOW.minusSeconds(61)));
health.beforeCall(active);
assertThat(probes).isEqualTo(2);
assertThat(store().getProvider(a.getProviderId()).getHealthState()).isEqualTo("UP");
health.failed(active, new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "offline"));
health.succeeded(active);
assertThat(store().getProvider(a.getProviderId()).getDownUntil()).isNull();
}
/*******************************************************************************
** Provider-private signatures from a successful tool round cannot reach B.
*******************************************************************************/
@Test
void midTurnSwitchRebuildsNeutralHistory() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
link(a, b);
script("a").thenRespond(request -> new ModelResponse().withParts(new ArrayList<>(List.of(ModelPart.toolUse("tool-1", "count_records", input("table", "blog_post")))))
.withProviderState("private-signature").withStopReason(ModelResponse.STOP_TOOL_USE))
.thenError(ModelProviderException.Kind.OVERLOADED, "busy");
script("b").thenRespond(request ->
{
assertThat(request.getMessages()).allMatch(message -> message.getProviderState() == null);
assertThat(request.getMessages()).anyMatch(message -> message.getParts().stream().anyMatch(part -> part.getType() == ModelPart.Type.TOOL_RESULT));
return (new ModelResponse().withParts(new ArrayList<>(List.of(ModelPart.text("Recovered after tool")))).withStopReason(ModelResponse.STOP_END_TURN));
});
TurnRunner.Outcome result = runner.run(USER, "count posts");
assertThat(result.text()).isEqualTo("Recovered after tool");
assertThat(result.toolCalls()).hasSize(1);
assertThat(turn(result).getFallbackCount()).isEqualTo(1);
assertThat(script("a").getRequests().get(1).getMessages()).anyMatch(message -> "private-signature".equals(message.getProviderState()));
}
/*******************************************************************************
** Even buffered text or a tool start makes the failure terminal.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"text", "tool"})
void partialOutputNeverFallsBack(String output) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
link(a, b);
scripts.put("https://a.invalid", new ScriptedModelProvider("a")
{
/*******************************************************************************
** Emit a partial event, then fail the same call.
*******************************************************************************/
@Override
public StreamHandle stream(ModelRequest request, ModelEventListener listener)
{
if("text".equals(output))
{
listener.onTextDelta("partial");
}
else
{
listener.onToolUseStart("tool-1", "count_records");
}
CompletableStreamHandle handle = new CompletableStreamHandle();
handle.fail(new ModelProviderException(ModelProviderException.Kind.NETWORK, null, "offline"));
return (handle);
}
});
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.end()).containsEntry("status", "ERROR");
assertThat(script("b").getRequests()).isEmpty();
assertThat(turn(result).getFallbackCount()).isZero();
assertThat(store().getProvider(a.getProviderId()).getHealthState()).isEqualTo("DOWN");
if("text".equals(output))
{
assertThat(result.text()).isEqualTo("partial");
}
}
/*******************************************************************************
** Failed local probes open the circuit without sending model requests.
*******************************************************************************/
@Test
void failedProbeAndHomeHealth() throws Exception
{
AssistantModel a = model("a", "LOCAL", true);
healthy = false;
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.end()).containsEntry("status", "ERROR");
assertThat(script("a").getRequests()).isEmpty();
assertThat(store().getProvider(a.getProviderId()).getHealthState()).isEqualTo("DOWN");
String html = AssistantHomeView.render(AssistantHomeData.load(config.getRuntime()), "");
assertThat(html).contains("DOWN", "Down until", "0 fallbacks");
}
/*******************************************************************************
** Classifier tokens count before the answer call can dispatch.
*******************************************************************************/
@Test
void classifierConsumesBudget() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
profile("CLASSIFIER", a, b, c);
store().insert(new AssistantBudget().withName("Small allowance").withScopeType("USER").withScopeValue(USER)
.withPeriod("DAY").withMetric("TOKENS").withLimitValue(BigDecimal.valueOf(100)).withAtLimitAction("STOP"));
script("c").thenText("SIMPLE");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.end()).containsEntry("status", "ERROR");
assertThat(script("a").getRequests()).isEmpty();
assertThat(store().query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class)).extracting(AssistantUsage::getPurpose).containsExactly("ROUTE");
}
/*******************************************************************************
** Conversation context supplies tool exhaustion, pending writes and history.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"tool_limit", "pending", "history"})
void heuristicConversationContext(String context) throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
profile("HEURISTIC", a, b, null);
script("a").thenText("Simple");
TurnRunner.Outcome first = runner.run(USER, "hello");
assertThat(turn(first).getRouteDecision()).isEqualTo("SIMPLE");
if("tool_limit".equals(context))
{
store().updateFields(AssistantTurn.TABLE_NAME, new QRecord().withValue("id", first.turnId()).withValue("stopReason", "tool_limit"));
}
else if("pending".equals(context))
{
store().insert(new AssistantPendingAction().withUserId(USER).withConversationId(first.conversationId()).withStatus("PENDING"));
}
else
{
for(Integer i = 0; i < 8; i++)
{
store().insert(new AssistantTurn().withUserId(USER).withConversationId(first.conversationId()).withStatus("COMPLETE"));
}
}
script("b").thenText("Complex");
TurnRunner.Outcome result = runner.run(USER, first.conversationId(), "hello", null);
assertThat(result.text()).isEqualTo("Complex");
assertThat(turn(result).getRouteDecision()).isEqualTo("COMPLEX");
}
/*******************************************************************************
** Without a profile, escalation supplies the complex model and effort.
*******************************************************************************/
@Test
void codeLevelEscalation() throws Exception
{
config.getModels().withEscalationEnabled(true).withEscalationModel("complex-default").withEscalationEffort("high");
provider.thenText("Complex");
TurnRunner.Outcome result = runner.run(USER, "why");
assertThat(result.text()).isEqualTo("Complex");
assertThat(provider.getRequests().get(0).getModel()).isEqualTo("complex-default");
assertThat(provider.getRequests().get(0).getEffort()).isEqualTo("high");
}
/*******************************************************************************
** Four edges are allowed; a fifth is rejected even when added at the tail.
*******************************************************************************/
@Test
void fourHopBoundary() throws Exception
{
AssistantModel a = model("a", "CLOUD", true);
AssistantModel b = model("b", "CLOUD", true);
AssistantModel c = model("c", "CLOUD", true);
AssistantModel d = store().insert(new AssistantModel().withProviderId(c.getProviderId()).withModelId("d").withEnabled(true));
AssistantModel e = store().insert(new AssistantModel().withProviderId(c.getProviderId()).withModelId("e").withEnabled(true));
AssistantModel f = store().insert(new AssistantModel().withProviderId(c.getProviderId()).withModelId("f").withEnabled(true));
link(a, b);
link(b, c);
link(c, d);
link(d, e);
assertThatThrownBy(() -> link(e, f)).hasMessageContaining("four hops");
script("a").thenError(ModelProviderException.Kind.OVERLOADED, "busy");
script("b").thenError(ModelProviderException.Kind.OVERLOADED, "busy");
script("c").thenError(ModelProviderException.Kind.OVERLOADED, "busy").thenError(ModelProviderException.Kind.OVERLOADED, "busy").thenText("Recovered");
TurnRunner.Outcome result = runner.run(USER, "hello");
assertThat(result.text()).isEqualTo("Recovered");
assertThat(turn(result).getModel()).isEqualTo("e");
assertThat(turn(result).getFallbackCount()).isEqualTo(4);
String html = AssistantHomeView.render(AssistantHomeData.load(config.getRuntime()), "");
assertThat(html).contains("4 fallbacks");
}
}
@@ -17,11 +17,6 @@
package com.kingsrook.qbits.assistant.settings;
import java.io.Serializable;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
import com.kingsrook.qbits.assistant.api.AssistantApiException;
import com.kingsrook.qbits.assistant.engine.ConnectionTest;
@@ -71,6 +66,11 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import java.io.Serializable;
import java.math.BigDecimal;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
@@ -516,7 +516,7 @@ class AssistantSettingsTest extends BaseAssistantTest
asAdmin();
String html = render();
assertThat(html).contains("No providers yet").contains("No assistant use in the last 30 days").contains("No tool calls yet")
.contains("No failed model calls").contains("No one has used the assistant this month").doesNotContain("<table");
.contains("No failed model calls").contains("No one has used the assistant in the last 30 days").doesNotContain("<table");
}
@@ -226,9 +226,13 @@ public class AssistantTestFixture
update("page", new QRecord().withValue("id", 12).withValue("metaDescription", "Who I am and what I build."));
update("page", new QRecord().withValue("id", 12).withValue("content", "Body of page 12, rewritten with a longer story about the home lab."));
insert("media", List.of(
new QRecord().withValue("fileName", "cover.heic").withValue("mimeType", "image/heic"),
new QRecord().withValue("fileName", "logo.png").withValue("mimeType", "image/png").withValue("width", 512)));
List<QRecord> media = new ArrayList<>();
for(Integer i = 0; i < 101; i++)
{
media.add(new QRecord().withValue("fileName", "image-" + i + (i < 81 ? ".jpg" : ".png"))
.withValue("mimeType", i < 81 ? "image/jpeg" : "image/png").withValue("width", 512));
}
insert("media", media);
List<QRecord> stats = new ArrayList<>();
for(int d = 1; d <= 5; d++)
@@ -180,6 +180,7 @@ public class GoldenTasks
},
o -> live(o, "search_records", "/app/blog_post/1")));
tasks.add(PresentationGoldenTask.task());
return (tasks);
}
@@ -0,0 +1,171 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.testing;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.fasterxml.jackson.core.type.TypeReference;
import com.kingsrook.qbits.assistant.engine.TurnEvent;
import com.kingsrook.qbits.assistant.provider.ModelPart;
import com.kingsrook.qbits.assistant.provider.ModelResponse;
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
import static com.kingsrook.qbits.assistant.testing.GoldenTasks.map;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Owner's image breakdown, shared by deterministic and live provider evals.
*******************************************************************************/
public class PresentationGoldenTask
{
/*******************************************************************************
** Counts come from UI API tool results, including the omitted zero types.
*******************************************************************************/
public static GoldenTasks.Task task()
{
return (new GoldenTasks.Task("images-by-type-presentation", AssistantTestFixture.EDITOR, "How many images do I have by type?", provider ->
{
List<Object[]> calls = new ArrayList<>();
for(String mime : List.of("image/jpeg", "image/png", "image/webp", "image/gif", "image/svg+xml"))
{
calls.add(new Object[] {"count_records", map("table", "media", "filter", GoldenTasks.filter("AND", GoldenTasks.criterion("mimeType", "EQUALS", mime)))});
}
provider.thenToolUses(null, calls).thenRespond(request ->
{
List<ModelPart> results = EvalSupport.lastToolResults(request);
Integer jpeg = ((Number) EvalSupport.unwrap(results.get(0).getText()).get("count")).intValue();
Integer png = ((Number) EvalSupport.unwrap(results.get(1).getText()).get("count")).intValue();
for(Integer i = 2; i < results.size(); i++)
{
assertThat(((Number) EvalSupport.unwrap(results.get(i).getText()).get("count")).intValue()).isZero();
}
return (new ModelResponse().withParts(List.of(ModelPart.toolUse("present", "present_results", input(jpeg, png)))).withStopReason(ModelResponse.STOP_TOOL_USE));
}).thenText("");
}, PresentationGoldenTask::check, PresentationGoldenTask::check));
}
/*******************************************************************************
** The presentation input, deliberately using null percentage placeholders.
*******************************************************************************/
public static Map<String, Object> input(Integer jpeg, Integer png)
{
return (map("summary", (jpeg + png) + " media files are images: " + jpeg + " JPEG and " + png + " PNG.",
"metrics", List.of(map("label", "Images", "value", jpeg + png)),
"table", map("title", "Media by type", "columns", List.of(map("label", "Type"), map("label", "Count"), map("label", "Share", "percentOfColumn", 1)),
"rows", List.of(java.util.Arrays.asList("JPEG", jpeg, null), java.util.Arrays.asList("PNG", png, null)),
"rowFilter", map("tableName", "media", "fieldName", "mimeType"), "rowKeys", List.of("image/jpeg", "image/png")),
"chart", map("chartType", "pie", "title", "Share by type", "labelColumn", 0, "valueColumn", 1),
"filterLinks", List.of(filterLink("JPEG", "image/jpeg"), filterLink("PNG", "image/png"))));
}
/*******************************************************************************
** A raw MIME criterion for the presentation tool.
*******************************************************************************/
private static Map<String, Object> filterLink(String label, String mime)
{
return (map("label", "Open " + label + " media", "tableName", "media", "criteria", List.of(map("fieldName", "mimeType", "operator", "EQUALS", "values", List.of(mime)))));
}
/*******************************************************************************
** Check actual block semantics, allowing a live model to choose labels and
** column order. Values, shares and filter criteria must remain exact.
*******************************************************************************/
public static void check(TurnRunner.Outcome outcome)
{
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
assertThat(outcome.toolCalls()).anyMatch(call -> "count_records".equals(call.getToolName()) && "OK".equals(call.getOutcome()));
assertThat(outcome.events(TurnEvent.PRESENTATION)).hasSize(1);
Map<String, Object> part = object(outcome.events(TurnEvent.PRESENTATION).get(0).get("part"));
assertPart(part);
}
/*******************************************************************************
** Also used against the persisted part, not only the streamed event.
*******************************************************************************/
public static void assertPart(Map<String, Object> part)
{
assertThat(part.get("type")).isEqualTo("presentation");
assertThat((String) part.get("summary")).isNotBlank().doesNotContain("\n").hasSizeLessThanOrEqualTo(300);
assertThat((String) part.get("markdown")).contains("\n|");
Map<String, Object> table = block(part, "table");
List<?> rows = (List<?>) table.get("rows");
assertThat(rows).hasSize(2);
assertThat(rows).anySatisfy(row -> assertThat(object(object(row).get("display")).values()).contains("81", "80.2%"));
assertThat(rows).anySatisfy(row -> assertThat(object(object(row).get("display")).values()).contains("20", "19.8%"));
Map<String, Object> chart = block(part, "chart");
assertThat(chart.get("chartType")).isEqualTo("pie");
List<?> points = (List<?>) chart.get("points");
assertThat(points).hasSize(2);
assertThat(points.stream().map(p -> ((Number) object(p).get("value")).intValue()).toList()).containsExactlyInAnyOrder(81, 20);
List<?> links = (List<?>) block(part, "filterLinks").get("items");
assertThat(links).hasSize(2);
List<Object> mimes = new ArrayList<>();
for(Object entry : links)
{
String href = (String) object(entry).get("href");
assertThat(href).startsWith("/app/media?filter=");
try
{
Map<String, Object> filter = JsonUtils.toObject(URLDecoder.decode(href.substring(href.indexOf('=') + 1), StandardCharsets.UTF_8), new TypeReference<Map<String, Object>>() {});
assertThat(filter).containsEntry("booleanOperator", "AND");
List<?> criteria = (List<?>) filter.get("criteria");
assertThat(criteria).hasSize(1);
Map<String, Object> criterion = object(criteria.get(0));
assertThat(criterion).containsEntry("fieldName", "mimeType").containsEntry("operator", "EQUALS");
mimes.addAll((List<?>) criterion.get("values"));
}
catch(Exception e)
{
throw (new AssertionError("Invalid filtered list href", e));
}
}
assertThat(mimes).containsExactlyInAnyOrder("image/jpeg", "image/png");
}
/*******************************************************************************
** Find one block of a given kind.
*******************************************************************************/
public static Map<String, Object> block(Map<String, Object> part, String kind)
{
return (((List<?>) part.get("blocks")).stream().map(PresentationGoldenTask::object).filter(block -> kind.equals(block.get("kind"))).findFirst().orElseThrow());
}
/*******************************************************************************
** Typed test view of a JSON object.
*******************************************************************************/
@SuppressWarnings("unchecked")
public static Map<String, Object> object(Object value)
{
return ((Map<String, Object>) value);
}
}
@@ -59,7 +59,8 @@ public class TurnRunner
*******************************************************************************/
public String text()
{
return (events.stream().filter(e -> TurnEvent.TEXT_DELTA.equals(e.getType())).map(e -> String.valueOf(e.getData().get("text"))).collect(Collectors.joining()));
return (events.stream().filter(e -> TurnEvent.TEXT_DELTA.equals(e.getType()) || TurnEvent.PRESENTATION.equals(e.getType()))
.map(e -> TurnEvent.PRESENTATION.equals(e.getType()) ? String.valueOf(((Map<?, ?>) e.getData().get("part")).get("markdown")) : String.valueOf(e.getData().get("text"))).collect(Collectors.joining()));
}
@@ -0,0 +1,275 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.tools;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.assistant.engine.MessageParts;
import com.kingsrook.qbits.assistant.engine.PolicyService;
import com.kingsrook.qbits.assistant.engine.TurnEvent;
import com.kingsrook.qbits.assistant.model.AssistantMessage;
import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qbits.assistant.testing.PresentationGoldenTask;
import com.kingsrook.qbits.assistant.testing.TurnRunner;
import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.assistant.testing.PresentationGoldenTask.block;
import static com.kingsrook.qbits.assistant.testing.PresentationGoldenTask.object;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Presentation persistence, permission boundaries, formatting and input limits.
*******************************************************************************/
class PresentResultsToolTest extends BaseAssistantTest
{
/*******************************************************************************
** Streamed, replayed and stored presentations agree; retention removes data.
*******************************************************************************/
@Test
void testPersistenceAndReplay() throws Exception
{
PresentationGoldenTask.task().script().accept(provider);
TurnRunner.Outcome outcome = runner.run(AssistantTestFixture.EDITOR, "How many images do I have by type?");
PresentationGoldenTask.check(outcome);
QContext.init(qInstance, AssistantTestFixture.ownerSession(AssistantTestFixture.EDITOR, config));
Map<String, Object> event = outcome.events(TurnEvent.PRESENTATION).get(0);
AssistantMessage message = runner.runtime().getStore().getMessage(AssistantTestFixture.EDITOR, (Integer) event.get("messageId"));
MessageParts parts = MessageParts.fromJson(message.getPartsJson());
Map<String, Object> part = parts.list().stream().filter(p -> "presentation".equals(p.get("type"))).findFirst().orElseThrow();
PresentationGoldenTask.assertPart(part);
assertThat(part).isEqualTo(com.kingsrook.qqq.backend.core.utils.JsonUtils.toObject(com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(event.get("part")), Map.class));
List<?> rows = (List<?>) block(part, "table").get("rows");
List<?> points = (List<?>) block(part, "chart").get("points");
List<?> links = (List<?>) block(part, "filterLinks").get("items");
for(Integer i = 0; i < 2; i++)
{
assertThat(object(rows.get(i)).get("href")).isEqualTo(object(links.get(i)).get("href"));
assertThat(object(points.get(i)).get("href")).isEqualTo(object(rows.get(i)).get("href"));
}
assertThat(runner.runtime().getStore().listEvents(AssistantTestFixture.EDITOR, outcome.turnId(), 0, 1000))
.anyMatch(e -> TurnEvent.PRESENTATION.equals(e.getEventType()));
assertThat(parts.text()).contains("81", "20", "\n|");
assertThat(parts.stripRecordData()).isTrue();
assertThat(parts.list()).noneMatch(p -> "presentation".equals(p.get("type")));
assertThat(outcome.toolCalls()).anySatisfy(call ->
{
assertThat(call.getToolName()).isEqualTo(PresentResultsTool.NAME);
assertThat(call.getInputJson()).contains("presentation");
assertThat(call.getResultJson()).contains("Do not repeat");
});
}
/*******************************************************************************
** The real UI metadata hides media from an analyst, as does presentation.
*******************************************************************************/
@Test
void testUiDenial() throws Exception
{
String user = AssistantTestFixture.ANALYST;
UiApiResult metadata = runner.runtime().getUiApiGateway().invoke(new UiApiCall("GET", "/metaData").withSessionUUID(user));
assertThat(object(metadata.json().get("tables"))).doesNotContainKey("media");
provider.thenToolUse(PresentResultsTool.NAME, input("summary", "The list is unavailable.", "recordLinks", List.of(input("label", "Image", "tableName", "media", "recordId", "1")))).thenText("");
TurnRunner.Outcome outcome = runner.run(user, "Open an image");
assertThat(outcome.toolCalls().get(0).getResultJson()).contains("table not visible through UI API: media");
Map<String, Object> part = object(outcome.events(TurnEvent.PRESENTATION).get(0).get("part"));
assertThat((List<?>) part.get("blocks")).isEmpty();
assertThat(outcome.toolCalls().get(0).getInputJson()).doesNotContain("recordId");
}
/*******************************************************************************
** Cache metadata once per turn and send the original session UUID.
*******************************************************************************/
@Test
void testMetadataCache() throws Exception
{
List<UiApiCall> calls = new ArrayList<>();
UiApiGateway gateway = new UiApiGateway()
{
/*******************************************************************************
** Fixture metadata gateway.
*******************************************************************************/
@Override
public boolean isInvocable(String method, String path)
{
return (true);
}
/*******************************************************************************
** Fixture metadata gateway.
*******************************************************************************/
@Override
public UiApiResult invoke(UiApiCall call)
{
calls.add(call);
return (new UiApiResult(200, "{\"tables\":{\"media\":{\"readPermission\":true}}}"));
}
};
UiApiToolbox.Context context = new UiApiToolbox.Context("editor-session", "UTC", gateway, PolicyService.ToolPolicy.defaults(), null);
PresentResultsTool tool = new PresentResultsTool(qInstance);
assertThat(tool.execute(context, "1", PresentationGoldenTask.input(81, 20)).getOutcome()).isEqualTo(AiToolOutcome.OK);
tool.execute(context, "2", PresentationGoldenTask.input(81, 20));
assertThat(calls).hasSize(1).allMatch(call -> "editor-session".equals(call.getSessionUUID()) && "/metaData".equals(call.getPathTemplate()));
new PresentResultsTool(qInstance).execute(context, "3", PresentationGoldenTask.input(81, 20));
assertThat(calls).hasSize(2);
}
/*******************************************************************************
** Unknown metadata is dropped independently without persisting its values.
*******************************************************************************/
@Test
void testUnknownMetadata()
{
AiToolResult result = execute(input("summary", "Available results.", "metrics", List.of(
input("label", "Unknown table", "value", 9876, "tableName", "missing"),
input("label", "Unknown field", "value", 5432, "tableName", "media", "fieldName", "missing"),
input("label", "Known", "value", 101))));
assertThat(result.getModelContent()).contains("Dropped metrics[0]", "unknown table", "Dropped metrics[1]", "unknown field");
assertThat(result.getRender().toString()).doesNotContain("9876", "5432");
assertThat((List<?>) block(result.getRender(), "metrics").get("items")).hasSize(1);
}
/*******************************************************************************
** Markdown punctuation, external URLs and pipes remain literal text.
*******************************************************************************/
@Test
void testUntrustedLabelsAndLinks()
{
AiToolResult result = execute(input("summary", "Available results.", "metrics", List.of(input("label", "[evil](https://evil.example) | <img>", "value", 1)),
"recordLinks", List.of(input("label", "Open", "tableName", "media", "recordId", "a/b?x#y")),
"actions", List.of(input("label", "Ask", "prompt", "Show images"), input("label", "Open media", "navigateTo", "media"), input("label", "Bad", "prompt", "Ask", "navigateTo", "media"))));
String markdown = (String) result.getRender().get("markdown");
assertThat(markdown).doesNotContain("https://", "<img>", "[evil]").contains("&#124;", "[Open](/app/media/a%2Fb%3Fx%23y)", "- Ask: Show images", "[Open media](/app/media)");
assertThat(result.getModelContent()).contains("exactly one");
}
/*******************************************************************************
** Formatting follows metadata while raw values remain numeric.
*******************************************************************************/
@Test
void testFormatting()
{
qInstance.getTable("media").addField(new QFieldMetaData("price", QFieldType.DECIMAL).withDisplayFormat("$%,.2f"));
AiToolResult result = execute(input("summary", "Amounts.", "metrics", List.of(input("label", "Amount", "value", 1234.5, "tableName", "media", "fieldName", "price"), input("label", "Count", "value", 12000)),
"table", input("columns", List.of(input("label", "Amount", "tableName", "media", "fieldName", "price")), "rows", List.of(List.of(1234.5)))));
assertThat(block(result.getRender(), "metrics").toString()).contains("$1,234.50", "12,000", "1234.5");
assertThat(block(result.getRender(), "table").toString()).contains("$1,234.50");
}
/*******************************************************************************
** Percentages cannot refer to labels, derived percentages or bad indexes.
*******************************************************************************/
@Test
void testInvalidPercentColumn()
{
for(Object source : List.of(0, 3, 1.5))
{
AiToolResult result = execute(input("summary", "Invalid share.", "table", input("columns", List.of(input("label", "Name"), input("label", "Share", "percentOfColumn", source)), "rows", List.of(List.of("JPEG", 0)))));
assertThat(result.getModelContent()).contains("Dropped table");
assertThat((List<?>) result.getRender().get("blocks")).isEmpty();
}
}
/*******************************************************************************
** Zero totals are defined, and charts never exceed twelve points.
*******************************************************************************/
@Test
void testChartLimitAndZeroTotal()
{
List<List<Object>> rows = new ArrayList<>();
for(Integer i = 0; i < 13; i++)
{
rows.add(List.of("Item " + i, 0, 0));
}
AiToolResult result = execute(input("summary", "Thirteen items.", "table", input("columns", List.of(input("label", "Item"), input("label", "Count"), input("label", "Share", "percentOfColumn", 1)), "rows", rows),
"chart", input("chartType", "bar", "labelColumn", 0, "valueColumn", 1)));
assertThat((List<?>) block(result.getRender(), "chart").get("points")).hasSize(12);
assertThat((List<?>) block(result.getRender(), "table").get("rows")).hasSize(13);
assertThat(result.getModelContent()).contains("truncated to 12");
assertThat(block(result.getRender(), "table").toString()).contains("0.0%");
}
/*******************************************************************************
** Operator validation does not coerce raw numeric/boolean filter values.
*******************************************************************************/
@Test
void testCriteriaValidation()
{
AiToolResult result = execute(input("summary", "Links.", "filterLinks", List.of(
input("label", "Invalid", "tableName", "media", "criteria", List.of(input("fieldName", "width", "operator", "INVENTED", "values", List.of(10)))),
input("label", "Valid", "tableName", "media", "criteria", List.of(input("fieldName", "width", "operator", "EQUALS", "values", List.of(512)))))));
assertThat(result.getModelContent()).contains("invalid criteria operator");
List<?> links = (List<?>) block(result.getRender(), "filterLinks").get("items");
String decoded = java.net.URLDecoder.decode((String) object(links.get(0)).get("href"), java.nio.charset.StandardCharsets.UTF_8);
assertThat(decoded).contains("\"values\":[512]");
}
/*******************************************************************************
** Invalid summaries produce an error tool result; no presentation is emitted.
*******************************************************************************/
@Test
void testInvalidSummaryAndAlwaysOffered() throws Exception
{
assertThat(runner.runtime().getContextBuilder().tools(new PolicyService.ToolPolicy(true, java.util.Set.of("navigate"), 20, true)))
.anyMatch(tool -> PresentResultsTool.NAME.equals(tool.getName()) && !Boolean.TRUE.equals(tool.getStrict()));
assertThat(execute(input("summary", "x".repeat(301))).getOutcome()).isEqualTo(AiToolOutcome.INVALID_INPUT);
provider.thenToolUse(PresentResultsTool.NAME, Map.of()).thenText("Unable to present.");
TurnRunner.Outcome outcome = runner.run(AssistantTestFixture.EDITOR, "Present");
assertThat(outcome.events(TurnEvent.PRESENTATION)).isEmpty();
assertThat(outcome.toolCalls().get(0).getOutcome()).isEqualTo("INVALID_INPUT");
}
/*******************************************************************************
** Use the fixture's real UI gateway for unit-level presentation checks.
*******************************************************************************/
private AiToolResult execute(Map<String, Object> input)
{
return (new PresentResultsTool(qInstance).execute(new UiApiToolbox.Context(AssistantTestFixture.EDITOR, "UTC", runner.runtime().getUiApiGateway(), PolicyService.ToolPolicy.defaults(), null), "present", input));
}
}