Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 james.maes
|
57673131ff
|
||
 james.maesandClaude Opus 5.5
|
c27994c432
|
No files matched your search
@@ -1,5 +1,14 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased - Answer presentation
|
||||
|
||||
- Add `present_results`: server-formatted metrics, tables, computed percentages,
|
||||
charts and validated in-app filter/record/action links.
|
||||
- Persist and stream presentation parts with a multi-line GFM fallback and
|
||||
record-data retention; add a static default presentation skill.
|
||||
- Add the owner's 81 JPEG / 20 PNG golden example, shared live eval coverage,
|
||||
permission/formatting/input tests and strict-schema budget coverage.
|
||||
|
||||
## Unreleased - P-e
|
||||
|
||||
- Add per-profile heuristic/classifier model routing, with budgeted ROUTE usage.
|
||||
|
||||
@@ -9,7 +9,7 @@ Design spec: `QRun-IO/qqq` `docs/specs/2026-10-11-qqq-assistant-qbit-design.md`.
|
||||
| `com.kingsrook.qbits:qbit-assistant` | The QBit: conversation tables, turn engine, `/qqq/v1/assistant` routes (JSON + Server-Sent Events), budgets, kill switch, the provider SPI, and a scripted provider for tests and demos. |
|
||||
| `com.kingsrook.qbits:qbit-assistant-provider-anthropic` | The Claude provider (`com.anthropic:anthropic-java`). Add it to use Claude. The core QBit has no LLM SDK dependency. |
|
||||
|
||||
The data access itself is done by the AI tool kernel in QQQ core (`com.kingsrook.qqq.backend.core.actions.ai.AiToolKernel`). The kernel checks table permissions and exposure rules, applies record security, and caps and redacts what the model sees. Every tool result reaches the model wrapped as untrusted data.
|
||||
Data tools call the QQQ UI API through `UiApiToolbox` and `UiApiGateway` with the signed-in user's session. The UI enforces permissions and record security; the assistant further narrows access and caps and redacts returned values. Data tool results reach the model wrapped as untrusted data. Only record-free navigation and help use the core AI tool kernel.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -203,6 +203,7 @@ Base path `/qqq/v1/assistant`. Every request is authenticated through the app's
|
||||
The stream sends these events, each with a `seq`:
|
||||
- `message.start`, `text.delta`
|
||||
- `tool.start`, `tool.result` (with `render` and `recordRefs`)
|
||||
- `presentation` (`messageId`, `part`: summary, blocks and server-built GFM markdown)
|
||||
- `ui.navigate`, `notice`
|
||||
- `usage`, `message.end`
|
||||
- `error`
|
||||
@@ -487,3 +488,16 @@ NETWORK/TIMEOUT marks a provider DOWN for 60 seconds; success marks it UP. Local
|
||||
health is probed when unknown/stale with a two-second timeout. Provider cards show
|
||||
health and down-until, failures show fallback totals, and turn records show the
|
||||
route and final destination. See [UPGRADE.md](UPGRADE.md) for the additive schema.
|
||||
|
||||
### Answer presentation
|
||||
|
||||
The always-available `present_results` tool turns retrieved results into metrics,
|
||||
tables, charts and actionable in-app links. The server validates the user's UI API
|
||||
visibility, formats field values, computes shares and builds all links. It emits
|
||||
and stores a `presentation` part with a complete GFM Markdown fallback. Profile
|
||||
skills can extend the built-in presentation instructions. See the
|
||||
[part, stream and input contract](docs/answer-presentation.md).
|
||||
|
||||
The image-breakdown golden task also runs in the existing opt-in live provider suite:
|
||||
`./mvnw verify -Dassistant.live=true -Dtest.excludedGroups= -Dgroups=live -Djacoco.skip=true -pl qbit-assistant-provider-anthropic -am`
|
||||
(with `ANTHROPIC_API_KEY` set). Live calls are excluded from normal verification.
|
||||
@@ -446,3 +446,36 @@ AI exposure NONE is not enough, because the assistant acts as the user and an ad
|
||||
- an "Explain access" action (for a user, and a table or a prompt, which operations the assistant could call and which it would be denied, with the reason);
|
||||
- a recent-denials widget on the app home.
|
||||
- **(d) Spec'd assistant endpoints:** move `/qqq/v1/assistant/...` (conversations, turns, events, feedback, cancel, config, actions) onto `AbstractEndpointSpec`s through the qbit spec extension point (`MiddlewareSpecExtension`, established prefix `/assistant`, QRun-IO/qqq#1128). The SSE turn and events streams stay raw routes until the spec layer can describe a stream.
|
||||
|
||||
## Answer presentation (owner priority 2026-10-13)
|
||||
|
||||
`present_results` is always offered and reads no records. It takes a direct answer,
|
||||
metrics, a table, an optional table-derived chart, filtered list links, record links
|
||||
and next-step actions. The server validates metadata and the user's UI API table
|
||||
visibility once per turn, formats values using QQQ field types/display formats,
|
||||
computes percentages and builds every `/app` href. Unknown or denied items are
|
||||
dropped with an explanation. The assistant must still retrieve every fact through
|
||||
data tools before presenting it.
|
||||
|
||||
The [presentation part and SSE contract](answer-presentation.md) defines `summary`,
|
||||
`blocks` and the complete server-built GFM `markdown` fallback. Each successful
|
||||
call emits `presentation` with `{messageId, part}` and persists that part; the tool
|
||||
response tells the model not to repeat its tables or links. Audit rows contain
|
||||
only accepted presentation data. Normal record-data retention removes these parts.
|
||||
|
||||
The built-in Presentation skill is compact static prompt text in the cached rules
|
||||
block. It asks for a one-sentence answer first, metrics for single numbers, tables
|
||||
for breakdowns/comparisons, charts for useful shares/trends and actual filter links
|
||||
whenever offering a list. Administrator profile skills append after it as before
|
||||
and extend it; there is no replacement boolean or extra permission surface.
|
||||
|
||||
The owner's one-line table was a frontend parsing defect: Next's `AssistantMarkdown`
|
||||
used react-markdown without `remark-gfm`, so table lines became a paragraph.
|
||||
The renderer fix belongs to `qqq-frontend-next`; this worktree preserves real
|
||||
newlines and provides structured blocks plus a proper multi-line GFM fallback.
|
||||
The frontend change is maintained separately and is not verified by this build.
|
||||
|
||||
The shared golden task asks "How many images do I have by type?": five UI count
|
||||
calls establish 81 JPEG, 20 PNG and zero other configured types, then a presentation
|
||||
with two rows, 80.2%/19.8% shares, two pie points and working MIME filter links.
|
||||
The same task is included in the opt-in `live` Anthropic eval suite.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Assistant answer presentation contract (v1)
|
||||
|
||||
Owner priority (2026-10-13): answers must ALWAYS be easy to read and act on. Bad example the owner saw, rendered on ONE line:
|
||||
"| Type | Count | |---|---| | JPEG (image/jpeg) | 81 | | PNG (image/png) | 20 | | WebP, GIF, SVG | 0 |" followed by "You can open the lists filtered by JPEG or PNG" with NO links.
|
||||
|
||||
Root cause found in Next: `AssistantMarkdown` uses react-markdown WITHOUT `remark-gfm`, so GFM tables are not parsed and the table's lines collapse into one paragraph (soft line breaks). Backend streaming preserves `\n` (events are JSON).
|
||||
|
||||
## New message part: `presentation`
|
||||
|
||||
An assistant message's `parts` (persisted in `assistant_message.parts_json`, streamed as an SSE event, returned by the conversation API) may contain, besides `text`, `notice`, `tool`/`render` parts etc., ONE part per `present_results` call:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "presentation",
|
||||
"summary": "101 media files are images: 81 JPEG and 20 PNG.",
|
||||
"markdown": "<server-built GFM fallback of the whole presentation, for clients that do not render blocks (CLI/TUI)>",
|
||||
"blocks": [
|
||||
{ "kind": "metrics", "items": [ { "label": "Images", "value": 101, "display": "101", "href": "/app/media?filter=..." } ] },
|
||||
{ "kind": "table", "title": "Media by type",
|
||||
"columns": [ { "key": "type", "label": "Type", "align": "left" },
|
||||
{ "key": "count", "label": "Count", "align": "right" },
|
||||
{ "key": "share", "label": "Share", "align": "right" } ],
|
||||
"rows": [ { "cells": { "type": "JPEG (image/jpeg)", "count": 81, "share": 0.802 },
|
||||
"display": { "type": "JPEG (image/jpeg)", "count": "81", "share": "80.2%" },
|
||||
"href": "/app/media?filter=..." } ] },
|
||||
{ "kind": "chart", "chartType": "pie", "title": "Share by type",
|
||||
"points": [ { "label": "JPEG", "value": 81, "display": "81", "href": "/app/media?filter=..." },
|
||||
{ "label": "PNG", "value": 20, "display": "20", "href": "/app/media?filter=..." } ] },
|
||||
{ "kind": "filterLinks", "items": [ { "label": "Open JPEG media (81)", "tableName": "media", "href": "/app/media?filter=..." } ] },
|
||||
{ "kind": "recordLinks", "items": [ { "label": "Sunset.jpg", "tableName": "media", "recordId": "12", "href": "/app/media/12" } ] },
|
||||
{ "kind": "actions", "items": [ { "label": "Show the PNG files", "prompt": "Show me the PNG media files" },
|
||||
{ "label": "Open media", "href": "/app/media" } ] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- `summary` is required, one sentence, the direct answer first.
|
||||
- `chartType` ∈ `bar`, `pie`, `line`; ≤ 12 points; values are numbers.
|
||||
- Every `href` is built SERVER-SIDE from metadata and is an `/app` path (same rules as `TurnEngine.isAppHref`). Filtered list links are `/app/<tableName>?filter=<URL-encoded JSON {"criteria":[{"fieldName":..,"operator":..,"values":[..]}],"booleanOperator":"AND"}>`, which Next's `deserializeFilter` accepts (JSON starting with `{`). The model never writes URLs.
|
||||
- `display` strings are formatted server-side from field metadata (`displayFormat`, e.g. `%,d`, `$%,.2f`; percent columns as `80.2%`); clients show `display` when present, else the raw value.
|
||||
- `actions` items have either `prompt` (send as the user's next message) or `href` (navigate), never both.
|
||||
- Unknown `kind`s must be ignored by clients (forward compatible); clients that do not render blocks render `markdown`.
|
||||
- The SSE event is `presentation` with `{ "messageId": <id>, "part": <the part above> }`; Next's reducer appends it to the assistant message's parts.
|
||||
|
||||
## Tool input and implementation
|
||||
|
||||
The input schema is [present-results.schema.json](../qbit-assistant/src/main/resources/present-results.schema.json). `table.rowKeys` supplies one raw filter value per row; `table.rowFilter` names the table, field and optional operator (EQUALS by default). Columns receive stable server keys `c0`, `c1`, etc. Input cells remain raw scalars; each row has a parallel display map. The example keys above are illustrative.
|
||||
|
||||
`PresentResultsTool` is instantiated per turn. It caches `GET /metaData` with that turn's session, checks table read visibility and instance fields, validates QQQ criteria operators and builds paths using RC.5 `AiLinkBuilder`. Filter links use URL-encoded JSON accepted by Next. Record ids are encoded as opaque path segments. Metadata denial or invalid item drops that item (an invalid table drops its table/chart), with a short explanation to the model; other items survive. No data query or permission elevation happens here.
|
||||
|
||||
Limits: summary 300 characters, display text 2,000 characters, table 20 columns/200 rows, chart 12 points, item lists 100 entries, filters 20 criteria/200 values. Chart truncation is reported to the model. Percent columns reference a raw numeric column; a zero total yields 0.0%. Numbers still must originate in earlier data tool results; presentation is formatting, not a new source of facts.
|
||||
|
||||
The schema uses small named, nonrecursive definitions to avoid deeply inlined structures. Each object component is at most two levels of inline containers; the resolved table/criteria structure necessarily has the nesting shown in the owner contract. Strict is off by default. Opting `present_results` into strict mode requires all properties (empty arrays/strings, empty table and -1 for unused percentage indexes), removes provider-unsupported length constraints, and retains server validation. This has zero optional parameters and one shared scalar union in the existing strict-budget test.
|
||||
|
||||
Presentation parts are persisted with the assistant message and replayed as `presentation` events with `{messageId, part}` (plus the standard event sequence). They are removed by record-data retention along with tool renders. Model history and text-only consumers can use the server-built Markdown. Unknown block kinds should be ignored. Plain labels must be rendered as text, never parsed as links or HTML; fallback Markdown escapes them. Tool audit input records the accepted presentation only, excluding dropped values.
|
||||
@@ -20,7 +20,7 @@
|
||||
<!-- Multi-module build. The qqq-orb sets the revision here; the modules -->
|
||||
<!-- inherit it, and the flatten plugin resolves it in the published POMs. -->
|
||||
<properties>
|
||||
<revision>0.1.0-RC.8</revision>
|
||||
<revision>0.1.0-SNAPSHOT</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
|
||||
+18
@@ -108,6 +108,24 @@ class AnthropicRequestContractTest
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Presentation remains inside the actual serialized strict request budget.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testPresentationStrictOptIn() throws Exception
|
||||
{
|
||||
Map<String, Object> body = requestBody(new AssistantQBitConfig().withModelProvider(new ScriptedModelProvider()).withStrictToolNames(Set.of("present_results")));
|
||||
assertThat(violations(body)).isEmpty();
|
||||
assertThat(strictBudgetViolations(body)).isEmpty();
|
||||
assertThat(tools(body)).anySatisfy(tool ->
|
||||
{
|
||||
assertThat(tool.get("name")).isEqualTo("present_results");
|
||||
assertThat(tool.get("strict")).isEqualTo(true);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The RC.3 shape (every tool strict) is recognized as over budget, and a
|
||||
** host can still opt a few tools in.
|
||||
|
||||
@@ -34,6 +34,7 @@ import com.kingsrook.qbits.assistant.provider.ModelPart;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelRequest;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
|
||||
import com.kingsrook.qbits.assistant.provider.SystemBlock;
|
||||
import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
|
||||
import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolDefinition;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolKernel;
|
||||
@@ -116,6 +117,7 @@ public class ContextBuilder
|
||||
.withStrict(isStrict(definition.getName(), !Boolean.FALSE.equals(definition.getStrict()))));
|
||||
}
|
||||
}
|
||||
byName.put(PresentResultsTool.NAME, PresentResultsTool.definition(isStrict(PresentResultsTool.NAME, true)));
|
||||
return (new ArrayList<>(byName.values()));
|
||||
}
|
||||
|
||||
|
||||
+19
-4
@@ -31,7 +31,7 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
|
||||
** assistantMessage.partsJson and returned by the API:
|
||||
** {type:"text", text} | {type:"tool", toolCallId, name, summary, outcome, render?}
|
||||
** | {type:"record_refs", refs:[...]} | {type:"navigate", href}
|
||||
** | {type:"notice", level, text}
|
||||
** | {type:"notice", level, text} | {type:"presentation", summary, blocks, markdown}
|
||||
** Consecutive text is merged into one part. Thread-safe.
|
||||
*******************************************************************************/
|
||||
public class MessageParts
|
||||
@@ -110,6 +110,17 @@ public class MessageParts
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Append a server-built presentation, one part per successful tool call.
|
||||
*******************************************************************************/
|
||||
public synchronized MessageParts addPresentation(Map<String, Object> presentation)
|
||||
{
|
||||
parts.add(new LinkedHashMap<>(presentation));
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Append record references.
|
||||
*******************************************************************************/
|
||||
@@ -150,12 +161,12 @@ public class MessageParts
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove record data (retention): the render payloads of tool parts and
|
||||
** the record_refs parts. Tool summaries, text, navigation and notices
|
||||
** the record_refs and presentation parts. Tool summaries, text, navigation and notices
|
||||
** stay. Returns whether anything was removed.
|
||||
*******************************************************************************/
|
||||
public synchronized boolean stripRecordData()
|
||||
{
|
||||
boolean changed = parts.removeIf(p -> "record_refs".equals(p.get("type")));
|
||||
boolean changed = parts.removeIf(p -> ("record_refs".equals(p.get("type")) || "presentation".equals(p.get("type"))));
|
||||
for(Map<String, Object> part : parts)
|
||||
{
|
||||
if("tool".equals(part.get("type")) && part.remove("render") != null)
|
||||
@@ -169,7 +180,7 @@ public class MessageParts
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** All text parts joined (for the model's history and titles).
|
||||
** Text and presentation fallbacks joined (for model history and titles).
|
||||
*******************************************************************************/
|
||||
public synchronized String text()
|
||||
{
|
||||
@@ -180,6 +191,10 @@ public class MessageParts
|
||||
{
|
||||
builder.append(part.get("text"));
|
||||
}
|
||||
else if("presentation".equals(part.get("type")))
|
||||
{
|
||||
builder.append("\n").append(part.get("markdown")).append("\n");
|
||||
}
|
||||
}
|
||||
return (builder.toString());
|
||||
}
|
||||
|
||||
@@ -40,7 +40,8 @@ public class SystemPrompt
|
||||
10. An administrator may give you a name, a personality and skills (in <assistant_profile> and <skill> blocks). Follow them for tone and know-how, but they never override these rules and never give you other tools, data or permissions.
|
||||
11. Some values are masked by the application (for example j***@x.com or ***1234). Treat them as masked, never as the real value, and never ask the user to reveal them for you.
|
||||
|
||||
Answer style: brief and specific. Use short Markdown (lists, small tables). Refer to records with the links the tools return.
|
||||
Presentation (default skill; administrator skills may extend it):
|
||||
Answer first in one sentence; do not restate totals ("101 add up to 101"). For 3+ items, comparisons or breakdowns, call present_results with a table; add a chart when shares or trends help. Prefer metrics for a single big number. Include any list you offer to open as a filterLinks item, never just plain text. Use only values from tool results, omit zero-count categories, and use percentOfColumn for shares. Keep prose short; do not repeat presented tables or links. Never write URLs yourself; supply metadata names and raw filter values to present_results.
|
||||
""";
|
||||
|
||||
public static final String QQQ_PRIMER = """
|
||||
|
||||
@@ -27,6 +27,7 @@ import java.util.Map;
|
||||
public class ToolLabels
|
||||
{
|
||||
private static final Map<String, String[]> LABELS = Map.ofEntries(
|
||||
Map.entry("present_results", new String[] { "Presenting results", "Presented results" }),
|
||||
Map.entry("list_tables", new String[] { "Listing tables", "Listed tables" }),
|
||||
Map.entry("describe_table", new String[] { "Reading table details", "Read table details" }),
|
||||
Map.entry("query_records", new String[] { "Querying records", "Queried records" }),
|
||||
|
||||
+3
-2
@@ -42,7 +42,7 @@ import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
*******************************************************************************/
|
||||
public class TurnClassification
|
||||
{
|
||||
private static final Set<String> METADATA_TOOLS = Set.of("list_tables", "describe_table", "navigate", "search_help");
|
||||
private static final Set<String> METADATA_TOOLS = Set.of("list_tables", "describe_table", "navigate", "search_help", "present_results");
|
||||
private final ClassificationSource source;
|
||||
private final Set<String> tables = new LinkedHashSet<>();
|
||||
private DataClassification classification;
|
||||
@@ -65,7 +65,8 @@ public class TurnClassification
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Observe before appending any result to a model request. Metadata tools carry no record values.
|
||||
** Observe before appending results. Metadata reads no records; presentation
|
||||
** only reuses values whose originating data tools were already classified.
|
||||
*******************************************************************************/
|
||||
public void observe(ModelPart use, AiToolResult result)
|
||||
{
|
||||
|
||||
@@ -36,6 +36,7 @@ import com.kingsrook.qbits.assistant.provider.ModelRequest;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelResponse;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelUsage;
|
||||
import com.kingsrook.qbits.assistant.provider.StreamHandle;
|
||||
import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
|
||||
import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiRecordRef;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolCall;
|
||||
@@ -101,6 +102,7 @@ public class TurnEngine implements Runnable
|
||||
|
||||
private static final Set<AiToolOutcome> ERROR_OUTCOMES = Set.of(AiToolOutcome.DENIED, AiToolOutcome.NOT_FOUND, AiToolOutcome.INVALID_INPUT, AiToolOutcome.ERROR);
|
||||
|
||||
private final PresentResultsTool presenter;
|
||||
private final AssistantRuntime runtime;
|
||||
private final AssistantQBitConfig config;
|
||||
private final AssistantStore store;
|
||||
@@ -146,6 +148,7 @@ public class TurnEngine implements Runnable
|
||||
AssistantTurn turn, String userText, Map<String, Object> pageContext, PolicyService.EffectivePolicy policy, RunningTurn running, String softBudgetNotice)
|
||||
{
|
||||
this.runtime = runtime;
|
||||
this.presenter = new PresentResultsTool(runtime.getQInstance());
|
||||
this.config = runtime.getConfig();
|
||||
this.store = runtime.getStore();
|
||||
this.session = session;
|
||||
@@ -689,7 +692,11 @@ public class TurnEngine implements Runnable
|
||||
{
|
||||
if(!running.isCancelled())
|
||||
{
|
||||
if(ContextBuilder.KERNEL_TOOLS.contains(call.getName()))
|
||||
if(PresentResultsTool.NAME.equals(call.getName()))
|
||||
{
|
||||
results[index] = presenter.execute(toolContext(), call.getId(), call.getInput());
|
||||
}
|
||||
else if(ContextBuilder.KERNEL_TOOLS.contains(call.getName()))
|
||||
{
|
||||
///////////////////////////////////////////////////
|
||||
// navigate / search_help: no records are read //
|
||||
@@ -798,7 +805,8 @@ public class TurnEngine implements Runnable
|
||||
}
|
||||
|
||||
Map<String, Object> data = map("messageId", assistantMessage.getId(), "toolCallId", use.getToolUseId(), "name", use.getToolName(), "outcome", outcome, "summary", summary);
|
||||
if(result.getRender() != null)
|
||||
Boolean presentation = PresentResultsTool.NAME.equals(use.getToolName()) && AiToolOutcome.OK.equals(result.getOutcome()) && result.getRender() != null;
|
||||
if(result.getRender() != null && !presentation)
|
||||
{
|
||||
data.put("render", result.getRender());
|
||||
}
|
||||
@@ -808,7 +816,12 @@ public class TurnEngine implements Runnable
|
||||
}
|
||||
emit(TurnEvent.TOOL_RESULT, data);
|
||||
|
||||
parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, result.getRender());
|
||||
parts.addTool(use.getToolUseId(), use.getToolName(), summary, outcome, presentation ? null : result.getRender());
|
||||
if(presentation)
|
||||
{
|
||||
parts.addPresentation(result.getRender());
|
||||
emit(TurnEvent.PRESENTATION, map("messageId", assistantMessage.getId(), "part", result.getRender()));
|
||||
}
|
||||
parts.addRecordRefs(refs);
|
||||
|
||||
String href = result.getNavigateHref();
|
||||
@@ -826,7 +839,7 @@ public class TurnEngine implements Runnable
|
||||
.withUserId(userId)
|
||||
.withToolCallId(use.getToolUseId())
|
||||
.withToolName(use.getToolName())
|
||||
.withInputJson(truncate(redactInput(use.getInput()), MAX_INPUT_JSON_CHARS))
|
||||
.withInputJson(truncate(redactInput(PresentResultsTool.NAME.equals(use.getToolName()) ? (presentation ? result.getRender() : Map.of()) : use.getInput()), MAX_INPUT_JSON_CHARS))
|
||||
.withOutcome(outcome)
|
||||
.withPermissionCheck(result.getPermissionCheck())
|
||||
.withTableName(result.getTableName())
|
||||
|
||||
@@ -33,6 +33,7 @@ public class TurnEvent
|
||||
public static final String TEXT_DELTA = "text.delta";
|
||||
public static final String TOOL_START = "tool.start";
|
||||
public static final String TOOL_RESULT = "tool.result";
|
||||
public static final String PRESENTATION = "presentation";
|
||||
public static final String UI_NAVIGATE = "ui.navigate";
|
||||
public static final String NOTICE = "notice";
|
||||
public static final String USAGE = "usage";
|
||||
|
||||
+727
@@ -0,0 +1,727 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.assistant.tools;
|
||||
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.math.BigDecimal;
|
||||
import java.math.MathContext;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.function.Function;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
|
||||
import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
|
||||
import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.output.AiLinkBuilder;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Builds presentations from previously retrieved values, never reading records.
|
||||
** One instance belongs to one turn: UI metadata, including denials, is cached
|
||||
** only for that turn and session. All output links are built by the server.
|
||||
*******************************************************************************/
|
||||
public class PresentResultsTool
|
||||
{
|
||||
public static final String NAME = "present_results";
|
||||
|
||||
private final QInstance qInstance;
|
||||
private UiApiResult metadata;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Create a turn-scoped presenter.
|
||||
*******************************************************************************/
|
||||
public PresentResultsTool(QInstance qInstance)
|
||||
{
|
||||
this.qInstance = qInstance;
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stable schema. Strict opt-in requires empty placeholders instead of many
|
||||
** optional productions, keeping the existing strict grammar budget small.
|
||||
*******************************************************************************/
|
||||
public static ModelToolDefinition definition(Boolean strict)
|
||||
{
|
||||
try(InputStream stream = PresentResultsTool.class.getResourceAsStream("/present-results.schema.json"))
|
||||
{
|
||||
Map<String, Object> schema = JsonUtils.toObject(new String(stream.readAllBytes(), StandardCharsets.UTF_8), new TypeReference<Map<String, Object>>() {});
|
||||
if(strict)
|
||||
{
|
||||
requireProperties(schema);
|
||||
}
|
||||
return (new ModelToolDefinition().withName(NAME).withDescription("Present an answer as metrics, a table, a chart and actionable links. Use only data from tool results; never supply URLs.")
|
||||
.withInputSchema(schema).withStrict(strict));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalStateException("Cannot load presentation schema", e));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Eliminate optional productions for strict mode without adding unions.
|
||||
*******************************************************************************/
|
||||
private static void requireProperties(Object node)
|
||||
{
|
||||
if(node instanceof Map<?, ?>)
|
||||
{
|
||||
Map<String, Object> schema = object(node);
|
||||
schema.remove("maxLength");
|
||||
if(schema.get("properties") instanceof Map<?, ?> properties)
|
||||
{
|
||||
schema.put("required", new ArrayList<>(properties.keySet()));
|
||||
}
|
||||
schema.values().forEach(PresentResultsTool::requireProperties);
|
||||
}
|
||||
else if(node instanceof List<?> list)
|
||||
{
|
||||
list.forEach(PresentResultsTool::requireProperties);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Preserve valid items when others are dropped; never echo rejected values.
|
||||
*******************************************************************************/
|
||||
public synchronized AiToolResult execute(UiApiToolbox.Context context, String callId, Map<String, Object> input)
|
||||
{
|
||||
AiToolResult result = new AiToolResult().withToolCallId(callId).withToolName(NAME);
|
||||
try
|
||||
{
|
||||
require(input != null && input.get("summary") instanceof String, "summary required");
|
||||
String summary = text(input.get("summary"));
|
||||
require(!summary.isBlank() && summary.length() <= 300, "summary must contain 1 to 300 characters");
|
||||
List<String> notes = new ArrayList<>();
|
||||
List<Map<String, Object>> blocks = new ArrayList<>();
|
||||
addItems(blocks, "metrics", input.get("metrics"), notes, item -> metric(context, item));
|
||||
Map<String, Object> table = null;
|
||||
if(input.get("table") != null)
|
||||
{
|
||||
try
|
||||
{
|
||||
table = table(context, object(input.get("table")));
|
||||
if(!list(table.get("columns")).isEmpty())
|
||||
{
|
||||
blocks.add(table);
|
||||
}
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
notes.add("Dropped table: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
if(input.get("chart") != null && table != null && !list(table.get("rows")).isEmpty())
|
||||
{
|
||||
try
|
||||
{
|
||||
blocks.add(chart(object(input.get("chart")), table, notes));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
notes.add("Dropped chart: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
if(input.get("chart") != null && (table == null || list(table.get("rows")).isEmpty()))
|
||||
{
|
||||
notes.add("Dropped chart: requires a valid nonempty table.");
|
||||
}
|
||||
addItems(blocks, "filterLinks", input.get("filterLinks"), notes, item -> filterLink(context, item));
|
||||
addItems(blocks, "recordLinks", input.get("recordLinks"), notes, item -> recordLink(context, item));
|
||||
addItems(blocks, "actions", input.get("actions"), notes, item -> action(context, item));
|
||||
Map<String, Object> part = map("type", "presentation", "summary", summary, "blocks", blocks, "markdown", markdown(summary, blocks));
|
||||
return (result.withOutcome(AiToolOutcome.OK).withSummary("Presented results").withRender(part)
|
||||
.withPermissionCheck(metadata == null ? "presentation:no table references" : "uiApi:GET /metaData:" + metadata.status())
|
||||
.withModelContent("Presented to the user: " + summary + (summary.endsWith(".") ? "" : ".") + " Do not repeat the table or links in text." + (notes.isEmpty() ? "" : "\n" + String.join("\n", notes))));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
return (result.withOutcome(AiToolOutcome.INVALID_INPUT).withSummary("Invalid presentation").withModelContent(e.getMessage()));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve tables through the same metadata operation and session as the UI.
|
||||
*******************************************************************************/
|
||||
private QTableMetaData visibleTable(UiApiToolbox.Context context, String name)
|
||||
{
|
||||
QTableMetaData table = qInstance.getTable(name);
|
||||
require(table != null, "unknown table " + name);
|
||||
if(metadata == null)
|
||||
{
|
||||
try
|
||||
{
|
||||
metadata = context.gateway().invoke(new UiApiCall("GET", "/metaData").withSessionUUID(context.sessionUUID()).withUserTimezone(context.userTimezone()));
|
||||
}
|
||||
catch(QException e)
|
||||
{
|
||||
metadata = new UiApiResult(503, "{}");
|
||||
}
|
||||
}
|
||||
require(metadata.isSuccess(), "GET /metaData answered HTTP " + metadata.status());
|
||||
Map<String, Object> tables = object(object(metadata.json()).get("tables"));
|
||||
require(tables.containsKey(name) && Boolean.TRUE.equals(object(tables.get(name)).get("readPermission")), "table not visible through UI API: " + name);
|
||||
return (table);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A field reference must be paired with a visible table reference.
|
||||
*******************************************************************************/
|
||||
private QFieldMetaData field(UiApiToolbox.Context context, Map<String, Object> item)
|
||||
{
|
||||
String tableName = text(item.get("tableName"));
|
||||
String fieldName = text(item.get("fieldName"));
|
||||
require(fieldName.isEmpty() || !tableName.isEmpty(), "fieldName requires tableName");
|
||||
QTableMetaData table = tableName.isEmpty() ? null : visibleTable(context, tableName);
|
||||
QFieldMetaData field = fieldName.isEmpty() ? null : table.getFields().get(fieldName);
|
||||
require(fieldName.isEmpty() || field != null, "unknown field " + fieldName);
|
||||
return (field);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A metric preserves its numeric value and adds the metadata display.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> metric(UiApiToolbox.Context context, Map<String, Object> item)
|
||||
{
|
||||
QFieldMetaData field = field(context, item);
|
||||
number(item.get("value"));
|
||||
return (map("label", label(item), "value", item.get("value"), "display", display(item.get("value"), field)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Build table cells and percentages from raw values, never model displays.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> table(UiApiToolbox.Context context, Map<String, Object> input)
|
||||
{
|
||||
List<?> columns = list(input.get("columns"));
|
||||
List<?> rows = list(input.get("rows"));
|
||||
require(columns.size() <= 20 && rows.size() <= 200, "table exceeds 20 columns or 200 rows");
|
||||
require(rows.isEmpty() || !columns.isEmpty(), "rows require columns");
|
||||
List<QFieldMetaData> fields = new ArrayList<>();
|
||||
List<Map<String, Object>> outputColumns = new ArrayList<>();
|
||||
List<BigDecimal> totals = new ArrayList<>();
|
||||
for(Integer c = 0; c < columns.size(); c++)
|
||||
{
|
||||
Map<String, Object> column = object(columns.get(c));
|
||||
fields.add(field(context, column));
|
||||
BigDecimal total = BigDecimal.ZERO;
|
||||
Boolean numeric = !rows.isEmpty();
|
||||
for(Object row : rows)
|
||||
{
|
||||
require(list(row).size() == columns.size(), "row width must match columns");
|
||||
Object value = list(row).get(c);
|
||||
scalar(value);
|
||||
if(value instanceof Number)
|
||||
{
|
||||
total = total.add(number(value));
|
||||
}
|
||||
else
|
||||
{
|
||||
numeric = false;
|
||||
}
|
||||
}
|
||||
totals.add(numeric ? total : null);
|
||||
outputColumns.add(map("key", "c" + c, "label", label(column), "align", numeric || percentColumn(column) >= 0 ? "right" : "left"));
|
||||
}
|
||||
Map<String, Object> rowFilter = input.get("rowFilter") == null ? Map.of() : object(input.get("rowFilter"));
|
||||
List<?> rowKeys = input.get("rowKeys") == null ? List.of() : list(input.get("rowKeys"));
|
||||
Boolean hasFilter = !text(rowFilter.get("tableName")).isEmpty();
|
||||
require(!hasFilter || rowKeys.size() == rows.size(), "rowKeys must match rows");
|
||||
QFieldMetaData filterField = field(context, rowFilter);
|
||||
require(!hasFilter || filterField != null, "rowFilter requires a field");
|
||||
List<Map<String, Object>> outputRows = new ArrayList<>();
|
||||
for(Integer r = 0; r < rows.size(); r++)
|
||||
{
|
||||
Map<String, Object> cells = new LinkedHashMap<>();
|
||||
Map<String, Object> displays = new LinkedHashMap<>();
|
||||
for(Integer c = 0; c < columns.size(); c++)
|
||||
{
|
||||
Integer source = percentColumn(object(columns.get(c)));
|
||||
Object value = list(rows.get(r)).get(c);
|
||||
String formatted;
|
||||
if(source >= 0)
|
||||
{
|
||||
require(source < columns.size() && totals.get(source) != null && percentColumn(object(columns.get(source))) < 0, "percentOfColumn must reference a numeric raw column");
|
||||
BigDecimal share = totals.get(source).signum() == 0 ? BigDecimal.ZERO : number(list(rows.get(r)).get(source)).divide(totals.get(source), MathContext.DECIMAL64);
|
||||
value = share;
|
||||
formatted = String.format(Locale.US, "%.1f%%", share.multiply(BigDecimal.valueOf(100)));
|
||||
}
|
||||
else
|
||||
{
|
||||
formatted = display(value, fields.get(c));
|
||||
}
|
||||
cells.put("c" + c, value);
|
||||
displays.put("c" + c, formatted);
|
||||
}
|
||||
Map<String, Object> row = map("cells", cells, "display", displays);
|
||||
if(hasFilter)
|
||||
{
|
||||
Object key = rowKeys.get(r);
|
||||
scalar(key);
|
||||
Map<String, Object> criterion = map("fieldName", rowFilter.get("fieldName"), "operator", rowFilter.getOrDefault("operator", "EQUALS"), "values", java.util.Collections.singletonList(key));
|
||||
row.put("href", filterHref(context, text(rowFilter.get("tableName")), List.of(criterion)));
|
||||
}
|
||||
outputRows.add(row);
|
||||
}
|
||||
return (map("kind", "table", "title", text(input.get("title")), "columns", outputColumns, "rows", outputRows));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A chart uses the table's raw values, formatted values and row links.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> chart(Map<String, Object> input, Map<String, Object> table, List<String> notes)
|
||||
{
|
||||
String type = text(input.get("chartType"));
|
||||
require(List.of("bar", "pie", "line").contains(type), "invalid chartType");
|
||||
Integer labelColumn = index(input.get("labelColumn"));
|
||||
Integer valueColumn = index(input.get("valueColumn"));
|
||||
Integer width = list(table.get("columns")).size();
|
||||
require(labelColumn >= 0 && valueColumn >= 0 && labelColumn < width && valueColumn < width, "chart column index out of range");
|
||||
List<Map<String, Object>> points = new ArrayList<>();
|
||||
List<?> rows = list(table.get("rows"));
|
||||
for(Object entry : rows)
|
||||
{
|
||||
Map<String, Object> row = object(entry);
|
||||
Object value = object(row.get("cells")).get("c" + valueColumn);
|
||||
number(value);
|
||||
if(points.size() < 12)
|
||||
{
|
||||
Map<String, Object> point = map("label", object(row.get("display")).get("c" + labelColumn), "value", value, "display", object(row.get("display")).get("c" + valueColumn));
|
||||
if(row.containsKey("href"))
|
||||
{
|
||||
point.put("href", row.get("href"));
|
||||
}
|
||||
points.add(point);
|
||||
}
|
||||
}
|
||||
if(rows.size() > 12)
|
||||
{
|
||||
notes.add("Chart truncated to 12 points; the table retains all rows.");
|
||||
}
|
||||
return (map("kind", "chart", "chartType", type, "title", text(input.get("title")), "points", points));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A filtered list link, including only validated metadata and raw values.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> filterLink(UiApiToolbox.Context context, Map<String, Object> item)
|
||||
{
|
||||
String tableName = text(item.get("tableName"));
|
||||
return (map("label", label(item), "tableName", tableName, "href", filterHref(context, tableName, list(item.get("criteria")))));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Serialize the UI's JSON filter format. The table path uses core navigate's
|
||||
** helper; filter values stay raw inside the URL-encoded JSON.
|
||||
*******************************************************************************/
|
||||
private String filterHref(UiApiToolbox.Context context, String tableName, List<?> criteria)
|
||||
{
|
||||
visibleTable(context, tableName);
|
||||
require(!criteria.isEmpty() && criteria.size() <= 20, "filter requires 1 to 20 criteria");
|
||||
List<Map<String, Object>> output = new ArrayList<>();
|
||||
for(Object entry : criteria)
|
||||
{
|
||||
Map<String, Object> criterion = object(entry);
|
||||
String fieldName = text(criterion.get("fieldName"));
|
||||
require(!fieldName.isBlank(), "filter requires fieldName");
|
||||
field(context, map("tableName", tableName, "fieldName", fieldName));
|
||||
String operator = text(criterion.get("operator"));
|
||||
operator = operator.isEmpty() ? "EQUALS" : operator;
|
||||
try
|
||||
{
|
||||
QCriteriaOperator.valueOf(operator);
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
throw (new IllegalArgumentException("invalid criteria operator"));
|
||||
}
|
||||
List<?> values = list(criterion.get("values"));
|
||||
require(values.size() <= 200, "too many filter values");
|
||||
values.forEach(PresentResultsTool::scalar);
|
||||
output.add(map("fieldName", fieldName, "operator", operator, "values", values));
|
||||
}
|
||||
return (safeHref(AiLinkBuilder.objectHref(tableName) + "?filter=" + URLEncoder.encode(JsonUtils.toJson(map("criteria", output, "booleanOperator", "AND")), StandardCharsets.UTF_8)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Record ids are opaque path segments, encoded by core's link helper.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> recordLink(UiApiToolbox.Context context, Map<String, Object> item)
|
||||
{
|
||||
String tableName = text(item.get("tableName"));
|
||||
visibleTable(context, tableName);
|
||||
Object id = item.get("recordId");
|
||||
require(id instanceof String || id instanceof Number, "recordId must be a string or number");
|
||||
String recordId = text(id);
|
||||
require(!recordId.isBlank() && !List.of(".", "..").contains(recordId), "invalid recordId");
|
||||
return (map("label", label(item), "tableName", tableName, "recordId", recordId, "href", safeHref(AiLinkBuilder.recordHref(tableName, recordId))));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Actions are either a next-message prompt or a visible table destination.
|
||||
*******************************************************************************/
|
||||
private Map<String, Object> action(UiApiToolbox.Context context, Map<String, Object> item)
|
||||
{
|
||||
String prompt = text(item.get("prompt"));
|
||||
String destination = text(item.get("navigateTo"));
|
||||
require(prompt.isBlank() != destination.isBlank(), "action requires exactly one of prompt or navigateTo");
|
||||
if(!destination.isBlank())
|
||||
{
|
||||
visibleTable(context, destination);
|
||||
return (map("label", label(item), "href", safeHref(AiLinkBuilder.objectHref(destination))));
|
||||
}
|
||||
return (map("label", label(item), "prompt", prompt));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Preserve independently valid items and report dropped positions.
|
||||
*******************************************************************************/
|
||||
private static void addItems(List<Map<String, Object>> blocks, String kind, Object input, List<String> notes, Function<Map<String, Object>, Map<String, Object>> build)
|
||||
{
|
||||
if(input == null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
List<?> entries = list(input);
|
||||
require(entries.size() <= 100, "too many " + kind);
|
||||
List<Map<String, Object>> items = new ArrayList<>();
|
||||
for(Integer i = 0; i < entries.size(); i++)
|
||||
{
|
||||
try
|
||||
{
|
||||
items.add(build.apply(object(entries.get(i))));
|
||||
}
|
||||
catch(IllegalArgumentException e)
|
||||
{
|
||||
notes.add("Dropped " + kind + "[" + i + "]: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
if(!items.isEmpty())
|
||||
{
|
||||
blocks.add(map("kind", kind, "items", items));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Metadata formats use QQQ field coercion and Java display format strings.
|
||||
*******************************************************************************/
|
||||
private static String display(Object value, QFieldMetaData field)
|
||||
{
|
||||
if(value == null)
|
||||
{
|
||||
return ("");
|
||||
}
|
||||
if(field != null)
|
||||
{
|
||||
try
|
||||
{
|
||||
Object typed = ValueUtils.getValueAsFieldType(field.getType(), value);
|
||||
if(field.getDisplayFormat() != null)
|
||||
{
|
||||
return (String.format(Locale.US, field.getDisplayFormat(), typed));
|
||||
}
|
||||
value = typed;
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new IllegalArgumentException("value does not match field format"));
|
||||
}
|
||||
}
|
||||
if(value instanceof Number)
|
||||
{
|
||||
BigDecimal decimal = number(value);
|
||||
return (decimal.stripTrailingZeros().scale() <= 0 ? String.format(Locale.US, "%,d", decimal.toBigIntegerExact()) : decimal.toPlainString());
|
||||
}
|
||||
return (text(value));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Complete GFM fallback with real newlines; chart data is already in table.
|
||||
*******************************************************************************/
|
||||
private static String markdown(String summary, List<Map<String, Object>> blocks)
|
||||
{
|
||||
StringBuilder markdown = new StringBuilder(escape(summary)).append("\n");
|
||||
for(Map<String, Object> block : blocks)
|
||||
{
|
||||
if("table".equals(block.get("kind")))
|
||||
{
|
||||
markdown.append("\n").append(escape(text(block.get("title")))).append("\n\n|");
|
||||
List<?> columns = list(block.get("columns"));
|
||||
for(Object column : columns)
|
||||
{
|
||||
markdown.append(' ').append(escape(text(object(column).get("label")))).append(" |");
|
||||
}
|
||||
markdown.append("\n|");
|
||||
for(Object column : columns)
|
||||
{
|
||||
markdown.append("right".equals(object(column).get("align")) ? " ---: |" : " --- |");
|
||||
}
|
||||
for(Object entry : list(block.get("rows")))
|
||||
{
|
||||
Map<String, Object> row = object(entry);
|
||||
markdown.append("\n|");
|
||||
for(Integer c = 0; c < columns.size(); c++)
|
||||
{
|
||||
String value = escape(text(object(row.get("display")).get("c" + c)));
|
||||
markdown.append(' ').append(c == 0 && row.containsKey("href") ? "[" + value + "](" + row.get("href") + ")" : value).append(" |");
|
||||
}
|
||||
}
|
||||
markdown.append("\n");
|
||||
}
|
||||
else if(block.get("items") instanceof List<?> items)
|
||||
{
|
||||
markdown.append("\n");
|
||||
for(Object entry : items)
|
||||
{
|
||||
Map<String, Object> item = object(entry);
|
||||
String label = escape(text(item.get("label")));
|
||||
markdown.append("- ");
|
||||
if(item.containsKey("href"))
|
||||
{
|
||||
markdown.append('[').append(label).append("](").append(item.get("href")).append(')');
|
||||
}
|
||||
else
|
||||
{
|
||||
markdown.append(label);
|
||||
if(item.containsKey("display") || item.containsKey("prompt"))
|
||||
{
|
||||
markdown.append(": ").append(escape(text(item.getOrDefault("display", item.get("prompt")))));
|
||||
}
|
||||
}
|
||||
markdown.append("\n");
|
||||
}
|
||||
}
|
||||
}
|
||||
return (markdown.toString());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Literal text only: no Markdown links, HTML or GFM URL autolinks from labels.
|
||||
*******************************************************************************/
|
||||
private static String escape(String value)
|
||||
{
|
||||
StringBuilder output = new StringBuilder();
|
||||
for(char character : value.toCharArray())
|
||||
{
|
||||
if("\\`*_{}[]()<>|!&#:.@~".indexOf(character) >= 0)
|
||||
{
|
||||
output.append("&#").append((int) character).append(';');
|
||||
}
|
||||
else
|
||||
{
|
||||
output.append(character);
|
||||
}
|
||||
}
|
||||
return (output.toString());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reject unsafe paths even if metadata contains unexpected names.
|
||||
*******************************************************************************/
|
||||
private static String safeHref(String href)
|
||||
{
|
||||
require(AiLinkBuilder.isSafeAppPath(href), "unsafe application path");
|
||||
return (href);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A column with no percentage uses -1 in strict mode.
|
||||
*******************************************************************************/
|
||||
private static Integer percentColumn(Map<String, Object> column)
|
||||
{
|
||||
return (column.get("percentOfColumn") == null ? -1 : index(column.get("percentOfColumn")));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Indexes must be integral; never silently truncate model input.
|
||||
*******************************************************************************/
|
||||
private static Integer index(Object value)
|
||||
{
|
||||
try
|
||||
{
|
||||
return (number(value).intValueExact());
|
||||
}
|
||||
catch(ArithmeticException e)
|
||||
{
|
||||
throw (new IllegalArgumentException("index must be an integer"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Finite numeric input only.
|
||||
*******************************************************************************/
|
||||
private static BigDecimal number(Object value)
|
||||
{
|
||||
require(value instanceof Number, "numeric value required");
|
||||
try
|
||||
{
|
||||
return (new BigDecimal(value.toString()));
|
||||
}
|
||||
catch(NumberFormatException e)
|
||||
{
|
||||
throw (new IllegalArgumentException("finite numeric value required"));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cells and filter values remain raw JSON scalars.
|
||||
*******************************************************************************/
|
||||
private static void scalar(Object value)
|
||||
{
|
||||
require(value == null || value instanceof String || value instanceof Number || value instanceof Boolean, "scalar value required");
|
||||
if(value instanceof Number)
|
||||
{
|
||||
number(value);
|
||||
}
|
||||
if(value instanceof String)
|
||||
{
|
||||
text(value);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bounded single-line display text.
|
||||
*******************************************************************************/
|
||||
private static String text(Object value)
|
||||
{
|
||||
require(value == null || value instanceof String || value instanceof Number || value instanceof Boolean, "text or scalar required");
|
||||
String text = value == null ? "" : String.valueOf(value);
|
||||
require(text.length() <= 2000, "text exceeds 2000 characters");
|
||||
return (text.replaceAll("[\\p{Cntrl}\\u2028\\u2029]", " "));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Labels are mandatory plain text.
|
||||
*******************************************************************************/
|
||||
private static String label(Map<String, Object> item)
|
||||
{
|
||||
String label = text(item.get("label"));
|
||||
require(!label.isBlank(), "label required");
|
||||
return (label);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validate a JSON object without converting arbitrary values to strings.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
private static Map<String, Object> object(Object value)
|
||||
{
|
||||
require(value instanceof Map<?, ?>, "object required");
|
||||
return ((Map<String, Object>) value);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validate a JSON array.
|
||||
*******************************************************************************/
|
||||
private static List<?> list(Object value)
|
||||
{
|
||||
require(value instanceof List<?>, "array required");
|
||||
return ((List<?>) value);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Ordered output maps allow null raw cells.
|
||||
*******************************************************************************/
|
||||
private static Map<String, Object> map(Object... pairs)
|
||||
{
|
||||
Map<String, Object> map = new LinkedHashMap<>();
|
||||
for(Integer i = 0; i < pairs.length; i += 2)
|
||||
{
|
||||
map.put((String) pairs[i], pairs[i + 1]);
|
||||
}
|
||||
return (map);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Validation failures contain only server-authored explanations.
|
||||
*******************************************************************************/
|
||||
private static void require(Boolean condition, String message)
|
||||
{
|
||||
if(!condition)
|
||||
{
|
||||
throw (new IllegalArgumentException(message));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"maxLength": 300
|
||||
},
|
||||
"metrics": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/metric"
|
||||
}
|
||||
},
|
||||
"table": {
|
||||
"$ref": "#/$defs/table"
|
||||
},
|
||||
"chart": {
|
||||
"$ref": "#/$defs/chart"
|
||||
},
|
||||
"filterLinks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/filterLink"
|
||||
}
|
||||
},
|
||||
"recordLinks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/recordLink"
|
||||
}
|
||||
},
|
||||
"actions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/action"
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"summary"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"$defs": {
|
||||
"cell": {
|
||||
"type": [
|
||||
"string",
|
||||
"number",
|
||||
"boolean",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"metric": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"value": {
|
||||
"type": "number"
|
||||
},
|
||||
"tableName": {
|
||||
"type": "string"
|
||||
},
|
||||
"fieldName": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"label",
|
||||
"value"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"column": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"tableName": {
|
||||
"type": "string"
|
||||
},
|
||||
"fieldName": {
|
||||
"type": "string"
|
||||
},
|
||||
"percentOfColumn": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"label"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"rowFilter": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"tableName": {
|
||||
"type": "string"
|
||||
},
|
||||
"fieldName": {
|
||||
"type": "string"
|
||||
},
|
||||
"operator": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"tableName",
|
||||
"fieldName"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"criterion": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"fieldName": {
|
||||
"type": "string"
|
||||
},
|
||||
"operator": {
|
||||
"type": "string"
|
||||
},
|
||||
"values": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/cell"
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"fieldName",
|
||||
"operator",
|
||||
"values"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"filterLink": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"tableName": {
|
||||
"type": "string"
|
||||
},
|
||||
"criteria": {
|
||||
"$ref": "#/$defs/criteria"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"label",
|
||||
"tableName",
|
||||
"criteria"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"recordLink": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"tableName": {
|
||||
"type": "string"
|
||||
},
|
||||
"recordId": {
|
||||
"$ref": "#/$defs/cell"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"label",
|
||||
"tableName",
|
||||
"recordId"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"action": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"prompt": {
|
||||
"type": "string"
|
||||
},
|
||||
"navigateTo": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"label"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"chart": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"chartType": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"bar",
|
||||
"pie",
|
||||
"line"
|
||||
]
|
||||
},
|
||||
"title": {
|
||||
"type": "string"
|
||||
},
|
||||
"labelColumn": {
|
||||
"type": "integer"
|
||||
},
|
||||
"valueColumn": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"chartType",
|
||||
"labelColumn",
|
||||
"valueColumn"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"row": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/cell"
|
||||
}
|
||||
},
|
||||
"columns": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/column"
|
||||
}
|
||||
},
|
||||
"rows": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/row"
|
||||
}
|
||||
},
|
||||
"rowKeys": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/cell"
|
||||
}
|
||||
},
|
||||
"criteria": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/criterion"
|
||||
}
|
||||
},
|
||||
"table": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": {
|
||||
"type": "string"
|
||||
},
|
||||
"columns": {
|
||||
"$ref": "#/$defs/columns"
|
||||
},
|
||||
"rows": {
|
||||
"$ref": "#/$defs/rows"
|
||||
},
|
||||
"rowFilter": {
|
||||
"$ref": "#/$defs/rowFilter"
|
||||
},
|
||||
"rowKeys": {
|
||||
"$ref": "#/$defs/rowKeys"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"columns",
|
||||
"rows"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"description": "Present only facts from previous tool results. Optional sections may be omitted. In strict mode use empty arrays, empty strings, empty table rows/columns and -1 for unused indexes; every property must be supplied. rowKeys belongs to table, one raw key per row. Percent columns may use null cells; the server computes them."
|
||||
}
|
||||
+4
@@ -93,6 +93,10 @@ class ClassificationTest extends BaseAssistantTest
|
||||
assertThat(record.getClassification()).isEqualTo(DataClassification.PHI);
|
||||
page.observe(ModelPart.toolUse("1", "describe_table", Map.of("table", "page")), new AiToolResult());
|
||||
assertThat(page.getClassification()).isEqualTo(DataClassification.PUBLIC);
|
||||
page.observe(ModelPart.toolUse("present", "present_results", Map.of("summary", "Available results")), new AiToolResult());
|
||||
assertThat(page.getClassification()).isEqualTo(DataClassification.PUBLIC);
|
||||
record.observe(ModelPart.toolUse("present", "present_results", Map.of("summary", "Available results")), new AiToolResult());
|
||||
assertThat(record.getClassification()).isEqualTo(DataClassification.PHI);
|
||||
page.observe(ModelPart.toolUse("2", "search_records", Map.of()), new AiToolResult().withModelContent("<qqq_data>{\"results\":[{\"table\":\"media\"}]}</qqq_data>"));
|
||||
assertThat(page.getClassification()).isEqualTo(DataClassification.PII);
|
||||
page.observe(ModelPart.toolUse("3", "search_records", Map.of()), new AiToolResult());
|
||||
|
||||
+7
-3
@@ -226,9 +226,13 @@ public class AssistantTestFixture
|
||||
update("page", new QRecord().withValue("id", 12).withValue("metaDescription", "Who I am and what I build."));
|
||||
update("page", new QRecord().withValue("id", 12).withValue("content", "Body of page 12, rewritten with a longer story about the home lab."));
|
||||
|
||||
insert("media", List.of(
|
||||
new QRecord().withValue("fileName", "cover.heic").withValue("mimeType", "image/heic"),
|
||||
new QRecord().withValue("fileName", "logo.png").withValue("mimeType", "image/png").withValue("width", 512)));
|
||||
List<QRecord> media = new ArrayList<>();
|
||||
for(Integer i = 0; i < 101; i++)
|
||||
{
|
||||
media.add(new QRecord().withValue("fileName", "image-" + i + (i < 81 ? ".jpg" : ".png"))
|
||||
.withValue("mimeType", i < 81 ? "image/jpeg" : "image/png").withValue("width", 512));
|
||||
}
|
||||
insert("media", media);
|
||||
|
||||
List<QRecord> stats = new ArrayList<>();
|
||||
for(int d = 1; d <= 5; d++)
|
||||
|
||||
@@ -180,6 +180,7 @@ public class GoldenTasks
|
||||
},
|
||||
o -> live(o, "search_records", "/app/blog_post/1")));
|
||||
|
||||
tasks.add(PresentationGoldenTask.task());
|
||||
return (tasks);
|
||||
}
|
||||
|
||||
|
||||
+171
@@ -0,0 +1,171 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.assistant.testing;
|
||||
|
||||
|
||||
import java.net.URLDecoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.kingsrook.qbits.assistant.engine.TurnEvent;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelPart;
|
||||
import com.kingsrook.qbits.assistant.provider.ModelResponse;
|
||||
import com.kingsrook.qqq.backend.core.utils.JsonUtils;
|
||||
import static com.kingsrook.qbits.assistant.testing.GoldenTasks.map;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Owner's image breakdown, shared by deterministic and live provider evals.
|
||||
*******************************************************************************/
|
||||
public class PresentationGoldenTask
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Counts come from UI API tool results, including the omitted zero types.
|
||||
*******************************************************************************/
|
||||
public static GoldenTasks.Task task()
|
||||
{
|
||||
return (new GoldenTasks.Task("images-by-type-presentation", AssistantTestFixture.EDITOR, "How many images do I have by type?", provider ->
|
||||
{
|
||||
List<Object[]> calls = new ArrayList<>();
|
||||
for(String mime : List.of("image/jpeg", "image/png", "image/webp", "image/gif", "image/svg+xml"))
|
||||
{
|
||||
calls.add(new Object[] {"count_records", map("table", "media", "filter", GoldenTasks.filter("AND", GoldenTasks.criterion("mimeType", "EQUALS", mime)))});
|
||||
}
|
||||
provider.thenToolUses(null, calls).thenRespond(request ->
|
||||
{
|
||||
List<ModelPart> results = EvalSupport.lastToolResults(request);
|
||||
Integer jpeg = ((Number) EvalSupport.unwrap(results.get(0).getText()).get("count")).intValue();
|
||||
Integer png = ((Number) EvalSupport.unwrap(results.get(1).getText()).get("count")).intValue();
|
||||
for(Integer i = 2; i < results.size(); i++)
|
||||
{
|
||||
assertThat(((Number) EvalSupport.unwrap(results.get(i).getText()).get("count")).intValue()).isZero();
|
||||
}
|
||||
return (new ModelResponse().withParts(List.of(ModelPart.toolUse("present", "present_results", input(jpeg, png)))).withStopReason(ModelResponse.STOP_TOOL_USE));
|
||||
}).thenText("");
|
||||
}, PresentationGoldenTask::check, PresentationGoldenTask::check));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The presentation input, deliberately using null percentage placeholders.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> input(Integer jpeg, Integer png)
|
||||
{
|
||||
return (map("summary", (jpeg + png) + " media files are images: " + jpeg + " JPEG and " + png + " PNG.",
|
||||
"metrics", List.of(map("label", "Images", "value", jpeg + png)),
|
||||
"table", map("title", "Media by type", "columns", List.of(map("label", "Type"), map("label", "Count"), map("label", "Share", "percentOfColumn", 1)),
|
||||
"rows", List.of(java.util.Arrays.asList("JPEG", jpeg, null), java.util.Arrays.asList("PNG", png, null)),
|
||||
"rowFilter", map("tableName", "media", "fieldName", "mimeType"), "rowKeys", List.of("image/jpeg", "image/png")),
|
||||
"chart", map("chartType", "pie", "title", "Share by type", "labelColumn", 0, "valueColumn", 1),
|
||||
"filterLinks", List.of(filterLink("JPEG", "image/jpeg"), filterLink("PNG", "image/png"))));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A raw MIME criterion for the presentation tool.
|
||||
*******************************************************************************/
|
||||
private static Map<String, Object> filterLink(String label, String mime)
|
||||
{
|
||||
return (map("label", "Open " + label + " media", "tableName", "media", "criteria", List.of(map("fieldName", "mimeType", "operator", "EQUALS", "values", List.of(mime)))));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Check actual block semantics, allowing a live model to choose labels and
|
||||
** column order. Values, shares and filter criteria must remain exact.
|
||||
*******************************************************************************/
|
||||
public static void check(TurnRunner.Outcome outcome)
|
||||
{
|
||||
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
|
||||
assertThat(outcome.toolCalls()).anyMatch(call -> "count_records".equals(call.getToolName()) && "OK".equals(call.getOutcome()));
|
||||
assertThat(outcome.events(TurnEvent.PRESENTATION)).hasSize(1);
|
||||
Map<String, Object> part = object(outcome.events(TurnEvent.PRESENTATION).get(0).get("part"));
|
||||
assertPart(part);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Also used against the persisted part, not only the streamed event.
|
||||
*******************************************************************************/
|
||||
public static void assertPart(Map<String, Object> part)
|
||||
{
|
||||
assertThat(part.get("type")).isEqualTo("presentation");
|
||||
assertThat((String) part.get("summary")).isNotBlank().doesNotContain("\n").hasSizeLessThanOrEqualTo(300);
|
||||
assertThat((String) part.get("markdown")).contains("\n|");
|
||||
Map<String, Object> table = block(part, "table");
|
||||
List<?> rows = (List<?>) table.get("rows");
|
||||
assertThat(rows).hasSize(2);
|
||||
assertThat(rows).anySatisfy(row -> assertThat(object(object(row).get("display")).values()).contains("81", "80.2%"));
|
||||
assertThat(rows).anySatisfy(row -> assertThat(object(object(row).get("display")).values()).contains("20", "19.8%"));
|
||||
Map<String, Object> chart = block(part, "chart");
|
||||
assertThat(chart.get("chartType")).isEqualTo("pie");
|
||||
List<?> points = (List<?>) chart.get("points");
|
||||
assertThat(points).hasSize(2);
|
||||
assertThat(points.stream().map(p -> ((Number) object(p).get("value")).intValue()).toList()).containsExactlyInAnyOrder(81, 20);
|
||||
List<?> links = (List<?>) block(part, "filterLinks").get("items");
|
||||
assertThat(links).hasSize(2);
|
||||
List<Object> mimes = new ArrayList<>();
|
||||
for(Object entry : links)
|
||||
{
|
||||
String href = (String) object(entry).get("href");
|
||||
assertThat(href).startsWith("/app/media?filter=");
|
||||
try
|
||||
{
|
||||
Map<String, Object> filter = JsonUtils.toObject(URLDecoder.decode(href.substring(href.indexOf('=') + 1), StandardCharsets.UTF_8), new TypeReference<Map<String, Object>>() {});
|
||||
assertThat(filter).containsEntry("booleanOperator", "AND");
|
||||
List<?> criteria = (List<?>) filter.get("criteria");
|
||||
assertThat(criteria).hasSize(1);
|
||||
Map<String, Object> criterion = object(criteria.get(0));
|
||||
assertThat(criterion).containsEntry("fieldName", "mimeType").containsEntry("operator", "EQUALS");
|
||||
mimes.addAll((List<?>) criterion.get("values"));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
throw (new AssertionError("Invalid filtered list href", e));
|
||||
}
|
||||
}
|
||||
assertThat(mimes).containsExactlyInAnyOrder("image/jpeg", "image/png");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Find one block of a given kind.
|
||||
*******************************************************************************/
|
||||
public static Map<String, Object> block(Map<String, Object> part, String kind)
|
||||
{
|
||||
return (((List<?>) part.get("blocks")).stream().map(PresentationGoldenTask::object).filter(block -> kind.equals(block.get("kind"))).findFirst().orElseThrow());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Typed test view of a JSON object.
|
||||
*******************************************************************************/
|
||||
@SuppressWarnings("unchecked")
|
||||
public static Map<String, Object> object(Object value)
|
||||
{
|
||||
return ((Map<String, Object>) value);
|
||||
}
|
||||
}
|
||||
@@ -59,7 +59,8 @@ public class TurnRunner
|
||||
*******************************************************************************/
|
||||
public String text()
|
||||
{
|
||||
return (events.stream().filter(e -> TurnEvent.TEXT_DELTA.equals(e.getType())).map(e -> String.valueOf(e.getData().get("text"))).collect(Collectors.joining()));
|
||||
return (events.stream().filter(e -> TurnEvent.TEXT_DELTA.equals(e.getType()) || TurnEvent.PRESENTATION.equals(e.getType()))
|
||||
.map(e -> TurnEvent.PRESENTATION.equals(e.getType()) ? String.valueOf(((Map<?, ?>) e.getData().get("part")).get("markdown")) : String.valueOf(e.getData().get("text"))).collect(Collectors.joining()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
+275
@@ -0,0 +1,275 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.assistant.tools;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.assistant.engine.MessageParts;
|
||||
import com.kingsrook.qbits.assistant.engine.PolicyService;
|
||||
import com.kingsrook.qbits.assistant.engine.TurnEvent;
|
||||
import com.kingsrook.qbits.assistant.model.AssistantMessage;
|
||||
import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
|
||||
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
|
||||
import com.kingsrook.qbits.assistant.testing.PresentationGoldenTask;
|
||||
import com.kingsrook.qbits.assistant.testing.TurnRunner;
|
||||
import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
|
||||
import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
|
||||
import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
|
||||
import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.assistant.testing.PresentationGoldenTask.block;
|
||||
import static com.kingsrook.qbits.assistant.testing.PresentationGoldenTask.object;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Presentation persistence, permission boundaries, formatting and input limits.
|
||||
*******************************************************************************/
|
||||
class PresentResultsToolTest extends BaseAssistantTest
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Streamed, replayed and stored presentations agree; retention removes data.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testPersistenceAndReplay() throws Exception
|
||||
{
|
||||
PresentationGoldenTask.task().script().accept(provider);
|
||||
TurnRunner.Outcome outcome = runner.run(AssistantTestFixture.EDITOR, "How many images do I have by type?");
|
||||
PresentationGoldenTask.check(outcome);
|
||||
QContext.init(qInstance, AssistantTestFixture.ownerSession(AssistantTestFixture.EDITOR, config));
|
||||
Map<String, Object> event = outcome.events(TurnEvent.PRESENTATION).get(0);
|
||||
AssistantMessage message = runner.runtime().getStore().getMessage(AssistantTestFixture.EDITOR, (Integer) event.get("messageId"));
|
||||
MessageParts parts = MessageParts.fromJson(message.getPartsJson());
|
||||
Map<String, Object> part = parts.list().stream().filter(p -> "presentation".equals(p.get("type"))).findFirst().orElseThrow();
|
||||
PresentationGoldenTask.assertPart(part);
|
||||
assertThat(part).isEqualTo(com.kingsrook.qqq.backend.core.utils.JsonUtils.toObject(com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(event.get("part")), Map.class));
|
||||
List<?> rows = (List<?>) block(part, "table").get("rows");
|
||||
List<?> points = (List<?>) block(part, "chart").get("points");
|
||||
List<?> links = (List<?>) block(part, "filterLinks").get("items");
|
||||
for(Integer i = 0; i < 2; i++)
|
||||
{
|
||||
assertThat(object(rows.get(i)).get("href")).isEqualTo(object(links.get(i)).get("href"));
|
||||
assertThat(object(points.get(i)).get("href")).isEqualTo(object(rows.get(i)).get("href"));
|
||||
}
|
||||
assertThat(runner.runtime().getStore().listEvents(AssistantTestFixture.EDITOR, outcome.turnId(), 0, 1000))
|
||||
.anyMatch(e -> TurnEvent.PRESENTATION.equals(e.getEventType()));
|
||||
assertThat(parts.text()).contains("81", "20", "\n|");
|
||||
assertThat(parts.stripRecordData()).isTrue();
|
||||
assertThat(parts.list()).noneMatch(p -> "presentation".equals(p.get("type")));
|
||||
assertThat(outcome.toolCalls()).anySatisfy(call ->
|
||||
{
|
||||
assertThat(call.getToolName()).isEqualTo(PresentResultsTool.NAME);
|
||||
assertThat(call.getInputJson()).contains("presentation");
|
||||
assertThat(call.getResultJson()).contains("Do not repeat");
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The real UI metadata hides media from an analyst, as does presentation.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testUiDenial() throws Exception
|
||||
{
|
||||
String user = AssistantTestFixture.ANALYST;
|
||||
UiApiResult metadata = runner.runtime().getUiApiGateway().invoke(new UiApiCall("GET", "/metaData").withSessionUUID(user));
|
||||
assertThat(object(metadata.json().get("tables"))).doesNotContainKey("media");
|
||||
provider.thenToolUse(PresentResultsTool.NAME, input("summary", "The list is unavailable.", "recordLinks", List.of(input("label", "Image", "tableName", "media", "recordId", "1")))).thenText("");
|
||||
TurnRunner.Outcome outcome = runner.run(user, "Open an image");
|
||||
assertThat(outcome.toolCalls().get(0).getResultJson()).contains("table not visible through UI API: media");
|
||||
Map<String, Object> part = object(outcome.events(TurnEvent.PRESENTATION).get(0).get("part"));
|
||||
assertThat((List<?>) part.get("blocks")).isEmpty();
|
||||
assertThat(outcome.toolCalls().get(0).getInputJson()).doesNotContain("recordId");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Cache metadata once per turn and send the original session UUID.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testMetadataCache() throws Exception
|
||||
{
|
||||
List<UiApiCall> calls = new ArrayList<>();
|
||||
UiApiGateway gateway = new UiApiGateway()
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Fixture metadata gateway.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public boolean isInvocable(String method, String path)
|
||||
{
|
||||
return (true);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixture metadata gateway.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public UiApiResult invoke(UiApiCall call)
|
||||
{
|
||||
calls.add(call);
|
||||
return (new UiApiResult(200, "{\"tables\":{\"media\":{\"readPermission\":true}}}"));
|
||||
}
|
||||
};
|
||||
UiApiToolbox.Context context = new UiApiToolbox.Context("editor-session", "UTC", gateway, PolicyService.ToolPolicy.defaults(), null);
|
||||
PresentResultsTool tool = new PresentResultsTool(qInstance);
|
||||
assertThat(tool.execute(context, "1", PresentationGoldenTask.input(81, 20)).getOutcome()).isEqualTo(AiToolOutcome.OK);
|
||||
tool.execute(context, "2", PresentationGoldenTask.input(81, 20));
|
||||
assertThat(calls).hasSize(1).allMatch(call -> "editor-session".equals(call.getSessionUUID()) && "/metaData".equals(call.getPathTemplate()));
|
||||
new PresentResultsTool(qInstance).execute(context, "3", PresentationGoldenTask.input(81, 20));
|
||||
assertThat(calls).hasSize(2);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Unknown metadata is dropped independently without persisting its values.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testUnknownMetadata()
|
||||
{
|
||||
AiToolResult result = execute(input("summary", "Available results.", "metrics", List.of(
|
||||
input("label", "Unknown table", "value", 9876, "tableName", "missing"),
|
||||
input("label", "Unknown field", "value", 5432, "tableName", "media", "fieldName", "missing"),
|
||||
input("label", "Known", "value", 101))));
|
||||
assertThat(result.getModelContent()).contains("Dropped metrics[0]", "unknown table", "Dropped metrics[1]", "unknown field");
|
||||
assertThat(result.getRender().toString()).doesNotContain("9876", "5432");
|
||||
assertThat((List<?>) block(result.getRender(), "metrics").get("items")).hasSize(1);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Markdown punctuation, external URLs and pipes remain literal text.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testUntrustedLabelsAndLinks()
|
||||
{
|
||||
AiToolResult result = execute(input("summary", "Available results.", "metrics", List.of(input("label", "[evil](https://evil.example) | <img>", "value", 1)),
|
||||
"recordLinks", List.of(input("label", "Open", "tableName", "media", "recordId", "a/b?x#y")),
|
||||
"actions", List.of(input("label", "Ask", "prompt", "Show images"), input("label", "Open media", "navigateTo", "media"), input("label", "Bad", "prompt", "Ask", "navigateTo", "media"))));
|
||||
String markdown = (String) result.getRender().get("markdown");
|
||||
assertThat(markdown).doesNotContain("https://", "<img>", "[evil]").contains("|", "[Open](/app/media/a%2Fb%3Fx%23y)", "- Ask: Show images", "[Open media](/app/media)");
|
||||
assertThat(result.getModelContent()).contains("exactly one");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Formatting follows metadata while raw values remain numeric.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testFormatting()
|
||||
{
|
||||
qInstance.getTable("media").addField(new QFieldMetaData("price", QFieldType.DECIMAL).withDisplayFormat("$%,.2f"));
|
||||
AiToolResult result = execute(input("summary", "Amounts.", "metrics", List.of(input("label", "Amount", "value", 1234.5, "tableName", "media", "fieldName", "price"), input("label", "Count", "value", 12000)),
|
||||
"table", input("columns", List.of(input("label", "Amount", "tableName", "media", "fieldName", "price")), "rows", List.of(List.of(1234.5)))));
|
||||
assertThat(block(result.getRender(), "metrics").toString()).contains("$1,234.50", "12,000", "1234.5");
|
||||
assertThat(block(result.getRender(), "table").toString()).contains("$1,234.50");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Percentages cannot refer to labels, derived percentages or bad indexes.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testInvalidPercentColumn()
|
||||
{
|
||||
for(Object source : List.of(0, 3, 1.5))
|
||||
{
|
||||
AiToolResult result = execute(input("summary", "Invalid share.", "table", input("columns", List.of(input("label", "Name"), input("label", "Share", "percentOfColumn", source)), "rows", List.of(List.of("JPEG", 0)))));
|
||||
assertThat(result.getModelContent()).contains("Dropped table");
|
||||
assertThat((List<?>) result.getRender().get("blocks")).isEmpty();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Zero totals are defined, and charts never exceed twelve points.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testChartLimitAndZeroTotal()
|
||||
{
|
||||
List<List<Object>> rows = new ArrayList<>();
|
||||
for(Integer i = 0; i < 13; i++)
|
||||
{
|
||||
rows.add(List.of("Item " + i, 0, 0));
|
||||
}
|
||||
AiToolResult result = execute(input("summary", "Thirteen items.", "table", input("columns", List.of(input("label", "Item"), input("label", "Count"), input("label", "Share", "percentOfColumn", 1)), "rows", rows),
|
||||
"chart", input("chartType", "bar", "labelColumn", 0, "valueColumn", 1)));
|
||||
assertThat((List<?>) block(result.getRender(), "chart").get("points")).hasSize(12);
|
||||
assertThat((List<?>) block(result.getRender(), "table").get("rows")).hasSize(13);
|
||||
assertThat(result.getModelContent()).contains("truncated to 12");
|
||||
assertThat(block(result.getRender(), "table").toString()).contains("0.0%");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Operator validation does not coerce raw numeric/boolean filter values.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testCriteriaValidation()
|
||||
{
|
||||
AiToolResult result = execute(input("summary", "Links.", "filterLinks", List.of(
|
||||
input("label", "Invalid", "tableName", "media", "criteria", List.of(input("fieldName", "width", "operator", "INVENTED", "values", List.of(10)))),
|
||||
input("label", "Valid", "tableName", "media", "criteria", List.of(input("fieldName", "width", "operator", "EQUALS", "values", List.of(512)))))));
|
||||
assertThat(result.getModelContent()).contains("invalid criteria operator");
|
||||
List<?> links = (List<?>) block(result.getRender(), "filterLinks").get("items");
|
||||
String decoded = java.net.URLDecoder.decode((String) object(links.get(0)).get("href"), java.nio.charset.StandardCharsets.UTF_8);
|
||||
assertThat(decoded).contains("\"values\":[512]");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Invalid summaries produce an error tool result; no presentation is emitted.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void testInvalidSummaryAndAlwaysOffered() throws Exception
|
||||
{
|
||||
assertThat(runner.runtime().getContextBuilder().tools(new PolicyService.ToolPolicy(true, java.util.Set.of("navigate"), 20, true)))
|
||||
.anyMatch(tool -> PresentResultsTool.NAME.equals(tool.getName()) && !Boolean.TRUE.equals(tool.getStrict()));
|
||||
assertThat(execute(input("summary", "x".repeat(301))).getOutcome()).isEqualTo(AiToolOutcome.INVALID_INPUT);
|
||||
provider.thenToolUse(PresentResultsTool.NAME, Map.of()).thenText("Unable to present.");
|
||||
TurnRunner.Outcome outcome = runner.run(AssistantTestFixture.EDITOR, "Present");
|
||||
assertThat(outcome.events(TurnEvent.PRESENTATION)).isEmpty();
|
||||
assertThat(outcome.toolCalls().get(0).getOutcome()).isEqualTo("INVALID_INPUT");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Use the fixture's real UI gateway for unit-level presentation checks.
|
||||
*******************************************************************************/
|
||||
private AiToolResult execute(Map<String, Object> input)
|
||||
{
|
||||
return (new PresentResultsTool(qInstance).execute(new UiApiToolbox.Context(AssistantTestFixture.EDITOR, "UTC", runner.runtime().getUiApiGateway(), PolicyService.ToolPolicy.defaults(), null), "present", input));
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user