fix(tools): a tool call does not hold a monitor while it waits for another thread (one and two processors) #115

Merged
james.maes merged 1 commits from fix/tool-call-admission-does-not-pin into develop 2026-11-17 21:43:15 +00:00
5 changed files with 211 additions and 5 deletions

No files matched your search

+6
View File
@@ -1,5 +1,11 @@
# Changelog
## Unreleased - Tool calls no longer wait out their time limit on a server with one or two processors
- **A tool call does not hold a monitor while it waits for another thread.** A tool call is a virtual thread. The first call of a turn works out what the person is offered, which asks the UI API on another virtual thread and waits for it; `ToolCallService` did that inside `synchronized(admission)`, and the other calls of the batch waited to enter the same monitor. A virtual thread that waits while it holds a monitor, or waits to enter one, keeps its carrier thread, and there are only as many carriers as processors: with two processors a batch of tool calls, and with one processor every first tool call of a turn, waited 60 seconds (`toolCallTimeoutSeconds`), after which the person was offered no write and no process tool for that turn. An admission is now guarded by a `ReentrantLock`; a virtual thread parked on it gives its carrier up. What the lock guards did not change.
- **The build runs it with few processors.** `FewProcessorsTest` (one tool call; a batch of six with a write and a process tool among them) runs three times: with the machine's processors, and in JVMs with one and with two (`-XX:ActiveProcessorCount`, the surefire executions `one-processor` and `two-processors`). Without the fix it fails in the one-processor run after a minute per test.
- Nothing for a host to change.
## Unreleased - The assistant runs a process the signed-in person may run; four review findings closed
Builds package A4 of the actions design (`docs/actions-and-mcp.md`, section 4), with four findings of an independent review, each described by the guarantee it gives. **One behaviour change needs a host's attention before deploying: a table's own bulk edit and bulk delete become runnable through the assistant, and any other process once the host lists it (see `UPGRADE.md`).**
+1
View File
@@ -70,6 +70,7 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def
- `ToolCallService` trusts no caller and takes nothing it can find out itself. A caller hands it a `tools.registry.ToolCall` (a `SignedInPerson`, the call's id, name and input) and nothing else: the service reads the policy fresh on every call (`PolicyService.current(true)`, narrowed for a scheduled task's conversation; never the cache), builds the tool `Context` with the gateway the runtime handed it and keeps the proposed writes itself.
- The boundary holds at compile time for code outside the assistant's packages, not only by ArchUnit. Keep it so: `ToolCallRequest`, `ConfirmedToolCall` and `UiApiToolbox.Context` are final classes with no public constructor, made only through `of(ToolCallService.Permit, ...)`; the permit has a private constructor and the service never hands its own out. `AssistantRuntime` has no public getter for the tool registry, the UI API gateway, a toolbox or its tool families: it hands them to the collaborators it builds. Never add a public way to any of them, and never a public method on the service that runs a tool without writing its row.
- Its public entry points: `executeAndLog(call, conversationId, hipaaRoute)` for a caller with no turn (the service makes the scope); `confirm(person, pendingActionId)` and `cancel(person, pendingActionId)`, called only by `AssistantApi` (ArchUnit); `offers(person, toolName)` and `offersAny(person, toolNames)`; `readContext(person, timeout)` (only `AssistantApi` and `RagService`, ArchUnit); `expiredIfDue(userId, action)`; `failAbandonedActions(now)` for the retention process. `execute(call, scope)`, `log(call, scope, outcome)` and `withdraw(call, scope)` are package-private, for the turn engine only.
- An admission of `ToolCallService` is guarded by its `ReentrantLock`, never by `synchronized`: a tool call is a virtual thread, and what a person is offered is worked out by waiting for another one (`UiApiToolFamily.writableTables`, `ProcessToolFamily.runnableProcesses`). Never hold a monitor across a wait for a task of the runtime's executor: with one or two processors the task then never runs (`FewProcessorsTest`, run by the build with `-XX:ActiveProcessorCount=1` and `=2`).
- On every call, before the tool is reached, it checks: a signed-in person, assistant access, the kill switch, that the scope's conversation and turn are that person's, the policy (unreadable refuses), the hard budget (that a spending limit is in force at all, then the limits; read on every call, never remembered per scope; one aggregate query per budget through `AssistantStore.sumUsage`, never the usage rows) and that the tool is offered to this person now. What a caller still supplies (rate limiting, per-request call limits and timeouts, classification, the HIPAA route) is listed in the service's javadoc.
- `SignedInPerson` binds a session to the one credential forwarded to the UI API. It has no public constructor: `ofRequest(Context)` (only `AssistantSpecs`, ArchUnit; it forwards only the credential the session was resolved from) or `ofSession(qInstance, session)` (the session's own id is the credential: the authentication module resolves it, it must be the user the session names, and the person IS the session the module resolved, never the object passed in: only its time zone is carried over). Never make a person from a session that was not checked, never read access, the owner key or a permission from a session object a caller handed over, never pass a session and credentials separately, and never let the turn engine make a person.
- The toolboxes cannot be called around the service: their entry points that run a tool are package-private, the runtime has no getter for a toolbox (each family is handed its own), and no class outside `tools/` calls into it except through the registry (ArchUnit, with a short allow-list: the runtime's wiring, the value types, and `UiApiToolbox.read`, the four read-only tools the starter prompts and the knowledge index use).
+44
View File
@@ -31,6 +31,12 @@
<description>In-app AI assistant QBit for QQQ: conversations, turn engine, SSE routes, budgets and provider registry</description>
<url>https://github.com/QRun-IO/qbit-assistant</url>
<properties>
<!-- what JaCoCo's agent adds to; empty when coverage is skipped, so that the -->
<!-- executions below that add a JVM option of their own still have a value -->
<argLine></argLine>
</properties>
<dependencies>
<dependency>
<groupId>com.kingsrook.qbits</groupId>
@@ -152,6 +158,44 @@
<excludedGroups>${test.excludedGroups}</excludedGroups>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
</configuration>
<executions>
<!-- a server with one or two processors: tool calls are virtual threads, and a -->
<!-- monitor held across a wait for another one leaves no carrier thread for it. -->
<!-- FewProcessorsTest runs again in a JVM that has that few (it fails there, by -->
<!-- waiting out a time limit, if such a wait comes back). -->
<execution>
<id>one-processor</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<argLine>@{argLine} -XX:ActiveProcessorCount=1</argLine>
<includes>
<include>**/FewProcessorsTest.java</include>
</includes>
<reportNameSuffix>one-processor</reportNameSuffix>
<systemPropertyVariables>
<assistant.fewProcessors>1</assistant.fewProcessors>
</systemPropertyVariables>
</configuration>
</execution>
<execution>
<id>two-processors</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<argLine>@{argLine} -XX:ActiveProcessorCount=2</argLine>
<includes>
<include>**/FewProcessorsTest.java</include>
</includes>
<reportNameSuffix>two-processors</reportNameSuffix>
<systemPropertyVariables>
<assistant.fewProcessors>2</assistant.fewProcessors>
</systemPropertyVariables>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<!-- the eval fixture and harness are reused by provider-module live evals: -->
@@ -27,6 +27,7 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Supplier;
import com.fasterxml.jackson.core.type.TypeReference;
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
@@ -308,6 +309,18 @@ public class ToolCallService
*******************************************************************************/
private static final class Admission
{
///////////////////////////////////////////////////////////////////////
// what guards an admission is a lock of java.util.concurrent, never //
// a monitor. A call is a virtual thread, and working out what a //
// person is offered waits for another one (the families ask the UI //
// API on the runtime's executor). A virtual thread that waits while //
// it holds a monitor, or waits to enter one, keeps its carrier //
// thread; with as few carriers as processors, the thread waited for //
// then never runs, and every call of a batch waits out its time //
// limit. Parked on this lock a virtual thread gives its carrier up //
///////////////////////////////////////////////////////////////////////
private final ReentrantLock lock = new ReentrantLock();
private QSession session;
private UiApiCredentials credentials;
private Integer conversationId;
@@ -513,7 +526,8 @@ public class ToolCallService
Admission admission = admission(call.person(), scope);
ToolCallRequest request;
synchronized(admission)
admission.lock.lock();
try
{
AiToolResult refused = refusal(call, admission, effective);
if(refused != null)
@@ -522,6 +536,10 @@ public class ToolCallService
}
request = ToolCallRequest.of(PERMIT, session, context(call.person(), admission, call.toolCallId()), call.toolCallId(), call.toolName(), call.input(), scope);
}
finally
{
admission.lock.unlock();
}
return (registry.find(call.toolName()).execute(request));
}
}
@@ -656,7 +674,8 @@ public class ToolCallService
*******************************************************************************/
private Integer propose(Admission admission, String userId, String toolCallId, UiApiToolbox.ProposedWrite write) throws QException
{
synchronized(admission)
admission.lock.lock();
try
{
if(admission.cutShort.contains(toolCallId))
{
@@ -683,6 +702,10 @@ public class ToolCallService
admission.proposed.put(toolCallId, action.getId());
return (action.getId());
}
finally
{
admission.lock.unlock();
}
}
@@ -705,7 +728,8 @@ public class ToolCallService
{
AssistantSecurity.ownerKey(session, runtime.getConfig());
Admission admission = admission(call.person(), scope);
synchronized(admission)
admission.lock.lock();
try
{
admission.cutShort.add(call.toolCallId());
Integer actionId = admission.proposed.get(call.toolCallId());
@@ -714,6 +738,10 @@ public class ToolCallService
store.withdrawPendingAction(call.person().userId(), actionId, WITHDRAWN_SUMMARY, runtime.getClock().instant());
}
}
finally
{
admission.lock.unlock();
}
}
catch(QException | RuntimeException e)
{
@@ -893,7 +921,8 @@ public class ToolCallService
{
admission = new Admission();
}
synchronized(admission)
admission.lock.lock();
try
{
if(!admission.bound)
{
@@ -902,6 +931,10 @@ public class ToolCallService
boolean same = admission.session == person.session() && Objects.equals(admission.conversationId, scope.conversationId()) && Objects.equals(admission.turnId, scope.turnId());
return (same ? admission : bind(new Admission(), person, scope));
}
finally
{
admission.lock.unlock();
}
}
@@ -1432,11 +1465,16 @@ public class ToolCallService
Admission admission = admission(call.person(), scope);
boolean owned;
String channel;
synchronized(admission)
admission.lock.lock();
try
{
owned = admission.owned;
channel = admission.channel;
}
finally
{
admission.lock.unlock();
}
String inputJson = inputRedactor.redact(JsonUtils.toJson(inputForLog(tool, call, result)), new HashSet<>());
store.insert(new AssistantToolCall()
.withTurnId(owned ? scope.turnId() : null)
@@ -0,0 +1,117 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.engine;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qbits.assistant.testing.TurnRunner;
import org.junit.jupiter.api.Test;
import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.AUTHOR;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Tool calls on a server with one or two processors.
**
** A tool call is a virtual thread. The first call of a turn works out what
** the person is offered, which asks the UI API on another virtual thread and
** waits for it. When that wait is made while a monitor is held (and the
** other calls of the batch wait to enter the same monitor), every waiting
** thread keeps its carrier, there are only as many carriers as processors,
** and the thread waited for never runs: the calls wait out their time limit
** (60 seconds), and the person is then offered no write and no process tool
** for that turn.
**
** The build runs this class three times: with the processors of the machine,
** and again in JVMs that have one and two (the surefire executions
** one-processor and two-processors, -XX:ActiveProcessorCount). With few
** processors it fails, by taking a minute and by the tools it then refuses,
** as soon as a monitor is held across such a wait again.
*******************************************************************************/
class FewProcessorsTest extends BaseAssistantTest
{
private static final long LIMIT_MILLIS = 30_000;
/*******************************************************************************
** The executions that say they have few processors have them.
*******************************************************************************/
@Test
void theJvmHasTheProcessorsItsExecutionNames()
{
Integer few = Integer.getInteger("assistant.fewProcessors");
if(few != null)
{
assertThat(Runtime.getRuntime().availableProcessors()).isEqualTo(few);
}
}
/*******************************************************************************
** One tool call, the first of its turn: it is answered at once, and what
** the person is offered was worked out (a write tool is offered to a
** person who may change something, so the proposal is made).
*******************************************************************************/
@Test
void oneToolCallIsAnsweredAtOnce() throws Exception
{
provider.thenToolUse("update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title"))).thenText("Please confirm.");
long started = System.currentTimeMillis();
TurnRunner.Outcome outcome = runner.run(AUTHOR, "Rename the first post");
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS);
assertThat(outcome.toolCalls()).singleElement().satisfies(call ->
{
assertThat(call.getOutcome()).as("the write tool was offered: " + call.getResultSummary()).isEqualTo("OK");
assertThat(call.getPendingActionId()).isNotNull();
});
}
/*******************************************************************************
** A batch of tool calls in one reply, more of them than there are
** processors: all are answered at once, in order, and the write and the
** process tool among them were offered.
*******************************************************************************/
@Test
void aBatchOfToolCallsIsAnsweredAtOnce() throws Exception
{
Map<String, Object> all = Map.of("booleanOperator", "AND", "criteria", List.of());
List<Object[]> calls = new ArrayList<>();
calls.add(new Object[] { "count_records", Map.of("table", "blog_post", "filter", all) });
calls.add(new Object[] { "count_records", Map.of("table", "page", "filter", all) });
calls.add(new Object[] { "list_tables", Map.of() });
calls.add(new Object[] { "list_processes", Map.of() });
calls.add(new Object[] { "update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title")) });
calls.add(new Object[] { "count_records", Map.of("table", "media", "filter", all) });
provider.thenToolUses("Checking.", calls).thenText("Done.");
long started = System.currentTimeMillis();
TurnRunner.Outcome outcome = runner.run(AUTHOR, "Counts, and rename the first post");
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS);
assertThat(outcome.toolCalls()).extracting(AssistantToolCall::getToolName).containsExactly("count_records", "count_records", "list_tables", "list_processes", "update_record", "count_records");
assertThat(outcome.toolCalls()).as("every call ran, and the write and the process tool were offered").allSatisfy(call -> assertThat(call.getOutcome()).as(call.getToolName()).isEqualTo("OK"));
}
}