fix(tools): a tool call does not hold a monitor while it waits for another thread (one and two processors) #115
No files matched your search
@@ -1,5 +1,11 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased - Tool calls no longer wait out their time limit on a server with one or two processors
|
||||
|
||||
- **A tool call does not hold a monitor while it waits for another thread.** A tool call is a virtual thread. The first call of a turn works out what the person is offered, which asks the UI API on another virtual thread and waits for it; `ToolCallService` did that inside `synchronized(admission)`, and the other calls of the batch waited to enter the same monitor. A virtual thread that waits while it holds a monitor, or waits to enter one, keeps its carrier thread, and there are only as many carriers as processors: with two processors a batch of tool calls, and with one processor every first tool call of a turn, waited 60 seconds (`toolCallTimeoutSeconds`), after which the person was offered no write and no process tool for that turn. An admission is now guarded by a `ReentrantLock`; a virtual thread parked on it gives its carrier up. What the lock guards did not change.
|
||||
- **The build runs it with few processors.** `FewProcessorsTest` (one tool call; a batch of six with a write and a process tool among them) runs three times: with the machine's processors, and in JVMs with one and with two (`-XX:ActiveProcessorCount`, the surefire executions `one-processor` and `two-processors`). Without the fix it fails in the one-processor run after a minute per test.
|
||||
- Nothing for a host to change.
|
||||
|
||||
## Unreleased - The assistant runs a process the signed-in person may run; four review findings closed
|
||||
|
||||
Builds package A4 of the actions design (`docs/actions-and-mcp.md`, section 4), with four findings of an independent review, each described by the guarantee it gives. **One behaviour change needs a host's attention before deploying: a table's own bulk edit and bulk delete become runnable through the assistant, and any other process once the host lists it (see `UPGRADE.md`).**
|
||||
|
||||
@@ -70,6 +70,7 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def
|
||||
- `ToolCallService` trusts no caller and takes nothing it can find out itself. A caller hands it a `tools.registry.ToolCall` (a `SignedInPerson`, the call's id, name and input) and nothing else: the service reads the policy fresh on every call (`PolicyService.current(true)`, narrowed for a scheduled task's conversation; never the cache), builds the tool `Context` with the gateway the runtime handed it and keeps the proposed writes itself.
|
||||
- The boundary holds at compile time for code outside the assistant's packages, not only by ArchUnit. Keep it so: `ToolCallRequest`, `ConfirmedToolCall` and `UiApiToolbox.Context` are final classes with no public constructor, made only through `of(ToolCallService.Permit, ...)`; the permit has a private constructor and the service never hands its own out. `AssistantRuntime` has no public getter for the tool registry, the UI API gateway, a toolbox or its tool families: it hands them to the collaborators it builds. Never add a public way to any of them, and never a public method on the service that runs a tool without writing its row.
|
||||
- Its public entry points: `executeAndLog(call, conversationId, hipaaRoute)` for a caller with no turn (the service makes the scope); `confirm(person, pendingActionId)` and `cancel(person, pendingActionId)`, called only by `AssistantApi` (ArchUnit); `offers(person, toolName)` and `offersAny(person, toolNames)`; `readContext(person, timeout)` (only `AssistantApi` and `RagService`, ArchUnit); `expiredIfDue(userId, action)`; `failAbandonedActions(now)` for the retention process. `execute(call, scope)`, `log(call, scope, outcome)` and `withdraw(call, scope)` are package-private, for the turn engine only.
|
||||
- An admission of `ToolCallService` is guarded by its `ReentrantLock`, never by `synchronized`: a tool call is a virtual thread, and what a person is offered is worked out by waiting for another one (`UiApiToolFamily.writableTables`, `ProcessToolFamily.runnableProcesses`). Never hold a monitor across a wait for a task of the runtime's executor: with one or two processors the task then never runs (`FewProcessorsTest`, run by the build with `-XX:ActiveProcessorCount=1` and `=2`).
|
||||
- On every call, before the tool is reached, it checks: a signed-in person, assistant access, the kill switch, that the scope's conversation and turn are that person's, the policy (unreadable refuses), the hard budget (that a spending limit is in force at all, then the limits; read on every call, never remembered per scope; one aggregate query per budget through `AssistantStore.sumUsage`, never the usage rows) and that the tool is offered to this person now. What a caller still supplies (rate limiting, per-request call limits and timeouts, classification, the HIPAA route) is listed in the service's javadoc.
|
||||
- `SignedInPerson` binds a session to the one credential forwarded to the UI API. It has no public constructor: `ofRequest(Context)` (only `AssistantSpecs`, ArchUnit; it forwards only the credential the session was resolved from) or `ofSession(qInstance, session)` (the session's own id is the credential: the authentication module resolves it, it must be the user the session names, and the person IS the session the module resolved, never the object passed in: only its time zone is carried over). Never make a person from a session that was not checked, never read access, the owner key or a permission from a session object a caller handed over, never pass a session and credentials separately, and never let the turn engine make a person.
|
||||
- The toolboxes cannot be called around the service: their entry points that run a tool are package-private, the runtime has no getter for a toolbox (each family is handed its own), and no class outside `tools/` calls into it except through the registry (ArchUnit, with a short allow-list: the runtime's wiring, the value types, and `UiApiToolbox.read`, the four read-only tools the starter prompts and the knowledge index use).
|
||||
|
||||
@@ -31,6 +31,12 @@
|
||||
<description>In-app AI assistant QBit for QQQ: conversations, turn engine, SSE routes, budgets and provider registry</description>
|
||||
<url>https://github.com/QRun-IO/qbit-assistant</url>
|
||||
|
||||
<properties>
|
||||
<!-- what JaCoCo's agent adds to; empty when coverage is skipped, so that the -->
|
||||
<!-- executions below that add a JVM option of their own still have a value -->
|
||||
<argLine></argLine>
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
@@ -152,6 +158,44 @@
|
||||
<excludedGroups>${test.excludedGroups}</excludedGroups>
|
||||
<redirectTestOutputToFile>true</redirectTestOutputToFile>
|
||||
</configuration>
|
||||
<executions>
|
||||
<!-- a server with one or two processors: tool calls are virtual threads, and a -->
|
||||
<!-- monitor held across a wait for another one leaves no carrier thread for it. -->
|
||||
<!-- FewProcessorsTest runs again in a JVM that has that few (it fails there, by -->
|
||||
<!-- waiting out a time limit, if such a wait comes back). -->
|
||||
<execution>
|
||||
<id>one-processor</id>
|
||||
<goals>
|
||||
<goal>test</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<argLine>@{argLine} -XX:ActiveProcessorCount=1</argLine>
|
||||
<includes>
|
||||
<include>**/FewProcessorsTest.java</include>
|
||||
</includes>
|
||||
<reportNameSuffix>one-processor</reportNameSuffix>
|
||||
<systemPropertyVariables>
|
||||
<assistant.fewProcessors>1</assistant.fewProcessors>
|
||||
</systemPropertyVariables>
|
||||
</configuration>
|
||||
</execution>
|
||||
<execution>
|
||||
<id>two-processors</id>
|
||||
<goals>
|
||||
<goal>test</goal>
|
||||
</goals>
|
||||
<configuration>
|
||||
<argLine>@{argLine} -XX:ActiveProcessorCount=2</argLine>
|
||||
<includes>
|
||||
<include>**/FewProcessorsTest.java</include>
|
||||
</includes>
|
||||
<reportNameSuffix>two-processors</reportNameSuffix>
|
||||
<systemPropertyVariables>
|
||||
<assistant.fewProcessors>2</assistant.fewProcessors>
|
||||
</systemPropertyVariables>
|
||||
</configuration>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<!-- the eval fixture and harness are reused by provider-module live evals: -->
|
||||
|
||||
+43
-5
@@ -27,6 +27,7 @@ import java.util.Map;
|
||||
import java.util.Objects;
|
||||
import java.util.Set;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.locks.ReentrantLock;
|
||||
import java.util.function.Supplier;
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import com.kingsrook.qbits.assistant.AssistantQBitConfig;
|
||||
@@ -308,6 +309,18 @@ public class ToolCallService
|
||||
*******************************************************************************/
|
||||
private static final class Admission
|
||||
{
|
||||
///////////////////////////////////////////////////////////////////////
|
||||
// what guards an admission is a lock of java.util.concurrent, never //
|
||||
// a monitor. A call is a virtual thread, and working out what a //
|
||||
// person is offered waits for another one (the families ask the UI //
|
||||
// API on the runtime's executor). A virtual thread that waits while //
|
||||
// it holds a monitor, or waits to enter one, keeps its carrier //
|
||||
// thread; with as few carriers as processors, the thread waited for //
|
||||
// then never runs, and every call of a batch waits out its time //
|
||||
// limit. Parked on this lock a virtual thread gives its carrier up //
|
||||
///////////////////////////////////////////////////////////////////////
|
||||
private final ReentrantLock lock = new ReentrantLock();
|
||||
|
||||
private QSession session;
|
||||
private UiApiCredentials credentials;
|
||||
private Integer conversationId;
|
||||
@@ -513,7 +526,8 @@ public class ToolCallService
|
||||
|
||||
Admission admission = admission(call.person(), scope);
|
||||
ToolCallRequest request;
|
||||
synchronized(admission)
|
||||
admission.lock.lock();
|
||||
try
|
||||
{
|
||||
AiToolResult refused = refusal(call, admission, effective);
|
||||
if(refused != null)
|
||||
@@ -522,6 +536,10 @@ public class ToolCallService
|
||||
}
|
||||
request = ToolCallRequest.of(PERMIT, session, context(call.person(), admission, call.toolCallId()), call.toolCallId(), call.toolName(), call.input(), scope);
|
||||
}
|
||||
finally
|
||||
{
|
||||
admission.lock.unlock();
|
||||
}
|
||||
return (registry.find(call.toolName()).execute(request));
|
||||
}
|
||||
}
|
||||
@@ -656,7 +674,8 @@ public class ToolCallService
|
||||
*******************************************************************************/
|
||||
private Integer propose(Admission admission, String userId, String toolCallId, UiApiToolbox.ProposedWrite write) throws QException
|
||||
{
|
||||
synchronized(admission)
|
||||
admission.lock.lock();
|
||||
try
|
||||
{
|
||||
if(admission.cutShort.contains(toolCallId))
|
||||
{
|
||||
@@ -683,6 +702,10 @@ public class ToolCallService
|
||||
admission.proposed.put(toolCallId, action.getId());
|
||||
return (action.getId());
|
||||
}
|
||||
finally
|
||||
{
|
||||
admission.lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -705,7 +728,8 @@ public class ToolCallService
|
||||
{
|
||||
AssistantSecurity.ownerKey(session, runtime.getConfig());
|
||||
Admission admission = admission(call.person(), scope);
|
||||
synchronized(admission)
|
||||
admission.lock.lock();
|
||||
try
|
||||
{
|
||||
admission.cutShort.add(call.toolCallId());
|
||||
Integer actionId = admission.proposed.get(call.toolCallId());
|
||||
@@ -714,6 +738,10 @@ public class ToolCallService
|
||||
store.withdrawPendingAction(call.person().userId(), actionId, WITHDRAWN_SUMMARY, runtime.getClock().instant());
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
admission.lock.unlock();
|
||||
}
|
||||
}
|
||||
catch(QException | RuntimeException e)
|
||||
{
|
||||
@@ -893,7 +921,8 @@ public class ToolCallService
|
||||
{
|
||||
admission = new Admission();
|
||||
}
|
||||
synchronized(admission)
|
||||
admission.lock.lock();
|
||||
try
|
||||
{
|
||||
if(!admission.bound)
|
||||
{
|
||||
@@ -902,6 +931,10 @@ public class ToolCallService
|
||||
boolean same = admission.session == person.session() && Objects.equals(admission.conversationId, scope.conversationId()) && Objects.equals(admission.turnId, scope.turnId());
|
||||
return (same ? admission : bind(new Admission(), person, scope));
|
||||
}
|
||||
finally
|
||||
{
|
||||
admission.lock.unlock();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1432,11 +1465,16 @@ public class ToolCallService
|
||||
Admission admission = admission(call.person(), scope);
|
||||
boolean owned;
|
||||
String channel;
|
||||
synchronized(admission)
|
||||
admission.lock.lock();
|
||||
try
|
||||
{
|
||||
owned = admission.owned;
|
||||
channel = admission.channel;
|
||||
}
|
||||
finally
|
||||
{
|
||||
admission.lock.unlock();
|
||||
}
|
||||
String inputJson = inputRedactor.redact(JsonUtils.toJson(inputForLog(tool, call, result)), new HashSet<>());
|
||||
store.insert(new AssistantToolCall()
|
||||
.withTurnId(owned ? scope.turnId() : null)
|
||||
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.assistant.engine;
|
||||
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import com.kingsrook.qbits.assistant.model.AssistantToolCall;
|
||||
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
|
||||
import com.kingsrook.qbits.assistant.testing.TurnRunner;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.AUTHOR;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Tool calls on a server with one or two processors.
|
||||
**
|
||||
** A tool call is a virtual thread. The first call of a turn works out what
|
||||
** the person is offered, which asks the UI API on another virtual thread and
|
||||
** waits for it. When that wait is made while a monitor is held (and the
|
||||
** other calls of the batch wait to enter the same monitor), every waiting
|
||||
** thread keeps its carrier, there are only as many carriers as processors,
|
||||
** and the thread waited for never runs: the calls wait out their time limit
|
||||
** (60 seconds), and the person is then offered no write and no process tool
|
||||
** for that turn.
|
||||
**
|
||||
** The build runs this class three times: with the processors of the machine,
|
||||
** and again in JVMs that have one and two (the surefire executions
|
||||
** one-processor and two-processors, -XX:ActiveProcessorCount). With few
|
||||
** processors it fails, by taking a minute and by the tools it then refuses,
|
||||
** as soon as a monitor is held across such a wait again.
|
||||
*******************************************************************************/
|
||||
class FewProcessorsTest extends BaseAssistantTest
|
||||
{
|
||||
private static final long LIMIT_MILLIS = 30_000;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The executions that say they have few processors have them.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void theJvmHasTheProcessorsItsExecutionNames()
|
||||
{
|
||||
Integer few = Integer.getInteger("assistant.fewProcessors");
|
||||
if(few != null)
|
||||
{
|
||||
assertThat(Runtime.getRuntime().availableProcessors()).isEqualTo(few);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One tool call, the first of its turn: it is answered at once, and what
|
||||
** the person is offered was worked out (a write tool is offered to a
|
||||
** person who may change something, so the proposal is made).
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void oneToolCallIsAnsweredAtOnce() throws Exception
|
||||
{
|
||||
provider.thenToolUse("update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title"))).thenText("Please confirm.");
|
||||
long started = System.currentTimeMillis();
|
||||
TurnRunner.Outcome outcome = runner.run(AUTHOR, "Rename the first post");
|
||||
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
|
||||
assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS);
|
||||
assertThat(outcome.toolCalls()).singleElement().satisfies(call ->
|
||||
{
|
||||
assertThat(call.getOutcome()).as("the write tool was offered: " + call.getResultSummary()).isEqualTo("OK");
|
||||
assertThat(call.getPendingActionId()).isNotNull();
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A batch of tool calls in one reply, more of them than there are
|
||||
** processors: all are answered at once, in order, and the write and the
|
||||
** process tool among them were offered.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void aBatchOfToolCallsIsAnsweredAtOnce() throws Exception
|
||||
{
|
||||
Map<String, Object> all = Map.of("booleanOperator", "AND", "criteria", List.of());
|
||||
List<Object[]> calls = new ArrayList<>();
|
||||
calls.add(new Object[] { "count_records", Map.of("table", "blog_post", "filter", all) });
|
||||
calls.add(new Object[] { "count_records", Map.of("table", "page", "filter", all) });
|
||||
calls.add(new Object[] { "list_tables", Map.of() });
|
||||
calls.add(new Object[] { "list_processes", Map.of() });
|
||||
calls.add(new Object[] { "update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title")) });
|
||||
calls.add(new Object[] { "count_records", Map.of("table", "media", "filter", all) });
|
||||
provider.thenToolUses("Checking.", calls).thenText("Done.");
|
||||
|
||||
long started = System.currentTimeMillis();
|
||||
TurnRunner.Outcome outcome = runner.run(AUTHOR, "Counts, and rename the first post");
|
||||
assertThat(outcome.end()).containsEntry("status", "COMPLETE");
|
||||
assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS);
|
||||
assertThat(outcome.toolCalls()).extracting(AssistantToolCall::getToolName).containsExactly("count_records", "count_records", "list_tables", "list_processes", "update_record", "count_records");
|
||||
assertThat(outcome.toolCalls()).as("every call ran, and the write and the process tool were offered").allSatisfy(call -> assertThat(call.getOutcome()).as(call.getToolName()).isEqualTo("OK"));
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user