From f4b279b95d92ad7fab8619606f8dca0d665b3a4c Mon Sep 17 00:00:00 2001 From: James Maes Date: Tue, 17 Nov 2026 13:26:08 -0600 Subject: [PATCH] fix(tools): a tool call does not hold a monitor while it waits for another thread A tool call is a virtual thread, and the first call of a turn works out what the person is offered by waiting for another one. ToolCallService did that inside synchronized(admission), which keeps the carrier thread: with two processors a batch of tool calls, and with one processor every first tool call of a turn, waited out toolCallTimeoutSeconds, and the person was then offered no write and no process tool for that turn. An admission is now guarded by a ReentrantLock. FewProcessorsTest runs in JVMs with one and with two processors (two surefire executions) and fails there without the fix. Co-Authored-By: Claude Fable 5.2 --- CHANGELOG.md | 6 + CLAUDE.md | 1 + qbit-assistant/pom.xml | 44 +++++++ .../assistant/engine/ToolCallService.java | 48 ++++++- .../assistant/engine/FewProcessorsTest.java | 117 ++++++++++++++++++ 5 files changed, 211 insertions(+), 5 deletions(-) create mode 100644 qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/FewProcessorsTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 302ed00..2383b1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## Unreleased - Tool calls no longer wait out their time limit on a server with one or two processors + +- **A tool call does not hold a monitor while it waits for another thread.** A tool call is a virtual thread. The first call of a turn works out what the person is offered, which asks the UI API on another virtual thread and waits for it; `ToolCallService` did that inside `synchronized(admission)`, and the other calls of the batch waited to enter the same monitor. A virtual thread that waits while it holds a monitor, or waits to enter one, keeps its carrier thread, and there are only as many carriers as processors: with two processors a batch of tool calls, and with one processor every first tool call of a turn, waited 60 seconds (`toolCallTimeoutSeconds`), after which the person was offered no write and no process tool for that turn. An admission is now guarded by a `ReentrantLock`; a virtual thread parked on it gives its carrier up. What the lock guards did not change. +- **The build runs it with few processors.** `FewProcessorsTest` (one tool call; a batch of six with a write and a process tool among them) runs three times: with the machine's processors, and in JVMs with one and with two (`-XX:ActiveProcessorCount`, the surefire executions `one-processor` and `two-processors`). Without the fix it fails in the one-processor run after a minute per test. +- Nothing for a host to change. + ## Unreleased - The assistant runs a process the signed-in person may run; four review findings closed Builds package A4 of the actions design (`docs/actions-and-mcp.md`, section 4), with four findings of an independent review, each described by the guarantee it gives. **One behaviour change needs a host's attention before deploying: a table's own bulk edit and bulk delete become runnable through the assistant, and any other process once the host lists it (see `UPGRADE.md`).** diff --git a/CLAUDE.md b/CLAUDE.md index 550da0c..f9eeb0b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -70,6 +70,7 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def - `ToolCallService` trusts no caller and takes nothing it can find out itself. A caller hands it a `tools.registry.ToolCall` (a `SignedInPerson`, the call's id, name and input) and nothing else: the service reads the policy fresh on every call (`PolicyService.current(true)`, narrowed for a scheduled task's conversation; never the cache), builds the tool `Context` with the gateway the runtime handed it and keeps the proposed writes itself. - The boundary holds at compile time for code outside the assistant's packages, not only by ArchUnit. Keep it so: `ToolCallRequest`, `ConfirmedToolCall` and `UiApiToolbox.Context` are final classes with no public constructor, made only through `of(ToolCallService.Permit, ...)`; the permit has a private constructor and the service never hands its own out. `AssistantRuntime` has no public getter for the tool registry, the UI API gateway, a toolbox or its tool families: it hands them to the collaborators it builds. Never add a public way to any of them, and never a public method on the service that runs a tool without writing its row. - Its public entry points: `executeAndLog(call, conversationId, hipaaRoute)` for a caller with no turn (the service makes the scope); `confirm(person, pendingActionId)` and `cancel(person, pendingActionId)`, called only by `AssistantApi` (ArchUnit); `offers(person, toolName)` and `offersAny(person, toolNames)`; `readContext(person, timeout)` (only `AssistantApi` and `RagService`, ArchUnit); `expiredIfDue(userId, action)`; `failAbandonedActions(now)` for the retention process. `execute(call, scope)`, `log(call, scope, outcome)` and `withdraw(call, scope)` are package-private, for the turn engine only. + - An admission of `ToolCallService` is guarded by its `ReentrantLock`, never by `synchronized`: a tool call is a virtual thread, and what a person is offered is worked out by waiting for another one (`UiApiToolFamily.writableTables`, `ProcessToolFamily.runnableProcesses`). Never hold a monitor across a wait for a task of the runtime's executor: with one or two processors the task then never runs (`FewProcessorsTest`, run by the build with `-XX:ActiveProcessorCount=1` and `=2`). - On every call, before the tool is reached, it checks: a signed-in person, assistant access, the kill switch, that the scope's conversation and turn are that person's, the policy (unreadable refuses), the hard budget (that a spending limit is in force at all, then the limits; read on every call, never remembered per scope; one aggregate query per budget through `AssistantStore.sumUsage`, never the usage rows) and that the tool is offered to this person now. What a caller still supplies (rate limiting, per-request call limits and timeouts, classification, the HIPAA route) is listed in the service's javadoc. - `SignedInPerson` binds a session to the one credential forwarded to the UI API. It has no public constructor: `ofRequest(Context)` (only `AssistantSpecs`, ArchUnit; it forwards only the credential the session was resolved from) or `ofSession(qInstance, session)` (the session's own id is the credential: the authentication module resolves it, it must be the user the session names, and the person IS the session the module resolved, never the object passed in: only its time zone is carried over). Never make a person from a session that was not checked, never read access, the owner key or a permission from a session object a caller handed over, never pass a session and credentials separately, and never let the turn engine make a person. - The toolboxes cannot be called around the service: their entry points that run a tool are package-private, the runtime has no getter for a toolbox (each family is handed its own), and no class outside `tools/` calls into it except through the registry (ArchUnit, with a short allow-list: the runtime's wiring, the value types, and `UiApiToolbox.read`, the four read-only tools the starter prompts and the knowledge index use). diff --git a/qbit-assistant/pom.xml b/qbit-assistant/pom.xml index 63f1557..2e4d947 100644 --- a/qbit-assistant/pom.xml +++ b/qbit-assistant/pom.xml @@ -31,6 +31,12 @@ In-app AI assistant QBit for QQQ: conversations, turn engine, SSE routes, budgets and provider registry https://github.com/QRun-IO/qbit-assistant + + + + + + com.kingsrook.qbits @@ -152,6 +158,44 @@ ${test.excludedGroups} true + + + + + + + one-processor + + test + + + @{argLine} -XX:ActiveProcessorCount=1 + + **/FewProcessorsTest.java + + one-processor + + 1 + + + + + two-processors + + test + + + @{argLine} -XX:ActiveProcessorCount=2 + + **/FewProcessorsTest.java + + two-processors + + 2 + + + + diff --git a/qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ToolCallService.java b/qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ToolCallService.java index 162e969..d2aa120 100644 --- a/qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ToolCallService.java +++ b/qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ToolCallService.java @@ -27,6 +27,7 @@ import java.util.Map; import java.util.Objects; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.locks.ReentrantLock; import java.util.function.Supplier; import com.fasterxml.jackson.core.type.TypeReference; import com.kingsrook.qbits.assistant.AssistantQBitConfig; @@ -308,6 +309,18 @@ public class ToolCallService *******************************************************************************/ private static final class Admission { + /////////////////////////////////////////////////////////////////////// + // what guards an admission is a lock of java.util.concurrent, never // + // a monitor. A call is a virtual thread, and working out what a // + // person is offered waits for another one (the families ask the UI // + // API on the runtime's executor). A virtual thread that waits while // + // it holds a monitor, or waits to enter one, keeps its carrier // + // thread; with as few carriers as processors, the thread waited for // + // then never runs, and every call of a batch waits out its time // + // limit. Parked on this lock a virtual thread gives its carrier up // + /////////////////////////////////////////////////////////////////////// + private final ReentrantLock lock = new ReentrantLock(); + private QSession session; private UiApiCredentials credentials; private Integer conversationId; @@ -513,7 +526,8 @@ public class ToolCallService Admission admission = admission(call.person(), scope); ToolCallRequest request; - synchronized(admission) + admission.lock.lock(); + try { AiToolResult refused = refusal(call, admission, effective); if(refused != null) @@ -522,6 +536,10 @@ public class ToolCallService } request = ToolCallRequest.of(PERMIT, session, context(call.person(), admission, call.toolCallId()), call.toolCallId(), call.toolName(), call.input(), scope); } + finally + { + admission.lock.unlock(); + } return (registry.find(call.toolName()).execute(request)); } } @@ -656,7 +674,8 @@ public class ToolCallService *******************************************************************************/ private Integer propose(Admission admission, String userId, String toolCallId, UiApiToolbox.ProposedWrite write) throws QException { - synchronized(admission) + admission.lock.lock(); + try { if(admission.cutShort.contains(toolCallId)) { @@ -683,6 +702,10 @@ public class ToolCallService admission.proposed.put(toolCallId, action.getId()); return (action.getId()); } + finally + { + admission.lock.unlock(); + } } @@ -705,7 +728,8 @@ public class ToolCallService { AssistantSecurity.ownerKey(session, runtime.getConfig()); Admission admission = admission(call.person(), scope); - synchronized(admission) + admission.lock.lock(); + try { admission.cutShort.add(call.toolCallId()); Integer actionId = admission.proposed.get(call.toolCallId()); @@ -714,6 +738,10 @@ public class ToolCallService store.withdrawPendingAction(call.person().userId(), actionId, WITHDRAWN_SUMMARY, runtime.getClock().instant()); } } + finally + { + admission.lock.unlock(); + } } catch(QException | RuntimeException e) { @@ -893,7 +921,8 @@ public class ToolCallService { admission = new Admission(); } - synchronized(admission) + admission.lock.lock(); + try { if(!admission.bound) { @@ -902,6 +931,10 @@ public class ToolCallService boolean same = admission.session == person.session() && Objects.equals(admission.conversationId, scope.conversationId()) && Objects.equals(admission.turnId, scope.turnId()); return (same ? admission : bind(new Admission(), person, scope)); } + finally + { + admission.lock.unlock(); + } } @@ -1432,11 +1465,16 @@ public class ToolCallService Admission admission = admission(call.person(), scope); boolean owned; String channel; - synchronized(admission) + admission.lock.lock(); + try { owned = admission.owned; channel = admission.channel; } + finally + { + admission.lock.unlock(); + } String inputJson = inputRedactor.redact(JsonUtils.toJson(inputForLog(tool, call, result)), new HashSet<>()); store.insert(new AssistantToolCall() .withTurnId(owned ? scope.turnId() : null) diff --git a/qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/FewProcessorsTest.java b/qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/FewProcessorsTest.java new file mode 100644 index 0000000..ed7326f --- /dev/null +++ b/qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/engine/FewProcessorsTest.java @@ -0,0 +1,117 @@ +/* + * Copyright 2026 Kingsrook, LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.assistant.engine; + + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.assistant.model.AssistantToolCall; +import com.kingsrook.qbits.assistant.testing.BaseAssistantTest; +import com.kingsrook.qbits.assistant.testing.TurnRunner; +import org.junit.jupiter.api.Test; +import static com.kingsrook.qbits.assistant.testing.AssistantTestFixture.AUTHOR; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** Tool calls on a server with one or two processors. + ** + ** A tool call is a virtual thread. The first call of a turn works out what + ** the person is offered, which asks the UI API on another virtual thread and + ** waits for it. When that wait is made while a monitor is held (and the + ** other calls of the batch wait to enter the same monitor), every waiting + ** thread keeps its carrier, there are only as many carriers as processors, + ** and the thread waited for never runs: the calls wait out their time limit + ** (60 seconds), and the person is then offered no write and no process tool + ** for that turn. + ** + ** The build runs this class three times: with the processors of the machine, + ** and again in JVMs that have one and two (the surefire executions + ** one-processor and two-processors, -XX:ActiveProcessorCount). With few + ** processors it fails, by taking a minute and by the tools it then refuses, + ** as soon as a monitor is held across such a wait again. + *******************************************************************************/ +class FewProcessorsTest extends BaseAssistantTest +{ + private static final long LIMIT_MILLIS = 30_000; + + + + /******************************************************************************* + ** The executions that say they have few processors have them. + *******************************************************************************/ + @Test + void theJvmHasTheProcessorsItsExecutionNames() + { + Integer few = Integer.getInteger("assistant.fewProcessors"); + if(few != null) + { + assertThat(Runtime.getRuntime().availableProcessors()).isEqualTo(few); + } + } + + + + /******************************************************************************* + ** One tool call, the first of its turn: it is answered at once, and what + ** the person is offered was worked out (a write tool is offered to a + ** person who may change something, so the proposal is made). + *******************************************************************************/ + @Test + void oneToolCallIsAnsweredAtOnce() throws Exception + { + provider.thenToolUse("update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title"))).thenText("Please confirm."); + long started = System.currentTimeMillis(); + TurnRunner.Outcome outcome = runner.run(AUTHOR, "Rename the first post"); + assertThat(outcome.end()).containsEntry("status", "COMPLETE"); + assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS); + assertThat(outcome.toolCalls()).singleElement().satisfies(call -> + { + assertThat(call.getOutcome()).as("the write tool was offered: " + call.getResultSummary()).isEqualTo("OK"); + assertThat(call.getPendingActionId()).isNotNull(); + }); + } + + + + /******************************************************************************* + ** A batch of tool calls in one reply, more of them than there are + ** processors: all are answered at once, in order, and the write and the + ** process tool among them were offered. + *******************************************************************************/ + @Test + void aBatchOfToolCallsIsAnsweredAtOnce() throws Exception + { + Map all = Map.of("booleanOperator", "AND", "criteria", List.of()); + List calls = new ArrayList<>(); + calls.add(new Object[] { "count_records", Map.of("table", "blog_post", "filter", all) }); + calls.add(new Object[] { "count_records", Map.of("table", "page", "filter", all) }); + calls.add(new Object[] { "list_tables", Map.of() }); + calls.add(new Object[] { "list_processes", Map.of() }); + calls.add(new Object[] { "update_record", Map.of("table", "blog_post", "id", "1", "values", Map.of("title", "A new title")) }); + calls.add(new Object[] { "count_records", Map.of("table", "media", "filter", all) }); + provider.thenToolUses("Checking.", calls).thenText("Done."); + + long started = System.currentTimeMillis(); + TurnRunner.Outcome outcome = runner.run(AUTHOR, "Counts, and rename the first post"); + assertThat(outcome.end()).containsEntry("status", "COMPLETE"); + assertThat(System.currentTimeMillis() - started).as("no call waited out its time limit").isLessThan(LIMIT_MILLIS); + assertThat(outcome.toolCalls()).extracting(AssistantToolCall::getToolName).containsExactly("count_records", "count_records", "list_tables", "list_processes", "update_record", "count_records"); + assertThat(outcome.toolCalls()).as("every call ran, and the write and the process tool were offered").allSatisfy(call -> assertThat(call.getOutcome()).as(call.getToolName()).isEqualTo("OK")); + } +} -- 2.54.0