chore(bulk-load): tag the jobs-launch stand-in with the kernel PR (QRun-IO/qqq#1178)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## Unreleased - P-f
+
+- Add OpenAI-compatible embeddings (including Azure), explicit CHAT/EMBEDDING
+  model kinds, editable discovery hints, and budgeted EMBED usage.
+- Add admin Knowledge sources with streamed Review/Result Index now and Clear
+  index actions. Indexing uses the clicking admin's UI API session, chunks text,
+  skips unchanged hashes and reconciles disappeared records.
+- Add content-free in-memory and optional JDBC pgvector stores, per-instance
+  schemas, HNSW cosine search and transactional record replacement.
+- Add permission-filtered `search_knowledge`, fresh UI API reads, existing
+  masking/redaction, PHI embedding guards, classification propagation and citations.
+- Add embedding HTTP contracts, real UI API RAG security/golden tests and
+  always-on JDBC contracts; the pgvector container test skips without Docker.
 ## Unreleased - Answer presentation
 
 - Add `present_results`: server-formatted metrics, tables, computed percentages,

CLAUDE.md

@@ -46,6 +46,8 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def
 - `qbit-assistant-provider-bedrock`: Claude through the Anthropic Bedrock backend, AWS default credentials and SigV4; IRSA/web-identity hosts supply STS.
 - `qbit-assistant-provider-vertex`: Claude through the Anthropic Vertex backend and Gemini through OpenAI-compatible Chat Completions, with refreshed Google ADC tokens.
 
+- `qbit-assistant-rag-pgvector`: optional host-supplied JDBC vector store, isolated by instance schema; no QQQ RDBMS internals.
+
 ## Invariants (do not break)
 
 - **Sessions:**
@@ -53,6 +55,7 @@ Always use the wrapper: Maven 3.10 cannot read the parent POM. The committed def
   - Every route uses `ExecutorSessionUtils.setupSession` (the user's own session).
   - `AssistantSecurity` only ever adds the user's OWN id key, or the all-access key for `assistant.admin` holders.
 - **Data access:**
+  - The vector index stores no record content: only source/table/key/field/chunk, content hash and embedding. Retrieval re-reads every hit through the UI API as the asking user, then applies the existing get_record exposure/masking path.
   - Only `persistence/` calls QueryAction, InsertAction and the other data actions (ArchUnit).
   - Data tools run only through `UiApiToolbox` → `UiApiGateway` (spec'd `/qqq/v1` operations with the user's `sessionUUID`); never core data actions for user data. Only `navigate` and `search_help` use `AiToolKernel.execute`.
   - Data access (`DataAccessService`) and the policy only ever narrow what the UI API allows; a table whose AI exposure is NONE stays NONE.
@@ -99,4 +102,4 @@ QQQ style:
 - `develop` publishes SNAPSHOTs.
 - `release/*` publishes RCs; the orb bumps `<revision>` in the root POM.
 - `v*` tags publish releases.
-- All five modules and the parent POM publish. The flatten plugin resolves `${revision}`.
+- All six modules and the parent POM publish. The flatten plugin resolves `${revision}`.

README.md

@@ -9,6 +9,68 @@ Design spec: `QRun-IO/qqq` `docs/specs/2026-10-11-qqq-assistant-qbit-design.md`.
 | `com.kingsrook.qbits:qbit-assistant` | The QBit: conversation tables, turn engine, `/qqq/v1/assistant` routes (JSON + Server-Sent Events), budgets, kill switch, the provider SPI, and a scripted provider for tests and demos. |
 | `com.kingsrook.qbits:qbit-assistant-provider-anthropic` | The Claude provider (`com.anthropic:anthropic-java`). Add it to use Claude. The core QBit has no LLM SDK dependency. |
 
+Data tools use the QQQ UI API (`/qqq/v1`) with the signed-in user's session. The UI API enforces permissions and record locks; assistant data-access policy and field exposure can only narrow access. Only record-free navigation and help use the core AI tool kernel. Every tool result reaches the model wrapped as untrusted data.
+
+## RAG setup
+
+Knowledge retrieval is optional. The vector index stores **no record content**:
+only source/table/key/field/chunk identifiers, SHA-256 hashes and embeddings.
+Indexing reads through the UI API as the clicking administrator. Retrieval
+re-reads each hit through that same API as the asking user, silently filters
+403/404 results, applies `get_record` field exposure/masking/redaction and returns
+fresh snippets with `/app/...` citations. `search_knowledge` follows `allowedTools`
+and is offered only with an enabled source the user may read in the UI.
+
+1. Register the OpenAI-compatible provider module and configure the embedding
+   endpoint through the existing provider settings/URL allowlist. Discover or add
+   a model, set **Model kind = EMBEDDING** and its dimensions. Known discovery
+   defaults are `bge-m3` (1024) and `nomic-embed-text` (768); these remain editable.
+   CHAT models cannot be selected for embedding, and EMBEDDING models cannot answer
+   chat turns. Each embedding batch contains at most 64 inputs.
+2. Use the default `InMemoryVectorStore` for tests/small hosts, or add
+   `com.kingsrook.qbits:qbit-assistant-rag-pgvector` at the QBit's version for
+   persistence. The host supplies its JDBC driver and DataSource; this module
+   does not depend on QQQ RDBMS internals.
+3. For pgvector, have the DBA install the extension once in the chosen database:
+   `CREATE EXTENSION IF NOT EXISTS vector`. Give the application a distinct schema
+   per QQQ instance. Configure the store before producing the QBit:
+
+   ```java
+   PgVectorStore vectors = new PgVectorStore(dataSource, 1024)
+      .withSchema("assistant_rag_my_instance");
+   vectors.ensureSchema();
+   config.withVectorStore(vectors);
+   config.withUiApiBaseUrl("http://127.0.0.1:8000");
+   ```
+
+   The UI API URL is this server's own listening address. Configure it explicitly
+   so Index now works before any chat request has taught the runtime its local
+   port. The store creates a content-free `assistant_rag_chunk` table and an HNSW
+   cosine index. Its fixed dimension must match every source using the store;
+   different models are searched separately. No credentials go in source rows.
+   See the [pgvector installation/index documentation](https://github.com/pgvector/pgvector)
+   and the reference DDL in [schema-postgres.sql](docs/schema-postgres.sql).
+4. Open **Assistant → Knowledge sources** as an `assistant.admin`. Add the table,
+   comma-separated text fields, optional title field and QQQ query-filter JSON,
+   embedding model, and chunk size/overlap (defaults 1500/200). Tables with AI data
+   access NONE, `assistant_*` tables, and fields with NONE/METADATA_ONLY exposure
+   are refused. MASKED fields are masked before embedding and in snippets.
+5. Select **Index now**. Review makes no embedding calls; Result reports records
+   seen, chunks embedded, unchanged, removed and errors. Re-running unchanged
+   content sends no embeddings. A complete successful scan removes disappeared
+   records; failed scans preserve unvisited entries. **Clear index** removes only
+   that source's vectors. Changes after indexing are read fresh, but semantic
+   ranking becomes current only after re-indexing.
+
+Embedding usage has purpose `EMBED`, counts toward budgets, and observes the kill
+switch and provider concurrency limits. PHI sources and queries in PHI
+conversations require an eligible embedding provider; the eligibility flag is an
+operator assertion, as for chat. Automatic budget downgrades cannot change an
+embedding model's vector space, so an incompatible downgrade stops the call.
+
+Scheduling is deferred to P-g. Its designated-user runner can call
+`RagService.index` within that user's authenticated context; P-f creates no system
+session, stored admin credentials, scheduler or background content crawler.
 Data tools call the QQQ UI API through `UiApiToolbox` and `UiApiGateway` with the signed-in user's session. The UI enforces permissions and record security; the assistant further narrows access and caps and redacts returned values. Data tool results reach the model wrapped as untrusted data. Only record-free navigation and help use the core AI tool kernel.
 
 ## Requirements

UPGRADE.md

@@ -1,5 +1,36 @@
 # Upgrading qbit-assistant
 
+## P-f: knowledge retrieval
+
+Apply the **P-f** assistant-table DDL in
+[docs/schema-postgres.sql](docs/schema-postgres.sql) before deployment. It adds
+`assistant_model.model_kind` (existing rows default to CHAT),
+`embedding_dimensions`, and the admin-only `assistant_rag_source` table.
+No sources are seeded. Existing chat selections remain unchanged; explicitly
+EMBEDDING models are excluded from chat, classifier, fallback and downgrade
+selection. Correct discovery's kind/dimensions if a model name is ambiguous.
+
+For persistent vectors, add `qbit-assistant-rag-pgvector` at the same version as
+the QBit, supply the host's JDBC DataSource/driver, and install pgvector in the
+database. A DBA runs `CREATE EXTENSION IF NOT EXISTS vector`; the application
+role needs schema creation rights for initial setup (or a DBA-created schema),
+then normal DML rights. `ensureSchema()` checks for an existing extension first
+and does not require extension installation privileges when it is already present.
+The optional pgvector DDL is a **commented reference block**, separate from the
+assistant-table migration. See [RAG setup](README.md#rag-setup).
+
+Use a distinct validated schema for each QQQ instance. `vector(N)` fixes dimensions
+for that store; changing dimensions requires a new schema and re-indexing.
+Sources can use different models with that dimension; ranking is restricted to
+sources using the query's model. If the weights behind an unchanged model id
+change, clear and rebuild its sources. Editing source chunk/filter/model settings
+or registry model identity/dimensions automatically clears affected vectors.
+
+Every indexing batch produces EMBED usage and updates existing budget rollups.
+PHI source content and PHI conversation queries require a HIPAA-eligible embedding
+provider. No background runner or designated credentials are installed in P-f.
+The default in-memory store is lost on restart and must be indexed again.
+
 ## P-e: profile routing and provider fallback
 
 Apply the additive **P-e** ALTER TABLE statements at the end of

docs/DESIGN.md

@@ -422,6 +422,53 @@ AI exposure NONE is not enough, because the assistant acts as the user and an ad
 - Admin record screens, additive Postgres DDL and the flow-layout home expose these
   settings and counters without granting AI access to assistant configuration.
 
+### P-f as built
+
+- `AssistantModelProvider.embed` is optional and unsupported by default. The
+  OpenAI-compatible implementation shares chat's JDK HTTP authentication, TLS,
+  Azure deployment paths, timeout and sanitized error mapping. A request accepts
+  1–64 inputs; `EmbeddingService` batches larger jobs and records each call as
+  EMBED, including daily rollups and provider/model budget accounting.
+- Registry rows have CHAT/EMBEDDING kinds and embedding dimensions. Legacy rows
+  remain CHAT. Discovery suggests EMBEDDING for ids containing embed, bge, e5,
+  gte or nomic; administrators can correct it. Chat routing/fallback/downgrades
+  cannot select an embedding model. Embedding resolution is explicit and checks
+  fresh provider enablement and HIPAA eligibility before dispatch.
+- `assistant_rag_source` is an admin-only navigation table. Its text fields must
+  exist and allow VALUES/MASKED exposure; application data access must allow READ.
+  Index now and Clear index are standard streamed ETL record actions with Review
+  and Result, never navigation entries. Indexing pages the UI API as the clicking
+  admin, reuses get_record's masking path, chunks deterministically and compares
+  hashes. Complete scans alone prune disappeared records; per-record replacement
+  also removes obsolete field chunks. Configuration/model identity changes clear
+  stale vectors. No source text is saved on a source or vector row.
+- `VectorStore` supplies upsert/delete/search/count plus content-free chunk
+  reconciliation and atomic record replacement. The in-memory implementation uses
+  exact cosine; the optional pgvector module uses host JDBC, a validated instance
+  schema, bound values and an HNSW cosine index. Sources from other embedding
+  models are excluded before ranking. The DBA installs pgvector; existing
+  installations do not require extension-installation privileges on startup.
+- `KnowledgeTool` is separate from the unchanged `UiApiToolbox`. A small
+  ContextBuilder hook adds it only for an enabled source, policy permission and
+  UI-readable source table. Search embeds the question, requests 4× the result
+  limit, then re-reads each hit through get_record with the asking user's session.
+  403/404 hits contribute only to the filtered count. Safe current values produce
+  snippets and `/app` citations; record refs propagate touched tables into P-c
+  classification and the normal tool-call log. The citation instruction is in
+  the static rules block. The result limit is capped at eight.
+- PHI source content and PHI conversation queries only reach eligible embedding
+  providers. Embedding budgets, kill switch and provider concurrency checks apply
+  to each batch. A budget downgrade into another vector space stops embedding.
+- Tests exercise real fixture UI API reads, denials, record locks, field masking,
+  PHI refusal, classification, reconciliation, embedding wire contracts and a
+  golden cited answer. JDBC shape/transaction tests always run; a pgvector:pg16
+  Testcontainers test skips on hosts without Docker.
+- P-g may invoke the manual service within a designated user's authenticated
+  context. P-f supplies no scheduled runner or stored/elevated session. Multiple
+  replicas should serialize manual re-indexing of a source at the host level;
+  per-record pgvector replacement is transactional, while scan reconciliation is
+  not a snapshot of concurrently changing application data.
+
 ### Phases (each a green PR)
 | | Scope | Depends on |
 |---|---|---|

docs/schema-postgres.sql

@@ -477,3 +477,50 @@ ALTER TABLE assistant_turn ADD COLUMN route_decision VARCHAR(16);
 ALTER TABLE assistant_turn ADD COLUMN route_source VARCHAR(16);
 ALTER TABLE assistant_turn ADD COLUMN fallback_count INTEGER DEFAULT 0;
 ALTER TABLE assistant_turn ADD COLUMN provider_id INTEGER REFERENCES assistant_provider(id);
+
+
+-- P-f: embedding registry and admin-only knowledge-source configuration.
+ALTER TABLE assistant_model ADD COLUMN model_kind VARCHAR(16) DEFAULT 'CHAT'
+   CHECK (model_kind IN ('CHAT', 'EMBEDDING'));
+ALTER TABLE assistant_model ADD COLUMN embedding_dimensions INTEGER
+   CHECK (embedding_dimensions > 0);
+
+CREATE TABLE assistant_rag_source (
+   id SERIAL PRIMARY KEY,
+   name VARCHAR(200) NOT NULL,
+   enabled BOOLEAN DEFAULT TRUE,
+   table_name VARCHAR(200) NOT NULL,
+   field_names VARCHAR(500) NOT NULL,
+   title_field_name VARCHAR(200),
+   filter_json TEXT,
+   embedding_model_id INTEGER NOT NULL REFERENCES assistant_model(id),
+   chunk_chars INTEGER DEFAULT 1500 CHECK (chunk_chars BETWEEN 100 AND 16000),
+   chunk_overlap_chars INTEGER DEFAULT 200 CHECK (chunk_overlap_chars >= 0 AND chunk_overlap_chars < chunk_chars),
+   last_indexed_at TIMESTAMPTZ,
+   last_index_status VARCHAR(200),
+   last_index_error VARCHAR(500),
+   indexed_record_count INTEGER,
+   indexed_chunk_count INTEGER,
+   create_date TIMESTAMPTZ,
+   modify_date TIMESTAMPTZ
+);
+
+-- Optional pgvector part: reference only. Run the extension step as a DBA.
+-- PgVectorStore.ensureSchema() performs the remainder for the host's schema/N.
+-- Use a distinct schema per QQQ instance; match N to the embedding model.
+-- CREATE EXTENSION IF NOT EXISTS vector;
+-- CREATE SCHEMA IF NOT EXISTS assistant_rag;
+-- CREATE TABLE IF NOT EXISTS assistant_rag.assistant_rag_chunk (
+--    id BIGSERIAL PRIMARY KEY,
+--    source_id INT NOT NULL,
+--    table_name TEXT NOT NULL,
+--    record_pk TEXT NOT NULL,
+--    field_name TEXT NOT NULL,
+--    chunk_index INT NOT NULL,
+--    content_hash CHAR(64) NOT NULL,
+--    embedding vector(1024) NOT NULL,
+--    indexed_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
+--    UNIQUE(source_id, table_name, record_pk, field_name, chunk_index)
+-- );
+-- CREATE INDEX IF NOT EXISTS assistant_rag_chunk_cosine
+--    ON assistant_rag.assistant_rag_chunk USING hnsw (embedding vector_cosine_ops);

docs/screenshots/README.md

@@ -35,3 +35,8 @@ Light and dark, 390 and 1440 px:
 - `p-provider-local-*`: Local GPU (Ollama), keyless LOCAL settings, discovery/test results and discovered models.
 - `p-model-*`: a discovered model's capabilities, compute prices and connection-test result.
 - `p-home-providers-*`: the home's provider summary and usage from the executed demo calls, including disabled OpenRouter with its key Not set.
+
+Answer presentation (owner priority, 2026-10-13), taken against `OpenAiDemoServer` with the Next UI build of QRun-IO/qqq-frontend-next#111 (`next dev` proxied to the demo). The demo's OpenAI-compatible mock answers "How many images do I have by type?" the way a model would through the real adapter: five `count_records` calls through the UI API, then `present_results`.
+
+Light and dark, 390 and 1440 px:
+- `panel-presentation-*`: the summary line, an Images metric, the "Media by type" table (counts and the server-computed 80.2% and 19.8%, each row a filtered-list link), the pie chart, two filter links (Open JPEG media, Open PNG media) and a follow-up action.

pom.xml

@@ -55,6 +55,7 @@
       <module>qbit-assistant-provider-openai</module>
       <module>qbit-assistant-provider-bedrock</module>
       <module>qbit-assistant-provider-vertex</module>
+      <module>qbit-assistant-rag-pgvector</module>
    </modules>
 
    <licenses>

qbit-assistant-provider-openai/src/main/java/com/kingsrook/qbits/assistant/provider/openai/OpenAiCompatibleModelProvider.java

@@ -38,6 +38,8 @@ import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
 import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
 import com.kingsrook.qbits.assistant.provider.CompletableStreamHandle;
+import com.kingsrook.qbits.assistant.provider.EmbeddingRequest;
+import com.kingsrook.qbits.assistant.provider.EmbeddingResponse;
 import com.kingsrook.qbits.assistant.provider.ModelCapabilities;
 import com.kingsrook.qbits.assistant.provider.ModelEventListener;
 import com.kingsrook.qbits.assistant.provider.ModelPart;
@@ -82,6 +84,84 @@ public class OpenAiCompatibleModelProvider implements AssistantModelProvider
 
 
 
+   /*******************************************************************************
+    ** One bounded batch, with the deadline covering headers and the entire body.
+    ** Restore input order from the response indexes; reject malformed vectors.
+    *******************************************************************************/
+   @Override
+   public EmbeddingResponse embed(EmbeddingRequest request) throws ModelProviderException
+   {
+      ObjectNode body = OpenAiHttp.JSON.createObjectNode();
+      body.put("model", request.model());
+      body.set("input", OpenAiHttp.JSON.valueToTree(request.input()));
+      var pending = http.client.sendAsync(http.request(request.model(), http.settings.getRequestTimeoutSeconds(), "embeddings")
+         .POST(java.net.http.HttpRequest.BodyPublishers.ofString(body.toString())).build(), HttpResponse.BodyHandlers.ofString());
+      try
+      {
+         HttpResponse<String> response = pending.get(http.settings.getRequestTimeoutSeconds(), TimeUnit.SECONDS);
+         if(response.statusCode() != 200)
+         {
+            throw (http.error(response.statusCode(), response.body(), OpenAiHttp.requestId(response), false));
+         }
+         JsonNode json = OpenAiHttp.JSON.readTree(response.body());
+         if(json.has("error"))
+         {
+            throw (http.error(500, response.body(), OpenAiHttp.requestId(response), false));
+         }
+         float[][] vectors = new float[request.input().size()][];
+         Integer dimensions = null;
+         for(JsonNode item : json.path("data"))
+         {
+            Integer index = item.path("index").asInt(-1);
+            JsonNode values = item.path("embedding");
+            if(index < 0 || index >= vectors.length || vectors[index] != null || !values.isArray() || values.isEmpty()
+               || (dimensions != null && dimensions != values.size()))
+            {
+               throw (new IOException("Invalid embedding response."));
+            }
+            dimensions = values.size();
+            float[] vector = new float[dimensions];
+            for(Integer i = 0; i < vector.length; i++)
+            {
+               if(!values.get(i).isNumber())
+               {
+                  throw (new IOException("Invalid embedding coordinate."));
+               }
+               vector[i] = (float) values.get(i).asDouble();
+               if(!Float.isFinite(vector[i]))
+               {
+                  throw (new IOException("Non-finite embedding coordinate."));
+               }
+            }
+            vectors[index] = vector;
+         }
+         if(java.util.Arrays.stream(vectors).anyMatch(java.util.Objects::isNull))
+         {
+            throw (new IOException("Missing embedding."));
+         }
+         return (new EmbeddingResponse(List.of(vectors), new ModelUsage().withInputTokens(json.path("usage").path("prompt_tokens").asLong()), OpenAiHttp.requestId(response)));
+      }
+      catch(java.util.concurrent.TimeoutException e)
+      {
+         throw (new ModelProviderException(ModelProviderException.Kind.TIMEOUT, null, "The provider did not answer in time."));
+      }
+      catch(InterruptedException e)
+      {
+         Thread.currentThread().interrupt();
+         throw (new ModelProviderException(ModelProviderException.Kind.NETWORK, null, "The embedding request was interrupted."));
+      }
+      catch(java.util.concurrent.ExecutionException | IOException e)
+      {
+         throw (OpenAiHttp.transportError(e));
+      }
+      finally
+      {
+         pending.cancel(true);
+      }
+   }
+
+
+
    /*******************************************************************************
     ** Conservative defaults; registry rows can override model-specific facts.
     *******************************************************************************/

qbit-assistant-provider-openai/src/main/java/com/kingsrook/qbits/assistant/provider/openai/OpenAiHttp.java

@@ -84,13 +84,23 @@ final class OpenAiHttp
     ** Azure uses deployment paths and a version query parameter.
     *******************************************************************************/
    HttpRequest.Builder request(String model, Integer seconds) throws ModelProviderException
+   {
+      return (request(model, seconds, "chat/completions"));
+   }
+
+
+
+   /*******************************************************************************
+    ** Chat and embeddings share authentication, TLS, deadlines and Azure paths.
+    *******************************************************************************/
+   HttpRequest.Builder request(String model, Integer seconds, String operation) throws ModelProviderException
    {
       try
       {
-         String path = model == null ? "/models" : "/chat/completions";
+         String path = model == null ? "/models" : "/" + operation;
          if("AZURE_OPENAI".equals(settings.getKind()))
          {
-            path = model == null ? "/openai/models" : "/openai/deployments/" + encode(model) + "/chat/completions";
+            path = model == null ? "/openai/models" : "/openai/deployments/" + encode(model) + "/" + operation;
             path += "?api-version=" + encode(settings.getApiVersion() == null || settings.getApiVersion().isBlank() ? "2024-10-21" : settings.getApiVersion());
          }
          HttpRequest.Builder builder = HttpRequest.newBuilder(URI.create(base + path)).timeout(Duration.ofSeconds(seconds)).header("Content-Type", "application/json");

qbit-assistant-provider-openai/src/test/java/com/kingsrook/qbits/assistant/provider/openai/OpenAiEmbeddingTest.java

@@ -0,0 +1,133 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.provider.openai;
+
+
+import java.util.List;
+import com.kingsrook.qbits.assistant.provider.EmbeddingRequest;
+import com.kingsrook.qbits.assistant.provider.ModelProviderException;
+import com.kingsrook.qbits.assistant.provider.ProviderSettings;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Embedding wire contracts through the same local HTTP harness as chat.
+ *******************************************************************************/
+class OpenAiEmbeddingTest
+{
+   /*******************************************************************************
+    ** Input arrays, response indexes, usage and authentication are preserved.
+    *******************************************************************************/
+   @Test
+   void shapeOrderAndAzure() throws Exception
+   {
+      for(String kind : List.of("OPENAI", "AZURE_OPENAI", "OLLAMA"))
+      {
+         try(MockOpenAiServer server = new MockOpenAiServer())
+         {
+            server.enqueue(200, "{\"data\":[{\"index\":1,\"embedding\":[0,1]},{\"index\":0,\"embedding\":[1,0]}],\"usage\":{\"prompt_tokens\":7}}");
+            var provider = new OpenAiCompatibleModelProvider(new ProviderSettings().withKind(kind).withBaseUrl(server.url()).withApiKey("synthetic-key").withApiVersion("2024-10-21"));
+            var response = provider.embed(new EmbeddingRequest("bge-m3", List.of("first", "second")));
+            assertThat(response.vectors()).hasSize(2);
+            assertThat(response.vectors().getFirst()).containsExactly(1, 0);
+            assertThat(response.usage().getInputTokens()).isEqualTo(7L);
+            var captured = server.requests.getFirst();
+            assertThat(captured.path()).isEqualTo(kind.equals("AZURE_OPENAI") ? "/v1/openai/deployments/bge-m3/embeddings?api-version=2024-10-21" : "/v1/embeddings");
+            var body = OpenAiHttp.JSON.readTree(captured.body());
+            assertThat(body.path("model").asText()).isEqualTo("bge-m3");
+            assertThat(body.path("input").size()).isEqualTo(2);
+            assertThat(body.has("messages")).isFalse();
+            assertThat(body.has("stream")).isFalse();
+            assertThat(kind.equals("AZURE_OPENAI") ? captured.apiKey() : captured.authorization()).contains("synthetic-key");
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Reuse chat's safe error categories, without echoing credentials.
+    *******************************************************************************/
+   @Test
+   void errorsAndMalformedVectors() throws Exception
+   {
+      try(MockOpenAiServer server = new MockOpenAiServer())
+      {
+         var provider = new OpenAiCompatibleModelProvider(new ProviderSettings().withKind("OPENAI").withBaseUrl(server.url()).withApiKey("synthetic-key"));
+         server.enqueue(429, "{\"error\":{\"message\":\"synthetic-key rejected\"}}");
+         assertThatThrownBy(() -> provider.embed(new EmbeddingRequest("embed", List.of("text"))))
+            .isInstanceOfSatisfying(ModelProviderException.class, e ->
+            {
+               assertThat(e.getKind()).isEqualTo(ModelProviderException.Kind.RATE_LIMITED);
+               assertThat(e.getMessage()).doesNotContain("synthetic-key");
+            });
+         for(String body : List.of("{}", "{\"data\":[{\"index\":0,\"embedding\":[\"not numeric\"]}]}",
+            "{\"data\":[{\"index\":2,\"embedding\":[1]}]}", "{\"data\":[{\"index\":0,\"embedding\":[1]},{\"index\":0,\"embedding\":[1]}]}"))
+         {
+            server.enqueue(200, body);
+            assertThatThrownBy(() -> provider.embed(new EmbeddingRequest("embed", List.of("text")))).isInstanceOf(ModelProviderException.class);
+         }
+         assertThatThrownBy(() -> new EmbeddingRequest("embed", java.util.Collections.nCopies(65, "text"))).isInstanceOf(IllegalArgumentException.class);
+         assertThatThrownBy(() -> new EmbeddingRequest("embed", List.of())).isInstanceOf(IllegalArgumentException.class);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Deadline covers a stalled response body, not only HTTP headers.
+    *******************************************************************************/
+   @Test
+   void bodyTimeout() throws Exception
+   {
+      try(MockOpenAiServer server = new MockOpenAiServer())
+      {
+         server.stall(true);
+         var provider = new OpenAiCompatibleModelProvider(new ProviderSettings().withKind("OLLAMA").withBaseUrl(server.url()).withRequestTimeoutSeconds(1));
+         assertThatThrownBy(() -> provider.embed(new EmbeddingRequest("embed", List.of("text"))))
+            .isInstanceOfSatisfying(ModelProviderException.class, e -> assertThat(e.getKind()).isEqualTo(ModelProviderException.Kind.TIMEOUT));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The transport accepts a complete 64-input batch and rejects overflow
+    ** before sending. EmbeddingService owns splitting larger indexing jobs.
+    *******************************************************************************/
+   @Test
+   void maximumBatch() throws Exception
+   {
+      try(MockOpenAiServer server = new MockOpenAiServer())
+      {
+         var body = OpenAiHttp.JSON.createObjectNode();
+         var data = body.putArray("data");
+         for(Integer i = 0; i < 64; i++)
+         {
+            data.addObject().put("index", i).putArray("embedding").add(1).add(0);
+         }
+         server.enqueue(200, body.toString());
+         var provider = new OpenAiCompatibleModelProvider(new ProviderSettings().withKind("OLLAMA").withBaseUrl(server.url()));
+         assertThat(provider.embed(new EmbeddingRequest("embed", java.util.Collections.nCopies(64, "text"))).vectors()).hasSize(64);
+         assertThat(OpenAiHttp.JSON.readTree(server.requests.getFirst().body()).path("input").size()).isEqualTo(64);
+         assertThatThrownBy(() -> provider.embed(new EmbeddingRequest("embed", java.util.Collections.nCopies(65, "text")))).isInstanceOf(IllegalArgumentException.class);
+         assertThat(server.requests).hasSize(1);
+      }
+   }
+}

qbit-assistant-provider-openai/src/test/java/com/kingsrook/qbits/assistant/provider/openai/demo/OpenAiDemoServer.java

@@ -21,8 +21,13 @@ import java.io.IOException;
 import java.math.BigDecimal;
 import java.net.InetSocketAddress;
 import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
 import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
 import com.kingsrook.qbits.assistant.model.AssistantModel;
@@ -35,6 +40,9 @@ import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
 import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
 import com.kingsrook.qbits.assistant.provider.openai.OpenAiCompatibleProviderType;
 import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.EvalSupport;
+import com.kingsrook.qbits.assistant.testing.GoldenTasks;
+import com.kingsrook.qbits.assistant.testing.PresentationGoldenTask;
 import com.kingsrook.qbits.assistant.testing.TurnRunner;
 import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory;
 import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
@@ -61,6 +69,8 @@ public class OpenAiDemoServer extends AbstractQQQApplication
 {
    private static final String ADMIN = "admin-demo";
    private static final String MODEL = "qwen3:30b-a3b";
+   private static final String IMAGE_QUESTION = "How many images do I have by type?";
+   private static final ObjectMapper JSON = new ObjectMapper();
    private static QInstance instance;
 
 
@@ -160,6 +170,8 @@ public class OpenAiDemoServer extends AbstractQQQApplication
       {
          throw (new QException("Demo assistant turn failed: " + outcome.errorCode()));
       }
+      TurnRunner.Outcome presentation = new TurnRunner(instance, config).run(ADMIN, IMAGE_QUESTION);
+      PresentationGoldenTask.check(presentation);
       instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("mock").withType(QAuthenticationType.MOCK)
          .withCustomizer(new QCodeReference(DemoSessionCustomizer.class)));
       instance.setHomeAppName(AssistantMetaDataBuilder.APP_NAME);
@@ -167,6 +179,7 @@ public class OpenAiDemoServer extends AbstractQQQApplication
       System.out.println("Assistant home: http://localhost:" + port + "/app/" + AssistantMetaDataBuilder.APP_NAME);
       System.out.println("Local provider: http://localhost:" + port + "/app/assistant_provider/" + providerId);
       System.out.println("Local model: http://localhost:" + port + "/app/assistant_model/" + modelId);
+      System.out.println("Image presentation conversation id: " + presentation.conversationId());
    }
 
 
@@ -211,7 +224,7 @@ public class OpenAiDemoServer extends AbstractQQQApplication
    {
       try(exchange)
       {
-         exchange.getRequestBody().readAllBytes();
+         byte[] request = exchange.getRequestBody().readAllBytes();
          String path = exchange.getRequestURI().getPath();
          String body;
          if("GET".equals(exchange.getRequestMethod()) && "/v1/models".equals(path))
@@ -227,16 +240,7 @@ public class OpenAiDemoServer extends AbstractQQQApplication
          else if("POST".equals(exchange.getRequestMethod()) && "/v1/chat/completions".equals(path))
          {
             exchange.getResponseHeaders().set("Content-Type", "text/event-stream");
-            body = """
-               data: {"id":"demo-completion","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello from the local demo provider."}}]}
-
-               data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
-
-               data: {"choices":[],"usage":{"prompt_tokens":24,"completion_tokens":8,"total_tokens":32}}
-
-               data: [DONE]
-
-               """;
+            body = completion(JSON.readTree(request));
          }
          else
          {
@@ -248,4 +252,84 @@ public class OpenAiDemoServer extends AbstractQQQApplication
          exchange.getResponseBody().flush();
       }
    }
+
+
+
+   /*******************************************************************************
+    ** Follow the latest user turn, so connection tests and repeated chats are
+    ** independent. Counts are read from the adapter's actual tool messages.
+    *******************************************************************************/
+   private static String completion(JsonNode request) throws IOException
+   {
+      String question = "";
+      Map<String, Integer> counts = new LinkedHashMap<>();
+      boolean presented = false;
+      for(JsonNode message : request.path("messages"))
+      {
+         if("user".equals(message.path("role").asText()))
+         {
+            question = message.path("content").asText();
+            counts.clear();
+            presented = false;
+         }
+         else if("tool".equals(message.path("role").asText()) && IMAGE_QUESTION.equals(question))
+         {
+            String id = message.path("tool_call_id").asText();
+            if(id.startsWith("count-image/"))
+            {
+               counts.put(id.substring("count-".length()), ((Number) EvalSupport.unwrap(message.path("content").asText()).get("count")).intValue());
+            }
+            else if("present-images".equals(id))
+            {
+               presented = true;
+            }
+         }
+      }
+
+      Map<String, Object> delta = Map.of("role", "assistant", "content", "Hello from the local demo provider.");
+      String finish = "stop";
+      if(IMAGE_QUESTION.equals(question) && !presented)
+      {
+         List<Map<String, Object>> calls = new ArrayList<>();
+         if(counts.isEmpty())
+         {
+            for(String mime : List.of("image/jpeg", "image/png", "image/webp", "image/gif", "image/svg+xml"))
+            {
+               calls.add(toolCall(calls.size(), "count-" + mime, "count_records", Map.of("table", "media", "filter",
+                  GoldenTasks.filter("AND", GoldenTasks.criterion("mimeType", "EQUALS", mime)))));
+            }
+         }
+         else
+         {
+            if(counts.size() != 5 || counts.get("image/webp") != 0 || counts.get("image/gif") != 0 || counts.get("image/svg+xml") != 0)
+            {
+               throw (new IOException("Unexpected demo image counts."));
+            }
+            Map<String, Object> input = PresentationGoldenTask.input(counts.get("image/jpeg"), counts.get("image/png"));
+            input.put("actions", List.of(Map.of("label", "Show the PNG files", "prompt", "Show the PNG files")));
+            calls.add(toolCall(0, "present-images", "present_results", input));
+         }
+         delta = Map.of("role", "assistant", "tool_calls", calls);
+         finish = "tool_calls";
+      }
+      else if(presented)
+      {
+         delta = Map.of("role", "assistant", "content", "You can open either image type above.");
+      }
+      return ("data: " + JSON.writeValueAsString(Map.of("id", "demo-completion", "object", "chat.completion.chunk",
+         "choices", List.of(Map.of("index", 0, "delta", delta)))) + "\n\n"
+         + "data: " + JSON.writeValueAsString(Map.of("choices", List.of(Map.of("index", 0, "delta", Map.of(), "finish_reason", finish)))) + "\n\n"
+         + "data: {\"choices\":[],\"usage\":{\"prompt_tokens\":24,\"completion_tokens\":8,\"total_tokens\":32}}\n\n"
+         + "data: [DONE]\n\n");
+   }
+
+
+
+   /*******************************************************************************
+    ** Encode tool arguments as a JSON string, as OpenAI streaming requires.
+    *******************************************************************************/
+   private static Map<String, Object> toolCall(Integer index, String id, String name, Map<String, Object> arguments) throws IOException
+   {
+      return (Map.of("index", index, "id", id, "type", "function", "function", Map.of("name", name, "arguments", JSON.writeValueAsString(arguments))));
+   }
 }

qbit-assistant-provider-openai/src/test/java/com/kingsrook/qbits/assistant/provider/openai/demo/README.md

@@ -29,6 +29,11 @@ Startup prints these URLs (with the selected port):
 - Local provider: `http://localhost:8099/app/assistant_provider/1`
 - Priced and tested model: `http://localhost:8099/app/assistant_model/1`
 
+Startup also prints `Image presentation conversation id: <id>`. Open that
+conversation in the assistant panel history as `admin-demo` to capture the
+answer to "How many images do I have by type?". The id is assigned on each run;
+the conversation and its presentation are memory-backed and reset on restart.
+
 Every browser session is `admin-demo`. Startup inserts an enabled, default,
 keyless Local GPU (Ollama) provider, marked LOCAL and HIPAA eligible for the
 synthetic demo, and a disabled CLOUD OpenRouter provider whose key is Not set.
@@ -40,8 +45,14 @@ mock: `qwen3:30b-a3b`, `gpt-oss:20b`, `qwen3-coder:30b`, and `bge-m3`.
 A normal update sets qwen3's input/output compute prices to $0.05/$0.10 per
 million tokens. The provider and model Test connection processes then each
 make a real HTTP completion call, recording two TEST usage rows and the
-last-test fields. One assistant turn records a third usage row with purpose
-TURN. Discovery timestamps/status also come from the actual process.
+last-test fields. The greeting turn records a third usage row with purpose
+TURN. An image breakdown turn then makes three more model calls through the
+real OpenAI adapter and engine: five MIME-filtered `count_records` calls,
+`present_results` using the returned counts, and a short closing message.
+The fixture seeds 81 JPEG and 20 PNG rows (no WebP, GIF or SVG). The presentation
+includes a table with computed shares, filtered row links, a pie chart, JPEG
+and PNG filter links, and a "Show the PNG files" prompt action.
+Discovery timestamps/status also come from the actual process.
 
 Replies and token counts are synthetic mock API responses (24 input / 8 output
 tokens per completion), not live GPU measurements or billed cloud usage.

qbit-assistant-rag-pgvector/pom.xml

@@ -0,0 +1,130 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Copyright 2026 Kingsrook, LLC
+  ~
+  ~ Licensed under the Apache License, Version 2.0 (the "License");
+  ~ you may not use this file except in compliance with the License.
+  ~ You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+   <modelVersion>4.0.0</modelVersion>
+
+   <parent>
+      <groupId>com.kingsrook.qbits</groupId>
+      <artifactId>qbit-assistant-parent</artifactId>
+      <version>${revision}</version>
+   </parent>
+
+   <artifactId>qbit-assistant-rag-pgvector</artifactId>
+   <packaging>jar</packaging>
+
+   <name>QBit Assistant - pgvector</name>
+   <description>Content-free per-instance vector storage through JDBC</description>
+   <url>https://github.com/QRun-IO/qbit-assistant</url>
+
+   <dependencies>
+      <dependency>
+         <groupId>com.kingsrook.qbits</groupId>
+         <artifactId>qbit-assistant</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>com.kingsrook.qqq</groupId>
+         <artifactId>qqq-backend-core</artifactId>
+      </dependency>
+      <dependency>
+         <groupId>org.testcontainers</groupId>
+         <artifactId>junit-jupiter</artifactId>
+         <version>1.21.4</version>
+         <scope>test</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.testcontainers</groupId>
+         <artifactId>postgresql</artifactId>
+         <version>1.21.4</version>
+         <scope>test</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.postgresql</groupId>
+         <artifactId>postgresql</artifactId>
+         <version>42.7.14</version>
+         <scope>test</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.junit.jupiter</groupId>
+         <artifactId>junit-jupiter</artifactId>
+         <scope>test</scope>
+      </dependency>
+      <dependency>
+         <groupId>org.assertj</groupId>
+         <artifactId>assertj-core</artifactId>
+         <scope>test</scope>
+      </dependency>
+   </dependencies>
+
+   <build>
+      <plugins>
+         <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-surefire-plugin</artifactId>
+            <configuration>
+               <!-- live evals call the real API: opt-in only -->
+               <excludedGroups>${test.excludedGroups}</excludedGroups>
+               <redirectTestOutputToFile>true</redirectTestOutputToFile>
+            </configuration>
+         </plugin>
+         <plugin>
+            <groupId>org.jacoco</groupId>
+            <artifactId>jacoco-maven-plugin</artifactId>
+            <executions>
+               <execution>
+                  <id>prepare-agent</id>
+                  <goals>
+                     <goal>prepare-agent</goal>
+                  </goals>
+               </execution>
+               <execution>
+                  <id>report</id>
+                  <phase>verify</phase>
+                  <goals>
+                     <goal>report</goal>
+                  </goals>
+               </execution>
+               <execution>
+                  <id>check</id>
+                  <goals>
+                     <goal>check</goal>
+                  </goals>
+                  <configuration>
+                     <rules>
+                        <rule>
+                           <element>BUNDLE</element>
+                           <limits>
+                              <limit>
+                                 <counter>INSTRUCTION</counter>
+                                 <value>COVEREDRATIO</value>
+                                 <minimum>0.70</minimum>
+                              </limit>
+                              <limit>
+                                 <counter>CLASS</counter>
+                                 <value>COVEREDRATIO</value>
+                                 <minimum>0.90</minimum>
+                              </limit>
+                           </limits>
+                        </rule>
+                     </rules>
+                  </configuration>
+               </execution>
+            </executions>
+         </plugin>
+      </plugins>
+   </build>
+</project>

qbit-assistant-rag-pgvector/src/main/java/com/kingsrook/qbits/assistant/rag/pgvector/PgVectorStore.java

@@ -0,0 +1,452 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag.pgvector;
+
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.StringJoiner;
+import javax.sql.DataSource;
+import com.kingsrook.qbits.assistant.rag.ChunkVector;
+import com.kingsrook.qbits.assistant.rag.VectorStore;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** Host-supplied JDBC only, without QQQ RDBMS internals. No text is ever stored.
+ *******************************************************************************/
+public class PgVectorStore implements VectorStore
+{
+   private final ConnectionSupplier connections;
+   private final Integer dimensions;
+   private String schema = "assistant_rag";
+
+
+
+   /*******************************************************************************
+    ** The host owns credentials and connection pooling; each call closes its lease.
+    *******************************************************************************/
+   @FunctionalInterface
+   public interface ConnectionSupplier
+   {
+      /*******************************************************************************
+       ** Return a connection owned by this caller until it closes it.
+       *******************************************************************************/
+      Connection open() throws SQLException;
+   }
+
+
+
+   /*******************************************************************************
+    ** Use any host DataSource and a fixed vector dimension for this instance.
+    *******************************************************************************/
+   public PgVectorStore(DataSource dataSource, Integer dimensions)
+   {
+      this(dataSource::getConnection, dimensions);
+   }
+
+
+
+   /*******************************************************************************
+    ** HNSW vector cosine indexes support up to 2000 dimensions.
+    *******************************************************************************/
+   public PgVectorStore(ConnectionSupplier connections, Integer dimensions)
+   {
+      if(connections == null || dimensions == null || dimensions < 1 || dimensions > 2000)
+      {
+         throw (new IllegalArgumentException("A connection supplier and 1 to 2000 dimensions are required."));
+      }
+      this.connections = connections;
+      this.dimensions = dimensions;
+   }
+
+
+
+   /*******************************************************************************
+    ** Configure before use. Identifiers cannot be parameters, so validate strictly.
+    *******************************************************************************/
+   public PgVectorStore withSchema(String schema)
+   {
+      if(schema == null || !schema.matches("[a-z_][a-z0-9_]{0,62}"))
+      {
+         throw (new IllegalArgumentException("The vector schema must be a lower-case SQL identifier of at most 63 characters."));
+      }
+      this.schema = schema;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** The DBA normally installs vector. An existing extension needs no elevated
+    ** privileges; if absent, attempt installation and report an actionable error.
+    *******************************************************************************/
+   public void ensureSchema() throws QException
+   {
+      try(Connection connection = connections.open())
+      {
+         try(PreparedStatement check = connection.prepareStatement("SELECT 1 FROM pg_extension WHERE extname = ?"))
+         {
+            check.setString(1, "vector");
+            try(ResultSet rows = check.executeQuery())
+            {
+               if(!rows.next())
+               {
+                  try(PreparedStatement extension = connection.prepareStatement("CREATE EXTENSION IF NOT EXISTS vector"))
+                  {
+                     extension.execute();
+                  }
+                  catch(SQLException e)
+                  {
+                     throw (new QException("Ask the DBA to run CREATE EXTENSION IF NOT EXISTS vector in this database, then retry.", e));
+                  }
+               }
+            }
+         }
+         for(String sql : List.of("CREATE SCHEMA IF NOT EXISTS " + schema,
+            "CREATE TABLE IF NOT EXISTS " + table() + " (id BIGSERIAL PRIMARY KEY, source_id INT NOT NULL, table_name TEXT NOT NULL, record_pk TEXT NOT NULL, field_name TEXT NOT NULL, chunk_index INT NOT NULL, content_hash CHAR(64) NOT NULL, embedding vector(" + dimensions + ") NOT NULL, indexed_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(source_id, table_name, record_pk, field_name, chunk_index))",
+            "CREATE INDEX IF NOT EXISTS assistant_rag_chunk_cosine ON " + table() + " USING hnsw (embedding vector_cosine_ops)"))
+         {
+            try(PreparedStatement statement = connection.prepareStatement(sql))
+            {
+               statement.execute();
+            }
+         }
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Validate the entire batch before sending anything to JDBC.
+    *******************************************************************************/
+   @Override
+   public void upsert(List<ChunkVector> chunks) throws QException
+   {
+      chunks.forEach(chunk -> vector(chunk.embedding()));
+      try(Connection connection = connections.open())
+      {
+         upsert(connection, chunks);
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Values, including vectors and arbitrary record keys, are bound parameters.
+    *******************************************************************************/
+   private void upsert(Connection connection, List<ChunkVector> chunks) throws SQLException
+   {
+      String sql = "INSERT INTO " + table() + " (source_id, table_name, record_pk, field_name, chunk_index, content_hash, embedding) VALUES (?, ?, ?, ?, ?, ?, ?::vector) ON CONFLICT (source_id, table_name, record_pk, field_name, chunk_index) DO UPDATE SET content_hash = EXCLUDED.content_hash, embedding = EXCLUDED.embedding, indexed_at = CURRENT_TIMESTAMP";
+      try(PreparedStatement statement = connection.prepareStatement(sql))
+      {
+         for(ChunkVector chunk : chunks)
+         {
+            statement.setInt(1, chunk.source());
+            statement.setString(2, chunk.table());
+            statement.setString(3, chunk.pk());
+            statement.setString(4, chunk.field());
+            statement.setInt(5, chunk.chunkIndex());
+            statement.setString(6, chunk.contentHash());
+            statement.setString(7, vector(chunk.embedding()));
+            statement.addBatch();
+         }
+         statement.executeBatch();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A record replacement is transactional, including empty replacements.
+    *******************************************************************************/
+   @Override
+   public void replaceRecord(Integer source, String table, String pk, List<ChunkVector> chunks) throws QException
+   {
+      for(ChunkVector chunk : chunks)
+      {
+         vector(chunk.embedding());
+         if(!source.equals(chunk.source()) || !table.equals(chunk.table()) || !pk.equals(chunk.pk()))
+         {
+            throw (new IllegalArgumentException("Replacement chunks must belong to the specified record."));
+         }
+      }
+      try(Connection connection = connections.open())
+      {
+         Boolean autoCommit = connection.getAutoCommit();
+         connection.setAutoCommit(false);
+         try
+         {
+            deleteRecord(connection, source, table, pk);
+            upsert(connection, chunks);
+            connection.commit();
+         }
+         catch(SQLException e)
+         {
+            connection.rollback();
+            throw (e);
+         }
+         finally
+         {
+            connection.setAutoCommit(autoCommit);
+         }
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Remove one record, always scoped to its source.
+    *******************************************************************************/
+   @Override
+   public void deleteRecord(Integer source, String table, String pk) throws QException
+   {
+      try(Connection connection = connections.open())
+      {
+         deleteRecord(connection, source, table, pk);
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Shared transactional delete statement.
+    *******************************************************************************/
+   private void deleteRecord(Connection connection, Integer source, String table, String pk) throws SQLException
+   {
+      try(PreparedStatement statement = connection.prepareStatement("DELETE FROM " + table() + " WHERE source_id = ? AND table_name = ? AND record_pk = ?"))
+      {
+         statement.setInt(1, source);
+         statement.setString(2, table);
+         statement.setString(3, pk);
+         statement.executeUpdate();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Clear a source, preserving every other source in the instance.
+    *******************************************************************************/
+   @Override
+   public void deleteSource(Integer source) throws QException
+   {
+      try(Connection connection = connections.open(); PreparedStatement statement = connection.prepareStatement("DELETE FROM " + table() + " WHERE source_id = ?"))
+      {
+         statement.setInt(1, source);
+         statement.executeUpdate();
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** HNSW cosine ordering; tables are values, never interpolated SQL identifiers.
+    *******************************************************************************/
+   @Override
+   public List<Hit> search(float[] query, Integer k, Set<String> tables) throws QException
+   {
+      return (search(query, k, tables, Set.of()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Apply enabled compatible source restrictions before the nearest-hit limit.
+    *******************************************************************************/
+   @Override
+   public List<Hit> search(float[] query, Integer k, Set<String> tables, Set<Integer> sources) throws QException
+   {
+      if(tables.isEmpty() || k <= 0)
+      {
+         return (List.of());
+      }
+      String vector = vector(query);
+      String sql = "SELECT source_id, table_name, record_pk, field_name, chunk_index, 1 - (embedding <=> ?::vector) AS score FROM " + table()
+         + " WHERE table_name IN (" + String.join(", ", Collections.nCopies(tables.size(), "?")) + ")"
+         + (sources.isEmpty() ? "" : " AND source_id IN (" + String.join(", ", Collections.nCopies(sources.size(), "?")) + ")")
+         + " ORDER BY embedding <=> ?::vector LIMIT ?";
+      try(Connection connection = connections.open(); PreparedStatement statement = connection.prepareStatement(sql))
+      {
+         Integer parameter = 1;
+         statement.setString(parameter++, vector);
+         for(String table : tables)
+         {
+            statement.setString(parameter++, table);
+         }
+         for(Integer source : sources)
+         {
+            statement.setInt(parameter++, source);
+         }
+         statement.setString(parameter++, vector);
+         statement.setInt(parameter, k);
+         List<Hit> hits = new ArrayList<>();
+         try(ResultSet rows = statement.executeQuery())
+         {
+            while(rows.next())
+            {
+               Double score = rows.getDouble("score");
+               if(Double.isFinite(score))
+               {
+                  hits.add(new Hit(rows.getInt("source_id"), rows.getString("table_name"), rows.getString("record_pk"), rows.getString("field_name"), rows.getInt("chunk_index"), score));
+               }
+            }
+         }
+         return (hits);
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Count chunks for the source's status screen.
+    *******************************************************************************/
+   @Override
+   public Long count(Integer source) throws QException
+   {
+      try(Connection connection = connections.open(); PreparedStatement statement = connection.prepareStatement("SELECT COUNT(*) FROM " + table() + " WHERE source_id = ?"))
+      {
+         statement.setInt(1, source);
+         try(ResultSet rows = statement.executeQuery())
+         {
+            rows.next();
+            return (rows.getLong(1));
+         }
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Content-free reconciliation data. Embedding text is pgvector's numeric form.
+    *******************************************************************************/
+   @Override
+   public List<ChunkVector> chunks(Integer source) throws QException
+   {
+      try(Connection connection = connections.open(); PreparedStatement statement = connection.prepareStatement("SELECT table_name, record_pk, field_name, chunk_index, content_hash, embedding::text FROM " + table() + " WHERE source_id = ?"))
+      {
+         statement.setInt(1, source);
+         List<ChunkVector> chunks = new ArrayList<>();
+         try(ResultSet rows = statement.executeQuery())
+         {
+            while(rows.next())
+            {
+               chunks.add(new ChunkVector(source, rows.getString("table_name"), rows.getString("record_pk"), rows.getString("field_name"), rows.getInt("chunk_index"), rows.getString("content_hash"), parse(rows.getString("embedding"))));
+            }
+         }
+         return (chunks);
+      }
+      catch(SQLException e)
+      {
+         throw (failure(e));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Qualified name uses only the validated instance schema.
+    *******************************************************************************/
+   private String table()
+   {
+      return (schema + ".assistant_rag_chunk");
+   }
+
+
+
+   /*******************************************************************************
+    ** Serialize only finite coordinates with the configured dimension.
+    *******************************************************************************/
+   private String vector(float[] vector)
+   {
+      if(vector == null || vector.length != dimensions)
+      {
+         throw (new IllegalArgumentException("Vector dimensions differ from the configured pgvector schema."));
+      }
+      StringJoiner values = new StringJoiner(",", "[", "]");
+      for(float value : vector)
+      {
+         if(!Float.isFinite(value))
+         {
+            throw (new IllegalArgumentException("Vector coordinates must be finite."));
+         }
+         values.add(Float.toString(value));
+      }
+      return (values.toString());
+   }
+
+
+
+   /*******************************************************************************
+    ** Parse the database's numeric representation without any source content.
+    *******************************************************************************/
+   private float[] parse(String text)
+   {
+      String[] parts = text.substring(1, text.length() - 1).split(",");
+      float[] vector = new float[parts.length];
+      for(Integer i = 0; i < parts.length; i++)
+      {
+         vector[i] = Float.parseFloat(parts[i]);
+      }
+      vector(vector);
+      return (vector);
+   }
+
+
+
+   /*******************************************************************************
+    ** Do not expose database URLs, SQL values or credentials in tool responses.
+    *******************************************************************************/
+   private static QException failure(SQLException cause)
+   {
+      return (new QException("The knowledge vector store is unavailable. Check its schema, dimensions and database permissions.", cause));
+   }
+}

qbit-assistant-rag-pgvector/src/test/java/com/kingsrook/qbits/assistant/rag/pgvector/PgVectorContainerTest.java

@@ -0,0 +1,76 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag.pgvector;
+
+
+import java.sql.DriverManager;
+import java.util.List;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.rag.ChunkVector;
+import com.kingsrook.qbits.assistant.rag.TextChunks;
+import org.junit.jupiter.api.Test;
+import org.testcontainers.containers.PostgreSQLContainer;
+import org.testcontainers.junit.jupiter.Container;
+import org.testcontainers.junit.jupiter.Testcontainers;
+import org.testcontainers.utility.DockerImageName;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Real pgvector contract; skipped automatically on hosts without Docker.
+ *******************************************************************************/
+@Testcontainers(disabledWithoutDocker = true)
+class PgVectorContainerTest
+{
+   @Container
+   static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>(DockerImageName.parse("pgvector/pgvector:pg16").asCompatibleSubstituteFor("postgres"));
+
+
+
+   /*******************************************************************************
+    ** DDL idempotence, cosine ranking, upsert, deletes and instance isolation.
+    *******************************************************************************/
+   @Test
+   void realVectorLifecycle() throws Exception
+   {
+      PgVectorStore.ConnectionSupplier connections = () -> DriverManager.getConnection(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword());
+      PgVectorStore store = new PgVectorStore(connections, 3).withSchema("instance_one");
+      PgVectorStore other = new PgVectorStore(connections, 3).withSchema("instance_two");
+      store.ensureSchema();
+      store.ensureSchema();
+      other.ensureSchema();
+      ChunkVector first = new ChunkVector(1, "article", "1", "body", 0, TextChunks.hash("one"), new float[] { 1, 0, 0 });
+      ChunkVector second = new ChunkVector(1, "article", "2", "body", 0, TextChunks.hash("two"), new float[] { 0, 1, 0 });
+      store.upsert(List.of(first, second));
+      store.upsert(List.of(first));
+      assertThat(store.count(1)).isEqualTo(2);
+      assertThat(other.count(1)).isZero();
+      assertThat(store.search(new float[] { 1, 0, 0 }, 1, Set.of("article"))).singleElement().satisfies(hit ->
+      {
+         assertThat(hit.pk()).isEqualTo("1");
+         assertThat(hit.score()).isEqualTo(1.0);
+      });
+      assertThat(store.chunks(1)).hasSize(2);
+      store.replaceRecord(1, "article", "1", List.of());
+      assertThat(store.count(1)).isEqualTo(1);
+      store.deleteRecord(1, "article", "2");
+      assertThat(store.count(1)).isZero();
+      store.upsert(List.of(first));
+      store.deleteSource(1);
+      assertThat(store.count(1)).isZero();
+   }
+}

qbit-assistant-rag-pgvector/src/test/java/com/kingsrook/qbits/assistant/rag/pgvector/PgVectorStoreTest.java

@@ -0,0 +1,228 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag.pgvector;
+
+
+import java.lang.reflect.Proxy;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.sql.DataSource;
+import com.kingsrook.qbits.assistant.rag.ChunkVector;
+import com.kingsrook.qbits.assistant.rag.TextChunks;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Always-on JDBC SQL-shape and transaction tests, with no Docker dependency.
+ *******************************************************************************/
+class PgVectorStoreTest
+{
+   private final List<String> sql = new ArrayList<>();
+   private final List<Map<Integer, Object>> parameters = new ArrayList<>();
+   private final List<String> operations = new ArrayList<>();
+   private Boolean extensionExists = true;
+   private Boolean failBatch = false;
+   private Boolean failExtension = false;
+
+
+
+   /*******************************************************************************
+    ** JDBC recording connection; values never appear in SQL text.
+    *******************************************************************************/
+   private Connection connection()
+   {
+      return ((Connection) Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] { Connection.class }, (proxy, method, args) ->
+      {
+         operations.add(method.getName());
+         if(method.getName().equals("prepareStatement"))
+         {
+            return (statement((String) args[0]));
+         }
+         return (method.getName().equals("getAutoCommit") ? true : null);
+      }));
+   }
+
+
+
+   /*******************************************************************************
+    ** Capture each binding and emulate rows or SQL failures.
+    *******************************************************************************/
+   private PreparedStatement statement(String query)
+   {
+      sql.add(query);
+      Map<Integer, Object> values = new HashMap<>();
+      parameters.add(values);
+      return ((PreparedStatement) Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] { PreparedStatement.class }, (proxy, method, args) ->
+      {
+         if(method.getName().startsWith("set"))
+         {
+            values.put((Integer) args[0], args[1]);
+         }
+         if(method.getName().equals("executeQuery"))
+         {
+            return (rows(query));
+         }
+         if(method.getName().equals("executeBatch"))
+         {
+            if(failBatch)
+            {
+               throw (new SQLException("synthetic batch failure"));
+            }
+            return (new int[] { 1 });
+         }
+         if(method.getName().equals("executeUpdate"))
+         {
+            return (1);
+         }
+         if(method.getName().equals("execute"))
+         {
+            if(failExtension && query.startsWith("CREATE EXTENSION"))
+            {
+               throw (new SQLException("extension privilege required"));
+            }
+            return (true);
+         }
+         return (null);
+      }));
+   }
+
+
+
+   /*******************************************************************************
+    ** One content-free result; the extension probe can return no rows.
+    *******************************************************************************/
+   private ResultSet rows(String query)
+   {
+      Boolean[] first = { true };
+      return ((ResultSet) Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] { ResultSet.class }, (proxy, method, args) ->
+      {
+         if(method.getName().equals("next"))
+         {
+            Boolean next = first[0] && (!query.contains("pg_extension") || extensionExists);
+            first[0] = false;
+            return (next);
+         }
+         return (switch(method.getName())
+         {
+            case "getString" -> switch(String.valueOf(args[0]))
+            {
+               case "content_hash" -> TextChunks.hash("synthetic");
+               case "embedding" -> "[1,0,0]";
+               case "record_pk" -> "pk'; --";
+               case "field_name" -> "body";
+               default -> "article";
+            };
+            case "getInt" -> "chunk_index".equals(args[0]) ? 0 : 1;
+            case "getLong" -> 1L;
+            case "getDouble" -> 1.0;
+            default -> null;
+         });
+      }));
+   }
+
+
+
+   /*******************************************************************************
+    ** Exercise every statement with values that would be unsafe to interpolate.
+    *******************************************************************************/
+   @Test
+   void statementsAreBoundAndContentFree() throws Exception
+   {
+      PgVectorStore store = new PgVectorStore(this::connection, 3).withSchema("instance_one");
+      store.ensureSchema();
+      assertThat(sql).anyMatch(s -> s.contains("embedding vector(3)"));
+      assertThat(sql).anyMatch(s -> s.contains("USING hnsw (embedding vector_cosine_ops)"));
+      assertThat(sql).noneMatch(s -> s.contains("CREATE EXTENSION"));
+      ChunkVector chunk = new ChunkVector(1, "article", "pk'; --", "body", 0, TextChunks.hash("synthetic"), new float[] { 1, 0, 0 });
+      store.upsert(List.of(chunk));
+      store.replaceRecord(1, "article", "pk'; --", List.of(chunk));
+      assertThat(store.search(new float[] { 1, 0, 0 }, 8, Set.of("article", "injection'; --"))).hasSize(1);
+      assertThat(store.search(new float[] { 1, 0, 0 }, 8, Set.of("article"), Set.of(1))).hasSize(1);
+      assertThat(store.chunks(1)).singleElement().satisfies(v -> assertThat(v.embedding()).containsExactly(1, 0, 0));
+      assertThat(store.count(1)).isEqualTo(1);
+      store.deleteRecord(1, "article", "pk'; --");
+      store.deleteSource(1);
+      assertThat(sql).anyMatch(s -> s.contains("?::vector") && s.contains("ON CONFLICT"));
+      assertThat(sql).anyMatch(s -> s.contains("ORDER BY embedding <=> ?::vector LIMIT ?"));
+      assertThat(sql).noneMatch(s -> s.contains("pk'; --") || s.contains("injection'; --") || s.contains("synthetic"));
+      assertThat(parameters).anyMatch(p -> p.containsValue("pk'; --"));
+      assertThat(parameters).anyMatch(p -> p.containsValue("injection'; --"));
+      assertThat(operations).contains("commit", "close", "setAutoCommit");
+      assertThat(store.search(new float[] { 1, 0, 0 }, 8, Set.of())).isEmpty();
+      assertThat(store.search(new float[] { 1, 0, 0 }, 0, Set.of("article"))).isEmpty();
+      DataSource dataSource = (DataSource) Proxy.newProxyInstance(getClass().getClassLoader(), new Class<?>[] { DataSource.class }, (p, method, args) -> connection());
+      assertThat(new PgVectorStore(dataSource, 3).count(1)).isEqualTo(1);
+   }
+
+
+
+   /*******************************************************************************
+    ** DBA failure is actionable; failed replacement rolls back before closing.
+    *******************************************************************************/
+   @Test
+   void extensionAndRollback() throws Exception
+   {
+      extensionExists = false;
+      PgVectorStore store = new PgVectorStore(this::connection, 3);
+      store.ensureSchema();
+      assertThat(sql).contains("CREATE EXTENSION IF NOT EXISTS vector");
+      failExtension = true;
+      assertThatThrownBy(store::ensureSchema).hasMessageContaining("DBA");
+      failBatch = true;
+      assertThatThrownBy(() -> store.replaceRecord(1, "article", "1", List.of())).hasMessageContaining("vector store");
+      assertThat(operations).contains("rollback");
+      PgVectorStore down = new PgVectorStore(() ->
+      {
+         throw (new SQLException("offline"));
+      }, 3);
+      assertThatThrownBy(down::ensureSchema).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.count(1)).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.chunks(1)).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.upsert(List.of())).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.deleteRecord(1, "a", "1")).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.deleteSource(1)).hasMessageContaining("vector store");
+      assertThatThrownBy(() -> down.search(new float[] { 1, 0, 0 }, 8, Set.of("a"))).hasMessageContaining("vector store");
+   }
+
+
+
+   /*******************************************************************************
+    ** Identifier and coordinate validation happens before opening a connection.
+    *******************************************************************************/
+   @Test
+   void invalidInputsNeverReachJdbc()
+   {
+      PgVectorStore store = new PgVectorStore(this::connection, 3);
+      for(String name : List.of("a.b", "bad; DROP SCHEMA public", "UPPER", "a".repeat(64)))
+      {
+         assertThatThrownBy(() -> store.withSchema(name)).isInstanceOf(IllegalArgumentException.class);
+      }
+      assertThatThrownBy(() -> new PgVectorStore(this::connection, 2001)).isInstanceOf(IllegalArgumentException.class);
+      assertThatThrownBy(() -> store.search(new float[] { 1 }, 1, Set.of("a"))).isInstanceOf(IllegalArgumentException.class);
+      assertThatThrownBy(() -> store.search(new float[] { Float.NaN, 0, 0 }, 1, Set.of("a"))).isInstanceOf(IllegalArgumentException.class);
+      assertThat(operations).isEmpty();
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantQBitConfig.java

@@ -17,7 +17,6 @@
 package com.kingsrook.qbits.assistant;
 
 
-import javax.net.ssl.SSLContext;
 import java.io.IOException;
 import java.nio.file.Files;
 import java.nio.file.Path;
@@ -29,6 +28,7 @@ import java.util.Map;
 import java.util.Set;
 import java.util.function.Function;
 import java.util.regex.Pattern;
+import javax.net.ssl.SSLContext;
 import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.kingsrook.qbits.assistant.classification.ClassificationSource;
 import com.kingsrook.qbits.assistant.classification.ConfiguredClassificationSource;
@@ -37,6 +37,7 @@ import com.kingsrook.qbits.assistant.engine.ModelPricing;
 import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
 import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
 import com.kingsrook.qbits.assistant.provider.TrustedCaBundle;
+import com.kingsrook.qbits.assistant.rag.VectorStore;
 import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
 import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
@@ -59,6 +60,8 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
  *******************************************************************************/
 public class AssistantQBitConfig implements QBitConfig
 {
+   private VectorStore vectorStore;
+
    public static final String ACCESS_PERMISSION        = "assistant.hasAccess";
    public static final String ADMIN_PERMISSION         = "assistant.admin";
    public static final String DEFAULT_OWNER_KEY_TYPE   = "assistant_user_id";
@@ -130,6 +133,7 @@ public class AssistantQBitConfig implements QBitConfig
    //////////////////////////////////////////////////////////////////////////
    private String                    uiApiBaseUrl;
    private UiApiGateway              uiApiGateway;
+   private com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadRunner bulkLoadRunner;
    private Integer                   pendingActionTtlSeconds  = 900;
    private List<String>              customRedactionPatterns  = new ArrayList<>();
    private Integer                   killSwitchPollMillis     = 2000;
@@ -1199,6 +1203,38 @@ public class AssistantQBitConfig implements QBitConfig
 
 
 
+   /*******************************************************************************
+    ** Getter for bulkLoadRunner (null: the durable jobs launch,
+    ** JobsLaunchBulkLoadRunner) - how bulk_load_run starts a load.
+    *******************************************************************************/
+   public com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadRunner getBulkLoadRunner()
+   {
+      return (this.bulkLoadRunner);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for bulkLoadRunner
+    *******************************************************************************/
+   public void setBulkLoadRunner(com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadRunner bulkLoadRunner)
+   {
+      this.bulkLoadRunner = bulkLoadRunner;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for bulkLoadRunner
+    *******************************************************************************/
+   public AssistantQBitConfig withBulkLoadRunner(com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadRunner bulkLoadRunner)
+   {
+      this.bulkLoadRunner = bulkLoadRunner;
+      return (this);
+   }
+
+
+
    /*******************************************************************************
     ** Getter for uiApiGateway (null: the loopback stand-in)
     *******************************************************************************/
@@ -1788,4 +1824,35 @@ public class AssistantQBitConfig implements QBitConfig
       configuredClassifications.withClassification(tableName, fieldName, classification);
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for vectorStore
+    *******************************************************************************/
+   public VectorStore getVectorStore()
+   {
+      return (vectorStore);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for vectorStore
+    *******************************************************************************/
+   public void setVectorStore(VectorStore vectorStore)
+   {
+      this.vectorStore = vectorStore;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for vectorStore
+    *******************************************************************************/
+   public AssistantQBitConfig withVectorStore(VectorStore vectorStore)
+   {
+      this.vectorStore = vectorStore;
+      return (this);
+   }
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantQBitProducer.java

@@ -26,6 +26,7 @@ import com.kingsrook.qbits.assistant.home.AssistantHome;
 import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
 import com.kingsrook.qbits.assistant.metadata.AssistantRecordScreens;
 import com.kingsrook.qbits.assistant.processes.DiscoverModelsProcess;
+import com.kingsrook.qbits.assistant.processes.KnowledgeIndexProcess;
 import com.kingsrook.qbits.assistant.processes.TestConnectionProcess;
 import com.kingsrook.qbits.assistant.processes.TestModelConnectionProcess;
 import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
@@ -179,6 +180,12 @@ public class AssistantQBitProducer implements QBitMetaDataProducer<AssistantQBit
          qInstance.addProcess(process);
       }
 
+      for(QProcessMetaData process : KnowledgeIndexProcess.processes())
+      {
+         process.setSourceQBitName(qBitMetaData.getName());
+         qInstance.addProcess(process);
+      }
+
       qInstance.addWidget(AssistantHome.widget());
       qInstance.addWidget(AssistantAccessWidget.widget());
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/AssistantRuntime.java

@@ -17,6 +17,12 @@
 package com.kingsrook.qbits.assistant;
 
 
+import java.time.Clock;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.regex.Pattern;
 import com.kingsrook.qbits.assistant.access.AccessExplainer;
 import com.kingsrook.qbits.assistant.access.DataAccessService;
 import com.kingsrook.qbits.assistant.engine.BudgetService;
@@ -27,6 +33,11 @@ import com.kingsrook.qbits.assistant.engine.ProviderService;
 import com.kingsrook.qbits.assistant.engine.RateLimiter;
 import com.kingsrook.qbits.assistant.engine.RunningTurn;
 import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qbits.assistant.rag.InMemoryVectorStore;
+import com.kingsrook.qbits.assistant.rag.RagService;
+import com.kingsrook.qbits.assistant.rag.VectorStore;
+import com.kingsrook.qbits.assistant.tools.KnowledgeTool;
+import com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadTools;
 import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
 import com.kingsrook.qbits.assistant.uiapi.LoopbackUiApiGateway;
 import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
@@ -37,12 +48,6 @@ import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
-import java.time.Clock;
-import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.regex.Pattern;
 
 
 /*******************************************************************************
@@ -52,6 +57,10 @@ import java.util.regex.Pattern;
  *******************************************************************************/
 public class AssistantRuntime
 {
+   private final RagService ragService;
+   private final KnowledgeTool knowledgeTool;
+   private final BulkLoadTools bulkLoadTools;
+
    private final QInstance                        qInstance;
    private final AssistantQBitConfig              config;
    private final Clock                            clock;
@@ -106,7 +115,11 @@ public class AssistantRuntime
       this.toolbox = new UiApiToolbox(qInstance, dataAccessService, exposurePolicy, new AiRedactor(settings.getCustomRedactionPatterns()),
          config.getMaxCellChars() == null ? AiToolKernelSettings.DEFAULT_MAX_CELL_CHARS : config.getMaxCellChars());
       this.uiApiGateway = config.getUiApiGateway() != null ? config.getUiApiGateway() : new LoopbackUiApiGateway(this::uiApiBaseUrl, qInstance);
-      this.contextBuilder = new ContextBuilder(config, kernel, toolbox);
+      VectorStore vectors = config.getVectorStore() == null ? new InMemoryVectorStore() : config.getVectorStore();
+      this.ragService = new RagService(this, vectors);
+      this.knowledgeTool = new KnowledgeTool(this, ragService, vectors);
+      this.bulkLoadTools = new BulkLoadTools(qInstance, dataAccessService, config.getBulkLoadRunner());
+      this.contextBuilder = new ContextBuilder(config, kernel, toolbox).withKnowledgeTool(knowledgeTool).withBulkLoadTools(bulkLoadTools);
    }
 
 
@@ -417,4 +430,34 @@ public class AssistantRuntime
    {
       return (budgetService);
    }
+
+
+   /*******************************************************************************
+    ** Manual indexing service, always using the current caller's session.
+    *******************************************************************************/
+   public RagService getRagService()
+   {
+      return (ragService);
+   }
+
+
+
+   /*******************************************************************************
+    ** Permission-filtered retrieval tool.
+    *******************************************************************************/
+   public KnowledgeTool getKnowledgeTool()
+   {
+      return (knowledgeTool);
+   }
+
+
+
+   /*******************************************************************************
+    ** The bulk-load tools (saved profiles, attached files, dry runs, loads).
+    *******************************************************************************/
+   public BulkLoadTools getBulkLoadTools()
+   {
+      return (bulkLoadTools);
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/api/AssistantApi.java

@@ -46,6 +46,7 @@ import com.kingsrook.qbits.assistant.model.AssistantTurn;
 import com.kingsrook.qbits.assistant.model.AssistantTurnEvent;
 import com.kingsrook.qbits.assistant.persistence.AssistantStore;
 import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
+import com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadTools;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
 import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
@@ -598,7 +599,9 @@ public class AssistantApi
       }
       PolicyService.EffectivePolicy policy = runtime.getPolicyService().current(true);
       UiApiToolbox.Context context = new UiApiToolbox.Context(session.getUuid(), session.getValue(QSession.VALUE_KEY_USER_TIMEZONE), runtime.getUiApiGateway(), policy.tools(), null);
-      UiApiToolbox.Outcome outcome = runtime.getToolbox().executeConfirmedWrite(context, action.getToolCallId(), action.getToolName(), action.getTableName(), action.getRecordId(), values);
+      UiApiToolbox.Outcome outcome = BulkLoadTools.WRITE_TOOLS.contains(action.getToolName())
+         ? runtime.getBulkLoadTools().executeConfirmed(context, action.getToolCallId(), action.getToolName(), action.getTableName(), values)
+         : runtime.getToolbox().executeConfirmedWrite(context, action.getToolCallId(), action.getToolName(), action.getTableName(), action.getRecordId(), values);
       AiToolResult result = outcome.getResult();
       boolean ok = result.getOutcome() == AiToolOutcome.OK;
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ConnectionTest.java

@@ -21,6 +21,7 @@ import java.time.Duration;
 import java.util.ArrayList;
 import java.util.List;
 import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.classification.DataClassification;
 import com.kingsrook.qbits.assistant.model.AssistantModel;
 import com.kingsrook.qbits.assistant.model.AssistantProvider;
 import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
@@ -31,6 +32,7 @@ import com.kingsrook.qbits.assistant.provider.ModelRequest;
 import com.kingsrook.qbits.assistant.provider.ModelResponse;
 import com.kingsrook.qbits.assistant.provider.RegistryModelProvider;
 import com.kingsrook.qbits.assistant.provider.StreamHandle;
+import com.kingsrook.qbits.assistant.rag.EmbeddingService;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 
@@ -209,10 +211,17 @@ public final class ConnectionTest
          {
             throw (new ProviderService.ProviderSetupException("The model's provider no longer exists."));
          }
-         AssistantProviderType type = runtime.getConfig().getProviderTypes().get(row.getProviderType());
-         var settings = runtime.getProviderService().settings(row);
-         var provider = new RegistryModelProvider(type.create(settings), model);
-         result = call(runtime, new ProviderService.Active(row, row.getProviderType(), model.getModelId(), provider), userId);
+         if(!model.isChat())
+         {
+            result = testEmbedding(runtime, model);
+         }
+         else
+         {
+            AssistantProviderType type = runtime.getConfig().getProviderTypes().get(row.getProviderType());
+            var settings = runtime.getProviderService().settings(row);
+            var provider = new RegistryModelProvider(type.create(settings), model);
+            result = call(runtime, new ProviderService.Active(row, row.getProviderType(), model.getModelId(), provider), userId);
+         }
       }
       catch(ProviderService.ProviderSetupException e)
       {
@@ -226,6 +235,27 @@ public final class ConnectionTest
 
 
 
+   /*******************************************************************************
+    ** An embedding model is tested with embeddings, never a chat completion.
+    ** The normal embedding budget, eligibility and usage rules still apply.
+    *******************************************************************************/
+   private static Result testEmbedding(AssistantRuntime runtime, AssistantModel model)
+   {
+      Long started = System.nanoTime();
+      try
+      {
+         new EmbeddingService(runtime).embed(model.getId(), List.of("Connection test."),
+            DataClassification.PUBLIC, "", null, null);
+         return (new Result(STATUS_OK, (int) ((System.nanoTime() - started) / 1_000_000), model.getModelId(), null));
+      }
+      catch(QException e)
+      {
+         return (new Result(STATUS_ERROR, null, model.getModelId(), e.getMessage()));
+      }
+   }
+
+
+
 
    /*******************************************************************************
     ** Map the provider error category.

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ContextBuilder.java

@@ -34,10 +34,13 @@ import com.kingsrook.qbits.assistant.provider.ModelPart;
 import com.kingsrook.qbits.assistant.provider.ModelRequest;
 import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
 import com.kingsrook.qbits.assistant.provider.SystemBlock;
+import com.kingsrook.qbits.assistant.tools.KnowledgeTool;
 import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
+import com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadTools;
 import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolDefinition;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolKernel;
+import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
@@ -64,11 +67,14 @@ public class ContextBuilder
 
    public static final int MAX_CONTEXT_CHARS = 512;
    public static final int MAX_SELECTED_IDS  = 100;
+   public static final int MAX_ATTACHMENTS   = 5;
 
    private final AssistantQBitConfig config;
    private final AiToolKernel        kernel;
    private final UiApiToolbox        toolbox;
    private volatile String           appDigest;
+   private KnowledgeTool knowledgeTool;
+   private BulkLoadTools bulkLoadTools;
 
 
 
@@ -117,7 +123,18 @@ public class ContextBuilder
                .withStrict(isStrict(definition.getName(), !Boolean.FALSE.equals(definition.getStrict()))));
          }
       }
+      if(knowledgeTool != null && knowledgeTool.offered(policy, QContext.getQSession()))
+      {
+         byName.put(KnowledgeTool.NAME, knowledgeTool.definition());
+      }
       byName.put(PresentResultsTool.NAME, PresentResultsTool.definition(isStrict(PresentResultsTool.NAME, true)));
+      if(bulkLoadTools != null && bulkLoadTools.offered())
+      {
+         for(ModelToolDefinition definition : bulkLoadTools.definitions(policy, toolbox.anyWritableTable()))
+         {
+            byName.put(definition.getName(), definition);
+         }
+      }
       return (new ArrayList<>(byName.values()));
    }
 
@@ -345,6 +362,32 @@ public class ContextBuilder
          }
       }
 
+      ////////////////////////////////////////////////////////////////////
+      // files the user attached in the panel (already uploaded through  //
+      // the UI API as the user): only their opaque refs and names       //
+      ////////////////////////////////////////////////////////////////////
+      if(raw.get("attachments") instanceof List<?> attachments)
+      {
+         List<Map<String, Object>> kept = new ArrayList<>();
+         for(Object attachment : attachments)
+         {
+            if(attachment instanceof Map<?, ?> a && scalar(a.get("uploadRef")) != null && kept.size() < MAX_ATTACHMENTS)
+            {
+               Map<String, Object> one = new LinkedHashMap<>();
+               one.put("uploadRef", scalar(a.get("uploadRef")));
+               if(scalar(a.get("fileName")) != null)
+               {
+                  one.put("fileName", scalar(a.get("fileName")));
+               }
+               kept.add(one);
+            }
+         }
+         if(!kept.isEmpty())
+         {
+            clean.put("attachments", kept);
+         }
+      }
+
       if(JsonUtils.toJson(clean).length() > 4000)
       {
          clean.remove("filter");
@@ -372,4 +415,26 @@ public class ContextBuilder
       return (null);
    }
 
+
+
+   /*******************************************************************************
+    ** Additive optional tool hook; UI API toolbox stays independently maintained.
+    *******************************************************************************/
+   public ContextBuilder withKnowledgeTool(KnowledgeTool knowledgeTool)
+   {
+      this.knowledgeTool = knowledgeTool;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for the bulk-load tools.
+    *******************************************************************************/
+   public ContextBuilder withBulkLoadTools(BulkLoadTools bulkLoadTools)
+   {
+      this.bulkLoadTools = bulkLoadTools;
+      return (this);
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ModelDiscovery.java

@@ -85,6 +85,7 @@ public final class ModelDiscovery
                }
                ModelCapabilities caps = provider.getCapabilities(model.modelId());
                runtime.getStore().insert(new AssistantModel().withProviderId(row.getId()).withModelId(model.modelId())
+                  .withModelKind(embedding(model.modelId()) ? "EMBEDDING" : "CHAT").withEmbeddingDimensions(dimensions(model.modelId()))
                   .withDisplayName(model.displayName()).withEnabled(local || discovered.size() <= 20)
                   .withSource("DISCOVERED").withDiscoveredAt(runtime.getClock().instant())
                   .withSupportsTools(caps.getTools()).withSupportsVision(caps.getVision()).withSupportsStreaming(caps.getStreaming())
@@ -115,4 +116,33 @@ public final class ModelDiscovery
          return (result);
       }
    }
+
+
+   /*******************************************************************************
+    ** Discovery is a suggestion; an administrator can correct the kind.
+    *******************************************************************************/
+   public static Boolean embedding(String id)
+   {
+      String lower = id.toLowerCase(java.util.Locale.ROOT);
+      return (List.of("embed", "bge", "e5", "gte", "nomic").stream().anyMatch(lower::contains));
+   }
+
+
+
+   /*******************************************************************************
+    ** Known local defaults; other models must declare their dimensions.
+    *******************************************************************************/
+   private static Integer dimensions(String id)
+   {
+      if(id.contains("bge-m3"))
+      {
+         return (1024);
+      }
+      if(id.contains("nomic-embed-text"))
+      {
+         return (768);
+      }
+      return (null);
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ProviderService.java

@@ -186,6 +186,16 @@ public class ProviderService
     ** with a message for the admin when the row cannot be used.
     *******************************************************************************/
    public Active forRow(AssistantProvider row, String fallbackModel) throws ProviderSetupException
+   {
+      return (forRow(row, fallbackModel, false));
+   }
+
+
+
+   /*******************************************************************************
+    ** Resolve only the requested kind; embedding calls never select chat models.
+    *******************************************************************************/
+   private Active forRow(AssistantProvider row, String fallbackModel, Boolean embedding) throws ProviderSetupException
    {
       if(Boolean.TRUE.equals(config.getHipaaMode()) && !Boolean.TRUE.equals(row.getHipaaEligible()))
       {
@@ -205,7 +215,7 @@ public class ProviderService
       String model = StringUtils.hasContent(row.getModel()) ? row.getModel() : null;
       if(model == null && !models.isEmpty())
       {
-         model = models.stream().filter(m -> Boolean.TRUE.equals(m.getEnabled())).map(AssistantModel::getModelId).findFirst().orElse(null);
+         model = models.stream().filter(m -> Boolean.TRUE.equals(m.getEnabled()) && !m.isChat().equals(embedding)).map(AssistantModel::getModelId).findFirst().orElse(null);
          if(model == null)
          {
             throw (new ProviderSetupException("All models of this provider are disabled. Enable a model first."));
@@ -217,7 +227,7 @@ public class ProviderService
       }
       String selected = model;
       AssistantModel registry = models.stream().filter(m -> java.util.Objects.equals(selected, m.getModelId())).findFirst().orElse(null);
-      if(registry != null && !Boolean.TRUE.equals(registry.getEnabled()))
+      if(registry != null && (!Boolean.TRUE.equals(registry.getEnabled()) || registry.isChat().equals(embedding)))
       {
          throw (new ProviderSetupException("The selected model is disabled in this provider's model registry."));
       }
@@ -752,7 +762,7 @@ public class ProviderService
    public Active resolveModel(Integer modelId, Boolean requiresHipaa, Boolean requireTools) throws QException
    {
       AssistantModel model = store.listModels().stream().filter(m -> modelId.equals(m.getId()) && Boolean.TRUE.equals(m.getEnabled())).findFirst().orElse(null);
-      if(model == null)
+      if(model == null || !model.isChat())
       {
          return (null);
       }
@@ -816,4 +826,28 @@ public class ProviderService
    {
       return (left != null && right != null && java.util.Objects.equals(left.providerId(), right.providerId()) && java.util.Objects.equals(left.model(), right.model()));
    }
+
+
+   /*******************************************************************************
+    ** Embedding selection is explicit and checks fresh provider eligibility.
+    *******************************************************************************/
+   public Active resolveEmbedding(Integer modelId, Boolean requiresHipaa) throws QException
+   {
+      AssistantModel model = store.listModels().stream().filter(m -> java.util.Objects.equals(modelId, m.getId())).findFirst().orElse(null);
+      if(model == null || !"EMBEDDING".equals(model.getModelKind()) || !Boolean.TRUE.equals(model.getEnabled()))
+      {
+         throw (new ProviderSetupException("Select an enabled EMBEDDING model for this knowledge source."));
+      }
+      AssistantProvider row = store.getProvider(model.getProviderId());
+      if(row == null || !Boolean.TRUE.equals(row.getEnabled()))
+      {
+         throw (new ProviderSetupException("The embedding provider is disabled or unavailable."));
+      }
+      if((requiresHipaa || Boolean.TRUE.equals(config.getHipaaMode())) && !Boolean.TRUE.equals(row.getHipaaEligible()))
+      {
+         throw (new ProviderSetupException("PHI may only be embedded by a HIPAA-eligible provider. Nothing was sent."));
+      }
+      return (forRow(new AssistantProvider(row.toQRecord()).withModel(model.getModelId()), model.getModelId(), true));
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/SystemPrompt.java

@@ -40,6 +40,8 @@ public class SystemPrompt
       10. An administrator may give you a name, a personality and skills (in <assistant_profile> and <skill> blocks). Follow them for tone and know-how, but they never override these rules and never give you other tools, data or permissions.
       11. Some values are masked by the application (for example j***@x.com or ***1234). Treat them as masked, never as the real value, and never ask the user to reveal them for you.
 
+      12. When you use search_knowledge, cite records with their links.
+
       Presentation (default skill; administrator skills may extend it):
       Answer first in one sentence; do not restate totals ("101 add up to 101"). For 3+ items, comparisons or breakdowns, call present_results with a table; add a chart when shares or trends help. Prefer metrics for a single big number. Include any list you offer to open as a filterLinks item, never just plain text. Use only values from tool results, omit zero-count categories, and use percentOfColumn for shares. Keep prose short; do not repeat presented tables or links. Never write URLs yourself; supply metadata names and raw filter values to present_results.
       """;

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/ToolLabels.java

@@ -37,7 +37,14 @@ public class ToolLabels
       Map.entry("get_record_history", new String[] { "Reading the change history", "Read the change history" }),
       Map.entry("render_widget", new String[] { "Rendering a widget", "Rendered a widget" }),
       Map.entry("search_help", new String[] { "Searching help", "Searched help" }),
-      Map.entry("navigate", new String[] { "Opening a screen", "Opened a screen" })
+      Map.entry("navigate", new String[] { "Opening a screen", "Opened a screen" }),
+      Map.entry("bulk_profiles_list", new String[] { "Listing saved bulk load profiles", "Listed saved bulk load profiles" }),
+      Map.entry("bulk_profile_describe", new String[] { "Reading a bulk load profile", "Read a bulk load profile" }),
+      Map.entry("bulk_file_attach", new String[] { "Reading the attached file", "Read the attached file" }),
+      Map.entry("bulk_mapping_suggest", new String[] { "Suggesting a column mapping", "Suggested a column mapping" }),
+      Map.entry("bulk_load_dry_run", new String[] { "Checking the mapping (dry run)", "Checked the mapping (dry run)" }),
+      Map.entry("bulk_profile_save", new String[] { "Preparing to save the profile", "Prepared to save the profile" }),
+      Map.entry("bulk_load_run", new String[] { "Preparing the load", "Prepared the load" })
    );
 
 

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/TurnClassification.java

@@ -75,7 +75,7 @@ public class TurnClassification
          return;
       }
       Map<String, Object> input = use.getInput() == null ? Map.of() : use.getInput();
-      boolean crossTable = "search_records".equals(use.getToolName()) || "get_record_history".equals(use.getToolName());
+      boolean crossTable = "search_knowledge".equals(use.getToolName()) || "search_records".equals(use.getToolName()) || "get_record_history".equals(use.getToolName());
       String table = result.getTableName() != null ? result.getTableName()
          : (input.get("table") instanceof String name ? name : (input.get("tableName") instanceof String name ? name : null));
       if(crossTable)
@@ -98,6 +98,10 @@ public class TurnClassification
          }
          collectTables(result.getRender(), touched);
          collectResultTables(result.getModelContent(), touched);
+         if("search_knowledge".equals(use.getToolName()) && touched.isEmpty())
+         {
+            return;
+         }
          if(touched.isEmpty() || touched.stream().anyMatch(name -> QContext.getQInstance().getTable(name) == null))
          {
             touchUnknown();

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/TurnEngine.java

@@ -17,6 +17,21 @@
 package com.kingsrook.qbits.assistant.engine;
 
 
+import java.io.Serializable;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
 import com.kingsrook.qbits.assistant.AssistantRuntime;
 import com.kingsrook.qbits.assistant.model.AssistantConversation;
@@ -36,7 +51,9 @@ import com.kingsrook.qbits.assistant.provider.ModelRequest;
 import com.kingsrook.qbits.assistant.provider.ModelResponse;
 import com.kingsrook.qbits.assistant.provider.ModelUsage;
 import com.kingsrook.qbits.assistant.provider.StreamHandle;
+import com.kingsrook.qbits.assistant.tools.KnowledgeTool;
 import com.kingsrook.qbits.assistant.tools.PresentResultsTool;
+import com.kingsrook.qbits.assistant.tools.bulkload.BulkLoadTools;
 import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
 import com.kingsrook.qqq.backend.core.actions.ai.AiRecordRef;
 import com.kingsrook.qqq.backend.core.actions.ai.AiToolCall;
@@ -50,21 +67,6 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
 import com.kingsrook.qqq.backend.core.utils.StringUtils;
-import java.io.Serializable;
-import java.time.Duration;
-import java.time.Instant;
-import java.time.LocalDate;
-import java.time.ZoneOffset;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.Future;
-import java.util.concurrent.Semaphore;
-import java.util.concurrent.TimeUnit;
 import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
 
 
@@ -711,8 +713,20 @@ public class TurnEngine implements Runnable
                         QContext.init(runtime.getQInstance(), session);
                         try
                         {
-                           outcomes[index] = runtime.getToolbox().execute(toolContext(), call.getId(), call.getName(), call.getInput());
-                           results[index] = outcomes[index].getResult();
+                           if(KnowledgeTool.NAME.equals(call.getName()))
+                           {
+                              results[index] = runtime.getKnowledgeTool().execute(toolContext(), call.getId(), call.getInput(), classification.getClassification(), conversation.getId(), turn.getId());
+                           }
+                           else if(BulkLoadTools.NAMES.contains(call.getName()))
+                           {
+                              outcomes[index] = runtime.getBulkLoadTools().execute(toolContext(), call.getId(), call.getName(), call.getInput(), hipaaRoute());
+                              results[index] = outcomes[index].getResult();
+                           }
+                           else
+                           {
+                              outcomes[index] = runtime.getToolbox().execute(toolContext(), call.getId(), call.getName(), call.getInput());
+                              results[index] = outcomes[index].getResult();
+                           }
                         }
                         finally
                         {
@@ -848,9 +862,9 @@ public class TurnEngine implements Runnable
             .withRowCount(result.getRowCount())
             .withDurationMs(result.getDurationMs())
             .withResultSummary(summary)
-            .withResultJson(truncate(result.getModelContent(), MAX_RESULT_JSON_CHARS))
+            .withResultJson(truncate(BulkLoadTools.NAMES.contains(use.getToolName()) ? BulkLoadTools.forLog(result.getModelContent()) : result.getModelContent(), MAX_RESULT_JSON_CHARS))
             .withRedactionsJson(result.getRedactions() == null || result.getRedactions().isEmpty() ? null : JsonUtils.toJson(result.getRedactions()))
-            .withUiApiOperation(extras == null ? null : extras.getOperation())
+            .withUiApiOperation(extras == null ? ("search_knowledge".equals(use.getToolName()) ? "GET /table/{tableName}/{primaryKey}" : null) : extras.getOperation())
             .withHttpStatus(extras == null ? null : extras.getHttpStatus())
             .withAuditHref(extras == null ? null : extras.getAuditHref())
             .withPendingActionId(extras == null ? null : extras.getPendingActionId()));
@@ -868,6 +882,21 @@ public class TurnEngine implements Runnable
     ** API), the gateway, the policy's tool settings, and the pending-action
     ** store for writes that need confirmation.
     *******************************************************************************/
+   private boolean hipaaRoute()
+   {
+      ///////////////////////////////////////////////////////////////////////
+      // the X-QQQ-AI-Route the bulk-load tools send: the turn's provider's //
+      // HIPAA eligibility, never anything the model says                  //
+      ///////////////////////////////////////////////////////////////////////
+      return (active != null && active.row() != null && Boolean.TRUE.equals(active.row().getHipaaEligible()));
+   }
+
+
+
+   /*******************************************************************************
+    ** The UI API tool context: the user's session, the gateway, the policy's
+    ** tools and the pending-action hook.
+    *******************************************************************************/
    private UiApiToolbox.Context toolContext()
    {
       return (new UiApiToolbox.Context(session.getUuid(), userTimezone(), runtime.getUiApiGateway(), policy.tools(), this::proposeWrite));
@@ -923,7 +952,7 @@ public class TurnEngine implements Runnable
    /*******************************************************************************
     ** Only same-origin application links are passed on.
     *******************************************************************************/
-   static boolean isAppHref(String href)
+   public static boolean isAppHref(String href)
    {
       return (href != null && (href.equals("/app") || href.startsWith("/app/")) && !href.contains("//") && !href.contains("\\"));
    }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/engine/UsageRecorder.java

@@ -39,6 +39,7 @@ public final class UsageRecorder
 
    public static final String PURPOSE_TURN = "TURN";
    public static final String PURPOSE_TEST = "TEST";
+   public static final String PURPOSE_EMBED = "EMBED";
    public static final String PURPOSE_ROUTE = "ROUTE";
 
    public static final String STATUS_OK        = "OK";

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantMetaDataBuilder.java

@@ -30,6 +30,7 @@ import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
 import com.kingsrook.qbits.assistant.model.AssistantFeedback;
 import com.kingsrook.qbits.assistant.model.AssistantMessage;
 import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
 import com.kingsrook.qbits.assistant.model.AssistantModelPrice;
 import com.kingsrook.qbits.assistant.model.AssistantPendingAction;
 import com.kingsrook.qbits.assistant.model.AssistantPolicy;
@@ -110,7 +111,7 @@ public class AssistantMetaDataBuilder
     ** calls on a conversation; model calls on a provider) or reached from the
     ** home. Processes are table and record actions, never nav entries.
     *******************************************************************************/
-   public static final List<String> NAV_TABLES = List.of(AssistantBudget.TABLE_NAME, AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
+   public static final List<String> NAV_TABLES = List.of(AssistantRagSource.TABLE_NAME, AssistantBudget.TABLE_NAME, AssistantProvider.TABLE_NAME, AssistantProfile.TABLE_NAME, AssistantSkill.TABLE_NAME, AssistantDataAccess.TABLE_NAME, AssistantConversation.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantPolicy.TABLE_NAME);
 
    private final AssistantQBitConfig config;
 
@@ -186,6 +187,13 @@ public class AssistantMetaDataBuilder
          .withUniqueKey(new UniqueKey("name"))
          .withRecordLabelFormat("%s").withRecordLabelFields("name"));
 
+      QCodeReference ragCustomizer = new QCodeReference(AssistantRagSourceCustomizer.class);
+      tables.add(text(base(AssistantRagSource.TABLE_NAME, "Knowledge sources", "manage_search", AssistantRagSource.class), "filterJson")
+         .withRecordLabelFormat("%s").withRecordLabelFields("name")
+         .withCustomizer(TableCustomizers.PRE_INSERT_RECORD, ragCustomizer).withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, ragCustomizer)
+         .withCustomizer(TableCustomizers.POST_INSERT_RECORD, ragCustomizer).withCustomizer(TableCustomizers.POST_UPDATE_RECORD, ragCustomizer)
+         .withCustomizer(TableCustomizers.POST_DELETE_RECORD, ragCustomizer));
+
       tables.add(settings(base(AssistantModel.TABLE_NAME, "Assistant Model", "memory", AssistantModel.class))
          .withUniqueKey(new UniqueKey("providerId", "modelId"))
          .withRecordLabelFormat("%s").withRecordLabelFields("modelId"));

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantRagSourceCustomizer.java

@@ -0,0 +1,125 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.metadata;
+
+
+import java.io.Serializable;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
+
+
+/*******************************************************************************
+ ** Validate source configuration; invalidate vectors whenever their meaning changes.
+ *******************************************************************************/
+public class AssistantRagSourceCustomizer implements TableCustomizerInterface
+{
+   private static final List<String> SEMANTIC_FIELDS = List.of("tableName", "fieldNames", "filterJson", "embeddingModelId", "chunkChars", "chunkOverlapChars");
+
+
+
+   /*******************************************************************************
+    ** Validate merged edits, not partial update payloads. Status writes remain
+    ** possible even after a host removes a table or disables a model.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean isPreview, Optional<List<QRecord>> oldRecordList) throws QException
+   {
+      Optional<Map<Serializable, QRecord>> old = oldRecordListToMap("id", oldRecordList);
+      AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
+      for(QRecord record : records)
+      {
+         if(record.getValues().keySet().stream().allMatch(name -> name.equals("id") || name.startsWith("lastIndex") || name.startsWith("indexed") || name.endsWith("Date")))
+         {
+            continue;
+         }
+         QRecord merged = new QRecord();
+         old.map(m -> m.get(record.getValue("id"))).ifPresent(o -> merged.setValues(new LinkedHashMap<>(o.getValues())));
+         record.getValues().forEach(merged::setValue);
+         for(var entry : Map.<String, Serializable>of("enabled", true, "chunkChars", 1500, "chunkOverlapChars", 200).entrySet())
+         {
+            if(merged.getValue(entry.getKey()) == null)
+            {
+               record.setValue(entry.getKey(), entry.getValue());
+               merged.setValue(entry.getKey(), entry.getValue());
+            }
+         }
+         if(runtime != null)
+         {
+            for(String error : runtime.getRagService().validate(new AssistantRagSource(merged)))
+            {
+               record.addError(new BadInputStatusMessage(error));
+            }
+         }
+      }
+      return (records);
+   }
+
+
+
+   /*******************************************************************************
+    ** Old vectors cannot survive a model, filter or chunk-layout change.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> postInsertOrUpdate(AbstractActionInput input, List<QRecord> records, Optional<List<QRecord>> oldRecordList) throws QException
+   {
+      Optional<Map<Serializable, QRecord>> old = oldRecordListToMap("id", oldRecordList);
+      AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
+      if(runtime != null && old.isPresent())
+      {
+         for(QRecord record : records)
+         {
+            QRecord previous = old.get().get(record.getValue("id"));
+            if(previous != null && SEMANTIC_FIELDS.stream().anyMatch(name -> record.getValues().containsKey(name) && !Objects.equals(previous.getValue(name), record.getValue(name))))
+            {
+               runtime.getRagService().clear(new AssistantRagSource(previous));
+            }
+         }
+      }
+      return (records);
+   }
+
+
+
+   /*******************************************************************************
+    ** Deleted sources no longer have searchable vectors.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> postDelete(DeleteInput input, List<QRecord> records) throws QException
+   {
+      AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
+      if(runtime != null)
+      {
+         for(QRecord record : records)
+         {
+            runtime.getRagService().remove(new AssistantRagSource(record));
+         }
+      }
+      return (records);
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantRecordScreens.java

@@ -17,6 +17,11 @@
 package com.kingsrook.qbits.assistant.metadata;
 
 
+import java.util.ArrayList;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
 import com.kingsrook.qbits.assistant.AssistantModelRouting;
 import com.kingsrook.qbits.assistant.AssistantQBitConfig;
 import com.kingsrook.qbits.assistant.access.DataAccessMode;
@@ -37,6 +42,7 @@ import com.kingsrook.qbits.assistant.model.AssistantPolicy;
 import com.kingsrook.qbits.assistant.model.AssistantProfile;
 import com.kingsrook.qbits.assistant.model.AssistantProfileSkill;
 import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
 import com.kingsrook.qbits.assistant.model.AssistantSkill;
 import com.kingsrook.qbits.assistant.model.AssistantToolCall;
 import com.kingsrook.qbits.assistant.model.AssistantTurn;
@@ -47,6 +53,9 @@ import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
 import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
 import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceWithProperties;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.AdornmentType;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.DisplayFormat;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.FieldAdornment;
@@ -64,10 +73,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleVal
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier;
-import java.util.ArrayList;
-import java.util.LinkedHashSet;
-import java.util.List;
-import java.util.Set;
 
 
 /*******************************************************************************
@@ -184,6 +189,7 @@ public final class AssistantRecordScreens
          case AssistantUsageDaily.TABLE_NAME -> usageDaily(table);
          case AssistantProvider.TABLE_NAME -> provider(table);
          case AssistantModel.TABLE_NAME -> model(table);
+         case AssistantRagSource.TABLE_NAME -> knowledge(table);
          case AssistantModelPrice.TABLE_NAME -> price(table);
          case AssistantUsage.TABLE_NAME -> usage(table);
          case AssistantDataAccess.TABLE_NAME -> dataAccess(table);
@@ -286,17 +292,30 @@ public final class AssistantRecordScreens
       return (List.of(
          new QPossibleValueSource().withName(PROVIDER_TYPE_PVS_NAME).withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING)
             .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY).withEnumValues(types),
-         new QPossibleValueSource().withName(MODEL_PVS_NAME).withType(QPossibleValueSourceType.ENUM).withIdType(QFieldType.STRING)
-            .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY).withEnumValues(modelValues),
-         enumSource("assistant_provider_kind", kinds),
+         settingsSource(MODEL_PVS_NAME, modelValues, true, true),
+         settingsSource("assistant_price_model", modelValues, true, false),
+         settingsSource("assistant_provider_kind", kinds, false, false),
          enumSource("assistant_provider_location", List.of(new QPossibleValue<>("LOCAL", "Local"), new QPossibleValue<>("CLOUD", "Cloud"))),
          enumSource("assistant_routing_mode", List.of(new QPossibleValue<>("OFF", "Off"), new QPossibleValue<>("HEURISTIC", "Heuristic"), new QPossibleValue<>("CLASSIFIER", "Classifier"))),
+         enumSource("assistant_model_kind", List.of(new QPossibleValue<>("CHAT", "Chat"), new QPossibleValue<>("EMBEDDING", "Embedding"))),
          enumSource("assistant_model_source", List.of(new QPossibleValue<>("DISCOVERED", "Discovered"), new QPossibleValue<>("MANUAL", "Manual"))),
          QPossibleValueSource.newForTable(AssistantProvider.TABLE_NAME)));
    }
 
 
 
+   /*******************************************************************************
+    ** Contingent searches use RC.5's otherValues input.
+    *******************************************************************************/
+   private static QPossibleValueSource settingsSource(String name, List<QPossibleValue<?>> union, Boolean models, Boolean providerRow)
+   {
+      return (new QPossibleValueSource().withName(name).withType(QPossibleValueSourceType.CUSTOM).withIdType(QFieldType.STRING)
+         .withValueFormatAndFields(PVSValueFormatAndFields.LABEL_ONLY)
+         .withCustomCodeReference(new QCodeReferenceWithProperties(AssistantSettingsPossibleValueProvider.class, Map.of("union", new ArrayList<>(union), "models", models, "providerRow", providerRow))));
+   }
+
+
+
    /*******************************************************************************
     ** Display label shared by provider settings and home cards. Custom kinds
     ** retain their registered name.
@@ -332,6 +351,11 @@ public final class AssistantRecordScreens
    private static void provider(QTableMetaData table)
    {
       pvs(table, "providerType", PROVIDER_TYPE_PVS_NAME, "Type");
+      /////////////////////////////////////////////////////////////////////////
+      // Add possibleValueSourceDependsOn("providerType") to model and kind   //
+      // once the next QQQ RC includes core feature/contingent-possible-values. //
+      // See QQQ docs/CONTINGENT-POSSIBLE-VALUES.md.                           //
+      /////////////////////////////////////////////////////////////////////////
       pvs(table, "model", MODEL_PVS_NAME, "Model");
       pvs(table, "kind", "assistant_provider_kind", "Kind");
       pvs(table, "location", "assistant_provider_location", "Location");
@@ -380,6 +404,11 @@ public final class AssistantRecordScreens
    private static void price(QTableMetaData table)
    {
       pvs(table, "providerType", PROVIDER_TYPE_PVS_NAME, "Provider Type");
+      /////////////////////////////////////////////////////////////////////////
+      // Add possibleValueSourceDependsOn("providerType") when the next QQQ  //
+      // RC includes it (QQQ docs/CONTINGENT-POSSIBLE-VALUES.md).              //
+      /////////////////////////////////////////////////////////////////////////
+      pvs(table, "model", "assistant_price_model", "Model");
       for(String field : List.of("inputUsdPerMTok", "outputUsdPerMTok", "cacheWrite5mUsdPerMTok", "cacheWrite1hUsdPerMTok"))
       {
          help(table, field, "USD per million tokens.");
@@ -888,6 +917,7 @@ public final class AssistantRecordScreens
    {
       link(table, "providerId", AssistantProvider.TABLE_NAME, "Provider");
       pvs(table, "source", "assistant_model_source", "Source");
+      pvs(table, "modelKind", "assistant_model_kind", "Model kind");
       pvs(table, "fallbackModelId", "assistant_registry_model", "Fallback model");
       table.getField("contextWindowTokens").withLabel("Context window (tokens)");
       table.getField("maxOutputTokens").withLabel("Max output tokens");
@@ -896,7 +926,7 @@ public final class AssistantRecordScreens
       table.getField("outputUsdPerMTok").withLabel("Output (USD per million tokens)");
       table.getField("lastTestLatencyMs").withLabel("Last test latency (ms)");
       sections(table,
-         section("model", "Model", "memory", Tier.T1, "id", "providerId", "modelId", "displayName", "enabled", "source", "discoveredAt", "fallbackModelId"),
+         section("model", "Model", "memory", Tier.T1, "id", "providerId", "modelId", "displayName", "enabled", "modelKind", "embeddingDimensions", "source", "discoveredAt", "fallbackModelId"),
          section("capabilities", "Capabilities", "settings", Tier.T2, "supportsTools", "supportsVision", "supportsStreaming", "supportsReasoningEffort", "contextWindowTokens", "maxOutputTokens"),
          section("prices", "Prices", "payments", Tier.T2, "inputUsdPerMTok", "outputUsdPerMTok"),
          section("lastTest", "Last connection test", "network_check", Tier.T2, "lastTestAt", "lastTestStatus", "lastTestLatencyMs", "lastTestError"));
@@ -936,4 +966,22 @@ public final class AssistantRecordScreens
          section("budget", "Budget", "savings", Tier.T1, "id", "name", "enabled", "scopeType", "scopeValue", "period", "metric", "limitValue"),
          section("thresholds", "Thresholds", "notifications", Tier.T2, "warnPercent", "atLimitAction", "downgradeModelId", "hardCeilingPercent", "notes"));
    }
+
+
+   /*******************************************************************************
+    ** Standard source editor and status, with explicit embedding model selection.
+    *******************************************************************************/
+   private static void knowledge(QTableMetaData table)
+   {
+      pvs(table, "embeddingModelId", "assistant_registry_model", "Embedding model");
+      table.getField("embeddingModelId").withPossibleValueSourceFilter(new QQueryFilter()
+         .withCriteria("modelKind", QCriteriaOperator.EQUALS, "EMBEDDING").withCriteria("enabled", QCriteriaOperator.EQUALS, true));
+      help(table, "embeddingModelId", "Select an enabled EMBEDDING model. Chat models are refused on save.");
+      help(table, "filterJson", "Optional QQQ query filter JSON. Index now reads it through the UI API as the clicking administrator.");
+      sections(table,
+         section("source", "Knowledge source", "manage_search", Tier.T1, "id", "name", "enabled", "tableName", "fieldNames", "titleFieldName", "filterJson"),
+         section("embedding", "Embedding", "memory", Tier.T2, "embeddingModelId", "chunkChars", "chunkOverlapChars"),
+         section("status", "Last index", "info", Tier.T2, "lastIndexedAt", "lastIndexStatus", "lastIndexError", "indexedRecordCount", "indexedChunkCount"));
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantSettingsCustomizer.java

@@ -83,6 +83,18 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
       {
          for(QRecord record : records)
          {
+            if(record.getValue("id") == null && record.getValue("modelKind") == null)
+            {
+               record.setValue("modelKind", "CHAT");
+            }
+            if(record.getValue("modelKind") != null && !List.of("CHAT", "EMBEDDING").contains(record.getValueString("modelKind")))
+            {
+               record.addError(new BadInputStatusMessage("Model kind must be CHAT or EMBEDDING."));
+            }
+            if(record.getValue("embeddingDimensions") != null && record.getValueInteger("embeddingDimensions") < 1)
+            {
+               record.addError(new BadInputStatusMessage("Embedding dimensions must be positive."));
+            }
             if(record.getValue("id") == null)
             {
                if(record.getValue("source") == null)
@@ -171,6 +183,25 @@ public class AssistantSettingsCustomizer implements TableCustomizerInterface
             }
          }
       }
+      if(AssistantModel.TABLE_NAME.equals(tableName(input)) && oldRecordList.isPresent())
+      {
+         Map<Serializable, QRecord> previous = oldRecordListToMap("id", oldRecordList).orElse(Map.of());
+         for(QRecord record : records)
+         {
+            QRecord before = previous.get(record.getValue("id"));
+            if(before != null && List.of("providerId", "modelId", "modelKind", "embeddingDimensions").stream()
+               .anyMatch(name -> record.getValues().containsKey(name) && !java.util.Objects.equals(before.getValue(name), record.getValue(name))))
+            {
+               for(var source : runtime.getRagService().sources())
+               {
+                  if(java.util.Objects.equals(source.getEmbeddingModelId(), record.getValueInteger("id")))
+                  {
+                     runtime.getRagService().clear(source);
+                  }
+               }
+            }
+         }
+      }
       runtime.getProviderService().invalidate();
       return (records);
    }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/metadata/AssistantSettingsPossibleValueProvider.java

@@ -0,0 +1,134 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.metadata;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
+import com.kingsrook.qqq.backend.core.actions.values.QCustomPossibleValueProvider;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
+import com.kingsrook.qqq.backend.core.model.metadata.code.InitializableViaCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReferenceWithProperties;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
+
+
+/*******************************************************************************
+ ** Provider settings choices, narrowed by the selected provider type.
+ *******************************************************************************/
+public class AssistantSettingsPossibleValueProvider implements QCustomPossibleValueProvider<String>, InitializableViaCodeReference
+{
+   private final Map<String, QPossibleValue<String>> union = new LinkedHashMap<>();
+   private Boolean models;
+   private Boolean providerRow;
+
+
+
+   /*******************************************************************************
+    ** Price rows share model choices but their id is never a provider id.
+    *******************************************************************************/
+   @Override
+   public void initialize(QCodeReference codeReference)
+   {
+      Map<String, Serializable> properties = ((QCodeReferenceWithProperties) codeReference).getProperties();
+      models = (Boolean) properties.get("models");
+      providerRow = (Boolean) properties.get("providerRow");
+      for(Object object : (List<?>) properties.get("union"))
+      {
+         QPossibleValue<?> value = (QPossibleValue<?>) object;
+         union.put(String.valueOf(value.getId()), new QPossibleValue<>(String.valueOf(value.getId()), value.getLabel()));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Saved values resolve across types, including disabled registry models.
+    *******************************************************************************/
+   @Override
+   public QPossibleValue<String> getPossibleValue(Serializable idValue) throws QException
+   {
+      String id = String.valueOf(idValue);
+      if(union.containsKey(id))
+      {
+         return (union.get(id));
+      }
+      if(models && idValue != null)
+      {
+         for(AssistantModel model : new AssistantStore().listModels())
+         {
+            if(id.equals(model.getModelId()))
+            {
+               return (new QPossibleValue<>(id, id));
+            }
+         }
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    ** Keep union fallback for callers that do not supply a known provider type.
+    *******************************************************************************/
+   @Override
+   public List<QPossibleValue<String>> search(SearchPossibleValueSourceInput input) throws QException
+   {
+      Map<String, Serializable> otherValues = input.getOtherValues();
+      AssistantProviderType type = otherValues == null ? null : AssistantRuntime.of(QContext.getQInstance()).getConfig().getProviderTypes().get(otherValues.get("providerType"));
+      Map<String, QPossibleValue<String>> values = new LinkedHashMap<>();
+      if(type == null)
+      {
+         values.putAll(union);
+      }
+      else
+      {
+         for(String id : models ? type.getModels() : type.getKinds())
+         {
+            values.put(id, new QPossibleValue<>(id, models ? id : AssistantRecordScreens.providerKindLabel(id)));
+         }
+         if(models && providerRow && otherValues.get("id") != null)
+         {
+            AssistantStore store = new AssistantStore();
+            String providerId = String.valueOf(otherValues.get("id"));
+            AssistantProvider provider = store.listProviders().stream()
+               .filter(row -> providerId.equals(String.valueOf(row.getId())) && type.getType().equals(row.getProviderType())).findFirst().orElse(null);
+            if(provider != null)
+            {
+               for(AssistantModel model : store.listModels())
+               {
+                  if(provider.getId().equals(model.getProviderId()) && Boolean.TRUE.equals(model.getEnabled()) && "CHAT".equals(model.getModelKind()))
+                  {
+                     values.put(model.getModelId(), new QPossibleValue<>(model.getModelId(), model.getModelId()));
+                  }
+               }
+            }
+         }
+      }
+      return (completeCustomPVSSearch(input, new ArrayList<>(values.values())));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/model/AssistantModel.java

@@ -32,6 +32,12 @@ public class AssistantModel extends QRecordEntity
 {
    public static final String TABLE_NAME = "assistant_model";
 
+   @QField(maxLength = 16)
+   private String modelKind;
+
+   @QField()
+   private Integer embeddingDimensions;
+
    @QField()
    private Integer fallbackModelId;
 
@@ -798,4 +804,76 @@ public class AssistantModel extends QRecordEntity
       this.fallbackModelId = fallbackModelId;
       return (this);
    }
+
+
+
+   /*******************************************************************************
+    ** Getter for modelKind
+    *******************************************************************************/
+   public String getModelKind()
+   {
+      return (modelKind);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modelKind
+    *******************************************************************************/
+   public void setModelKind(String modelKind)
+   {
+      this.modelKind = modelKind;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modelKind
+    *******************************************************************************/
+   public AssistantModel withModelKind(String modelKind)
+   {
+      this.modelKind = modelKind;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for embeddingDimensions
+    *******************************************************************************/
+   public Integer getEmbeddingDimensions()
+   {
+      return (embeddingDimensions);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for embeddingDimensions
+    *******************************************************************************/
+   public void setEmbeddingDimensions(Integer embeddingDimensions)
+   {
+      this.embeddingDimensions = embeddingDimensions;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for embeddingDimensions
+    *******************************************************************************/
+   public AssistantModel withEmbeddingDimensions(Integer embeddingDimensions)
+   {
+      this.embeddingDimensions = embeddingDimensions;
+      return (this);
+   }
+
+
+   /*******************************************************************************
+    ** Legacy registry rows remain chat models until explicitly classified.
+    *******************************************************************************/
+   public Boolean isChat()
+   {
+      return (modelKind == null || "CHAT".equals(modelKind));
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/model/AssistantRagSource.java

@@ -0,0 +1,629 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.model;
+
+
+import java.time.Instant;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.data.QField;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
+
+
+/*******************************************************************************
+ ** Admin-owned indexing configuration. Source records never contain indexed text.
+ *******************************************************************************/
+public class AssistantRagSource extends QRecordEntity
+{
+   public static final String TABLE_NAME = "assistant_rag_source";
+
+   @QField(isEditable = false, isPrimaryKey = true)
+   private Integer id;
+
+   @QField(isRequired = true, maxLength = 200)
+   private String name;
+
+   @QField()
+   private Boolean enabled;
+
+   @QField(isRequired = true, maxLength = 200)
+   private String tableName;
+
+   @QField(isRequired = true, maxLength = 500)
+   private String fieldNames;
+
+   @QField(maxLength = 200)
+   private String titleFieldName;
+
+   @QField()
+   private String filterJson;
+
+   @QField(isRequired = true)
+   private Integer embeddingModelId;
+
+   @QField()
+   private Integer chunkChars;
+
+   @QField()
+   private Integer chunkOverlapChars;
+
+   @QField(isEditable = false)
+   private Instant lastIndexedAt;
+
+   @QField(isEditable = false, maxLength = 200)
+   private String lastIndexStatus;
+
+   @QField(isEditable = false, maxLength = 500)
+   private String lastIndexError;
+
+   @QField(isEditable = false)
+   private Integer indexedRecordCount;
+
+   @QField(isEditable = false)
+   private Integer indexedChunkCount;
+
+   @QField(isEditable = false)
+   private Instant createDate;
+
+   @QField(isEditable = false)
+   private Instant modifyDate;
+
+
+   /*******************************************************************************
+    ** Empty entity for framework mapping.
+    *******************************************************************************/
+   public AssistantRagSource()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Map the source's own configuration record.
+    *******************************************************************************/
+   public AssistantRagSource(QRecord record) throws QException
+   {
+      populateFromQRecord(record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for id
+    *******************************************************************************/
+   public Integer getId()
+   {
+      return (id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for id
+    *******************************************************************************/
+   public void setId(Integer id)
+   {
+      this.id = id;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for id
+    *******************************************************************************/
+   public AssistantRagSource withId(Integer id)
+   {
+      this.id = id;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for name
+    *******************************************************************************/
+   public String getName()
+   {
+      return (name);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for name
+    *******************************************************************************/
+   public void setName(String name)
+   {
+      this.name = name;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for name
+    *******************************************************************************/
+   public AssistantRagSource withName(String name)
+   {
+      this.name = name;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for enabled
+    *******************************************************************************/
+   public Boolean getEnabled()
+   {
+      return (enabled);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for enabled
+    *******************************************************************************/
+   public void setEnabled(Boolean enabled)
+   {
+      this.enabled = enabled;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for enabled
+    *******************************************************************************/
+   public AssistantRagSource withEnabled(Boolean enabled)
+   {
+      this.enabled = enabled;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for tableName
+    *******************************************************************************/
+   public String getTableName()
+   {
+      return (tableName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for tableName
+    *******************************************************************************/
+   public void setTableName(String tableName)
+   {
+      this.tableName = tableName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for tableName
+    *******************************************************************************/
+   public AssistantRagSource withTableName(String tableName)
+   {
+      this.tableName = tableName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for fieldNames
+    *******************************************************************************/
+   public String getFieldNames()
+   {
+      return (fieldNames);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for fieldNames
+    *******************************************************************************/
+   public void setFieldNames(String fieldNames)
+   {
+      this.fieldNames = fieldNames;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for fieldNames
+    *******************************************************************************/
+   public AssistantRagSource withFieldNames(String fieldNames)
+   {
+      this.fieldNames = fieldNames;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for titleFieldName
+    *******************************************************************************/
+   public String getTitleFieldName()
+   {
+      return (titleFieldName);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for titleFieldName
+    *******************************************************************************/
+   public void setTitleFieldName(String titleFieldName)
+   {
+      this.titleFieldName = titleFieldName;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for titleFieldName
+    *******************************************************************************/
+   public AssistantRagSource withTitleFieldName(String titleFieldName)
+   {
+      this.titleFieldName = titleFieldName;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for filterJson
+    *******************************************************************************/
+   public String getFilterJson()
+   {
+      return (filterJson);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for filterJson
+    *******************************************************************************/
+   public void setFilterJson(String filterJson)
+   {
+      this.filterJson = filterJson;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for filterJson
+    *******************************************************************************/
+   public AssistantRagSource withFilterJson(String filterJson)
+   {
+      this.filterJson = filterJson;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for embeddingModelId
+    *******************************************************************************/
+   public Integer getEmbeddingModelId()
+   {
+      return (embeddingModelId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for embeddingModelId
+    *******************************************************************************/
+   public void setEmbeddingModelId(Integer embeddingModelId)
+   {
+      this.embeddingModelId = embeddingModelId;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for embeddingModelId
+    *******************************************************************************/
+   public AssistantRagSource withEmbeddingModelId(Integer embeddingModelId)
+   {
+      this.embeddingModelId = embeddingModelId;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for chunkChars
+    *******************************************************************************/
+   public Integer getChunkChars()
+   {
+      return (chunkChars);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for chunkChars
+    *******************************************************************************/
+   public void setChunkChars(Integer chunkChars)
+   {
+      this.chunkChars = chunkChars;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for chunkChars
+    *******************************************************************************/
+   public AssistantRagSource withChunkChars(Integer chunkChars)
+   {
+      this.chunkChars = chunkChars;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for chunkOverlapChars
+    *******************************************************************************/
+   public Integer getChunkOverlapChars()
+   {
+      return (chunkOverlapChars);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for chunkOverlapChars
+    *******************************************************************************/
+   public void setChunkOverlapChars(Integer chunkOverlapChars)
+   {
+      this.chunkOverlapChars = chunkOverlapChars;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for chunkOverlapChars
+    *******************************************************************************/
+   public AssistantRagSource withChunkOverlapChars(Integer chunkOverlapChars)
+   {
+      this.chunkOverlapChars = chunkOverlapChars;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for lastIndexedAt
+    *******************************************************************************/
+   public Instant getLastIndexedAt()
+   {
+      return (lastIndexedAt);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for lastIndexedAt
+    *******************************************************************************/
+   public void setLastIndexedAt(Instant lastIndexedAt)
+   {
+      this.lastIndexedAt = lastIndexedAt;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for lastIndexedAt
+    *******************************************************************************/
+   public AssistantRagSource withLastIndexedAt(Instant lastIndexedAt)
+   {
+      this.lastIndexedAt = lastIndexedAt;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for lastIndexStatus
+    *******************************************************************************/
+   public String getLastIndexStatus()
+   {
+      return (lastIndexStatus);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for lastIndexStatus
+    *******************************************************************************/
+   public void setLastIndexStatus(String lastIndexStatus)
+   {
+      this.lastIndexStatus = lastIndexStatus;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for lastIndexStatus
+    *******************************************************************************/
+   public AssistantRagSource withLastIndexStatus(String lastIndexStatus)
+   {
+      this.lastIndexStatus = lastIndexStatus;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for lastIndexError
+    *******************************************************************************/
+   public String getLastIndexError()
+   {
+      return (lastIndexError);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for lastIndexError
+    *******************************************************************************/
+   public void setLastIndexError(String lastIndexError)
+   {
+      this.lastIndexError = lastIndexError;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for lastIndexError
+    *******************************************************************************/
+   public AssistantRagSource withLastIndexError(String lastIndexError)
+   {
+      this.lastIndexError = lastIndexError;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for indexedRecordCount
+    *******************************************************************************/
+   public Integer getIndexedRecordCount()
+   {
+      return (indexedRecordCount);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for indexedRecordCount
+    *******************************************************************************/
+   public void setIndexedRecordCount(Integer indexedRecordCount)
+   {
+      this.indexedRecordCount = indexedRecordCount;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for indexedRecordCount
+    *******************************************************************************/
+   public AssistantRagSource withIndexedRecordCount(Integer indexedRecordCount)
+   {
+      this.indexedRecordCount = indexedRecordCount;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for indexedChunkCount
+    *******************************************************************************/
+   public Integer getIndexedChunkCount()
+   {
+      return (indexedChunkCount);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for indexedChunkCount
+    *******************************************************************************/
+   public void setIndexedChunkCount(Integer indexedChunkCount)
+   {
+      this.indexedChunkCount = indexedChunkCount;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for indexedChunkCount
+    *******************************************************************************/
+   public AssistantRagSource withIndexedChunkCount(Integer indexedChunkCount)
+   {
+      this.indexedChunkCount = indexedChunkCount;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for createDate
+    *******************************************************************************/
+   public Instant getCreateDate()
+   {
+      return (createDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for createDate
+    *******************************************************************************/
+   public void setCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for createDate
+    *******************************************************************************/
+   public AssistantRagSource withCreateDate(Instant createDate)
+   {
+      this.createDate = createDate;
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for modifyDate
+    *******************************************************************************/
+   public Instant getModifyDate()
+   {
+      return (modifyDate);
+   }
+
+
+
+   /*******************************************************************************
+    ** Setter for modifyDate
+    *******************************************************************************/
+   public void setModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+   }
+
+
+
+   /*******************************************************************************
+    ** Fluent setter for modifyDate
+    *******************************************************************************/
+   public AssistantRagSource withModifyDate(Instant modifyDate)
+   {
+      this.modifyDate = modifyDate;
+      return (this);
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/processes/KnowledgeIndexProcess.java

@@ -0,0 +1,231 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.processes;
+
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.classification.DataClassification;
+import com.kingsrook.qbits.assistant.metadata.AssistantMetaDataBuilder;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
+import com.kingsrook.qbits.assistant.rag.RagService;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
+import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
+
+
+/*******************************************************************************
+ ** Standard Review/Result record actions. No scheduler or service-user runner.
+ *******************************************************************************/
+public final class KnowledgeIndexProcess
+{
+   public static final String INDEX = "assistantKnowledgeIndex";
+   public static final String CLEAR = "assistantKnowledgeClear";
+
+
+
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private KnowledgeIndexProcess()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Both processes are table actions, never navigation entries.
+    *******************************************************************************/
+   public static List<QProcessMetaData> processes()
+   {
+      List<QProcessMetaData> processes = new ArrayList<>();
+      for(Boolean clear : List.of(false, true))
+      {
+         Map<String, Serializable> values = new HashMap<>();
+         values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, AssistantRagSource.TABLE_NAME);
+         values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, AssistantRagSource.TABLE_NAME);
+         values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
+         values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
+         values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, clear ? "Clear vectors for the selected knowledge sources." : "Read content through the UI API as you and embed changed chunks. Embedding usage counts toward budgets.");
+         QProcessMetaData process = StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, clear ? ClearReview.class : Review.class, clear ? Clear.class : Index.class, values);
+         process.setName(clear ? CLEAR : INDEX);
+         process.setLabel(clear ? "Clear index" : "Index now");
+         process.setIcon(new QIcon(clear ? "delete_sweep" : "manage_search"));
+         process.setTableName(AssistantRagSource.TABLE_NAME);
+         process.setPermissionRules(AssistantMetaDataBuilder.adminRules());
+         processes.add(process);
+      }
+      return (processes);
+   }
+
+
+
+   /*******************************************************************************
+    ** Review validates configuration without sending content to any provider.
+    *******************************************************************************/
+   public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface
+   {
+      private final ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
+
+
+
+      /*******************************************************************************
+       ** Source records only; application records are read at Result time.
+       *******************************************************************************/
+      @Override
+      public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
+      {
+         AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
+         for(QRecord record : input.getRecords())
+         {
+            AssistantRagSource source = new AssistantRagSource(record);
+            List<String> errors = problems(runtime, source);
+            lines.add(TestConnectionProcess.single(errors.isEmpty() ? Status.INFO : Status.ERROR,
+               source.getName() + (errors.isEmpty() ? ": source selected; no content has been sent." : ": " + String.join("; ", errors)), source.getId()));
+            output.getRecords().add(record);
+         }
+      }
+
+
+
+      /*******************************************************************************
+       ** Standard streamed ETL Review lines.
+       *******************************************************************************/
+      @Override
+      public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
+      {
+         return (lines);
+      }
+
+
+
+      /*******************************************************************************
+       ** Review reports PHI/model setup blocks before any indexing dispatch.
+       *******************************************************************************/
+      protected List<String> problems(AssistantRuntime runtime, AssistantRagSource source) throws QException
+      {
+         List<String> errors = runtime.getRagService().validate(source);
+         if(errors.isEmpty())
+         {
+            try
+            {
+               runtime.getProviderService().resolveEmbedding(source.getEmbeddingModelId(),
+                  runtime.getConfig().getClassificationSource().highest(source.getTableName()) == DataClassification.PHI);
+            }
+            catch(QException e)
+            {
+               errors.add(e.getMessage());
+            }
+         }
+         return (errors);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Broken or disabled source configuration must never prevent clearing it.
+    *******************************************************************************/
+   public static class ClearReview extends Review
+   {
+      /*******************************************************************************
+       ** Clearing has no model or source-content prerequisites.
+       *******************************************************************************/
+      @Override
+      protected List<String> problems(AssistantRuntime runtime, AssistantRagSource source)
+      {
+         return (List.of());
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Result performs indexing under the clicking administrator's QContext.
+    *******************************************************************************/
+   public static class Index extends AbstractLoadStep implements ProcessSummaryProviderInterface
+   {
+      protected final ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
+
+
+
+      /*******************************************************************************
+       ** Page of source configuration records, not application content.
+       *******************************************************************************/
+      @Override
+      public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
+      {
+         AssistantRuntime runtime = AssistantRuntime.of(QContext.getQInstance());
+         for(QRecord record : input.getRecords())
+         {
+            AssistantRagSource source = new AssistantRagSource(record);
+            RagService.Result result = runtime.getRagService().index(source);
+            lines.add(TestConnectionProcess.single(result.errors() == 0 ? Status.OK : Status.ERROR, result.summary(), source.getId()));
+            output.getRecords().add(record);
+         }
+      }
+
+
+
+      /*******************************************************************************
+       ** Counts from this run, suitable for the standard Result screen.
+       *******************************************************************************/
+      @Override
+      public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
+      {
+         return (lines);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Clear shares the process presentation but makes no embedding calls.
+    *******************************************************************************/
+   public static class Clear extends Index
+   {
+      /*******************************************************************************
+       ** Remove only the selected sources' vectors.
+       *******************************************************************************/
+      @Override
+      public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
+      {
+         for(QRecord record : input.getRecords())
+         {
+            AssistantRagSource source = new AssistantRagSource(record);
+            AssistantRuntime.of(QContext.getQInstance()).getRagService().clear(source);
+            lines.add(TestConnectionProcess.single(Status.OK, source.getName() + ": index cleared.", source.getId()));
+            output.getRecords().add(record);
+         }
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/provider/AssistantModelProvider.java

@@ -58,4 +58,14 @@ public interface AssistantModelProvider
       return (null);
    }
 
+
+
+   /*******************************************************************************
+    ** Optional embedding capability. Unsupported adapters fail before transport.
+    *******************************************************************************/
+   default EmbeddingResponse embed(EmbeddingRequest request) throws ModelProviderException
+   {
+      throw (new ModelProviderException(ModelProviderException.Kind.INVALID_REQUEST, null, "Embeddings are unsupported by this provider."));
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/provider/EmbeddingRequest.java

@@ -0,0 +1,39 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.provider;
+
+
+import java.util.List;
+
+
+/*******************************************************************************
+ ** One embedding batch. Providers accept at most 64 inputs per request.
+ *******************************************************************************/
+public record EmbeddingRequest(String model, List<String> input)
+{
+   /*******************************************************************************
+    ** Snapshot the inputs and reject empty or oversized batches before transport.
+    *******************************************************************************/
+   public EmbeddingRequest
+   {
+      input = List.copyOf(input);
+      if(model == null || model.isBlank() || input.isEmpty() || input.size() > 64)
+      {
+         throw (new IllegalArgumentException("An embedding request needs a model and 1 to 64 inputs."));
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/provider/EmbeddingResponse.java

@@ -0,0 +1,28 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.provider;
+
+
+import java.util.List;
+
+
+/*******************************************************************************
+ ** Ordered vectors and billable usage. No input text is retained in the response.
+ *******************************************************************************/
+public record EmbeddingResponse(List<float[]> vectors, ModelUsage usage, String requestId)
+{
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/provider/RegistryModelProvider.java

@@ -94,4 +94,15 @@ public class RegistryModelProvider implements AssistantModelProvider
    {
       return (delegate.countTokens(request));
    }
+
+
+   /*******************************************************************************
+    ** Preserve the delegate's optional embedding capability.
+    *******************************************************************************/
+   @Override
+   public EmbeddingResponse embed(EmbeddingRequest request) throws ModelProviderException
+   {
+      return (delegate.embed(request));
+   }
+
 }

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/ChunkVector.java

@@ -0,0 +1,55 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+/*******************************************************************************
+ ** Content-free index entry. The embedding is copied on entry and on read.
+ *******************************************************************************/
+public record ChunkVector(Integer source, String table, String pk, String field, Integer chunkIndex, String contentHash, float[] embedding)
+{
+   /*******************************************************************************
+    ** No mutable vector can alter an indexed entry after insertion.
+    *******************************************************************************/
+   public ChunkVector
+   {
+      if(source == null || table == null || pk == null || field == null || chunkIndex == null || chunkIndex < 0
+         || contentHash == null || !contentHash.matches("[a-f0-9]{64}") || embedding == null || embedding.length == 0)
+      {
+         throw (new IllegalArgumentException("Incomplete chunk vector metadata."));
+      }
+      embedding = embedding.clone();
+      for(float value : embedding)
+      {
+         if(!Float.isFinite(value))
+         {
+            throw (new IllegalArgumentException("Embedding coordinates must be finite."));
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Return a defensive copy.
+    *******************************************************************************/
+   @Override
+   public float[] embedding()
+   {
+      return (embedding.clone());
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/EmbeddingService.java

@@ -0,0 +1,170 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.time.LocalDate;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.AssistantSecurity;
+import com.kingsrook.qbits.assistant.classification.DataClassification;
+import com.kingsrook.qbits.assistant.engine.BudgetService;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.engine.ProviderService;
+import com.kingsrook.qbits.assistant.engine.UsageRecorder;
+import com.kingsrook.qbits.assistant.model.AssistantDataAudit;
+import com.kingsrook.qbits.assistant.provider.EmbeddingRequest;
+import com.kingsrook.qbits.assistant.provider.EmbeddingResponse;
+import com.kingsrook.qbits.assistant.provider.ModelProviderException;
+import com.kingsrook.qbits.assistant.provider.ModelResponse;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** Embedding dispatch uses fresh eligibility, kill-switch and budget checks per batch.
+ *******************************************************************************/
+public class EmbeddingService
+{
+   private final AssistantRuntime runtime;
+
+
+
+   /*******************************************************************************
+    ** Bind the per-instance accounting and provider services.
+    *******************************************************************************/
+   public EmbeddingService(AssistantRuntime runtime)
+   {
+      this.runtime = runtime;
+   }
+
+
+
+   /*******************************************************************************
+    ** Batch at most 64 inputs. Every dispatched batch records EMBED usage,
+    ** including failures, and updates the existing daily budget rollups.
+    *******************************************************************************/
+   public List<float[]> embed(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId) throws QException
+   {
+      List<float[]> vectors = new ArrayList<>();
+      for(Integer offset = 0; offset < texts.size(); offset += 64)
+      {
+         vectors.addAll(batch(modelId, texts.subList(offset, Math.min(offset + 64, texts.size())), classification, tables, conversationId, turnId));
+      }
+      return (vectors);
+   }
+
+
+
+   /*******************************************************************************
+    ** Audit protected dispatch refusals without any text, before transport.
+    *******************************************************************************/
+   public ProviderService.Active resolve(Integer modelId, DataClassification classification, String tables, Integer conversationId, Integer turnId) throws QException
+   {
+      try
+      {
+         return (runtime.getProviderService().resolveEmbedding(modelId, classification == DataClassification.PHI));
+      }
+      catch(ProviderService.ProviderSetupException e)
+      {
+         if(classification == DataClassification.PHI || Boolean.TRUE.equals(runtime.getConfig().getHipaaMode()))
+         {
+            var model = runtime.getStore().listModels().stream().filter(m -> Objects.equals(m.getId(), modelId)).findFirst().orElse(null);
+            var provider = model == null ? null : runtime.getStore().getProvider(model.getProviderId());
+            runtime.getStore().insert(new AssistantDataAudit().withUserId(AssistantSecurity.userId(QContext.getQSession())).withConversationId(conversationId).withTurnId(turnId)
+               .withProviderId(provider == null ? null : provider.getId()).withProviderName(provider == null ? null : provider.getName())
+               .withModel(model == null ? null : model.getModelId()).withHipaaEligible(provider != null && Boolean.TRUE.equals(provider.getHipaaEligible()))
+               .withClassification(classification.name()).withTables(tables).withDecision("BLOCKED").withReason(e.getMessage()).withCreateDate(runtime.getClock().instant()));
+         }
+         throw (e);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Never downgrade an embedding call into a different vector space.
+    *******************************************************************************/
+   private List<float[]> batch(Integer modelId, List<String> texts, DataClassification classification, String tables, Integer conversationId, Integer turnId) throws QException
+   {
+      String userId = AssistantSecurity.userId(QContext.getQSession());
+      ProviderService.Active active = resolve(modelId, classification, tables, conversationId, turnId);
+      PolicyService.EffectivePolicy policy = runtime.getPolicyService().current(true);
+      BudgetService.Decision budget = runtime.getBudgetService().evaluate(userId, active, classification == DataClassification.PHI, false);
+      Long cap = ProviderService.capMicros(active.row());
+      if(policy.killSwitch())
+      {
+         throw (new QException("The assistant kill switch is on; embeddings were not sent."));
+      }
+      if(budget.stopped() || !Objects.equals(active.model(), budget.active().model()) || !Objects.equals(active.providerId(), budget.active().providerId())
+         || runtime.getPolicyService().checkConfiguredBudget(policy, userId, 0, 0).hardExceeded()
+         || (cap != null && runtime.getProviderService().monthToDateCostMicros(active.providerId()) >= cap))
+      {
+         throw (new QException("The embedding budget has been reached; embeddings were not sent."));
+      }
+      if(classification.isAtLeast(DataClassification.PII))
+      {
+         runtime.getStore().insert(new AssistantDataAudit().withUserId(userId).withConversationId(conversationId).withTurnId(turnId)
+            .withProviderId(active.providerId()).withProviderName(active.row().getName()).withModel(active.model())
+            .withHipaaEligible(Boolean.TRUE.equals(active.row().getHipaaEligible())).withClassification(classification.name())
+            .withTables(tables).withDecision("ALLOWED").withReason("Embedding provider permitted for this classification.").withCreateDate(runtime.getClock().instant()));
+      }
+      Semaphore gate = runtime.getProviderService().gate(active.row());
+      Boolean acquired = gate == null || gate.tryAcquire();
+      if(!acquired)
+      {
+         throw (new QException("Embedding provider concurrency limit reached."));
+      }
+      Long started = System.nanoTime();
+      ModelResponse response = null;
+      ModelProviderException error = null;
+      try
+      {
+         EmbeddingResponse embedded = active.provider().embed(new EmbeddingRequest(active.model(), texts));
+         response = new ModelResponse().withModel(active.model()).withRequestId(embedded.requestId()).withUsage(embedded.usage());
+         if(embedded.vectors().size() != texts.size())
+         {
+            throw (new ModelProviderException(ModelProviderException.Kind.OTHER, null, "The embedding provider returned the wrong number of vectors."));
+         }
+         return (embedded.vectors());
+      }
+      catch(ModelProviderException e)
+      {
+         error = e;
+         throw (new QException("Embedding request failed (" + e.getKind() + "). Check provider settings and retry."));
+      }
+      finally
+      {
+         if(gate != null)
+         {
+            gate.release();
+         }
+         var pricing = runtime.getProviderService().pricing(active, active.model());
+         runtime.getStore().insert(UsageRecorder.row(new UsageRecorder.Call(userId, conversationId, turnId, active, UsageRecorder.PURPOSE_EMBED),
+            response, error, (System.nanoTime() - started) / 1_000_000, pricing));
+         var usage = response == null ? (error == null ? null : error.getPartialUsage()) : response.getUsage();
+         if(usage != null)
+         {
+            runtime.getStore().addUsage(userId, LocalDate.now(runtime.getClock().withZone(ZoneOffset.UTC)), usage, pricing.costMicros(usage), false);
+         }
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/InMemoryVectorStore.java

@@ -0,0 +1,157 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+/*******************************************************************************
+ ** Exact cosine search for tests and small hosts. Never stores source text.
+ *******************************************************************************/
+public class InMemoryVectorStore implements VectorStore
+{
+   private final Map<List<Object>, ChunkVector> vectors = new LinkedHashMap<>();
+
+
+
+   /*******************************************************************************
+    ** Upsert immutable entries.
+    *******************************************************************************/
+   @Override
+   public synchronized void upsert(List<ChunkVector> chunks)
+   {
+      for(ChunkVector chunk : chunks)
+      {
+         vectors.put(List.of(chunk.source(), chunk.table(), chunk.pk(), chunk.field(), chunk.chunkIndex()), chunk);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Remove a record within its source only.
+    *******************************************************************************/
+   @Override
+   public synchronized void deleteRecord(Integer source, String table, String pk)
+   {
+      vectors.values().removeIf(v -> v.source().equals(source) && v.table().equals(table) && v.pk().equals(pk));
+   }
+
+
+
+   /*******************************************************************************
+    ** A source can be cleared independently of other sources.
+    *******************************************************************************/
+   @Override
+   public synchronized void deleteSource(Integer source)
+   {
+      vectors.values().removeIf(v -> v.source().equals(source));
+   }
+
+
+
+   /*******************************************************************************
+    ** Search only the named tables and vectors with compatible dimensions.
+    *******************************************************************************/
+   @Override
+   public synchronized List<Hit> search(float[] query, Integer k, Set<String> tables)
+   {
+      return (search(query, k, tables, Set.of()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Different embedding models are different spaces even at equal dimensions.
+    *******************************************************************************/
+   @Override
+   public synchronized List<Hit> search(float[] query, Integer k, Set<String> tables, Set<Integer> sources)
+   {
+      List<Hit> hits = new ArrayList<>();
+      for(ChunkVector vector : vectors.values())
+      {
+         if(tables.contains(vector.table()) && (sources.isEmpty() || sources.contains(vector.source())) && query.length == vector.embedding().length)
+         {
+            Double score = cosine(query, vector.embedding());
+            if(Double.isFinite(score))
+            {
+               hits.add(new Hit(vector.source(), vector.table(), vector.pk(), vector.field(), vector.chunkIndex(), score));
+            }
+         }
+      }
+      return (hits.stream().sorted(Comparator.comparing(Hit::score).reversed()).limit(Math.max(0, k)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    ** Zero vectors have no direction and never match.
+    *******************************************************************************/
+   private static Double cosine(float[] a, float[] b)
+   {
+      Double dot = 0.0;
+      Double left = 0.0;
+      Double right = 0.0;
+      for(Integer i = 0; i < a.length; i++)
+      {
+         dot += (double) a[i] * b[i];
+         left += (double) a[i] * a[i];
+         right += (double) b[i] * b[i];
+      }
+      return (left == 0 || right == 0 ? Double.NaN : dot / Math.sqrt(left * right));
+   }
+
+
+
+   /*******************************************************************************
+    ** Count by source.
+    *******************************************************************************/
+   @Override
+   public synchronized Long count(Integer source)
+   {
+      return (vectors.values().stream().filter(v -> v.source().equals(source)).count());
+   }
+
+
+
+   /*******************************************************************************
+    ** Snapshot the content-free entries for reconciliation.
+    *******************************************************************************/
+   @Override
+   public synchronized List<ChunkVector> chunks(Integer source)
+   {
+      return (vectors.values().stream().filter(v -> v.source().equals(source)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    ** Readers see either complete version of this record.
+    *******************************************************************************/
+   @Override
+   public synchronized void replaceRecord(Integer source, String table, String pk, List<ChunkVector> chunks)
+   {
+      deleteRecord(source, table, pk);
+      upsert(chunks);
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/RagService.java

@@ -0,0 +1,446 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
+import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
+import com.kingsrook.qqq.backend.core.actions.ai.exposure.AiExposurePolicy;
+import com.kingsrook.qqq.backend.core.actions.ai.output.AiRedactor;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+
+
+/*******************************************************************************
+ ** Manual indexing as the clicking administrator, and shared safe UI API reads.
+ *******************************************************************************/
+public class RagService
+{
+   private final AssistantRuntime runtime;
+   private final VectorStore vectors;
+   private final AiExposurePolicy exposure;
+   private final UiApiToolbox reader;
+   private final EmbeddingService embeddings;
+
+
+
+   /*******************************************************************************
+    ** Reuse get_record's helpers through its public execution path, without
+    ** imposing the chat cell-length cap on indexing or snippet boundaries.
+    *******************************************************************************/
+   public RagService(AssistantRuntime runtime, VectorStore vectors)
+   {
+      this.runtime = runtime;
+      this.vectors = vectors;
+      var settings = AssistantRuntime.kernelSettings(runtime.getConfig());
+      exposure = new AiExposurePolicy(settings);
+      reader = new UiApiToolbox(runtime.getQInstance(), runtime.getDataAccessService(), exposure, new AiRedactor(settings.getCustomRedactionPatterns()), Integer.MAX_VALUE);
+      embeddings = new EmbeddingService(runtime);
+   }
+
+
+
+   /*******************************************************************************
+    ** Configuration only; never query application data through AssistantStore.
+    *******************************************************************************/
+   public List<AssistantRagSource> sources() throws QException
+   {
+      return (runtime.getStore().query(AssistantRagSource.TABLE_NAME, new QQueryFilter(), AssistantRagSource.class));
+   }
+
+
+
+   /*******************************************************************************
+    ** Source definitions only narrow existing AI table/field exposure.
+    *******************************************************************************/
+   public List<String> validate(AssistantRagSource source) throws QException
+   {
+      List<String> errors = new ArrayList<>();
+      QTableMetaData table = runtime.getQInstance().getTable(source.getTableName());
+      if(table == null || source.getTableName().startsWith("assistant_") || !runtime.getDataAccessService().forTable(source.getTableName()).mode().allowsRead())
+      {
+         errors.add("Knowledge sources require an application table with AI data access READ or READ_WRITE, never NONE or assistant_*.");
+         return (errors);
+      }
+      if(fields(source).isEmpty())
+      {
+         errors.add("Select at least one text field.");
+      }
+      Set<String> selected = new LinkedHashSet<>(fields(source));
+      if(source.getTitleFieldName() != null && !source.getTitleFieldName().isBlank())
+      {
+         selected.add(source.getTitleFieldName());
+      }
+      for(String name : selected)
+      {
+         QFieldMetaData field = table.getFields().get(name);
+         AiExposure level = field == null ? AiExposure.NONE : exposure.getFieldExposure(table, field);
+         if(field == null || !Set.of(QFieldType.STRING, QFieldType.TEXT).contains(field.getType()) || !level.allowsValues())
+         {
+            errors.add("Knowledge fields must be text fields with VALUES or MASKED exposure: " + name);
+         }
+      }
+      if(size(source) < 100 || size(source) > 16000 || overlap(source) < 0 || overlap(source) >= size(source))
+      {
+         errors.add("Chunk size must be 100 to 16000; overlap must be nonnegative and smaller than chunk size.");
+      }
+      AssistantModel model = runtime.getStore().listModels().stream().filter(m -> Objects.equals(m.getId(), source.getEmbeddingModelId())).findFirst().orElse(null);
+      if(model == null || !"EMBEDDING".equals(model.getModelKind()) || !Boolean.TRUE.equals(model.getEnabled()) || model.getEmbeddingDimensions() == null || model.getEmbeddingDimensions() < 1)
+      {
+         errors.add("Select an enabled EMBEDDING model with positive embedding dimensions.");
+      }
+      try
+      {
+         filter(source);
+      }
+      catch(QException e)
+      {
+         errors.add("The query filter must be valid QQQ query filter JSON.");
+      }
+      return (errors);
+   }
+
+
+
+   /*******************************************************************************
+    ** Transient safe values from get_record, with the asking user's session.
+    *******************************************************************************/
+   public UiApiToolbox.Outcome read(UiApiToolbox.Context context, String table, String pk)
+   {
+      PolicyService.ToolPolicy policy = context.policy();
+      UiApiToolbox.Context internal = new UiApiToolbox.Context(context.sessionUUID(), context.userTimezone(), context.gateway(),
+         new PolicyService.ToolPolicy(true, Set.of(), policy == null ? 50 : policy.maxRowsPerCall(), policy == null || policy.redactPii()), null);
+      return (reader.execute(internal, "knowledge-read", UiApiToolbox.GET_RECORD, Map.of("table", table, "id", pk)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Read already masked/redacted fields from the existing record render.
+    *******************************************************************************/
+   public static Map<String, String> values(UiApiToolbox.Outcome outcome)
+   {
+      Map<String, String> values = new LinkedHashMap<>();
+      if(outcome.getResult().getRender() instanceof Map<?, ?> render && render.get("fields") instanceof List<?> fields)
+      {
+         for(Object item : fields)
+         {
+            if(item instanceof Map<?, ?> field && field.get("name") instanceof String name && field.get("value") != null)
+            {
+               values.put(name, String.valueOf(field.get("value")));
+            }
+         }
+      }
+      return (values);
+   }
+
+
+
+   /*******************************************************************************
+    ** Index result counters contain no record values or provider diagnostics.
+    *******************************************************************************/
+   public record Result(Integer recordsSeen, Integer embedded, Integer unchanged, Integer removed, Integer errors, String reason)
+   {
+      /*******************************************************************************
+       ** Standard process summary wording.
+       *******************************************************************************/
+      public String summary()
+      {
+         return ("Records seen: " + recordsSeen + "; chunks embedded: " + embedded + "; unchanged: " + unchanged + "; removed: " + removed + "; errors: " + errors
+            + (reason == null ? "" : ". " + reason));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Complete scans alone remove disappeared records. Failed or denied pages
+    ** never erase previously indexed records. Re-indexing is deliberately manual.
+    *******************************************************************************/
+   public synchronized Result index(AssistantRagSource source) throws QException
+   {
+      requireAdmin();
+      Integer seen = 0;
+      Integer embedded = 0;
+      Integer unchanged = 0;
+      Integer removed = 0;
+      String reason = null;
+      try
+      {
+         List<String> errors = validate(source);
+         if(!errors.isEmpty())
+         {
+            throw (new QException(errors.getFirst()));
+         }
+         var classification = runtime.getConfig().getClassificationSource().highest(source.getTableName());
+         embeddings.resolve(source.getEmbeddingModelId(), classification, source.getTableName(), null, null);
+         Map<String, List<ChunkVector>> old = new LinkedHashMap<>();
+         vectors.chunks(source.getId()).forEach(v -> old.computeIfAbsent(v.pk(), key -> new ArrayList<>()).add(v));
+         Set<String> seenIds = new LinkedHashSet<>();
+         QTableMetaData table = runtime.getQInstance().getTable(source.getTableName());
+         QQueryFilter filter = filter(source);
+         filter.setOrderBys(List.of(new QFilterOrderBy(table.getPrimaryKeyField(), true)));
+         filter.setLimit(100);
+         UiApiToolbox.Context context = context();
+         for(Integer skip = 0; ; skip += 100)
+         {
+            checkKillSwitch();
+            filter.setSkip(skip);
+            UiApiResult page = runtime.getUiApiGateway().invoke(new UiApiCall("POST", "/table/{tableName}/query").withPathParam("tableName", source.getTableName())
+               .withSessionUUID(context.sessionUUID()).withUserTimezone(context.userTimezone()).withJsonBody("{\"filter\":" + JsonUtils.toJson(filter) + "}"));
+            if(!page.isSuccess())
+            {
+               throw (new QException("Index query answered HTTP " + page.status() + "; existing chunks were preserved."));
+            }
+            if(!(page.json() instanceof Map<?, ?> json) || !(json.get("records") instanceof List<?> records))
+            {
+               throw (new QException("The UI API returned an invalid query page; existing chunks were preserved."));
+            }
+            for(Object item : records)
+            {
+               if(!(item instanceof Map<?, ?> record) || !(record.get("values") instanceof Map<?, ?> raw) || raw.get(table.getPrimaryKeyField()) == null)
+               {
+                  throw (new QException("The UI API returned a record without its primary key."));
+               }
+               String pk = String.valueOf(raw.get(table.getPrimaryKeyField()));
+               if(!seenIds.add(pk))
+               {
+                  throw (new QException("Records moved during paging. Retry indexing; disappeared-record cleanup was skipped."));
+               }
+               seen++;
+               checkKillSwitch();
+               UiApiToolbox.Outcome read = read(context, source.getTableName(), pk);
+               if(read.getResult().getOutcome() != AiToolOutcome.OK)
+               {
+                  throw (new QException("Index record read was refused; existing chunks were preserved."));
+               }
+               List<ChunkVector> previous = old.getOrDefault(pk, List.of());
+               List<ChunkVector> next = new ArrayList<>();
+               List<String> texts = new ArrayList<>();
+               List<String> names = new ArrayList<>();
+               List<Integer> indexes = new ArrayList<>();
+               Map<String, String> values = values(read);
+               for(String field : fields(source))
+               {
+                  List<String> chunks = TextChunks.split(values.getOrDefault(field, ""), size(source), overlap(source));
+                  for(Integer i = 0; i < chunks.size(); i++)
+                  {
+                     Integer index = i;
+                     String hash = TextChunks.hash(chunks.get(i));
+                     ChunkVector same = previous.stream().filter(v -> v.field().equals(field) && v.chunkIndex().equals(index) && v.contentHash().equals(hash)).findFirst().orElse(null);
+                     if(same != null)
+                     {
+                        next.add(same);
+                        unchanged++;
+                     }
+                     else
+                     {
+                        texts.add(chunks.get(i));
+                        names.add(field);
+                        indexes.add(i);
+                     }
+                  }
+               }
+               List<float[]> generated = embeddings.embed(source.getEmbeddingModelId(), texts, classification, source.getTableName(), null, null);
+               Integer dimensions = runtime.getStore().listModels().stream().filter(m -> m.getId().equals(source.getEmbeddingModelId())).findFirst().orElseThrow().getEmbeddingDimensions();
+               for(Integer i = 0; i < generated.size(); i++)
+               {
+                  if(generated.get(i).length != dimensions)
+                  {
+                     throw (new QException("Embedding dimensions differ from the model registry; no record replacement was made."));
+                  }
+                  next.add(new ChunkVector(source.getId(), source.getTableName(), pk, names.get(i), indexes.get(i), TextChunks.hash(texts.get(i)), generated.get(i)));
+               }
+               embedded += generated.size();
+               removed += (int) previous.stream().filter(v -> next.stream().noneMatch(n -> n.field().equals(v.field()) && n.chunkIndex().equals(v.chunkIndex()))).count();
+               if(!texts.isEmpty() || next.size() != previous.size())
+               {
+                  vectors.replaceRecord(source.getId(), source.getTableName(), pk, next);
+               }
+            }
+            if(records.size() < 100)
+            {
+               break;
+            }
+         }
+         checkKillSwitch();
+         for(var entry : old.entrySet())
+         {
+            if(!seenIds.contains(entry.getKey()))
+            {
+               vectors.deleteRecord(source.getId(), source.getTableName(), entry.getKey());
+               removed += entry.getValue().size();
+            }
+         }
+      }
+      catch(QException e)
+      {
+         reason = e.getMessage();
+      }
+      catch(RuntimeException e)
+      {
+         reason = "Indexing failed. Check the source and vector store configuration.";
+      }
+      Integer chunkCount = source.getIndexedChunkCount();
+      try
+      {
+         chunkCount = vectors.count(source.getId()).intValue();
+      }
+      catch(QException e)
+      {
+         reason = "The vector store is unavailable; the indexed chunk count could not be refreshed.";
+      }
+      Result result = new Result(seen, embedded, unchanged, removed, reason == null ? 0 : 1, reason);
+      runtime.getStore().updateFields(AssistantRagSource.TABLE_NAME, new QRecord().withValue("id", source.getId())
+         .withValue("lastIndexedAt", runtime.getClock().instant()).withValue("lastIndexStatus", reason == null ? "OK" : "ERROR")
+         .withValue("lastIndexError", reason == null ? null : reason.substring(0, Math.min(500, reason.length())))
+         .withValue("indexedRecordCount", seen).withValue("indexedChunkCount", chunkCount));
+      return (result);
+   }
+
+
+
+   /*******************************************************************************
+    ** Clear is an admin record action, independent of the current model setup.
+    *******************************************************************************/
+   public synchronized void clear(AssistantRagSource source) throws QException
+   {
+      requireAdmin();
+      vectors.deleteSource(source.getId());
+      runtime.getStore().updateFields(AssistantRagSource.TABLE_NAME, new QRecord().withValue("id", source.getId()).withValue("indexedRecordCount", 0)
+         .withValue("indexedChunkCount", 0).withValue("lastIndexStatus", "CLEARED").withValue("lastIndexError", null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Only a real clicking administrator may launch manual indexing.
+    *******************************************************************************/
+   private static void requireAdmin() throws QException
+   {
+      if(QContext.getQSession() == null || !QContext.getQSession().hasPermission(AssistantQBitConfig.ADMIN_PERMISSION))
+      {
+         throw (new QException("Knowledge indexing requires assistant.admin."));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Stop between UI API reads as well as before every embedding dispatch.
+    *******************************************************************************/
+   private void checkKillSwitch() throws QException
+   {
+      if(runtime.getPolicyService().current(true).killSwitch())
+      {
+         throw (new QException("The assistant kill switch is on."));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Current caller context, never a stored or service session.
+    *******************************************************************************/
+   public UiApiToolbox.Context context() throws QException
+   {
+      return (new UiApiToolbox.Context(QContext.getQSession().getUuid(), QContext.getQSession().getValue("userTimezone"), runtime.getUiApiGateway(), runtime.getPolicyService().current(true).tools(), null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Configured text field names in stable order.
+    *******************************************************************************/
+   public static List<String> fields(AssistantRagSource source)
+   {
+      return (source.getFieldNames() == null ? List.of() : java.util.Arrays.stream(source.getFieldNames().split(",")).map(String::trim).filter(s -> !s.isEmpty()).distinct().toList());
+   }
+
+
+
+   /*******************************************************************************
+    ** Default chunk size.
+    *******************************************************************************/
+   public static Integer size(AssistantRagSource source)
+   {
+      return (source.getChunkChars() == null ? 1500 : source.getChunkChars());
+   }
+
+
+
+   /*******************************************************************************
+    ** Default overlap.
+    *******************************************************************************/
+   public static Integer overlap(AssistantRagSource source)
+   {
+      return (source.getChunkOverlapChars() == null ? 200 : source.getChunkOverlapChars());
+   }
+
+
+
+   /*******************************************************************************
+    ** Source filter is QQQ filter JSON, passed unchanged apart from paging/order.
+    *******************************************************************************/
+   private static QQueryFilter filter(AssistantRagSource source) throws QException
+   {
+      try
+      {
+         QQueryFilter filter = source.getFilterJson() == null || source.getFilterJson().isBlank() ? new QQueryFilter() : JsonUtils.toObject(source.getFilterJson(), QQueryFilter.class);
+         if(filter == null)
+         {
+            throw (new IllegalArgumentException("null filter"));
+         }
+         return (filter);
+      }
+      catch(Exception e)
+      {
+         throw (new QException("The query filter must be valid QQQ query filter JSON."));
+      }
+   }
+   /*******************************************************************************
+    ** Remove a deleted source without attempting to update its missing row.
+    *******************************************************************************/
+   public synchronized void remove(AssistantRagSource source) throws QException
+   {
+      requireAdmin();
+      vectors.deleteSource(source.getId());
+   }
+
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/TextChunks.java

@@ -0,0 +1,80 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+
+
+/*******************************************************************************
+ ** Deterministic overlapping chunks. Text exists only during indexing/retrieval.
+ *******************************************************************************/
+public final class TextChunks
+{
+   /*******************************************************************************
+    ** Utility class.
+    *******************************************************************************/
+   private TextChunks()
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Fixed character boundaries are reproduced on the fresh UI API value.
+    *******************************************************************************/
+   public static List<String> split(String text, Integer size, Integer overlap)
+   {
+      if(size == null || overlap == null || size < 1 || overlap < 0 || overlap >= size)
+      {
+         throw (new IllegalArgumentException("Chunk overlap must be nonnegative and smaller than chunk size."));
+      }
+      List<String> chunks = new ArrayList<>();
+      for(Integer start = 0; start < text.length(); start += size - overlap)
+      {
+         Integer end = Math.min(text.length(), start + size);
+         chunks.add(text.substring(start, end));
+         if(end == text.length())
+         {
+            break;
+         }
+      }
+      return (chunks);
+   }
+
+
+
+   /*******************************************************************************
+    ** SHA-256 permits unchanged detection without retaining any text.
+    *******************************************************************************/
+   public static String hash(String text)
+   {
+      try
+      {
+         return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(text.getBytes(StandardCharsets.UTF_8))));
+      }
+      catch(NoSuchAlgorithmException e)
+      {
+         throw (new IllegalStateException("SHA-256 is required by the Java runtime.", e));
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/rag/VectorStore.java

@@ -0,0 +1,76 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** Per-instance vectors only. Every hit must be re-read through the UI API.
+ *******************************************************************************/
+public interface VectorStore
+{
+   /*******************************************************************************
+    ** Source identity disambiguates field/chunk layouts on the same table.
+    *******************************************************************************/
+   record Hit(Integer source, String table, String pk, String field, Integer chunkIndex, Double score)
+   {
+   }
+
+   /*******************************************************************************
+    ** Upsert by source/table/key/field/chunk, without storing content.
+    *******************************************************************************/
+   void upsert(List<ChunkVector> chunks) throws QException;
+
+   /*******************************************************************************
+    ** Delete all chunks for a record within a source.
+    *******************************************************************************/
+   void deleteRecord(Integer source, String table, String pk) throws QException;
+
+   /*******************************************************************************
+    ** Remove a source's entire index.
+    *******************************************************************************/
+   void deleteSource(Integer source) throws QException;
+
+   /*******************************************************************************
+    ** Cosine similarity, highest first; an empty table set returns no hits.
+    *******************************************************************************/
+   List<Hit> search(float[] query, Integer k, Set<String> tables) throws QException;
+
+   /*******************************************************************************
+    ** Restrict ranking to compatible enabled sources, before applying the limit.
+    *******************************************************************************/
+   List<Hit> search(float[] query, Integer k, Set<String> tables, Set<Integer> sources) throws QException;
+
+   /*******************************************************************************
+    ** Count indexed chunks.
+    *******************************************************************************/
+   Long count(Integer source) throws QException;
+
+   /*******************************************************************************
+    ** Content-free reconciliation seam for unchanged and disappeared chunks.
+    *******************************************************************************/
+   List<ChunkVector> chunks(Integer source) throws QException;
+
+   /*******************************************************************************
+    ** Atomically replace one record, removing shortened or removed fields.
+    *******************************************************************************/
+   void replaceRecord(Integer source, String table, String pk, List<ChunkVector> chunks) throws QException;
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/KnowledgeTool.java

@@ -0,0 +1,227 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools;
+
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.AssistantRuntime;
+import com.kingsrook.qbits.assistant.classification.DataClassification;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.engine.TurnEngine;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
+import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
+import com.kingsrook.qbits.assistant.rag.EmbeddingService;
+import com.kingsrook.qbits.assistant.rag.RagService;
+import com.kingsrook.qbits.assistant.rag.TextChunks;
+import com.kingsrook.qbits.assistant.rag.VectorStore;
+import com.kingsrook.qqq.backend.core.actions.ai.AiRecordRef;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
+import com.kingsrook.qqq.backend.core.actions.ai.output.AiDataEnvelope;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+
+
+/*******************************************************************************
+ ** Permission-filtered semantic retrieval. Every candidate is re-read as the caller.
+ *******************************************************************************/
+public class KnowledgeTool
+{
+   public static final String NAME = "search_knowledge";
+   private final AssistantRuntime runtime;
+   private final RagService rag;
+   private final VectorStore vectors;
+
+
+
+   /*******************************************************************************
+    ** Bind the existing services; the vector store contains no record content.
+    *******************************************************************************/
+   public KnowledgeTool(AssistantRuntime runtime, RagService rag, VectorStore vectors)
+   {
+      this.runtime = runtime;
+      this.rag = rag;
+      this.vectors = vectors;
+   }
+
+
+
+   /*******************************************************************************
+    ** A source must be enabled and readable in the UI before offering the tool.
+    *******************************************************************************/
+   public Boolean offered(PolicyService.ToolPolicy policy, QSession session)
+   {
+      if(session == null || (policy != null && !policy.allows(NAME)))
+      {
+         return (false);
+      }
+      try
+      {
+         Set<String> tables = new LinkedHashSet<>();
+         for(AssistantRagSource source : rag.sources())
+         {
+            if(Boolean.TRUE.equals(source.getEnabled()) && runtime.getDataAccessService().forTable(source.getTableName()).mode().allowsRead() && rag.validate(source).isEmpty())
+            {
+               tables.add(source.getTableName());
+            }
+         }
+         if(tables.isEmpty())
+         {
+            return (false);
+         }
+         return (runtime.getAccessExplainer().explain(runtime.getUiApiGateway(), session.getUuid(), policy, null, null).stream()
+            .anyMatch(t -> tables.contains(t.table()) && t.uiRead() && t.mode().allowsRead()));
+      }
+      catch(QException e)
+      {
+         return (false);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Stable, small schema in the same sorted tool list as other tools.
+    *******************************************************************************/
+   public ModelToolDefinition definition()
+   {
+      return (new ModelToolDefinition().withName(NAME).withDescription("Search indexed application knowledge. Cite returned records using their links.").withStrict(false)
+         .withInputSchema(Map.of("type", "object", "properties", Map.of("query", Map.of("type", "string"),
+            "tables", Map.of("type", "array", "items", Map.of("type", "string")), "limit", Map.of("type", "integer", "minimum", 1, "maximum", 8)), "required", List.of("query"))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Expected denials disclose only a filtered count, never hit identifiers.
+    *******************************************************************************/
+   public AiToolResult execute(UiApiToolbox.Context context, String callId, Map<String, Object> input, DataClassification classification, Integer conversationId, Integer turnId)
+   {
+      AiToolResult result = new AiToolResult().withToolName(NAME).withToolCallId(callId);
+      Long started = System.nanoTime();
+      try
+      {
+         if(context.policy() != null && !context.policy().allows(NAME))
+         {
+            return (result.withOutcome(AiToolOutcome.DENIED).withSummary("Not allowed").withModelContent("The policy does not allow search_knowledge."));
+         }
+         if(input == null || !(input.get("query") instanceof String query) || query.isBlank() || query.length() > 8000
+            || (input.containsKey("tables") && (!(input.get("tables") instanceof List<?> requested) || requested.stream().anyMatch(t -> !(t instanceof String))))
+            || (input.containsKey("limit") && (!(input.get("limit") instanceof Number n) || n.doubleValue() != n.intValue() || n.intValue() < 1)))
+         {
+            return (result.withOutcome(AiToolOutcome.INVALID_INPUT).withSummary("Invalid search").withModelContent("Supply a nonempty query (at most 8000 characters), optional table names and a positive integer limit."));
+         }
+         Integer limit = input.get("limit") instanceof Number n ? Math.min(8, n.intValue()) : 5;
+         Set<?> requested = input.get("tables") instanceof List<?> list ? new LinkedHashSet<>(list) : Set.of();
+         Map<Integer, AssistantRagSource> sources = new LinkedHashMap<>();
+         Map<Integer, Set<String>> modelTables = new LinkedHashMap<>();
+         for(AssistantRagSource source : rag.sources())
+         {
+            if(Boolean.TRUE.equals(source.getEnabled()) && (requested.isEmpty() || requested.contains(source.getTableName()))
+               && runtime.getDataAccessService().forTable(source.getTableName()).mode().allowsRead() && rag.validate(source).isEmpty())
+            {
+               sources.put(source.getId(), source);
+               modelTables.computeIfAbsent(source.getEmbeddingModelId(), key -> new LinkedHashSet<>()).add(source.getTableName());
+            }
+         }
+         List<VectorStore.Hit> hits = new ArrayList<>();
+         EmbeddingService embeddings = new EmbeddingService(runtime);
+         for(var model : modelTables.entrySet())
+         {
+            float[] vector = embeddings.embed(model.getKey(), List.of(query), classification, String.join(", ", model.getValue()), conversationId, turnId).getFirst();
+            Set<Integer> compatible = new LinkedHashSet<>();
+            sources.values().stream().filter(s -> model.getKey().equals(s.getEmbeddingModelId())).forEach(s -> compatible.add(s.getId()));
+            for(VectorStore.Hit hit : vectors.search(vector, limit * 4, model.getValue(), compatible))
+            {
+               AssistantRagSource source = sources.get(hit.source());
+               if(source != null && model.getKey().equals(source.getEmbeddingModelId()) && source.getTableName().equals(hit.table()) && RagService.fields(source).contains(hit.field()))
+               {
+                  hits.add(hit);
+               }
+            }
+         }
+         hits.sort(Comparator.comparing(VectorStore.Hit::score).reversed());
+         List<Map<String, Object>> records = new ArrayList<>();
+         Set<List<String>> returned = new LinkedHashSet<>();
+         Integer filtered = 0;
+         for(VectorStore.Hit hit : hits)
+         {
+            if(records.size() >= limit)
+            {
+               break;
+            }
+            if(runtime.getPolicyService().current(true).killSwitch())
+            {
+               throw (new QException("The assistant kill switch is on."));
+            }
+            UiApiToolbox.Outcome read = rag.read(context, hit.table(), hit.pk());
+            if(read.getResult().getOutcome() == AiToolOutcome.DENIED || read.getResult().getOutcome() == AiToolOutcome.NOT_FOUND)
+            {
+               filtered++;
+               continue;
+            }
+            if(read.getResult().getOutcome() != AiToolOutcome.OK)
+            {
+               throw (new QException("A knowledge record could not be read through the UI API. Retry the search."));
+            }
+            Map<String, String> values = RagService.values(read);
+            AssistantRagSource source = sources.get(hit.source());
+            List<String> chunks = TextChunks.split(values.getOrDefault(hit.field(), ""), RagService.size(source), RagService.overlap(source));
+            if(hit.chunkIndex() >= chunks.size() || !returned.add(List.of(hit.table(), hit.pk(), hit.field(), String.valueOf(hit.chunkIndex()))))
+            {
+               filtered++;
+               continue;
+            }
+            String href = String.valueOf(((Map<?, ?>) read.getResult().getRender()).get("href"));
+            if(!TurnEngine.isAppHref(href))
+            {
+               filtered++;
+               continue;
+            }
+            String title = values.getOrDefault(source.getTitleFieldName(), hit.pk());
+            title = title.substring(0, Math.min(300, title.length()));
+            String snippet = chunks.get(hit.chunkIndex());
+            snippet = snippet.substring(0, Math.min(1500, snippet.length()));
+            records.add(Map.of("table", hit.table(), "recordId", hit.pk(), "title", title, "snippet", snippet, "score", hit.score(), "href", href));
+            result.getRecordRefs().add(new AiRecordRef().withTableName(hit.table()).withId(hit.pk()).withLabel(title).withHref(href));
+            result.getRedactions().addAll(read.getResult().getRedactions());
+         }
+         Map<String, Object> content = Map.of("results", records, "filtered", filtered);
+         return (result.withOutcome(AiToolOutcome.OK).withRowCount(records.size()).withSummary("Found " + records.size() + " knowledge results; filtered " + filtered)
+            .withPermissionCheck("uiApi:GET /table/{tableName}/{primaryKey}").withModelContent(AiDataEnvelope.wrap(NAME, null, JsonUtils.toJson(content))));
+      }
+      catch(QException e)
+      {
+         return (result.withOutcome(AiToolOutcome.ERROR).withSummary("Knowledge search failed").withModelContent(e.getMessage()));
+      }
+      catch(RuntimeException e)
+      {
+         return (result.withOutcome(AiToolOutcome.ERROR).withSummary("Knowledge search failed").withModelContent("Knowledge search could not complete. Check the source and embedding configuration."));
+      }
+      finally
+      {
+         result.withDurationMs((System.nanoTime() - started) / 1_000_000);
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/UiApiToolbox.java

@@ -203,6 +203,21 @@ public class UiApiToolbox
 
 
 
+      /*******************************************************************************
+       ** An outcome built by another tool family (e.g. the bulk-load tools), so
+       ** TurnEngine and the confirm route log it the same way.
+       *******************************************************************************/
+      public static Outcome of(AiToolResult result, String operation, Integer httpStatus, Integer pendingActionId)
+      {
+         Outcome outcome = new Outcome(result);
+         outcome.operation = operation;
+         outcome.httpStatus = httpStatus;
+         outcome.pendingActionId = pendingActionId;
+         return (outcome);
+      }
+
+
+
       /*******************************************************************************
        ** Getter for result
        *******************************************************************************/

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/bulkload/BulkLoadRunner.java

@@ -0,0 +1,57 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools.bulkload;
+
+
+import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+
+
+/*******************************************************************************
+ ** How bulk_load_run starts a load (design spec #1171 section 3.4): one seam,
+ ** so the run can move from today's path to durable jobs without the tool
+ ** changing. Every implementation goes through the /qqq/v1 UI API as the user.
+ *******************************************************************************/
+public interface BulkLoadRunner
+{
+   /*******************************************************************************
+    ** A load to start: the target table, the upload, and the saved profile.
+    *******************************************************************************/
+   record Request(String sessionUUID, String userTimezone, UiApiGateway gateway, String tableName, String uploadRef, Integer savedBulkLoadProfileId)
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** What started: status uses the durable job status names (jobs_job.status:
+    ** AWAITING_CONFIRMATION, QUEUED, RUNNING, COMPLETED,
+    ** COMPLETED_WITH_ERRORS, FAILED, ...); jobId when it runs as a job,
+    ** processUUID when it ran synchronously; a short summary; the HTTP status.
+    *******************************************************************************/
+   record RunHandle(String status, Long jobId, String processUUID, String summary, String operation, Integer httpStatus)
+   {
+   }
+
+
+
+   /*******************************************************************************
+    ** Start the load. Throws when it cannot start at all (the handle's status
+    ** reports a load that started and failed).
+    *******************************************************************************/
+   RunHandle start(Request request) throws QException;
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/bulkload/BulkLoadTools.java

@@ -0,0 +1,883 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools.bulkload;
+
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.access.DataAccessService;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.provider.ModelToolDefinition;
+import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
+import com.kingsrook.qqq.backend.core.actions.ai.output.AiDataEnvelope;
+import com.kingsrook.qqq.backend.core.actions.ai.tools.AiJsonSchema;
+import com.kingsrook.qqq.backend.core.actions.ai.tools.AiToolException;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qqq.backend.core.utils.StringUtils;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+import org.json.JSONArray;
+import org.json.JSONObject;
+
+
+/*******************************************************************************
+ ** The assistant's bulk-load tools (design spec #1171 section 3.4): use and
+ ** create saved bulk load profiles from a file the user attached in the chat.
+ ** Every call goes through the /qqq/v1 UI API with the user's own session, so
+ ** table and process permissions, record locks and the PHI rule are enforced
+ ** by the server, never by the model:
+ ** - bulk_profiles_list / bulk_profile_describe: the user's (and shared)
+ **   saved profiles, through the savedBulkLoadProfile table (record locks);
+ ** - bulk_file_attach: an upload the panel already made (by its ref), with
+ **   its column profiles (samples only where the server allows);
+ ** - bulk_mapping_suggest: core's suggester, then the model refines it;
+ ** - bulk_load_dry_run: the table's real bulk-insert validation, no store;
+ ** - bulk_profile_save (write): storeSavedBulkLoadProfile, and shares through
+ **   insertSharedRecord, after the user confirms;
+ ** - bulk_load_run (write): through {@link BulkLoadRunner}, after the user
+ **   confirms (and, as a durable job, again on its job record).
+ **
+ ** The AI route header (X-QQQ-AI-Route: hipaa|standard) is set here from the
+ ** turn's provider, never from the model's input. Tool-call logs keep
+ ** profiles and counts, never sample values (see {@link #forLog}).
+ **
+ ** STAND-IN(core=#1179, removeBy=RC.8): the profile, suggestMapping and
+ ** validate operations ship in a QQQ RC after 4.1.0-RC.5; until then the
+ ** tools report that the server does not offer them.
+ *******************************************************************************/
+public class BulkLoadTools
+{
+   public static final String LIST     = "bulk_profiles_list";
+   public static final String DESCRIBE = "bulk_profile_describe";
+   public static final String ATTACH   = "bulk_file_attach";
+   public static final String SUGGEST  = "bulk_mapping_suggest";
+   public static final String DRY_RUN  = "bulk_load_dry_run";
+   public static final String SAVE     = "bulk_profile_save";
+   public static final String RUN      = "bulk_load_run";
+
+   public static final Set<String> NAMES       = Set.of(LIST, DESCRIBE, ATTACH, SUGGEST, DRY_RUN, SAVE, RUN);
+   public static final Set<String> WRITE_TOOLS = Set.of(SAVE, RUN);
+
+   public static final String PROFILE_TABLE = "savedBulkLoadProfile";
+   public static final String ROUTE_HEADER  = "X-QQQ-AI-Route";
+
+   private static final Set<String> LOG_STRIPPED_KEYS = Set.of("sampleValues", "bulletsOfText", "valueMappings");
+
+   private final QInstance         qInstance;
+   private final DataAccessService dataAccess;
+   private final BulkLoadRunner    runner;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public BulkLoadTools(QInstance qInstance, DataAccessService dataAccess, BulkLoadRunner runner)
+   {
+      this.qInstance = qInstance;
+      this.dataAccess = dataAccess;
+      this.runner = runner == null ? new JobsLaunchBulkLoadRunner() : runner;
+   }
+
+
+
+   /*******************************************************************************
+    ** Offered when the instance has saved bulk load profiles.
+    *******************************************************************************/
+   public boolean offered()
+   {
+      return (qInstance != null && qInstance.getTable(PROFILE_TABLE) != null);
+   }
+
+
+
+   /*******************************************************************************
+    ** The definitions the policy allows (writes only when anyWritable).
+    *******************************************************************************/
+   public List<ModelToolDefinition> definitions(PolicyService.ToolPolicy policy, boolean anyWritable)
+   {
+      List<ModelToolDefinition> definitions = new ArrayList<>();
+      if(!offered())
+      {
+         return (definitions);
+      }
+
+      Map<String, Object> table   = AiJsonSchema.string("The table to load into (from list_tables).");
+      Map<String, Object> upload  = AiJsonSchema.string("The upload ref of the file the user attached (from the conversation's attachments).");
+      Map<String, Object> profile = new LinkedHashMap<>(Map.of("type", "object", "description", """
+         A v1 bulk load profile: {"version":"v1","layout":"FLAT","hasHeaderRow":true,"fieldList":[{"fieldName":"...","columnIndex":0, \
+         "headerName":"...","defaultValue":null,"doValueMapping":false,"valueMappings":{"file value":"field value"}}]}."""));
+
+      add(definitions, policy, LIST, "List the saved bulk load profiles (the user's own and those shared with them) for a table.",
+         AiJsonSchema.object(AiJsonSchema.properties("tableName", table), List.of("tableName")));
+      add(definitions, policy, DESCRIBE, "Read one saved bulk load profile: its table, owner and column mapping.",
+         AiJsonSchema.object(AiJsonSchema.properties("id", AiJsonSchema.integer("The profile id (from bulk_profiles_list).")), List.of("id")));
+      add(definitions, policy, ATTACH, """
+         Look at a file the user attached for a bulk load: whether row 1 is a header, and each column's type, blanks, distinct count, \
+         pattern and (only where the server allows) a few sample values.""",
+         AiJsonSchema.object(AiJsonSchema.properties("uploadRef", upload, "tableName", table), List.of("uploadRef", "tableName")));
+      add(definitions, policy, SUGGEST, """
+         Suggest a bulk load profile for an attached file: the server's header matching, with the columns it left unmapped, required \
+         fields it did not fill, and fields whose values need mapping. Refine the profile yourself from the headers and column \
+         profiles, then check it with bulk_load_dry_run.""",
+         AiJsonSchema.object(AiJsonSchema.properties("uploadRef", upload, "tableName", table), List.of("uploadRef", "tableName")));
+      add(definitions, policy, DRY_RUN, """
+         Dry-run a bulk load profile over an attached file through the table's real bulk insert validation. Nothing is stored. \
+         Returns counts, summary lines and error groups with row numbers. SAMPLE checks the first 100 rows; FULL the whole file. \
+         Fix the profile and run it again until there are no errors (or explain the rest to the user).""",
+         AiJsonSchema.object(AiJsonSchema.properties("uploadRef", upload, "tableName", table, "profile", profile,
+            "mode", AiJsonSchema.enumOf(List.of("SAMPLE", "FULL"), "SAMPLE (default) or FULL.")), List.of("uploadRef", "tableName", "profile")));
+
+      if(anyWritable)
+      {
+         Map<String, Object> share = AiJsonSchema.object(AiJsonSchema.properties(
+            "audienceType", AiJsonSchema.enumOf(List.of("user", "group"), "Share with a user (email) or a group."),
+            "audienceId", AiJsonSchema.string("The user's email or the group's name."),
+            "scope", AiJsonSchema.enumOf(List.of("READ_ONLY", "READ_WRITE"), "The share's scope (default READ_ONLY).")), List.of("audienceType", "audienceId"));
+         add(definitions, policy, SAVE, """
+            Save a bulk load profile (after a clean dry run) under a name, optionally sharing it. The user confirms before it is saved; \
+            it is saved as the user, and shares are limited to what the user may share.""",
+            AiJsonSchema.object(AiJsonSchema.properties("tableName", table, "label", AiJsonSchema.string("The profile's name."), "profile", profile,
+               "shares", AiJsonSchema.array(share, "Optional shares.")), List.of("tableName", "label", "profile")));
+         add(definitions, policy, RUN, """
+            Load an attached file into a table with a saved profile. The user confirms first; a large load runs as a background job that \
+            the user starts from its job record.""",
+            AiJsonSchema.object(AiJsonSchema.properties("tableName", table, "uploadRef", upload, "profileId", AiJsonSchema.integer("The saved profile id.")),
+               List.of("tableName", "uploadRef", "profileId")));
+      }
+      return (definitions);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void add(List<ModelToolDefinition> definitions, PolicyService.ToolPolicy policy, String name, String description, Map<String, Object> schema)
+   {
+      if(policy == null || policy.allows(name))
+      {
+         definitions.add(new ModelToolDefinition().withName(name).withDescription(description).withInputSchema(schema).withStrict(false));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Run a tool. hipaaRoute is the turn's provider's HIPAA eligibility (sent
+    ** as X-QQQ-AI-Route; the model cannot set it).
+    *******************************************************************************/
+   public UiApiToolbox.Outcome execute(UiApiToolbox.Context context, String toolCallId, String toolName, Map<String, Object> input, boolean hipaaRoute)
+   {
+      AiToolResult        result  = new AiToolResult().withToolCallId(toolCallId).withToolName(toolName);
+      Map<String, Object> in      = input == null ? Map.of() : input;
+      Call                call    = new Call(context, hipaaRoute);
+      Integer             actionId = null;
+      long                started = System.nanoTime();
+      try
+      {
+         if(context.policy() != null && !context.policy().allows(toolName))
+         {
+            deny(result, "policy:tool", "The administrator has not allowed the " + toolName + " tool.");
+            return (UiApiToolbox.Outcome.of(result, null, null, null));
+         }
+
+         String tableName = in.get("tableName") instanceof String t ? t.strip() : null;
+         if(tableName != null)
+         {
+            result.setTableName(tableName);
+            if(!dataAccess.forTable(tableName).mode().allowsRead())
+            {
+               deny(result, "dataAccess:" + dataAccess.forTable(tableName).mode(), "The assistant may not use the " + tableName + " table.");
+               return (UiApiToolbox.Outcome.of(result, null, null, null));
+            }
+         }
+
+         switch(toolName)
+         {
+            case LIST -> list(call, result, required(in, "tableName"));
+            case DESCRIBE -> describe(call, result, requiredInteger(in, "id"));
+            case ATTACH -> attach(call, result, required(in, "tableName"), required(in, "uploadRef"));
+            case SUGGEST -> suggest(call, result, required(in, "tableName"), required(in, "uploadRef"));
+            case DRY_RUN -> dryRun(call, result, required(in, "tableName"), required(in, "uploadRef"), requiredMap(in, "profile"), in.get("mode") instanceof String m ? m : "SAMPLE");
+            case SAVE, RUN -> actionId = propose(context, result, toolName, required(in, "tableName"), in);
+            default -> throw (AiToolException.invalidInput("Unknown tool \"" + toolName + "\"."));
+         }
+      }
+      catch(AiToolException e)
+      {
+         result.withOutcome(e.getOutcome() == null ? AiToolOutcome.INVALID_INPUT : e.getOutcome()).withSummary(shortText(e.getMessage())).withModelContent(e.getMessage());
+      }
+      catch(QException | RuntimeException e)
+      {
+         result.withOutcome(AiToolOutcome.ERROR).withSummary("Failed").withModelContent("The tool failed: " + shortText(e.getMessage()));
+      }
+      finally
+      {
+         result.setDurationMs((System.nanoTime() - started) / 1_000_000);
+      }
+      return (UiApiToolbox.Outcome.of(result, call.operation, call.httpStatus, actionId));
+   }
+
+
+
+   /*******************************************************************************
+    ** Run a confirmed write (from a pending action) with the confirming user's
+    ** session.
+    *******************************************************************************/
+   public UiApiToolbox.Outcome executeConfirmed(UiApiToolbox.Context context, String toolCallId, String toolName, String tableName, Map<String, Object> values)
+   {
+      AiToolResult result  = new AiToolResult().withToolCallId(toolCallId).withToolName(toolName).withTableName(tableName);
+      Call         call    = new Call(context, false);
+      long         started = System.nanoTime();
+      try
+      {
+         if(!WRITE_TOOLS.contains(toolName))
+         {
+            throw (AiToolException.invalidInput("Not a write tool: " + toolName));
+         }
+         DataAccessService.Decision decision = dataAccess.forTable(SAVE.equals(toolName) ? PROFILE_TABLE : tableName);
+         if(!decision.mode().allowsWrite())
+         {
+            deny(result, "dataAccess:" + decision.mode(), "The assistant may not write here (data access " + decision.mode() + ", " + decision.source() + ").");
+            return (UiApiToolbox.Outcome.of(result, null, null, null));
+         }
+         if(SAVE.equals(toolName))
+         {
+            save(call, result, tableName, values);
+         }
+         else
+         {
+            run(call, result, context, tableName, values);
+         }
+      }
+      catch(AiToolException e)
+      {
+         result.withOutcome(e.getOutcome() == null ? AiToolOutcome.INVALID_INPUT : e.getOutcome()).withSummary(shortText(e.getMessage())).withModelContent(e.getMessage());
+      }
+      catch(QException | RuntimeException e)
+      {
+         result.withOutcome(AiToolOutcome.ERROR).withSummary("Failed").withModelContent("The write failed: " + shortText(e.getMessage()));
+      }
+      finally
+      {
+         result.setDurationMs((System.nanoTime() - started) / 1_000_000);
+      }
+      return (UiApiToolbox.Outcome.of(result, call.operation, call.httpStatus, null));
+   }
+
+
+
+   ////////////////////////////////////////////////////////////////////////////
+   // read tools                                                             //
+   ////////////////////////////////////////////////////////////////////////////
+
+
+
+   /*******************************************************************************
+    ** POST /table/savedBulkLoadProfile/query: the profiles the record locks let
+    ** the user see, for one table.
+    *******************************************************************************/
+   private void list(Call call, AiToolResult result, String tableName) throws QException
+   {
+      Map<String, Object> filter = Map.of(
+         "criteria", List.of(Map.of("fieldName", "tableName", "operator", "EQUALS", "values", List.of(tableName))),
+         "orderBys", List.of(Map.of("fieldName", "label", "isAscending", true)));
+      UiApiResult response = call.invoke(new UiApiCall("POST", "/table/{tableName}/query").withPathParam("tableName", PROFILE_TABLE)
+         .withJsonBody(Map.of("filter", filter)));
+      if(response == null || !response.isSuccess())
+      {
+         fail(result, call, response);
+         return;
+      }
+
+      List<Map<String, Object>> rows = new ArrayList<>();
+      for(Object record : list(response.json().get("records")))
+      {
+         Map<String, Object> values = map(map(record).get("values"));
+         Map<String, Object> row    = new LinkedHashMap<>();
+         row.put("id", values.get("id"));
+         row.put("label", values.get("label"));
+         row.put("owner", values.get("userId"));
+         row.put("fieldCount", fieldList(values.get("mappingJson")).size());
+         rows.add(row);
+      }
+      result.withOutcome(AiToolOutcome.OK).withRowCount(rows.size()).withSummary("Found " + rows.size() + " saved profile" + (rows.size() == 1 ? "" : "s"))
+         .withPermissionCheck(call.permissionCheck())
+         .withModelContent(AiDataEnvelope.wrap(LIST, PROFILE_TABLE, JsonUtils.toJson(Map.of("tableName", tableName, "profiles", rows))));
+   }
+
+
+
+   /*******************************************************************************
+    ** GET /table/savedBulkLoadProfile/{id}: one profile (when the locks allow).
+    *******************************************************************************/
+   private void describe(Call call, AiToolResult result, Integer id) throws QException
+   {
+      UiApiResult response = call.invoke(new UiApiCall("GET", "/table/{tableName}/{primaryKey}").withPathParam("tableName", PROFILE_TABLE).withPathParam("primaryKey", id));
+      if(response == null || !response.isSuccess())
+      {
+         fail(result, call, response);
+         return;
+      }
+      Map<String, Object> values = map(map(response.json().get("record")).get("values"));
+      Map<String, Object> out    = new LinkedHashMap<>();
+      out.put("id", values.get("id"));
+      out.put("label", values.get("label"));
+      out.put("tableName", values.get("tableName"));
+      out.put("owner", values.get("userId"));
+      out.put("profile", values.get("mappingJson") == null ? null : new JSONObject(String.valueOf(values.get("mappingJson"))).toMap());
+      result.setTableName(String.valueOf(values.get("tableName")));
+      result.withOutcome(AiToolOutcome.OK).withRowCount(1).withSummary("Read profile " + values.get("label"))
+         .withPermissionCheck(call.permissionCheck())
+         .withModelContent(AiDataEnvelope.wrap(DESCRIBE, PROFILE_TABLE, JsonUtils.toJson(out)));
+   }
+
+
+
+   /*******************************************************************************
+    ** GET /bulkLoad/uploads/{uploadRef}/profile.
+    *******************************************************************************/
+   private void attach(Call call, AiToolResult result, String tableName, String uploadRef) throws QException
+   {
+      UiApiResult response = call.invoke(new UiApiCall("GET", "/bulkLoad/uploads/{uploadRef}/profile").withPathParam("uploadRef", uploadRef)
+         .withQueryParam("tableName", tableName).withQueryParam("sampleCount", 5));
+      if(response == null || !response.isSuccess())
+      {
+         fail(result, call, response);
+         return;
+      }
+      Map<String, Object> json    = response.json();
+      int                 columns = list(json.get("columns")).size();
+      result.withOutcome(AiToolOutcome.OK).withRowCount(columns).withSummary("Read " + json.get("fileName") + ": " + columns + " columns")
+         .withPermissionCheck(call.permissionCheck())
+         .withModelContent(AiDataEnvelope.wrap(ATTACH, tableName, response.body()));
+   }
+
+
+
+   /*******************************************************************************
+    ** POST /bulkLoad/{tableName}/suggestMapping.
+    *******************************************************************************/
+   private void suggest(Call call, AiToolResult result, String tableName, String uploadRef) throws QException
+   {
+      UiApiResult response = call.invoke(new UiApiCall("POST", "/bulkLoad/{tableName}/suggestMapping").withPathParam("tableName", tableName)
+         .withJsonBody(Map.of("uploadRef", uploadRef)));
+      if(response == null || !response.isSuccess())
+      {
+         fail(result, call, response);
+         return;
+      }
+      Map<String, Object> json   = response.json();
+      int                 mapped = list(map(json.get("profile")).get("fieldList")).size();
+      result.withOutcome(AiToolOutcome.OK).withRowCount(mapped).withSummary("Suggested a mapping of " + mapped + " field" + (mapped == 1 ? "" : "s"))
+         .withPermissionCheck(call.permissionCheck())
+         .withModelContent(AiDataEnvelope.wrap(SUGGEST, tableName, response.body()));
+   }
+
+
+
+   /*******************************************************************************
+    ** POST /bulkLoad/{tableName}/validate (nothing is stored).
+    *******************************************************************************/
+   private void dryRun(Call call, AiToolResult result, String tableName, String uploadRef, Map<String, Object> profile, String mode) throws QException
+   {
+      UiApiResult response = call.invoke(new UiApiCall("POST", "/bulkLoad/{tableName}/validate").withPathParam("tableName", tableName)
+         .withJsonBody(Map.of("uploadRef", uploadRef, "profile", profile, "mode", "FULL".equalsIgnoreCase(mode) ? "FULL" : "SAMPLE"))
+         .withTimeout(Duration.ofSeconds(120)));
+      if(response == null || !response.isSuccess())
+      {
+         fail(result, call, response);
+         return;
+      }
+      Map<String, Object> json   = response.json();
+      Integer             ok     = ValueUtils.getValueAsInteger(json.get("okCount"));
+      Integer             errors = ValueUtils.getValueAsInteger(json.get("errorCount"));
+      result.withOutcome(AiToolOutcome.OK).withRowCount(ok)
+         .withSummary("Dry run (" + json.get("mode") + "): " + ok + " ok, " + errors + " with errors")
+         .withPermissionCheck(call.permissionCheck())
+         .withRender(render("bulk_load_dry_run", tableName, json))
+         .withModelContent(AiDataEnvelope.wrap(DRY_RUN, tableName, response.body()));
+   }
+
+
+
+   ////////////////////////////////////////////////////////////////////////////
+   // write tools                                                            //
+   ////////////////////////////////////////////////////////////////////////////
+
+
+
+   /*******************************************************************************
+    ** Propose a save or a run as a pending action (the user confirms in the
+    ** panel; nothing happens until then).
+    *******************************************************************************/
+   private Integer propose(UiApiToolbox.Context context, AiToolResult result, String toolName, String tableName, Map<String, Object> in) throws QException
+   {
+      DataAccessService.Decision decision = dataAccess.forTable(SAVE.equals(toolName) ? PROFILE_TABLE : tableName);
+      if(!decision.mode().allowsWrite())
+      {
+         deny(result, "dataAccess:" + decision.mode(), "The assistant may only read here (data access " + decision.mode() + ", " + decision.source() + "). Writes need READ_WRITE.");
+         return (null);
+      }
+
+      Map<String, Object> values  = new LinkedHashMap<>();
+      String              summary;
+      if(SAVE.equals(toolName))
+      {
+         String label = required(in, "label");
+         values.put("label", label);
+         values.put("profile", requiredMap(in, "profile"));
+         if(in.get("shares") instanceof List<?> shares && !shares.isEmpty())
+         {
+            values.put("shares", shares);
+         }
+         summary = "Save the bulk load profile \"" + label + "\" for " + tableName + (values.containsKey("shares") ? " and share it" : "");
+      }
+      else
+      {
+         values.put("uploadRef", required(in, "uploadRef"));
+         values.put("profileId", requiredInteger(in, "profileId"));
+         summary = "Load the attached file into " + tableName + " with saved profile " + values.get("profileId");
+      }
+
+      if(context.pendingActions() == null)
+      {
+         throw (new QException("The " + toolName + " action could not be proposed."));
+      }
+      String  operation = SAVE.equals(toolName) ? "POST /processes/{processName}/init" : "POST " + JobsLaunchBulkLoadRunner.LAUNCH_OPERATION_PATH;
+      Integer actionId  = context.pendingActions().propose(new UiApiToolbox.ProposedWrite(result.getToolCallId(), toolName, operation, tableName, null, values, summary));
+      if(actionId == null)
+      {
+         throw (new QException("The " + toolName + " action could not be proposed."));
+      }
+
+      Map<String, Object> render = new LinkedHashMap<>();
+      render.put("type", "pending_action");
+      render.put("actionId", actionId);
+      render.put("toolName", toolName);
+      render.put("tableName", tableName);
+      render.put("summary", summary);
+      result.withOutcome(AiToolOutcome.OK).withSummary("Waiting for your confirmation").withRender(render)
+         .withModelContent("Proposed (pending action " + actionId + "): " + summary + ". Nothing happens until the user confirms it in the panel.");
+      return (actionId);
+   }
+
+
+
+   /*******************************************************************************
+    ** Confirmed save: storeSavedBulkLoadProfile, then insertSharedRecord per
+    ** share (core lets only the owner share).
+    *******************************************************************************/
+   private void save(Call call, AiToolResult result, String tableName, Map<String, Object> values) throws QException
+   {
+      Map<String, Object> processValues = new LinkedHashMap<>();
+      processValues.put("tableName", tableName);
+      processValues.put("label", values.get("label"));
+      processValues.put("mappingJson", JsonUtils.toJson(values.get("profile")));
+      processValues.put("isBulkEdit", false);
+      UiApiResult response = call.invoke(new UiApiCall("POST", "/processes/{processName}/init").withPathParam("processName", "storeSavedBulkLoadProfile")
+         .withFormParam("values", JsonUtils.toJson(processValues)));
+      String error = processError(response);
+      if(error != null)
+      {
+         result.withOutcome(response == null || response.isSuccess() ? AiToolOutcome.ERROR : outcomeFor(response.status())).withSummary(shortText(error)).withPermissionCheck(call.permissionCheck())
+            .withModelContent("The profile was not saved: " + error);
+         return;
+      }
+      Integer id = ValueUtils.getValueAsInteger(map(map(list(map(response.json().get("values")).get("savedBulkLoadProfileList")).stream().findFirst().orElse(null)).get("values")).get("id"));
+
+      List<String> shared = new ArrayList<>();
+      List<String> failed = new ArrayList<>();
+      for(Object share : list(values.get("shares")))
+      {
+         Map<String, Object> s = map(share);
+         Map<String, Object> shareValues = new LinkedHashMap<>();
+         shareValues.put("tableName", PROFILE_TABLE);
+         shareValues.put("recordId", id);
+         shareValues.put("audienceType", s.get("audienceType"));
+         shareValues.put("audienceId", s.get("audienceId"));
+         shareValues.put("scopeId", s.get("scope") == null ? "READ_ONLY" : s.get("scope"));
+         UiApiResult shareResponse = call.invoke(new UiApiCall("POST", "/processes/{processName}/init").withPathParam("processName", "insertSharedRecord")
+            .withFormParam("values", JsonUtils.toJson(shareValues)));
+         String shareError = processError(shareResponse);
+         (shareError == null ? shared : failed).add(s.get("audienceType") + " " + s.get("audienceId") + (shareError == null ? "" : " (" + shortText(shareError) + ")"));
+      }
+
+      String summary = "Saved profile \"" + values.get("label") + "\" (id " + id + ")" + (shared.isEmpty() ? "" : ", shared with " + String.join(", ", shared))
+         + (failed.isEmpty() ? "" : "; not shared with " + String.join(", ", failed));
+      result.withOutcome(AiToolOutcome.OK).withRowCount(1).withSummary(shortText(summary)).withPermissionCheck(call.permissionCheck())
+         .withModelContent(JsonUtils.toJson(Map.of("id", id == null ? "" : id, "label", values.get("label"), "shared", shared, "notShared", failed)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Confirmed run: through the runner seam.
+    *******************************************************************************/
+   private void run(Call call, AiToolResult result, UiApiToolbox.Context context, String tableName, Map<String, Object> values) throws QException
+   {
+      BulkLoadRunner.RunHandle handle = runner.start(new BulkLoadRunner.Request(context.sessionUUID(), context.userTimezone(), context.gateway(), tableName,
+         String.valueOf(values.get("uploadRef")), ValueUtils.getValueAsInteger(values.get("profileId"))));
+      call.operation = handle.operation();
+      call.httpStatus = handle.httpStatus();
+      boolean ok = !"FAILED".equals(handle.status()) && !"NOT_AVAILABLE".equals(handle.status());
+      Map<String, Object> out = new LinkedHashMap<>();
+      out.put("status", handle.status());
+      out.put("jobId", handle.jobId());
+      out.put("processUUID", handle.processUUID());
+      out.put("summary", handle.summary());
+      result.withOutcome(ok ? AiToolOutcome.OK : AiToolOutcome.ERROR).withSummary(shortText(handle.summary())).withPermissionCheck(call.permissionCheck())
+         .withModelContent(JsonUtils.toJson(out));
+   }
+
+
+
+   ////////////////////////////////////////////////////////////////////////////
+   // helpers                                                                //
+   ////////////////////////////////////////////////////////////////////////////
+
+
+
+   /*******************************************************************************
+    ** A tool's input or result as kept in the tool-call log: profiles and
+    ** counts, never sample values, example bullets or value mappings.
+    *******************************************************************************/
+   public static String forLog(String json)
+   {
+      if(json == null)
+      {
+         return (null);
+      }
+      try
+      {
+         int start = json.indexOf('{');
+         int end   = json.lastIndexOf('}');
+         if(start < 0 || end < start)
+         {
+            return (json);
+         }
+         JSONObject object = new JSONObject(json.substring(start, end + 1));
+         strip(object);
+         return (object.toString());
+      }
+      catch(RuntimeException e)
+      {
+         return ("{\"note\":\"not logged\"}");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void strip(Object node)
+   {
+      if(node instanceof JSONObject object)
+      {
+         for(String key : new ArrayList<>(object.keySet()))
+         {
+            if(LOG_STRIPPED_KEYS.contains(key))
+            {
+               object.remove(key);
+            }
+            else
+            {
+               strip(object.get(key));
+            }
+         }
+      }
+      else if(node instanceof JSONArray array)
+      {
+         for(int i = 0; i < array.length(); i++)
+         {
+            strip(array.get(i));
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A compact render for the panel (counts and error groups, no values).
+    *******************************************************************************/
+   private static Map<String, Object> render(String type, String tableName, Map<String, Object> json)
+   {
+      Map<String, Object> render = new LinkedHashMap<>();
+      render.put("type", type);
+      render.put("tableName", tableName);
+      render.put("mode", json.get("mode"));
+      render.put("okCount", json.get("okCount"));
+      render.put("errorCount", json.get("errorCount"));
+      List<Map<String, Object>> errors = new ArrayList<>();
+      for(Object error : list(json.get("errors")))
+      {
+         Map<String, Object> e = map(error);
+         errors.add(new LinkedHashMap<>(Map.of("rows", String.valueOf(e.get("rowNumbers")), "field", String.valueOf(e.get("fieldName")), "message", String.valueOf(e.get("message")))));
+      }
+      render.put("errors", errors);
+      return (render);
+   }
+
+
+
+   /*******************************************************************************
+    ** The error of a process init call (HTTP or the process's ERROR type), or
+    ** null when it completed.
+    *******************************************************************************/
+   private static String processError(UiApiResult response)
+   {
+      if(response == null)
+      {
+         return ("No response.");
+      }
+      if(!response.isSuccess())
+      {
+         return (response.errorMessage() == null ? "HTTP " + response.status() : response.errorMessage());
+      }
+      Map<String, Object> json = response.json();
+      if("ERROR".equals(json.get("type")))
+      {
+         return (String.valueOf(json.getOrDefault("userFacingError", json.getOrDefault("error", "The process failed."))));
+      }
+      return (null);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void fail(AiToolResult result, Call call, UiApiResult response)
+   {
+      if(response == null)
+      {
+         result.withOutcome(AiToolOutcome.ERROR).withSummary("Not available")
+            .withModelContent("This server does not offer " + call.operation + " yet (it needs a newer QQQ). Use the table's Bulk Load screen instead.");
+         return;
+      }
+      AiToolOutcome kind    = outcomeFor(response.status());
+      String        message = response.errorMessage() == null ? "HTTP " + response.status() : response.errorMessage();
+      result.withOutcome(kind).withSummary(kind == AiToolOutcome.DENIED ? "Not allowed" : shortText(message)).withPermissionCheck(call.permissionCheck())
+         .withModelContent(call.operation + " answered HTTP " + response.status() + ": " + shortText(message));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static AiToolOutcome outcomeFor(int status)
+   {
+      return (switch(status)
+      {
+         case 401, 403 -> AiToolOutcome.DENIED;
+         case 404 -> AiToolOutcome.NOT_FOUND;
+         case 400, 409, 422 -> AiToolOutcome.INVALID_INPUT;
+         default -> AiToolOutcome.ERROR;
+      });
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static void deny(AiToolResult result, String check, String message)
+   {
+      result.withOutcome(AiToolOutcome.DENIED).withSummary("Not allowed").withPermissionCheck(check).withModelContent(message);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static String required(Map<String, Object> in, String name) throws AiToolException
+   {
+      if(in.get(name) instanceof String s && StringUtils.hasContent(s))
+      {
+         return (s.strip());
+      }
+      throw (AiToolException.invalidInput(name + " is required."));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static Integer requiredInteger(Map<String, Object> in, String name) throws AiToolException
+   {
+      try
+      {
+         Integer value = ValueUtils.getValueAsInteger(in.get(name));
+         if(value != null)
+         {
+            return (value);
+         }
+      }
+      catch(RuntimeException e)
+      {
+         // fall through
+      }
+      throw (AiToolException.invalidInput(name + " must be a number."));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static Map<String, Object> requiredMap(Map<String, Object> in, String name) throws AiToolException
+   {
+      Object value = in.get(name);
+      if(value instanceof String s && s.strip().startsWith("{"))
+      {
+         return (new JSONObject(s).toMap());
+      }
+      if(value instanceof Map<?, ?> m && !m.isEmpty())
+      {
+         return ((Map<String, Object>) m);
+      }
+      throw (AiToolException.invalidInput(name + " is required (a JSON object)."));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static List<Object> fieldList(Object mappingJson)
+   {
+      if(mappingJson == null)
+      {
+         return (List.of());
+      }
+      try
+      {
+         return (new JSONObject(String.valueOf(mappingJson)).optJSONArray("fieldList") == null ? List.of()
+            : new JSONObject(String.valueOf(mappingJson)).getJSONArray("fieldList").toList());
+      }
+      catch(RuntimeException e)
+      {
+         return (List.of());
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static Map<String, Object> map(Object value)
+   {
+      return (value instanceof Map<?, ?> m ? (Map<String, Object>) m : Map.of());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @SuppressWarnings("unchecked")
+   private static List<Object> list(Object value)
+   {
+      return (value instanceof List<?> l ? (List<Object>) l : List.of());
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static String shortText(String text)
+   {
+      if(text == null)
+      {
+         return ("");
+      }
+      return (text.length() > 300 ? text.substring(0, 297) + "..." : text);
+   }
+
+
+
+   /*******************************************************************************
+    ** One tool call's UI API invocations: the user's session, the route header,
+    ** and the last operation and status (for the log).
+    *******************************************************************************/
+   private static class Call
+   {
+      private final UiApiToolbox.Context context;
+      private final boolean              hipaaRoute;
+      private String                     operation;
+      private Integer                    httpStatus;
+
+
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      Call(UiApiToolbox.Context context, boolean hipaaRoute)
+      {
+         this.context = context;
+         this.hipaaRoute = hipaaRoute;
+      }
+
+
+
+      /*******************************************************************************
+       ** Invoke, or null when the server does not offer the operation (an
+       ** older QQQ).
+       *******************************************************************************/
+      UiApiResult invoke(UiApiCall uiApiCall) throws QException
+      {
+         operation = uiApiCall.operation();
+         if(!context.gateway().isInvocable(uiApiCall.getMethod(), uiApiCall.getPathTemplate()))
+         {
+            return (null);
+         }
+         uiApiCall.withSessionUUID(context.sessionUUID()).withUserTimezone(context.userTimezone()).withInvokedBy("assistant")
+            .withHeader(ROUTE_HEADER, hipaaRoute ? "hipaa" : "standard");
+         UiApiResult response = context.gateway().invoke(uiApiCall);
+         httpStatus = response.status();
+         return (response);
+      }
+
+
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      String permissionCheck()
+      {
+         return ("uiApi:" + operation + ":" + httpStatus);
+      }
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/tools/bulkload/JobsLaunchBulkLoadRunner.java

@@ -0,0 +1,90 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools.bulkload;
+
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.utils.ValueUtils;
+
+
+/*******************************************************************************
+ ** The default {@link BulkLoadRunner}: the durable jobs launch,
+ ** POST /qqq/v1/processes/{table}.bulkInsert/jobs with the upload ref and the
+ ** saved profile (design spec #1171 sections 2.4 and 3.4). From the assistant
+ ** channel core creates the job AWAITING_CONFIRMATION, so it starts only when
+ ** the user confirms it outside the assistant.
+ **
+ ** STAND-IN(core=#1178, removeBy=RC.8): until the jobs kernel ships in a QQQ
+ ** RC, the launch is not a spec'd operation on the server; the runner then
+ ** says so instead of guessing at another path (a load needs the upload ref,
+ ** which the bulk-insert process endpoints cannot take).
+ *******************************************************************************/
+public class JobsLaunchBulkLoadRunner implements BulkLoadRunner
+{
+   public static final String LAUNCH_OPERATION_PATH = "/processes/{processName}/jobs";
+
+   public static final String NOT_AVAILABLE = "Running a bulk load from the assistant needs the durable jobs launch "
+      + "(POST /qqq/v1/processes/{process}/jobs), which this server does not have yet. Open the table's Bulk Load "
+      + "screen and choose the saved profile instead.";
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public RunHandle start(Request request) throws QException
+   {
+      String processName = request.tableName() + ".bulkInsert";
+      if(!request.gateway().isInvocable("POST", LAUNCH_OPERATION_PATH))
+      {
+         return (new RunHandle("NOT_AVAILABLE", null, null, NOT_AVAILABLE, "POST " + LAUNCH_OPERATION_PATH, null));
+      }
+
+      Map<String, Object> body = new LinkedHashMap<>();
+      body.put("uploadRefs", List.of(request.uploadRef()));
+      body.put("savedBulkLoadProfileId", request.savedBulkLoadProfileId());
+      body.put("mode", "EXECUTE");
+      body.put("label", "Bulk load into " + request.tableName() + " (assistant)");
+
+      UiApiResult response = request.gateway().invoke(new UiApiCall("POST", LAUNCH_OPERATION_PATH)
+         .withPathParam("processName", processName)
+         .withJsonBody(body)
+         .withSessionUUID(request.sessionUUID())
+         .withUserTimezone(request.userTimezone())
+         .withInvokedBy("assistant"));
+
+      if(!response.isSuccess())
+      {
+         return (new RunHandle("FAILED", null, null, response.errorMessage() == null ? "HTTP " + response.status() : response.errorMessage(),
+            "POST " + LAUNCH_OPERATION_PATH, response.status()));
+      }
+
+      Map<String, Object> json   = response.json();
+      Long                jobId  = ValueUtils.getValueAsLong(json.get("jobId"));
+      String              status = json.get("status") == null ? "QUEUED" : String.valueOf(json.get("status"));
+      String summary = "AWAITING_CONFIRMATION".equals(status)
+         ? "Job " + jobId + " is waiting for your confirmation on its job record before it starts."
+         : "Job " + jobId + " is " + status.toLowerCase() + ".";
+      return (new RunHandle(status, jobId, null, summary, "POST " + LAUNCH_OPERATION_PATH, response.status()));
+   }
+}

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/uiapi/LoopbackUiApiGateway.java

@@ -30,6 +30,7 @@ import java.util.LinkedHashSet;
 import java.util.Map;
 import java.util.Set;
 import java.util.StringJoiner;
+import java.util.UUID;
 import java.util.function.Supplier;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
@@ -64,6 +65,9 @@ public class LoopbackUiApiGateway implements UiApiGateway
    public static final String INVOKED_BY_HEADER = "X-QQQ-Invoked-By";
    public static final String TIMEZONE_HEADER   = "X-QQQ-UserTimezone";
 
+   /** Headers a call cannot set itself (lower case). */
+   private static final Set<String> RESERVED_HEADERS = Set.of("cookie", "authorization", "x-api-key", "content-type", "accept", INVOKED_BY_HEADER.toLowerCase(), TIMEZONE_HEADER.toLowerCase());
+
    private static final Set<String> SPEC_OPERATIONS = specOperations();
 
    private final Supplier<String> baseUrlSupplier;
@@ -204,8 +208,31 @@ public class LoopbackUiApiGateway implements UiApiGateway
       {
          builder.header(TIMEZONE_HEADER, call.getUserTimezone());
       }
+      call.getHeaders().forEach((name, value) ->
+      {
+         if(!RESERVED_HEADERS.contains(name.toLowerCase()))
+         {
+            builder.header(name, value);
+         }
+      });
 
-      if(call.getJsonBody() != null)
+      if(!call.getFormParams().isEmpty())
+      {
+         ////////////////////////////////////////////////////////////////
+         // multipart/form-data text fields (the /processes endpoints) //
+         ////////////////////////////////////////////////////////////////
+         String        boundary = "qqq-assistant-" + UUID.randomUUID();
+         StringBuilder body     = new StringBuilder();
+         String        lineEnd  = "\r\n";
+         call.getFormParams().forEach((name, value) -> body.append("--").append(boundary).append(lineEnd)
+            .append("Content-Disposition: form-data; name=\"").append(name.replace("\"", "")).append("\"").append(lineEnd)
+            .append("Content-Type: text/plain; charset=UTF-8").append(lineEnd).append(lineEnd)
+            .append(value).append(lineEnd));
+         body.append("--").append(boundary).append("--").append(lineEnd);
+         builder.header("Content-Type", "multipart/form-data; boundary=" + boundary);
+         builder.method(call.getMethod(), HttpRequest.BodyPublishers.ofString(body.toString(), StandardCharsets.UTF_8));
+      }
+      else if(call.getJsonBody() != null)
       {
          builder.header("Content-Type", "application/json");
          builder.method(call.getMethod(), HttpRequest.BodyPublishers.ofString(call.getJsonBody() instanceof String s ? s : JsonUtils.toJson(call.getJsonBody()), StandardCharsets.UTF_8));

qbit-assistant/src/main/java/com/kingsrook/qbits/assistant/uiapi/UiApiCall.java

@@ -33,6 +33,8 @@ public class UiApiCall
    private final String              pathTemplate;
    private final Map<String, String> pathParams  = new LinkedHashMap<>();
    private final Map<String, String> queryParams = new LinkedHashMap<>();
+   private final Map<String, String> formParams  = new LinkedHashMap<>();
+   private final Map<String, String> headers     = new LinkedHashMap<>();
    private Object                    jsonBody;
    private String                    sessionUUID;
    private String                    userTimezone;
@@ -128,6 +130,58 @@ public class UiApiCall
 
 
 
+   /*******************************************************************************
+    ** Add a multipart form field (ignored when null). A call with form fields
+    ** is sent as multipart/form-data (the /processes endpoints), mirroring
+    ** core's UiApiRequest.withFormParam.
+    *******************************************************************************/
+   public UiApiCall withFormParam(String name, Object value)
+   {
+      if(value != null)
+      {
+         formParams.put(name, String.valueOf(value));
+      }
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for formParams
+    *******************************************************************************/
+   public Map<String, String> getFormParams()
+   {
+      return (formParams);
+   }
+
+
+
+   /*******************************************************************************
+    ** Add a request header the invoker sets (e.g. X-QQQ-AI-Route); ignored
+    ** when null. The session cookie and the invoked-by marker cannot be
+    ** overridden this way.
+    *******************************************************************************/
+   public UiApiCall withHeader(String name, Object value)
+   {
+      if(name != null && value != null)
+      {
+         headers.put(name, String.valueOf(value));
+      }
+      return (this);
+   }
+
+
+
+   /*******************************************************************************
+    ** Getter for headers
+    *******************************************************************************/
+   public Map<String, String> getHeaders()
+   {
+      return (headers);
+   }
+
+
+
    /*******************************************************************************
     ** Fluent setter for jsonBody
     *******************************************************************************/

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/AssistantQBitProducerTest.java

@@ -88,7 +88,7 @@ class AssistantQBitProducerTest extends BaseAssistantTest
       // the nav: the home plus the main tables; processes are actions    //
       //////////////////////////////////////////////////////////////////////
       assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getChildren()).extracting(c -> c.getName())
-         .containsExactly("assistant_budget", "assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_model_price", "assistant_policy");
+         .containsExactly("assistant_rag_source", "assistant_budget", "assistant_provider", "assistant_profile", "assistant_skill", "assistant_data_access", "assistant_conversation", "assistant_model_price", "assistant_policy");
       assertThat(qInstance.getApp(AssistantMetaDataBuilder.APP_NAME).getWidgets()).containsExactly("assistantHome", "assistantAccess");
       assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME).getTableName()).isEqualTo("assistant_conversation");
       assertThat(qInstance.getProcess(AssistantMetaDataBuilder.RETENTION_PROCESS_NAME)).isNotNull();

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/rag/RagTest.java

@@ -0,0 +1,578 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.rag;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.classification.DataClassification;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.engine.TurnClassification;
+import com.kingsrook.qbits.assistant.model.AssistantConversation;
+import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.model.AssistantPolicy;
+import com.kingsrook.qbits.assistant.model.AssistantProvider;
+import com.kingsrook.qbits.assistant.model.AssistantRagSource;
+import com.kingsrook.qbits.assistant.model.AssistantUsage;
+import com.kingsrook.qbits.assistant.processes.KnowledgeIndexProcess;
+import com.kingsrook.qbits.assistant.provider.AssistantModelProvider;
+import com.kingsrook.qbits.assistant.provider.AssistantProviderType;
+import com.kingsrook.qbits.assistant.provider.EmbeddingRequest;
+import com.kingsrook.qbits.assistant.provider.EmbeddingResponse;
+import com.kingsrook.qbits.assistant.provider.ModelPart;
+import com.kingsrook.qbits.assistant.provider.ModelUsage;
+import com.kingsrook.qbits.assistant.provider.fake.ScriptedModelProvider;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qbits.assistant.tools.KnowledgeTool;
+import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qqq.backend.core.actions.ai.AiExposure;
+import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolResult;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
+import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Real UI API indexing/retrieval, with deterministic local embeddings and no credentials.
+ *******************************************************************************/
+class RagTest extends BaseAssistantTest
+{
+   private final InMemoryVectorStore vectors = new InMemoryVectorStore();
+   private final Embeddings embeddings = new Embeddings();
+   private AssistantRagSource source;
+   private AssistantModel model;
+   private AssistantProvider embeddingProvider;
+
+
+
+   /*******************************************************************************
+    ** A hashed bag of words; capture batches solely to assert routing and masking.
+    *******************************************************************************/
+   static class Embeddings extends ScriptedModelProvider
+   {
+      final List<EmbeddingRequest> batches = new ArrayList<>();
+
+
+
+      /*******************************************************************************
+       ** Same text always yields the same 16-dimensional vector.
+       *******************************************************************************/
+      @Override
+      public EmbeddingResponse embed(EmbeddingRequest request)
+      {
+         batches.add(request);
+         List<float[]> vectors = new ArrayList<>();
+         for(String text : request.input())
+         {
+            float[] vector = new float[16];
+            for(String word : text.toLowerCase(java.util.Locale.ROOT).split("[^a-z0-9]+"))
+            {
+               vector[Math.floorMod(word.hashCode(), vector.length)]++;
+            }
+            vectors.add(vector);
+         }
+         return (new EmbeddingResponse(vectors, new ModelUsage().withInputTokens((long) request.input().size()), "embed-test"));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Keyless local type with the same provider instance for chat and embeddings.
+    *******************************************************************************/
+   class Type implements AssistantProviderType
+   {
+      /*******************************************************************************
+       ** Test type identity.
+       *******************************************************************************/
+      @Override
+      public String getType()
+      {
+         return ("rag-test");
+      }
+
+
+
+      /*******************************************************************************
+       ** Test label.
+       *******************************************************************************/
+      @Override
+      public String getLabel()
+      {
+         return ("RAG test");
+      }
+
+
+
+      /*******************************************************************************
+       ** Chat default deliberately differs from the embedding model.
+       *******************************************************************************/
+      @Override
+      public List<String> getModels()
+      {
+         return (List.of("chat", "bge-m3", "nomic-embed-text"));
+      }
+
+
+
+      /*******************************************************************************
+       ** No credentials for this in-process fake.
+       *******************************************************************************/
+      @Override
+      public String getDefaultApiKeyEnvName()
+      {
+         return (null);
+      }
+
+
+
+      /*******************************************************************************
+       ** Never contacted; satisfies provider configuration validation.
+       *******************************************************************************/
+      @Override
+      public String getDefaultBaseUrl()
+      {
+         return ("http://localhost:11434/v1");
+      }
+
+
+
+      /*******************************************************************************
+       ** Local test location.
+       *******************************************************************************/
+      @Override
+      public String getDefaultLocation(String kind)
+      {
+         return ("LOCAL");
+      }
+
+
+
+      /*******************************************************************************
+       ** Return the test recording provider.
+       *******************************************************************************/
+      @Override
+      public AssistantModelProvider create(String key, String base)
+      {
+         return (embeddings);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Configure only local fakes and the optional PHI fixture table.
+    *******************************************************************************/
+   @Override
+   protected void customizeConfig(AssistantQBitConfig config)
+   {
+      config.withVectorStore(vectors).withProviderType(new Type()).withDataClassification("patient", DataClassification.PHI);
+   }
+
+
+
+   /*******************************************************************************
+    ** Registered EMBEDDING model and a source in the real metadata-backed store.
+    *******************************************************************************/
+   @BeforeEach
+   void sources() throws Exception
+   {
+      as(AssistantTestFixture.ADMIN);
+      embeddingProvider = config.getRuntime().getStore().insert(new AssistantProvider().withName("Local embeddings").withProviderType("rag-test")
+         .withModel("chat").withEnabled(true).withHipaaEligible(false));
+      model = config.getRuntime().getStore().insert(new AssistantModel().withProviderId(embeddingProvider.getId()).withModelId("bge-m3")
+         .withModelKind("EMBEDDING").withEmbeddingDimensions(16).withEnabled(true));
+      config.getRuntime().getStore().insert(new AssistantModel().withProviderId(embeddingProvider.getId()).withModelId("chat").withEnabled(true));
+      source = config.getRuntime().getStore().insert(new AssistantRagSource().withName("Blog knowledge").withTableName("blog_post")
+         .withFieldNames("title,content").withTitleFieldName("title").withEmbeddingModelId(model.getId()));
+      assertThat(source.getChunkChars()).isEqualTo(1500);
+   }
+
+
+
+   /*******************************************************************************
+    ** The session used both for configuration reads and UI API calls.
+    *******************************************************************************/
+   private void as(String user)
+   {
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(user, config));
+   }
+
+
+
+   /*******************************************************************************
+    ** Retrieval helper, retaining the caller's exact session UUID.
+    *******************************************************************************/
+   private AiToolResult search(String user, DataClassification classification) throws Exception
+   {
+      as(user);
+      return (config.getRuntime().getKnowledgeTool().execute(config.getRuntime().getRagService().context(), "search-1", Map.of("query", "Drain upgrade verify", "limit", 8), classification, null, null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Unchanged content performs no embedding; disappeared records are removed.
+    *******************************************************************************/
+   @Test
+   void indexesReconcilesAndReports() throws Exception
+   {
+      RagService rag = config.getRuntime().getRagService();
+      RunBackendStepInput input = new RunBackendStepInput().withRecords(List.of(source.toQRecord()));
+      RunBackendStepOutput output = new RunBackendStepOutput();
+      KnowledgeIndexProcess.Review review = new KnowledgeIndexProcess.Review();
+      review.runOnePage(input, output);
+      assertThat(review.getProcessSummary(output, false)).hasSize(1);
+      assertThat(embeddings.batches).isEmpty();
+      KnowledgeIndexProcess.Index index = new KnowledgeIndexProcess.Index();
+      index.runOnePage(input, new RunBackendStepOutput());
+      assertThat(index.getProcessSummary(output, true)).hasSize(1);
+      assertThat(vectors.count(source.getId())).isPositive();
+      Integer calls = embeddings.batches.size();
+      RagService.Result unchanged = rag.index(source);
+      assertThat(unchanged.errors()).as(unchanged.reason()).isZero();
+      assertThat(unchanged.embedded()).isZero();
+      assertThat(unchanged.unchanged()).isEqualTo(vectors.count(source.getId()).intValue());
+      assertThat(embeddings.batches).hasSize(calls);
+      assertThat(vectors.chunks(source.getId())).allSatisfy(v -> assertThat(v.contentHash()).matches("[a-f0-9]{64}"));
+      assertThat(ChunkVector.class.getRecordComponents()).extracting(java.lang.reflect.RecordComponent::getName)
+         .containsExactly("source", "table", "pk", "field", "chunkIndex", "contentHash", "embedding");
+      assertThat(config.getRuntime().getUiApiGateway().invoke(new UiApiCall("DELETE", "/table/{tableName}/{primaryKey}")
+         .withPathParam("tableName", "blog_post").withPathParam("primaryKey", "1").withSessionUUID(AssistantTestFixture.ADMIN)).status()).isEqualTo(200);
+      RagService.Result changed = rag.index(source);
+      assertThat(changed.errors()).as(changed.reason()).isZero();
+      assertThat(changed.removed()).isEqualTo(2);
+      assertThat(vectors.chunks(source.getId())).noneMatch(v -> v.pk().equals("1"));
+      new KnowledgeIndexProcess.Clear().runOnePage(input, new RunBackendStepOutput());
+      assertThat(vectors.count(source.getId())).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** UI read permission is checked again for every vector hit, with no identifiers
+    ** or snippets leaking from denied records. Exposure is reused from get_record.
+    *******************************************************************************/
+   @Test
+   void retrievesAsUserAndFiltersUiDenials() throws Exception
+   {
+      assertThat(config.getRuntime().getRagService().index(source).errors()).isZero();
+      AiToolResult allowed = search(AssistantTestFixture.EDITOR, DataClassification.PUBLIC);
+      assertThat(allowed.getOutcome()).as(allowed.getModelContent()).isEqualTo(AiToolOutcome.OK);
+      assertThat(allowed.getRowCount()).isPositive();
+      assertThat(allowed.getModelContent()).contains("Drain, upgrade, verify.", "/app/blog_post/").doesNotContain(AssistantTestFixture.HIDDEN_SENTINEL, AssistantTestFixture.PASSWORD_SENTINEL);
+      assertThat(config.getRuntime().getKnowledgeTool().offered(PolicyService.ToolPolicy.defaults(), QContext.getQSession())).isTrue();
+      AiToolResult denied = search(AssistantTestFixture.ANALYST, DataClassification.PUBLIC);
+      assertThat(denied.getOutcome()).isEqualTo(AiToolOutcome.OK);
+      assertThat(denied.getRowCount()).isZero();
+      assertThat(denied.getSummary()).doesNotEndWith("filtered 0");
+      assertThat(denied.getRecordRefs()).isEmpty();
+      assertThat(denied.getModelContent()).doesNotContain("blog_post", "Drain", "/app/");
+      assertThat(config.getRuntime().getKnowledgeTool().offered(PolicyService.ToolPolicy.defaults(), QContext.getQSession())).isFalse();
+      assertThat(config.getRuntime().getUiApiGateway().invoke(new UiApiCall("GET", "/table/{tableName}/{primaryKey}")
+         .withPathParam("tableName", "blog_post").withPathParam("primaryKey", "1").withSessionUUID(AssistantTestFixture.ANALYST)).status()).isEqualTo(403);
+   }
+
+
+
+   /*******************************************************************************
+    ** Fresh field/table restrictions narrow both index validation and tool offering.
+    *******************************************************************************/
+   @Test
+   void validatesExposureAndMasksBeforeEmbedding() throws Exception
+   {
+      RagService rag = config.getRuntime().getRagService();
+      assertThat(rag.validate(new AssistantRagSource(source.toQRecord()).withFieldNames("internalNotes"))).isNotEmpty();
+      assertThat(rag.validate(new AssistantRagSource(source.toQRecord()).withFieldNames("id"))).isNotEmpty();
+      assertThat(rag.validate(new AssistantRagSource(source.toQRecord()).withTableName("assistant_model"))).isNotEmpty();
+      assertThat(rag.validate(new AssistantRagSource(source.toQRecord()).withChunkChars(100).withChunkOverlapChars(100))).isNotEmpty();
+      qInstance.getTable("blog_post").getField("content").withSupplementalMetaData(new AiFieldMetaData().withExposure(AiExposure.MASKED));
+      assertThat(rag.index(source).errors()).isZero();
+      assertThat(embeddings.batches.stream().flatMap(b -> b.input().stream())).noneMatch(s -> s.contains("Drain, upgrade"));
+      assertThat(search(AssistantTestFixture.EDITOR, DataClassification.PUBLIC).getModelContent()).doesNotContain("Drain, upgrade");
+      as(AssistantTestFixture.ADMIN);
+      config.getRuntime().getStore().insert(new AssistantDataAccess().withObjectType("TABLE").withObjectName("blog_post").withMode("NONE"));
+      assertThat(rag.validate(source)).anyMatch(s -> s.contains("NONE"));
+      assertThat(config.getRuntime().getKnowledgeTool().offered(PolicyService.ToolPolicy.defaults(), QContext.getQSession())).isFalse();
+      assertThat(search(AssistantTestFixture.EDITOR, DataClassification.PUBLIC).getRowCount()).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** PHI dispatch fails before text leaves the instance; eligible retrieval
+    ** contributes table classification through the existing record-ref mechanism.
+    *******************************************************************************/
+   @Test
+   void phiEmbeddingGuardAndClassification() throws Exception
+   {
+      AssistantRagSource patient = config.getRuntime().getStore().insert(new AssistantRagSource().withName("Patients").withTableName("patient")
+         .withFieldNames("diagnosis").withEmbeddingModelId(model.getId()));
+      RagService.Result blocked = config.getRuntime().getRagService().index(patient);
+      assertThat(blocked.errors()).isEqualTo(1);
+      assertThat(blocked.reason()).contains("HIPAA-eligible");
+      assertThat(embeddings.batches).isEmpty();
+      assertThat(search(AssistantTestFixture.EDITOR, DataClassification.PHI).getModelContent()).contains("HIPAA-eligible");
+      assertThat(embeddings.batches).isEmpty();
+      as(AssistantTestFixture.ADMIN);
+      config.getRuntime().getStore().updateFields(AssistantProvider.TABLE_NAME, new QRecord().withValue("id", embeddingProvider.getId()).withValue("hipaaEligible", true));
+      assertThat(config.getRuntime().getRagService().index(patient).errors()).isZero();
+      as(AssistantTestFixture.EDITOR);
+      AiToolResult result = config.getRuntime().getKnowledgeTool().execute(config.getRuntime().getRagService().context(), "phi", Map.of("query", "diagnosis", "tables", List.of("patient")), DataClassification.PUBLIC, null, null);
+      assertThat(result.getRowCount()).isPositive();
+      TurnClassification classification = new TurnClassification(config.getClassificationSource(), new AssistantConversation().withMaxClassification("PUBLIC"), null);
+      classification.observe(ModelPart.toolUse("phi", KnowledgeTool.NAME, Map.of()), result);
+      assertThat(classification.getClassification()).isEqualTo(DataClassification.PHI);
+   }
+
+
+
+   /*******************************************************************************
+    ** Batches are capped, accounted as EMBED, and blocked by the kill switch.
+    *******************************************************************************/
+   @Test
+   void batchesUsageAndSelection() throws Exception
+   {
+      new EmbeddingService(config.getRuntime()).embed(model.getId(), java.util.Collections.nCopies(130, "word"), DataClassification.PUBLIC, "blog_post", null, null);
+      assertThat(embeddings.batches).extracting(b -> b.input().size()).containsExactly(64, 64, 2);
+      assertThat(config.getRuntime().getStore().query(AssistantUsage.TABLE_NAME, new QQueryFilter(), AssistantUsage.class))
+         .hasSize(3).allSatisfy(row -> assertThat(row.getPurpose()).isEqualTo("EMBED"));
+      assertThat(config.getRuntime().getProviderService().resolveModel(model.getId(), false)).isNull();
+      assertThat(config.getRuntime().getProviderService().forRow(new AssistantProvider(embeddingProvider.toQRecord()).withModel(null), "chat").model()).isEqualTo("chat");
+      assertThatThrownBy(() -> config.getRuntime().getProviderService().resolveEmbedding(-1, false)).hasMessageContaining("EMBEDDING");
+      assertThat(com.kingsrook.qbits.assistant.engine.ModelDiscovery.run(config.getRuntime(), embeddingProvider).ok()).isTrue();
+      assertThat(config.getRuntime().getStore().listModels()).filteredOn(m -> m.getModelId().equals("nomic-embed-text"))
+         .singleElement().satisfies(m ->
+         {
+            assertThat(m.getModelKind()).isEqualTo("EMBEDDING");
+            assertThat(m.getEmbeddingDimensions()).isEqualTo(768);
+         });
+      config.getRuntime().getStore().insert(new AssistantPolicy().withName("Stop").withKillSwitch(true));
+      assertThatThrownBy(() -> new EmbeddingService(config.getRuntime()).embed(model.getId(), List.of("blocked"), DataClassification.PUBLIC, "blog_post", null, null)).hasMessageContaining("kill switch");
+      assertThat(embeddings.batches).hasSize(3);
+      assertThat(com.kingsrook.qbits.assistant.engine.ConnectionTest.runModel(config.getRuntime(), model, AssistantTestFixture.ADMIN).status()).isEqualTo("ERROR");
+   }
+
+
+
+   /*******************************************************************************
+    ** A full golden turn cites fresh knowledge and writes the standard tool log.
+    *******************************************************************************/
+   @Test
+   @Tag("golden")
+   void goldenAnswerCitesKnowledge() throws Exception
+   {
+      assertThat(config.getRuntime().getRagService().index(source).errors()).isZero();
+      embeddings.thenToolUse(KnowledgeTool.NAME, Map.of("query", "Drain upgrade verify"))
+         .thenRespond(com.kingsrook.qbits.assistant.testing.EvalSupport.answer(request ->
+         {
+            Map<String, Object> data = com.kingsrook.qbits.assistant.testing.EvalSupport.firstResult(request);
+            Map<?, ?> record = (Map<?, ?>) ((List<?>) data.get("results")).getFirst();
+            return (record.get("snippet") + " [Source](" + record.get("href") + ")");
+         }));
+      var outcome = runner.run(AssistantTestFixture.EDITOR, "Find the knowledge about safely upgrading");
+      assertThat(outcome.errorCode()).isNull();
+      assertThat(outcome.text()).contains("[Source](/app/blog_post/");
+      assertThat(outcome.toolCalls()).singleElement().satisfies(call ->
+      {
+         assertThat(call.getToolName()).isEqualTo(KnowledgeTool.NAME);
+         assertThat(call.getOutcome()).isEqualTo("OK");
+         assertThat(call.getUiApiOperation()).startsWith("GET /table/");
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Defensive vectors and chunk boundary behavior, including invalid inputs.
+    *******************************************************************************/
+   @Test
+   void vectorAndChunkContracts() throws Exception
+   {
+      float[] raw = { 1, 0 };
+      ChunkVector chunk = new ChunkVector(1, "table", "key", "body", 0, TextChunks.hash("text"), raw);
+      raw[0] = 9;
+      assertThat(chunk.embedding()).containsExactly(1, 0);
+      chunk.embedding()[0] = 5;
+      assertThat(chunk.embedding()).containsExactly(1, 0);
+      InMemoryVectorStore store = new InMemoryVectorStore();
+      store.upsert(List.of(chunk));
+      assertThat(store.search(new float[] { 1, 0 }, 1, Set.of("table"))).singleElement().satisfies(hit -> assertThat(hit.score()).isEqualTo(1.0));
+      assertThat(store.search(new float[] { 0, 0 }, 1, Set.of("table"))).isEmpty();
+      assertThat(store.search(new float[] { 1 }, 1, Set.of("table"))).isEmpty();
+      assertThat(store.search(raw, 1, Set.of())).isEmpty();
+      store.replaceRecord(1, "table", "key", List.of());
+      assertThat(store.count(1)).isZero();
+      assertThat(TextChunks.split("abcdefgh", 4, 1)).containsExactly("abcd", "defg", "gh");
+      assertThat(TextChunks.split("", 4, 0)).isEmpty();
+      assertThatThrownBy(() -> TextChunks.split("a", 1, 1)).isInstanceOf(IllegalArgumentException.class);
+      assertThatThrownBy(() -> new ChunkVector(1, "t", "p", "f", 0, "invalid", raw)).isInstanceOf(IllegalArgumentException.class);
+      assertThatThrownBy(() -> new ChunkVector(1, "t", "p", "f", 0, TextChunks.hash("x"), new float[] { Float.NaN })).isInstanceOf(IllegalArgumentException.class);
+      assertThatThrownBy(() -> new ScriptedModelProvider().embed(new EmbeddingRequest("chat", List.of("text")))).hasMessageContaining("unsupported");
+   }
+
+
+
+   /*******************************************************************************
+    ** Test auth supplies a record-lock key for each real user's UI API session.
+    *******************************************************************************/
+   public static class LockedSessions extends AssistantTestFixture.TestSessionCustomizer
+   {
+      /*******************************************************************************
+       ** The admin can index all records; each editor can see only its own key.
+       *******************************************************************************/
+      @Override
+      public void customizeSession(com.kingsrook.qqq.backend.core.model.metadata.QInstance instance, com.kingsrook.qqq.backend.core.model.session.QSession session, Map<String, Object> context)
+      {
+         super.customizeSession(instance, session, context);
+         session.withSecurityKeyValue("rag_owner", session.getUuid());
+         if(session.getUuid().startsWith("admin"))
+         {
+            session.withSecurityKeyValue("rag_all", true);
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The owner and non-owner hit identical UI record-lock enforcement.
+    *******************************************************************************/
+   @Test
+   void recordLocksFilterNonOwner() throws Exception
+   {
+      qInstance.addSecurityKeyType(new com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType().withName("rag_owner").withAllAccessKeyName("rag_all"));
+      qInstance.getAuthentication().withCustomizer(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(LockedSessions.class));
+      var table = qInstance.getTable("blog_post");
+      table.withRecordSecurityLock(new com.kingsrook.qqq.backend.core.model.metadata.security.RecordSecurityLock().withFieldName("slug").withSecurityKeyType("rag_owner"));
+      assertThat(config.getRuntime().getUiApiGateway().invoke(new UiApiCall("PATCH", "/table/{tableName}/{primaryKey}")
+         .withPathParam("tableName", "blog_post").withPathParam("primaryKey", "1").withSessionUUID(AssistantTestFixture.ADMIN)
+         .withJsonBody(Map.of("slug", AssistantTestFixture.EDITOR))).status()).isEqualTo(200);
+      RagService.Result indexed = config.getRuntime().getRagService().index(source);
+      assertThat(indexed.errors()).as(indexed.reason()).isZero();
+      AiToolResult owner = search(AssistantTestFixture.EDITOR, DataClassification.PUBLIC);
+      assertThat(owner.getRowCount()).as(owner.getModelContent()).isPositive();
+      assertThat(owner.getRecordRefs()).allSatisfy(ref -> assertThat(String.valueOf(ref.getId())).isEqualTo("1"));
+      AiToolResult other = search(AssistantTestFixture.OTHER_EDITOR, DataClassification.PUBLIC);
+      assertThat(other.getRowCount()).isZero();
+      assertThat(other.getSummary()).doesNotEndWith("filtered 0");
+      Integer status = config.getRuntime().getUiApiGateway().invoke(new UiApiCall("GET", "/table/{tableName}/{primaryKey}")
+         .withPathParam("tableName", "blog_post").withPathParam("primaryKey", "1").withSessionUUID(AssistantTestFixture.OTHER_EDITOR)).status();
+      assertThat(status).isIn(403, 404);
+   }
+
+
+
+   /*******************************************************************************
+    ** Filter paging and chunk shrinkage reconcile without holding record content.
+    *******************************************************************************/
+   @Test
+   void sourceFilterAndChangedChunks() throws Exception
+   {
+      source.setFilterJson("{\"criteria\":[{\"fieldName\":\"id\",\"operator\":\"EQUALS\",\"values\":[1]}]}");
+      source.setFieldNames("content");
+      source.setChunkChars(100);
+      source.setChunkOverlapChars(20);
+      config.getRuntime().getStore().update(source);
+      var gateway = config.getRuntime().getUiApiGateway();
+      var update = new UiApiCall("PATCH", "/table/{tableName}/{primaryKey}").withPathParam("tableName", "blog_post").withPathParam("primaryKey", "1").withSessionUUID(AssistantTestFixture.ADMIN);
+      assertThat(gateway.invoke(update.withJsonBody(Map.of("content", "Knowledge ".repeat(40)))).status()).isEqualTo(200);
+      RagService.Result first = config.getRuntime().getRagService().index(source);
+      assertThat(first.errors()).as(first.reason()).isZero();
+      assertThat(first.recordsSeen()).isEqualTo(1);
+      assertThat(first.embedded()).isGreaterThan(1);
+      assertThat(gateway.invoke(update.withJsonBody(Map.of("content", "Short updated knowledge"))).status()).isEqualTo(200);
+      assertThat(search(AssistantTestFixture.EDITOR, DataClassification.PUBLIC).getModelContent()).contains("Short updated knowledge").doesNotContain("Knowledge Knowledge");
+      as(AssistantTestFixture.ADMIN);
+      RagService.Result shortened = config.getRuntime().getRagService().index(source);
+      assertThat(shortened.errors()).as(shortened.reason()).isZero();
+      assertThat(shortened.embedded()).isEqualTo(1);
+      assertThat(shortened.removed()).isEqualTo(first.embedded() - 1);
+      source.setFieldNames("title");
+      config.getRuntime().getStore().update(source);
+      assertThat(vectors.count(source.getId())).isZero();
+      assertThat(config.getRuntime().getRagService().validate(new AssistantRagSource(source.toQRecord()).withFilterJson("{"))).isNotEmpty();
+      as(AssistantTestFixture.EDITOR);
+      assertThatThrownBy(() -> config.getRuntime().getRagService().index(source)).hasMessageContaining("assistant.admin");
+   }
+
+
+
+   /*******************************************************************************
+    ** Standard process execution reaches both Review and Result backend steps.
+    *******************************************************************************/
+   @Test
+   void standardProcessAndPolicy() throws Exception
+   {
+      var input = new com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput();
+      input.setProcessName(KnowledgeIndexProcess.INDEX);
+      input.setCallback(com.kingsrook.qqq.backend.core.actions.processes.QProcessCallbackFactory.forPrimaryKey("id", source.getId()));
+      input.setFrontendStepBehavior(com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput.FrontendStepBehavior.SKIP);
+      new com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction().execute(input);
+      assertThat(vectors.count(source.getId())).isPositive();
+      assertThat(com.kingsrook.qbits.assistant.engine.ConnectionTest.runModel(config.getRuntime(), model, AssistantTestFixture.ADMIN).status()).isEqualTo("OK");
+      RunBackendStepOutput clearOutput = new RunBackendStepOutput();
+      KnowledgeIndexProcess.ClearReview clearReview = new KnowledgeIndexProcess.ClearReview();
+      clearReview.runOnePage(new RunBackendStepInput().withRecords(List.of(source.toQRecord())), clearOutput);
+      assertThat(clearReview.getProcessSummary(clearOutput, false)).hasSize(1);
+      var policy = new PolicyService.ToolPolicy(true, Set.of("search_help"), 50, true);
+      assertThat(config.getRuntime().getKnowledgeTool().offered(policy, QContext.getQSession())).isFalse();
+      var context = new UiApiToolbox.Context(AssistantTestFixture.ADMIN, null, config.getRuntime().getUiApiGateway(), policy, null);
+      assertThat(config.getRuntime().getKnowledgeTool().execute(context, "denied", Map.of("query", "x"), DataClassification.PUBLIC, null, null).getOutcome()).isEqualTo(AiToolOutcome.DENIED);
+      assertThat(config.getRuntime().getKnowledgeTool().execute(config.getRuntime().getRagService().context(), "invalid", Map.of(), DataClassification.PUBLIC, null, null).getOutcome()).isEqualTo(AiToolOutcome.INVALID_INPUT);
+      config.getBudgets().withUserDailyTokensHard(1L);
+      assertThatThrownBy(() -> new EmbeddingService(config.getRuntime()).embed(model.getId(), List.of("blocked"), DataClassification.PUBLIC, "blog_post", null, null)).hasMessageContaining("budget");
+   }
+
+
+
+   /*******************************************************************************
+    ** A multi-page scan does not stop at the first UI API query page.
+    *******************************************************************************/
+   @Test
+   void indexesMultiplePages() throws Exception
+   {
+      List<QRecord> records = new ArrayList<>();
+      for(Integer i = 0; i < 105; i++)
+      {
+         records.add(new QRecord().withValue("title", "Extra page " + i).withValue("content", "Paged knowledge " + i));
+      }
+      new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput("page").withRecords(records));
+      source.setTableName("page");
+      source.setFieldNames("content");
+      config.getRuntime().getStore().update(source);
+      RagService.Result result = config.getRuntime().getRagService().index(source);
+      assertThat(result.errors()).as(result.reason()).isZero();
+      assertThat(result.recordsSeen()).isEqualTo(117);
+      assertThat(vectors.count(source.getId())).isEqualTo(117);
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/settings/AssistantSettingsTest.java

@@ -66,6 +66,13 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
 import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import com.kingsrook.qbits.assistant.model.AssistantModel;
+import com.kingsrook.qbits.assistant.persistence.AssistantStore;
+import com.kingsrook.qqq.backend.core.actions.values.SearchPossibleValueSourceAction;
+import com.kingsrook.qqq.backend.core.actions.values.QCustomPossibleValueProvider;
+import com.kingsrook.qqq.backend.core.model.actions.values.SearchPossibleValueSourceInput;
+import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
 import java.io.Serializable;
 import java.math.BigDecimal;
 import java.util.HashMap;
@@ -222,8 +229,9 @@ class AssistantSettingsTest extends BaseAssistantTest
     ** Tables, nav, actions, child lists and exposure.
     *******************************************************************************/
    @Test
-   void testMetaData()
+   void testMetaData() throws Exception
    {
+      asAdmin();
       for(String name : List.of(AssistantProvider.TABLE_NAME, AssistantModelPrice.TABLE_NAME, AssistantUsage.TABLE_NAME))
       {
          QTableMetaData table = qInstance.getTable(name);
@@ -237,7 +245,7 @@ class AssistantSettingsTest extends BaseAssistantTest
       assertThat(qInstance.getProcess(TestConnectionProcess.NAME).getLabel()).isEqualTo("Test connection");
       assertThat(qInstance.getWidget(AssistantHome.WIDGET_NAME)).isNotNull();
       assertThat(qInstance.getPossibleValueSource(AssistantRecordScreens.PROVIDER_TYPE_PVS_NAME).getEnumValues()).extracting(v -> String.valueOf(v.getId())).containsExactly(FAKE_TYPE);
-      assertThat(qInstance.getPossibleValueSource(AssistantRecordScreens.MODEL_PVS_NAME).getEnumValues()).extracting(v -> String.valueOf(v.getId())).contains("fake-model-1", "claude-sonnet-5-5");
+      assertThat(new SearchPossibleValueSourceAction().execute(new SearchPossibleValueSourceInput().withPossibleValueSourceName(AssistantRecordScreens.MODEL_PVS_NAME)).getResults()).extracting(v -> String.valueOf(v.getId())).contains("fake-model-1", "claude-sonnet-5-5");
       assertThat(qInstance.getTable(AssistantProvider.TABLE_NAME).getSections()).anyMatch(s -> s.getWidgetName() != null);
       assertThat(qInstance.getTable(AssistantProvider.TABLE_NAME).getSections()).anyMatch(s -> s.getFieldNames() != null && s.getFieldNames().contains("apiKeyStatus"));
    }
@@ -570,4 +578,112 @@ class AssistantSettingsTest extends BaseAssistantTest
       return (((RawHTML) output.getWidgetData()).getHtml());
    }
 
+
+
+   /*******************************************************************************
+    ** Provider models are scoped, with only this row's enabled chat models added.
+    *******************************************************************************/
+   @Test
+   void testContingentProviderModels() throws Exception
+   {
+      asAdmin();
+      Integer id = providerId();
+      AssistantStore store = new AssistantStore();
+      store.insert(new AssistantModel().withProviderId(id).withModelId("registered-chat").withEnabled(true).withModelKind("CHAT"));
+      store.insert(new AssistantModel().withProviderId(id).withModelId("disabled-chat").withEnabled(false).withModelKind("CHAT"));
+      store.insert(new AssistantModel().withProviderId(id).withModelId("embedding").withEnabled(true).withModelKind("EMBEDDING"));
+      Integer otherId = insertProvider(Map.of("name", "Other")).getRecords().get(0).getValueInteger("id");
+      store.insert(new AssistantModel().withProviderId(otherId).withModelId("other-chat").withEnabled(true).withModelKind("CHAT"));
+      assertThat(settingsValues(AssistantRecordScreens.MODEL_PVS_NAME, Map.of("providerType", FAKE_TYPE, "id", id.toString())))
+         .extracting(value -> String.valueOf(value.getId())).containsExactly("fake-model-1", "fake-model-2", "registered-chat");
+      assertThat(settingsValues(AssistantRecordScreens.MODEL_PVS_NAME, Map.of("providerType", FAKE_TYPE)))
+         .extracting(value -> String.valueOf(value.getId())).containsExactly("fake-model-1", "fake-model-2");
+      assertThat(new SearchPossibleValueSourceAction().execute(new SearchPossibleValueSourceInput()
+         .withPossibleValueSourceName(AssistantRecordScreens.MODEL_PVS_NAME).withOtherValues(Map.of("providerType", FAKE_TYPE)).withSearchTerm("fake-model-2")).getResults())
+         .extracting(value -> String.valueOf(value.getId())).containsExactly("fake-model-2");
+   }
+
+
+
+   /*******************************************************************************
+    ** Kinds use the selected type and keep their existing labels.
+    *******************************************************************************/
+   @Test
+   void testContingentProviderKinds() throws Exception
+   {
+      assertThat(settingsValues("assistant_provider_kind", Map.of("providerType", FAKE_TYPE)))
+         .extracting(value -> String.valueOf(value.getId())).containsExactly("FAKE");
+      assertThat(settingsValues("assistant_provider_kind", Map.of())).anySatisfy(value ->
+      {
+         assertThat(value.getId()).isEqualTo("AZURE_OPENAI");
+         assertThat(value.getLabel()).isEqualTo("Azure OpenAI");
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** A price id must not be interpreted as a provider id.
+    *******************************************************************************/
+   @Test
+   void testContingentPriceModels() throws Exception
+   {
+      asAdmin();
+      Integer id = providerId();
+      new AssistantStore().insert(new AssistantModel().withProviderId(id).withModelId("registered-chat").withEnabled(true).withModelKind("CHAT"));
+      String source = qInstance.getTable(AssistantModelPrice.TABLE_NAME).getField("model").getPossibleValueSourceName();
+      assertThat(settingsValues(source, Map.of("providerType", FAKE_TYPE, "id", id)))
+         .extracting(value -> String.valueOf(value.getId())).containsExactly("fake-model-1", "fake-model-2");
+   }
+
+
+
+   /*******************************************************************************
+    ** Missing and unknown types retain the original union, including pricing.
+    *******************************************************************************/
+   @Test
+   void testContingentUnionFallback() throws Exception
+   {
+      for(String source : List.of(AssistantRecordScreens.MODEL_PVS_NAME, "assistant_price_model", "assistant_provider_kind"))
+      {
+         List<QPossibleValue<?>> union = settingsValues(source, null);
+         assertThat(settingsValues(source, Map.of())).usingRecursiveComparison().isEqualTo(union);
+         assertThat(settingsValues(source, Map.of("providerType", "unknown"))).usingRecursiveComparison().isEqualTo(union);
+      }
+      assertThat(settingsValues(AssistantRecordScreens.MODEL_PVS_NAME, null)).extracting(value -> String.valueOf(value.getId()))
+         .contains("fake-model-1", "claude-sonnet-5-5");
+   }
+
+
+
+   /*******************************************************************************
+    ** Saved values resolve without a selected type, even disabled registry ids.
+    *******************************************************************************/
+   @Test
+   void testSavedPossibleValue() throws Exception
+   {
+      asAdmin();
+      Integer id = providerId();
+      new AssistantStore().insert(new AssistantModel().withProviderId(id).withModelId("saved-model").withEnabled(false).withModelKind("CHAT"));
+      for(String source : List.of(AssistantRecordScreens.MODEL_PVS_NAME, "assistant_price_model", "assistant_provider_kind"))
+      {
+         QCustomPossibleValueProvider<?> custom = QCodeLoader.getAdHoc(QCustomPossibleValueProvider.class, qInstance.getPossibleValueSource(source).getCustomCodeReference());
+         String saved = source.equals("assistant_provider_kind") ? "OPENAI" : "saved-model";
+         assertThat(custom.getPossibleValue(saved).getId()).isEqualTo(saved);
+         assertThat(custom.getPossibleValue("not-known")).isNull();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Exercise the registered source through QQQ's search action.
+    *******************************************************************************/
+   private List<QPossibleValue<?>> settingsValues(String source, Map<String, Serializable> otherValues) throws Exception
+   {
+      asAdmin();
+      return (new SearchPossibleValueSourceAction().execute(new SearchPossibleValueSourceInput()
+         .withPossibleValueSourceName(source).withOtherValues(otherValues)).getResults());
+   }
+
 }

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/tools/bulkload/BulkLoadToolsTest.java

@@ -0,0 +1,350 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools.bulkload;
+
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import com.kingsrook.qbits.assistant.AssistantQBitConfig;
+import com.kingsrook.qbits.assistant.access.DataAccessMode;
+import com.kingsrook.qbits.assistant.api.AssistantApi;
+import com.kingsrook.qbits.assistant.engine.PolicyService;
+import com.kingsrook.qbits.assistant.model.AssistantDataAccess;
+import com.kingsrook.qbits.assistant.model.AssistantPendingAction;
+import com.kingsrook.qbits.assistant.model.AssistantToolCall;
+import com.kingsrook.qbits.assistant.provider.ModelPart;
+import com.kingsrook.qbits.assistant.provider.ModelResponse;
+import com.kingsrook.qbits.assistant.testing.AssistantTestFixture;
+import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
+import com.kingsrook.qbits.assistant.testing.TurnRunner;
+import com.kingsrook.qbits.assistant.tools.UiApiToolbox;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qqq.backend.core.actions.ai.AiToolOutcome;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import org.json.JSONObject;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static com.kingsrook.qbits.assistant.testing.EvalSupport.firstResult;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** The bulk-load tools: definitions, the UI API calls they make (as the
+ ** user, with the route header from the provider, never the model), pending
+ ** actions for writes, the confirmed save and run, and log hygiene. Plus the
+ ** golden eval: a messy CSV -> profile -> dry run with an error -> fixed ->
+ ** saved -> loaded.
+ *******************************************************************************/
+class BulkLoadToolsTest extends BaseAssistantTest
+{
+   private final FakeBulkLoadServer server  = new FakeBulkLoadServer(true);
+   private final List<BulkLoadRunner.Request> runs = new ArrayList<>();
+
+
+
+   /*******************************************************************************
+    ** The fake server as the gateway, and a recording runner.
+    *******************************************************************************/
+   @Override
+   protected void customizeConfig(AssistantQBitConfig config)
+   {
+      config.withUiApiGateway(server).withBulkLoadRunner(request ->
+      {
+         runs.add(request);
+         return (new BulkLoadRunner.RunHandle("AWAITING_CONFIRMATION", 42L, null, "Job 42 is waiting for your confirmation on its job record.", "POST /processes/{processName}/jobs", 201));
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** The saved-profile tables (so the tools are offered), and write access.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws QException
+   {
+      new SavedBulkLoadProfileMetaDataProvider().defineAll(qInstance, AssistantTestFixture.BACKEND, null);
+      setAccess(BulkLoadTools.PROFILE_TABLE, DataAccessMode.READ_WRITE);
+      setAccess("blog_post", DataAccessMode.READ_WRITE);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterEach
+   void clear()
+   {
+      QContext.clear();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private void setAccess(String tableName, DataAccessMode mode) throws QException
+   {
+      QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+      try
+      {
+         config.getRuntime().getStore().insert(new AssistantDataAccess().withObjectType(AssistantDataAccess.TYPE_TABLE).withObjectName(tableName).withMode(mode.name()));
+         config.getRuntime().getDataAccessService().invalidate();
+      }
+      finally
+      {
+         QContext.clear();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** One tool call as the turn engine makes it.
+    *******************************************************************************/
+   private UiApiToolbox.Outcome tool(String toolName, Map<String, Object> in, boolean hipaa)
+   {
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(AssistantTestFixture.ADMIN, config));
+      try
+      {
+         return (config.getRuntime().getBulkLoadTools().execute(new UiApiToolbox.Context("session-1", null, server, PolicyService.ToolPolicy.defaults(),
+            write -> config.getRuntime().getStore().insert(new AssistantPendingAction().withUserId(AssistantTestFixture.ADMIN).withToolName(write.toolName())
+               .withTableName(write.tableName()).withUiApiOperation(write.operation()).withParamsJson(JsonUtils.toJson(write.values()))
+               .withSummary(write.summary()).withStatus(AssistantPendingAction.STATUS_PENDING).withExpiresAt(java.time.Instant.now().plusSeconds(600))).getId()),
+            "call-1", toolName, in, hipaa));
+      }
+      finally
+      {
+         QContext.clear();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testDefinitionsAreOffered()
+   {
+      List<String> names = config.getRuntime().getContextBuilder().tools().stream().map(d -> d.getName()).toList();
+      assertThat(names).contains(BulkLoadTools.LIST, BulkLoadTools.DESCRIBE, BulkLoadTools.ATTACH, BulkLoadTools.SUGGEST, BulkLoadTools.DRY_RUN);
+      assertThat(config.getRuntime().getBulkLoadTools().definitions(PolicyService.ToolPolicy.defaults(), true))
+         .extracting(d -> d.getName()).containsExactlyInAnyOrderElementsOf(BulkLoadTools.NAMES);
+      assertThat(config.getRuntime().getBulkLoadTools().definitions(PolicyService.ToolPolicy.defaults(), false))
+         .extracting(d -> d.getName()).doesNotContain(BulkLoadTools.SAVE, BulkLoadTools.RUN);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testReadToolsCallTheUiApiWithTheProvidersRoute()
+   {
+      UiApiToolbox.Outcome list = tool(BulkLoadTools.LIST, Map.of("tableName", "blog_post"), false);
+      assertThat(list.getResult().getOutcome()).isEqualTo(AiToolOutcome.OK);
+      assertThat(list.getResult().getModelContent()).contains("Posts from the CMS export");
+      assertThat(list.getOperation()).isEqualTo("POST /table/{tableName}/query");
+
+      UiApiToolbox.Outcome describe = tool(BulkLoadTools.DESCRIBE, Map.of("id", 3), false);
+      assertThat(describe.getResult().getModelContent()).contains("fieldList");
+
+      ////////////////////////////////////////////////////////////////////////
+      // the route header comes from the provider flag, whatever the input //
+      ////////////////////////////////////////////////////////////////////////
+      tool(BulkLoadTools.ATTACH, Map.of("tableName", "blog_post", "uploadRef", "up_1", "X-QQQ-AI-Route", "hipaa", "route", "hipaa"), false);
+      tool(BulkLoadTools.ATTACH, Map.of("tableName", "blog_post", "uploadRef", "up_1"), true);
+      List<UiApiCall> attaches = server.calls("GET /bulkLoad/uploads/{uploadRef}/profile");
+      assertThat(attaches.get(0).getHeaders()).containsEntry(BulkLoadTools.ROUTE_HEADER, "standard");
+      assertThat(attaches.get(1).getHeaders()).containsEntry(BulkLoadTools.ROUTE_HEADER, "hipaa");
+      assertThat(attaches.get(0).getSessionUUID()).isEqualTo("session-1");
+
+      UiApiToolbox.Outcome missing = tool(BulkLoadTools.ATTACH, Map.of("tableName", "blog_post", "uploadRef", "nope"), false);
+      assertThat(missing.getResult().getOutcome()).isEqualTo(AiToolOutcome.NOT_FOUND);
+
+      UiApiToolbox.Outcome invalid = tool(BulkLoadTools.DRY_RUN, Map.of("tableName", "blog_post", "uploadRef", "up_1"), false);
+      assertThat(invalid.getResult().getOutcome()).isEqualTo(AiToolOutcome.INVALID_INPUT);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testDataAccessAndAnOlderServer() throws QException
+   {
+      setAccess("page", DataAccessMode.NONE);
+      UiApiToolbox.Outcome denied = tool(BulkLoadTools.SUGGEST, Map.of("tableName", "page", "uploadRef", "up_1"), false);
+      assertThat(denied.getResult().getOutcome()).isEqualTo(AiToolOutcome.DENIED);
+      assertThat(server.getCalls()).isEmpty();
+
+      setAccess("media", DataAccessMode.READ);
+      UiApiToolbox.Outcome readOnly = tool(BulkLoadTools.RUN, Map.of("tableName", "media", "uploadRef", "up_1", "profileId", 7), false);
+      assertThat(readOnly.getResult().getOutcome()).isEqualTo(AiToolOutcome.DENIED);
+
+      BulkLoadTools        older   = new BulkLoadTools(qInstance, config.getRuntime().getDataAccessService(), null);
+      FakeBulkLoadServer   oldApi  = new FakeBulkLoadServer(false);
+      QContext.init(qInstance, AssistantTestFixture.ownerSession(AssistantTestFixture.ADMIN, config));
+      UiApiToolbox.Outcome outcome = older.execute(new UiApiToolbox.Context("s", null, oldApi, null, null), "c", BulkLoadTools.SUGGEST,
+         Map.of("tableName", "blog_post", "uploadRef", "up_1"), false);
+      assertThat(outcome.getResult().getOutcome()).isEqualTo(AiToolOutcome.ERROR);
+      assertThat(outcome.getResult().getModelContent()).contains("does not offer");
+      assertThat(oldApi.getCalls()).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testJobsLaunchRunnerSaysWhenTheLaunchIsMissing() throws QException
+   {
+      BulkLoadRunner.RunHandle handle = new JobsLaunchBulkLoadRunner().start(new BulkLoadRunner.Request("s", null, server, "blog_post", "up_1", 7));
+      assertThat(handle.status()).isEqualTo("NOT_AVAILABLE");
+      assertThat(handle.summary()).contains("durable jobs launch");
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Test
+   void testForLogDropsSamplesAndValues()
+   {
+      String logged = BulkLoadTools.forLog("<qqq_data source=\"x\">{\"columns\":[{\"header\":\"A\",\"sampleValues\":[\"" + FakeBulkLoadServer.SAMPLE_SENTINEL
+         + "\"]}],\"summaryLines\":[{\"bulletsOfText\":[\"Row 3 [x]\"]}],\"profile\":{\"fieldList\":[{\"valueMappings\":{\"secret\":1}}]}}</qqq_data>");
+      assertThat(logged).doesNotContain(FakeBulkLoadServer.SAMPLE_SENTINEL).doesNotContain("Row 3 [x]").doesNotContain("secret").contains("header");
+      assertThat(BulkLoadTools.forLog(null)).isNull();
+      assertThat(BulkLoadTools.forLog("plain")).isEqualTo("plain");
+   }
+
+
+
+   /*******************************************************************************
+    ** Golden eval (owner request 2026-10-13): upload a CSV with messy headers;
+    ** the assistant proposes a profile, validates it, fixes one error, saves
+    ** it (shared with a group, after the user confirms) and runs the load
+    ** (after the user confirms). The scripted model stands in for the LLM;
+    ** its fix reads the dry run's error from the tool result.
+    *******************************************************************************/
+   @Test
+   @SuppressWarnings("unchecked")
+   void testGoldenMessyCsvToSavedProfileAndLoad() throws Exception
+   {
+      Map<String, Object> attachments = Map.of("attachments", List.of(Map.of("uploadRef", "up_1", "fileName", "posts.csv")));
+      AtomicReference<String> suggested = new AtomicReference<>();
+
+      provider.thenToolUse(BulkLoadTools.ATTACH, Map.of("tableName", "blog_post", "uploadRef", "up_1"))
+         .thenToolUse(BulkLoadTools.SUGGEST, Map.of("tableName", "blog_post", "uploadRef", "up_1"))
+         .thenRespond(request ->
+         {
+            Map<String, Object> suggestion = firstResult(request);
+            suggested.set(JsonUtils.toJson(suggestion.get("profile")));
+            return (toolUse("dry1", BulkLoadTools.DRY_RUN, Map.of("tableName", "blog_post", "uploadRef", "up_1", "profile", suggestion.get("profile"))));
+         })
+         .thenRespond(request ->
+         {
+            ////////////////////////////////////////////////////////////////
+            // read the error, fix it: default the missing required field //
+            ////////////////////////////////////////////////////////////////
+            Map<String, Object> dryRun = firstResult(request);
+            assertThat(dryRun.get("errorCount")).isEqualTo(2);
+            String missingField = String.valueOf(((Map<?, ?>) ((List<?>) dryRun.get("errors")).get(0)).get("fieldName"));
+            Map<String, Object> profile = new JSONObject(suggested.get()).toMap();
+            List<Map<String, Object>> fields = new ArrayList<>((List<Map<String, Object>>) profile.get("fieldList"));
+            fields.add(new LinkedHashMap<>(Map.of("fieldName", missingField, "defaultValue", "DRAFT")));
+            profile.put("fieldList", fields);
+            return (toolUse("dry2", BulkLoadTools.DRY_RUN, Map.of("tableName", "blog_post", "uploadRef", "up_1", "profile", profile, "mode", "FULL")));
+         })
+         .thenRespond(request ->
+         {
+            Map<String, Object> profile = new JSONObject(suggested.get()).toMap();
+            ((List<Map<String, Object>>) profile.get("fieldList")).add(new LinkedHashMap<>(Map.of("fieldName", "status", "defaultValue", "DRAFT")));
+            return (toolUse("save", BulkLoadTools.SAVE, Map.of("tableName", "blog_post", "label", "CMS post export", "profile", profile,
+               "shares", List.of(Map.of("audienceType", "group", "audienceId", "kof22-admins")))));
+         })
+         .thenText("The file maps cleanly once Status defaults to DRAFT (2 rows, no errors). Confirm to save the profile \"CMS post export\".");
+
+      TurnRunner.Outcome first = runner.run(AssistantTestFixture.ADMIN, null, "Load the attached posts.csv into blog posts and save the mapping for next time.", attachments);
+      List<String> tools = first.toolCalls().stream().map(AssistantToolCall::getToolName).toList();
+      assertThat(tools).containsExactly(BulkLoadTools.ATTACH, BulkLoadTools.SUGGEST, BulkLoadTools.DRY_RUN, BulkLoadTools.DRY_RUN, BulkLoadTools.SAVE);
+
+      List<UiApiCall> validates = server.calls("POST /bulkLoad/{tableName}/validate");
+      assertThat(validates).hasSize(2);
+      assertThat(validates).allSatisfy(c -> assertThat(c.getHeaders()).containsEntry(BulkLoadTools.ROUTE_HEADER, "standard"));
+      assertThat(first.toolCalls().get(2).getResultSummary()).contains("2 with errors");
+      assertThat(first.toolCalls().get(3).getResultSummary()).contains("0 with errors");
+      assertThat(first.toolCalls()).allSatisfy(c -> assertThat(String.valueOf(c.getResultJson()) + c.getInputJson()).doesNotContain(FakeBulkLoadServer.SAMPLE_SENTINEL));
+      assertThat(server.calls("POST /processes/{processName}/init")).as("nothing is saved before the user confirms").isEmpty();
+
+      Integer saveActionId = first.toolCalls().get(4).getPendingActionId();
+      assertThat(saveActionId).isNotNull();
+      QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+      Map<String, Object> saved = new AssistantApi(config.getRuntime(), QContext.getQSession()).confirmAction(saveActionId);
+      QContext.clear();
+      assertThat(saved.get("status")).isEqualTo(AssistantPendingAction.STATUS_EXECUTED);
+      List<UiApiCall> processCalls = server.calls("POST /processes/{processName}/init");
+      assertThat(processCalls).extracting(c -> c.getPathParams().get("processName")).containsExactly("storeSavedBulkLoadProfile", "insertSharedRecord");
+      Map<String, Object> stored = FakeBulkLoadServer.formValues(processCalls.get(0));
+      assertThat(stored).containsEntry("label", "CMS post export").containsEntry("tableName", "blog_post");
+      assertThat(String.valueOf(stored.get("mappingJson"))).contains("DRAFT");
+      assertThat(FakeBulkLoadServer.formValues(processCalls.get(1))).containsEntry("audienceType", "group").containsEntry("audienceId", "kof22-admins")
+         .containsEntry("recordId", 7).containsEntry("scopeId", "READ_ONLY");
+
+      ////////////////////////////////////////////////////////////////
+      // second turn: run the load with the saved profile; confirm  //
+      ////////////////////////////////////////////////////////////////
+      provider.thenToolUse(BulkLoadTools.RUN, Map.of("tableName", "blog_post", "uploadRef", "up_1", "profileId", 7))
+         .thenText("The load is queued as job 42; start it from its job record when you are ready.");
+      TurnRunner.Outcome second = runner.run(AssistantTestFixture.ADMIN, first.conversationId(), "Now load it.", attachments);
+      AssistantToolCall runCall = second.toolCalls().get(0);
+      assertThat(runCall.getToolName()).isEqualTo(BulkLoadTools.RUN);
+      assertThat(runs).as("nothing runs before the user confirms").isEmpty();
+
+      QContext.init(qInstance, AssistantTestFixture.sessionFor(AssistantTestFixture.ADMIN));
+      Map<String, Object> ran = new AssistantApi(config.getRuntime(), QContext.getQSession()).confirmAction(runCall.getPendingActionId());
+      QContext.clear();
+      assertThat(ran.get("outcome")).isEqualTo("OK");
+      assertThat(runs).hasSize(1);
+      assertThat(runs.get(0).savedBulkLoadProfileId()).isEqualTo(7);
+      assertThat(runs.get(0).uploadRef()).isEqualTo("up_1");
+      assertThat(runs.get(0).tableName()).isEqualTo("blog_post");
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static ModelResponse toolUse(String id, String name, Map<String, Object> input)
+   {
+      return (new ModelResponse().withParts(List.of(ModelPart.toolUse("toolu_" + id, name, input))).withStopReason(ModelResponse.STOP_TOOL_USE));
+   }
+}

qbit-assistant/src/test/java/com/kingsrook/qbits/assistant/tools/bulkload/FakeBulkLoadServer.java

@@ -0,0 +1,176 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.assistant.tools.bulkload;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qbits.assistant.uiapi.UiApiCall;
+import com.kingsrook.qbits.assistant.uiapi.UiApiGateway;
+import com.kingsrook.qbits.assistant.uiapi.UiApiResult;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import org.json.JSONArray;
+import org.json.JSONObject;
+
+
+/*******************************************************************************
+ ** A stand-in for the server's bulk-load UI API operations (core's
+ ** /bulkLoad/... and the profile processes), recording every call. Its
+ ** validate answers like core's dry run: the blog_post status field is
+ ** required, so a profile that neither maps nor defaults it has errors.
+ *******************************************************************************/
+public class FakeBulkLoadServer implements UiApiGateway
+{
+   public static final String SAMPLE_SENTINEL = "SAMPLE_SENTINEL_5b1e";
+
+   private static final Set<String> OPERATIONS = Set.of(
+      "GET /bulkLoad/uploads/{uploadRef}/profile", "POST /bulkLoad/{tableName}/suggestMapping", "POST /bulkLoad/{tableName}/validate",
+      "POST /table/{tableName}/query", "GET /table/{tableName}/{primaryKey}", "POST /processes/{processName}/init");
+
+   private final List<UiApiCall> calls = new ArrayList<>();
+   private final boolean         offersBulkLoad;
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   public FakeBulkLoadServer(boolean offersBulkLoad)
+   {
+      this.offersBulkLoad = offersBulkLoad;
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public boolean isInvocable(String method, String pathTemplate)
+   {
+      String operation = method + " " + pathTemplate;
+      return (OPERATIONS.contains(operation) && (offersBulkLoad || !pathTemplate.startsWith("/bulkLoad")));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @Override
+   public synchronized UiApiResult invoke(UiApiCall call)
+   {
+      calls.add(call);
+      String operation = call.operation();
+      return (switch(operation)
+      {
+         case "GET /bulkLoad/uploads/{uploadRef}/profile" -> "up_1".equals(call.getPathParams().get("uploadRef"))
+            ? ok(new JSONObject().put("uploadRef", "up_1").put("fileName", "posts.csv").put("hasHeaderRow", true).put("rowsProfiled", 2)
+            .put("columns", new JSONArray()
+               .put(new JSONObject().put("columnIndex", 0).put("header", "Post Title").put("mappedFieldName", "title").put("samplesAllowed", true)
+                  .put("sampleValues", new JSONArray().put(SAMPLE_SENTINEL)))
+               .put(new JSONObject().put("columnIndex", 1).put("header", "Teaser").put("samplesAllowed", false))))
+            : new UiApiResult(404, "{\"error\":\"Upload not found.\"}");
+         case "POST /bulkLoad/{tableName}/suggestMapping" -> ok(new JSONObject()
+            .put("profile", new JSONObject().put("version", "v1").put("layout", "FLAT").put("hasHeaderRow", true)
+               .put("fieldList", new JSONArray().put(new JSONObject().put("fieldName", "title").put("columnIndex", 0).put("headerName", "Post Title"))))
+            .put("unmappedColumns", new JSONArray().put(new JSONObject().put("columnIndex", 1).put("header", "Teaser")))
+            .put("unmappedRequiredFields", new JSONArray().put(new JSONObject().put("fieldName", "status").put("label", "Status"))));
+         case "POST /bulkLoad/{tableName}/validate" -> validate(call);
+         case "POST /table/{tableName}/query" -> ok(new JSONObject().put("records", new JSONArray()
+            .put(new JSONObject().put("values", new JSONObject().put("id", 3).put("label", "Posts from the CMS export").put("userId", "admin-1")
+               .put("mappingJson", "{\"version\":\"v1\",\"fieldList\":[{\"fieldName\":\"title\",\"columnIndex\":0}]}")))));
+         case "GET /table/{tableName}/{primaryKey}" -> ok(new JSONObject().put("record", new JSONObject().put("values", new JSONObject().put("id", 3)
+            .put("label", "Posts from the CMS export").put("tableName", "blog_post").put("userId", "admin-1")
+            .put("mappingJson", "{\"version\":\"v1\",\"fieldList\":[{\"fieldName\":\"title\",\"columnIndex\":0}]}"))));
+         case "POST /processes/{processName}/init" -> "storeSavedBulkLoadProfile".equals(call.getPathParams().get("processName"))
+            ? ok(new JSONObject().put("type", "COMPLETE").put("values", new JSONObject().put("savedBulkLoadProfileList",
+            new JSONArray().put(new JSONObject().put("values", new JSONObject().put("id", 7).put("label", "x"))))))
+            : ok(new JSONObject().put("type", "COMPLETE").put("values", new JSONObject()));
+         default -> new UiApiResult(404, "{\"error\":\"no such operation\"}");
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Errors unless status is mapped or defaulted.
+    *******************************************************************************/
+   private UiApiResult validate(UiApiCall call)
+   {
+      JSONObject body       = new JSONObject(JsonUtils.toJson(call.getJsonBody()));
+      JSONArray  fieldList  = body.getJSONObject("profile").getJSONArray("fieldList");
+      boolean    hasStatus  = false;
+      for(int i = 0; i < fieldList.length(); i++)
+      {
+         JSONObject field = fieldList.getJSONObject(i);
+         if("status".equals(field.optString("fieldName")) && (field.has("columnIndex") || !field.isNull("defaultValue") && field.has("defaultValue")))
+         {
+            hasStatus = true;
+         }
+      }
+      JSONObject result = new JSONObject().put("mode", body.optString("mode", "SAMPLE")).put("processName", "blog_post.bulkInsert").put("recordCount", 2);
+      if(hasStatus)
+      {
+         return (ok(result.put("okCount", 2).put("errorCount", 0).put("errors", new JSONArray())));
+      }
+      return (ok(result.put("okCount", 0).put("errorCount", 2).put("errors", new JSONArray()
+         .put(new JSONObject().put("rowNumbers", "Rows 2-3").put("fieldName", "status").put("errorCode", "REQUIRED").put("message", "Missing value in required field: Status")))));
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   private static UiApiResult ok(JSONObject json)
+   {
+      return (new UiApiResult(200, json.toString()));
+   }
+
+
+
+   /*******************************************************************************
+    ** The calls, in order.
+    *******************************************************************************/
+   public synchronized List<UiApiCall> getCalls()
+   {
+      return (new ArrayList<>(calls));
+   }
+
+
+
+   /*******************************************************************************
+    ** Calls of one operation.
+    *******************************************************************************/
+   public synchronized List<UiApiCall> calls(String operation)
+   {
+      return (calls.stream().filter(c -> c.operation().equals(operation)).toList());
+   }
+
+
+
+   /*******************************************************************************
+    ** A form value of a process call, as JSON.
+    *******************************************************************************/
+   public static Map<String, Object> formValues(UiApiCall call)
+   {
+      return (new JSONObject(call.getFormParams().get("values")).toMap());
+   }
+}