feat: RAG over instance content, permission-filtered through the UI API (P-f) #33

2026-10-14T00:54:46Z

KofTwentyTwo commented

2026-10-14 00:54:46 +00:00

(Migrated from github.com)

P-f: RAG over the instance's content, permission-filtered through the UI API

Spec: docs/DESIGN.md part P, item 5. This is based on develop. Merge order: #32 (answer presentation, owner priority) goes first; I'll then merge develop into this branch, since both touch SystemPrompt, ContextBuilder and TurnEngine.

Security model

The vector index holds no record content: only table, primary key, field, chunk index, content hash and embedding. Text is read through /qqq/v1 with a real user's session:

at index time, as the admin who clicks Index now;
at answer time, as the asking user.

So retrieval returns only what that user may read in the UI.

Pieces

Embeddings: the EmbeddingProvider SPI, implemented by the OpenAI-compatible adapter (/embeddings, batches of up to 64, the Azure path, the same TLS, auth and timeouts).
- assistant_model gains model_kind (CHAT or EMBEDDING) and embedding_dimensions. Discovery recognizes bge, e5, gte, nomic and embed models. Chat selection, fallback and downgrade ignore embedding models.
- Embedding calls are EMBED usage and count toward budgets.
- The PHI rule applies: PHI sources and PHI-conversation queries are embedded only by eligible providers.
Vector store: a VectorStore SPI with an in-memory store, plus the new module qbit-assistant-rag-pgvector:
- one schema per instance, with assistant_rag_chunk and an HNSW cosine index;
- parameterized SQL;
- ensureSchema, plus a documented DBA step for CREATE EXTENSION vector.
Knowledge sources: assistant_rag_source is in the nav as "Knowledge sources": table, fields, filter, embedding model and chunking.
- Validation: no assistant_* tables, data access must not be NONE, and fields with AI exposure NONE or METADATA_ONLY are not allowed.
- Record actions: Index now (streamed ETL with Review and Result; skips unchanged chunks by hash and removes vanished records) and Clear index.
search_knowledge tool:
1. Embeds the query and runs the vector search.
2. Re-reads every hit with GET /table/{t}/{pk}, using the asking user's session. Hits that return 403 or 404 are dropped and counted only as "filtered".
3. Applies AI field exposure and masking.
4. Returns snippets with /app citations.
Results raise the turn's classification (P-c).
Scheduled re-indexing comes with P-g.

Tests

275 in total, plus 1 that runs only with Docker; all green in ./mvnw verify with the JaCoCo gates.

Indexing: chunking and hashing; a re-run embeds nothing; deleted records are removed; the index holds no text.
Retrieval:
- a permitted user gets hits with citations;
- a user without read permission gets zero hits (the same 403 as the UI);
- record locks filter results;
- data access NONE refused;
- exposure NONE fields refused; masked fields stay masked;
- PHI with a non-eligible embedding provider is refused;
- classification is raised.
Embeddings: contract tests (batching, Azure path, errors).
pgvector: SQL-shape tests, and the Testcontainers test against pgvector/pgvector:pg16, which passes locally with Docker.

Also in this PR: contingent possible values (owner request)

assistant_provider.model and kind, and assistant_model_price.model, now depend on providerType. Each is a CUSTOM possible-value provider that reads otherValues.providerType:

it returns that type's models, plus the provider's enabled chat models from the registry, and that type's kinds;
without a providerType, it returns the union, so saved rows and API callers keep working;
getPossibleValue resolves any saved id.

possibleValueSourceDependsOn("providerType") is added once the next QQQ RC ships it. There are five new tests in AssistantSettingsTest.

🤖 Generated with Claude Code

## P-f: RAG over the instance's content, permission-filtered through the UI API Spec: `docs/DESIGN.md` part P, item 5. This is based on `develop`. **Merge order:** #32 (answer presentation, owner priority) goes first; I'll then merge develop into this branch, since both touch `SystemPrompt`, `ContextBuilder` and `TurnEngine`. ### Security model The vector index holds **no record content**: only table, primary key, field, chunk index, content hash and embedding. Text is read through `/qqq/v1` with a real user's session: - at index time, as the admin who clicks **Index now**; - at answer time, as the asking user. So retrieval returns only what that user may read in the UI. ### Pieces - **Embeddings:** the `EmbeddingProvider` SPI, implemented by the OpenAI-compatible adapter (`/embeddings`, batches of up to 64, the Azure path, the same TLS, auth and timeouts). - `assistant_model` gains `model_kind` (CHAT or EMBEDDING) and `embedding_dimensions`. Discovery recognizes bge, e5, gte, nomic and embed models. Chat selection, fallback and downgrade ignore embedding models. - Embedding calls are `EMBED` usage and count toward budgets. - The PHI rule applies: PHI sources and PHI-conversation queries are embedded only by eligible providers. - **Vector store:** a `VectorStore` SPI with an in-memory store, plus the new module **`qbit-assistant-rag-pgvector`**: - one schema per instance, with `assistant_rag_chunk` and an HNSW cosine index; - parameterized SQL; - `ensureSchema`, plus a documented DBA step for `CREATE EXTENSION vector`. - **Knowledge sources:** **`assistant_rag_source`** is in the nav as "Knowledge sources": table, fields, filter, embedding model and chunking. - Validation: no `assistant_*` tables, data access must not be NONE, and fields with AI exposure NONE or METADATA_ONLY are not allowed. - Record actions: **Index now** (streamed ETL with Review and Result; skips unchanged chunks by hash and removes vanished records) and **Clear index**. - **`search_knowledge` tool:** 1. Embeds the query and runs the vector search. 2. Re-reads **every hit** with `GET /table/{t}/{pk}`, using the asking user's session. Hits that return 403 or 404 are dropped and counted only as "filtered". 3. Applies AI field exposure and masking. 4. Returns snippets with `/app` citations. Results raise the turn's classification (P-c). - Scheduled re-indexing comes with P-g. ### Tests 275 in total, plus 1 that runs only with Docker; all green in `./mvnw verify` with the JaCoCo gates. - **Indexing:** chunking and hashing; a re-run embeds nothing; deleted records are removed; the index holds no text. - **Retrieval:** - a permitted user gets hits with citations; - a user without read permission gets zero hits (the same 403 as the UI); - record locks filter results; - data access NONE refused; - exposure NONE fields refused; masked fields stay masked; - PHI with a non-eligible embedding provider is refused; - classification is raised. - **Embeddings:** contract tests (batching, Azure path, errors). - **pgvector:** SQL-shape tests, and **the Testcontainers test against `pgvector/pgvector:pg16`, which passes locally with Docker.** ### Also in this PR: contingent possible values (owner request) `assistant_provider.model` and `kind`, and `assistant_model_price.model`, now depend on `providerType`. Each is a CUSTOM possible-value provider that reads `otherValues.providerType`: - it returns that type's models, plus the provider's enabled chat models from the registry, and that type's kinds; - without a `providerType`, it returns the union, so saved rows and API callers keep working; - `getPossibleValue` resolves any saved id. `possibleValueSourceDependsOn("providerType")` is added once the next QQQ RC ships it. There are five new tests in `AssistantSettingsTest`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-assistant#33