Compare commits

..
Author SHA1 Message Date
james.maesandClaude Opus 5.5 1bae6e26ba test(defaults): keep the reference DDL's DO $$ block whole and skip it on H2
The foreign-key block is PL/pgSQL; the H2 loader in
AssistantDefaultsChangelogTest now splits it as one statement and skips it.
SchemaForeignKeyPostgresTest runs it on real Postgres.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:55:19 -05:00
james.maesandClaude Opus 5.5 80d529e741 Merge feature/schema-indexes (with origin/develop) into feature/foreign-keys
Conflicts in CHANGELOG.md and qbit-assistant/pom.xml: kept develop's
durable-turns and starter-defaults entries and its H2/Liquibase test
dependencies alongside the Testcontainers/Postgres ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:55:19 -05:00
james.maesandClaude Opus 5.5 b50bc4a5c7 Merge origin/develop into feature/schema-indexes
Conflicts in UPGRADE.md and docs/schema-postgres.sql: kept develop's
durable-turns and starter-defaults notes and DDL; the index audit block
now follows the durable-turns block at the end of the schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:51:27 -05:00
james.maes 6a1bac6a9e Merge pull request #52 from QRun-IO/fix/assistant-defaults-rollback
fix(defaults): empty rollbacks; a rollback never deletes host data
2026-10-14 17:12:49 -05:00
james.maesandClaude Opus 5.5 9c3af04a91 fix(defaults): empty rollbacks so a rollback never deletes host data
A seeded row cannot be told apart from a host's own row with the same name
('default' profile, policy, starter skill names), and Liquibase also rolls
back changesets it marked ran, so the name-based rollback lines could delete
a host's data. Every seed changeset now rolls back to nothing; the test runs
a real Liquibase rollback over host-owned rows and checks they survive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:08:13 -05:00
james.maes 0bfa4e4d01 Merge pull request #49 from QRun-IO/feature/durable-turns
fix: durable assistant turns - lease, heartbeat, cross-pod Stop, never refuse a question
2026-10-14 17:06:18 -05:00
james.maesandClaude Opus 5.5 fdf8d72367 feat(schema): add the missing foreign keys with orphan quarantine
Reference DDL gains a guarded, re-runnable foreign key block after the
index audit: the conversation tree CASCADEs (matching
AssistantStore.deleteConversation and retention), the message/turn cycle,
usage and the data audit are SET NULL so cost rows and the audit trail
survive, provider references from usage, budget accounting and the data
audit are NO ACTION, profile-skill links CASCADE, and
assistant_model.provider_id becomes CASCADE (fk_assistant_model_provider,
as on kof22). Orphans are recorded in qqq_fk_quarantine first. Every FK
column now leads an index.

SchemaForeignKeyPostgresTest applies the DDL on Postgres (Testcontainers)
and fails any declared relation without a foreign key, a wrong delete
rule, an unindexed FK column, or a broken conversation delete order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:23:25 -05:00
james.maesandClaude Opus 5.5 8aac6cf92d docs(schema): add the missing lookup, retention and FK indexes
The reference Postgres schema gains six IF NOT EXISTS indexes:
- assistant_usage (turn_id): ScheduledTurn.cost
- assistant_turn_event (create_date), assistant_tool_call (create_date) and
  assistant_conversation (modify_date): the retention filters
- assistant_scheduled_run (conversation_id) and (turn_id): the ON DELETE
  SET NULL foreign keys

UPGRADE.md notes them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:26:08 -05:00
7 changed files with 624 additions and 14 deletions

No files matched your search

+9
View File
@@ -1,5 +1,12 @@
# Changelog
## Unreleased - Foreign keys
- Reference DDL: add the missing foreign keys (conversation tree, usage, data audit,
budget accounting, profile skills) with orphan quarantine and their indexes; make
`assistant_model.provider_id` `ON DELETE CASCADE`. A Postgres guardrail test fails
any declared relation without a foreign key.
## Unreleased - Durable turns
- Renew user-owned run leases with heartbeats, poll cross-replica Stop, enforce
@@ -23,6 +30,8 @@
## Unreleased - Starter defaults
- Every seed changeset's rollback is empty: a seeded row cannot be told apart from a host's own row with the same name, and Liquibase also rolls back changesets it marked ran, so a name-based undo would delete host data.
- Fix the defaults changelog: every precondition is a portable `sql-check` (the formatted-SQL `table-exists` form failed to parse); the policy, budget and prices are seeded only into empty tables and the profile only when no default exists, so an existing host never changes on deploy. The test now runs the changelog through Liquibase 4.33, including an already configured host and a missing table.
- The Setup card prices the default model the way calls are costed (registry, price rows, configured prices); "Add rows for every table" reads fresh data-access rows.
+28
View File
@@ -1,5 +1,33 @@
# Upgrading qbit-assistant
## Foreign keys (2026-10-14)
Recommended: apply the "Foreign keys" block at the end of
`docs/schema-postgres.sql` (after the index audit). It is guarded and safe to re-run.
- Adds the missing foreign keys across the conversation tables, usage, data audit,
budget accounting and profile skills, with the indexes they need.
- Detail rows of a conversation `CASCADE`; the message/turn cycle, usage and the data
audit use `SET NULL`, so usage keeps its cost rows when a conversation is deleted.
- Usage, budget accounting and the data audit hold their provider (`NO ACTION`):
disable a provider with history instead of deleting it.
- `assistant_model.provider_id` changes from `NO ACTION` to `CASCADE`
(constraint renamed `fk_assistant_model_provider`).
- Rows that point at a missing parent are first copied to `qqq_fk_quarantine`
(created if absent), then nulled, or deleted for the `NOT NULL` profile-skill
links. Check that table after the upgrade.
No code change.
## Index audit (2026-10-14)
Recommended, not required: apply the "Index audit" block at the end of
`docs/schema-postgres.sql`. It adds six indexes, every one `IF NOT EXISTS`:
- usage by turn (read on each scheduled run)
- the three retention date filters (turn events, tool calls, inactive conversations)
- the two `ON DELETE SET NULL` foreign keys on `assistant_scheduled_run`
No code change.
## Durable turns (Unreleased)
Apply the **Durable turns** block at the end of `docs/schema-postgres.sql` before
+109
View File
@@ -592,3 +592,112 @@ CREATE INDEX assistant_turn_user_status ON assistant_turn (user_id, status);
-- Check for pre-existing duplicate message.end rows before applying this index.
CREATE UNIQUE INDEX assistant_turn_event_terminal ON assistant_turn_event (turn_id)
WHERE event_type = 'message.end';
-- Index audit (2026-10-14): lookup, retention and FK paths with no leading index.
-- ScheduledTurn.cost reads usage by turn on every scheduled run.
CREATE INDEX IF NOT EXISTS assistant_usage_turn ON assistant_usage (turn_id);
-- Retention: deleteTurnEventsBefore, clearToolResultsBefore and listConversationsInactiveSince filter by date.
CREATE INDEX IF NOT EXISTS assistant_turn_event_create_date ON assistant_turn_event (create_date);
CREATE INDEX IF NOT EXISTS assistant_tool_call_create_date ON assistant_tool_call (create_date);
CREATE INDEX IF NOT EXISTS assistant_conversation_modify_date ON assistant_conversation (modify_date);
-- ON DELETE SET NULL children: deleting a conversation or turn would scan scheduled runs without these.
CREATE INDEX IF NOT EXISTS assistant_scheduled_run_conversation ON assistant_scheduled_run (conversation_id);
CREATE INDEX IF NOT EXISTS assistant_scheduled_run_turn ON assistant_scheduled_run (turn_id);
-- Foreign keys (2026-10-14). Run on new and existing databases; every step is guarded
-- and re-runnable. The rules follow how the qbit deletes:
-- * AssistantStore.deleteConversation (also the retention purge) deletes turn events,
-- tool calls, feedback, messages, turns, data audit and pending actions, then the
-- conversation. The conversation's own detail rows CASCADE, so an admin delete of a
-- conversation cleans up the same way.
-- * The message <-> turn cycle (turn.user_message_id / assistant_message_id and
-- message.turn_id) and the other optional pointers are SET NULL. Rows are inserted
-- parent first, so nothing is deferred.
-- * assistant_usage keeps its cost rows when a conversation goes (SET NULL), and the
-- data audit trail is never cascaded into (SET NULL).
-- * Provider references from usage, budget accounting and the data audit are NO ACTION
-- (like assistant_turn.provider_id): disable a provider with history, do not delete
-- it. A provider's model rows are its own (CASCADE).
-- * assistant_budget_usage.usage_id has no foreign key on purpose: the projection
-- outlives its usage row so budgets keep counting the spend.
-- Rows that already point at a missing parent are recorded in qqq_fk_quarantine first,
-- then nulled (nullable columns) or deleted with the full row recorded (NOT NULL).
CREATE INDEX IF NOT EXISTS assistant_message_parent_message ON assistant_message (parent_message_id);
CREATE INDEX IF NOT EXISTS assistant_message_turn ON assistant_message (turn_id);
CREATE INDEX IF NOT EXISTS assistant_turn_user_message ON assistant_turn (user_message_id);
CREATE INDEX IF NOT EXISTS assistant_turn_assistant_message ON assistant_turn (assistant_message_id);
CREATE INDEX IF NOT EXISTS assistant_turn_provider ON assistant_turn (provider_id);
CREATE INDEX IF NOT EXISTS assistant_tool_call_pending_action ON assistant_tool_call (pending_action_id);
CREATE INDEX IF NOT EXISTS assistant_feedback_message ON assistant_feedback (message_id);
CREATE INDEX IF NOT EXISTS assistant_pending_action_turn ON assistant_pending_action (turn_id);
CREATE INDEX IF NOT EXISTS assistant_data_audit_provider ON assistant_data_audit (provider_id);
CREATE INDEX IF NOT EXISTS assistant_budget_usage_provider ON assistant_budget_usage (provider_id);
CREATE INDEX IF NOT EXISTS assistant_profile_skill_skill ON assistant_profile_skill (skill_id);
CREATE INDEX IF NOT EXISTS assistant_budget_downgrade_model ON assistant_budget (downgrade_model_id);
CREATE INDEX IF NOT EXISTS assistant_conversation_scheduled_task ON assistant_conversation (scheduled_task_id);
CREATE INDEX IF NOT EXISTS assistant_model_fallback_model ON assistant_model (fallback_model_id);
CREATE INDEX IF NOT EXISTS assistant_profile_simple_model ON assistant_profile (simple_model_id);
CREATE INDEX IF NOT EXISTS assistant_profile_complex_model ON assistant_profile (complex_model_id);
CREATE INDEX IF NOT EXISTS assistant_profile_classifier_model ON assistant_profile (classifier_model_id);
CREATE INDEX IF NOT EXISTS assistant_rag_source_embedding_model ON assistant_rag_source (embedding_model_id);
CREATE INDEX IF NOT EXISTS assistant_scheduled_task_profile ON assistant_scheduled_task (profile_id);
CREATE TABLE IF NOT EXISTS qqq_fk_quarantine (id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, table_name VARCHAR(100) NOT NULL, column_name VARCHAR(100) NOT NULL, row_id VARCHAR(100), missing_parent_id VARCHAR(100), action VARCHAR(20) NOT NULL, row_data TEXT, changeset VARCHAR(200), quarantined_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP);
DO $$
DECLARE
r RECORD;
old TEXT;
BEGIN
FOR r IN SELECT * FROM (VALUES
('assistant_message', 'conversation_id', 'assistant_conversation', 'CASCADE', TRUE),
('assistant_message', 'parent_message_id', 'assistant_message', 'SET NULL', TRUE),
('assistant_message', 'turn_id', 'assistant_turn', 'SET NULL', TRUE),
('assistant_turn', 'conversation_id', 'assistant_conversation', 'CASCADE', TRUE),
('assistant_turn', 'user_message_id', 'assistant_message', 'SET NULL', TRUE),
('assistant_turn', 'assistant_message_id', 'assistant_message', 'SET NULL', TRUE),
('assistant_turn_event', 'turn_id', 'assistant_turn', 'CASCADE', TRUE),
('assistant_tool_call', 'conversation_id', 'assistant_conversation', 'CASCADE', TRUE),
('assistant_tool_call', 'turn_id', 'assistant_turn', 'CASCADE', TRUE),
('assistant_tool_call', 'pending_action_id', 'assistant_pending_action', 'SET NULL', TRUE),
('assistant_feedback', 'conversation_id', 'assistant_conversation', 'CASCADE', TRUE),
('assistant_feedback', 'message_id', 'assistant_message', 'CASCADE', TRUE),
('assistant_pending_action', 'conversation_id', 'assistant_conversation', 'CASCADE', TRUE),
('assistant_pending_action', 'turn_id', 'assistant_turn', 'SET NULL', TRUE),
('assistant_data_audit', 'conversation_id', 'assistant_conversation', 'SET NULL', TRUE),
('assistant_data_audit', 'turn_id', 'assistant_turn', 'SET NULL', TRUE),
('assistant_usage', 'conversation_id', 'assistant_conversation', 'SET NULL', TRUE),
('assistant_usage', 'turn_id', 'assistant_turn', 'SET NULL', TRUE),
('assistant_usage', 'provider_id', 'assistant_provider', 'NO ACTION', TRUE),
('assistant_data_audit', 'provider_id', 'assistant_provider', 'NO ACTION', TRUE),
('assistant_budget_usage', 'provider_id', 'assistant_provider', 'NO ACTION', TRUE),
('assistant_profile_skill', 'profile_id', 'assistant_profile', 'CASCADE', FALSE),
('assistant_profile_skill', 'skill_id', 'assistant_skill', 'CASCADE', FALSE)
) AS v(child, col, parent, rule, nullable)
LOOP
IF NOT EXISTS (SELECT 1 FROM information_schema.table_constraints WHERE constraint_schema = current_schema() AND LOWER(constraint_name) = 'fk_' || r.child || '_' || r.col) THEN
IF r.nullable THEN
EXECUTE format('INSERT INTO qqq_fk_quarantine (table_name, column_name, row_id, missing_parent_id, action, changeset) '
|| 'SELECT %L, %L, c.id::text, c.%I::text, ''NULLED'', ''qbit-assistant 2026-10-14 foreign keys'' FROM %I c '
|| 'WHERE c.%I IS NOT NULL AND NOT EXISTS (SELECT 1 FROM %I p WHERE p.id = c.%I)', r.child, r.col, r.col, r.child, r.col, r.parent, r.col);
EXECUTE format('UPDATE %I c SET %I = NULL WHERE c.%I IS NOT NULL AND NOT EXISTS (SELECT 1 FROM %I p WHERE p.id = c.%I)', r.child, r.col, r.col, r.parent, r.col);
ELSE
EXECUTE format('INSERT INTO qqq_fk_quarantine (table_name, column_name, row_id, missing_parent_id, action, row_data, changeset) '
|| 'SELECT %L, %L, c.id::text, c.%I::text, ''DELETED'', row_to_json(c)::text, ''qbit-assistant 2026-10-14 foreign keys'' FROM %I c '
|| 'WHERE NOT EXISTS (SELECT 1 FROM %I p WHERE p.id = c.%I)', r.child, r.col, r.col, r.child, r.parent, r.col);
EXECUTE format('DELETE FROM %I c WHERE NOT EXISTS (SELECT 1 FROM %I p WHERE p.id = c.%I)', r.child, r.parent, r.col);
END IF;
EXECUTE format('ALTER TABLE %I ADD CONSTRAINT %I FOREIGN KEY (%I) REFERENCES %I (id) ON DELETE %s', r.child, 'fk_' || r.child || '_' || r.col, r.col, r.parent, r.rule);
END IF;
END LOOP;
-- assistant_model.provider_id: the inline NO ACTION key becomes CASCADE, named as on kof22.
IF NOT EXISTS (SELECT 1 FROM information_schema.table_constraints WHERE constraint_schema = current_schema() AND LOWER(constraint_name) = 'fk_assistant_model_provider') THEN
FOR old IN SELECT conname FROM pg_constraint
WHERE contype = 'f' AND conrelid = 'assistant_model'::regclass AND confrelid = 'assistant_provider'::regclass
LOOP
EXECUTE format('ALTER TABLE assistant_model DROP CONSTRAINT %I', old);
END LOOP;
ALTER TABLE assistant_model ADD CONSTRAINT fk_assistant_model_provider FOREIGN KEY (provider_id) REFERENCES assistant_provider (id) ON DELETE CASCADE;
END IF;
END $$;
+19
View File
@@ -87,6 +87,25 @@
<version>4.33.0</version>
<scope>test</scope>
</dependency>
<!-- the reference DDL guardrail (SchemaForeignKeyPostgresTest) -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>1.21.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<version>1.21.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.7.14</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
@@ -9,6 +9,9 @@
-- Rules for this file:
-- * insert-only: every row is inserted only when its natural key is absent,
-- so an administrator's edits always survive and nothing is ever updated;
-- * rollback is empty: a seeded row cannot be told apart from a host's own
-- row with the same name, and Liquibase also rolls back changesets it marked
-- ran, so undoing would delete the host's data (remove seeded rows by hand);
-- * no secrets: environment variable NAMES only, never values;
-- * portable SQL (PostgreSQL and H2): INSERT ... SELECT ... WHERE NOT EXISTS;
-- * every changeset has a precondition on its table, so a host that has not
@@ -22,7 +25,7 @@
INSERT INTO assistant_policy (name, kill_switch, writes_require_confirmation, redact_pii, max_rows_per_call, create_date, modify_date)
SELECT 'default', FALSE, TRUE, TRUE, 50, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_policy);
-- rollback DELETE FROM assistant_policy WHERE name = 'default';
-- rollback empty
-- changeset qbit-assistant:defaults-2
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -37,7 +40,7 @@ Which tables can I look at?
What changed in the last 7 days?',
TRUE, TRUE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_profile WHERE name = 'default' OR is_default = TRUE);
-- rollback DELETE FROM assistant_profile WHERE name = 'default';
-- rollback empty
-- changeset qbit-assistant:defaults-3
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -53,7 +56,7 @@ SELECT 'find-and-count', 'Finding, filtering and counting records.',
5. If the result was truncated, say so and offer the link instead of listing more.',
TRUE, 'find,show,list,which,how many,count,filter,search,look up', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_skill WHERE name = 'find-and-count');
-- rollback DELETE FROM assistant_skill WHERE name = 'find-and-count';
-- rollback empty
-- changeset qbit-assistant:defaults-4
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -68,7 +71,7 @@ SELECT 'explain-a-record', 'Summarizing a record and who changed it.',
4. Link to the record.',
TRUE, 'explain,summarize,summary,tell me about,what is,status,history,changed,who changed,audit', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_skill WHERE name = 'explain-a-record');
-- rollback DELETE FROM assistant_skill WHERE name = 'explain-a-record';
-- rollback empty
-- changeset qbit-assistant:defaults-5
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -83,7 +86,7 @@ SELECT 'safe-changes', 'Proposing creates, updates and deletes safely.',
4. For many rows from a file, use the bulk load tools and always run bulk_load_dry_run and show its summary before bulk_load_run.',
TRUE, 'change,update,set,edit,create,add,new,delete,remove,fix,rename,import,load', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_skill WHERE name = 'safe-changes');
-- rollback DELETE FROM assistant_skill WHERE name = 'safe-changes';
-- rollback empty
-- changeset qbit-assistant:defaults-6
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -97,7 +100,7 @@ SELECT 'access-questions', 'Explaining why data is missing or not available.',
3. Suggest the next step: check the filter, open the list screen, or ask an administrator for access.',
TRUE, 'cannot see,can''t see,missing,not found,denied,access,permission,not allowed,hidden', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_skill WHERE name = 'access-questions');
-- rollback DELETE FROM assistant_skill WHERE name = 'access-questions';
-- rollback empty
-- changeset qbit-assistant:defaults-7
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -109,7 +112,7 @@ FROM assistant_profile p, assistant_skill s
WHERE p.name = 'default'
AND s.name IN ('find-and-count', 'explain-a-record', 'safe-changes', 'access-questions')
AND NOT EXISTS (SELECT 1 FROM assistant_profile_skill ps WHERE ps.profile_id = p.id AND ps.skill_id = s.id);
-- rollback DELETE FROM assistant_profile_skill WHERE profile_id IN (SELECT id FROM assistant_profile WHERE name = 'default') AND skill_id IN (SELECT id FROM assistant_skill WHERE name IN ('find-and-count', 'explain-a-record', 'safe-changes', 'access-questions'));
-- rollback empty
-- changeset qbit-assistant:defaults-8
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -120,7 +123,7 @@ SELECT 'Monthly spend (whole instance)', TRUE, 'INSTANCE', 'MONTH', 'COST_USD',
'Starter budget. Warns everyone at 80% of $50 a month, switches to the cheapest allowed model at 100% (or stops when there is none), and stops at 120%. Set Downgrade Model to choose the fallback; add an override to raise it temporarily.',
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM assistant_budget);
-- rollback DELETE FROM assistant_budget WHERE name = 'Monthly spend (whole instance)';
-- rollback empty
-- changeset qbit-assistant:defaults-9
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -143,7 +146,7 @@ FROM (VALUES
('anthropic', 'claude-haiku-4-5', 1.00, 5.00, 0.10, 1.25, 2.00, 'Anthropic list price, 2026-10')
) AS v (provider_type, model, input_price, output_price, cache_read, cache_write5m, cache_write1h, notes)
WHERE NOT EXISTS (SELECT 1 FROM assistant_model_price p);
-- rollback SELECT 1; -- prices may have been edited; keep them
-- rollback empty
-- changeset qbit-assistant:defaults-10
-- preconditions onFail:MARK_RAN onError:MARK_RAN
@@ -156,4 +159,4 @@ SELECT 'Example: weekly change summary', FALSE,
CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
FROM (SELECT MIN(id) AS id FROM assistant_profile WHERE name = 'default') p
WHERE NOT EXISTS (SELECT 1 FROM assistant_scheduled_task WHERE name = 'Example: weekly change summary');
-- rollback DELETE FROM assistant_scheduled_task WHERE name = 'Example: weekly change summary' AND enabled = FALSE;
-- rollback empty
@@ -144,6 +144,33 @@ class AssistantDefaultsChangelogTest extends BaseAssistantTest
/*******************************************************************************
** Rolling the changelog back never deletes data: not the host's own rows
** (changesets marked ran included), not the seeded ones.
*******************************************************************************/
@Test
void rollbackKeepsEveryRow() throws Exception
{
try(Connection connection = freshSchema(); Statement statement = connection.createStatement())
{
statement.execute("INSERT INTO assistant_policy (name, kill_switch) VALUES ('default', FALSE)");
statement.execute("INSERT INTO assistant_profile (name, display_name, is_default, enabled) VALUES ('default', 'Hal', TRUE, TRUE)");
statement.execute("INSERT INTO assistant_skill (name, instructions) VALUES ('find-and-count', 'The host wrote this.')");
runChangelog(connection);
Map<String, Integer> before = counts(connection);
Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(connection));
new Liquibase(CHANGELOG.substring(1), new ClassLoaderResourceAccessor(), database).rollback(10, (String) null);
assertThat(ranChangesets(connection)).isEmpty();
assertThat(counts(connection)).isEqualTo(before);
assertThat(scalar(connection, "SELECT display_name FROM assistant_profile WHERE name = 'default'")).isEqualTo("Hal");
assertThat(scalar(connection, "SELECT instructions FROM assistant_skill WHERE name = 'find-and-count'")).isEqualTo("The host wrote this.");
}
}
/*******************************************************************************
** A host without one of the tables: that changeset is marked ran, the
** rest apply, nothing fails.
@@ -174,7 +201,7 @@ class AssistantDefaultsChangelogTest extends BaseAssistantTest
for(String changeset : changesets)
{
assertThat(changeset).contains("-- preconditions onFail:MARK_RAN onError:MARK_RAN").contains("-- precondition-sql-check expectedResult:0 ")
.doesNotContain("precondition-table-exists").contains("-- rollback ").contains("NOT EXISTS");
.doesNotContain("precondition-table-exists").contains("-- rollback empty").contains("NOT EXISTS");
}
for(String table : SEEDED_TABLES)
{
@@ -318,6 +345,11 @@ class AssistantDefaultsChangelogTest extends BaseAssistantTest
{
for(String sql : statements(ddl))
{
if(sql.startsWith("DO $$"))
{
// the foreign-key block is PL/pgSQL (Postgres only); SchemaForeignKeyPostgresTest runs it on real Postgres
continue;
}
Matcher m = Pattern.compile("^(CREATE TABLE|ALTER TABLE) (assistant_[a-z_]+)\\b(.*)", Pattern.DOTALL).matcher(sql);
if(m.find() && DDL_TABLES.contains(m.group(2)) && (m.group(1).startsWith("CREATE") || m.group(3).strip().startsWith("ADD COLUMN")))
{
@@ -393,7 +425,8 @@ class AssistantDefaultsChangelogTest extends BaseAssistantTest
/*******************************************************************************
** SQL statements, without comment lines, split at a ';' that ends a line.
** SQL statements, without comment lines, split at a ';' that ends a line;
** a DO $$ ... $$ block stays one statement.
*******************************************************************************/
static List<String> statements(String sql)
{
@@ -406,14 +439,36 @@ class AssistantDefaultsChangelogTest extends BaseAssistantTest
text.append(stripped).append("\n");
}
}
List<String> rs = new ArrayList<>();
List<String> rs = new ArrayList<>();
StringBuilder block = null;
for(String statement : text.toString().split(";\\s*\\n"))
{
String stripped = statement.strip();
if(block == null && stripped.startsWith("DO $$") && !stripped.endsWith("$$"))
{
// a dollar-quoted block has its own semicolons: keep it whole up to its closing $$
block = new StringBuilder(stripped);
continue;
}
if(block != null)
{
block.append(";\n").append(stripped);
if(stripped.endsWith("$$"))
{
rs.add(block.toString());
block = null;
}
continue;
}
if(!statement.isBlank())
{
rs.add(statement.strip().replaceAll(";$", ""));
rs.add(stripped.replaceAll(";$", ""));
}
}
if(block != null)
{
rs.add(block.toString());
}
return (rs);
}
@@ -0,0 +1,387 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.assistant.persistence;
import java.nio.file.Files;
import java.nio.file.Path;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.assistant.testing.BaseAssistantTest;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.PostgreSQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Guardrail for the reference DDL (docs/schema-postgres.sql) on a real
** PostgreSQL: every relation the qbit declares (a table possible-value source
** on an assistant_ field, or a join between assistant_ tables), plus the
** relations only the code knows, has a foreign key with the intended ON DELETE
** rule, and every foreign key column leads an index. A new relation without a
** foreign key fails here unless it is added to EXEMPT with a reason.
*******************************************************************************/
@Testcontainers(disabledWithoutDocker = true)
class SchemaForeignKeyPostgresTest extends BaseAssistantTest
{
@Container
static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>(DockerImageName.parse("pgvector/pgvector:pg16").asCompatibleSubstituteFor("postgres"));
/////////////////////////////////////////////////////////////////////////
// relations the meta-data does not declare (no possible-value source) //
/////////////////////////////////////////////////////////////////////////
private static final List<Relation> CODE_ONLY = List.of(
new Relation("assistant_data_audit", "turn_id", "assistant_turn"),
new Relation("assistant_data_audit", "provider_id", "assistant_provider"),
new Relation("assistant_budget_usage", "provider_id", "assistant_provider"));
//////////////////////////////////////////////////////////////////
// relations that deliberately have no foreign key, with reason //
//////////////////////////////////////////////////////////////////
private static final Map<Relation, String> EXEMPT = Map.of(
new Relation("assistant_budget_usage", "usage_id", "assistant_usage"),
"budget accounting projection: it must outlive its usage row so budgets keep counting the spend");
///////////////////////////////////////////////////////////////////////////
// the intended ON DELETE rule (pg confdeltype: a NO ACTION, c CASCADE, //
// n SET NULL) - see the foreign key block in docs/schema-postgres.sql //
///////////////////////////////////////////////////////////////////////////
private static final Map<Relation, String> RULES = new LinkedHashMap<>();
static
{
rule("assistant_message", "conversation_id", "assistant_conversation", "c");
rule("assistant_message", "parent_message_id", "assistant_message", "n");
rule("assistant_message", "turn_id", "assistant_turn", "n");
rule("assistant_turn", "conversation_id", "assistant_conversation", "c");
rule("assistant_turn", "user_message_id", "assistant_message", "n");
rule("assistant_turn", "assistant_message_id", "assistant_message", "n");
rule("assistant_turn_event", "turn_id", "assistant_turn", "c");
rule("assistant_tool_call", "conversation_id", "assistant_conversation", "c");
rule("assistant_tool_call", "turn_id", "assistant_turn", "c");
rule("assistant_tool_call", "pending_action_id", "assistant_pending_action", "n");
rule("assistant_feedback", "conversation_id", "assistant_conversation", "c");
rule("assistant_feedback", "message_id", "assistant_message", "c");
rule("assistant_pending_action", "conversation_id", "assistant_conversation", "c");
rule("assistant_pending_action", "turn_id", "assistant_turn", "n");
rule("assistant_data_audit", "conversation_id", "assistant_conversation", "n");
rule("assistant_data_audit", "turn_id", "assistant_turn", "n");
rule("assistant_data_audit", "provider_id", "assistant_provider", "a");
rule("assistant_usage", "conversation_id", "assistant_conversation", "n");
rule("assistant_usage", "turn_id", "assistant_turn", "n");
rule("assistant_usage", "provider_id", "assistant_provider", "a");
rule("assistant_budget_usage", "provider_id", "assistant_provider", "a");
rule("assistant_profile_skill", "profile_id", "assistant_profile", "c");
rule("assistant_profile_skill", "skill_id", "assistant_skill", "c");
rule("assistant_model", "provider_id", "assistant_provider", "c");
rule("assistant_scheduled_run", "conversation_id", "assistant_conversation", "n");
rule("assistant_scheduled_run", "turn_id", "assistant_turn", "n");
}
/*******************************************************************************
**
*******************************************************************************/
record Relation(String child, String column, String parent)
{
@Override
public String toString()
{
return (child + "." + column + " -> " + parent);
}
}
/*******************************************************************************
**
*******************************************************************************/
private static void rule(String child, String column, String parent, String rule)
{
RULES.put(new Relation(child, column, parent), rule);
}
/*******************************************************************************
** Apply the whole reference DDL, then its foreign key block a second time
** (it must be re-runnable on an existing database).
*******************************************************************************/
@BeforeAll
static void applyDdl() throws Exception
{
String ddl = Files.readString(Path.of("..", "docs", "schema-postgres.sql"));
try(Connection connection = connect(); Statement statement = connection.createStatement())
{
statement.execute(ddl);
int start = ddl.indexOf("-- Foreign keys (2026-10-14)");
if(start >= 0)
{
statement.execute(ddl.substring(start));
}
}
}
/*******************************************************************************
** Every declared or code-only relation has a foreign key, unless exempt.
*******************************************************************************/
@Test
void everyRelationHasAForeignKey() throws Exception
{
Set<Relation> relations = new LinkedHashSet<>(declaredRelations());
assertThat(relations).as("declared relations found in the meta-data").hasSizeGreaterThan(20);
relations.addAll(CODE_ONLY);
Map<Relation, String> foreignKeys = foreignKeys();
List<String> missing = new ArrayList<>();
for(Relation relation : relations)
{
if(!foreignKeys.containsKey(relation) && !EXEMPT.containsKey(relation))
{
missing.add(relation.toString());
}
}
assertThat(missing).as("relations without a foreign key (add one to docs/schema-postgres.sql, or to EXEMPT with a reason)").isEmpty();
for(Relation exempt : EXEMPT.keySet())
{
assertThat(foreignKeys).as("exempt relation " + exempt + " must stay without a foreign key").doesNotContainKey(exempt);
}
}
/*******************************************************************************
** The ON DELETE rules match how the qbit deletes.
*******************************************************************************/
@Test
void deleteRulesMatchTheDeletePaths() throws Exception
{
Map<Relation, String> foreignKeys = foreignKeys();
for(Map.Entry<Relation, String> entry : RULES.entrySet())
{
assertThat(foreignKeys.get(entry.getKey())).as(entry.getKey().toString()).isEqualTo(entry.getValue());
}
}
/*******************************************************************************
** Every foreign key column is the first column of some index.
*******************************************************************************/
@Test
void everyForeignKeyColumnLeadsAnIndex() throws Exception
{
List<String> unindexed = new ArrayList<>();
try(Connection connection = connect(); Statement statement = connection.createStatement();
ResultSet rs = statement.executeQuery("""
SELECT c.conrelid::regclass::text, a.attname
FROM pg_constraint c
JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = c.conkey[1]
WHERE c.contype = 'f' AND c.connamespace = 'public'::regnamespace
AND NOT EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = c.conrelid AND i.indkey[0] = c.conkey[1])
"""))
{
while(rs.next())
{
unindexed.add(rs.getString(1) + "." + rs.getString(2));
}
}
assertThat(unindexed).isEmpty();
}
/*******************************************************************************
** AssistantStore.deleteConversation's delete order works under the foreign
** keys; usage (cost accounting) survives with its pointers nulled. A plain
** delete of a conversation (admin screen) cascades through the
** message <-> turn cycle and keeps usage and the data audit.
*******************************************************************************/
@Test
void conversationDeletesWorkUnderTheForeignKeys() throws Exception
{
try(Connection connection = connect(); Statement statement = connection.createStatement())
{
connection.setAutoCommit(false);
for(int c : new int[] { 1, 2 })
{
int m = c * 10;
statement.execute("INSERT INTO assistant_provider (id, name, provider_type) VALUES (" + c + ", 'p" + c + "', 'anthropic')");
statement.execute("INSERT INTO assistant_conversation (id, user_id) VALUES (" + c + ", 'u')");
statement.execute("INSERT INTO assistant_message (id, conversation_id, user_id, role) VALUES (" + m + ", " + c + ", 'u', 'USER')");
statement.execute("INSERT INTO assistant_turn (id, conversation_id, user_id, user_message_id, provider_id) VALUES (" + c + ", " + c + ", 'u', " + m + ", NULL)");
statement.execute("INSERT INTO assistant_message (id, conversation_id, user_id, role, turn_id, parent_message_id) VALUES (" + (m + 1) + ", " + c + ", 'u', 'ASSISTANT', " + c + ", " + m + ")");
statement.execute("UPDATE assistant_message SET turn_id = " + c + " WHERE id = " + m);
statement.execute("UPDATE assistant_turn SET assistant_message_id = " + (m + 1) + " WHERE id = " + c);
statement.execute("INSERT INTO assistant_turn_event (turn_id, user_id) VALUES (" + c + ", 'u')");
statement.execute("INSERT INTO assistant_pending_action (id, conversation_id, turn_id, user_id) VALUES (" + c + ", " + c + ", " + c + ", 'u')");
statement.execute("INSERT INTO assistant_tool_call (conversation_id, turn_id, pending_action_id, user_id) VALUES (" + c + ", " + c + ", " + c + ", 'u')");
statement.execute("INSERT INTO assistant_feedback (conversation_id, message_id, user_id) VALUES (" + c + ", " + (m + 1) + ", 'u')");
statement.execute("INSERT INTO assistant_data_audit (id, conversation_id, turn_id, provider_id, user_id) VALUES (" + c + ", " + c + ", " + c + ", " + c + ", 'u')");
statement.execute("INSERT INTO assistant_usage (id, conversation_id, turn_id, provider_id, user_id) VALUES (" + c + ", " + c + ", " + c + ", " + c + ", 'u')");
}
////////////////////////////////////////////////////////////
// conversation 1: AssistantStore.deleteConversation order //
////////////////////////////////////////////////////////////
statement.execute("DELETE FROM assistant_turn_event WHERE turn_id IN (SELECT id FROM assistant_turn WHERE conversation_id = 1)");
for(String table : List.of("assistant_tool_call", "assistant_feedback", "assistant_message", "assistant_turn", "assistant_data_audit", "assistant_pending_action"))
{
statement.execute("DELETE FROM " + table + " WHERE conversation_id = 1");
}
statement.execute("DELETE FROM assistant_conversation WHERE id = 1");
assertThat(queryString(statement, "SELECT conversation_id || '/' || turn_id FROM assistant_usage WHERE id = 1")).as("usage kept, pointers nulled").isNull();
assertThat(queryString(statement, "SELECT provider_id::text FROM assistant_usage WHERE id = 1")).isEqualTo("1");
//////////////////////////////////////////////
// conversation 2: a plain conversation delete //
//////////////////////////////////////////////
statement.execute("DELETE FROM assistant_conversation WHERE id = 2");
for(String table : List.of("assistant_message", "assistant_turn", "assistant_tool_call", "assistant_feedback", "assistant_pending_action"))
{
assertThat(queryString(statement, "SELECT COUNT(*)::text FROM " + table + " WHERE conversation_id = 2")).as(table).isEqualTo("0");
}
assertThat(queryString(statement, "SELECT COUNT(*)::text FROM assistant_turn_event")).isEqualTo("0");
assertThat(queryString(statement, "SELECT (conversation_id IS NULL AND turn_id IS NULL)::text FROM assistant_data_audit WHERE id = 2")).as("data audit kept").isEqualTo("true");
assertThat(queryString(statement, "SELECT (conversation_id IS NULL AND turn_id IS NULL)::text FROM assistant_usage WHERE id = 2")).as("usage kept").isEqualTo("true");
connection.rollback();
}
}
/*******************************************************************************
** Possible-value-source fields and joins between assistant_ tables.
*******************************************************************************/
private List<Relation> declaredRelations()
{
List<Relation> relations = new ArrayList<>();
for(QTableMetaData table : qInstance.getTables().values())
{
if(!table.getName().startsWith("assistant_"))
{
continue;
}
for(QFieldMetaData field : table.getFields().values())
{
QPossibleValueSource pvs = field.getPossibleValueSourceName() == null ? null : qInstance.getPossibleValueSource(field.getPossibleValueSourceName());
if(pvs != null && pvs.getType() == QPossibleValueSourceType.TABLE && pvs.getTableName().startsWith("assistant_"))
{
relations.add(new Relation(table.getName(), snake(field.getName()), pvs.getTableName()));
}
}
}
for(QJoinMetaData join : qInstance.getJoins().values())
{
if(!join.getLeftTable().startsWith("assistant_") || !join.getRightTable().startsWith("assistant_"))
{
continue;
}
for(JoinOn on : join.getJoinOns())
{
if("id".equals(on.getLeftField()))
{
relations.add(new Relation(join.getRightTable(), snake(on.getRightField()), join.getLeftTable()));
}
else if("id".equals(on.getRightField()))
{
relations.add(new Relation(join.getLeftTable(), snake(on.getLeftField()), join.getRightTable()));
}
}
}
return (relations);
}
/*******************************************************************************
** Single-column foreign keys in the database, with their delete rule.
*******************************************************************************/
private static Map<Relation, String> foreignKeys() throws Exception
{
Map<Relation, String> foreignKeys = new LinkedHashMap<>();
try(Connection connection = connect(); Statement statement = connection.createStatement();
ResultSet rs = statement.executeQuery("""
SELECT c.conrelid::regclass::text, a.attname, c.confrelid::regclass::text, c.confdeltype
FROM pg_constraint c
JOIN pg_attribute a ON a.attrelid = c.conrelid AND a.attnum = c.conkey[1]
WHERE c.contype = 'f' AND array_length(c.conkey, 1) = 1 AND c.connamespace = 'public'::regnamespace
"""))
{
while(rs.next())
{
foreignKeys.put(new Relation(rs.getString(1), rs.getString(2), rs.getString(3)), rs.getString(4));
}
}
return (foreignKeys);
}
/*******************************************************************************
**
*******************************************************************************/
private static String queryString(Statement statement, String sql) throws Exception
{
try(ResultSet rs = statement.executeQuery(sql))
{
return (rs.next() ? rs.getString(1) : null);
}
}
/*******************************************************************************
**
*******************************************************************************/
private static String snake(String camel)
{
return (camel.replaceAll("([A-Z])", "_$1").toLowerCase());
}
/*******************************************************************************
**
*******************************************************************************/
private static Connection connect() throws Exception
{
return (DriverManager.getConnection(POSTGRES.getJdbcUrl(), POSTGRES.getUsername(), POSTGRES.getPassword()));
}
}