Memory: facts_vec never written — semantic search permanently empty (FTS5-only RRF) #12

New Issue

2026-05-12T19:03:11Z

KofTwentyTwo commented

2026-05-12 19:03:11 +00:00

(Migrated from github.com)

Summary

facts_vec table is never written. Semantic search branch is permanently empty. Hybrid SQL's vc CTE returns zero rows on every call. RRF degenerates to FTS5-only ranking. Semantic recall ("dog" → "Toby" via embedding similarity) silently doesn't work despite the schema being present and vec0 loaded.

Evidence

File: packages/Memory/Sources/Memory/MemoryStore.swift:132-246 (applyOp / insertNewFact)
Live DB: SELECT COUNT(*) FROM facts_vec_rowids → 0 despite 1 fact present
SELECT COUNT(*) FROM facts → 1 (Toby fact has FTS row but no vec row)
vec_version() → v0.1.6 (extension loaded fine)
No INSERT INTO facts_vec exists anywhere in MemoryStore.swift, MemoryExtractionOrchestrator.process, or MemoryExtractor.extract

Root cause / hypothesis

applyOp inserts into facts and the schema's FTS5 trigger mirrors facts_fts automatically — but nothing inserts into facts_vec. The embedder client (OllamaEmbeddingClient) exists and works for HybridSearch, but no producer wires it to the writer side. Schema-vs-writer impedance mismatch — same shape as today's FK calibration bug.

Recommended fix

Inside insertNewFact, after INSERT INTO facts succeeds and newId is known, run the query embedder against "\(subject) \(predicate) \(object)" and INSERT INTO facts_vec(fact_id, embedding) VALUES (?, ?) inside the same transaction.

Two options for embedder injection:

(a) MemoryStore.init takes embedder: any EmbeddingProviding?; skip vec INSERT when nil
(b) Move embed+insert into MemoryExtractionOrchestrator.process after applyOp returns the new Fact

Failure mode discipline: log + leave the FTS row (degraded but searchable) rather than rolling back the fact row.

Test

Add MemoryStoreApplyOpVecInsertTests:

Given MemoryStore with embedder injected, after applyOp(.add) the facts_vec table contains exactly one row matching the new fact_id
Given embedder injection fails (nil), applyOp(.add) still succeeds and writes facts + FTS row; facts_vec row absent

Add parity assertion to MemoryBootHealthProbe: facts_vec row count tracks facts row count ± 1.

Source

Audit: .planning/audit-2026-05-12/memory-audit.md (2026-05-12) — finding M-1 CRITICAL

Cross-references

Related: #8 (M-4 Memory surface — natural fix landing site)
Related: M-6 finding (MemoryBootHealthProbe evidence misleading; parity check is structural defense against M-1-style bugs)

## Summary `facts_vec` table is never written. Semantic search branch is permanently empty. Hybrid SQL's `vc` CTE returns zero rows on every call. RRF degenerates to FTS5-only ranking. Semantic recall ("dog" → "Toby" via embedding similarity) silently doesn't work despite the schema being present and vec0 loaded. ## Evidence - File: `packages/Memory/Sources/Memory/MemoryStore.swift:132-246` (`applyOp` / `insertNewFact`) - Live DB: `SELECT COUNT(*) FROM facts_vec_rowids` → `0` despite 1 fact present - `SELECT COUNT(*) FROM facts` → `1` (Toby fact has FTS row but no vec row) - `vec_version()` → `v0.1.6` (extension loaded fine) - No `INSERT INTO facts_vec` exists anywhere in `MemoryStore.swift`, `MemoryExtractionOrchestrator.process`, or `MemoryExtractor.extract` ## Root cause / hypothesis `applyOp` inserts into `facts` and the schema's FTS5 trigger mirrors `facts_fts` automatically — but nothing inserts into `facts_vec`. The embedder client (`OllamaEmbeddingClient`) exists and works for `HybridSearch`, but no producer wires it to the writer side. Schema-vs-writer impedance mismatch — same shape as today's FK calibration bug. ## Recommended fix Inside `insertNewFact`, after `INSERT INTO facts` succeeds and `newId` is known, run the query embedder against `"\(subject) \(predicate) \(object)"` and `INSERT INTO facts_vec(fact_id, embedding) VALUES (?, ?)` inside the same transaction. Two options for embedder injection: - (a) `MemoryStore.init` takes `embedder: any EmbeddingProviding?`; skip vec INSERT when nil - (b) Move embed+insert into `MemoryExtractionOrchestrator.process` after `applyOp` returns the new `Fact` Failure mode discipline: log + leave the FTS row (degraded but searchable) rather than rolling back the fact row. ## Test Add `MemoryStoreApplyOpVecInsertTests`: - Given `MemoryStore` with embedder injected, after `applyOp(.add)` the `facts_vec` table contains exactly one row matching the new `fact_id` - Given embedder injection fails (nil), `applyOp(.add)` still succeeds and writes `facts` + FTS row; `facts_vec` row absent Add parity assertion to `MemoryBootHealthProbe`: `facts_vec` row count tracks `facts` row count ± 1. ## Source - Audit: `.planning/audit-2026-05-12/memory-audit.md` (2026-05-12) — finding M-1 CRITICAL ## Cross-references - Related: #8 (M-4 Memory surface — natural fix landing site) - Related: M-6 finding (`MemoryBootHealthProbe` evidence misleading; parity check is structural defense against M-1-style bugs)

KofTwentyTwo commented

2026-05-13 11:51:43 +00:00

(Migrated from github.com)

Closed by v0.1 cluster merges on 2026-05-13. Develop now at a8c7f1a0.

Closed by v0.1 cluster merges on 2026-05-13. Develop now at `a8c7f1a0`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#12