Memory: hybridSearchSQL passes raw user text to FTS5 MATCH — punctuation crashes queries #16

New Issue

2026-05-12T19:04:01Z

KofTwentyTwo commented

2026-05-12 19:04:01 +00:00

(Migrated from github.com)

Summary

MemoryQueries.hybridSearchSQL passes raw user text directly to facts_fts MATCH ? — punctuation crashes the query. Real user queries like "what is my dog's name?", "remember when I said X?", or anything with ?, :, -, ", *, or FTS5-reserved operators throws fts5: syntax error at the SQLite layer.

Evidence

File: packages/Memory/Sources/Memory/MemoryQueries.swift:33-63
File: packages/Memory/Sources/Memory/HybridSearch.swift:32-57
File: packages/Memory/Sources/Memory/MemoryStore.swift:311-339 (runHybridSearchSQL)
Live probe: sqlite3 jarvis.db "SELECT rowid FROM facts_fts WHERE facts_fts MATCH 'what is my dog name?' LIMIT 10" → Error: stepping, fts5: syntax error near "?". Same query without ? works.
No sanitization in packages/Memory/Sources/ or packages/MCP/Sources/MCP/InProcess/SearchMemoryTool.swift

Root cause / hypothesis

HybridSearch.searchFacts passes query straight through to MemoryStore.runHybridSearchSQL which binds it to MATCH ? positional parameter. FTS5's MATCH syntax accepts a small grammar (terms, AND/OR/NOT/NEAR, column filters, phrase quoting with "); raw natural language with ? or ' is a syntax error.

Today's replay shows zero search_memory calls (#14 / M-3 keeps it dormant), so we have no live evidence yet — but the moment M-3 is fixed, every natural-language query with punctuation will fail.

Recommended fix

Add an FTS5-safe query rewriter in HybridSearch.searchFacts before calling the store. Two-line approach:

let safeQuery = query
  .components(separatedBy: .punctuationCharacters)
  .joined(separator: " ")
  .components(separatedBy: .whitespacesAndNewlines)
  .filter { !$0.isEmpty }
  .map { "\"\($0)\"" }
  .joined(separator: " ")

Turns "what is my dog's name?" into "what" "is" "my" "dog" "s" "name". Phrase-OR query, FTS5-safe. Long-term: actual query AST.

Test

Unit: HybridSearchTests parameterized over ["what is my dog's name?", "remember when?", "name:Toby", "what's *up*"] — assert no thrown error and non-empty/empty results as appropriate
Integration: M-3 fix lands the agent calling search_memory with natural language; this test prevents the regression

Source

Audit: .planning/audit-2026-05-12/memory-audit.md (2026-05-12) — finding M-5 HIGH

Cross-references

Related: #14 (M-3 search_memory discoverability) — must land together for proactive recall to actually work

## Summary `MemoryQueries.hybridSearchSQL` passes raw user text directly to `facts_fts MATCH ?` — punctuation crashes the query. Real user queries like `"what is my dog's name?"`, `"remember when I said X?"`, or anything with `?`, `:`, `-`, `"`, `*`, or FTS5-reserved operators throws `fts5: syntax error` at the SQLite layer. ## Evidence - File: `packages/Memory/Sources/Memory/MemoryQueries.swift:33-63` - File: `packages/Memory/Sources/Memory/HybridSearch.swift:32-57` - File: `packages/Memory/Sources/Memory/MemoryStore.swift:311-339` (`runHybridSearchSQL`) - Live probe: `sqlite3 jarvis.db "SELECT rowid FROM facts_fts WHERE facts_fts MATCH 'what is my dog name?' LIMIT 10"` → `Error: stepping, fts5: syntax error near "?"`. Same query without `?` works. - No sanitization in `packages/Memory/Sources/` or `packages/MCP/Sources/MCP/InProcess/SearchMemoryTool.swift` ## Root cause / hypothesis `HybridSearch.searchFacts` passes `query` straight through to `MemoryStore.runHybridSearchSQL` which binds it to `MATCH ?` positional parameter. FTS5's MATCH syntax accepts a small grammar (terms, AND/OR/NOT/NEAR, column filters, phrase quoting with `"`); raw natural language with `?` or `'` is a syntax error. Today's replay shows zero `search_memory` calls (#14 / M-3 keeps it dormant), so we have no live evidence yet — but the moment M-3 is fixed, every natural-language query with punctuation will fail. ## Recommended fix Add an FTS5-safe query rewriter in `HybridSearch.searchFacts` before calling the store. Two-line approach: ```swift let safeQuery = query .components(separatedBy: .punctuationCharacters) .joined(separator: " ") .components(separatedBy: .whitespacesAndNewlines) .filter { !$0.isEmpty } .map { "\"\($0)\"" } .joined(separator: " ") ``` Turns `"what is my dog's name?"` into `"what" "is" "my" "dog" "s" "name"`. Phrase-OR query, FTS5-safe. Long-term: actual query AST. ## Test - Unit: `HybridSearchTests` parameterized over `["what is my dog's name?", "remember when?", "name:Toby", "what's *up*"]` — assert no thrown error and non-empty/empty results as appropriate - Integration: M-3 fix lands the agent calling `search_memory` with natural language; this test prevents the regression ## Source - Audit: `.planning/audit-2026-05-12/memory-audit.md` (2026-05-12) — finding M-5 HIGH ## Cross-references - Related: #14 (M-3 search_memory discoverability) — must land together for proactive recall to actually work

KofTwentyTwo commented

2026-05-13 11:51:50 +00:00

(Migrated from github.com)

Closed by v0.1 cluster merges on 2026-05-13. Develop now at a8c7f1a0.

Closed by v0.1 cluster merges on 2026-05-13. Develop now at `a8c7f1a0`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#16