KofTwentyTwo/Jarvis
2
0 Fork 0
Code Issues 111 Pull Requests 16 Actions Packages Projects Releases 4 Wiki Activity

Memory: replay log poisoned by synthetic memory-trigger-<hash> turn_ids; all batches FK-fail since 18:48 #13

New Issue
Closed
opened 2026-05-12 19:03:24 +00:00 by KofTwentyTwo · 1 comment
KofTwentyTwo commented 2026-05-12 19:03:24 +00:00 (Migrated from github.com)
Copy link
Copy source

Summary

Replay log poisoned: synthetic memory-trigger-<hash> turn_ids break ALL replay flushes since 2026-05-12T18:48:47.211Z. Every replay batch fails with SQLITE_CONSTRAINT_FOREIGNKEY (code 19); ReplayLog re-buffers the poisoned batch and retries forever. At 3 consecutive failures it escalates to CRITICAL. Real turn events queued behind the poison are silently lost.

Evidence

  • File: App/AppDelegate.swift:2783-2797 (AppDelegateMemoryReplaySink.record(_:forTriggerTurnId:))
  • File: packages/Replay/Sources/Replay/Schema.swift:15 (PRAGMA foreign_keys=ON)
  • File: packages/Replay/Sources/Replay/ReplayLog.swift:195-260 (flushPending batch transaction)
  • Log: ~/Library/Logs/Jarvis/replay.2026-05-12.log shows ~50+ chronic-failure entries with turn_ids=...,memory-trigger-7747877241524896942, batch_size=31 growing, consecutive_failures=23+
  • DB probe: SELECT kind, COUNT(*) FROM events GROUP BY kind in replay.sqlite shows zero memory_mutation rows ever, despite 1 fact written today

Root cause / hypothesis

MemoryStore.recordMemoryMutation (line 248-272) routes through the sink with a synthetic TurnID = "memory-trigger-7747877241524896942" (FNV hash formatted as string). replay.sqlite's events.turn_id REFERENCES turns(turn_id) is a real FK with foreign_keys=ON. No row in turns has that synthetic id → SQLITE_CONSTRAINT_FOREIGNKEY on INSERT. flushPending wraps the whole batch in one transaction, so the poisoned event takes every real event in the same batch down on rollback. Re-buffer logic doesn't filter; same poison gets retried forever.

Recommended fix

Two layers, pick at least one:

  1. Don't synthesize a fake turn_id. Thread the real triggering TurnID (UUID) through MemoryExtractionCoordinator → Orchestrator.process → applyOp → recordMemoryMutation. The orchestrator already has job.turnId — pass it through alongside the Int64 hash, use the real one when emitting to replay. OR add a sentinel turns row up-front for memory mutations (session_id = current, turn_id = "memory-system").

  2. Defensive batch-splitting in ReplayLog.flushPending. When a batch fails, retry events individually so one poisoned row can't take 30 others with it. Bounds blast radius even if a new poisoned-id pathway sneaks in later.

Layer 2 alone is the structural defense; layer 1 is the surgical fix.

Test

  • Regression: feed a synthetic forTriggerTurnId not present in turns → assert ReplayLog flush succeeds for the OTHER events in the batch and only the poisoned event is dropped/logged
  • Add observability assertion: ReplayLog exposes consecutive-failure count to MemoryBootHealthProbe

Source

  • Audit: .planning/audit-2026-05-12/memory-audit.md (2026-05-12) — finding M-2 CRITICAL

Cross-references

  • Related: M-8 (LOW) — MemoryReplaySink.record is fire-and-forget; failures unobservable from MemoryStore (compounds visibility gap)
  • Related: M-4 (HIGH) — same FNV-hash shape exists in facts.source_turn_id
## Summary Replay log poisoned: synthetic `memory-trigger-<hash>` turn_ids break ALL replay flushes since `2026-05-12T18:48:47.211Z`. Every replay batch fails with `SQLITE_CONSTRAINT_FOREIGNKEY` (code 19); ReplayLog re-buffers the poisoned batch and retries forever. At 3 consecutive failures it escalates to CRITICAL. Real turn events queued behind the poison are silently lost. ## Evidence - File: `App/AppDelegate.swift:2783-2797` (`AppDelegateMemoryReplaySink.record(_:forTriggerTurnId:)`) - File: `packages/Replay/Sources/Replay/Schema.swift:15` (`PRAGMA foreign_keys=ON`) - File: `packages/Replay/Sources/Replay/ReplayLog.swift:195-260` (`flushPending` batch transaction) - Log: `~/Library/Logs/Jarvis/replay.2026-05-12.log` shows ~50+ chronic-failure entries with `turn_ids=...,memory-trigger-7747877241524896942`, `batch_size=31` growing, `consecutive_failures=23+` - DB probe: `SELECT kind, COUNT(*) FROM events GROUP BY kind` in `replay.sqlite` shows zero `memory_mutation` rows ever, despite 1 fact written today ## Root cause / hypothesis `MemoryStore.recordMemoryMutation` (line 248-272) routes through the sink with a synthetic `TurnID = "memory-trigger-7747877241524896942"` (FNV hash formatted as string). `replay.sqlite`'s `events.turn_id REFERENCES turns(turn_id)` is a real FK with `foreign_keys=ON`. No row in `turns` has that synthetic id → `SQLITE_CONSTRAINT_FOREIGNKEY` on INSERT. `flushPending` wraps the whole batch in one transaction, so the poisoned event takes every real event in the same batch down on rollback. Re-buffer logic doesn't filter; same poison gets retried forever. ## Recommended fix Two layers, pick at least one: 1. **Don't synthesize a fake turn_id.** Thread the real triggering `TurnID` (UUID) through `MemoryExtractionCoordinator` → `Orchestrator.process` → `applyOp` → `recordMemoryMutation`. The orchestrator already has `job.turnId` — pass it through alongside the Int64 hash, use the real one when emitting to replay. OR add a sentinel `turns` row up-front for memory mutations (`session_id = current`, `turn_id = "memory-system"`). 2. **Defensive batch-splitting in `ReplayLog.flushPending`.** When a batch fails, retry events individually so one poisoned row can't take 30 others with it. Bounds blast radius even if a new poisoned-id pathway sneaks in later. Layer 2 alone is the structural defense; layer 1 is the surgical fix. ## Test - Regression: feed a synthetic `forTriggerTurnId` not present in `turns` → assert ReplayLog flush succeeds for the OTHER events in the batch and only the poisoned event is dropped/logged - Add observability assertion: `ReplayLog` exposes consecutive-failure count to `MemoryBootHealthProbe` ## Source - Audit: `.planning/audit-2026-05-12/memory-audit.md` (2026-05-12) — finding M-2 CRITICAL ## Cross-references - Related: M-8 (LOW) — `MemoryReplaySink.record` is fire-and-forget; failures unobservable from MemoryStore (compounds visibility gap) - Related: M-4 (HIGH) — same FNV-hash shape exists in `facts.source_turn_id`
KofTwentyTwo commented 2026-05-13 11:51:45 +00:00 (Migrated from github.com)
Copy link
Copy source

Closed by v0.1 cluster merges on 2026-05-13. Develop now at a8c7f1a0.

Closed by v0.1 cluster merges on 2026-05-13. Develop now at `a8c7f1a0`.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
area:agent
area:build
area:bus
area:devtool
area:hud
area:logging
area:mcp
area:memory
area:vision
area:voice
audit-finding
bug
Something isn't working
 carry-forward
comment-cleanup
Comment / docstring cleanup per .planning/code-commenting-guide.md
 dependencies
Dependency updates
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 epic
good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 javascript
Pull requests that update javascript code
 migration
question
Further information is requested
 severity:critical
severity:high
severity:low
severity:medium
wontfix
This will not be worked on
No labels area:logging area:memory audit-finding bug severity:critical
Milestone
No items
No Milestone v0.1 — It actually works (basic capabilities)
Projects
Clear projects
No projects
Assignees
Clear assignees
forge-recovery (Forge Recovery) james.maes (James Maes)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#13