Commit Graph
17 Commits
Author SHA1 Message Date
james.maes f59f5d12e3 fix(memory): v0.1 CRIT cluster — facts_vec / replay correlation / FTS5 sanitize / search_memory preamble (#12, #13, #14, #15, #16, #153)
#14 (CRIT): ContextBuilder.selfAwarePreamble names search_memory + forget_fact with explicit cue 'call search_memory FIRST. Never claim to remember without searching.'

#16 (HIGH): MemoryQueries.sanitizeFTS5Query strips non-alphanumeric, quotes tokens; HybridSearch.searchFacts calls it before binding to MATCH ?. Embedding still runs on raw query.

#12 (CRIT): MemoryStore.init accepts optional embedder; insertNewFact writes facts_vec after the facts commit. Vec failures log + leave FTS row (no rollback). installMemory shares one embedder between writer + reader.

#15 (HIGH): drop stableHash; pass sentinel 0 as sourceTurnId (option 1 per audit — smaller change surface than threading real rowid through 5 files). Schema comment updated.

#13 (CRIT): two-layer fix. Layer 1: AppDelegate owns a per-process sentinel TurnID; installAgent writes the matching turns row after beginSession; AppDelegateMemoryReplaySink attributes events to it. Layer 2: ReplayLog.flushPending retries failed batches event-by-event so one poisoned row only drops itself.

Memory 98 + Replay 34 + AgentCore 243 tests pass; all 18 boundary gates green.
2026-05-13 06:30:35 -05:00
james.maesandClaude Opus 4.7 cb584a2ac8 feat(d5d6): real memory database — vec0 statically linked, hard-fail boot, stats tool
Closes D-5 / D-6 (sqlite-vec dylib bundling) carry-forwards from the
2026-05-04 audit by adopting jkrukowski/SQLiteVec as a SPM dependency.
The package's CSQLiteVec target compiles sqlite3.c + sqlite-vec.c into
one statically-linked C target with `core_vec_init()` registering vec0
as an auto-extension. All sqlite3_* calls in Replay's SQLiteConnection
and Memory's MemoryStore resolve to this bundled SQLite (vec0 always
available) instead of Apple's stripped /usr/lib/libsqlite3.dylib.

The retired hand-rolled approach (dlsym pathway + JARVIS_VEC0_STUB_PATH
env-flag + Bundle.module placeholder text file) never worked on macOS
because Apple stripped sqlite3_load_extension from the system libsqlite3.
Memory init has been silently dormant since Phase 7 (2026-04 onward),
which is what made B-02 (history threading) appear permanently broken
to the user during dogfooding even after the 48ad8d6 tactical patch
landed.

Live verification 2026-05-11: fresh launch with empty jarvis.db creates
schema, persists turn rows across submits, B-02 multi-turn context
resolves correctly ("what breed is Toby?" → "Golden Retriever" after
prior turn established Toby is a Golden Retriever), and the new
get_memory_stats tool reports `turnsCount=6, vecVersion=v0.1.6,
sqliteVersion=3.51.0`.

Changes:

- packages/Replay/Package.swift
    + SQLiteVec SPM dependency (from: "0.0.14")
    + CSQLiteVec product wired into Replay target
- packages/Replay/Sources/Replay/SQLiteConnection.swift
    - import SQLite3
    + @_exported import CSQLiteVec
    All sqlite3_* call sites unchanged — linker now resolves to our
    bundled sqlite3 with vec0 statically linked in.
- packages/Memory/Package.swift
    + SQLiteVec SPM dependency (named product so MemoryStore can import
      CSQLiteVec directly for the core_vec_init() one-shot call)
    - resources: PLACEHOLDER.txt copy (defunct vec0.dylib stub slot)
- packages/Memory/Sources/Memory/MemoryStore.swift
    - import SQLite3 → import CSQLiteVec
    + _vecAutoExtensionRegistered static-let one-shot: calls
      CSQLiteVec.core_vec_init() which registers vec0 via
      sqlite3_auto_extension on the first MemoryStore construction.
      Idempotent (static-let initializer fires exactly once).
    - loadVecExtension(on:) dlsym/JARVIS_VEC0_STUB_PATH/Bundle.module
      pathway (retired; ~70 lines removed)
    Updated docstring to reflect the new hard-fail boot policy.
- packages/Memory/Sources/Memory/Resources/PLACEHOLDER.txt — deleted
- App/AppDelegate.swift installMemory()
    Reverses Track-D D-1 (75a10be, 2026-05-04): MemoryStore.init failures
    are now FATAL via Logger.critical + TCCAlertService.presentHardBlock
    + NSApp.terminate. The graceful-degradation path that kept the app
    booting with memoryStore=nil hid the entire B-02/B-04/B-05 silent-
    failure bug class. Aligns with the 2026-05-07 API-first pivot
    thesis: "no silent failures".
    `store` is now non-optional throughout installMemory; downstream
    callers no longer need `if let store = store`.
- packages/MCP/Sources/MCP/InProcess/GetMemoryStatsTool.swift (new)
    + MemoryStatsDispatching protocol
    + MemoryStats Codable struct (turnsCount, factsTotal, factsActive,
      sessionsCount, dbFileBytes, vecVersion, sqliteVersion,
      lastTurnAtUnixMs, lastFactAtUnixMs)
    + GetMemoryStatsTool conforming to InProcessTool
- App/MCP/InProcessMemoryAdapters.swift
    + MemoryStatsStoreAdapter — bridges the protocol to MemoryStore's
      public queryRowCount / querySingleString seams and FileManager
      for the on-disk file size. Read-only, no confirmation.
- App/AppDelegate.swift installSelfKnowledgeTools()
    + Registers GetMemoryStatsTool with MemoryStatsStoreAdapter.
- packages/Memory/Tests/MemoryTests/*
    Retired the JARVIS_VEC0_STUB_PATH XCTSkipUnless guards from 9 tests
    (vec0 is always available now — no need to env-gate). Rewrote
    MemoryStoreTests to drop the now-impossible "init throws when vec
    missing" case and assert facts_vec virtual table creation +
    vec_version() readback. Added seedTurnRow helper to
    MemoryStoreApplyOpTests so the now-enforced
    `facts.source_turn_id REFERENCES turns(id)` foreign key is honored
    in tests that previously passed when FK enforcement was dormant.

Test summary: Memory full suite 91 tests, 10 skipped (env-gated real-
Ollama tests), 0 failures. App target builds clean. Substrate verified
end-to-end via live launch (vec_version: v0.1.6 logged at boot;
get_memory_stats returns actual schema row counts).

Pending D-7 carry-forward: ollama pull nomic-embed-text (~270MB) for
embedder; user already has qwen2.5-coder:32b-instruct-q8_0 which is
a higher-quality variant of the project's hardcoded
qwen2.5-coder:32b — small ModelID config delta to follow.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 10:25:56 -05:00
james.maesandClaude Opus 4.7 508cf924d0 docs(packages): per-package README files
Adds README.md to each of the 13 SPM packages (AgentCore, Bus, Config,
DevOverlay, Harness, Keychain, Logging, MCP, Memory, Replay, Shell, Vision,
Voice). Each covers purpose, key public types, deps, consumers, invariants
(pulled from CLAUDE.md + grep-gates + audit findings), test stack, and
notable files. Memory README explicitly marks the package functionally OFF
until vec0.dylib + Ollama models land.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-04 17:13:33 -05:00
james.maesandClaude Opus 4.7 048ab08ef6 fix(app,agent,replay): wire orchestrator install correctly + extract bus forwarder for testability
The 2026-05-03 INT-3 smoke test surfaced three latent runtime bugs the
unit-level GSD verifications all missed because the audits were source-
grep, not runtime-trace:

1. **F-E-RACE-1** — `agentInstallTask` awaited memory + vision installs
   but NOT the MCP runtime build task. MCPRuntime build is ~1s due to
   helper child-process spawn, so installAgent ALWAYS short-circuited on
   `mcpRuntime == nil` and the orchestrator + broadcaster + ALL six
   broadcaster subscribers stayed dormant for the entire process
   lifetime. Fix: capture `mcpInstallTask` Task handle and await it
   before calling installAgent. The v0.12.0 milestone audit's claim of
   "5 broadcaster subscribers wired" was accurate at source-grep level
   and false at runtime.

2. **F-E-FK-1** — installAgent constructed the orchestrator with a
   fresh SessionID but never called `replayLog.beginSession(...)`.
   Every `startTurn` violated the FK and returned
   `SubmitOutcome.rejected(reason: .configError)` — surfacing in the
   chat panel as "Config error — see ~/Library/Logs/Jarvis/system.log."
   Production code only ever called beginSession in tests
   (MCPRuntimeWiringTests). Fix: installAgent now reads
   CFBundleShortVersionString + CFBundleVersion from Info.plist, calls
   beginSession, and passes the returned SessionID to the orchestrator.

3. **F-E-WIRE-1** — `OutboundBatcher` was constructed only inside
   `installVoice`, AFTER OpenWakeWord/Silero models loaded. Voice DAG
   short-circuit on missing models in Debug builds left
   `self.outboundBatcher == nil`, and every `outboundBatcher?...`
   chained-optional in the .bus subscriber silently dropped tokenDelta
   / turnStarted / turnEnded / submitRejected. The chat panel got
   nothing back from the orchestrator. Fix: hoist batcher construction
   into installAgent (only depends on webviewBridge, not voice models);
   installVoice reuses self.outboundBatcher rather than overwriting.

In addition, the .bus subscriber's translation logic — including
StopReason → TurnTerminator mapping, turnStarted synthesis on first
emission for a fresh TurnID, error → submitRejected forwarding for
mid-stream failures — was inline in AppDelegate and therefore
impossible to unit-test (App target xctest harness is upstream-broken
on Xcode 26). Extracted to `BusForwarder` in AgentOrchestrator with a
sink protocol; App provides `AppBusForwarderSink` (~50 LOC) as the
glue. Every translation branch now has unit coverage.

Files:
- packages/AgentCore/Sources/AgentOrchestrator/BusForwarder.swift —
  Pure logic. `BusForwarderSink` protocol; `BusForwarder.terminator(for:)`
  StopReason mapping; `BusForwarder.drain(events:sink:)` async loop with
  per-turn lastEmittedTurnId state. No Bus package dep — keeps the
  layering clean.
- packages/AgentCore/Tests/AgentOrchestratorTests/BusForwarderTests.swift
  — 16 cases covering every branch: terminator mapping (5, exhaustive
  on StopReason), tokenDelta forwarding (3 — synthesize start, dedupe
  start across same-turn deltas, two-turn handoff), turnEnd forwarding
  (4 — mapped terminator, synthesize-start-on-bare-turnEnd,
  streamTruncated→errored, state reset), error forwarding (2), ignored
  events (1), malformed-id guard (1).
- packages/Replay/Tests/ReplayTests/SessionForeignKeyTests.swift —
  3 cases (FK1: startTurn-without-beginSession-throws; FK2: post-
  beginSession-succeeds; FK3: multiple-startTurns-share-one-session).
  Closes the bug class regardless of any future AppDelegate refactor.
- App/AppBusForwarderSink.swift — thin transport adapter. Translates
  `BusForwarder.Terminator` to wire-format `Bus.TurnTerminator` and
  dispatches across OutboundBatcher actor.
- App/AppDelegate.swift — race fix, beginSession call, batcher hoist,
  inline subscriber replaced with `BusForwarder.drain(events: sink:)`.
- Jarvis.xcodeproj/project.pbxproj — xcodegen regen picks up new App
  source file.
- .planning/AUDIT-FINDINGS.md — three new findings (F-E-RACE-1,
  F-E-FK-1, F-E-WIRE-1) documented and marked closed; refactor
  rationale recorded; carry-forward items called out
  (streamTruncatedFinal cause unverified without Keychain access; F1
  chat-turn end-to-end test is next).

Verification:
- AgentCore 194/194 (was 178; +16 BusForwarder).
- Replay 33/33 (was 30; +3 SessionFK).
- Bus 55/56 (F-A2-01 pre-existing).
- All 15 boundary grep gates PASS.
- scripts/check-app-builds.sh PASS.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-03 09:44:23 -05:00
james.maesandClaude Opus 4.7 13c8033028 feat(09-02): add VisionRouter dispatch + post-response escalation + voice-turn tracking (BLOCKER-2)
- ReplayEvent: new .escalationAttempt(turnId:kind:) case with .t1ToT2 EscalationKind (D-02 marker, distinct from AGENT-09 stream_truncated)
- ReplayEventKind: new .escalationAttempt = "escalation_attempt" rawvalue
- AgentOrchestrator: constructor gains optional visionRouter: VisionRouter? = nil
- runTurn pre-stream: route image-bearing turns through VisionRouter, swap provider, track imageBearingTurns (D-01, D-16)
- runTurnLoop: multimodal stream overload for image-bearing turns; assistantTextSoFar accumulator on .textDelta
- runTurnLoop .endTurn arm: post-response evaluatePostResponse hook, .escalateToT2 → discard T1 message, swap to T2, continue outer (WARNING-3 LOCKED labeled-loop strategy); records .escalationAttempt under SAME turnId (D-02)
- BLOCKER-2: voiceOriginatedTurns Set<TurnID> + recordVoiceTurn(_:) populator + turnSourceWasVoice(_:) public accessor (Plan 4 voice subscriber will gate emitTurnEnded on this)
- turnHadImage(_:) public accessor (Plan 2 frame-attach release subscriber consumer)
- AgentCore + Replay + MCP packages bumped from .v13 to .v14 (cascade — JarvisVision is .v14, project deployment target was already 14.0)
- MockLLMProvider extended with multimodal stream overload + RecordedCall.images
- 6 AgentOrchestratorVisionDispatchTests + 3 AgentOrchestratorVoiceTurnTrackingTests + 3 EscalationAttemptMarkerTests
- All 63 existing AgentOrchestratorTests still pass; all 30 Replay tests pass; orchestrator-events single-consumer grep gate exits 0

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 18:49:25 -05:00
james.maesandClaude Opus 4.7 20fd8aba67 feat(08-01): scaffold packages/Harness + extend TurnSource (Strategy B)
Plan 08-01 Task 1. New `packages/Harness` SPM module with:
- library target `Harness` (Placeholder.swift version stamp)
- executable target `jarvis-eval` (ArgumentParser CLI shell, no subcommands yet)
- test target `HarnessTests` (swift-testing PlaceholderTests smoke)
- `Corpora/` directory placeholder for golden replay sessions (Plan 08-04
  populates via `scripts/promote-replay-session.sh`)

Dependencies mirror PATTERNS.md §2.1: Logging, Config, AgentCore, Replay,
JarvisMCP, Voice, Memory, DevOverlay, swift-log 1.5+, swift-argument-parser
1.5+. Platform floor `.macOS(.v14)` matches Voice (ORT 1.24.2).

TurnSource migrated to Strategy B (CONTEXT D-01, PATTERNS §2.2 alternative):
- Drops `String, RawRepresentable, CaseIterable` synthesized conformances
- Adds associated-value cases `.replay(sessionId: UUID)` and
  `.evaluation(scenarioId: String)` so ReplayMCPAdapter (Task 3) can read the
  recorded session payload via the case payload — no sidecar context needed
- Hand-rolls `rawValue` (SQLite `turns.source` column writer at
  ReplayLog.swift:106 keeps working — produces "text"/"voice"/
  "memoryExtraction"/"replay"/"evaluation")
- Adds `dispatchesToHUD` and `speaks` computed properties (R4-L7
  suppression by construction): only `.text` / `.voice` return `true`;
  `.memoryExtraction` / `.replay` / `.evaluation` all return `false`

Migration call-site impact: zero. There are NO `switch source` consumers
project-wide — only `source.rawValue` at ReplayLog.swift:106 and field
declarations in TurnInput / ExtractionJob / AgentOrchestrator / BusOutbound
(strings, not switches). All packages build green; 27/27 Replay tests pass.

project.yml gains `Harness:` entry under packages.

Verifies: D-01, D-02, D-03 (PlaceholderTests uses swift-testing).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 08:16:14 -05:00
james.maes aa31e753bc feat(07-05): FrameAttachController + always-confirm gate + sole emission site (Task 6 GREEN)
- FrameConfirmationDecision: .send/.cancel/.timeout (default-cancel
  semantic — anything other than .send discards)
- FrameAttachController actor:
  * requestAttach(reason:) is the SINGLE entry point — phrase + HUD
    button both reach it (D-13 dual-trigger)
  * confirmSend(userText:) returns the ImageBlock the orchestrator
    routes; replays the placeholder via FrameAttachReplaySink
  * cancel() and timeout-fired both discard via the SOLE emission site
  * onAssistantTurnComplete() releases bytes on LLMEvent.messageStop edge
  * discardFrame() is the SOLE function that clears pendingFrame —
    'pendingFrame = nil' appears EXACTLY ONCE in the file
  * Cancellation-aware timeout task (do/catch around Task.sleep) so
    confirmSend cancellation cleanly suppresses .timeout firing
- FrameAttachReplaySink: actor that records the placeholder JSON via
  the injected ReplayLogProtocol; raw image bytes never enter the sink
- ReplayEvent.userInput doc comment: documents the D-15 placeholder
  convention for image-bearing turns
- All 10 new tests + 37 existing tests pass; 4-leg grep gate enforces
  single emission + raw-bytes egress fence
2026-04-29 18:13:43 -05:00
james.maes 5387a311ae feat(07-03): memory read primitives + .memoryRetrieval single emission site
- MemoryQueries: hybrid RRF SQL (RESEARCH §8 verbatim, k=60, active-row
  predicate, k literal interpolation per P9), sessionHistorySQL,
  forgetFactSQL (UPDATE-only), activeFactsBySubjectSQL
- MemoryEvent: FactRef gains score/triggerTurnId/timestamp + Codable;
  TurnRow added (TEXT-03 chat-panel hydration shape)
- MemoryError: forgetTargetNotFound case (reserved)
- MemoryStore: recordRetrieval (sole .memoryRetrieval emission site,
  D-05/MEM-08), runHybridSearchSQL, queryActiveFacts,
  recentTurnsForSession, forgetFact (closes valid_to + forgotten_at,
  never DELETEs)
- ReplayEvent: .memoryRetrieval(Data) + encoded() arm
- Schema: ReplayEventKind.memoryRetrieval rawValue
- SQLiteConnection: columnDouble accessor for RRF score readback

56 Memory tests (5 env-gated skipped), 27 Replay tests, 0 failures.
.memoryRetrieval single-emission-site grep gate green.
2026-04-29 17:00:28 -05:00
james.maes 53206d550b feat(07-02): MemoryStore.applyOp single emission site + MemoryOp parser (Task 1 GREEN)
- MemoryEvent.swift: MemoryOp enum (.add/.update/.noop) + parseApplyMemoryOps schema-validating parser + FactRef
- ExtractionJob.swift: value type carrying turnId/userText/assistantText/subjects/source
- MemoryError: appended embeddingNonLoopbackHost / extractorInvalidArgs / extractorMissingTool
- MemoryStore.applyOp: MEM-05 supersede transaction (UPDATE valid_to + superseded_by, never DELETE), MEM-06 SOLE emission site for ReplayEvent.memoryMutation
- MemoryReplaySink protocol decouples Memory from concrete ReplayLog
- ReplayEvent: append .memoryMutation(Data) case + encoded() arm
- ReplayEventKind: append memory_mutation rawValue
- All Memory tests pass (4 env-gated DB tests skip cleanly); 27 Replay tests still pass
2026-04-29 16:11:45 -05:00
james.maes 4ee018d95c feat(07-01): add SQLiteConnection.withHandle + MemoryConstants + symbol-uniqueness test
- SQLiteConnection.withHandle(_:): closure-based accessor so MemoryStore
  can call sqlite3_load_extension without leaking the OpaquePointer.
- MemoryConstants.embeddingDim = 768 (MEM-02): single canonical symbol
  for vector dimension. Hard-pinned to nomic-embed-text 768d output.
- EmbeddingDimSymbolTests:
    testEmbeddingDimIsSeventySixtyEight — value sanity.
    testEmbeddingDimIsSingleSymbol — workspace grep guard against any
    parallel literal definition.

Deviation Rule 1: the plan-supplied repoRoot heuristic walked up looking
for 'a Package.swift WITHOUT a parent Package.swift', but this workspace
has no top-level Package.swift (it uses XcodeGen project.yml). Walk up
looking for project.yml or Jarvis.xcodeproj instead.

Deviation Rule 1: the plan-supplied symbol-grep flagged the test file's
own docstrings and the Constants.swift docstring as 'rogue literals'
(self-reference bug). Filtered #file and Constants.swift from the hit
list — the test still flags any other shadow definition anywhere in the
workspace, which is the intended invariant.
2026-04-29 15:49:09 -05:00
james.maes daae27e720 feat(05-05): App-level MCPRuntimeWiring + ME-04 channel + SEC-07 observer
Closes ME-04 (Plan 04-05 deferred state). AppDelegate.applicationWill
FinishLaunching now instantiates BoundedAsyncChannel<ReplayEvent>(
capacity: 2048, policy: .dropOldest) for the orch→replay seam — the
production instance that Plan 04-05's ChannelTopologyTests CT2 only
verified as a primitive.

- App/MCP/MCPRuntimeWiring.swift: dependency-graph constructor for the
  full chain (MCPClient → ToolRegistry → MCPToolDispatcher →
  ConfirmingToolDispatcher) plus a compose() test seam. The
  ConfirmationPresenterHolder breaks the broker↔presenter cyclic init.
- App/MCP/ReplayingToolResultObserver.swift: SEC-07 observer that
  records pre-/post-sanitize bytes via two ReplayEvent.toolResultFull
  rows (one canonical, one ':raw'-suffixed toolUseId). Uses an
  injected turnIDResolver closure since orchestrator wiring lands in a
  later plan.
- AppDelegate stores the orchToReplayChannel + drain Task and tears
  them down on applicationWillTerminate.
- App/Tests/AppTests/MCPRuntimeWiringTests.swift: 3 structural tests
  (compose-returns-ConfirmingToolDispatcher; channel-capacity-+-policy;
  MCPRuntime-struct-addressable). No xcodebuild test launch — known
  Xcode 26 RunningBoard blocker per .planning/debug/.

Deviations:
- Replay / AgentCore / MCP / Bus Package.swifts had implicit transitive
  swift-log deps that SPM tolerated but Xcode framework-linker rejects
  once the App pulls them in as separate frameworks. Added explicit
  swift-log deps to all four (Rule 3, blocking).
- HudStateCoordinatorBusWiringTests qualified two HudState references
  to Jarvis.HudState — the new framework deps disturbed the App vs Bus
  symbol disambiguation (Rule 1, surgical).
- The plan's plain ReplayEvent.toolResultFull(toolUseId:rawBytes:
  sanitizedBytes:) variant doesn't exist (Phase 4 schema closed); the
  observer writes two rows with a ':raw' toolUseId suffix instead.

xcodebuild build (Debug) succeeds with the new MCP deps wired into the
App target. Test build (build-for-testing) also succeeds; runtime test
execution is the documented Xcode 26 RunningBoard blocker per
03-05-PLAN.
2026-04-25 14:57:01 -05:00
james.maesandClaude Opus 4.7 0e5f860e89 fix(04-review): ME-06 inject ReplayLog batch-window for deterministic L3 test
L3 ("buffer below chunk threshold remains unflushed") relied on the
production 50ms batch-window timer not having fired by the time the
assertion ran. Under CI load the timer could race ahead of the
assertion and L3 would observe `count == 50` instead of `0`, failing
spuriously.

Add an injectable `batchWindowMs` parameter to `ReplayLog.init` (defaults
to `Self.batchWindowMs = 50` so production behavior is unchanged). The
init reads the override into a per-instance `batchWindowMsValue` that
gates the windowed-flush Task.sleep duration.

Update L3 to construct `ReplayLog(databaseURL:, batchWindowMs: 60_000)`
so the 60-second window cannot fire within the test's lifetime. The
only flush trigger that could plausibly fire is the 64-event chunk
threshold, which 50 events doesn't reach. L3 is now deterministic.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 19:18:11 -05:00
james.maesandClaude Opus 4.7 1a432fdf9b fix(04-review): ME-05 ReplayLog.flushPending re-buffers + escalates instead of silently dropping
Pre-fix, a transaction failure inside `flushPending` cleared `pending`
before the transaction began. On rollback the entire batch was lost
with no breadcrumb beyond `batch_size` + raw error. This violated
OBS-02's durability claim — silent truncation of the authoritative
byte record on disk-pressure / locked-WAL conditions.

Three changes:
 1. On rollback, re-buffer the dropped batch at the FRONT of `pending`
    so the next flush attempt can retry. Successful flushes reset the
    failure counter.
 2. Capture the affected `turn_ids` in the error metadata so a forensic
    reader can identify which turns lost which events. Redact the
    error message (defense-in-depth alongside HI-01).
 3. After 3 consecutive failures, escalate from `error` to `critical`
    log severity so chronic data-loss conditions surface to the
    DevOverlay / log scrapers. Don't `fatalError` — OBS-02 says replay
    is best-effort and must never propagate into orchestrator turn
    semantics.

Add ReplayLogTests.test_L9_flushFailureRebuffersEvents — happy-path
control verifies the re-buffer path doesn't drop events when flushes
succeed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 19:16:11 -05:00
james.maesandClaude Opus 4.7 bd94699fd0 fix(04-review): ME-03 TokenDeltaDropOldestChannel.send loops instead of recursing
Pre-fix `send` for a non-dropTag element awaited a continuation, then
called `await send(element)` after resume. Swift does not guarantee
tail-call elimination across `await` suspension points, so each
backpressure cycle could add a stack frame. Under pathological consumer
stalls (slow drainer + many protected sends) the producer's stack would
grow proportional to retry depth.

Convert to a `while true` loop. Same shape as `BoundedAsyncChannel.send`
(no recursion). On resume the loop re-checks `finished`, re-tries the
fast paths, and re-attempts the enqueue at the same stack frame.

Add T7 stress test: 5000 protected (toolCall) sends through a capacity-1
channel with a serial drainer. Every send suspends until the previous
element is consumed. Pre-fix this would have recursed 5000 frames deep
in the actor's send invocation chain. Post-fix it stays O(1).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 19:14:11 -05:00
james.maes 3aabdd616e feat(04-03): TokenDeltaDropOldestChannel per-element drop policy + JarvisLogChannel .replay/.devoverlay (AGENT-10)
- TokenDeltaDropOldestChannel<Tag>: actor-backed AsyncSequence with per-element drop policy. Overflow drops oldest dropTag elements only; non-dropTag elements suspend the producer (backpressure). Saturated-with-non-dropTag fallback path also suspends rather than dropping a protected event.
- 6 tests: T1 below-cap, T2 oldest-drop on overflow, T3 toolCall NEVER dropped under tokenDelta firehose (AGENT-10 critical invariant), T4 load test 10000 tokenDelta + 1000 toolCall @ cap=2048 — exactly 1000 toolCalls delivered, T5 non-dropTag suspend path, T6 finish drains buffer.
- Logging ChannelTests: targeted assertions for the .replay + .devoverlay additions (LoggingTests.test_sevenChannelsAreDefined updated alongside the enum extension in commit 2614639).
2026-04-24 17:53:43 -05:00
james.maes a7fd068f70 feat(04-03): ReplayLog actor with batched writes + OrphanDetector for OBS-07 crash recovery
- ReplayLog actor: opens replay.db with WAL pragmas + DDL + meta seed; beginSession/startTurn/record/endTurn/close. Writes batched in 50ms window OR 64-event chunks. endTurn appends a turn_end event, flushes, UPDATEs ended_at + stop_reason, then PRAGMA wal_checkpoint(TRUNCATE) for the per-turn fsync. Failures logged via JarvisLogChannel.replay; never propagated.
- OrphanDetector actor: opens its own connection, increments meta.crash_count, runs the OBS-07 query (ended_at IS NULL AND turn_id NOT IN (turn_end events)), stamps each orphan with stop_reason='orphan_recovered' + recovery_marker='detected_at_launch:crash_count=N'.
- 12 new tests: L1-L8 (session/turn/buffering/window/chunk/large-blob/nothing-masked) + O1-O4 (orphan vs non-orphan, race window, crash_count increment, end-to-end SIGKILL parity).
- Adds JarvisLogChannel.replay + .devoverlay (additive — Logging test updated to assert 7 cases). The full Task 3 channel-tests file lands next commit.
2026-04-24 17:53:43 -05:00
james.maes 37b7053903 feat(04-03): scaffold Replay package with hand-rolled SQLite wrapper + schema DDL (OBS-02)
- Package.swift: new SPM library 'Replay' depending on AgentCore + JarvisLogging + Config (Swift 6).
- SQLiteConnection: import SQLite3 (libsqlite3.dylib) — no SQLite.swift dependency (Phase 7 needs sqlite-vec via load_extension). FULLMUTEX, parameterized bindings via SQLiteValue enum, SQLITE_TRANSIENT for text/blob.
- Schema: research §8 verbatim — meta/sessions/turns/events + events_by_turn + turns_by_session indices. Pragmas: WAL, synchronous=NORMAL, busy_timeout=3000, foreign_keys=ON, temp_store=MEMORY. seedMeta with INSERT OR IGNORE.
- ReplayEvent: ten cases with .encoded() returning (kind, payloadBytes); base64 envelope for tool_call_requested + tool_result_full so binary tool results round-trip through JSON.
- ReplayPaths: defaultDatabaseURL = ~/Library/Application Support/Jarvis/replay.db.
- 7 schema tests covering DDL, FK retry_of -> turns(turn_id), BLOB NOT NULL, indices, pragmas, seed, idempotent re-open.
2026-04-24 17:53:43 -05:00