fix: store scripted assistant content in eval + session-affinity recall boost (#431)
## Summary Two fixes for eval recall quality that improve the inflated CM-1 score from 3.69 → 4.38. ## Changes ### 1. Store scripted assistant turns in eval replay (`harness.ts`) The eval replays scripted conversation turns through the gateway, but the gateway stores its **own API response** (which differs from the scripted content) as the assistant temporal message. The scenario's actual content (e.g., "staleness check vs flock", "5000ms timeout error") was never stored in temporal and therefore never searchable via recall. Fix: After each scripted assistant turn is added to history, store it directly in temporal with the replay session ID. This makes the scenario's ground-truth content available for recall search. **Impact:** m4 (exact error message) went from 1.2 → 4.0. ### 2. Session-affinity RRF boost (`recall.ts`) When `scope=all` and `sessionID` is known, add extra RRF lists for same-session temporal and distillation results. This gives current-session content a ranking boost over cross-session LTM entries that may match keywords but lack session-specific context. ## Eval Results (CM-1, 400K inflation) | Metric | Before | After | |---|---|---| | Average score | 3.69 | **4.38** | | Questions >= 4.0 | 7/15 | **13/15** | | Questions = 5.0 | 5/15 | **9/15** | Remaining failures: m3 (2.2, LTM entry outranks session content), h4 (1.7, initial wrong hypothesis not found). ## Tests - 1752 pass, 0 fail - Typecheck clean across all 4 packages
B
Burak Yigit Kaya committed
48ffead2b470c2de8e120b0ea3b5e331d8581b0c
Parent: 72edf63
Committed by GitHub <noreply@github.com>
on 5/20/2026, 9:05:12 PM