Tuesday, July 7, 2026, 17:16:19 +0530 — the timestamp on 84e4d39, the commit where I finally built an eval for retrieval in fabri, the self-improving agent engine I build products on. fabri stores “guidelines” — short lessons an agent retrieves before it acts, mined from its own past runs. Four commits later, at 20:46:40 the same evening, I’d found and fixed two more bugs I didn’t know existed.
What I’d built on vibes
fabri’s memory loop — retrieve → run → mine → dedup → promote → retrieve — had been running unmeasured. By v0.9.x it had a config knob, memory.retrieval_strategy, that could be dense (embedding/vector similarity search), sparse (BM25 keyword search via SQLite FTS5), hybrid (RRF — Reciprocal Rank Fusion, merge two ranked lists by summing 1/(k+rank) per result — of both), or hybrid+mmr (hybrid, then Maximal Marginal Relevance re-ranks the fused list to spread out near-duplicate guidelines, trading recall for diversity). I’d shipped hybrid as an option, had opinions about which was better, and never once measured any of them against labeled queries. Every tuning decision up to that point was vibes wearing the costume of engineering judgment.
So I built the thing that should have existed from day one: an offline eval gate. tests/fixtures/retrieval_eval.json — 40 guidelines, 24 queries, top_k=5. 14 of those 24 queries have exactly one gold-labeled relevant guideline, 10 have two — recall@k is a fraction retrieved, not a binary hit. Not a mock — it drives the real _retrieve_inner code path a live agent calls. The runner (python -m fabri.benchmarks.retrieval_eval) computes recall@k and MRR — mean reciprocal rank, the mean over queries of 1/rank of the first relevant hit. A CI gate fails if a measured number drops below baseline − 0.05 — not exact equality, which was, in the docstring’s own words, “the shape of the July 2026 CI flakiness.”
I ran it for the first time expecting a baseline number to write down. Instead it caught a live bug on the first run.
Finding #1: BM25 had been doing nothing
_fts5_query in src/fabri/memory/embedded_store.py space-joined query tokens before handing them to SQLite’s FTS5 MATCH, which reads whitespace between terms as implicit AND. So a guideline had to contain every single word of a query to match at all — almost never true for natural language. Sparse and hybrid retrieval had been silently collapsing to dense — byte-identical recall, every time, since that code was written.
The fix: split on [\W_]+ instead of [^\w\s] (so read_file now contributes read and file as separate tokens), then join with " OR " instead of a space — “The OR is load-bearing,” per the comment I left in that diff. hybrid recall@5 went 0.79 → 0.94; sparse recall@5 went 0.79 → 0.88. I added test_hybrid_bm25_is_alive so this exact failure mode — hybrid silently degrading to a no-op — trips CI if it recurs. That commit also cleared a pre-existing red test, test_tool_name_with_underscore — same root cause, two symptoms.
Flipping the default on half the evidence
Right after, in a separate commit (17b48cc), I flipped the default from dense to hybrid on the strength of the recall@5 win I’d just measured. That was a mistake: I’d only looked at one number.
Running the same eval head-to-head after the flip told me so directly. BENCHMARKS.md: “Running the M4 eval head-to-head after the dense->hybrid flip showed hybrid only won at recall@5 while LOSING at recall@1/@3.” Hybrid’s recall@1 was 0.125 — same as dense’s — and recall@3 was worse than dense (0.604 vs 0.688). I’d shipped a regression and called it a win because I’d checked the wrong slice of the metric.
Finding #2a: k=60 doesn’t belong here
The first root cause was RRF, which fuses dense and sparse pools by scoring each candidate Σ 1/(k+rank). The classic default is k=60, built for web-scale search over pools with thousands of candidates. fabri fuses two short pools — roughly 2×top_k each, about ten candidates per pool at top_k=5 — and at k=60 that’s flat enough that “appears in both lists” outranks “is the single best match in either.” I’d never checked whether 60 fit this pool size. Agreement was beating relevance.
I added memory.rrf_k, defaulting to 20, with the comment “Lower = sharper rank discrimination; raise toward 60 only for very large stores.” hybrid recall@3 went 0.60 → 0.90. recall@5 was unchanged — the damage had been entirely at the top of the list, exactly where a fusion-constant mismatch would hide.
Finding #2b: the slot that was stealing rank 1
The second root cause ran deeper. fabri reserves guaranteed slots for success_pattern entries — “what worked last time” guidelines — so a learned strategy surfaces without a semantic match. Those slots were being injected at ranks 1–2, ahead of whatever the actual most-relevant guideline was. Only about 5 of the 24 eval queries actually wanted a success pattern — for the other 19, an irrelevant “here’s what worked before” entry sat at rank 1 instead.
Fixing it meant back-loading those reserved slots to the tail, with the reserved count computed as min(top_k // 2, top_k - 1) — a formula chosen specifically so rank 1 can never be a guaranteed slot, in code that runs, in the comment’s own words, “regardless of which retrieval strategy is active.” hybrid recall@1 went 0.13 → 0.58; MRR went 0.45 → 0.84. The detail I didn’t expect: dense recall@1 moved by the exact same amount, 0.13 → 0.58 — same shared code path, so this wasn’t a hybrid-only problem. Sparse recall@1 rose too, 0.17 → 0.50, but not identically: it ranks by BM25 score instead of a dense vector, so the same fix freed rank 1 for a different, smaller set of queries. Either way, the whole memory system had been capping its own best-case rank-1 accuracy, hiding in a slot-placement decision nobody had measured.
Both fixes landed in the same commit, ad442e4, alongside a trace-ordering bug the observability event (4101b5a) had exposed: threading session_id into retrieval made the retrieval trace event log before the run’s own start event, breaking an invariant a smoke test relied on. One-line fix: move log_event earlier.
What actually shipped
Final numbers, same 40-guideline/24-query fixture, top_k=5 — measured on the same fixture that found the RRF-k and slot-order wins above, which is circular and I know it. A held-out slice that never entered tuning is the honest fix, still outstanding.
| strategy | recall@1 | recall@3 | recall@5 | MRR |
|---|---|---|---|---|
| dense | 0.583 | 0.688 | 0.792 | 0.790 |
| sparse (BM25) | 0.500 | 0.792 | 0.875 | 0.772 |
| hybrid (shipped default) | 0.583 | 0.896 | 0.938 | 0.844 |
| hybrid+mmr | 0.583 | 0.625 | 0.729 | 0.804 |
hybrid+mmr is in that table because it’s a live option, not because it wins: it loses to hybrid everywhere and even to plain dense on recall@3 (0.625 vs 0.688). MMR trades recall for diversity, and this fixture doesn’t reward that trade — it stays opt-in, not the default.
CI gate floors got re-baselined at measured − 0.05 across the board (HYBRID_RECALL5_FLOOR = 0.89, HYBRID_RECALL3_FLOOR = 0.84, HYBRID_MRR_FLOOR = 0.79). Full suite: 914 passed.
docs/retrieval-tuning.md shipped in that same commit; its first line is the honest version of this post: “The defaults are eval-backed and good for most stores — if you’re just starting out, change nothing.” The rule it states in a blockquote is the one I broke and am now enforcing on myself: never flip a retrieval default blind.
I don’t think the eval gate is done. Reranking, query expansion, and an embedding upgrade are still on the roadmap, each explicitly gated on this same eval before it ships, though there’s no guarantee the next finding is this clean. Three of the last four “improvements” I’d shipped by feel — hybrid as a config option, the dense→hybrid default flip, the guaranteed success-pattern slots — were either doing nothing, actively regressing, or capping the ceiling on every other strategy at once: recall@1 alone moved from 0.13 to 0.58 once slot placement stopped hiding it.