Preindexed notes and memory retrieval
MemoryQuest sessions become 3,878 local Markdown notes across 50 users. 535 natural questions test retrieval of 2,029 required memories.
Aggregate results
| Mode | Recall@10 | Recall@20 | Exact@20 | nDCG@10 | Warm p95 |
|---|---|---|---|---|---|
lexical | 52.9% | 70.2% | 35.1% | 0.4145 | 35.81 ms |
blended | 54.5% | 74.9% | 44.9% | 0.4274 | 86.05 ms |
forced neural | 52.1% | 71.0% | 35.9% | 0.4129 | 33.14 ms |
blended is default daemon-backed semantic + lexical routing across CLI, MCP, Web, and TUI. When an implicit natural-language question initially returns overwhelmingly note-like files, default search runs two generic local memory probes concurrently and fuses file ranks. forced neural disables this adaptive path and runs neural retrieval for every query. lexical disables vectors.
Method
MemoryQuest is a Microsoft Research and University of Washington benchmark for implicit, context-dependent personal-memory retrieval. Exporter creates one Markdown note per session and indexes all notes once. Each historical question searches only its user's sessions dated at or before query date, matching official temporal protocol.
Only session date and raw conversation turns enter index. Topics, domains, required-memory flags, demographics, timelines, reasoning, and references stay outside indexed tree. All 2,029 references resolve to labeled sessions.
Dataset uses CC BY 4.0; generated corpus is not redistributed.
Published reference point
MemoryQuest paper reports 58.0% recall for query-only fact retrieval, 72.3% recall / 32.6% exact for GPT-4o PGR-TOT, and 74.8% / 34.8% for iterative DeepSeek-V3.2 PGR-TOT.
These are not direct leaderboard comparisons. Published systems retrieve LLM-extracted atomic facts, PGR gathers about 35 facts through multiple generated probes and 4.79 average query-time LLM calls, and GPT-5.2 judges whether references appear. ivygrep result uses top-20 raw session files, deterministic session labels, fixed local probes with no query-time LLM, and no LLM judge.
Where recall is still lost
Depth is largest measured ranking gap: default recall rises from 54.5% at 10 to 74.9% at 20. MemoryQuest intentionally makes required memories semantically distant from query and requires three or four sessions per question.
Forcing neural retrieval on every query reaches 71.0%, 3.9 percentage points below default. Default's gain comes from prospective rank fusion, not simply forcing existing vector pass. Remaining misses need better probe selection or structured fact memory.
Same-binary single-query control reaches 70.8% recall@20 and 36.3% complete recall@20 at 30.03 ms p95. Automatic local probes add 4.1 recall points and 8.6 complete-recall points.
What this establishes
ivygrep can preindex note-like text once, then retrieve related memories from natural-language questions without hosted inference. Measurement covers session retrieval, not final answer correctness.
Limits: conversations are synthetic, each released question needs three or four sessions, run uses one Linux host and one quality pass, and latency varies by hardware and load.
Reproduce
uv run scripts/export_memoryquest.py --output /tmp/ivygrep-memoryquest
uv run scripts/eval_code_retrieval.py --dataset /tmp/ivygrep-memoryquest --binary target/release/ig --mode lexical --output /tmp/memoryquest-lexical.json
uv run scripts/eval_code_retrieval.py --dataset /tmp/ivygrep-memoryquest --binary target/release/ig --mode blended --output /tmp/memoryquest-blended.json
uv run scripts/eval_code_retrieval.py --dataset /tmp/ivygrep-memoryquest --binary target/release/ig --mode neural --output /tmp/memoryquest-neural.json
uv run scripts/eval_code_retrieval.py --dataset /tmp/ivygrep-memoryquest --binary target/release/ig --mode blended --disable-memory-expansion --output /tmp/memoryquest-blended-control.json
uv run scripts/render_memory_benchmark.py --dataset /tmp/ivygrep-memoryquest --result /tmp/memoryquest-lexical.json --result /tmp/memoryquest-blended.json --result /tmp/memoryquest-neural.json --control-result /tmp/memoryquest-blended-control.json --source-commit 3891ae4db35b6e52bd18f353a7b83c82f6394966 --output-json docs/benchmarks/public-memory-retrieval-results.json --output-html docs/benchmarks/public-memory-retrieval.html