Public benchmark

Code-retrieval quality and cost

Measured with v1.2.16 at commit 2f4208d79493 (2026-09-20).

Profile public-core: 1000 regression queries across 4 public CoIR tasks, repeated 3 times. No private corpus or local path is included.

Corpus sampling: 1 of 4 corpora sampled to 5,000 documents (qrel documents retained, remainder dropped deterministically). Full CoIR corpora: codefeedback-st 156,526 documents (5,000 indexed). Indexed in full: codetrans-dl (816), codetrans-contest (1,008), cosqa (20,604). Scores on sampled corpora are not comparable to full-corpus CoIR leaderboard numbers.

1000regression queries

4public tasks

50languages

3repetitions

1/4corpora sampled

Aggregate results

Mode	nDCG@10	MRR@10	P@5	R@20	Warm p95	Index size
`lexical`	0.3421	0.2905	0.0760	0.5760	808.44 ms	38.17 MiB
`hash`	0.3411	0.2891	0.0757	0.5767	509.46 ms	71.02 MiB
`hybrid`	0.3415	0.2894	0.0760	0.5770	507.54 ms	71.02 MiB
`blended`	0.3533	0.2992	0.0786	0.5920	518.10 ms	107.35 MiB
`neural`	0.3537	0.2996	0.0786	0.5930	513.99 ms	107.35 MiB

Population variance, phase timings, peak RSS, binary identity, dataset revisions, and checksums are retained in the raw JSON.

Change from frozen baseline

neural improves nDCG@10 by +52.22% and MRR@10 by +55.70% over hash at commit 49b1571de77a. The raw JSON retains every task and run.

Per-task quality

Task	Mode	nDCG@10	MRR@10	R@20
`codetrans-dl`	`lexical`	0.3182	0.2061	0.7667
`codetrans-dl`	`hash`	0.3203	0.2073	0.7722
`codetrans-dl`	`hybrid`	0.3203	0.2073	0.7722
`codetrans-dl`	`blended`	0.3179	0.1997	0.7944
`codetrans-dl`	`neural`	0.3169	0.1984	0.7944
`codetrans-contest`	`lexical`	0.6024	0.5623	0.7421
`codetrans-contest`	`hash`	0.5959	0.5544	0.7451
`codetrans-contest`	`hybrid`	0.5978	0.5559	0.7466
`codetrans-contest`	`blended`	0.6217	0.5814	0.7647
`codetrans-contest`	`neural`	0.6212	0.5808	0.7647
`cosqa`	`lexical`	0.1459	0.1074	0.3680
`cosqa`	`hash`	0.1459	0.1074	0.3660
`cosqa`	`hybrid`	0.1459	0.1074	0.3660
`cosqa`	`blended`	0.1528	0.1099	0.3780
`cosqa`	`neural`	0.1541	0.1113	0.3800
`codefeedback-st`	`lexical`	0.7950	0.7616	0.9091
`codefeedback-st`	`hash`	0.7962	0.7631	0.9091
`codefeedback-st`	`hybrid`	0.7957	0.7626	0.9091
`codefeedback-st`	`blended`	0.8316	0.8063	0.9192
`codefeedback-st`	`neural`	0.8318	0.8066	0.9192

Every retained task remains visible so aggregate improvements cannot hide regressions.

Scope

Matrix covers natural-language and code-to-code retrieval. Exact-search tools require a separate exact-query workload.

Public-core is regression evidence, not an unseen-query generalization set: it overlaps the checkout-reference reranker's fit data. Runtime-reported model bytes match the checksum-bound fit ledger. Its query set and acceptance gates are unchanged.

Artifacts use pinned public datasets and retain source revisions and checksums in raw JSON. Deterministic samples: codefeedback-st (99 queries, 5000 documents). Dataset cards do not declare licenses for: codetrans-dl, codetrans-contest, cosqa, codefeedback-st. Treat downloaded corpora as evaluation inputs; do not redistribute them without checking upstream terms.