Public benchmark

Code-retrieval quality and cost

Profile sota-challenge: 600 held-out queries across 6 public CoIR tasks, repeated 3 times. No private corpus or local path is included.

600held-out queries

6public tasks

3languages

3repetitions

2048query char limit

Aggregate results

Mode	nDCG@10	MRR@10	P@5	R@20	Warm p95	Index size
`hash`	0.5900	0.5582	0.1287	0.7000	688.94 ms	45.43 MiB
`hybrid`	0.5896	0.5578	0.1287	0.7000	722.46 ms	45.45 MiB
`blended`	0.5955	0.5652	0.1300	0.7000	778.19 ms	65.54 MiB
`neural`	0.5963	0.5661	0.1303	0.7000	798.38 ms	65.55 MiB

Population variance, phase timings, peak RSS, binary identity, dataset revisions, and checksums are retained in the raw JSON.

Task	Mode	nDCG@10	MRR@10	R@20
`stackoverflow-qa`	`hash`	0.5829	0.5231	0.7700
`stackoverflow-qa`	`hybrid`	0.5810	0.5206	0.7700
`stackoverflow-qa`	`blended`	0.5988	0.5458	0.7700
`stackoverflow-qa`	`neural`	0.5988	0.5458	0.7700
`apps`	`hash`	0.0033	0.0014	0.0200
`apps`	`hybrid`	0.0033	0.0014	0.0200
`apps`	`blended`	0.0033	0.0014	0.0200
`apps`	`neural`	0.0033	0.0014	0.0200
`codefeedback-mt`	`hash`	0.5657	0.5112	0.7600
`codefeedback-mt`	`hybrid`	0.5657	0.5112	0.7600
`codefeedback-mt`	`blended`	0.5788	0.5243	0.7600
`codefeedback-mt`	`neural`	0.5788	0.5243	0.7600
`synthetic-text2sql`	`hash`	0.8662	0.8366	0.9800
`synthetic-text2sql`	`hybrid`	0.8662	0.8366	0.9800
`synthetic-text2sql`	`blended`	0.8658	0.8330	0.9800
`synthetic-text2sql`	`neural`	0.8683	0.8363	0.9800
`CodeSearchNet-python`	`hash`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`hybrid`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`blended`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`neural`	0.9400	0.9400	0.9400
`CodeSearchNet-java`	`hash`	0.5816	0.5370	0.7300
`CodeSearchNet-java`	`hybrid`	0.5816	0.5370	0.7300
`CodeSearchNet-java`	`blended`	0.5865	0.5465	0.7300
`CodeSearchNet-java`	`neural`	0.5885	0.5488	0.7300

Every retained task remains visible so aggregate improvements cannot hide regressions.

Matrix covers held-out natural-language and code-to-code retrieval. Exact-search tools require a separate exact-query workload.