Public benchmark

Challenge retrieval

Measured with v1.2.16 at commit e9119a4f683f (2026-09-21).

Profile sota-challenge: 600 verified fit-disjoint diagnostic queries across 6 public CoIR tasks, repeated 3 times. No private corpus or local path is included.

Corpus sampling: 6 of 6 corpora sampled to 5,000 documents (qrel documents retained, remainder dropped deterministically). Full CoIR corpora: stackoverflow-qa 19,931 documents (5,000 indexed), apps 8,765 documents (5,000 indexed), codefeedback-mt 66,383 documents (5,000 indexed), synthetic-text2sql 105,851 documents (5,000 indexed), CodeSearchNet-python 280,310 documents (5,000 indexed), CodeSearchNet-java 180,866 documents (5,000 indexed). Scores on sampled corpora are not comparable to full-corpus CoIR leaderboard numbers.

600verified fit-disjoint diagnostic queries

6public tasks

3languages

3repetitions

6/6corpora sampled

2048query char limit

Aggregate results

Mode	nDCG@10	MRR@10	P@5	R@20	Warm p95	Index size
`hash`	0.6310	0.6104	0.1352	0.7067	1745.60 ms	52.26 MiB
`hybrid`	0.6231	0.6027	0.1313	0.7117	2191.91 ms	52.26 MiB
`blended`	0.6385	0.6206	0.1327	0.7183	2091.78 ms	76.17 MiB
`neural`	0.6503	0.6320	0.1377	0.7150	1777.92 ms	76.16 MiB

Population variance, phase timings, peak RSS, binary identity, dataset revisions, and checksums are retained in the raw JSON.

Per-task quality

Task	Mode	nDCG@10	MRR@10	R@20
`stackoverflow-qa`	`hash`	0.6730	0.6435	0.7700
`stackoverflow-qa`	`hybrid`	0.6497	0.6199	0.7900
`stackoverflow-qa`	`blended`	0.6631	0.6352	0.7900
`stackoverflow-qa`	`neural`	0.6851	0.6592	0.7700
`apps`	`hash`	0.0238	0.0190	0.0400
`apps`	`hybrid`	0.0254	0.0212	0.0400
`apps`	`blended`	0.0400	0.0367	0.0500
`apps`	`neural`	0.0393	0.0358	0.0500
`codefeedback-mt`	`hash`	0.6079	0.5650	0.7600
`codefeedback-mt`	`hybrid`	0.5780	0.5394	0.7200
`codefeedback-mt`	`blended`	0.5904	0.5541	0.7200
`codefeedback-mt`	`neural`	0.6314	0.5947	0.7600
`synthetic-text2sql`	`hash`	0.8704	0.8410	0.9900
`synthetic-text2sql`	`hybrid`	0.8699	0.8405	0.9900
`synthetic-text2sql`	`blended`	0.8777	0.8504	0.9900
`synthetic-text2sql`	`neural`	0.8826	0.8570	0.9900
`CodeSearchNet-python`	`hash`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`hybrid`	0.9336	0.9317	0.9500
`CodeSearchNet-python`	`blended`	0.9336	0.9317	0.9500
`CodeSearchNet-python`	`neural`	0.9400	0.9400	0.9400
`CodeSearchNet-java`	`hash`	0.6713	0.6538	0.7400
`CodeSearchNet-java`	`hybrid`	0.6819	0.6634	0.7800
`CodeSearchNet-java`	`blended`	0.7265	0.7158	0.8100
`CodeSearchNet-java`	`neural`	0.7235	0.7051	0.7800

Every retained task remains visible so aggregate improvements cannot hide regressions.

Scope

Matrix covers natural-language and code-to-code retrieval. Exact-search tools require a separate exact-query workload.

Runtime-reported model bytes match the checksum-bound fit ledger, with zero repository-qualified fit-ID overlap. This does not prove semantic independence, absence of development-time exposure, or reranker invocation on every query. The diagnostic does not replace the public-core release gate.

Artifacts use pinned public datasets and retain source revisions and checksums in raw JSON. Deterministic samples: stackoverflow-qa (100 queries, 5000 documents), apps (100 queries, 5000 documents), codefeedback-mt (100 queries, 5000 documents), synthetic-text2sql (100 queries, 5000 documents), CodeSearchNet-python (100 queries, 5000 documents), CodeSearchNet-java (100 queries, 5000 documents). Dataset cards do not declare licenses for: stackoverflow-qa, apps, codefeedback-mt, synthetic-text2sql. Treat downloaded corpora as evaluation inputs; do not redistribute them without checking upstream terms. This refresh was measured with a native glibc build on a shared 16-CPU Linux host, not a shipped musl archive or the runner used for the previous report; shared-host load was not isolated, so latency and resource costs are descriptive, not a controlled before/after comparison. The late dense-fusion calibration used half of the stackoverflow-qa, apps, and codefeedback-mt samples. Zero overlap with the learned reranker's fit ledger does not make those tasks unseen evaluations of the complete ranking stack. CodeSearchNet-python, CodeSearchNet-java, and synthetic-text2sql were not used for that calibration.