Public benchmark

Held-out challenge retrieval

Measured with v1.1.9 (2026-07-09).

Profile sota-challenge: 600 public queries (executed model unverified) across 6 public CoIR tasks, repeated 3 times. No private corpus or local path is included.

Corpus sampling: 6 of 6 corpora sampled to 5,000 documents (qrel documents retained, remainder dropped deterministically). Full CoIR corpora: stackoverflow-qa 19,931 documents (5,000 indexed), apps 8,765 documents (5,000 indexed), codefeedback-mt 66,383 documents (5,000 indexed), synthetic-text2sql 105,851 documents (5,000 indexed), CodeSearchNet-python 280,310 documents (5,000 indexed), CodeSearchNet-java 180,866 documents (5,000 indexed). Scores on sampled corpora are not comparable to full-corpus CoIR leaderboard numbers.

600public queries (executed model unverified)

6public tasks

3languages

3repetitions

6/6corpora sampled

2048query char limit

Aggregate results

Mode	nDCG@10	MRR@10	P@5	R@20	Warm p95	Index size
`hash`	0.5900	0.5582	0.1287	0.7000	688.94 ms	45.43 MiB
`hybrid`	0.5896	0.5578	0.1287	0.7000	722.46 ms	45.45 MiB
`blended`	0.5955	0.5652	0.1300	0.7000	778.19 ms	65.54 MiB
`neural`	0.5963	0.5661	0.1303	0.7000	798.38 ms	65.55 MiB

Population variance, phase timings, peak RSS, binary identity, dataset revisions, and checksums are retained in the raw JSON.

Per-task quality

Task	Mode	nDCG@10	MRR@10	R@20
`stackoverflow-qa`	`hash`	0.5829	0.5231	0.7700
`stackoverflow-qa`	`hybrid`	0.5810	0.5206	0.7700
`stackoverflow-qa`	`blended`	0.5988	0.5458	0.7700
`stackoverflow-qa`	`neural`	0.5988	0.5458	0.7700
`apps`	`hash`	0.0033	0.0014	0.0200
`apps`	`hybrid`	0.0033	0.0014	0.0200
`apps`	`blended`	0.0033	0.0014	0.0200
`apps`	`neural`	0.0033	0.0014	0.0200
`codefeedback-mt`	`hash`	0.5657	0.5112	0.7600
`codefeedback-mt`	`hybrid`	0.5657	0.5112	0.7600
`codefeedback-mt`	`blended`	0.5788	0.5243	0.7600
`codefeedback-mt`	`neural`	0.5788	0.5243	0.7600
`synthetic-text2sql`	`hash`	0.8662	0.8366	0.9800
`synthetic-text2sql`	`hybrid`	0.8662	0.8366	0.9800
`synthetic-text2sql`	`blended`	0.8658	0.8330	0.9800
`synthetic-text2sql`	`neural`	0.8683	0.8363	0.9800
`CodeSearchNet-python`	`hash`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`hybrid`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`blended`	0.9400	0.9400	0.9400
`CodeSearchNet-python`	`neural`	0.9400	0.9400	0.9400
`CodeSearchNet-java`	`hash`	0.5816	0.5370	0.7300
`CodeSearchNet-java`	`hybrid`	0.5816	0.5370	0.7300
`CodeSearchNet-java`	`blended`	0.5865	0.5465	0.7300
`CodeSearchNet-java`	`neural`	0.5885	0.5488	0.7300

Every retained task remains visible so aggregate improvements cannot hide regressions.

Scope

Matrix covers natural-language and code-to-code retrieval. Exact-search tools require a separate exact-query workload.

This artifact does not record verified checkout-reference fit-ID disjointness. Executed-model fit disjointness is unverified: no model-checksum attestation binds the binary to the reference. Matching model IDs do not identify model bytes. This diagnostic does not replace the public-core release gate.

Artifacts use pinned public datasets and retain source revisions and checksums in raw JSON. Deterministic samples: stackoverflow-qa (100 queries, 5000 documents), apps (100 queries, 5000 documents), codefeedback-mt (100 queries, 5000 documents), synthetic-text2sql (100 queries, 5000 documents), CodeSearchNet-python (100 queries, 5000 documents), CodeSearchNet-java (100 queries, 5000 documents). Dataset cards do not declare licenses for: stackoverflow-qa, apps, codefeedback-mt, synthetic-text2sql. Treat downloaded corpora as evaluation inputs; do not redistribute them without checking upstream terms.