Search quality benchmarks

Generated by packages/eval-harness (pnpm eval), against a scratch Postgres seeded with src/fixtures/seed-catalog.ts and embedded with the real local model (all-MiniLM-L6-v2, not a test double). Reproducible: DATABASE_URL=... pnpm eval from packages/eval-harness gives the same numbers on the same fixtures.

Last run: 2026-08-12, after the pgvector migration - numbers are numerically identical to the pre-pgvector JS-cosine implementation, since pgvector's <=> operator computes the same cosine distance, just in SQL instead of a JS loop. That equivalence is itself a useful check that the migration didn't change behavior.

Results

Query set: 15 queries, packages/eval-harness/src/fixtures/golden-queries.ts, across five categories (exact-name, paraphrase, filtered, no-result, adversarial). Catalog: 18 resources, packages/eval-harness/src/fixtures/seed-catalog.ts. Judgments: graded 0-3, packages/eval-harness/src/fixtures/judgments.ts - see that file and the harness's own README for the full rubric and the judge-methodology limitation (single-pass, same model that built the harness, no independent human audit yet - stated there directly, not smoothed over here).

What the numbers say

BM25-only scores meaningfully lower (0.442 vs 0.981 nDCG@10) than dense/fused. The query set is deliberately paraphrase-heavy - is it going to rain tomorrow, how much is bitcoin worth right now - and those share zero vocabulary with the resources that answer them (WeatherCo/"precipitation and temperature predictions", CoinWatch/"cryptocurrency price feed"). BM25 structurally cannot match on vocabulary that isn't there. This is the query set doing its job, not a weakness in the BM25 implementation - it's tested and proven correct in isolation (it correctly ranks a rare-term match above a common-term match, the thing naive counting or bare ts_rank get backwards).

fused+settlement and dense-only currently score identically on this fixture set. Stated directly because the spec this was built against explicitly asked for it: "fused should outperform either arm alone; if it doesn't, that's a real finding to report, not a bug to hide." At this catalog size (18 resources) and with this query mix, fusion isn't yet demonstrably beating the dense arm alone. Two candidate explanations, both checkable once the query/catalog set grows:

1. Most of the 15 queries are paraphrase queries where BM25 contributes little rank signal, so RRF fusion has little to add on top of dense alone for this particular mix.
2. RRF_K = 60 is the literature default, untuned for a catalog this small - a smaller k weights top ranks more heavily, which might matter more at 18 resources than at 14,000.

Neither has been root-caused yet; that's exactly what a larger, more adversarial query set (and eventually a judge model independent of this session) would clarify, not something to paper over by hand-picking queries that make fusion look better this run.

The adversarial query (weather forecast) is the one to watch as the corpus grows. It pits a genuine listing against a keyword-stuffed near-duplicate. Both currently score correctly ordered in this run - the genuine listing outranks the stuffed one - but this is exactly the kind of case where a fusion regression would first show up, which is why it's a dedicated regression case rather than something the harness would only catch by accident.

CI gate

.github/workflows/eval-harness.yml runs pnpm check-regression on every change touching packages/discovery/src/search/**, comparing the fused+settlement configuration against packages/eval-harness/baseline.json with a 0.02 absolute threshold per metric. A ranking change that's meant to move these numbers updates baseline.json deliberately, as its own reviewed change - the check exists to catch accidental regressions, not to freeze the numbers forever.

Reproducing this

# from packages/eval-harness
docker run -d --name eval-pg -e POSTGRES_PASSWORD=eval -e POSTGRES_DB=rialto_eval \
  -p 5432:5432 pgvector/pgvector:pg17
DATABASE_URL=postgresql://postgres:eval@localhost:5432/rialto_eval pnpm eval

First run downloads the embedding model (~90MB, cached afterward). Seeding + embedding 18 resources + scoring 15 queries against 3 configurations takes well under a minute on a warm model cache.