experiment · evergreen
The reranker that hurt
Recall@40 left 17 points on the table; bge-reranker-v2-m3 spent them and added seconds.
The textbook says: retrieve 40, rerank with a cross-encoder, hand over 8. On this corpus the textbook was wrong, and the numbers said so.
| Setting | Recall@8 | MRR | TR prose | p50 |
|---|---|---|---|---|
| auto + dense (no rerank) | 0.786 | 0.690 | 0.684 | 34 ms |
| hybrid, k=40 candidates | 0.952 | — | 0.895 | 39 ms |
| hybrid + bge-reranker-v2-m3 | 0.762 | 0.508 | 0.579 | 4389 ms |
| auto + bge-reranker-v2-m3 | 0.762 | 0.514 | 0.579 | 2050 ms |
- The headroom is real. Recall@40 is 0.95 — the right chunk is almost always in the candidate pool. A good reranker has 17 points to win.
- This reranker spent them. MRR dropped from 0.690 to 0.514 and the Turkish slice fell furthest. It also cost two to four seconds per query.
- As a "no answer" gate it was worse. It caught 12/12 negatives — and flagged 12/42 real answers as junk. A 29% false alarm rate.
A reranker is a hypothesis about your corpus. Test it like one.