Every RAG tutorial ends the same way. Retrieve your candidates, then run them through a cross-encoder to rerank, because bi-encoder similarity is a coarse signal and a cross-encoder reads the query and the document together. It is good advice. I followed it, implemented it, benchmarked it against the alternative, and then removed it from the pipeline.
This is the story of that decision, and of the evaluation harness that made it possible to make. The project is AzureMentor, a grounded question-answering system over the official Microsoft Azure documentation.
The system
AzureMentor answers questions about Azure using only what it can retrieve from the docs. The corpus is 31,700 chunks of official documentation. Every answer carries inline citations back to the source pages, and there is a hard guarantee against ungrounded output: if retrieval comes back empty, the system skips the LLM call entirely rather than letting the model improvise. An honest refusal beats a confident invention, particularly in a domain where a plausible-sounding wrong answer costs somebody an afternoon.
Retrieval is hybrid. BM25 runs over SQLite FTS5, dense vectors come from bge-small stored in Qdrant, and the two result lists are combined with reciprocal rank fusion. That combination is the single highest-leverage decision in the project.
Building the evaluation set first
The thing I would tell anyone starting a retrieval project: build the evaluation harness before you start tuning. Not after, when you are trying to justify choices you have already made. Before.
I built a ground-truth set of 450 questions with known correct source chunks, and measured two things. Hit rate at 5 asks whether the right chunk appears anywhere in the top five results. Mean reciprocal rank asks how close to the top it lands. Hit rate tells you whether the answer is reachable at all; MRR tells you whether the model has to wade through noise to find it.
Having those numbers before making changes turns every subsequent decision from an argument into a measurement. That is the whole point.
Hybrid retrieval earns its keep
The fused pipeline reaches a 0.933 hit rate at 5 and 0.835 MRR. That MRR figure is roughly seven percentage points better than either BM25 or dense retrieval achieves alone.
The reason is that the two methods fail differently. BM25 is lexical, so it is excellent when a question contains the exact name of a service or a specific error code, and useless when the user describes a concept in their own words. Dense retrieval is the mirror image: it handles paraphrase well and gets vague about precise identifiers. Azure documentation is full of near-identical service names, so a purely semantic index confuses things that share vocabulary, while a purely lexical index misses anyone who does not already know the right term to search for.
Reciprocal rank fusion is a good way to combine them because it depends only on the rank position within each list, not on the scores. BM25 scores and cosine similarities are not on comparable scales and normalising between them requires tuning that will not survive a corpus change. Ignoring the magnitudes sidesteps that problem entirely.
The reranker
With hybrid retrieval working, the obvious next move was a cross-encoder reranker over the fused candidates. I implemented it, along with query expansion, and swept six retrieval configurations through the same 450-question harness.
Reranking improved hit rate by 0.7 percentage points. The standard error on that measurement was around 1.2 points. The improvement was smaller than the noise in my own evaluation, which means I could not distinguish it from nothing at all. And it cost 5.7 times the retrieval latency.
I want to be precise about what that result does and does not say. It does not say cross-encoder reranking is a bad technique. It says that on this corpus, with this retrieval stack, at this hit rate, there was very little headroom left for the reranker to recover. Hybrid retrieval was already putting the right chunk in the top five 93.3% of the time. A reranker earns its latency when your first-stage retrieval is leaving real material on the table. Mine was not.
Had I skipped the evaluation set, I would have shipped the reranker. It is what the tutorials recommend, the code worked, and every anecdotal check I ran on it looked fine. I would have paid 5.7 times the latency for an improvement I could not measure, and I would never have known.
Keeping the code
Both the reranker and query expansion are still in the repository, implemented and benchmarked, just disabled. That is deliberate. The measurement is a property of this corpus at this size, not a permanent verdict. If the corpus grows, if the questions get harder, or if first-stage hit rate drops, the reranker becomes worth turning on again, and the switch is a config change rather than a rewrite.
Instrumentation
None of the above is possible without visibility, so the full request path is instrumented: latency broken out by stage, token cost per request, and LLM-as-judge relevance scoring on live traffic, all feeding a 13-panel Grafana dashboard. The ingestion pipeline is staged and resumable, because re-embedding 31,700 chunks after a crash three-quarters of the way through is a mistake you make exactly once. The whole thing runs under Docker Compose with 76 offline tests.
This part connects backwards to my day job more than it might look. I spend my working hours on message-driven backends where throughput and failure modes are the whole problem, and where you learn quickly that a system you cannot observe is a system you cannot reason about. Retrieval pipelines are no different. The domain changed; the instinct did not.
What I would tell someone starting
Build the evaluation set first, and make it big enough that your standard error is smaller than the improvements you care about. Mine was borderline at 450 questions, which is exactly why the reranking result landed inside the error bars rather than cleanly outside them.
Then measure everything you add, and be genuinely willing to remove it. The most useful output of this project was not the pipeline. It was a number that told me to delete work I had already finished.
The code is on GitHub, evaluation harness included.