RAG & Retrieval · Ground answers in evidence
What is a cross-encoder and why is it slower?
Short answer
A cross-encoder runs the query and passage through the model together for each pair, so scoring N candidates costs N forward passes. It is accurate but unsuitable for first-pass search at scale.
Bi-encoders precompute document vectors, so search is a vector lookup. Cross-encoders cannot precompute a document score without the query, because the interaction between the two matters. Every candidate requires a separate pass, which is why they are used on a shortlist rather than the full corpus.
- Use a cross-encoder on the top 20–100 candidates.
- Batch pairs to use the hardware efficiently.
- Measure p95 latency, not only average, because reranking is on the critical path.
- Consider a smaller reranker or a hosted service if latency dominates.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1Why can't you precompute cross-encoder scores for all documents?
- 2How would you reduce reranker latency?
- 3What happens to a relevance threshold when you switch rerankers?
Say it in the interview
A cross-encoder processes the query and passage together, so it needs a forward pass for every candidate. That interaction is what makes it accurate, but it also makes it far too slow for the first pass over a large corpus. I use it only on a shortlist from retrieval, batch the pairs, and watch p95 latency because it sits on the critical path. Changing rerankers also means recalibrating any score thresholds.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live RAG & Retrieval mock interview and get scored feedback on depth, clarity and confidence.