LLM Evals · Turn quality into a signal
When is semantic similarity misleading?
Short answer
When the answer needs specific facts, numbers, negations or citations. Embeddings capture general meaning, so a confident wrong answer can score close to the correct one.
Semantic similarity is useful for paraphrase and topic matching. It is weak for correctness. 'The refund window is 30 days' and 'the refund window is 90 days' are nearly identical in embedding space but one is wrong. Negation and entity swaps are also poorly captured.
- Use it for coverage or topic match, not factual accuracy.
- Check numbers, dates, names and negations with deterministic or claim-level tests.
- Prefer claim decomposition or entailment for factual grading.
- Validate the threshold on examples where the answer is subtly wrong.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1How would you catch a wrong number that is semantically close?
- 2What metric would you use for paraphrased explanations?
- 3Why does negation break similarity metrics?
Say it in the interview
Semantic similarity measures broad meaning, not factual correctness. Two answers can be almost identical in embedding space while one has the wrong number or a flipped negation. I use it for topic or paraphrase checks, but for facts I check claims, entities and numbers directly. I also sample high-scoring failures, because that is where a similarity metric is most likely to hide an error.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.