Turn quality into a signal
LLM Evals Interview Questions
Golden sets, judges, faithfulness, agent trajectories and CI gates. How to measure LLM quality without fooling yourself.
40 questions5 easy26 medium9 hard0/40 learned
01Why do LLM features need evaluations?Easyevals · quality · regression02What is eval-driven development?Easyevals · workflow · development03Offline vs online evaluation?Mediumevals · online · offline04What makes a good golden dataset?Mediumdataset · evals · labels05How do you pick metrics for a specific task?Mediummetrics · evals · design06What should an evaluation suite cover?Mediumcoverage · evals · edge cases07Who owns evals on a product team?Easyevals · ownership · process08How do you build a golden set from scratch?Mediumdataset · evals · process09How do you sample production traffic for evals?Mediumsampling · production · privacy10How do you write labeling guidelines?Mediumlabels · guidelines · quality11How do you measure agreement between labelers?Mediumlabels · agreement · statistics12How do you find and represent edge cases?Mediumedge cases · coverage · evals13When should you generate synthetic eval data?Mediumsynthetic data · dataset · evals14How do you prevent eval data leaking into training or prompts?Hardleakage · dataset · integrity15How does LLM-as-judge work?Mediumjudge · evals · grading16What biases do LLM judges have?Hardjudge · bias · reliability17Pairwise vs pointwise judging?Mediumjudge · comparison · evals18How do you design a scoring rubric?Mediumrubric · judge · quality19How do you calibrate a judge against humans?Hardjudge · calibration · human review20When is human review still necessary?Mediumhuman review · quality · safety21How do you keep judge cost under control?Mediumjudge · cost · evals22When is exact match the right metric?Easymetrics · exact match · deterministic23When is semantic similarity misleading?Mediummetrics · semantic similarity · evals24How do you measure groundedness?Mediumgroundedness · faithfulness · evals25How do you measure answer relevance?Mediumrelevance · evals · quality26How do you evaluate toxicity and safety?Mediumsafety · toxicity · evals27Should latency and cost be part of evals?Easylatency · cost · evals28How do you know an improvement is real?Hardstatistics · significance · evals29How do you run evals in CI without slowing the team?Mediumci · regression · automation30How do prompts and eval results stay linked?Mediumprompt versioning · reproducibility · evals31How do you A/B test an LLM feature?Mediumab testing · experiments · product32What do you monitor after launch?Mediummonitoring · production · quality33How do you turn user feedback into evals?Mediumfeedback · evals · iteration34How do you handle an AI quality incident?Hardincident response · safety · operations35How is evaluating an agent different from evaluating a chatbot?Hardagents · evals · trajectory36What is trajectory evaluation?Hardagents · trajectory · evals37How do you evaluate tool calls?Mediumtools · agents · evals38How do you evaluate multi-turn conversations?Hardmulti-turn · conversation · evals39What belongs in an eval harness?Mediumharness · tooling · evals40When do evals give false confidence?Hardevals · pitfalls · quality
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live LLM Evals mock interview and get scored feedback on depth, clarity and confidence.