AI Engineering · LLM applications that ship
How do you regression-test prompt changes?
Short answer
Run the same versioned eval set against the old and new prompt, compare per-case results and aggregate scores, and require a review of any case that flips from pass to fail.
Prompt edits are code changes with fuzzy behaviour. A wording tweak that fixes one failure often changes tone, format or refusal behaviour elsewhere. Regression testing makes those side effects visible before release.
- Keep prompts in version control with a changelog and an owner.
- Run the eval suite on both versions and diff the per-case verdicts, not just the average.
- Watch for format drift and refusal-rate changes, not only answer correctness.
- Add a new case for every production bug the change is meant to fix.
Follow-ups you should expect
Interviewers use these to check whether the first answer was understanding or memorisation.
- 1What would make you block a prompt release?
- 2How do you test prompt changes when the model provider also updates the model?
- 3Would you compare outputs side by side or only scores?
Say it in the interview
I treat prompts like code. They are versioned, and every change runs against the same eval set as the previous version. I compare per-case outcomes, because an average can improve while a critical case breaks. I also watch format and refusal rates, since prompt edits often change those quietly. Any case that flips to fail gets reviewed before release, and the bug that motivated the change becomes a permanent test.
Use this as a model, not a script. Saying it in your own words is what makes it survive follow-up pressure.
Related questions
Reading is step one
Now say it out loud.
Understanding an answer and delivering it under pressure are different skills. Rehearse this topic in a live AI Engineering mock interview and get scored feedback on depth, clarity and confidence.