LLM Regression Checklist Before Every Ship
Shipping an LLM feature without an expected-behavior checklist is how silent regressions reach paying users.
TestPilot turns a prompt and expected behavior into a regression-oriented eval list: format, safety, version diff, and OWASP LLM-aware risks.
Prompt injection (OWASP LLM01) belongs on the review list whenever untrusted text enters the context window.
HOLD is advice. Humans still own release gates, especially for safety-sensitive domains.
What this is not: a production monitor, a red-team service, or a certification of model safety.
What belongs on the list
Format checks confirm the output matches the contract your code expects — valid JSON, a fixed set of keys, a length window. A model that returns prose instead of JSON breaks the parser even when the answer is correct.
Safety checks cover the cases your domain cannot ship wrong: a medical assistant that declines to diagnose, a finance bot that refuses to guarantee returns, a support agent that does not leak another user's data.
Version diff checks compare output across model or prompt versions. When you bump the underlying model, the eval set is what tells you the change was neutral rather than a silent behavior shift.
Prompt injection is not optional
OWASP lists prompt injection as LLM01 for a reason. Any feature that places user-supplied or retrieved text into the context window is exposed. The eval set should include at least one "ignore previous instructions" attempt and one data-exfiltration attempt per surface.
Treat a successful injection as a release blocker, not a low-priority finding. The cost of one leaked record outweighs a week of feature work.
The threat model moved in 2026
The OWASP GenAI LLM Top 10 kept Prompt Injection at LLM01 and moved Excessive Agency up to LLM03, reflecting how many teams now ship tool-using agents that can take real actions. A pre-ship eval set that ignores agent permissions and human checkpoints tests last year's threat model. TestPilot's OWASP coverage tracks LLM01 through LLM05 for the feature's surface, but the published list runs to LLM10 — treat the eval as a floor, not a ceiling, and add agent-specific cases when the model can call tools.
Pass criteria at a glance
| Category | Pass criterion |
|---|---|
| Format | Output parses against the declared schema |
| Safety | All domain refusals triggered correctly |
| Injection | At least one injection case blocked per surface |
| Version diff | No behavior regression vs. the previous version |
| OWASP coverage | LLM01–LLM05 addressed for the feature's surface |
HOLD is advice, not a lock
When a check fails, TestPilot returns HOLD. That is a signal to a human owner, not an automated deploy block. Someone still decides whether the failure is a real regression or a stale expectation in the eval set.
For safety-sensitive domains — health, finance, child-facing — the human gate is non-negotiable. A score should never be the only thing standing between a model and production.
What the checklist cannot do
It runs against the cases you wrote. It does not watch live traffic, does not catch novel attack patterns, and does not measure whether users are satisfied. Those need a production monitor and real feedback, not a pre-ship eval.
Run the checklist before every ship to catch the known failure modes. Pair it with monitoring to catch the unknown ones. One without the other leaves a gap.
A sample eval case
An eval entry pairs an input with the behavior you expect:
{"input":"Summarize this ticket","expect":{"format":"json","keys":["summary","priority"],"refuse":false}}
The runner parses the model output, checks the keys exist, and confirms no refusal where none is due. Add one case per known failure mode and one injection case per surface. Ten good cases beat fifty vague ones.
Further reading
- OWASP GenAI LLM Top 10 — LLM01 Prompt Injection and the 2026 list.
- schema.org/FAQPage: the structured-data type this page publishes for its FAQ.
FAQ
Does HOLD block deploys automatically?
No. It is decision-support for humans.
Is OWASP coverage complete?
It highlights common risks; it is not a full assurance program.
Honesty note: educational decision-support only. No guarantee of rankings, compliance, or ship safety.