Most teams do not test prompts. They check them: run the new version on two or three inputs, read the output, decide it looks better, ship. That is not a test. A test is something that can fail without you watching, and tells you what broke.
This is the procedure we use, in the order you should build it. None of it requires a labeled dataset or an eval platform to start.
Step 0: know what kind of testing you are doing
Two different things get called "prompt testing," and conflating them is why teams stall:
- Benchmarking — is this prompt good in absolute terms? Needs labeled data, hundreds of cases, real effort.
- Regression testing — did my last change break something that used to work? Needs a baseline and a handful of frozen cases.
You want the second one first. It is cheap, it catches the failure that actually costs you weekends, and you can have it running this afternoon. Everything below is regression testing.
Step 1: build a golden set from real failures
A golden set is 30 to 50 inputs with confirmed-correct outputs, re-run on every change.
The mistake is assembling it up front from imagination. A set you invent in advance tests the cases you already thought about — which are, by definition, the ones your prompt already handles. Build it the other way:
- Start with five. Three normal cases, two you know are ugly.
- Pin every failure you actually hit. A bad output from production, a weird result during a demo, an edge case a teammate found. Each one becomes a permanent case.
- Weight toward the edges. Empty input, missing fields, hostile input, the longest realistic document, the ambiguous case where two answers are defensible.
- Keep the expected output, not just the input. "This should not crash" is a weak assertion. "The Impact line should read unknown" is a test.
A set built this way sharpens every time something goes wrong. A set built in advance stays theoretical forever.
Step 2: write pass criteria a second person could apply
Here is where most prompt tests quietly become vibes with extra steps. "Output should be helpful" is not a criterion. If two reasonable reviewers would disagree about whether a given output passes, you do not have a test.
Criteria that actually hold up come in three flavors, in descending order of reliability:
- Deterministic assertions. The output contains three labelled lines. The customer name never appears. The JSON parses and has these four keys. Use these wherever you can — they never drift and they cost nothing to run.
- Rubric scores with a floor. The instruction scores at least 70 overall and at least 6 on bounded behavior. Good for quality that resists a regex.
- Model-graded comparison. Does this output answer the question the reference answer answers? Most expensive, least stable, use last.
Reach for the cheapest one that can catch the failure you care about. Teams reflexively reach for an LLM judge and then discover they have to test the judge too.
Step 3: measure your evaluator's noise before you trust a delta
This is the step everyone skips and it invalidates everything downstream.
An LLM grading against a rubric is not a ruler. Run the same audit twice on the same unchanged text and the score moves — for us, typically about a point in either direction, though it varies a lot by artifact. On one triage prompt we measured a standard deviation of 3.06; on another, 0.82. Nearly four times the spread, same rubric, same model.
Which means a 2-point "improvement" and a 2-point coin flip look identical. Before you can call any change an improvement:
- Score the baseline more than once. One run is a sample, not a value.
- Repeat the candidate too. Comparing one run of A to one run of B compares two lucky draws.
- Ask whether the gap exceeds the noise. A Welch's t-test answers exactly this: is this difference bigger than what I get by changing nothing?
- Correct when you test ten dimensions at once. Ten dimensions at p < 0.05 produce a false positive roughly forty percent of the time. Benjamini-Hochberg fixes it.
If a single global threshold like "±1 point is fine" sounds appealing, note that it would have false-alarmed constantly on the noisy artifact above and slept through a real regression on the quiet one. Thresholds have to be per-artifact, and they have to be measured.
Step 4: put the gate in CI
A test you have to remember to run is a test that stops running in about three weeks.
Wire it into the same place your unit tests live. With Rubrkit that is one command:
npx rubrkit test
Fail the build when a golden case regresses or a rubric score drops below your bar. The point is not the tooling — it is that the bar travels with the instruction. When someone tidies up the prompt six weeks from now and quietly drops the PII rule, the build goes red instead of the customer's trust.
Step 5: watch for drift you did not cause
There is a failure mode none of the above catches: you ship nothing and the score moves anyway. The model updates underneath you. The rubric gets revised and a dimension starts meaning something slightly different.
Pin a version and re-audit it on a schedule. Because the artifact is frozen, any movement is by definition not your prompt. Make the first scheduled run do nothing but calibrate — measure that version's own normal range before it is allowed to call anything drift.
The order matters
If you only do one of these, do the golden set. If you do two, add the CI gate. The statistics matter enormously the moment you start making claims about improvement — but a frozen set of real failures, re-run automatically, catches more production bugs than any amount of significance testing on a set you never built.
Test the thing you would be embarrassed to break. Then make it impossible to break it quietly.