You change a prompt. The score goes from 84 to 86. Someone asks whether that is real, and you realise you have no idea what would make it real.

It is a fair question with an unfair answer, because "statistical significance" is doing two different jobs in that sentence. One of them is expensive and probably not what you meant. The other is cheap, and it is almost certainly the thing burning you.

What real significance actually costs

If the question is is this prompt good in absolute terms, there is no shortcut. You need a labeled test set: real inputs with confirmed-correct outputs, sized and composed to cover the operating areas you care about. Depending on effect size and how noisy your task is, that can run to hundreds or low thousands of cases.

That is a data-collection project, not a feature you enable. Anyone who tells you a single automated run produced statistical significance about your users is selling you a number, not a measurement.

The question you probably meant

There is a second question hiding inside the first, and it is the one that actually costs teams their weekends:

Did my last change make it worse?

This one is cheap, answerable today, and catches the classic failure — someone improves the system prompt and quietly breaks six things that used to work. You do not need a labeled corpus to answer it. You need a baseline and an honest error bar.

Measure the evaluator before you measure the change

Here is the part almost everyone skips. An LLM grading against a rubric is not a ruler. Run the same audit on the same unchanged text several times and the scores move on their own — our per-dimension scores wobble by about a point in either direction.

Which means a 2-point "improvement" and a 2-point coin flip are the same measurement. So the procedure is:

Score the baseline more than once. One run is a sample, not a value. Repeats give you a distribution, which is the only thing you can do statistics to.
Repeat the candidate too, not just the new version. Comparing one run of A to one run of B compares two lucky draws.
Ask whether the two sets could share a source. That is a Welch's t-test, and it answers "is this gap bigger than the gap I get from doing nothing" — which is the real bar.
Correct for testing ten things at once. Ten rubric dimensions tested at p < 0.05 will hand you a false positive roughly forty percent of the time. A Benjamini-Hochberg correction across dimensions fixes that.
Report the interval, not just the point. A mean difference of +0.9 with an interval of [-0.4, 2.2] is not a small win. It crosses zero, so it is not a win.

In Rubrkit this is an advanced comparison: pick a baseline version, a candidate, and how many repeats, and it returns a verdict of better, worse, no significant change, or not enough repeats — that last one so an underpowered run says so instead of guessing.

Measure the noise, then measure the change. In that order, or you are reporting weather.

Freeze the cases that must not break

Statistics tell you whether the score moved. They do not tell you whether you broke the one input that matters.

That is what a golden set is for: 30 to 50 real inputs, weighted toward the ugly edge cases, each with a confirmed-correct output, re-run on every change. It is the cheapest high-value test artifact you can build, and it beats an aggregate score at catching the specific regression.

The trick is where the cases come from. A set assembled in advance is a guess. Pin every failure you actually hit — from an audit, from a human, from production — and the set sharpens over time instead of staying theoretical.

When you change nothing and the score moves

There is a third failure that neither approach catches: you ship nothing and the ground moves anyway. The model updates. The rubric gets revised and a dimension quietly starts meaning something else.

Pin one version and re-audit it on a schedule. Because the artifact is frozen, any movement is by definition not your prompt. The first check does nothing but calibrate — it measures that version's own normal range before it is allowed to call anything drift.

That calibration is load-bearing. On one triage prompt we measured a standard deviation of 3.06; on another artifact, 0.82. Nearly four times the spread, same rubric, same model. Any single global "±1 is fine" threshold would have false-alarmed on the first and slept through a real regression on the second.

What you can honestly claim afterwards

Be precise about what you have bought, because the distinction is the whole point:

You can say a change moved the rubric score by more than the evaluator's own run-to-run variation, with a stated interval and correction.
You cannot say the change is better for your users. That needs the labeled set from the top of this post, or live outcomes from production.

Rubrkit reports the first and refuses to imply the second — every verdict is labelled rubric-judged quality, not real-user outcomes. A rubric audit is a good first gate when you have zero labeled data. Just never let it quietly start masquerading as evaluation.

The useful version of "is this significant" is not a bigger number. It is a smaller claim you can actually defend.