Looking for a promptfoo alternative? Decide what you are testing first

People search for a "promptfoo alternative" for four very different reasons, and only two of them are actually about promptfoo. Working out which one you are is worth more than any comparison table, so let us do that before the table.

This post is written by a team that builds a competing product. That is a reason to be suspicious of it, so here is the deal: the section that says keep using promptfoo is longer than the section that says otherwise, and every claim about promptfoo below comes from its own documentation.

What promptfoo actually is

promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. You declare evals in a config file rather than writing a test harness, point it at prompts and providers, and it runs your cases concurrently with caching.

It does two jobs. The first is security red-teaming — automated scans for prompt injection, data leakage, jailbreaks, and tool misuse. That work is genuinely excellent and it is why OpenAI agreed to acquire the company in March 2026, with the core staying open source under its current licence. We do not compete with that half at all: if adversarial testing is your goal, promptfoo is purpose-built for it and Rubrkit is not.

The second job is evals, which is the half this post is about. Its real substance there is the assertion library. You get deterministic checks — equals, contains, regex, is-json, is-sql, javascript, python, levenshtein, latency, cost — and model-graded ones — llm-rubric, g-eval, factuality, answer-relevance, context-faithfulness, moderation, classifier. Assertions carry weights, a test case scores as their weighted average, any assertion can be negated with a not- prefix, and a threshold turns the score into pass or fail. There is a GitHub Action for CI.

That is a good design, it is free, and if it fits your problem you should stop reading and go use it.

Reason one: "I cannot get it to tell me whether my prompt is good"

This is the most common reason, and promptfoo is not the problem.

promptfoo grades outputs against cases you wrote. Every assertion above assumes you can already finish the sentence "a correct answer would…". llm-rubric is the clearest example: it grades output against a rubric, and the rubric is a string you supply. The tool is a very good judge. You are still the one who has to write the law.

So if you arrived at promptfoo hoping it would tell you whether your prompt was any good and left empty-handed, the missing piece sits upstream of any eval runner. You do not have test cases yet. No alternative eval framework fixes that, because they all take the cases as input.

There are two honest ways out. Write the cases — genuinely the best answer, and how to test a prompt walks through doing it without fooling yourself. Or grade the instruction itself first, which is the thing we built and the one real architectural difference worth naming below.

Reason two: "the scores move around and I do not trust them"

Also not a promptfoo problem. It is a property of using a language model as a judge, and every tool in this category inherits it, ours included.

A model-graded assertion is a sample, not a measurement. Run the same unchanged text through llm-rubric five times and you will usually get five different numbers. If you have not measured how wide that spread is on your artifact, you cannot tell a real improvement from noise — and the spread is not a constant you can look up. Across our own corpus, measured standard deviation on the same rubric has ranged from 0.82 to 3.06 depending on the artifact. A single global "2 points is meaningful" rule would cry wolf on one file and sleep through a real regression on another.

The fix is repeats and statistics, not a different runner. We wrote up the arithmetic in how to get statistical significance for your prompts. You can do this on top of promptfoo yourself: run the case n times, keep the distribution instead of the last number, and compare with a t-test rather than an eyeball. It is not free — cost scales with repeats — and most teams who try it discover they had been shipping on a coin flip.

Reason three: "I need something upstream of test cases"

This one is real, and it is the only place we will claim a genuine difference.

Every eval framework, promptfoo included, is output-shaped: prompt in, output out, assertions on the output. That loop is the right one for a feature you have already specified. It is the wrong one on day zero, when the artifact you are worried about is a 400-line agent instruction, a skill file, or a system prompt that nobody has written a single test case for and that quietly says "handle edge cases appropriately."

That vagueness is a defect you can find by reading the instruction. You do not need a test case to know that "appropriately" is unfalsifiable, that there is no defined move for empty input, or that the output contract is missing. An output eval will eventually catch the symptom, on some input, sometimes. Reading the artifact catches the cause, deterministically, before you have spent a single token on a test run.

So Rubrkit grades the artifact rather than the output, against a fixed ten-dimension rubric — objective clarity, output specification, evaluation criteria, bounded behavior, context sufficiency, constraint quality, failure handling, reusability, maintainability, safety. The rubric being fixed is the whole point. With llm-rubric you author the criteria, which means your blind spots end up in the rubric too. A standard rubric asks the ten questions you did not think to ask, and it asks the same ten tomorrow, in CI, and in the pull request.

# Structural checks, fully local — no API, no credits
npx rubrkit validate "prompts/**/*.md"

# Score against the ten-dimension rubric
npx rubrkit test "prompts/**/*.md" --remote

# Gate the build
npx rubrkit audit team-agent-kit --fail-under 85 --ci

The CLI runs under npx on Node 22 or newer, and its exit codes distinguish failed the bar from could not run — the full loop is here.

Two things an output eval structurally cannot do

Beyond the upstream question, two capabilities fall out of grading a frozen artifact that an output-shaped runner cannot express.

Drift caused by the model, not by you. Pin a version and re-audit it on a cadence. The artifact has not changed, so any movement is the model or the rubric moving underneath you. This only works with per-artifact calibration — the first check measures that artifact's own mean and spread before it is allowed to report drift, for the 0.82-to-3.06 reason above. In a purely output-shaped world, a provider quietly shipping a new model version looks exactly like your prompt getting worse, and you go and edit a prompt that was fine.

Verdicts instead of scores. An advanced comparison audits a baseline and a candidate n times each and returns better, worse, no_significant_change, or underpowered — Welch's t-test per dimension with a 95% interval, Benjamini-Hochberg corrected because ten simultaneous tests otherwise manufacture false positives. underpowered matters more than it sounds: too few repeats reports itself instead of guessing.

Then golden sets, which are less clever and probably more useful: every failure anyone finds — from an audit, a human, or production — gets pinned as a case, so the set sharpens over time instead of staying theoretical. That is a growth loop, and it is the part promptfoo users tend to build by hand in a tests/ directory.

Reason four: "I do not want my eval tool owned by a model vendor"

This one is about promptfoo, and it is a legitimate thing to weigh rather than a gotcha.

The core stays MIT-licensed and model-agnostic, and OpenAI has said it will keep supporting existing customers. For most teams that is the end of it — an MIT licence you already have cannot be taken back, and self-hosting means nobody can change the terms underneath you. If you self-host promptfoo, this reason is weaker than it feels.

Where it is worth taking seriously is if your eval tool is the thing that decides whether a model change ships. An evaluator maintained by one of the vendors it grades is a conflict of interest even when everyone involved is acting in good faith, and roadmap attention follows the acquirer's priorities — here, agentic security inside OpenAI Frontier. If your reason for switching is neutrality, buy neutrality specifically: pick something independent and model-neutral, and check that it stays that way.

Where the line is

Read that table honestly and promptfoo wins several rows outright. It is open source, you can self-host it, its deterministic assertion library is far broader than ours and always will be, its red-teaming has no counterpart here, and it supports providers we do not. If your problem is "does this RAG pipeline return grounded answers across my 200 labelled questions," promptfoo is the correct tool and we are not competing for that job.

The full dimension-by-dimension matrix, including where we concede, lives on Rubrkit vs Promptfoo. If promptfoo is not the tool you were really comparing against, the alternatives index has the same treatment for Langfuse, Braintrust, and PromptLayer.

Using both, which is what we actually do

These are not substitutes. They sit at different points in the same pipeline:

# 1. Is the instruction itself well-formed? Cheap, needs no test cases.
npx rubrkit test "prompts/**/*.md" --remote --fail-under 80 --ci

# 2. Does the system built on it behave? Your cases, your assertions.
npx promptfoo eval

Step one is a gate you can run on a prompt the day it is written. Step two needs cases, and cases take time to accumulate — which is exactly what a golden set is for. Running step one first means the expensive run in step two gets spent on real behavioural questions, rather than on rediscovering that a clause was ambiguous, which a careful reader could have told you for free.

The honest limit

Rubrkit is a first gate, not the evaluation. A rubric score is a judgement about the artifact's quality as our rubric defines it. Even a statistically significant advanced comparison establishes that a change is better by the rubric, not merely different — it says nothing about whether your users are better served. That needs a labelled dataset or live outcomes, and any tool implying otherwise from an offline run is selling you something.

Which is the same reason a passing promptfoo suite is not proof your feature works. It is proof your cases pass. Both are worth having, and neither is the finish line.

If what you actually needed from promptfoo was test cases you did not have, start with the artifact. If you were switching for neutrality, buy neutrality on purpose. And if you had the cases all along, promptfoo was never your problem.