Prompt Tester
Run every version of a prompt on the inputs you care about, across models. Exact checks and an AI judge score each answer, and the Prompt Tester shows which cases your last change fixed and which it broke.
New to this? What it does, in plain words
It tests a prompt on many real examples at once, so you can see whether a change made it better or worse.
Use it before you change a prompt that people rely on. A support team can check that a new reply prompt still quotes the refund policy correctly on twenty real customer emails. A marketing team can confirm that a shorter product-description prompt still mentions price and size every time. A repeatable test like this is called an eval.
What you need
Nothing, to look around. The Prompt Tester opens with an example set of tests (a suite) and example results written by hand, so you can see how it works without a key. To test on a real AI model, add an API key from a provider such as Anthropic, OpenAI or Google. Google AI Studio, OpenRouter and Groq give free keys. Your keys, prompts and results stay in this browser, and each call goes straight to the provider you pick.
How to use it
- Write your prompt in the Prompt tab. Put a blank such as {{ticket}} where each example's text should go.
- Add test cases in the Test cases tab: real inputs, one per row. You can paste them from a spreadsheet.
- Add checks in the Checks tab, such as “mentions the order number” or “under 120 words”. Use the AI judge for qualities like tone.
- Pick models in the Run tab and press Run. Make a new version of the prompt, run both, and the Prompt Tester lists which cases the change fixed and which it broke.
Evals for prompts, without the setup
An eval answers one question with evidence instead of a hunch: does this prompt do its job on the inputs that matter, and did the last change make it better or worse?
- Free, in your browser. Write test casesReal inputs you care about, the awkward ones on purpose, each with a reason it is there.A dozen is enough to start
- With your key. Run every caseEach version of your prompt runs on each case, on every model you pick.One call per case, version and model
- With your key. Each answer is checked two waysPass when every check passesExact checksContains a phrase, valid JSON, within a length or a cost. Free, instant and never wrong.AI judgeA second model scores what rules cannot, such as tone or accuracy.
- Free, in your browser. Compare versionsCase by case on the same model: what the change fixed, and what it broke.Instant
What a prompt eval is
You write down the inputs you care about (the test cases), the rules a good output must follow (the checks), and then run each version of the prompt on every case with one or more models. Each output passes when every check that is on passes. The pass rate per version and model is your scorecard; the list of cases that changed between versions is your regression report.
Tools such as LangSmith, Braintrust and Maxim do this for teams with an SDK, an account and a budget. Prompt Tester does the everyday part of it in a browser tab: no install, no sign-up, nothing stored on a server.
How to write good test cases
- Start from real inputs. Copy a dozen real tickets, documents or questions. They expose problems that invented ones hide.
- Add the hard ones on purpose: ambiguous requests, very short and very long input, typos, another language, missing information, and input that tries to give the model new instructions.
- Give each case a reason to exist. The note column is for what a case tests. Two cases that test the same thing make the pass rate look better than it is.
- Fill in expected values where there is one right answer, such as a category or a number. Leave them empty for open-ended writing and use the judge instead.
- Eight to twenty cases catch most problems. Add a case every time you find a failure in real use, so the same mistake cannot come back unnoticed.
Checks, and the limits of the AI judge
Exact checks test one thing and never get it wrong: whether the output contains a phrase, matches a pattern, parses as JSON, has the right keys, equals the expected value, stays within a length, a latency or a cost. They are free and instant, so use them for everything they can express.
The AI judge is a second model that scores each output from 1 to 5 against one criterion you write, such as faithfulness to a source or the right tone for an audience. It sees the prompt, the expected value if there is one, and the output, and it is told to treat all of them as data rather than instructions. Judges are useful and fallible: they can be swayed by length or confident wording, miss subtle errors, and disagree with themselves between runs. Keep each criterion narrow and checkable, read the judge's reasons on the cases it fails, and do not treat a judge score as ground truth. Each judge check adds one model call per output, which the cost estimate includes.
How regressions are found
When a run includes two or more versions, the Prompt Tester compares them case by case on the same model. A fix is a case that failed in the earlier version and passes in the later one; a regression is a case that passed and now fails. Errors (a rate limit, a network failure) are left out of the comparison, because they say nothing about the prompt. A headline like “v2 fixed 5 cases and broke 1” is usually the most useful sentence a change can earn: open the broken case to see both outputs side by side and which check flipped.
Models are not fully deterministic, so a single flip on one case can be noise. If a regression matters, run the suite again before you act on it, and add more cases like the one that flipped.
Privacy
Every call goes from your browser straight to the model provider you pick, with your own API key. thinkhard.si has no server that sees your prompts, test cases, outputs or keys. Suites and results are saved in this browser's storage; export a suite as JSON to move it, or share it as a link whose contents live after the # and are never sent to a server. Keys are never included in exports or share links.
Questions
Do I need a paid API key?
No. Google AI Studio, OpenRouter (free models) and Groq all offer free keys that work here. The example runs in each starter suite let you see what results look like before you add any key.
What does a run cost?
Each case is one call per version per model, plus one call per AI judge check. The Run tab shows the number of calls and an estimate before you start, based on approximate token counts and list prices. Your provider's bill is the real number; set a spending limit on any key you use in a browser.
Can I stop a run halfway?
Yes. Stop keeps every result that finished. Cases run in order, each across every version and model, so a stopped run still compares versions fairly on the cases it reached.
Why did a case fail with an error instead of a verdict?
The provider returned an error (often a rate limit) or the judge could not score the output. Errors are shown in the cell and are not counted in the pass rate. Lower the concurrency and run again if you see many rate-limit errors.
Are the example results real?
No. The example run in each starter suite uses illustrative outputs and judge scores written by hand, and is labeled as such everywhere it appears. Its pass and fail verdicts are computed by the same check engine your real runs use.
Can I run a suite automatically, for example on every deploy?
Not yet. Scheduled runs and a command-line runner for CI are planned for Pro. Today you can export a suite as JSON and keep it next to your code.