How to choose an LLM: the cheapest model that passes.

Stop asking which model is best. Decide what good enough means for your task, test models from every price tier against that bar, and choose the cheapest one that clears it at your real volume. On the list prices we track, the same workload costs 100 times more on the most expensive models than on the cheapest, so an afternoon of testing usually pays for itself in the first month.

Why “best” is the wrong question

Leaderboards rank models on someone else’s tasks. The model at the top of a reasoning benchmark may be the right call for drafting a legal memo and a waste of money for tagging support tickets by topic. Rankings also turn over every few months, so a choice made on reputation is a choice you will be arguing about again by spring.

The useful question is narrower, and you can answer it: for this prompt, on these inputs, which models do the job, and what does each one cost at our volume?

Define the bar before you look at prices

Write the bar down first, or the prices will start writing it for you. A good bar has four parts.

• Must-pass checks. Rules a good answer never breaks: valid JSON, a category from your list, the order number quoted back, no promises about refunds. Each check is pass or fail.
• A quality score for what rules cannot catch, such as tone or faithfulness to a source. An AI judge can score each answer from 1 to 5 against a criterion you write. Set a minimum average.
• A speed limit, if a person is waiting. For live chat, how soon the answer starts matters more than how fast it finishes, so put a number on latency.
• The failure rate you can live with. Hamel Husain’s essay on why AI products need evals puts it well: “your pass rate is a product decision.” A draft that a person always reads can fail more often than a reply sent straight to a customer.

Then compress it into one sentence. For example: on our 40 test cases, every must-pass check passes, the judge averages at least 4 out of 5, and the first words appear within 2 seconds. A model clears that or it does not.

Price it at your volume, with real prices

Take a hypothetical workload: triage inbound customer emails and draft a first reply. Each call sends about 1,200 tokens (instructions plus the email) and gets back about 300. Volume is 50,000 emails a month. Here is the arithmetic for one model:

GPT-6.1 Sol, $2 input and $10 output per million tokens
  1,200 input  × $2   =  2,400
    300 output × $10  =  3,000
                         5,400 ÷ 1,000,000 = $0.0054 per call
  × 50,000 calls = $270 a month

And the same workload across the catalog:

The top row costs 100 times the bottom. Notice also that output is a fifth of the tokens but more than half the cost on every row. A length limit in the prompt (“at most 120 words”) is often the cheapest saving available, whichever model you pick.

Count the cost of a failure

A price table invites the wrong conclusion: pick the bottom row. The bottom row is only cheaper if it passes.

Suppose your test suite shows Claude Haiku 5.5 handling 94% of cases and Claude Opus 5.5 handling 99%, and every failure costs someone four minutes to catch and fix. These pass rates are made up for the example; yours will differ. At 50,000 emails a month:

Claude Haiku 5.5   6% of 50,000 = 3,000 failures × 4 min = 200 hours
Claude Opus 5.5    1% of 50,000 =   500 failures × 4 min ≈  33 hours

Extra people time with the cheaper model    ≈ 167 hours a month
Saved on the model bill     $540 - $13.50 = $526.50 a month

The cheaper model saves $526.50 and costs about 167 hours. Unless an hour of your team’s time is worth less than $3.16, the expensive model is the cheap one.

Now change one number. If Haiku had passed 99% as well, you would save $526.50 a month for nothing, every month. You cannot know which of those two worlds you are in without running the test, and that is the entire case for running it.

So compare models by cost per successful outcome: the model bill, plus the cost of catching and fixing what it gets wrong. Price per token is only the first line of that sum.

Run the test

1. Write the tests first. Twenty to fifty real inputs, including the awkward ones, with the checks and the judge criterion from your bar. The guide to testing a prompt walks through it.
2. Pick candidates across tiers. At least one top, one mid and one fast model, from more than one provider. If a model lets you set reasoning effort, treat each setting as a separate candidate, because higher effort bills more output tokens per call.
3. Look at one real request side by side. Model Compare runs it on up to six models with speed and cost for each. Turn on the judge to see Best value: the cheapest answer whose judge score is within one point of the top.
4. Run the full suite on the shortlist in Prompt Tester, with the same prompt and settings on every model. Add a cost check so any answer over your per-call budget fails.
5. Price the models that clear the bar in the token and cost calculator, with your real prompt, your volume and your caching.
6. Choose the cheapest that clears it, and keep the runner-up as a fallback for outages and price changes.

Sometimes the result is two models: a fast one for most requests and a stronger one for the cases it fails. Do that only if the failures share a trait you can detect before answering, such as very long emails or a particular language. Otherwise routing is one more thing to test and maintain.

Re-run when prices move

Your choice has a shelf life. These prices in our catalog are already scheduled to change, as listed on the vendors’ pages on October 20, 2026:

• Gemini 4 Argon: an introductory $2 / $10 until November 30, 2026, then $4 / $20.
• Gemini 3.8 Flash: $0.75 / $3.75 today, $1.50 / $7.50 from January 1, 2027 (Google’s pricing page).
• Mistral Large 4: a preview sale price of $0.68 / $2.09, against a regular $1.36 / $4.18.
• DeepSeek V4 Pro and DeepSeek Flash: listed at the peak-hour price, and half that outside the peak windows.

Budget at the price you will pay in three months, not the one on today’s invoice. Re-run the suite when a price changes, when a new model ships in a tier you use, and whenever you change the prompt. With the suite already written, each re-run takes an afternoon.

Keep reading

• What an LLM call really costsThe cost formula, caching, and where the money goes.Guide, ends in the AI Cost Calculator and Model Compare
• What a support chatbot costs at 10,000 conversations a dayTokens per conversation, caching and batch, with the arithmetic shown.Blog
• Why AI answers change every timeOne good run proves little. How many runs you need, and when it matters.Blog