The Real LLM Race Is Intelligence per Dollar

A model release can make headlines for beating a benchmark by a few points. For a team building with AI, the more useful question is often simpler: what does it cost to get a result we can actually use?

Imagine a coding assistant that produces a patch for a few cents, but leaves a developer with twenty minutes of cleanup. A more expensive model could be the better deal if its patch takes two minutes to review. The API bill captures only part of the cost.

That is what I mean by intelligence per dollar: useful work delivered within a budget, with an acceptable error rate and waiting time. A benchmark score helps us assess capability. It takes a workflow evaluation to establish value.

The evidence below brings together a September 2026 comparison and an older price series. I also propose a way to think about reliability, cost, and time together. The price data supports a specific claim: reaching some benchmark thresholds became much cheaper. It does not establish how quickly the total cost of reliable work has fallen.

What the price tag leaves out

Per-token pricing is easy to compare. The cost of finishing a task is harder.

Models can use different numbers of tokens, spend different amounts of time reasoning, or need several attempts. Tool calls and human review add costs of their own. A cheap response can become an expensive result if someone has to keep fixing it.

For a real deployment, I would track total spending across the workload and divide it by the number of accepted results. Failed attempts belong in that bill too. I would also measure completion time through validation, including how often tasks miss their deadline or never finish.

Artificial Analysis provides a useful starting point: a weighted average token cost per benchmark task, based on actual token consumption. This is still an evaluation cost, rather than a full production bill. Methodology and measurements.

The highlighted line shows the cost–quality frontier: among these eleven observations, each highlighted model offers a score that no cheaper option matches. This helps narrow a shortlist. It does not establish the best model for every application.

The distinction matters at the top. Claude Fable 5.1 and GPT-6 Astra both display a rounded score of 53, but their underlying scores are about 53.37 and 52.81. Both belong on this sample’s frontier. That small gap alone does not establish a meaningful performance difference.

This is a selected comparison at the effort settings recorded in the data. Adding other models or changing how much reasoning they use can change the frontier. Artificial Analysis’s broader v4.3 comparison includes other models and effort settings.

What became cheaper over time?

To compare across years, it helps to hold the target steady: how much did it cost to reach a given benchmark score?

Epoch AI’s historical dataset gives two examples on GPQA Diamond, a benchmark of difficult science questions:

• At the 33% threshold, matching GPT-4-0314’s recorded score, the lowest listed price fell from $37.50 to $0.1225 per million tokens between March 2023 and December 2024. That is roughly 306× cheaper.
• At the 56% threshold, matching the June 2024 Claude 3.5 Sonnet score, the price fell from $6.00 to $0.175 between June 2024 and February 2025: roughly 34× cheaper.

These calculations use Epoch’s published dataset and its blended token-price convention. Matching one score does not mean matching the original model’s capabilities across all tasks.

These are substantial declines, but their scope matters. They measure price per token at a benchmark threshold, without accounting for tokens consumed per completed task, retries, review, or latency. The source dataset ends in February 2025; this chart supplies no evidence about the subsequent nineteen months. See Epoch’s methodology.

For a product team, the useful implication is to revisit model choices. An option that was too expensive when a product was designed may become affordable later. Whether that saves money depends on how it performs in the product’s own workflow.

A proposed way to compare useful work

Price is only one part of the decision. I use Reliable Intelligence Yield (RIY) as a proposed framework for bringing quality, serious errors, cost, and time into the same discussion:

All four terms need the same task set and a fixed evaluation protocol. Unfinished tasks stay in the evaluation and receive the quality score and error penalty specified in advance. Keeping cost and time on a per-evaluated-task basis avoids counting only successful attempts.

A visible failure and a convincing mistake can have very different consequences. A broken code snippet may be caught immediately; a plausible but incorrect calculation may survive review. The error penalty expresses that difference. It needs a published severity scale, and any overlap with the quality score needs to be explicit so the same failure is not penalized twice by accident.

RIY is a proposal, not an established benchmark or a direct measurement of intelligence. Its ranking depends on how each term is defined. Multiplying cost by time also penalizes longer reasoning through both terms. That can be appropriate when both resources matter, but it is a choice rather than a law.

For many teams, a simpler rule will be easier to defend: set minimum quality and reliability requirements, set a deadline, then compare total cost per accepted result among the systems that qualify.

What happens when we try the idea on public data?

The next chart is an exploratory calculation, RIY-AA. It combines the Artificial Analysis index divided by 100, its AA-Omniscience non-hallucination measure, cost per task, and decode time in minutes. Dividing the index by 100 rescales it; it does not turn it into a probability of success.

The assumptions behind this chart matter more than the winner:

• The error measure is narrower than production reliability. AA-Omniscience defines hallucination rate as incorrect answers divided by incorrect, partially correct, and unanswered questions. Correct answers are excluded from that denominator. It measures guessing when a model fails to answer correctly, rather than the share of all tasks with serious errors. Benchmark paper.
• The inputs cover different tasks. The index combines ten evaluations; AA-Omniscience is one of them. Applying its error measure across the index adds a separate weight to one kind of behavior. Index composition.
• Decode time leaves out part of the wait. It excludes time to first token and overhead, so it is shorter than a complete workflow measurement. Time-per-task definition.

The calculation is also sensitive to values near 100% hallucination. A rate of 96% leaves a multiplier of 0.04; at 97%, it falls to 0.03, a 25% reduction. That is why the chart uses the underlying source values and rounds only the displayed results.

I would treat this as an illustration of how assumptions affect a ranking. I would not select a production model from this score alone. A useful next test would measure quality, serious errors, cost, and complete task time on the same workload, then check whether the ordering survives changes in weights and reasoning effort.

How I would use this when choosing a model

The practical decision starts with the work:

1. Define an acceptable result. Use representative tasks and acceptance criteria that a reviewer can apply consistently.
2. Set the limits. Decide how much delay and what kinds of errors the application can tolerate.
3. Measure the whole bill. Include failed attempts, retries, tools, and human verification. Report the completion rate alongside cost per accepted result.
4. Compare the qualifying options. A low price is useful only if the system meets the requirements.
5. Recheck when something changes. Model versions, prices, reasoning settings, and the workload can all affect the decision.

There is no universal exchange rate between benchmark points and business value. An extra point may matter greatly if it lets a system pass a necessary acceptance test. Elsewhere, the same gain may make no practical difference.

That is why intelligence per dollar deserves attention alongside the leaderboard. Better models expand what we can attempt. Lower costs, fewer consequential mistakes, and shorter waits determine how much of that work we can afford to do.

Sources

• Artificial Analysis: model measurements and Intelligence Index v4.3.
• AA-Omniscience: benchmark paper and benchmark introduction.
• Epoch AI: LLM inference price trends and underlying dataset, licensed under CC BY 4.0.