Why AI Benchmarks Do Not Predict Your Results

Last updated: October 2026

No. A benchmark is an average of tasks that are not yours, run on a corpus that is not yours. Twice this year I gave confident advice that had tested well for me, and twice it measurably backfired on somebody else's setup: once on model choice, once on document formatting. Benchmark your own workload instead. One afternoon, a fixed test set, one variable at a time.

The short version

• Results do not transfer between task types: the model that won a short one-shot generation lost a long build and then failed a third task outright, same two models, same method.
• Results do not transfer between corpora: the single biggest retrieval improvement I had measured cost a student almost 10% of his recall when he applied it to his own documents.
• Your own benchmark is cheap: a fixed test set, one variable changed at a time, and results tagged by category will tell you more in an afternoon than any leaderboard will.

Why does the cheaper model turn out more expensive on a longer job?

Because "cheaper" was a measurement of one task shape, and task shape is the variable that decides your bill.

I have now run the same two models against three genuinely different jobs. On a short one-shot generation, the pricier model per token used roughly a ninth of the tokens, finished in about half the time, and produced output two people could not tell apart from the alternative. On a long build from a written specification, the ranking flipped: slightly faster, considerably more expensive, and worse output. I wrote that pair up in detail in 5 things Claude Code pricing does not tell you, and at the time I thought two data points was enough to be careful.

Then the third test landed and it was not close. The job was transcript analysis, translation, and producing a plan. Starting from 20% of a usage allowance already consumed, the model I had once recommended burned the entire remaining four fifths in about twenty minutes and produced no output file at all. The same task handed to the other model finished properly for roughly 5% of the allowance.

Three tests, three task shapes, and only one of them made that model look good. The workflow I now use, and which I got from a working developer rather than from a benchmark, is to let it do the deliberating and hand the result to something else to build with. It thinks well. It finishes poorly.

Here is the part that makes this more than an anecdote, because there is a documented mechanism. Per published rates that model is priced at exactly double the other on both input and output, $10 per million input and $50 per million output against $5 and $25, all on Anthropic's pricing docs. Yet the vendor's own position is that it reduces costs by around 25% for typical workloads and up to 45% for highly agentic work. Both statements are true, and the reconciliation is the whole lesson of this post.

The savings come almost entirely from cached reads, which fell by 75% to $0.25 per million tokens. That model prices a cache hit at 0.025 times its input rate where the rest of the family uses 0.1 times, so the most expensive model in the line has the cheapest cache read in the line. Those savings only materialise in long agent loops that re-read the same cached context on every step. All three of my tests were one-shot jobs that never re-read a large cached context, so I paid the expensive half of that pricing structure and received none of the cheap half.

That is not a flaw in my testing. It is the correct answer for my workload and the wrong answer for somebody running agent loops all day. One honest caveat while I am here: I never controlled for the reasoning effort setting across those three runs, and reporting is explicit that effort is a dominant variable in what these models cost. Some of what I measured may be a configuration difference rather than a model difference.

Why did the best change I ever measured make a student's results worse?

This is the one that actually taught me something, because I was the source of the advice.

I had run a batch of retrieval experiments on a demo knowledge base and the largest single improvement did not come from tuning chunk sizes. It came from splitting documents on their markdown structure, so each chunk broke at a heading or a section boundary and arrived as a coherent unit rather than an arbitrary slice of characters. The embedding model never changed. The place the cuts landed did. Best result across accuracy, completeness and relevance, by a clear margin.

So I told a student to convert his corpus to markdown, and called it the biggest lever available to him. He did it. The conversion took about forty minutes.

His recall fell by almost 10% and his mean reciprocal rank by almost 7%. His own system reported three mechanisms. The conversion produced 186 chunks where the previous path produced 136, because blank lines and headings created extra boundaries and split context that used to sit together. It reported a fragmentation tax of around 3.5% across the corpus. And the serious one: zero of those 186 chunks carried a page reference, against 136 of 136 before, because the page numbers lived in comments that did not survive extraction.

That third finding is the one worth carrying away. His citation layer depended on page anchors to point a reader back at the source document. The conversion deleted them silently while the retrieval numbers still looked plausible, which is exactly the kind of failure nobody catches until somebody clicks a citation and lands nowhere.

The two results are not in conflict once you separate what each one tested. My experiment asked whether structural splitting beats arbitrary splitting on content that was already structured. His experiment asked whether changing your input format is free when every setting downstream was tuned around the old format. It is not. His pipeline had shaped itself around PDFs over months: the chunk sizing, the retrieval configuration, the citation layer. Swapping the input disturbed all of it.

His own system also flagged, correctly, that the comparison was not controlled. It tested markdown against his existing PDF pipeline, not against the best possible PDF handling. If you are new to this whole area, what RAG actually is in plain English will make the mechanics above easier to picture.

What can a public benchmark still tell you?

It can rank general capability, and that is genuinely useful for deciding what to shortlist. Fixed task sets like SWE-bench exist precisely so that claims can be compared on identical work, and a model that does badly there is unlikely to surprise you on your own code.

What a benchmark cannot do is carry a recommendation across a boundary it never tested. Both of my reversals are the same failure in different clothes:

Notice that neither original finding was wrong. Both were correct measurements of the thing actually measured. What failed was the leap from "this was true here" to "do this". That leap is what every leaderboard, every vendor comparison and every confident blog post is quietly asking you to make, mine included.

How do you benchmark your own workload?

Build a small fixed test set from real examples of your own work, and never change it once you start. Twenty or thirty cases is enough to be informative. If the set moves while you are testing, you have no comparison, only impressions.

Then change one variable at a time and record the result. One model swapped, or one setting, or one format, never two. This is tedious and it is the entire method. It is also why notebooks and scripted runs beat doing it by hand: when a comparison costs you two minutes you will actually run it, and when it costs an hour you will guess instead.

Tag your results by category rather than reading one aggregate number. A single score of 78% tells you nothing you can act on. The same 78% broken out by question type, with synthesis questions failing and lookups passing, tells you exactly where to work. That reframing is the difference between scoring yourself and diagnosing yourself.

And check what each change destroys, not only what it improves. Page numbers, section identifiers and document positions are citation infrastructure. Verify that a tool split your documents rather than paraphrasing them, by reading the tail of one chunk and finding that exact text in the source. It takes ten seconds and almost nobody does it. For the hands-on version of all of this, build your first RAG app walks the pipeline end to end, and the same discipline applied to a subscription decision is the whole method behind deciding whether Claude Max is worth it in a week.

Frequently Asked Questions

Do AI benchmarks matter at all?

Yes, for shortlisting. They tell you which tools are in the conversation and which are not. They cannot tell you which of two credible options will be better on your data, your task length and your existing pipeline, because none of those were in the test.

How many tests do I need before I trust a result?

More than one, and the second one should be a different shape of task rather than a repeat. My model ranking flipped between test one and test two and then collapsed on test three. A single careful test is a single sample, no matter how well you ran it.

What should I measure if I am not building a retrieval system?

Total cost and wall-clock time per finished piece of work, plus whether you had to redo it. Cost per token is the wrong unit because verbosity is the variable that moves your bill. A model with identical published rates can cost several times more on the same job.

Does my own benchmark have to be scientific?

It has to be consistent, which is a lower bar. Same test cases, one variable changed at a time, results written down. Most of the value comes from the writing down, because it stops you re-running an experiment you already did and forgetting the outcome.

How often should I re-run it?

When you change models, when you change a pipeline component, and when a vendor announces something that sounds like it applies to you. Keeping the test set fixed is what makes those re-runs cheap enough to bother with.

Someone published a benchmark that contradicts my results. Who is wrong?

Probably neither of you. Check what workload each number describes before you assume a conflict. The cost disagreement in this post resolved entirely once I worked out which pricing mechanism applied to my task shape and which did not.

Ready to find out what your own numbers say?

Most people skip this because it sounds like research, and it is closer to an afternoon of bookkeeping. If you want help building a test set for your actual work, deciding which variable to move first, and reading the results without fooling yourself, book a free 15-minute Discovery Call and bring the decision you are stuck on.

Written by Michael Murr for AI Tutor Code: private 1-on-1 online tutoring for professionals learning Python, AI tools, Data Science, ML, and LLM engineering. 200+ students taught, 3,000+ hours delivered.