Your AI Is Right 90% of the Time. Your Build Is Not

Last updated: September 2026

A student of mine wanted to strip the approval gates out of an autonomous build system. His argument was reasonable and I have heard versions of it from four people this year: the model's suggestions are right about 90% of the time, the gates slow everything down, so why sit there clicking approve. Then we did the arithmetic together on a shared screen, and the number that came back ended the discussion in about ninety seconds. Five patterns below, in the order they usually show up, all of them from August session notes and my own builds.

In this post: the arithmetic nobody runs · why 90 to 95 is a bigger jump than it looks · it finished and it was wrong · small projects are not exempt · gates where they compound

1. The arithmetic nobody runs

So how accurate is AI generated code? Per decision, good models land somewhere around 90 to 95%. Across a whole build, far lower, because a build is a chain of decisions and chains multiply. Ten decisions at 90% each finish clean about 35% of the time.

Here is the thing that gets missed. "Right 90% of the time" is a per-decision number. What you actually care about is whether the whole run is right, and a run is not one decision. It is a chain of them, each one built on the last. Chains multiply.

If every decision is independent and each is right 90% of the time, the odds that all of them are right is 0.9 raised to the number of decisions. That is the whole model. It is one line of arithmetic and almost nobody does it before hitting go.

Decisions in the chain	At 90% each	At 95% each
5	59.0%	77.4%
10	34.9%	59.9%
15	20.6%	46.3%
20	12.2%	35.8%
30	4.2%	21.5%

Ten decisions at 90% is roughly 35%. Not 90%, and not the 10% people sometimes guess when they overcorrect. About one run in three comes out clean. You need around 22 sequential decisions before you drop under 10%.

I want to be precise here because I have seen this number quoted badly in both directions. Thirty-five percent is not a catastrophe and it is not fine. It is the honest answer to "can I let this run unattended", and the honest answer is no, not without gates.

Two objections come up every time I show this table, and both deserve a straight answer.

The first is that the decisions are not independent, so the multiplication is wrong. Partly true. Real chains have correlated errors, and correlation cuts both ways: a model that understands your codebase well is more likely to be right at every step, which pushes the real number above the table, while a model that misread your intent at step one is close to guaranteed to be wrong at step nine, which pushes it below. What correlation does not do is rescue you. It changes the shape of the distribution, not the direction of travel.

The second objection is that not every wrong decision matters. Also true, and it is the more useful objection. A wrong choice of variable name does not propagate. A wrong choice of data structure propagates into everything built on top of it. That is not an argument that the arithmetic is wrong. It is an argument about which decisions belong in the count, which is exactly where pattern 5 ends up.

The hardest part in practice is counting honestly. People think of a task as one decision because they typed one prompt. Ask what the model actually chose along the way: the file layout, the library, the schema, the error handling, the naming, the test strategy, what to do at each ambiguity your prompt did not cover. That is not one decision. In my experience a single paragraph of instruction routinely contains fifteen to twenty implicit ones.

The fix: before any unattended run, count the decisions in the chain and raise 0.9 to that power. If the number frightens you, that is the number, not a pessimist's version of it.

2. Why 90 to 95 is a bigger jump than it looks

Five percentage points sounds like a rounding difference. On a single decision it is. Across a chain it is the difference between a workflow you can use and one you cannot.

Look along the twenty-decision row above. At 90% you are at 12.2%. At 95% you are at 35.8%. Same chain, five points of per-step accuracy, and roughly three times the chance of a clean run. That is why "use the better model for this step" is not a luxury purchase, and why one sloppy step early poisons everything after it.

Run your own numbers rather than trusting mine:

# Odds a chain of N decisions is entirely correct, at a given per-step accuracy.
for accuracy in (0.90, 0.95, 0.99):
    for steps in (5, 10, 20, 50):
        print(f"{accuracy:.0%} x {steps:2d} steps -> {accuracy ** steps:.1%}")

90% x  5 steps -> 59.0%
90% x 10 steps -> 34.9%
90% x 20 steps -> 12.2%
90% x 50 steps -> 0.5%
95% x  5 steps -> 77.4%
95% x 10 steps -> 59.9%
95% x 20 steps -> 35.8%
95% x 50 steps -> 7.7%
99% x  5 steps -> 95.1%
99% x 10 steps -> 90.4%
99% x 20 steps -> 81.8%
99% x 50 steps -> 60.5%

Notice the bottom block. Even at 99% per step, a fifty-step chain is a coin flip. There is no per-step accuracy high enough to make an arbitrarily long unattended chain safe. That is a structural property of multiplication, not a criticism of any particular model.

This is where model choice stops being about taste. Most people pick one model and run everything on it, which means they are paying top rates for the trivial steps and accepting mid-tier accuracy on the load-bearing ones. The compounding table argues for the opposite: spend on the decisions with the most downstream weight, economise everywhere else. Five points of accuracy on the schema decision is worth more than five points on forty cosmetic ones, because the schema is upstream of all forty.

There is a second lever people forget, and it is usually cheaper than a better model. Per-step accuracy is not a fixed property of the model, it is a property of the model plus what you gave it. A model working with your conventions loaded and current documentation available is measurably more likely to be right per step than the same model guessing from memory, which is the entire argument for loading your skills before you build anything. Raising per-step accuracy from 90 to 95 by fixing the context is the same win as buying a better model, and it usually costs nothing.

The fix: stop shopping for a model good enough to run unsupervised forever. Shorten the chain, break it, or raise per-step accuracy by fixing what the model has to work with.

3. It finished and it was wrong

This is the failure mode that costs people the most time, because nothing announces it.

During an unattended run on one of my own projects, the model generated a loading animation: a little car, wheels turning, the sort of thing you glance at for half a second. It ran to completion. No errors, no warnings, no failed build. The car had its brakes mounted in the wrong place.

Nobody died. It was a loading animation. But sit with what that tells you about the run it came from. Every gate had been removed so the thing could work through the night, and the output looked finished. If I had not actually looked at the car, that detail ships, and so does whatever else in that chain came out subtly wrong and never got looked at.

Worth saying plainly: the tools ship these gates on purpose. Claude Code's permission modes include a plan mode in which "Claude reads files and runs read-only shell commands to explore but doesn't edit your source files". That is a gate placed before the chain starts rather than an apology after it ends. Turning it off is a choice, and it is the kind of choice people make once and then never revisit.

This is the same shape as the skills failure I described earlier, where two students built for weeks with nothing loaded and the output looked perfectly reasonable the whole time. A build that errors is telling you the truth. A build that completes is only telling you it completed.

Compounding accuracy and invisible failure are the same problem wearing different clothes. The arithmetic says roughly two runs in three have something wrong in them. Silent failure says you will not be told which two.

There is a reason this hits people who are doing everything else right. If your tests pass, your build is green and your linter is quiet, you have confirmed that the code satisfies the rules you thought to write down in advance. None of those tools has an opinion about whether the brakes belong on that part of the car. Automated checks catch the failures you predicted. The compounding ones are, almost by definition, the failures nobody predicted, because a failure anyone predicted would have had a gate on it.

The practical version of this is that reviewing an unattended run is a different activity from reviewing your own work. When you write something yourself you carry a mental list of the bits you were unsure about, and you check those. After an unattended run that list does not exist. You have to construct it deliberately, by asking what the model must have decided in order to produce this, and then going and looking at those specific things.

The fix: treat "it ran to completion" as no evidence at all. Write down the three outputs that matter most before you start the run, and look at exactly those with your own eyes when it finishes.

4. Small projects are not exempt

The reflex is to assume this only bites on ambitious builds. It does not.

I have a Pomodoro timer. It is about as simple as a real project gets: start a clock, stop a clock, write a row. I tested it deliberately for a full week and it was still surfacing silent failures at the end of that week. Not crashes. Behaviour that looked correct until you checked it against what actually happened.

What it looked like	What was actually happening
A session vanished	It was never banked in the first place
The timer drifted	It legitimately spans its own pauses
Duplicate rows	One task switch split a session into two rows that sum correctly

Every one of those looked like a bug and only one of them was. That cuts both ways: some of what looks broken is correct, and some of what looks correct is broken. Either way you only find out by querying the data rather than reading the code and forming a theory.

That last point is the one I now teach hardest. The instinct when something looks wrong is to open the file that probably caused it and read until you spot the mistake. It feels productive and it is close to useless, because you are asking your own assumptions to audit themselves. Query the data first. Ask what actually got written, in what order, with what values, and let the answer tell you whether there is a bug at all. On that timer, several reported bugs turned out to be correct behaviour I had misunderstood, and I would never have discovered that by rereading the code.

Why does a project this small still misbehave after a week of testing? Because simplicity reduces the number of decisions, it does not change the arithmetic. A short chain at 90% is still a chain. Ten decisions is ten decisions whether they are about a timer or a trading system, and a week of casual use is not the same as ten deliberate checks against the data.

Most of what I do in 1-on-1 sessions is sit with someone while they check the thing they assumed was fine. It is unglamorous and it is where the bugs are.

The fix: budget verification time in proportion to how long the thing ran unattended, not in proportion to how complicated you think it is.

5. Gates where they compound

Here is the part that makes this useful rather than just a dunk on autonomy.

The multiplication only runs while nothing interrupts it. Verify a decision and confirm it is right, and that term stops being 0.9 and becomes roughly 1. The chain restarts from that point. You do not need a gate on every decision. You need gates on the decisions everything downstream depends on.

Anthropic's own engineering guidance on building agents says the same thing from the other direction, noting that the autonomous nature of agents brings "the potential for compounding errors" and recommending that agents "pause for human feedback at checkpoints or when encountering blockers." The arithmetic above is just that sentence with numbers attached.

Watch what a single well-placed gate does to a twenty-decision chain:

# One verified checkpoint splits a 20-step chain into two 10-step chains.
ungated = 0.9 ** 20
gated = (0.9 ** 10) ** 2  # same chain, but the midpoint is confirmed correct
print(f"no gate:  {ungated:.1%}")
print(f"one gate: {0.9 ** 10:.1%} to reach the checkpoint, then {0.9 ** 10:.1%} again")
print(f"and if the checkpoint is actually fixed when wrong: {0.9 ** 10:.1%} per half")

Reaching a verified midpoint is a ten-decision problem at 34.9%, and if you fix what is wrong when you get there, the second half is a fresh ten-decision problem rather than a continuation of a decaying one. You have turned one 12.2% run into two 34.9% runs with a repair step in between. That is the entire technique.

The decisions worth gating are the ones with the most downstream dependents: the schema, the data model, the core abstraction, the interface every later step calls. Gate the foundation, let the model run on the leaves. A handoff file the next session can read makes those checkpoints durable, so a gate you passed on Tuesday is still a gate on Thursday.

There is a failure mode on this side too, and I have watched people walk straight into it after seeing the table. They gate everything. Every step gets an approval prompt, the run takes four times as long, and after twenty clicks of approve the person is rubber-stamping without reading. A gate nobody genuinely evaluates is worse than no gate, because it produces the paperwork of verification and none of the substance, and it converts the arithmetic from something you respect into something you have learned to click past.

Three or four real gates beat twenty ceremonial ones. The test for whether a gate is real is simple: if you cannot say what you would be looking for when you reach it, it is not a gate, it is a speed bump. Write down the question before the run. "Is the schema right" is a gate. "Does this look ok" is not.

The other placement rule worth knowing is that gates are worth more early than late. A checkpoint at step two protects the eighteen decisions after it. The same checkpoint at step eighteen protects two. Most people put their attention at the end, where the output is, because that is where the result appears. The leverage is at the front, where the assumptions are.

The fix: find the three decisions that everything else is built on top of, put a real gate on exactly those, write down what you are checking for before the run starts, and leave the rest alone.

Start here

If you do one thing from this post, do the arithmetic on a workflow you already run unattended. Not a hypothetical one. The one you set going last week and did not watch.

Count the decisions in the chain, honestly. Most people undercount by half.
Raise your best guess at per-step accuracy to that power, using the script in pattern 2.
Find the earliest decision that most of the others depend on, and put a gate there before you run it again.

That takes about ten minutes and it usually changes how someone runs their agents permanently. If you want to go deeper on what these systems actually are before you start gating them, AI agents for beginners is the ground floor, and the mistakes professionals make with AI tools covers the habits that put people here in the first place. If you would rather do it with someone watching, that is what a Discovery Call is for.

From the session logs of AI Tutor Code: 1-on-1 Python and AI tutoring for working professionals. The patterns above come from real students and real builds, lightly anonymized.