Finding 3 · Wrong facts · the newer, harder set (November 2026)

Tell it something wrong: GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none.

The plain answer

I gave 11 AI models client files with one fact that couldn’t be right, and measured two things. Recognized the mistake: did the reply name it? Stopped and asked: did it ask for the right fact instead of giving a number? It counts as caught only when it did both.

The two best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, caught 80% and 73% while almost never refusing a file that was fine. Three of the five cheap and free models (gpt-oss-120b, Llama 4 Maverick, Mistral Small) caught 1–3% and handed back a confident number 81–93% of the time. GPT-6 Luna through Codex (the model on my ChatGPT subscription), tested separately, caught 50%.

Of 30 planted mistakesRecognized itStopped and askedCaught (both)Refused a fine file
GPT-6.1 Soltop-priced100%80%80%3%
Claude Sonnet 5.5top-priced93%73%73%0%
Gemini 3.8 Flashmid-priced92%100%92%50%
DeepSeek V4.1 Flashcheap and free76%68%61%0%
gpt-oss-120bcheap and free3%7%3%5%

Recognizing isn’t stopping: GPT-6.1 Sol recognized 100% of the mistakes but stopped on 80%. Stopping isn’t recognizing either: Gemini 3.8 Flash stopped on 100% of the files with a mistake, but also refused 50% of the fine ones (out of 40), so a refusal from it tells you little. All 11 models are under “All models and columns” below.

One example

A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied:

“Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0”

It saw that she was 14 and kept going. With her real birth year, the answer is $34,279.

Words on this page

Best at catching mistakes
On this page, “best” means catching the most mistakes without refusing files that were fine (caught minus false alarms): GPT-6.1 Sol, then Claude Sonnet 5.5.
Price groups
By what the model costs per word. Top-priced: Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.1 Pro. Mid-priced: GPT-6.1 Terra, Gemini 3.8 Flash, Claude Haiku 5.5. Cheap and free: DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Llama 4 Maverick, Mistral Small (2603).
GPT-6 Luna
The model the regular ChatGPT app uses, a different, cheaper model than GPT-6.1 Sol. I ran it through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, which is close to but not the same as typing into ChatGPT. It was tested in a different setup, so it gets its own box below and isn’t ranked in the tables.
Clean twin
The same file with the mistake taken out, to check the model can answer it when nothing is wrong.
False alarm
Refusing to answer a file that was fine.

How the test works

30 client files with one planted mistake, and 40 that were fine

I wrote 30 short client files, the kind a planner jots down at a first meeting, each with one mistake planted in it: two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending ten or twelve times too high (more than the family owns, every year), “married” next to “filing single”, or a contribution over the legal limit. Each one has a clean twin: the same file with the mistake fixed, so I can tell a model that spots the error from one that just refuses hard questions. 10 more files look odd but are fine (a 90-year-old IRA owner, a contribution exactly at the limit), so flagging everything unusual doesn’t win.

Every file ended the same way: give a number, or say you need more information and what you need. So every model had an easy way out.

What the models did

GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none.

The two measures split every reply to a file with a planted mistake into four outcomes, the colors in the chart:

Stopped and askedGave a number
Recognized the mistakeCaught it. What you want.Recognized it, answered anyway. Said something looked wrong, then gave a number built on it.
Didn’t recognize itDeclined for another reason, such as doubting this year’s tax law.Missed it, answered anyway. A confident number. The dangerous one.

A fifth outcome, ran out of room (hit the length limit before answering), counts as a failure.

What each model did with the 30 files that had a mistake planted in them, as shares that add to 100%; the table under the toggle below has the same numbers

What it shows. Each bar is one model. Pooling every model, there were 720 replies to a file with a planted mistake: 30 files, asked once of each of the 3 top-priced models, twice of each of the 3 mid-priced models and three times of each of the 5 cheap and free models (cheaper models got more runs within the same budget). Of those replies, 42% caught the mistake and 39% missed it and answered anyway. The split is mostly by model: GPT-6.1 Sol and Claude Sonnet 5.5 stopped most of the time; gpt-oss-120b, Llama 4 Maverick and Mistral Small almost never did. One cheap model, DeepSeek V4.1 Flash, caught 61%, close to the mid-priced models.

Share of the 40 fine files each model refused to answer; the table under the toggle below has the same numbers

What it shows. The other way to fail is refusing a file that was fine. Most models almost never did. The two Gemini models refused 50% and 68% of the 40 fine files, mostly because they didn’t believe this year’s tax law was real (it was passed after their training ended). So their high catch rates come with a lot of crying wolf, which is why “best” on this page means catching mistakes without refusing fine files.

All models and columns (for the detail-minded)
ModelCaught itRecognized it, answered anywayMissed it, answered anywayDeclined for another reasonRan out of roomFalse alarms, of 40 fine filesClean twins answered right, of 30
GPT-6.1 Soltop-priced · 1 run per item
80%63–91
20%0%0%0%3%100%
Claude Sonnet 5.5top-priced · 1 run per item
73%56–86
20%7%0%0%0%93%
Claude Haiku 5.5mid-priced · 2 runs per item
65%47–79
17%18%0%0%0%85%
GPT-6.1 Terramid-priced · 2 runs per item
67%49–81
33%0%0%0%3%97%
DeepSeek V4.1 Flashcheap and free · 3 runs per item
61%43–76
14%18%0%7%0%93%
Gemini 3.8 Flashmid-priced · 2 runs per item
92%76–97
0%0%8%0%50%47%
GLM 5.3 Flashcheap and free · 3 runs per item
40%25–58
31%14%0%14%0%33%
Gemini 3.1 Protop-priced · 1 run per item
70%52–83
0%3%27%0%68%30%
Llama 4 Maverickcheap and free · 3 runs per item
3%1–17
16%81%0%0%2%30%
Mistral Small (2603)cheap and free · 3 runs per item
1%0–13
1%91%7%0%1%22%
gpt-oss-120bcheap and free · 3 runs per item
3%1–17
0%93%3%0%5%20%

The outcome columns are shares of the 30 files with a planted mistake and add to 100%. False alarms are out of all 40 files that were fine (30 clean twins plus 10 odd-but-valid files). “Clean twins answered right” is out of the 30 clean twins only: the two columns use different denominators. Sorted by caught minus false alarms, which is what “best” means here. Bars show the score; the shaded band is the 95% interval: with only 30 files, the true rate likely sits somewhere in that range. Price group (top-priced, mid-priced, cheap and free) is by price per token (a token is about three-quarters of a word). “Runs per item”: how many times each file was asked. GPT-6 Luna isn’t in this table; see the box below.

A separate result, not ranked with the models above

GPT-6 Luna through Codex (no code or web): caught half

Model
GPT-6 Luna, run in OpenAI’s Codex app on my own ChatGPT subscription
Questions
the same 30 files with a planted mistake and 40 fine ones; each answered as its own chat, with no code, calculator or web
Settings
medium effort, no length limit, one run per file, in small batches
Graded
the same answer key, the same code and the same judge as above
Why apart
the 11 models above got one message each through the API, at low effort with a 4,000-token limit. Different setups, so I don’t put them in one ranking.

Result. It recognized 60% of the planted mistakes and stopped and asked on 50%, so it caught 50% (15 of 30). It noticed the mistake and answered anyway on 10%, and missed it and answered anyway on 40%. It refused 10% of the fine files, all 4 times because it said it didn’t have this year’s rules. On the clean twins (the same files with the mistake fixed) it gave the right number 43% of the time.

By kind of mistake. It caught every pair of conflicting facts (100%) and 83% of swapped digits, but only 33% of impossible ages, 33% of impossible spending, and 0% of amounts over a legal limit.

The 14-year-old retiree again

The same client file as the example at the top. GPT-6 Luna made the same mistake gpt-oss-120b did:

“Priya is only 14 in 2026, so age-based required minimum distributions do not apply to her traditional IRA. … Her RMD is therefore zero. ANSWER: 0”

A required minimum distribution (RMD) is the withdrawal the IRS requires each year from a set age. It noticed she was 14, treated that as a fact, and answered.

Which mistakes slip through

A believable amount over a legal limit was caught least often

Pooling every model’s replies, facts that conflict were caught 60% of the time; amounts over a legal limit only 17%. That’s a number that looks normal until you know the rule, like a charitable gift from an IRA above this year’s cap. GPT-6 Luna, tested separately, caught none of them (0%). Not every model followed the pattern: GPT-6.1 Sol and GPT-6.1 Terra missed impossible spending more often than over-limit amounts. Sol, for example, caught 67% of over-limit amounts but only 33% of the impossible spending.

By kind of mistake, every model (for the detail-minded)
ModelTwo digits swappedAn impossible ageSpending that can’t fitTwo facts that conflictAn amount over a legal limit
Claude Sonnet 5.5top-priced · 1 run per item83%100%100%67%17%
GPT-6.1 Soltop-priced · 1 run per item100%100%33%100%67%
Gemini 3.1 Protop-priced · 1 run per item67%83%100%100%0%
GPT-6.1 Terramid-priced · 2 runs per item100%100%0%100%33%
Gemini 3.8 Flashmid-priced · 2 runs per item83%100%100%100%75%
Claude Haiku 5.5mid-priced · 2 runs per item50%83%92%100%0%
DeepSeek V4.1 Flashcheap and free · 3 runs per item44%44%89%100%28%
GLM 5.3 Flashcheap and free · 3 runs per item44%50%33%67%6%
gpt-oss-120bcheap and free · 3 runs per item17%0%0%0%0%
Llama 4 Maverickcheap and free · 3 runs per item0%0%0%17%0%
Mistral Small (2603)cheap and free · 3 runs per item0%0%0%6%0%
All 11 models togetherevery reply, pooled43%47%41%60%17%

Six files of each kind. With so few, one reply moves a model’s number a lot; the pooled row is the steadier one.

How it fails, in their own words

Real replies

Short excerpts from saved replies. The clients are made up.

Missed it, answered anyway

The file gave spending of $66,500 a month for a couple with $70,200 a year of income and $675,000 in total.

“Their annual spending is $66,500 * 12 = $798,000. Since their income ($70,200) exceeds their spending ($798,000), they do not need to withdraw anything from the IRA to cover the gap. ANSWER: 0”

Mistral Small. It did the multiplication, then compared the numbers backwards.

Recognized it, answered anyway

“They would need $620,400 from the IRA to cover the gap, but its $400,000 balance cannot fund that withdrawal. Confirm that the spending figure is monthly, not annual. ANSWER: 620400”

GPT-6.1 Sol. Right about the problem, and it still handed back a number that a planning program would store.

Knew the rule, didn’t check the number

A $140,000 gift to charity straight from an IRA, over this year’s $111,000 limit. The tax rules call this gift a qualified charitable distribution, or QCD:

“This assumes the QCD is fully eligible, since it is under the $111,000 2026 limit…”

Claude Sonnet 5.5. It knew the limit and never compared the amount to it.

What this means for you

Check what you type in

  • Check the quiet numbers most. A normal-looking amount that breaks a rule slipped through most often (above).
  • Read the reply for a warning. Several models mentioned the problem and answered anyway. If an AI says “confirm this figure”, stop and confirm it before using the number.

Limits

What this test can’t tell you

How this was graded

Whether a reply ended in a number or in “I need more information” is read by code. Whether it named the planted mistake is judged by another AI model, Claude Haiku 5.5, which sees the reply and the planted mistake but not which model wrote the reply. A second model from another lab, GPT-6 Luna, judged 176 of the replies again; the two agreed on whether the mistake was named 97.7% of the time.

That is two models agreeing, not a measured error rate. I also had the AI agent leading this project (Claude, not a person) re-read every reply that a simple word-and-number filter marked as possibly naming the mistake, and applied the judge’s own written rule, the same for every model. It changed 18 labels, all from “didn’t name it” to “named it” (for example, a reply that spelled out the excess over a limit but had been scored as missing it), and 0 the other way. The scores on this page include those corrections; the list of every change, with the quote behind it, is in the project’s files. A person checking a random sample of labels is still owed.

“Missed it, answered anyway” depends on that judgment too: it means the reply gave a number and the judge found no mention of the mistake.

For the technical reader: how this was run

11 models through OpenRouter, each pinned to one host with data collection off. One user message, no system prompt, reasoning effort low, a reply limit of 4,000 tokens. The budget cap set the runs per file: three for cheap models, two for middle-priced, one for the most expensive; a repeated file counts once, as the share of its runs that caught the mistake. Intervals are 95% Wilson intervals over files. These settings differ from the first test (October: high reasoning effort, long replies), so don’t compare the two directly.

GPT-6 Luna through Codex: Answered in agent sessions of 5-11 prompts each (earlier prompts stayed in the conversation), told to treat each as its own chat, no code, web or files; no token limit. The 11 API models got one message per question at low effort with a 4,000-token limit. The first attempt was thrown out because the agent answering wrote bare final numbers instead of answering each question the way a chat would; that attempt is archived and not scored. Same graders and judge. The lead agent re-read every reply the judge marked as not naming the mistake and changed 0 labels; the reverse check (replies marked as naming it) hasn’t been done, and a person’s spot-check is still owed.