Finding 3 · Wrong facts · the newer, harder set (November 2026)
Tell it something wrong: GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none.
The plain answer
I gave 11 AI models client files with one fact that couldn’t be right, and measured two things. Recognized the mistake: did the reply name it? Stopped and asked: did it ask for the right fact instead of giving a number? It counts as caught only when it did both.
The two best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, caught 80% and 73% while almost never refusing a file that was fine. Three of the five cheap and free models (gpt-oss-120b, Llama 4 Maverick, Mistral Small) caught 1–3% and handed back a confident number 81–93% of the time. GPT-6 Luna through Codex (the model on my ChatGPT subscription), tested separately, caught 50%.
| Of 30 planted mistakes | Recognized it | Stopped and asked | Caught (both) | Refused a fine file |
|---|---|---|---|---|
| GPT-6.1 Soltop-priced | 100% | 80% | 80% | 3% |
| Claude Sonnet 5.5top-priced | 93% | 73% | 73% | 0% |
| Gemini 3.8 Flashmid-priced | 92% | 100% | 92% | 50% |
| DeepSeek V4.1 Flashcheap and free | 76% | 68% | 61% | 0% |
| gpt-oss-120bcheap and free | 3% | 7% | 3% | 5% |
Recognizing isn’t stopping: GPT-6.1 Sol recognized 100% of the mistakes but stopped on 80%. Stopping isn’t recognizing either: Gemini 3.8 Flash stopped on 100% of the files with a mistake, but also refused 50% of the fine ones (out of 40), so a refusal from it tells you little. All 11 models are under “All models and columns” below.
One example
A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied:
“Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0”
It saw that she was 14 and kept going. With her real birth year, the answer is $34,279.
Words on this page
- Best at catching mistakes
- On this page, “best” means catching the most mistakes without refusing files that were fine (caught minus false alarms): GPT-6.1 Sol, then Claude Sonnet 5.5.
- Price groups
- By what the model costs per word. Top-priced: Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.1 Pro. Mid-priced: GPT-6.1 Terra, Gemini 3.8 Flash, Claude Haiku 5.5. Cheap and free: DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Llama 4 Maverick, Mistral Small (2603).
- GPT-6 Luna
- The model the regular ChatGPT app uses, a different, cheaper model than GPT-6.1 Sol. I ran it through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, which is close to but not the same as typing into ChatGPT. It was tested in a different setup, so it gets its own box below and isn’t ranked in the tables.
- Clean twin
- The same file with the mistake taken out, to check the model can answer it when nothing is wrong.
- False alarm
- Refusing to answer a file that was fine.
How the test works
30 client files with one planted mistake, and 40 that were fine
I wrote 30 short client files, the kind a planner jots down at a first meeting, each with one mistake planted in it: two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending ten or twelve times too high (more than the family owns, every year), “married” next to “filing single”, or a contribution over the legal limit. Each one has a clean twin: the same file with the mistake fixed, so I can tell a model that spots the error from one that just refuses hard questions. 10 more files look odd but are fine (a 90-year-old IRA owner, a contribution exactly at the limit), so flagging everything unusual doesn’t win.
Every file ended the same way: give a number, or say you need more information and what you need. So every model had an easy way out.
What the models did
GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none.
The two measures split every reply to a file with a planted mistake into four outcomes, the colors in the chart:
| Stopped and asked | Gave a number | |
|---|---|---|
| Recognized the mistake | Caught it. What you want. | Recognized it, answered anyway. Said something looked wrong, then gave a number built on it. |
| Didn’t recognize it | Declined for another reason, such as doubting this year’s tax law. | Missed it, answered anyway. A confident number. The dangerous one. |
A fifth outcome, ran out of room (hit the length limit before answering), counts as a failure.
What it shows. Each bar is one model. Pooling every model, there were 720 replies to a file with a planted mistake: 30 files, asked once of each of the 3 top-priced models, twice of each of the 3 mid-priced models and three times of each of the 5 cheap and free models (cheaper models got more runs within the same budget). Of those replies, 42% caught the mistake and 39% missed it and answered anyway. The split is mostly by model: GPT-6.1 Sol and Claude Sonnet 5.5 stopped most of the time; gpt-oss-120b, Llama 4 Maverick and Mistral Small almost never did. One cheap model, DeepSeek V4.1 Flash, caught 61%, close to the mid-priced models.
What it shows. The other way to fail is refusing a file that was fine. Most models almost never did. The two Gemini models refused 50% and 68% of the 40 fine files, mostly because they didn’t believe this year’s tax law was real (it was passed after their training ended). So their high catch rates come with a lot of crying wolf, which is why “best” on this page means catching mistakes without refusing fine files.
All models and columns (for the detail-minded)
| Model | Caught it | Recognized it, answered anyway | Missed it, answered anyway | Declined for another reason | Ran out of room | False alarms, of 40 fine files | Clean twins answered right, of 30 |
|---|---|---|---|---|---|---|---|
| GPT-6.1 Soltop-priced · 1 run per item | 80%63–91 | 20% | 0% | 0% | 0% | 3% | 100% |
| Claude Sonnet 5.5top-priced · 1 run per item | 73%56–86 | 20% | 7% | 0% | 0% | 0% | 93% |
| Claude Haiku 5.5mid-priced · 2 runs per item | 65%47–79 | 17% | 18% | 0% | 0% | 0% | 85% |
| GPT-6.1 Terramid-priced · 2 runs per item | 67%49–81 | 33% | 0% | 0% | 0% | 3% | 97% |
| DeepSeek V4.1 Flashcheap and free · 3 runs per item | 61%43–76 | 14% | 18% | 0% | 7% | 0% | 93% |
| Gemini 3.8 Flashmid-priced · 2 runs per item | 92%76–97 | 0% | 0% | 8% | 0% | 50% | 47% |
| GLM 5.3 Flashcheap and free · 3 runs per item | 40%25–58 | 31% | 14% | 0% | 14% | 0% | 33% |
| Gemini 3.1 Protop-priced · 1 run per item | 70%52–83 | 0% | 3% | 27% | 0% | 68% | 30% |
| Llama 4 Maverickcheap and free · 3 runs per item | 3%1–17 | 16% | 81% | 0% | 0% | 2% | 30% |
| Mistral Small (2603)cheap and free · 3 runs per item | 1%0–13 | 1% | 91% | 7% | 0% | 1% | 22% |
| gpt-oss-120bcheap and free · 3 runs per item | 3%1–17 | 0% | 93% | 3% | 0% | 5% | 20% |
The outcome columns are shares of the 30 files with a planted mistake and add to 100%. False alarms are out of all 40 files that were fine (30 clean twins plus 10 odd-but-valid files). “Clean twins answered right” is out of the 30 clean twins only: the two columns use different denominators. Sorted by caught minus false alarms, which is what “best” means here. Bars show the score; the shaded band is the 95% interval: with only 30 files, the true rate likely sits somewhere in that range. Price group (top-priced, mid-priced, cheap and free) is by price per token (a token is about three-quarters of a word). “Runs per item”: how many times each file was asked. GPT-6 Luna isn’t in this table; see the box below.
A separate result, not ranked with the models above
GPT-6 Luna through Codex (no code or web): caught half
- Model
- GPT-6 Luna, run in OpenAI’s Codex app on my own ChatGPT subscription
- Questions
- the same 30 files with a planted mistake and 40 fine ones; each answered as its own chat, with no code, calculator or web
- Settings
- medium effort, no length limit, one run per file, in small batches
- Graded
- the same answer key, the same code and the same judge as above
- Why apart
- the 11 models above got one message each through the API, at low effort with a 4,000-token limit. Different setups, so I don’t put them in one ranking.
Result. It recognized 60% of the planted mistakes and stopped and asked on 50%, so it caught 50% (15 of 30). It noticed the mistake and answered anyway on 10%, and missed it and answered anyway on 40%. It refused 10% of the fine files, all 4 times because it said it didn’t have this year’s rules. On the clean twins (the same files with the mistake fixed) it gave the right number 43% of the time.
By kind of mistake. It caught every pair of conflicting facts (100%) and 83% of swapped digits, but only 33% of impossible ages, 33% of impossible spending, and 0% of amounts over a legal limit.
The 14-year-old retiree again
The same client file as the example at the top. GPT-6 Luna made the same mistake gpt-oss-120b did:
“Priya is only 14 in 2026, so age-based required minimum distributions do not apply to her traditional IRA. … Her RMD is therefore zero. ANSWER: 0”
A required minimum distribution (RMD) is the withdrawal the IRS requires each year from a set age. It noticed she was 14, treated that as a fact, and answered.
Which mistakes slip through
A believable amount over a legal limit was caught least often
Pooling every model’s replies, facts that conflict were caught 60% of the time; amounts over a legal limit only 17%. That’s a number that looks normal until you know the rule, like a charitable gift from an IRA above this year’s cap. GPT-6 Luna, tested separately, caught none of them (0%). Not every model followed the pattern: GPT-6.1 Sol and GPT-6.1 Terra missed impossible spending more often than over-limit amounts. Sol, for example, caught 67% of over-limit amounts but only 33% of the impossible spending.
By kind of mistake, every model (for the detail-minded)
| Model | Two digits swapped | An impossible age | Spending that can’t fit | Two facts that conflict | An amount over a legal limit |
|---|---|---|---|---|---|
| Claude Sonnet 5.5top-priced · 1 run per item | 83% | 100% | 100% | 67% | 17% |
| GPT-6.1 Soltop-priced · 1 run per item | 100% | 100% | 33% | 100% | 67% |
| Gemini 3.1 Protop-priced · 1 run per item | 67% | 83% | 100% | 100% | 0% |
| GPT-6.1 Terramid-priced · 2 runs per item | 100% | 100% | 0% | 100% | 33% |
| Gemini 3.8 Flashmid-priced · 2 runs per item | 83% | 100% | 100% | 100% | 75% |
| Claude Haiku 5.5mid-priced · 2 runs per item | 50% | 83% | 92% | 100% | 0% |
| DeepSeek V4.1 Flashcheap and free · 3 runs per item | 44% | 44% | 89% | 100% | 28% |
| GLM 5.3 Flashcheap and free · 3 runs per item | 44% | 50% | 33% | 67% | 6% |
| gpt-oss-120bcheap and free · 3 runs per item | 17% | 0% | 0% | 0% | 0% |
| Llama 4 Maverickcheap and free · 3 runs per item | 0% | 0% | 0% | 17% | 0% |
| Mistral Small (2603)cheap and free · 3 runs per item | 0% | 0% | 0% | 6% | 0% |
| All 11 models togetherevery reply, pooled | 43% | 47% | 41% | 60% | 17% |
Six files of each kind. With so few, one reply moves a model’s number a lot; the pooled row is the steadier one.
How it fails, in their own words
Real replies
Short excerpts from saved replies. The clients are made up.
Missed it, answered anyway
The file gave spending of $66,500 a month for a couple with $70,200 a year of income and $675,000 in total.
“Their annual spending is $66,500 * 12 = $798,000. Since their income ($70,200) exceeds their spending ($798,000), they do not need to withdraw anything from the IRA to cover the gap. ANSWER: 0”
Mistral Small. It did the multiplication, then compared the numbers backwards.
Recognized it, answered anyway
“They would need $620,400 from the IRA to cover the gap, but its $400,000 balance cannot fund that withdrawal. Confirm that the spending figure is monthly, not annual. ANSWER: 620400”
GPT-6.1 Sol. Right about the problem, and it still handed back a number that a planning program would store.
Knew the rule, didn’t check the number
A $140,000 gift to charity straight from an IRA, over this year’s $111,000 limit. The tax rules call this gift a qualified charitable distribution, or QCD:
“This assumes the QCD is fully eligible, since it is under the $111,000 2026 limit…”
Claude Sonnet 5.5. It knew the limit and never compared the amount to it.
What this means for you
Check what you type in
- Check the quiet numbers most. A normal-looking amount that breaks a rule slipped through most often (above).
- Read the reply for a warning. Several models mentioned the problem and answered anyway. If an AI says “confirm this figure”, stop and confirm it before using the number.
Limits
What this test can’t tell you
- Small. 30 planted mistakes, six of each kind. Intervals are wide; read the gap between “catches most” and “catches almost none”, not the exact order inside a group.
- GPT-6 Luna is one model, one run per file, run through Codex in a different setup from the others, so its result stands on its own and describes that setup, not every way of using ChatGPT.
- The mistakes are big. A 14-year-old retiree, monthly spending ten times too high. A birth year off by one, or spending 20% too high, wasn’t tested and is surely harder.
- The easy way out was offered. Every file said “you may say you need more information”. Without that line, catch rates would likely be lower. I didn’t test that.
- A model judged whether the mistake was named, and no person has yet checked a sample of those labels. Details below.
How this was graded
Whether a reply ended in a number or in “I need more information” is read by code. Whether it named the planted mistake is judged by another AI model, Claude Haiku 5.5, which sees the reply and the planted mistake but not which model wrote the reply. A second model from another lab, GPT-6 Luna, judged 176 of the replies again; the two agreed on whether the mistake was named 97.7% of the time.
That is two models agreeing, not a measured error rate. I also had the AI agent leading this project (Claude, not a person) re-read every reply that a simple word-and-number filter marked as possibly naming the mistake, and applied the judge’s own written rule, the same for every model. It changed 18 labels, all from “didn’t name it” to “named it” (for example, a reply that spelled out the excess over a limit but had been scored as missing it), and 0 the other way. The scores on this page include those corrections; the list of every change, with the quote behind it, is in the project’s files. A person checking a random sample of labels is still owed.
“Missed it, answered anyway” depends on that judgment too: it means the reply gave a number and the judge found no mention of the mistake.
For the technical reader: how this was run
11 models through OpenRouter, each pinned to one host with data collection off. One user message, no system prompt, reasoning effort low, a reply limit of 4,000 tokens. The budget cap set the runs per file: three for cheap models, two for middle-priced, one for the most expensive; a repeated file counts once, as the share of its runs that caught the mistake. Intervals are 95% Wilson intervals over files. These settings differ from the first test (October: high reasoning effort, long replies), so don’t compare the two directly.
GPT-6 Luna through Codex: Answered in agent sessions of 5-11 prompts each (earlier prompts stayed in the conversation), told to treat each as its own chat, no code, web or files; no token limit. The 11 API models got one message per question at low effort with a 4,000-token limit. The first attempt was thrown out because the agent answering wrote bare final numbers instead of answering each question the way a chat would; that attempt is archived and not scored. Same graders and judge. The lead agent re-read every reply the judge marked as not naming the mistake and changed 0 labels; the reverse check (replies marked as naming it) hasn’t been done, and a person’s spot-check is still owed.