Results
The numbers behind each finding
Each section below backs up one line of the home page, with the short table and a link to the full detail. How the questions were built and checked, and how the tests differ, is on Technical details.
Finding 1 · First test · October 2026
One rule at a time, it’s great
A one-rule question applies a single rule, like the tax on an income or this year’s required IRA withdrawal.
| Model | One rule | Several rules at once |
|---|---|---|
| Claude Sonnet 5.5Anthropic | 97%89–99 | 37%22–54 |
| Claude Haiku 5.5Anthropic | 97%89–99 | 17%7–34 |
| DeepSeek V4.1 FlashDeepSeek · free to download | 86%77–91 | 36%23–50 |
| GLM 5.3 FlashZ.ai · free to download | 68%58–77 | 27%16–41 |
| gpt-oss-120bOpenAI · free to download | 36%26–46 | 5%2–18 |
Claude Sonnet 5.5 got 97% of one-rule questions right, and Claude Haiku 5.5 got 97%. When several rules met in one household, Sonnet dropped to 37%. The shaded band behind each bar is the range the true score likely sits in.
Finding 2 · Newer, harder set · November 2026
Several rules at once, it depends heavily on which AI
42 harder questions, each making two or more rules meet: the largest Roth conversion that won’t raise Medicare premiums two years later, a widow’s first year of taxes, Arizona state tax for retirees. 11 models, at low reasoning effort with a 4,000-token reply limit.
| Model | Right, of the newer set |
|---|---|
| GPT-6.1 SolOpenAI · top-priced | 98% · 41/4288–100 |
| GPT-6.1 TerraOpenAI · mid-priced | 80% · 33.5/4265–89 |
| Claude Sonnet 5.5Anthropic · top-priced | 62% · 26/4247–75 |
| DeepSeek V4.1 FlashDeepSeek · cheap and free | 29% · 12/4217–44 |
| Llama 4 MaverickMeta · cheap and free | 6% · 2.5/422–17 |
GPT-6.1 Sol got 98%, GPT-6.1 Terra 80%, Claude Sonnet 5.5 62%, and all 5 cheap and free models under 30%. These settings differ from the October test, so don’t compare the two sets of scores.
Finding 2 · The ChatGPT model · November 2026
GPT-6 Luna
GPT-6 Luna, asked the same 42 hard questions through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, graded by the same answer key.
| Questions | GPT-6 Luna |
|---|---|
| First test · one rule | 71% · 42 of 59 |
| Newer, harder set | 17% · 7 of 42 |
| Wrong facts · caught the planted mistake | 50% · 15 of 30 |
It ran with more time to think and no length limit, in a different setup from the models above, so it isn’t ranked with them. Why, and how the run was done, is on Technical details.
Finding 2 · Calculator test · October 2026
A calculator that only gave the number didn’t fix it. One that showed the rule did.
Claude Haiku 5.5 on the 50 hardest first-test questions, asked three ways: on its own; with a calculator that sends back only the number; and with one that also shows the rule and the IRS form line behind it.
Result. 40% on its own, 62% with only the number, 100% with the rule shown. On 50 held-back questions nobody tuned against: 44% → 96%.
| Claude Haiku 5.5 | The hardest questions | Held-back questions | Dollars off per question, per year |
|---|---|---|---|
| On its own | 40% · 20/5028–54 | 44% · 22/5031–58 | $318held-back $216 |
| Calculator 1.0: just the number | 62% · 31/5048–74 | not asked | $229 |
| Calculator 1.1: the number, the rule and its source | 100% · 50/5093–100 | 96% · 48/5087–99 | $0held-back $4 |
The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions.
Not every model was rescued. On the newer set, DeepSeek V4.1 Flash rose from 29% to 79% with the calculator; three other cheap models stayed at 10% or less.
Finding 3 · Wrong facts · November 2026
It believes whatever you tell it
30 client files, each with one planted mistake (two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending far beyond what the family owns). A model caught it only if it named the mistake and stopped to ask instead of giving a number.
| Of 30 planted mistakes | Recognized it | Stopped and asked | Caught (both) | Refused a fine file |
|---|---|---|---|---|
| GPT-6.1 Soltop-priced | 100% | 80% | 80% | 3% |
| Claude Sonnet 5.5top-priced | 93% | 73% | 73% | 0% |
| Gemini 3.8 Flashmid-priced | 92% | 100% | 92% | 50% |
| DeepSeek V4.1 Flashcheap and free | 76% | 68% | 61% | 0% |
| gpt-oss-120bcheap and free | 3% | 7% | 3% | 5% |
Even the best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, missed 20–27%. Three of the cheap and free models (Mistral Small, gpt-oss-120b and Llama 4 Maverick) caught 1–3%. An AI judge decided whether a reply named the mistake; how that was checked is on Technical details.
One example
A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied:
“Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0”
It saw that she was 14 and kept going. With her real birth year, the answer is $34,279.