Results

The numbers behind each finding

Each section below backs up one line of the home page, with the short table and a link to the full detail. How the questions were built and checked, and how the tests differ, is on Technical details.

Finding 1 · First test · October 2026

One rule at a time, it’s great

A one-rule question applies a single rule, like the tax on an income or this year’s required IRA withdrawal.

ModelOne ruleSeveral rules at once
Claude Sonnet 5.5Anthropic
97%89–99
37%22–54
Claude Haiku 5.5Anthropic
97%89–99
17%7–34
DeepSeek V4.1 FlashDeepSeek · free to download
86%77–91
36%23–50
GLM 5.3 FlashZ.ai · free to download
68%58–77
27%16–41
gpt-oss-120bOpenAI · free to download
36%26–46
5%2–18

Claude Sonnet 5.5 got 97% of one-rule questions right, and Claude Haiku 5.5 got 97%. When several rules met in one household, Sonnet dropped to 37%. The shaded band behind each bar is the range the true score likely sits in.

Finding 2 · Newer, harder set · November 2026

Several rules at once, it depends heavily on which AI

42 harder questions, each making two or more rules meet: the largest Roth conversion that won’t raise Medicare premiums two years later, a widow’s first year of taxes, Arizona state tax for retirees. 11 models, at low reasoning effort with a 4,000-token reply limit.

ModelRight, of the newer set
GPT-6.1 SolOpenAI · top-priced
98% · 41/4288–100
GPT-6.1 TerraOpenAI · mid-priced
80% · 33.5/4265–89
Claude Sonnet 5.5Anthropic · top-priced
62% · 26/4247–75
DeepSeek V4.1 FlashDeepSeek · cheap and free
29% · 12/4217–44
Llama 4 MaverickMeta · cheap and free
6% · 2.5/422–17

GPT-6.1 Sol got 98%, GPT-6.1 Terra 80%, Claude Sonnet 5.5 62%, and all 5 cheap and free models under 30%. These settings differ from the October test, so don’t compare the two sets of scores.

Finding 2 · The ChatGPT model · November 2026

GPT-6 Luna

GPT-6 Luna, asked the same 42 hard questions through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, graded by the same answer key.

QuestionsGPT-6 Luna
First test · one rule71% · 42 of 59
Newer, harder set17% · 7 of 42
Wrong facts · caught the planted mistake50% · 15 of 30

It ran with more time to think and no length limit, in a different setup from the models above, so it isn’t ranked with them. Why, and how the run was done, is on Technical details.

Finding 2 · Calculator test · October 2026

A calculator that only gave the number didn’t fix it. One that showed the rule did.

Claude Haiku 5.5 on the 50 hardest first-test questions, asked three ways: on its own; with a calculator that sends back only the number; and with one that also shows the rule and the IRS form line behind it.

Result. 40% on its own, 62% with only the number, 100% with the rule shown. On 50 held-back questions nobody tuned against: 44% → 96%.

Claude Haiku 5.5The hardest questionsHeld-back questionsDollars off per question, per year
On its own
40% · 20/5028–54
44% · 22/5031–58
$318held-back $216
Calculator 1.0: just the number
62% · 31/5048–74
not asked$229
Calculator 1.1: the number, the rule and its source
100% · 50/5093–100
96% · 48/5087–99
$0held-back $4

The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions.

Not every model was rescued. On the newer set, DeepSeek V4.1 Flash rose from 29% to 79% with the calculator; three other cheap models stayed at 10% or less.

Finding 3 · Wrong facts · November 2026

It believes whatever you tell it

30 client files, each with one planted mistake (two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending far beyond what the family owns). A model caught it only if it named the mistake and stopped to ask instead of giving a number.

Of 30 planted mistakesRecognized itStopped and askedCaught (both)Refused a fine file
GPT-6.1 Soltop-priced100%80%80%3%
Claude Sonnet 5.5top-priced93%73%73%0%
Gemini 3.8 Flashmid-priced92%100%92%50%
DeepSeek V4.1 Flashcheap and free76%68%61%0%
gpt-oss-120bcheap and free3%7%3%5%

Even the best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, missed 20–27%. Three of the cheap and free models (Mistral Small, gpt-oss-120b and Llama 4 Maverick) caught 1–3%. An AI judge decided whether a reply named the mistake; how that was checked is on Technical details.

One example

A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied:

“Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0”

It saw that she was 14 and kept going. With her real birth year, the answer is $34,279.