PlannerBench · full report

Can you trust AI with retirement math?

Peyton Thalman · updated 2026-11-15 · two rounds of tests (October and November 2026), then an independent audit

Summary

I'm a financial advisor. More and more of the people I meet get their financial advice from ChatGPT, so I tested how often AI gets retirement math right, against an answer key I could prove.

  1. Simple questions: almost always right. On one-rule questions (this year's tax bracket, an IRA withdrawal), Claude Sonnet 5.5 and Claude Haiku 5.5 both scored 97% in the first test.
  2. Real-life questions: it depends which AI. When taxes, Social Security and Medicare meet in one household, scores spread out. In the first test, Sonnet got 37% of these and Haiku 17%, and models from several labs made the same misreading of a rule new in 2025. In the newer, harder set of 42 questions (11 models, low reasoning effort), GPT-6.1 Sol got 98%, Claude Sonnet 5.5 62%, and all five cheap models under 30%.
  3. A calculator helps, if it shows its work, and only for some models. Handed only the right number, Claude Haiku overruled it with its own wrong rule (40% → 62% on the 50 hardest first-test questions). When the calculator also showed the rule and the IRS form line, Haiku got 100%, and 96% on 50 questions held back. In the newer set, DeepSeek V4.1 Flash went from 29% to 79% with the calculator; three other cheap models called it and scored 10% or less, and the best model gained nothing.
  4. Tell it something wrong: GPT-6.1 Sol and Claude Sonnet 5.5 catch most of it, three cheap models almost none. This is the biggest open gap. In 30 client files with one planted mistake each, GPT-6.1 Sol caught 80% and Claude Sonnet 5.5 73%, with almost no false alarms. Three cheap models caught 1–3% and gave a confident number 81–93% of the time.

And ChatGPT? GPT-6 Luna on a ChatGPT subscription got 71% of one-rule questions but 17% of the hardest ones, and caught half the planted mistakes. It was a different setup, so it is reported in its own section below.

The plain answer. For one well-known rule, the best AIs are reliable. For a whole household, it depends heavily on which AI you ask: the cheap and free models I tested, at quick-answer settings, got most of the hard questions wrong. And the answer is only as good as what you type in: in my tests, three cheap models almost never questioned an impossible fact, and even the best missed about one in four. Check your inputs, and take big decisions to a professional.

How this was built. I directed AI coding agents (mostly Claude Code) and made the calls: what to test, what counts as right, what to spend, what to publish. Because AI did most of the work, the checks are set up so no agent is the only grader of its own work (the lead agent did re-check 18 judge labels; that is disclosed below, and a human spot-check is owed), and a model from another lab audited every published claim on 2026-11-15. It found 12 errors. They are fixed in this version and listed under Corrections.

Catch rate against false-alarm rate

Why I made this

I meet most people by knocking on their door. When someone tells me ChatGPT answered their retirement question, I ask how they know it's right. Usually they Googled it, and the top result was another AI's answer. I wanted to know, with evidence, which planning questions a model can be trusted with on its own, and which need verified math behind it.

What I built

The two rounds used different settings, so their scores are reported separately and never compared directly.

Findings 1 and 2: AI on its own

The first test (October)

ModelOne ruleSeveral rules at onceOverall
Claude Sonnet 5.597%37%84%
Claude Haiku 5.597%17%77%

Five more setups (DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Qwen3.8 Flash, Haiku through OpenRouter) are on the site's "AI on its own" page with 95% intervals; 10–15 questions per task per model.

One rule explains a large share of the misses. The 2025 tax law gave each person 65 or older an extra $6,000 deduction that shrinks by 6% of income over $150,000 on a joint return, worked per person. Models from several labs applied the phase-out to the couple as a whole and gave the same wrong answer. Across the two published first-test runs, this one misreading accounts for 99 of 342 wrong answers (29%), and 99 of the 139 wrong answers I could trace to a named misreading (71%). Pasting the IRS's two-line instructions in front of the question fixed most of them.

The newer, harder set (November)

Each of the 42 questions makes two or more rules meet: the largest Roth conversion that leaves Medicare premiums alone two years later; a conversion's cost in federal plus Arizona tax; taxable Social Security with muni interest and a charitable gift from an IRA (a QCD); an inherited IRA's 2026 required withdrawal under the 10-year rule; tax when part of a required withdrawal goes to charity; a widow's first year; Arizona tax for retirees. Each key comes from the planning engine and must match the second engine to the dollar. Sources and examples: v2/data/HARD.md.

ModelMakerAloneRan out of room$ per right answer
GPT-6.1 SolOpenAI98% [88–100]0%$0.0032
GPT-6.1 TerraOpenAI80% [65–89]0%$0.0019
Claude Sonnet 5.5Anthropic62% [47–75]0%$0.0173
Claude Haiku 5.5Anthropic52% [38–67]2%$0.0022
Gemini 3.1 ProGoogle29% [17–44]0%$0.0513
Gemini 3.8 FlashGoogle29% [17–44]36%$0.0195
DeepSeek V4.1 FlashDeepSeek (open)29% [17–44]71%$0.0036
GLM 5.3 FlashZ.ai (open)15% [8–29]20%$0.0063
gpt-oss-120bOpenAI (open)7% [2–19]0%$0.0016
Mistral SmallMistral (open)7% [2–19]0%$0.0087
Llama 4 MaverickMeta (open)6% [2–17]0%$0.0185

Finding 3: with a calculator

Hand it the number and it argues; show it the rule and it listens (October)

Claude Haiku 5.5, effort high, on the 50 hardest first-test questions:

AloneCalculator 1.0 (number only)Calculator 1.1 (number, rule, form line)
Public questions40% (20/50)62% (31/50)100% (50/50)
Held-back questions44%96% (48/50)

With calculator 1.0, most misses were Haiku calling the tool, getting the right answer, and replacing it with its own per-couple reading of the senior deduction, in its own words ("the tool returned $16,074, but I believe it overstates the senior deduction phase-out"). The engine's answers didn't change between 1.0 and 1.1; only the reply did. Both held-back misses were Haiku overruling the tool again. This was tested on one model; it shows that what a tool says back matters, not that it works for every model.

Whole-household decisions. Each question is a made-up household of about 16 facts and a real choice (how much to convert to a Roth, when to claim Social Security, how to give, how to empty an inherited IRA). One question from each of 10 families plus a trap, one try each:

ModelRight decision and every number within 2%, aloneWith the calculator
Claude Haiku 5.54/1111/11
Claude Sonnet 5.57/10 (3 ran out of room)11/11
DeepSeek V4.1 Flash2/1111/11

Every key agrees with the second engine on every field of every year, and every 2026 return agrees with PolicyEngine-US (203/203). With one fact left out (a Social Security amount or an IRA balance), DeepSeek with the calculator asked for it 9 times in 10 and got 9 of 10 fully right. Small samples: this shows the gap, not a ranking.

The newer set: a cheap model catches up, others don't (November)

Paired: each model's first run alone against its one run with the calculator, on the same 42 questions.

ModelAlone (first run)With the calculatorFixed / brokenCalled a tool
GPT-6.1 Sol41/4240/420 / 179%
GPT-6.1 Terra34/4236/423 / 181%
DeepSeek V4.1 Flash12/4233/4221 / 090%
Claude Haiku 5.520/4226/428 / 250%
GLM 5.3 Flash7/4218/4211 / 086%
gpt-oss-120b3/424/422 / 179%
Mistral Small3/423/423 / 379%
Llama 4 Maverick3/422/420 / 179%

Sonnet and the Gemini models were not run with the calculator (cost). The "Alone" column in the earlier table averages two runs for some models; this one uses the first run, the one the calculator run is paired with.

Finding 4: wrong facts

Setup. Each note is a short planner's intake file and one question. 30 notes have one planted mistake each, in five kinds: two digits swapped in a balance; a birth year that makes the client 14 (or 138); monthly spending inflated ten- or twelvefold, so the household would spend more than everything it owns in a year; facts that conflict ("married" next to "filing single"); an amount over a legal limit. Each has a clean twin that differs in only that fact, and 10 odd-but-valid notes (a 90-year-old IRA owner, a contribution at exactly the limit) make sure a model can't win by flagging anything unusual. Every prompt ends the same way: give a number, or end with ANSWER: NEED_INFO and say what you need. Examples: v2/data/BAD_INPUTS.md.

A reply to a corrupted note lands in one of five places:

Whether a reply ended in a number or NEED_INFO is read by a rule. Whether it named the planted problem was judged by Claude Haiku 5.5, which saw the reply and the planted problem but not which model wrote it. A second judge from another lab (GPT-6 Luna) re-judged 176 of the corrupted-note replies and agreed with the first on 97.7% of "named the problem" labels (kappa 0.95). Then the lead AI agent (Claude), not a person, adjudicated every reply the judge marked as not naming the problem (details below).

ModelCaught itRecognized it, answered anywayMissed it, answered anywayDeclined, other reasonRan out of roomFalse alarms (40 valid notes)Clean twins right (30)
GPT-6.1 Sol80% [63–90]20%0%0%0%3%100%
Claude Sonnet 5.573% [56–86]20%7%0%0%0%93%
GPT-6.1 Terra67% [49–81]33%0%0%0%3%97%
Claude Haiku 5.565% [47–79]17%18%0%0%0%85%
DeepSeek V4.1 Flash61% [43–76]14%18%0%7%0%93%
Gemini 3.8 Flash92% [76–97]0%0%8%0%50%47%
Gemini 3.1 Pro70% [52–83]0%3%27%0%68%30%
GLM 5.3 Flash40% [25–58]31%14%0%14%0%33%
Llama 4 Maverick3% [1–17]16%81%0%0%2%30%
gpt-oss-120b3% [1–17]0%93%3%0%5%20%
Mistral Small1% [0–13]1%91%7%0%1%22%

What each model did with a corrupted note

Adjudication. The audit found replies that plainly named the planted problem but were judged as not naming it. So the lead AI agent checked every one of the 322 corrupted-note replies the judge marked "not named": a transparent pre-filter (the planted numbers and terms appearing in the reply) picked 143 candidates, and each was read against the rule "the reply says, in its own words, that the corrupted fact is wrong, impossible, inconsistent, over a limit, or more than the household can fund". 18 labels changed, all from "missed" to "named": GPT-6.1 Terra 14, Sol 2, DeepSeek 2. A reverse check of 251 "caught" replies found none to change. Every override is in v2/runs/bad/adjudication.jsonl with the exact quote, and v2/results/ADJUDICATION.md has the method. The judgment call that matters most: 9 Terra spending replies say the IRA can't cover the withdrawal without questioning the spending figure. I counted them as "noticed". On a stricter reading, Terra's "missed it" rate would be 15% instead of 0%; its catch rate is the same either way. The adjudicator was the lead AI agent, not a person; a human spot-check is still owed. The judge-only numbers are kept in v2/results/summary.json (judge_only).

How it fails: real replies

Excerpts from saved replies (v2/runs/*/raw.jsonl). Personas are made up.

1. Missed it, answered anyway: computing on a fact that can't be true

The note gives a date of birth in 2012 for a retired school administrator with a $785,000 IRA and asks for her required withdrawal.

gpt-oss-120b: "Priya will be 14 years old throughout 2026, well below the required beginning age for RMDs (which is 73 under current law). Therefore, no required minimum distribution is required for 2026. ANSWER: 0"

It saw the age and reasoned from it. The real answer, with the real birth year, is $34,279.

The note gives spending of $66,500 a month for a couple with $70,200 of income and $675,000 in total.

Mistral Small: "Their annual spending is $66,500 * 12 = $798,000. Since their income ($70,200) exceeds their spending ($798,000), they do not need to withdraw anything from the IRA to cover the gap. ANSWER: 0"

The note lists an IRA balance of $624,900 and a custodian statement showing $642,900.

Llama 4 Maverick: "...the distribution period for an 82-year-old is 17.1. Thus, the RMD for 2026 is $624,900 / 17.1 = $36,543.86. ANSWER: 36544"

It picked one balance without a word, and used the life-expectancy table that was replaced in 2022.

2. Recognized it, answered anyway

GPT-6.1 Sol: "They would need $620,400 from the IRA to cover the gap, but its $400,000 balance cannot fund that withdrawal. Confirm that the spending figure is monthly, not annual. ANSWER: 620400"

The reply is right about the problem and still hands back a number that a downstream system would store.

Claude Sonnet 5.5 (a $46,000 401(k) deferral at age 43): "the $46,000 deferral exceeds the 2026 limit of $24,500 for someone under 50, so only $24,500 can be pre-tax; the excess $21,500 is taxable... ANSWER: 20774"

This one is defensible: it corrected the input to the legal limit and said so. A planner would still want to be asked.

GLM 5.3 Flash: "the notes say spending is '$56,500 a month,' which for this household almost certainly means $56,500 a year... On the annual reading... ANSWER: 0"

It guessed which fact the client meant, which is the behavior this test exists to catch.

3. Missing a wrong amount because the rule is half-remembered

A $140,000 qualified charitable distribution, over the 2026 limit of $111,000:

Claude Sonnet 5.5: "This assumes the QCD is fully eligible, since it is under the $111,000 2026 limit..."

It knew the limit and didn't compare the number to it.

4. False alarm: declining a valid note

Gemini 3.8 Flash (a clean note): "The cited 'One Big Beautiful Bill Act' and 'IRS Rev. Proc. 2025-32' are not actual federal tax authorities... ANSWER: NEED_INFO"

The law is real and was passed after the model's training data ends. For a firm this is still a failure: an answerable question got no answer.

5. Calculator offered, not used

Claude Haiku 5.5 (inherited IRA, key $9,960): "$248,000 ÷ 23.0 = $10,782.61... I did not run this through the calculator tools. The 27.0 factor for age 58 is from my memory of the Pub 590-B table. ANSWER: 10783"

6. Calculator called, then wrong

The question: tax for a single retiree whose required withdrawal partly goes to charity, with a pension and Social Security. gpt-oss made one call to the tax tool with the pension alone, left out the withdrawal entirely, and replied with a single line:

gpt-oss-120b: "ANSWER: 1115" (key: $9,614)

GPT-6.1 Sol (Arizona tax, key $395): took federal AGI and taxable Social Security from the tool, then: "Arizona's standard deduction does not include the federal age-65 addition... $17,900 × 2.5% = $447.50. ANSWER: 448"

Right about the standard deduction, and it missed Arizona's separate $2,100 exemption for people 65 and older. The tool gave it correct parts; the assembly was its own.

GPT-6 Luna through Codex (the model on my ChatGPT subscription)

Most people ask ChatGPT. So on 2026-11-15 the same questions went to GPT-6 Luna, the model on my ChatGPT subscription, run through OpenAI’s Codex app (not the ChatGPT chat window), graded with the same answer keys and graders. Luna answered each prompt as its own chat with no code, calculator or web, at medium effort with no token limit, in batches of 5–11 prompts per session. That is a different setup from the API runs (one message per question, fixed token limits), so these numbers sit beside the tables above, not inside them. In real ChatGPT the model may choose to run code; here it didn't, the same as the other models.

TestGPT-6 Luna
First test: one-rule questions71% (42/59)
First test: a few steps / several rules at once5% (1/20) / 0% (0/30)
First test: compliance93% (28/30)
Newer, harder set17% (7/42)
Wrong facts: caught / noticed, answered anyway / missed, answered anyway50% / 10% / 40% (of 30)
Wrong facts: false alarms (40 valid notes) / clean twins right (30)10% / 43%
Whole-household decisionsdeclined all 10 questions that had an answer; correctly declined the trap

Corrections

On 2026-11-15, GPT-6.1 Sol (a model from a different lab than the agents that built this) audited every published claim against the saved data, read-only. What it found and what changed:

#ErrorFix
1"Most wrong answers" came from the senior-deduction misreadingIt's 99 of 342 wrong answers (29%), and 71% of those traced to a named misreading; the text says so
2Some judge labels missed replies that named the planted mistakeEvery "not named" label adjudicated; 18 changed (Sol 73% → 80%, Terra 62% → 67%); method above
3"Every model was worst at over-limit amounts"False for Sol and Terra; now "lowest pooled catch rate", with the exceptions named
4reproduce.sh was described as rebuilding every number on the siteIt rebuilds and checks the first test's leaderboard; the newer set regrades with v2/grade.py; the text says which is which
5Replies that ran out of room were counted as declining for another reasonA separate "ran out of room" outcome (19 replies, 6 DeepSeek and 13 GLM), still counted as not caught
6Run-to-run stability measured caught/not-caught but was described as "same outcome"Both numbers reported and labeled
7Chart label "Caught the mistake and asked", but the grade doesn't check for a question; text said the other columns were rule-basedLabel is "Caught it": named the mistake and gave no number. Almost every caught reply also asked for the right fact (294 of 299), which is why the site says "stopped to ask"; the judge's role is described
8Twin pairs counted each repeated run as a separate itemRuns averaged within each of the 30 pairs
9The worked IRS form showed 2025 line numbers next to a 2026 questionLabeled as the 2025 form the 2026 rule comes from
10"Single IRMAA thresholds are half the joint ones"Not at the top tier ($500,000 single, $750,000 joint); corrected
11Whole-household key check quoted as 290/290203/203, matching checks/GAUNTLET.md
12"Spending ten times what the family owns"Monthly spending inflated ten- or twelvefold (about 1.2–1.7 times assets a year)

It also flagged claims stronger than the evidence, all now scoped: the home page combined numbers from different models and tests into one story; "at least as good" was claimed from overlapping intervals; "a product fix works for every model" rested on one model; a harness check with p = 1 was described as proof of neutral routing; the 4,000-token limit was per turn in the tool loop; PolicyEngine's check covers the first test's tax answers and 2026 returns, not Arizona. The full list is in docs/AUDIT-2026-11-15.md.

Limits

What I'd do next

  1. A person checks the judge and the key. Human labels on 100 replies, and a CPA sign-off on the answer key.
  2. Subtle mistakes. The same note with spending 1.2×, 2×, 5× and 10× too high, birth years off by 1, 5 and 50. Where each model starts to notice is the useful number.
  3. Take away the way out. Rerun with no mention of NEED_INFO, and with "always give a number", to measure how much the instruction is doing.
  4. Put validation in the calculator. A calculator that explains its rule beats one that returns a number; the same may hold for inputs. Have the engine reject a 14-year-old retiree, and measure whether cheap models pass that on.
  5. Three runs for everyone, high effort for the top models. About $40.

Reproduce

./reproduce.sh                        # first test: re-verify the key, regrade saved replies, rebuild the leaderboard
.venv/bin/python v2/build_hard.py     # 42 hard items; fails if the two engines disagree
.venv/bin/python v2/build_bad.py      # 70 notes; fails if the engine and the checker disagree
./v2/run_all.sh                       # every run, resumable, stops at the cap in v2/runs/ledger.json (costs money)
.venv/bin/python v2/grade.py judge    # then: grade.py judge --second (costs money)
.venv/bin/python v2/grade.py          # summary.json and RESULTS.md, free, from saved replies and adjudication.jsonl
.venv/bin/python v2/build_site.py     # charts, web/data/v2.json, web/report.html
.venv/bin/python src/bake_site.py     # web/ → site/, readable without JavaScript