Finding 2 · With a calculator

Give it a calculator it can trust, and some models get it right. Not all.

The plain answer

I turned my answer key into a calculator an AI model can use while it answers. It helps a lot, but only if the calculator shows its work and the model actually uses it well. It rescued some models, including a cheap one, and did nothing for others.

One example: it argued with the calculator

Claude Haiku 5.5 asked the calculator for a couple’s tax, got the right number, and replied:

“The calculation tool's figure of $21,639 applies the senior-deduction phase-out to each spouse separately, which I don't think the statute supports.”

It used its own wrong reading of a new 2025 tax break for people over 65 instead, and got the tax wrong. When the calculator also showed the rule and the IRS form behind the number, it stopped arguing.

What I mean by a “calculator”

In AI, a “tool” is a program a model can call in the middle of an answer, the way you’d reach for a calculator. Mine is the planning engine: code for 2026 federal tax, Social Security, required withdrawals from retirement accounts and Medicare premiums, checked against IRS, Social Security and Medicare publications. The model still has to decide what to type in and how to put the pieces together, which is where it can still go wrong.

Three calculator tests on this page

  1. Calculator test 1: number vs rule (October). One model, the hardest first-test questions. A right number wasn’t enough; the rule behind it was.
  2. Calculator test 2: who it rescues (November). Eight models on the newer, harder set. A cheap model jumped; three others didn’t move.
  3. Calculator test 3: whole-household decisions. Real choices over 25 years, three models. Plus GPT-6 Luna through Codex (the model on my ChatGPT subscription) without a calculator, and a short test: leave a fact out, and see if it asks.

Calculator test 1 of 3 · October 2026

Hand it the right number, and it argues. Show it the rule, and it listens.

Questions
the 50 hardest from the first test, plus 50 held-back ones
Model
Claude Haiku 5.5
Asked
three ways: on its own; with calculator 1.0 (sends back only the number); with 1.1 (the number, plus the rule and the IRS form line behind it)
Settings
high reasoning effort (lets the model think longer), long replies, one run per question
Graded
by code, against the answer key, within $1–2
Held out
yes: the held-back questions were kept secret and never used while building the calculator

Result. 40% on its own, 62% with only the number, 100% with the rule shown. On the held-back questions: 44% → 96%. The calculator’s numbers were the same in both versions; only what it said about them changed.

Claude Haiku 5.5The hardest questionsHeld-back questionsDollars off per question, per year
On its own
40% · 20/5028–54
44% · 22/5031–58
$318held-back $216
Calculator 1.0: just the number
62% · 31/5048–74
not asked$229
Calculator 1.1: the number, the rule and its source
100% · 50/5093–100
96% · 48/5087–99
$0held-back $4

The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions.

The question from the example at the top

The calculator’s answer (and the key)
$21,639
Haiku with calculator 1.0
$21,121Wrong
Haiku with calculator 1.1
$21,639Right

What version 1.1 sent back with the tax

  1. Income (MAGI) over the $150,000 threshold$39,225.00
  2. Reduction per person (6%)$2,353.50
  3. Senior deduction per person$3,646.50
  4. People 65 or older2
  5. Senior deduction$7,293.00

The senior deduction from the 2025 tax law (OBBBA), IRS Form 1040 Schedule 1-A Part V (lines 35-36b): $6,000 for each person 65 or older, reduced by 6% of MAGI over $150,000. The reduction is figured for each qualifying person separately: on a joint return each spouse's $6,000 is reduced by the full 6% of the excess, not one reduction off a combined $12,000.

  • A right number wasn’t enough. Across all 140 questions, calculator 1.0 lifted Haiku from 77% to 84% (13 fixed, 3 broken, p = 0.021); single-step questions were already at 97%. But 18 of the 22 misses left were Haiku getting the right number from the calculator and replacing it with its own, every time saying so.
  • It held on questions nobody tuned against. On 50 questions held back from a secret seed and never used while building the calculator: 44% on its own, 96% with calculator 1.1. Both misses were Haiku overruling the calculator again, with the rule in front of it.
  • Limits. One model, one run per question. The held-back questions came from the same generator, so they test the same kinds of rules. These are the hardest questions, which is why Haiku scores lower here than it does overall.

Calculator test 2 of 3 · November 2026

Who it rescues: a cheap model jumped, three others didn’t move

Questions
the 42 questions in the newer, harder set, each with several rules meeting in one household
Models
8 of the 11 models, on their own and with the calculator (Claude Sonnet 5.5 and the two Gemini models weren’t run with it, to save money)
Settings
low reasoning effort; replies capped at 4,000 tokens (a token is about three-quarters of a word); with the calculator, the cap applied to each of up to 8 back-and-forth turns
Compared
one run with the calculator against the first run on its own, question by question
Graded
by code, against the answer key, within $2–5
Held out
no: the newer set has no secret questions

Result. DeepSeek V4.1 Flash, a cheap model, went from 29% to 79% with the calculator setup: 21 questions fixed, none broken. That’s higher than Claude Sonnet 5.5 on its own (62%), at about 1/15 of the cost per right answer ($1.18 per 1,000 right answers against $17.33).

Not all of that gain is the calculator. The setup also let a model write more in total, because the reply cap applied to each turn. On its own, DeepSeek hit the cap before answering on 71% of questions; with the calculator, 9 of its replies went past 4,000 tokens in total. So part of the jump may come from that extra room. The conservative number: counting only replies that stayed within 4,000 tokens in total, it got 27/42 (64%). Higher than Sonnet’s score, but on only 42 questions that lead could still be luck.

ModelOn its own (first run)With the calculator (one run)
GPT-6.1 Soltop-priced · one run each way
98% · 41/4288–100
95% · 40/4284–99
DeepSeek V4.1 Flashcheap and free · one run each way
29% · 12/4217–44
79% · 33/4264–88
Claude Haiku 5.5mid-priced · one run each way
48% · 20/4233–62
62% · 26/4247–75
gpt-oss-120bcheap and free · one run each way
7% · 3/422–19
10% · 4/424–22
Claude Sonnet 5.5top-priced · reference: not run with the calculator
62% · 26/4247–75
not run

Four of the 8 models run both ways, plus Claude Sonnet 5.5 on its own for reference; all of them are in the first table under “All models and columns” below. Both columns are one run per question, on the same 42 questions. Highlighted: the calculator added more than 30 points. The small grey numbers and the shaded band behind each bar show the range the true score likely falls in, given so few questions. These settings differ from calculator test 1 above, so don’t compare the two sets of scores.

Called it, then wrong

A single retiree’s tax, with a pension, Social Security, and a required withdrawal partly given to charity. gpt-oss-120b made one call with the pension alone, left out the withdrawal, and replied with one line:

“ANSWER: 1115”

The right answer is $9,614.

Right parts, wrong assembly

GPT-6.1 Sol took the federal numbers from the calculator for an Arizona tax question, then did the state part itself:

“Arizona’s standard deduction does not include the federal age-65 addition… $17,900 × 2.5% = $447.50. ANSWER: 448”

It missed Arizona’s separate $2,100 exemption for people 65 and older. The right answer is $395. The calculator gave it correct parts; putting them together was its own work.

All models and columns (for the detail-minded)

Every model run both ways, one run each. Fixed: wrong on its own, right with the calculator; broken: the reverse. Called it: the share of questions where the model used the calculator at all.

ModelOn its own (first run)With the calculator (one run)Fixed / brokenCalled it
GPT-6.1 Soltop-priced · one run each way
98% · 41/4288–100
95% · 40/4284–99
0 / 179%
GPT-6.1 Terramid-priced · one run each way
81% · 34/4267–90
86% · 36/4272–93
3 / 181%
DeepSeek V4.1 Flashcheap and free · one run each way
29% · 12/4217–44
79% · 33/4264–88
21 / 090%
Claude Haiku 5.5mid-priced · one run each way
48% · 20/4233–62
62% · 26/4247–75
8 / 250%
GLM 5.3 Flashcheap and free · one run each way
17% · 7/428–31
43% · 18/4229–58
11 / 086%
gpt-oss-120bcheap and free · one run each way
7% · 3/422–19
10% · 4/424–22
2 / 179%
Mistral Small (2603)cheap and free · one run each way
7% · 3/422–19
7% · 3/422–19
3 / 379%
Llama 4 Maverickcheap and free · one run each way
7% · 3/422–19
5% · 2/421–16
0 / 179%
Claude Sonnet 5.5top-priced · reference: not run with the calculator
62% · 26/4247–75
not run––

Every model in the November set, with every run averaged:

ModelOn its ownWith the calculatorCalled itCost per 1,000 right, on its own → with it
GPT-6.1 Soltop-priced · one run each way
98%88–100
95%84–99
79%$3.22 → $3.19
GPT-6.1 Terramid-priced · on its own: 2 runs averaged · with it: 1 run
80%65–89
86%72–93
81%$1.89 → $1.79
DeepSeek V4.1 Flashcheap and free · on its own: 2 runs averaged · with it: 1 run
29%17–44
79%64–88
90%$3.64 → $1.18
Claude Sonnet 5.5top-priced · 1 run per item
62%47–75
not run–$17.33
Claude Haiku 5.5mid-priced · on its own: 2 runs averaged · with it: 1 run
52%38–67
62%47–75
50%$2.21 → $3.63
GLM 5.3 Flashcheap and free · on its own: 2 runs averaged · with it: 1 run
15%8–29
43%29–58
86%$6.30 → $2.46
Gemini 3.1 Protop-priced · 1 run per item
29%17–44
not run–$51.35
Gemini 3.8 Flashmid-priced · 1 run per item
29%17–44
not run–$19.47
gpt-oss-120bcheap and free · on its own: 2 runs averaged · with it: 1 run
7%2–19
10%4–22
79%$1.60 → $5.02
Mistral Small (2603)cheap and free · on its own: 2 runs averaged · with it: 1 run
7%2–19
7%2–19
79%$8.73 → $11.75
Llama 4 Maverickcheap and free · on its own: 2 runs averaged · with it: 1 run
6%2–17
5%1–16
79%$18.47 → $62.65

Unlike the table above, “On its own” here averages every run of a question (two for most models, one for the most expensive), so Claude Haiku 5.5 shows 52% here and 20/42 (its first run) above. Each row says how many runs went into each column. Price group is by the model’s price per token: top-priced, mid-priced, or cheap and free.

By kind of question (for the detail-minded)
ModelRoth conversion, Medicare-safeRoth conversion cost with Arizona taxTaxable Social SecurityInherited IRA withdrawalCharity from a required withdrawalWidow’s first yearArizona tax
GPT-6.1 Soltop-priced · on its own → with it100% → 100%100% → 100%100% → 100%100% → 100%100% → 100%100% → 100%83% → 67%
GPT-6.1 Terramid-priced · on its own → with it100% → 100%75% → 100%100% → 100%92% → 83%92% → 100%100% → 100%0% → 17%
DeepSeek V4.1 Flashcheap and free · on its own → with it67% → 100%0% → 83%100% → 100%33% → 50%0% → 100%0% → 83%0% → 33%
Claude Sonnet 5.5top-priced · on its own100%17%100%83%67%67%0%
Claude Haiku 5.5mid-priced · on its own → with it100% → 100%8% → 50%92% → 83%33% → 50%67% → 83%67% → 67%0% → 0%
GLM 5.3 Flashcheap and free · on its own → with it42% → 100%0% → 0%33% → 50%25% → 33%0% → 67%8% → 50%0% → 0%
Gemini 3.1 Protop-priced · on its own17%0%100%83%0%0%0%
Gemini 3.8 Flashmid-priced · on its own0%0%100%67%33%0%0%
gpt-oss-120bcheap and free · on its own → with it0% → 0%0% → 17%17% → 0%33% → 33%0% → 17%0% → 0%0% → 0%
Mistral Small (2603)cheap and free · on its own → with it0% → 0%0% → 0%17% → 50%25% → 0%8% → 0%0% → 0%0% → 0%
Llama 4 Maverickcheap and free · on its own → with it0% → 0%0% → 0%8% → 0%33% → 33%0% → 0%0% → 0%0% → 0%

Six questions of each kind. Each cell is right on its own (every run averaged) → right with the calculator. The kinds: the largest Roth conversion (moving money from a regular IRA to a Roth IRA and paying the tax now) that keeps Medicare premiums level; what a conversion costs in federal plus Arizona tax; how much Social Security is taxed when there’s tax-free bond interest and a gift to charity from an IRA; this year’s required withdrawal from an inherited IRA; tax when part of a required withdrawal goes to charity; a widow’s first tax year; Arizona tax for retirees.

Calculator test 3 of 3 · Whole-household decisions

Real retirement decisions: with the calculator, all three models made the right decision, with every number within 2%, every time

Questions
11 whole made-up households (about 16 facts each: Social Security, IRAs, a pension, savings), one from each of 10 kinds of decision plus a trap with no honest answer
Models
Claude Sonnet 5.5, Claude Haiku 5.5, DeepSeek V4.1 Flash
Settings
high reasoning effort; reply budget 64,000 tokens for the Claude models and 32,000 for DeepSeek; one try each
Graded
by code: “fully right” means the right decision and every number within 2% of the key
Held out
no: 33 held-back questions are built but not yet run

Result. Each choice is one people really face: how much to convert to a Roth, when to claim Social Security, how to give to charity, how fast to empty an IRA inherited from someone who died. Getting it right means combining at least three rules over 25 years. With the calculator, all 3 models got every question fully right: the right decision, and every number within 2% of the key. On its own, DeepSeek V4.1 Flash got 2/11.

ModelFully rightRan out of roomThinking per questionCost per question
Claude Haiku 5.5, on its own
36% · 4/1115–65
034k tokens$0.02
Claude Sonnet 5.5, on its own
70% · 7/1040–89
337k tokens$0.38
DeepSeek V4.1 Flash, on its own
18% · 2/115–48
027k tokens$0.01
Claude Haiku 5.5 + calculator
100% · 11/1174–100
09k tokens$0.01
Claude Sonnet 5.5 + calculator
100% · 11/1174–100
03k tokens$0.16
DeepSeek V4.1 Flash + calculator
100% · 11/1174–100
010k tokens$0.01

11 questions, one from each family plus a trap whose honest answer is “I can’t answer that.” Ran out of room: used its whole reply budget thinking and never answered (scored as not right). Sonnet alone shows 10 because one reply was cancelled to stay inside the run’s spending cap. The shaded band behind each bar is the range the true score likely falls in, given so few questions.

One question, both ways

Ray and Linda want to start Roth conversions from Ray's traditional IRA this year and want to know how much to convert.

The calculator’s best plan
$118,000 a year for 8 years (2026–2033)
DeepSeek V4.1 Flash, on its own
Convert nothing (a projected $134,445 less after-tax wealth over 25 years; a simulation, not a real household)Wrong
DeepSeek V4.1 Flash + calculator
$118,000 a year × 8 yearsRight

Confident, wrong, and the projected after-tax wealth it gives up over 25 years (simulated households, not real ones)

  1. 1How much to convert to Roth, and for how many years (DeepSeek V4.1 Flash, on its own). With the calculator: right.−$134,445
  2. 2Converting inside the senior-deduction phase-out (DeepSeek V4.1 Flash, on its own). With the calculator: right.−$72,529
  3. 3How much to convert to Roth, and for how many years (Claude Haiku 5.5, on its own). With the calculator: right.−$26,002
  • The calculator’s answers held. With the calculator, all 3 models made the right decision on every question, with every number within 2% of the key, and never went against it. They also thought far less: about 3k tokens for Sonnet instead of 37k.
  • Hard on purpose. Code checks that no single tax rate or skipped rule lands within 2% of the answer. The answers are rechecked by separate programs (203 of 203 agree; Technical details).
  • Still small. One try per question, three models. The intervals are wide on purpose: this shows the gap, not a ranking.

A separate result, not in the table above

GPT-6 Luna through Codex, without a calculator: it said it couldn’t

Questions
the same 11 households, trap included
Model
GPT-6 Luna through Codex (the model on my ChatGPT subscription, run in OpenAI’s Codex app), with no calculator, code or web
Settings
medium effort, no length limit, one try each; a different setup from the three models above
Graded
the same way: the right decision, and every number within 2%

Result. It declined all 10 questions that had an answer, typically like this: “A reliable answer requires a full multi-year projection of taxes, RMDs, premiums, and account balances.” (RMDs are the withdrawals the IRS requires from retirement accounts.) It also rightly declined the trap, so it scored 1 of 11.

Why this matters. That’s an honest answer, and it’s the case for a calculator. A real decision needs a 25-year projection. GPT-6 Luna declined every one; some other models answered some of these correctly without a calculator (Claude Sonnet 5.5 got 7 of 10), and with one, all three models tested got every decision right.

Short test: leave a fact out. Does it ask, or guess?

Questions
the same household questions (not the trap), each with the one deciding fact left out: a Social Security amount or an IRA balance. The file says something may be missing, not what.
Model
DeepSeek V4.1 Flash, on its own and with the calculator
Settings
as above; the model may ask a scripted client up to three questions before answering
Graded
the same way, against the full facts

Result. With the calculator, DeepSeek asked for the missing fact 9 times out of 10.

ModelFully rightAsked for the missing factCost per question
DeepSeek V4.1 Flash, on its own
40% · 2/512–77
5 of 5$0.01
DeepSeek V4.1 Flash + calculator
90% · 9/1060–98
9 of 10$0.01

One question per family (the trap has nothing to leave out). The alone run was stopped after 5 questions: answering without the calculator took 10–15 minutes each. Fully right is graded against the full facts, by the same grader as above. The shaded band behind each bar is the range the true score likely falls in, given so few questions.

What it asked (word for word)

  1. 1“Is Frank receiving Social Security now—and if so, what is his monthly benefit and at what age did he claim? If he is not yet claiming, what is his monthly benefit at full retirement age and when does he plan to claim?”
  2. 2“The options call for converting from Glen's traditional IRA, but the account list you gave me only has Maria's traditional IRA ($70,000) and Glen's Roth IRA — there's no Glen traditional IRA or 401(k) listed. …”
  3. 3“What is Ken's traditional IRA balance as of December 31, 2025, and is any of it after-tax money (i.e., does he have a Form 8606 basis)?”

Why guessing can’t pass

  • Every tested guess fails. Before a question is used, the engine answers it 4–5 times with the missing fact guessed (none, half, 75%, 125%, 150%). All 69 of 69 questions grade wrong under every guess.
  • The client never volunteers. It’s code, not a model: it answers only a question on the missing fact’s topic, and says nothing new to anything else.
  • Checked with stand-ins. A stand-in that asks the right question gets 69 of 69 fully right; one that guesses gets 0 of 69; one that asks vague questions never gets the fact (0 of 69).

See it work

Watch the calls, or see it in a chat

For the technical reader

The calculator is an MCP server (the Model Context Protocol, the standard way to plug tools into AI models). The benchmark runners and the chat add-on call the same code. The first test offered federal tax, taxable Social Security, required withdrawals, Social Security benefits, Medicare premiums and tax tables. The November set offered those plus a year-by-year scenario runner; the Roth-conversion solver and the chat front door were left out to keep the instructions short.

Calculator test 1: from calculator 1.0 (just the number) to 1.1 (the number plus the rule), the same questions both ways; exact McNemar test, p < 0.001. Shaded bands in every table are 95% intervals (Wilson).

Calculator test 2 (the 42 questions in the newer set; low effort; one run each way; graded by code within $2–5). Token budget: 4,000 output tokens per turn, up to 8 turns. 10 of 336 calculator replies used more than 4,000 in total, the longest 7,135.

DeepSeek V4.1 Flash with the calculator against Claude Sonnet 5.5 on its own, the same 42 questions, first run each: 10 only DeepSeek got right, 3 only Sonnet got right; exact McNemar p ≈ 0.0923. Higher observed score, not a significant difference. DeepSeek alone → with: 21 fixed, 0 broken, p < 0.001. Claude Haiku 5.5: 8 fixed, 2 broken, p = 0.1094.

First test: Claude Haiku 5.5 through Anthropic’s API. November set: every model through OpenRouter (a service that relays requests to many models), each pinned to one host. The calculator prices each choice; it never picks investments or products.