Technical details
Technical details
The plain answer
The planning engine (the program that works out the right answers) and a second, separately written program both compute every answer in the first test, and they must agree within $2. Outside tax software built by other people recomputes the tax answers. Another engine, written by a separate AI agent that was never allowed to see the planning engine’s code, agrees within $1 on every one of 4,000 random households, and every question in the newer set must match it. Two human checks are still owed: a CPA’s sign-off and a person spot-checking the AI judge in the wrong-facts test.
That’s the whole idea. The rest of this page is the detail: every test, the ChatGPT model, the checks, the statistics, the choices and mistakes behind the build, and the words used on this site and the limits. How I built it, in plain English, is on How I built it.
Every test
October and November used different settings, so compare within a test, not across. October: high reasoning effort, long replies. November: low effort, replies up to 4,000 tokens.
| Test | When | What it asked | How many | Which AIs |
|---|---|---|---|---|
| First test | Oct | One-rule questions, then several rules at once | 140–210 each, of 2,800 | six models |
| Newer, harder set | Nov | Harder questions where several rules meet | 42 | 11 models |
| GPT-6 Luna through Codex | Nov | The same questions, through my ChatGPT subscription | 42 + 139 from the first test | GPT-6 Luna |
| Calculator test 1: number vs rule | Oct | Does a calculator help if it shows only the number, or the rule too? | 50 + 50 held back | Claude Haiku 5.5 |
| Calculator test 2: who it rescues | Nov | Which models the calculator lifts | 42 | 8 models |
| Calculator test 3: whole-household decisions | Oct | Real choices over 25 years, like when to claim Social Security | 11 | 3 models |
| Wrong facts | Nov | Client files with one fact that can’t be right | 30 + 40 that were fine | 11 models + GPT-6 Luna |
All in 2026. Price groups, by what each model costs to use: top-priced: GPT-6.1 Sol, Claude Sonnet 5.5, Gemini 3.1 Pro · mid-priced: GPT-6.1 Terra, Gemini 3.8 Flash, Claude Haiku 5.5 · cheap and free: DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Llama 4 Maverick, Mistral Small.
About ChatGPT: GPT-6 Luna through Codex
The regular ChatGPT app uses GPT-6 Luna, a different, cheaper model than GPT-6.1 Sol (the one that got 98% of the hardest questions). I ran Luna through OpenAI’s Codex app (a ChatGPT-subscription tool for running tasks) with code and web turned off, in small batches, so it’s close to, but not the same as, typing into the ChatGPT window.
It also ran with more time to think and no length limit, so its 17% on the hardest questions isn’t ranked with the other models. How the run was done, and the first attempt that was thrown out, is on AI on its own.
The answer key, in plain words
Every number question has one exact answer, usually a dollar amount. The planning engine, a program I directed AI agents to build, computes it from the 2026 rules: IRS forms and publications, the Social Security Administration’s operating manual, and Medicare’s premium tables.
For those number questions a script does the grading, so nobody’s opinion is involved, including mine. The wrong-facts test is the exception: whether a reply flagged a planted mistake takes judgment, so an AI judge decides, checked as described below.
When AI models disagree with the key, the published source decides, not the models. In the first test, models from several companies agreed with each other and against the key on the new senior deduction. The IRS form, and separately PolicyEngine, showed the key was right.
The checks, with their scores
Each card says what was checked and how far the check reaches.
A separate AI agent wrote another engine, in a different programming language, from the written rules alone, and was never allowed to read the first. Both run on random households; every number for every year must agree within $1. Its first run found a real bug in the main engine. Every question in the newer, harder set also has to match it before it can be used.
PolicyEngine-US, open-source tax software written by other people, recomputes every tax answer in the first test, all within $2, the new senior deduction included; and 400 / 400 on the held-back questions (private ones never published). It doesn't model required withdrawals, Social Security claiming or Medicare premiums, so those rest on the published tables and the second engine.
Every option in every public whole-household question is rerun by the second engine, and its 2026 federal tax return by PolicyEngine, before the question can be used.
Before any model used the calculator, a script checked that calling it correctly reproduces every number answer in the first test: the 2,200 public questions that have a number, plus 550 held-back ones. The other 600 of the 2,800 public questions are compliance questions, answered with sentences or flags rather than a number, so there is nothing for a calculator to compute.
What checked the AI’s work
- The engines above. A second engine written by a separate agent that never read the first, and outside tax software written by people, recompute the answers.
- A second judge from another lab. In the wrong-facts test, an AI judge (Claude Haiku 5.5) reads each reply and records whether it flagged the problem. A judge from OpenAI (GPT-6 Luna) re-read 176 of those replies and agreed on 97.7%. Where a reply plainly named the problem but the judge marked it as missed, the lead AI agent corrected the label (18 replies). Two AIs agreeing, or one AI correcting another, is not the same as a person checking.
- The GPT-6 Luna run. GPT-6 Luna through Codex (the model on my ChatGPT subscription): its replies were graded by the same answer keys and the same scripts as every other model. On wrong facts the same judge (Claude Haiku 5.5) used the same instructions, and the lead AI agent read every reply the judge marked as not catching the mistake (12 replies; 0 labels changed). The replies the judge marked as caught haven’t been re-read yet. A first attempt was thrown out because most replies were a bare final number with no reasoning, which isn’t how ChatGPT answers a person; the details are on AI on its own.
- An outside audit. On November 15, 2026, OpenAI’s GPT-6.1 Sol read the whole project, without changing anything, and looked for errors. It found 12, all fixed on this version; the list is in the report.
- Still owed. A CPA (a licensed tax accountant) hasn’t signed off on the answer key yet; worked solutions are ready for one to check with a calculator. That would be a human check on top of the independent programs, which already agree. And no person has yet spot-checked the AI judge’s labels in the wrong-facts test.
What every first-test question must pass
- Published examples. Cases whose answers come straight from IRS, Social Security and Medicare tables must match exactly.
- Two independent programs. The planning engine and a second, separately written program compute every answer and must agree within $2. They agree on all 2,800 questions in the first test.
- Rules that must always hold. Thousands of random households test things like “more income never lowers tax” and “waiting a month never lowers a Social Security check.”
- Dataset health. No duplicates, and no task where one answer is common enough to guess.
- An outside tax model, where it reaches. PolicyEngine-US recomputes the first test’s tax answers and the 2026 federal returns in the whole-household decisions. It doesn’t cover required withdrawals, Social Security claiming or Medicare premiums.
The newer set’s questions must match the second engine before they’re used; its Arizona state-tax questions have no outside check. The human checks above are still open, so these are the automatic checks, not a sign-off.
For the technical reader: statistics, scripts and where each check lives
- Every accuracy carries a 95% interval: the range the true score likely sits in. Small runs show wide ranges on purpose.
- Comparisons are paired. “Alone vs. with the calculator” asks the same model the same questions and counts how many flipped each way, then uses McNemar’s test (a standard check of whether that many one-way flips could be luck).
- Failures are their own bucket. A reply that ran out of tokens is published and counted as not right, never quietly dropped.
- Mistakes are priced. Each wrong number in the first test is also scored in dollars a year off, so a ten-cent miss and a large one don’t count the same.
- Hosts are pinned. Every outside model runs on one named host with data collection off; the private set never goes to free endpoints.
- A held-back set. Held-back questions are private questions, made from a secret seed (a starting number for the generator), that pass the same checks and are never published, so a model that trained on the public set still meets new ones, and nothing is tuned on them.
- What rebuilds the newer set.
v2/grade.pyregrades it from saved replies;v2/build_hard.pyandv2/build_bad.pyrebuild its questions and stop if the engine and an independent check disagree. - Same model, two routes. Claude Haiku 5.5 answered the same 140 questions directly and through OpenRouter (a service that relays requests to many models): 77% and 77%. No difference detected in this check, which is not proof the route never matters. (checks/HARNESS.md)
- One script rebuilds the first test. reproduce.sh (a script anyone can run from the code) re-checks the first test’s answer key, regrades every saved first-test reply, rebuilds its scores and the calculator runs on its 50 hardest questions, and compares them with the published copy; any difference fails it. It doesn’t regenerate the questions or cover the newer set.
- Where each check’s record lives. Another engine, written without seeing the first: engine repo: verify/COMPARE.md (rules v1.4.1, 2026-10-23) · Outside tax code: checks/POLICYENGINE.md · Whole-household decisions, recomputed: checks/GAUNTLET.md · The calculator can answer everything: checks/tools_check.py (README log, 2026-10-21). (“engine repo” is the planning engine’s own code repository; verify/COMPARE.md there is the two engines’ comparison report.)
Behind the build
The main choices I made, the mistakes that got caught, and when each piece was built. The plain-English version is on How I built it.
The main choices I made, and why
- When the AI ignored the calculator, I changed what the calculator says back. I could have written a stronger prompt or used a bigger model. Instead the calculator now shows the rule and the IRS form line next to the number, which fixes the cause. I tested that on one model; how it carried over to others is on With a calculator.
- A few models, cheapest first, inside small budgets. October: two Claude models plus four low-cost models from other companies. November: 11 models from seven companies, plus the ChatGPT model tested separately. The goal was to show the gap and why it happens, not to rank every model.
- “Better plan” means more money left after taxes at the end. Not lowest tax or highest income. That’s what the family actually keeps.
- When two plans are close, say so. If two plans end within a rounding error of each other over 25 years, picking a “winner” would be false precision.
- The chat version keeps nothing people type. People enter real finances, so nothing is saved. The cost is no usage numbers, so quality is measured with made-up clients instead.
Mistakes the AI made while building, and what caught them
- A bug in the calculator. The second calculator disagreed on 3 of 2,000 households. One was a real error in mine: a 2027 tax threshold. After the fix, the two agreed on all 4,000 households in the next comparison.
- A Medicare premium off by one level. Income exactly at a cutoff was put one level too low. A check found it and it was fixed.
- Unfair questions. Reading the first wrong-facts replies, a model rightly asked about a fact the file never gave. 14 files were unclear. I fixed them, asked again, and left the old replies out of the scores.
- A spending cap that leaked. One run went $0.39 over its cap because requests already in progress weren’t counted. The cap now sets aside the most each request could cost.
- The first ChatGPT-model run cut corners. Asked to do dozens of questions at once, it wrote bare final numbers instead of answering each one like a chat would. I threw that run out and redid it in small batches. Lesson: when an AI does work in bulk, check how it did the work, not only the score.
- Grading and wording errors. The outside audit found replies that ran out of room counted the wrong way, some judge labels that missed a model naming the mistake, and sentences that claimed more than the data showed. All fixed; the report lists each one. On the wrong-facts test the lead AI also corrected 18 of the judge’s labels, every change listed; a person still needs to spot-check them.
Timeline
- Oct 15: the calculator (federal tax, Social Security, required withdrawals, Medicare premiums), checked for rule errors the same day.
- Oct 20: the first test: questions with a provable answer key, outside tax software as a referee, and the first models graded.
- Oct 21: the calculator as a tool AI can use. Version 1.0 gave only the number and got ignored; 1.1 showed the rule and the form line, and the AI mostly stopped arguing. Also the second calculator, written from the rules alone.
- Oct 22: whole-household decisions over 25 years, and the calculator inside Claude and ChatGPT, tested with made-up clients first.
- Nov 14: the harder questions and the wrong-facts test, after I saw the Claude models had nearly maxed out the simple questions and every fact I’d given them was true.
- Nov 15: an outside audit by OpenAI’s GPT-6.1 Sol. It found 12 errors, all fixed, and the site was rewritten to read simply.
Words on this site
- One rule
- One tax or retirement rule, like this year’s tax brackets or a required IRA withdrawal.
- Several rules at once
- Taxes, Social Security and Medicare affecting each other, the way they do in a real household.
- The planning engine
- The program I had built for the 2026 IRS, Social Security and Medicare rules. It computes the answer key, and it is the calculator the AI models could call.
- Best
- On scores, right most often. On wrong facts, catching the most mistakes without refusing files that were fine.
- Caught
- On wrong facts, the reply recognized the planted mistake and stopped to ask, with no number.
- AI judge
- Another AI model reading each wrong-facts reply to decide whether it named the mistake. No person has checked its labels yet.
- Held-back questions
- Secret questions never used while building, kept only to check that a fix is real.
- CPA
- A licensed tax accountant. A CPA’s sign-off on the answer key is still to come.
- PolicyEngine-US
- Outside tax software built by other people. It rechecks the first test’s tax answers.
- Price groups
- By what each model costs to use: top-priced, mid-priced, and cheap and free (listed under Every test).
The limits. Small samples; on the wrong-facts test, an AI judge decided whether a model named the mistake; no CPA sign-off yet.