How I built PlannerBench
I’m a financial advisor, not a programmer. I built all of this by telling AI coding tools what to build (mostly Claude Code) and checking what they made. A lot of it happened at 3:45 in the morning before the gym, on the stair stepper, or while I pushed my daughter in a stroller and worked on my laptop remotely from my phone. Running the latest full round of tests cost $4.51.
It started as a different project
I wanted to see if I could build my own version of the planning software advisors use, and fill it in by talking instead of typing everything in by hand. Getting the retirement math right turned out to be hard. Taxes, Social Security, and Medicare all affect each other, and making my version correct took a lot of work.
That made me wonder how the AI models do on this math on their own. In my testing they made a lot of mistakes.
Then it got personal
I meet a lot of my prospects by knocking doors. Many of them already have an advisor and ask ChatGPT their money questions anyway. When I ask how they know the answer is right, they say it seems right. They Google it, and the top result agrees, but that top result is usually Google’s AI. One prospect I’d been following up with told me they had a question about a 529 versus a UTMA account for their kid. They felt it was too dumb a question to ask me, so they asked ChatGPT. That’s fine for a simple question. On more complicated stuff, I don’t think people realize what risks they’re taking.
So I wanted an actual answer: how do you check whether the AI is right?
The parts, in plain English
The calculator (planning engine). A program that knows the 2026 rules for federal taxes, Social Security, Medicare premiums, and required withdrawals from retirement accounts. Give it a household and it works out the exact numbers. Think of a very focused TurboTax for retirement.
The answer key. Thousands of retirement questions, each with one exact answer, usually a dollar amount, worked out by the calculator.
Checking the checker. If the calculator is wrong, the whole test is wrong. So I had a separate AI write a second calculator from the written rules only, without ever seeing the first one’s code, in a different programming language. On 4,000 made-up households, the two have to agree within $1. The first time I compared them, they disagreed on 3 households, and that’s how I found a real bug in my calculator. Free outside tax software called PolicyEngine-US also rechecks the tax answers. When anything disagrees, the IRS, Social Security, or Medicare publication decides. A vote between AIs doesn’t.
The test. I gave the same questions to 11 AI models from 7 companies, plus GPT-6 Luna. A script grades the number answers, so nobody’s opinion decides what’s right, mine included.
Secret questions. I kept some questions locked away and never used them while I was building. It’s like not letting a student see the final exam. If a fix works on questions I never looked at, it’s a real fix and not me accidentally teaching to the test.
Giving the AI the calculator. My first idea was to let the AI use my calculator. The surprise was that it often ignored the right number and argued with it. What fixed it was changing what the calculator handed back. Along with the number, it now shows the rule it used and the IRS form line. After that, Claude Haiku went from 40% to 100% on 50 hard questions, and 96% on the 50 secret ones.
The wrong-facts test. At the doors, people give wrong information all the time: the wrong birthday, transposed digits, a guess at what they spend. So I made 30 client files that each had one impossible fact (like a birthday that makes a retiree 14) and 40 normal files. The question was whether the AI notices and asks, or just gives advice anyway. An AI graded these replies, and an AI from a different company re-graded 176 of them and agreed 97.7% of the time.
Having another AI audit everything. Since AI built most of this, I had OpenAI’s top model review the whole project, looking for mistakes and claims that went too far. It found 12 problems. I fixed all of them and published what they were.
What I learned
The models mostly fail by skipping steps. They assume things instead of asking, and they lean on what they remember from training instead of the current rules. When you tell them what they need to ask and what to think about before they answer, they do much better. Even cheap models get pretty close.
The expensive models do great on their own. GPT-6.1 Sol scored 98% on the hardest questions. But it sits behind a $20 a month subscription, and paying per question gets expensive fast across thousands of advisors and their clients. GPT-6 Luna, the cheaper model behind the regular ChatGPT app, scored 17% on its own. So for most people, the answer isn’t a smarter model. It’s asking the right questions in a set order, giving the model a calculator that shows its work, and having it flag things that don’t make sense.
Where I want to take this
I believe the best AI models will soon be able to do financial planning at the level of the best human advisors. People are already asking AI for advice, and before long it will be hard to tell the best AI advice apart from the best human advice. Right now the free and cheap models aren’t great at this, but the best models already do very well, at least on my small test. Eventually the free models will be as good as today’s best paid ones, and the best paid models will be at a level that’s hard to picture.
That’s why this matters now. Checking whether AI gets retirement math right, and making it better at it, needs to happen at a much bigger scale than one guy can do on his own. I’d like to take PlannerBench further as part of a team, with real resources behind it.
The limits
This is one person on a small budget, testing for a few dollars a round. The samples are small. No CPA has signed off on the answer key yet. An AI, not a person, graded the wrong-facts test. I tested the ChatGPT model through OpenAI’s Codex app with code and web access off, which is close to the ChatGPT app but not the same. All the details are in Technical details.
Who did what
I picked the problem, decided what counts as a correct answer, set the rules, and approved every test and every dollar spent. AI coding tools did the building and ran the tests. Separate AIs, outside tax software, and another company’s model checked the work.