# For AI labs

Source: https://plannerbench.org/labs.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt

**What this is:** A test set of retirement planning questions where every answer is exact and checked. The answers come from a calculator built on 2026 IRS, Social Security, and Medicare rules, confirmed by a second calculator written separately and by outside tax software. This is an area where real people are already asking AI for advice and acting on it.

## What it shows

- The top models are nearly maxed out. GPT-6.1 Sol scored 98% on the hardest set.
- Cheap models aren’t. They scored under 30%, and GPT-6 Luna scored 17%.
- None of them reliably catch wrong facts. Even the best models missed 20–27% of planted errors and gave advice anyway.

## Why it’s useful for training

Every number answer can be graded by a script, so there’s no human or AI judge in the loop for those. I think a cheap model could close most of the gap quickly with reinforcement learning (training it by rewarding correct answers). The wrong-facts files add a second thing to train: knowing when to stop and ask instead of answering.

## What I’d build next

Subtler, more realistic wrong facts. A CPA sign-off on the answer key. A person checking the AI judge’s labels.

[How I built it →](https://plannerbench.org/built.html)  [Technical details →](https://plannerbench.org/method.html)

---

PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com)

Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown.

Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice.
