For AI labs

What this is: A test set of retirement planning questions where every answer is exact and checked. The answers come from a calculator built on 2026 IRS, Social Security, and Medicare rules, confirmed by a second calculator written separately and by outside tax software. This is an area where real people are already asking AI for advice and acting on it.

What it shows

  • The top models are nearly maxed out. GPT-6.1 Sol scored 98% on the hardest set.
  • Cheap models aren’t. They scored under 30%, and GPT-6 Luna scored 17%.
  • None of them reliably catch wrong facts. Even the best models missed 20–27% of planted errors and gave advice anyway.

Why it’s useful for training

Every number answer can be graded by a script, so there’s no human or AI judge in the loop for those. I think a cheap model could close most of the gap quickly with reinforcement learning (training it by rewarding correct answers). The wrong-facts files add a second thing to train: knowing when to stop and ask instead of answering.

What I’d build next

Subtler, more realistic wrong facts. A CPA sign-off on the answer key. A person checking the AI judge’s labels.

How I built it → Technical details →