For financial firms
Your clients are already asking AI. Even in a retirement community where most people have an advisor, prospects tell me they ask ChatGPT first. One told me a 529 question felt too dumb to ask me. They trust it because it seems right and Google’s AI agrees.
What I found
- On single rules, good models are almost always right.
- On real retirement questions, where taxes, Social Security, and Medicare interact, it depends heavily on the model. The cheap and free models people actually use scored under 30%.
- When a client gets their own facts wrong, most models give confident advice anyway.
What fixes it, without paying for the most expensive model
- Ask the questions that matter in a set order, so nothing gets assumed.
- Give the AI a calculator that shows the rule behind every number.
- Have it flag facts that don’t make sense before it answers.
What a firm could do with this
- Know which questions need an advisor. The test data shows where AI does well and where it breaks.
- Offer a “check your AI answer” tool on your website. Someone brings the question they asked ChatGPT. The tool collects the facts that matter, flags what needs a professional, and asks if they’d like an advisor to reach out. People are using AI either way, so the firm might as well offer the version that checks itself.
- Keep costs down. Structure and checks let cheap models do the job, which matters across thousands of advisors and their clients.