ALMOST INTELLIGENT : Don’t let a chatbot’s confidence trick you into taking financial risks

Consider the following scenario. Suzy is 63, recently retired and trying to decide when to start receiving social security and how to manage her retirement savings to minimise the tax hit. She opens an AI chatbot, types in the details and gets a calm, well-organised and confident answer: claim now, convert this much, here is the reasoning.The chatbot sounds authoritative and even shows its work. So Suzy follows its guidance and never calls a financial planner. Maybe the advice was fine. But maybe it quietly ignored Suzy’s spouse being younger than her and in poor health, which can flip the social security maths. It also may have overlooked that the retirement savings plan conversion it suggested would push Suzy into paying higher medical aid premiums two years later.Suzy will not find out for a long time, if ever, whether this guidance was right for her. And the AI will never call back to say it was unsure.Suzy isn’t an exception. AI chatbots have entered everyday life with remarkable speed: a 2025 Pew Research Center survey found that 34% of American adults and 58% of those under 30 have used ChatGPT, roughly double the share two years earlier.A growing number are asking AI about money, and some are getting burned. According to a 2025 survey of 2,000 US adults by Pearl.com, a professional services platform, 19% said they lost more than $100 by following financial advice from an AI chatbot. Among Gen Z investors, that figure rose to 27%.These aren’t hypothetical risks. People are already paying for answers about their money that are confident – and wrong.As a finance professor who has been closely watching the spread of AI into personal finance, this is the part of the AI story that worries me most. And it’s not the part you usually hear about.We argue about AI the wrong wayThere are two seemingly opposite complaints about AI. One is that people trust it too much, treating a chatbot like an oracle, a tendency researchers call algorithm appreciation. The other is that people don’t trust it enough and dismiss its useful tools, a tendency known as algorithm aversion.I argue that these are actually two sides of the same coin, and what decides which side you see is whether you can tell when the AI is wrong.When an AI fails in an obvious way, you notice and lose confidence. So you’re more likely to seek a professional or another human you trust sooner than you otherwise would. That is the safe failure.The dangerous failure is the opposite. The answer is fluent, confident – and wrong. You have no way to catch it, so you keep managing the problem yourself long past when you should have asked for help.The trouble is that with money, the second kind of failure is the common kind.Fluent AI is not an accurate AIThree things make financial advice especially treacherous for AI. First, fluency is not accuracy. People naturally read a confident and well-articulated answer as competent.But how polished an answer sounds tells you almost nothing about whether it fits your situation or the accuracy of the proposed solution. A chatbot can be word-­perfect and still be wrong about your taxes, because your taxes depend on details it never asked about.Second, AI is least reliable exactly where the stakes are highest. AI tools are good at routine and general topics: what a tax-free savings account is, how compound interest works, the difference between a stock and a bond. But financial life is full of rare, complicated, one-time decisions such as exercising stock options, understanding the alternative minimum tax, drawing up a divorce settlement, and so on.I made a similar argument three years ago about AI trading on Wall Street. Because market crashes are rare, there’s little data for AI to learn from, so it can be most confident exactly where it is least informed.That worry hasn’t faded. Market watchers now caution that AI trading bots are creating fresh financial risks, and that same blind spot applies to your personal finances. Researchers call this uneven competence a “jagged frontier” – reliable with common cases, but unreliable for unusual ones. And in finance, the unusual cases tend to be the expensive ones.Third, you often can’t check the work. Financial advice is what economists call a “credence good”, like a mechanic’s diagnosis or a doctor’s recommendation. You often can’t tell whether the advice was good, sometimes for years.A mistaken tax move may not surface until an audit. A poor retirement annuity or living annuity drawdown strategy may not bite until the stock market slumps. Without quick feedback, the wrong-but-confident answer never gets corrected. This is why the Pearl numbers mentioned earlier are probably an undercount, since they capture only losses people noticed.Quiet failure is the one to watchNotice that the real harm in Suzy’s story isn’t a single dramatic mistake. It’s that a confident answer made Suzy feel no need to call a professional, so it never happened.The danger is not so much that you act on bad advice, but that you never seek good advice. The smoother and more reassuring the tool, the easier it is to stay in do-it-yourself mode past the point when you need outside help.Who is most at risk? In a study of a large robo-advising platform in India, coauthor Vishaal Baulkaran and I found that its users skew young, are predominantly male and tend to be smaller retail investors and professionals. And new account sign-ups rise during periods of high market ­volatility.In other words, the people leaning hardest on automated advice match that 27% figure among those Gen Zers who lost more than $100 while using a chatbot for financial advice. They reach for it just when markets turn turbulent and a wrong move is most costly.There is also an incentive worth naming. In my new analysis, I argue that a tool that earns its revenue by holding your attention has a good reason to sound confident and helpful: confidence keeps you on the platform.The catch is that the user it retains this way is sometimes the one who should have been handed off to a human. A system tuned to keep you engaged isn’t the same as one tuned to protect your financial future, and the two can point in different directions.The disruption is already under way, as wealth managers face what Bloomberg has called a chatbot reckoning. A single, new AI tax tool recently sent wealth management stocks sliding as investors bet that automated advice will eat into the business.How to be smart about using AIThese findings don’t mean that people should avoid AI for money advice. Used well, these tools are a valuable and free financial educator. This is also not to say that a financial adviser always has the right answers. As with finding any kind of specialist, it’s important to do your research first and make sure the person meets the kind of ­criteria laid out by financial regulators. Fee transparency is also crucial.But if you do turn to AI, the skill is knowing where to draw the line. Treat AI as a starting point, not a verdict. It’s excellent for learning concepts, drafting questions and getting oriented before a meeting. It can teach people the vocabulary to have a smarter conversation with an expert.But watch out for the signals that you have left its comfort zone and entered the territory where AI is weakest and a confident answer is least trustworthy. The red flags are tax consequences, anything irreversible and anything that turns on the specifics of your situation rather than a general rule.Estate questions, the drawdown of retirement savings, business structure and major one-time transactions all belong in this category. Those are the decisions that call for bringing in a human, such as a certified financial planner.And remember, confidence isn’t competence. When the answer about your money sounds most polished and most certain, that’s not a reason to relax. On the hardest questions, that smooth confidence is exactly the signal that you should pick up the phone and talk to an expert. DMThis article is republished from The Conversation under a Creative Commons license. Read the original article.This story appeared in our weekly DM168 newspaper, available countrywide for R35.