AI agents went shopping for TVs and laptops—and messed up nearly 50% of the time, study shows

As personal AI agents like Muse or Claude become part of daily life, more people are asking them to carry out shopping tasks. But there’s a problem: the agents aren’t very good at shopping. In a first-of-its-kind study, Visa instructed 22 different agents to purchase three everyday items—a TV, a laptop and a hair appliance—and, while some agents did significantly better than others, the overall success rate was only around 57%. The results of the study are available on a new website by Visa that shows the outcome of 15 separate shopping expeditions by each of the 22 agents. Unsurprisingly, the agents powered by more cutting edge AI models fared best, with the Aragon 4 version of Google’s Gemini and the Wonder version of Meta’s Making notching the highest scores. Both agents made a successful purchase 82% of the time. A Claude agent came in third with a 76% success rate, while an OpenAI agent finished 6th with a 64% success rate. The diagram below from Visa shows the performance of all 22 models. The vertical axis scores the agents’ success rate, while the horizontal one reflects the cost to run each shopping inquiry. (Visa notes that the five most expensive models succeeded on average 67% of the time, while the cheapest five scored 50%). The agents’ struggle to make a successful purchase is rooted in part in a simple reality: The motivations behind any given purchase are often complicated and, in many cases, consumers lack information and are not certain what they want. Visa’s experiment sought to capture this reality by, in the case of TV shoppers, assigning a list of attributes they needed in a television—for instance, compatibility with a child’s gaming console, a 30-day return window, and a model with Dolby Vision. As is the case with real world consumers, the purchaser (who in this case was also an agent) would volunteer some information, but not all of it. The upshot is that, for an AI agent to succeed in Visa’s experiment, it had to interact with the shopper in a way that would elicit the missing information in a reasonable time. Success amounted to buying a TV that matched the shopper’s needs but, in many cases, the agents bought the wrong product, or simply failed to suggest something suitable at all. “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate,” Visa’s Global Head of Growth, Rubail Birwadker, told Fortune. Birwadker noted that agent-driven shopping is still in its early days, but that it is advancing rapidly. He added that the focus of Visa’s experiment—product discovery—is just one element of this new type of commerce that retailers and AI companies are trying to figure out. For AI agents to enter the shopping mainstream, companies will also have to sort out smooth and secure payments—a challenge that Visa and its rival Mastercard are already seeking to address by educating merchants and building tools to facilitate so-called agentic commerce. There are also new and unresolved legal issues when it comes to AI agents going shopping. Most notably, it’s still unclear who should bear the liability when agents make rogue purchases or commit fraud. According to Birwadker, these questions will get resolved over time in the same way that an earlier generation of consumers and merchants learned to use ecommerce—he added, though, that agent-based shopping is going to be a very different experience than how we shop online today. In announcing the results of its agent shopping experiment, Visa also rolled out a website that allows consumers to interact with the agents used in its survey.