Visa shopping benchmark: 54% of failed runs still chose to…

The tests expose a product-matching problem for shopping agents and a catalog-preparation task for small retailers.

Visa released its Agentic Commerce Benchmark November 5, testing 22 AI models from 12 labs on 15 shopping challenges covering televisions, laptops and personal-care appliances.

The Next Web reports that the models passed 57.2% of runs and failed 42.8%, and that in 54% of failed runs, the agent still decided to buy despite missing requirements or choosing an unsuitable product.

Fortune describes shoppers withholding some needs until asked.

According to The Next Web, Visa says agents must uncover every shopper requirement and make the correct buy-or-not decision, including declining when nothing fits. Sourcing Journal’s report carried by Yahoo Finance adds that, to prevent lucky guesses, agents also had to uncover enough information to justify their decisions.

The Next Web reports that the five models uncovering the most hidden buying factors passed 70.2% of runs, against 44.4% for the five uncovering the fewest.

Visa reports three tries per scenario and a 34.2% figure for cases. Its results show that the best-performing models passed 82.2% of attempts; the lowest scored 28.9%.

Google and Meta agents tied for first, with a Claude agent third at 76% and an OpenAI agent sixth at 64%, Fortune reports.

The Next Web reports pass rates of 67.1% for the five most expensive models per run and 50.2% for the five cheapest.

Visa says checkout, payment and order placement are planned extensions. It also plans additional scenarios and product categories, according to The Next Web.

According to The Next Web, the benchmark is publicly available on Visa’s Payments Frontier site, and Visa has shared it with leading model providers for feedback.

“We’re not there yet from a trust or technology perspective,” Rubail Birwadker, Visa’s global head of growth, told Fortune.

In Visa’s television scenario, the shopper mentions a relative’s Samsung recommendation and a membership with a third-party supplier, according to Sourcing Journal’s report carried by Yahoo Finance. The agent must uncover the budget and other requirements.

Fortune identifies requirements including compatibility with a child’s gaming console, a 30-day return window and Dolby Vision support.

For small-store owners, the inference is to clarify product fit, attributes and policies, not a proven accuracy or sales remedy. Descriptions alone can’t fix agents failing to ask necessary questions.

Visa shopping benchmark: 54% of failed runs still chose to buy

Sources