The AI Consumer Index (ACE)

TL;DR

The paper introduces the AI Consumer Index (ACE), a benchmark to evaluate frontier AI models on everyday consumer tasks across shopping, food, gaming, and DIY, highlighting current model limitations.

Contribution

It presents the ACE benchmark with a novel grading methodology and evaluates leading models, revealing significant gaps between AI performance and consumer needs.

Findings

Top models score around 56%, with lower scores in shopping.

Models often hallucinate key information like prices.

Significant gap remains between AI capabilities and consumer requirements.

Abstract

We introduce the first version of the AI Consumer Index (ACE), a benchmark for assessing whether frontier AI models can perform everyday consumer tasks. ACE contains a hidden heldout set of 400 test cases, split across four consumer activities: shopping, food, gaming, and DIY. We are also open sourcing 80 cases as a devset with a CC-BY license. For the ACE leaderboard we evaluated 10 frontier models (with websearch turned on) using a novel grading methodology that dynamically checks whether relevant parts of the response are grounded in the retrieved web sources. GPT 5 (Thinking = High) is the top-performing model, scoring 56.1%, followed by o3 Pro (Thinking = On) at 55.2% and GPT 5.1 (Thinking = High) at 55.1%. Model scores differ across domains, and in Shopping the top model scores under 50\%. We find that models are prone to hallucinating key information, such as prices. ACE shows a substantial gap between the performance of even the best models and consumers' AI needs.