Beyond Benchmarks: How Users Evaluate AI Chat Assistants

TL;DR

This study compares user satisfaction, motivations, and frustrations across seven major AI chat platforms, revealing that users treat these tools as interchangeable utilities and that platform specialization sustains competition.

Contribution

It provides the first systematic cross-platform survey comparing user perceptions and motivations, highlighting the importance of platform differentiation and user behavior patterns.

Findings

Top platforms have similar satisfaction ratings despite differences in resources.

Most users switch between multiple platforms with low switching costs.

Different platforms attract users for distinct reasons, supporting market diversity.

Abstract

Automated benchmarks dominate the evaluation of large language models, yet no systematic study has compared user satisfaction, adoption motivations, and frustrations across competing platforms using a consistent instrument. We address this gap with a cross-platform survey of 388 active AI chat users, comparing satisfaction, adoption drivers, use case performance, and qualitative frustrations across seven major platforms: ChatGPT, Claude, Gemini, DeepSeek, Grok, Mistral, and Llama. Three broad findings emerge. First, the top three platforms (Claude, ChatGPT, and DeepSeek) receive statistically indistinguishable satisfaction ratings despite vast differences in funding, team size, and benchmark performance. Second, users treat these tools as interchangeable utilities rather than sticky ecosystems: over 80% use two or more platforms, and switching costs are negligible. Third, each platform attracts users for different reasons: ChatGPT for its interface, Claude for answer quality, DeepSeek through word-of-mouth, and Grok for its content policy, suggesting that specialization, not generalist dominance, sustains competition. Hallucination and content filtering remain the most common frustrations across all platforms. These findings offer an early empirical baseline for a market that benchmarks alone cannot characterize, and point toward competitive plurality rather than winner-take-all consolidation among engaged users.