Evaluating LLM Metrics Through Real-World Capabilities

TL;DR

This paper evaluates large language models based on real-world user tasks, identifying gaps in existing benchmarks and highlighting Google Gemini's superior performance in practical utility metrics.

Contribution

It introduces a human-centered evaluation framework for LLMs focusing on real-world capabilities and assesses how current benchmarks align with actual user needs.

Findings

Existing benchmarks have significant gaps in coverage and interpretability.

Google Gemini outperforms other models on real-world utility metrics.

Most benchmarks do not adequately measure practical usefulness.

Abstract

As generative AI becomes increasingly embedded in everyday workflows, it is important to evaluate its performance in ways that reflect real-world usage rather than abstract notions of intelligence. Unlike many existing benchmarks that assess general intelligence, our approach focuses on real-world utility, evaluating how well models support users in everyday tasks. While current benchmarks emphasize code generation or factual recall, users rely on AI for a much broader range of activities-from writing assistance and summarization to citation formatting and stylistic feedback. In this paper, we analyze large-scale survey data and usage logs to identify six core capabilities that represent how people commonly use Large Language Models (LLMs): Summarization, Technical Assistance, Reviewing Work, Data Structuring, Generation, and Information Retrieval. We then assess the extent to which existing benchmarks cover these capabilities, revealing significant gaps in coverage, efficiency measurement, and interpretability. Drawing on this analysis, we use human-centered criteria to identify gaps in how well current benchmarks reflect common usage that is grounded in five practical criteria: coherence, accuracy, clarity, relevance, and efficiency. For four of the six capabilities, we identify the benchmarks that best align with real-world tasks and use them to compare leading models. We find that Google Gemini outperforms other models-including OpenAI's GPT, xAI's Grok, Meta's LLaMA, Anthropic's Claude, DeepSeek, and Qwen from Alibaba-on these utility-focused metrics.