GameArena: Evaluating LLM Reasoning through Live Computer Games

TL;DR

GameArena is a novel interactive benchmark that evaluates large language models' reasoning skills through engaging, real-time gameplay, providing detailed insights into their reasoning processes in dynamic, human-in-the-loop settings.

Contribution

It introduces a new dynamic, interactive benchmark with specific reasoning tasks, enabling detailed analysis of LLM reasoning in real-world gameplay scenarios.

Findings

Collected over 2000 game sessions for analysis.

Assessed reasoning capabilities of five state-of-the-art LLMs.

Improved user engagement compared to existing benchmarks.

Abstract

Evaluating the reasoning abilities of large language models (LLMs) is challenging. Existing benchmarks often depend on static datasets, which are vulnerable to data contamination and may get saturated over time, or on binary live human feedback that conflates reasoning with other abilities. As the most prominent dynamic benchmark, Chatbot Arena evaluates open-ended questions in real-world settings, but lacks the granularity in assessing specific reasoning capabilities. We introduce GameArena, a dynamic benchmark designed to evaluate LLM reasoning capabilities through interactive gameplay with humans. GameArena consists of three games designed to test specific reasoning capabilities (e.g., deductive and inductive reasoning), while keeping participants entertained and engaged. We analyze the gaming data retrospectively to uncover the underlying reasoning processes of LLMs and measure their fine-grained reasoning capabilities. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of-the-art LLMs. Our user study with 100 participants suggests that GameArena improves user engagement compared to Chatbot Arena. For the first time, GameArena enables the collection of step-by-step LLM reasoning data in the wild.