CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

TL;DR

CivBench is a comprehensive benchmark for evaluating LLM-based agents' strategic decision-making in Civilization V, focusing on turn-level predictions and multi-agent long-horizon gameplay.

Contribution

It introduces a novel evaluation framework that captures strategic capabilities through turn-level signals, addressing limitations of outcome-only assessments.

Findings

CivBench effectively estimates strategic capabilities across diverse models.

Model-specific effects of agentic setups are revealed.

Distinct strategic profiles are identified beyond win/loss outcomes.

Abstract

Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide all three, and fewer still offer evaluation signals rich enough for long-horizon, multi-agent play. We introduce CivBench, a benchmark for LLM strategists (i.e., agentic setups) in multiplayer Civilization V. Because terminal win/loss is too sparse a signal in games spanning hundreds of turns and multiple opponents, CivBench trains models on turn-level game state to estimate victory probabilities throughout play, validated through predictive, construct, and convergent validity. Across 307 games with 7 LLMs and multiple CivBench agent conditions, we demonstrate CivBench's potential to estimate strategic capabilities as an unsaturated benchmark, reveal model-specific effects of agentic setup, and outline distinct strategic profiles not visible through outcome-only evaluation.