RPGBENCH: Evaluating Large Language Models as Role-Playing Game Engines

TL;DR

RPGBench introduces a comprehensive benchmark to evaluate large language models' ability to create and simulate text-based role-playing games, assessing creativity, coherence, and rule adherence through both objective and subjective measures.

Contribution

This work is the first to systematically evaluate LLMs as RPG engines using structured tasks and combined evaluation methods, setting a new standard for interactive storytelling assessment.

Findings

State-of-the-art LLMs generate engaging stories but struggle with consistent game mechanics.

Objective assessments verify rule adherence and state updates.

LLM-based judges evaluate content quality and role-playing depth.

Abstract

We present RPGBench, the first benchmark designed to evaluate large language models (LLMs) as text-based role-playing game (RPG) engines. RPGBench comprises two core tasks: Game Creation (GC) and Game Simulation (GS). In GC, an LLM must craft a valid and playable RPG world using a structured event-state representation, ensuring logical coherence and proper termination conditions. In GS, the LLM simulates interactive gameplay across multiple rounds while consistently updating states and enforcing game rules. To comprehensively assess performance, RPGBench integrates objective and subjective evaluation methodologies. Objective measures verify adherence to event mechanics and check variable updates without requiring human intervention. Subjective measures, such as content interestingness, action quality, and role-playing capability, are evaluated via an LLM-as-a-judge framework, where a strong LLM grades each candidate's outputs. Empirical results demonstrate that state-of-the-art LLMs can produce engaging stories but often struggle to implement consistent, verifiable game mechanics, particularly in long or complex scenarios. By combining structured, rule-based assessments with LLM-based judgments, RPGBench provides a new standard for evaluating how well LLMs can balance creativity, coherence, and complexity in text-based RPGs, opening avenues for more immersive and controllable interactive storytelling.