NARRA-Gym for Evaluating Interactive Narrative Agents

TL;DR

NARRA-Gym provides a comprehensive, executable benchmark environment for evaluating interactive narrative agents' ability to generate coherent, personalized stories while managing long-term context and user adaptation.

Contribution

It introduces NARRA-Gym, a novel environment for evaluating LLMs in interactive storytelling with detailed trajectory logging and multi-faceted assessment.

Findings

Significant variation in model performance across different evaluation dimensions.

Models fluent in story generation often struggle with robustness and personalization.

Interactive narrative serves as a valuable benchmark for long-horizon, user-adaptive LLM evaluation.

Abstract

Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality.