ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks

TL;DR

ChronoPlay introduces a novel framework for creating dynamic, authentic game RAG benchmarks by modeling dual content and community dynamics, enabling more realistic evaluation of retrieval-augmented systems in gaming.

Contribution

It presents the first automated framework for continuous, dual-dynamic game RAG benchmarks, integrating content updates and community authenticity.

Findings

First dynamic RAG benchmark for gaming created

Framework effectively tracks content and community changes

Benchmark reveals model performance under realistic conditions

Abstract

Retrieval Augmented Generation (RAG) systems are increasingly vital in dynamic domains like online gaming, yet the lack of a dedicated benchmark has impeded standardized evaluation in this area. The core difficulty lies in Dual Dynamics: the constant interplay between game content updates and the shifting focus of the player community. Furthermore, the necessity of automating such a benchmark introduces a critical requirement for player-centric authenticity to ensure generated questions are realistic. To address this integrated challenge, we introduce ChronoPlay, a novel framework for the automated and continuous generation of game RAG benchmarks. ChronoPlay utilizes a dual-dynamic update mechanism to track both forms of change, and a dual-source synthesis engine that draws from official sources and player community to ensure both factual correctness and authentic query patterns. We instantiate our framework on three distinct games to create the first dynamic RAG benchmark for the gaming domain, offering new insights into model performance under these complex and realistic conditions. Code is avaliable at: https://github.com/hly1998/ChronoPlay.