Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games

TL;DR

Orak is a comprehensive benchmark and dataset suite designed to evaluate and fine-tune Large Language Model agents across diverse video game genres, supporting reproducible research and development of versatile gaming AI.

Contribution

It introduces a unified evaluation framework, a plug-and-play interface, and a fine-tuning dataset to advance LLM-based game agents across multiple genres.

Findings

Established game leaderboards and battle arenas for LLM agents.

Conducted ablation studies on input modalities and agentic strategies.

Provided a fine-tuning dataset of expert gameplay trajectories.

Abstract

Large Language Model (LLM) agents are reshaping the game industry, by enabling more intelligent and human-preferable characters. Yet, current game benchmarks fall short of practical needs: they lack evaluations of diverse LLM capabilities across various game genres, studies of agentic modules crucial for complex gameplay, and fine-tuning datasets to adapt pre-trained LLMs into gaming agents. To fill these gaps, we present Orak, a benchmark for training and evaluating LLM agents across 12 popular video games spanning all major genres. Using a plug-and-play interface built on Model Context Protocol (MCP), Orak supports systematic and reproducible studies of agentic modules in varied game scenarios. We further release a fine-tuning dataset of expert LLM gameplay trajectories covering multiple genres, turning general LLMs into effective game agents. Orak offers a united evaluation framework, including game leaderboards, LLM battle arenas, and \fix{ablation studies} of input modality, agentic strategies, and fine-tuning effects, establishing a foundation towards versatile gaming agents. Code and datasets are available at https://github.com/krafton-ai/Orak and https://huggingface.co/datasets/KRAFTON/Orak.