KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

TL;DR

KORGym is a versatile, interactive evaluation platform with over fifty games designed to assess and analyze the reasoning capabilities of large language and vision models in complex, multi-turn scenarios.

Contribution

We introduce KORGym, a novel dynamic evaluation platform with diverse interactive games for comprehensive reasoning assessment of LLMs and VLMs.

Findings

Closed-source models outperform open-source ones.

Model performance varies with modality and reasoning strategies.

Reinforcement learning impacts model reasoning abilities.

Abstract

Recent advancements in large language models (LLMs) underscore the need for more comprehensive evaluation methods to accurately assess their reasoning capabilities. Existing benchmarks are often domain-specific and thus cannot fully capture an LLM's general reasoning potential. To address this limitation, we introduce the Knowledge Orthogonal Reasoning Gymnasium (KORGym), a dynamic evaluation platform inspired by KOR-Bench and Gymnasium. KORGym offers over fifty games in either textual or visual formats and supports interactive, multi-turn assessments with reinforcement learning scenarios. Using KORGym, we conduct extensive experiments on 19 LLMs and 8 VLMs, revealing consistent reasoning patterns within model families and demonstrating the superior performance of closed-source models. Further analysis examines the effects of modality, reasoning strategies, reinforcement learning techniques, and response length on model performance. We expect KORGym to become a valuable resource for advancing LLM reasoning research and developing evaluation methodologies suited to complex, interactive environments.