A Multi-Agent Pokemon Tournament for Evaluating Strategic Reasoning of Large Language Models

TL;DR

This paper introduces LLM Pokemon League, a tournament system where large language models act as AI trainers in Pokémon battles, serving as a new benchmark for evaluating strategic reasoning and decision-making in AI.

Contribution

The paper presents a novel competitive platform that uses LLMs as agents in a structured Pokémon battle environment to analyze their strategic reasoning and adaptability.

Findings

LLMs demonstrate varying levels of tactical depth.

The system captures detailed decision logs for analysis.

Pokémon League serves as a benchmark for AI strategic reasoning.

Abstract

This research presents LLM Pokemon League, a competitive tournament system that leverages Large Language Models (LLMs) as intelligent agents to simulate strategic decision-making in Pok\'emon battles. The platform is designed to analyze and compare the reasoning, adaptability, and tactical depth exhibited by different LLMs in a type-based, turn-based combat environment. By structuring the competition as a single-elimination tournament involving diverse AI trainers, the system captures detailed decision logs, including team-building rationale, action selection strategies, and switching decisions. The project enables rich exploration into comparative AI behavior, battle psychology, and meta-strategy development in constrained, rule-based game environments. Through this system, we investigate how modern LLMs understand, adapt, and optimize decisions under uncertainty, making Pok\'emon League a novel benchmark for AI research in strategic reasoning and competitive learning.