OpenGuanDan: A Large-Scale Imperfect Information Game Benchmark

TL;DR

OpenGuanDan introduces a large-scale, challenging Chinese card game benchmark designed to evaluate AI agents in imperfect information, multi-agent, and long-horizon decision-making scenarios, supporting both AI and human interactions.

Contribution

This work presents OpenGuanDan, a novel benchmark for GuanDan that enables comprehensive evaluation of AI agents in complex, imperfect information multi-player settings with an API for human-AI interaction.

Findings

Learning-based agents outperform rule-based ones.

Current AI agents are not yet superhuman.

OpenGuanDan is publicly available for research use.

Abstract

The advancement of data-driven artificial intelligence (AI), particularly machine learning, heavily depends on large-scale benchmarks. Despite remarkable progress across domains ranging from pattern recognition to intelligent decision-making in recent decades, exemplified by breakthroughs in board games, card games, and electronic sports games, there remains a pressing need for more challenging benchmarks to drive further research. To this end, this paper proposes OpenGuanDan, a novel benchmark that enables both efficient simulation of GuanDan (a popular four-player, multi-round Chinese card game) and comprehensive evaluation of both learning-based and rule-based GuanDan AI agents. OpenGuanDan poses a suite of nontrivial challenges, including imperfect information, large-scale information set and action spaces, a mixed learning objective involving cooperation and competition, long-horizon decision-making, variable action spaces, and dynamic team composition. These characteristics make it a demanding testbed for existing intelligent decision-making methods. Moreover, the independent API for each player allows human-AI interactions and supports integration with large language models. Empirically, we conduct two types of evaluations: (1) pairwise competitions among all GuanDan AI agents, and (2) human-AI matchups. Experimental results demonstrate that while current learning-based agents substantially outperform rule-based counterparts, they still fall short of achieving superhuman performance, underscoring the need for continued research in multi-agent intelligent decision-making domain. The project is publicly available at https://github.com/GameAI-NJUPT/OpenGuanDan.