LLM-PySC2: Starcraft II learning environment for Large Language Models

TL;DR

This paper introduces LLM-PySC2, a comprehensive StarCraft II environment designed for evaluating large language models' decision-making abilities, addressing previous platform limitations and supporting multi-agent collaboration with efficient interaction architecture.

Contribution

The paper presents the first complete pysc2 environment tailored for LLMs, enabling complex decision-making evaluation with multi-modal info and scalable interaction architecture.

Findings

LLMs can achieve victories in complex StarCraft II scenarios.

Pre-trained LLMs struggle with correct decisions without task instructions.

StarCraft II remains a challenging domain for large models.

Abstract

The tremendous potential has been demonstrated by large language models (LLMs) in intelligent decision-making problems, with unprecedented capabilities shown across diverse applications ranging from gaming AI systems to complex strategic planning frameworks. However, the StarCraft II platform, which has been widely adopted for validating decision-making algorithms in the past decade, has not yet provided substantial support for this emerging domain. To address issues that LLMs cannot interface with the hundreds of actions of the pysc2 backend and the lack of native support for multi-agent (MA) collaboration, we propose the LLM-PySC2 environment. This is the first environment that offers LLMs the complete pysc2 action space with sufficient multi-modal information and game Wiki knowledge. With an asynchronous query architecture, the environment efficiently interacts with LLMs that maintain a constant latency regardless of the scale of the agents' population. In the experiments, we evaluated LLMs' decision-making performance in both the macro-decision and micro-operation scenarios, with traditional StarCraft II Multi-Agent Challenge (SMAC) tasks and a series of new proposed. Results indicate that LLMs possess the potential to achieve victories in complex scenarios but cannot constantly generate correct decisions, especially in the recovered pysc2 action space and MA settings. Without task-relevant instructions, the pre-trained models suffer from issues such as hallucinations and inefficient collaboration. Our findings suggest that StarCraft II still challenges in the era of large models, revealing that there is a lot to do to develop an advanced LLM decision-making system, and the proposed LLM-PySC2 environment will support future development of LLM-based decision-making solutions.