SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC

TL;DR

SMAC-HARD is a new benchmark for multi-agent reinforcement learning in StarCraft that introduces diverse opponent strategies and self-play to better evaluate and improve algorithm robustness against varied adversaries.

Contribution

We propose SMAC-HARD, a novel benchmark with customizable opponent strategies and self-play support, addressing overfitting and evaluation limitations in existing SMAC environments.

Findings

State-of-the-art algorithms struggle with mixed strategy opponents.

Black-box testing reveals difficulty in policy transfer to unseen adversaries.

SMAC-HARD enhances robustness and generalization in MARL algorithms.

Abstract

The availability of challenging simulation environments is pivotal for advancing the field of Multi-Agent Reinforcement Learning (MARL). In cooperative MARL settings, the StarCraft Multi-Agent Challenge (SMAC) has gained prominence as a benchmark for algorithms following centralized training with decentralized execution paradigm. However, with continual advancements in SMAC, many algorithms now exhibit near-optimal performance, complicating the evaluation of their true effectiveness. To alleviate this problem, in this work, we highlight a critical issue: the default opponent policy in these environments lacks sufficient diversity, leading MARL algorithms to overfit and exploit unintended vulnerabilities rather than learning robust strategies. To overcome these limitations, we propose SMAC-HARD, a novel benchmark designed to enhance training robustness and evaluation comprehensiveness. SMAC-HARD supports customizable opponent strategies, randomization of adversarial policies, and interfaces for MARL self-play, enabling agents to generalize to varying opponent behaviors and improve model stability. Furthermore, we introduce a black-box testing framework wherein agents are trained without exposure to the edited opponent scripts but are tested against these scripts to evaluate the policy coverage and adaptability of MARL algorithms. We conduct extensive evaluations of widely used and state-of-the-art algorithms on SMAC-HARD, revealing the substantial challenges posed by edited and mixed strategy opponents. Additionally, the black-box strategy tests illustrate the difficulty of transferring learned policies to unseen adversaries. We envision SMAC-HARD as a critical step toward benchmarking the next generation of MARL algorithms, fostering progress in self-play methods for multi-agent systems. Our code is available at https://github.com/devindeng94/smac-hard.