Diversity is Strength: Mastering Football Full Game with Interactive Reinforcement Learning of Multiple AIs

TL;DR

This paper introduces a novel reinforcement learning framework called Diversity is Strength (DIS) that trains multiple AI agents with diverse strategies in complex multi-agent environments, leading to high-performance football AIs without human data.

Contribution

The paper proposes the DIS framework, which enhances AI diversity and strength through interconnected models and evaluation schemes, achieving state-of-the-art results in Google Research Football competitions.

Findings

Won 5v5 and 11v11 tracks in AI football competitions

AI exhibits rich, diverse strategies in complex multi-agent settings

Ablation studies confirm effectiveness of proposed modules

Abstract

Training AI with strong and rich strategies in multi-agent environments remains an important research topic in Deep Reinforcement Learning (DRL). The AI's strength is closely related to its diversity of strategies, and this relationship can guide us to train AI with both strong and rich strategies. To prove this point, we propose Diversity is Strength (DIS), a novel DRL training framework that can simultaneously train multiple kinds of AIs. These AIs are linked through an interconnected history model pool structure, which enhances their capabilities and strategy diversities. We also design a model evaluation and screening scheme to select the best models to enrich the model pool and obtain the final AI. The proposed training method provides diverse, generalizable, and strong AI strategies without using human data. We tested our method in an AI competition based on Google Research Football (GRF) and won the 5v5 and 11v11 tracks. The method enables a GRF AI to have a high level on both 5v5 and 11v11 tracks for the first time, which are under complex multi-agent environments. The behavior analysis shows that the trained AI has rich strategies, and the ablation experiments proved that the designed modules benefit the training process.