Learning Heterogeneous Agent Cooperation via Multiagent League Training

TL;DR

This paper introduces Heterogeneous League Training (HLT), a reinforcement learning algorithm designed to improve cooperation among diverse agents in multiagent systems by maintaining a policy pool and using a hyper-network to enhance behavioral diversity.

Contribution

The paper presents a novel RL algorithm, HLT, that effectively addresses challenges in heterogeneous multiagent systems, including non-stationarity and policy iteration issues, with a new policy league and diversity mechanism.

Findings

HLT increases success rates in cooperative heterogeneous tasks.

HLT effectively solves the policy version iteration problem.

HLT offers a practical method to evaluate learning difficulty of roles.

Abstract

Many multiagent systems in the real world include multiple types of agents with different abilities and functionality. Such heterogeneous multiagent systems have significant practical advantages. However, they also come with challenges compared with homogeneous systems for multiagent reinforcement learning, such as the non-stationary problem and the policy version iteration issue. This work proposes a general-purpose reinforcement learning algorithm named Heterogeneous League Training (HLT) to address heterogeneous multiagent problems. HLT keeps track of a pool of policies that agents have explored during training, gathering a league of heterogeneous policies to facilitate future policy optimization. Moreover, a hyper-network is introduced to increase the diversity of agent behaviors when collaborating with teammates having different levels of cooperation skills. We use heterogeneous benchmark tasks to demonstrate that (1) HLT promotes the success rate in cooperative heterogeneous tasks; (2) HLT is an effective approach to solving the policy version iteration problem; (3) HLT provides a practical way to assess the difficulty of learning each role in a heterogeneous team.