CCL: Collaborative Curriculum Learning for Sparse-Reward Multi-Agent Reinforcement Learning via Co-evolutionary Task Evolution

TL;DR

This paper introduces CCL, a curriculum learning framework that improves multi-agent reinforcement learning in sparse reward environments by co-evolving agents and tasks, leading to better training stability and performance.

Contribution

The paper presents a novel co-evolutionary curriculum learning approach that refines tasks and co-evolves agents and environment to address sparse rewards in multi-agent RL.

Findings

CCL outperforms existing methods in five cooperative tasks.

Co-evolution of agents and tasks enhances training stability.

The framework effectively handles sparse reward challenges.

Abstract

Sparse reward environments pose significant challenges in reinforcement learning, especially within multi-agent systems (MAS) where feedback is delayed and shared across agents, leading to suboptimal learning. We propose Collaborative Multi-dimensional Course Learning (CCL), a novel curriculum learning framework that addresses this by (1) refining intermediate tasks for individual agents, (2) using a variational evolutionary algorithm to generate informative subtasks, and (3) co-evolving agents with their environment to enhance training stability. Experiments on five cooperative tasks in the MPE and Hide-and-Seek environments show that CCL outperforms existing methods in sparse reward settings.