LAGMA: LAtent Goal-guided Multi-Agent Reinforcement Learning

TL;DR

LAGMA introduces a novel latent space approach for multi-agent reinforcement learning, enabling efficient goal-reaching trajectory generation and improved performance in complex cooperative tasks like StarCraft II and Google Research Football.

Contribution

The paper proposes a new method using a quantized latent space and trajectory generation to enhance goal-reaching in multi-agent reinforcement learning.

Findings

Achieves better performance than state-of-the-art baselines.

Effective in both dense and sparse reward settings.

Demonstrates applicability in complex environments like StarCraft II.

Abstract

In cooperative multi-agent reinforcement learning (MARL), agents collaborate to achieve common goals, such as defeating enemies and scoring a goal. However, learning goal-reaching paths toward such a semantic goal takes a considerable amount of time in complex tasks and the trained model often fails to find such paths. To address this, we present LAtent Goal-guided Multi-Agent reinforcement learning (LAGMA), which generates a goal-reaching trajectory in latent space and provides a latent goal-guided incentive to transitions toward this reference trajectory. LAGMA consists of three major components: (a) quantized latent space constructed via a modified VQ-VAE for efficient sample utilization, (b) goal-reaching trajectory generation via extended VQ codebook, and (c) latent goal-guided intrinsic reward generation to encourage transitions towards the sampled goal-reaching path. The proposed method is evaluated by StarCraft II with both dense and sparse reward settings and Google Research Football. Empirical results show further performance improvement over state-of-the-art baselines.