Representation Learning For Efficient Deep Multi-Agent Reinforcement Learning

TL;DR

This paper introduces MAPO-LSO, a novel representation learning method for multi-agent reinforcement learning that significantly improves sample efficiency and learning performance by leveraging latent space optimization techniques.

Contribution

It proposes a multi-agent extension of transition dynamics reconstruction and self-predictive learning to enhance MARL training with auxiliary representation learning.

Findings

Notable improvements in sample efficiency over vanilla MARL

Enhanced learning performance across diverse MARL tasks

Easy integration with existing MARL algorithms without extra hyperparameter tuning

Abstract

Sample efficiency remains a key challenge in multi-agent reinforcement learning (MARL). A promising approach is to learn a meaningful latent representation space through auxiliary learning objectives alongside the MARL objective to aid in learning a successful control policy. In our work, we present MAPO-LSO (Multi-Agent Policy Optimization with Latent Space Optimization) which applies a form of comprehensive representation learning devised to supplement MARL training. Specifically, MAPO-LSO proposes a multi-agent extension of transition dynamics reconstruction and self-predictive learning that constructs a latent state optimization scheme that can be trivially extended to current state-of-the-art MARL algorithms. Empirical results demonstrate MAPO-LSO to show notable improvements in sample efficiency and learning performance compared to its vanilla MARL counterpart without any additional MARL hyperparameter tuning on a diverse suite of MARL tasks.