Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning

TL;DR

This paper introduces IBORM, a method that uses the information bottleneck principle to learn compact, informative behavior representations in multi-agent reinforcement learning, leading to faster convergence and improved performance.

Contribution

It proposes a novel explicit behavior representation learning approach based on the information bottleneck, enhancing interpretability and efficiency in multi-agent RL.

Findings

IBORM achieves the fastest convergence among compared methods.

It results in the best policy performance.

The approach effectively balances information compression and utility.

Abstract

In multi-agent deep reinforcement learning, extracting sufficient and compact information of other agents is critical to attain efficient convergence and scalability of an algorithm. In canonical frameworks, distilling of such information is often done in an implicit and uninterpretable manner, or explicitly with cost functions not able to reflect the relationship between information compression and utility in representation. In this paper, we present Information-Bottleneck-based Other agents' behavior Representation learning for Multi-agent reinforcement learning (IBORM) to explicitly seek low-dimensional mapping encoder through which a compact and informative representation relevant to other agents' behaviors is established. IBORM leverages the information bottleneck principle to compress observation information, while retaining sufficient information relevant to other agents' behaviors used for cooperation decision. Empirical results have demonstrated that IBORM delivers the fastest convergence rate and the best performance of the learned policies, as compared with implicit behavior representation learning and explicit behavior representation learning without explicitly considering information compression and utility.