Learning Nearly Decomposable Value Functions Via Communication Minimization

TL;DR

This paper introduces a novel framework for multi-agent reinforcement learning that learns nearly decomposable value functions with minimal communication, improving scalability and efficiency in collaborative tasks.

Contribution

It proposes a new approach combining value function factorization with communication minimization using information-theoretic regularizers, compatible with existing methods like QMIX.

Findings

Significantly outperforms baseline methods on StarCraft benchmarks.

Reduces communication by over 80% without performance loss.

Demonstrates effective coordination with minimal messaging.

Abstract

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.