Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces LICA, a multi-agent reinforcement learning method that implicitly solves the credit assignment problem by combining a hypernetwork-based critic with adaptive entropy regularization, leading to superior performance in cooperative tasks.

Contribution

The paper proposes a novel implicit credit assignment approach using a hypernetwork critic and adaptive entropy regularization within the CTDE framework, enhancing cooperative multi-agent learning.

Findings

LICA outperforms previous methods on benchmarks

The hypernetwork critic effectively captures joint action value

Adaptive entropy regularization maintains exploration levels

Abstract

We present a multi-agent actor-critic method that aims to implicitly address the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment among agents may not require an explicit formulation as long as (1) the policy gradients derived from a centralized critic carry sufficient information for the decentralized agents to maximize their joint action value through optimal cooperation and (2) a sustained level of exploration is enforced throughout training. Under the centralized training with decentralized execution (CTDE) paradigm, we achieve the former by formulating the centralized critic as a hypernetwork such that a latent state representation is integrated into the policy gradients through its multiplicative association with the stochastic policies; to achieve the latter, we derive a simple technique called adaptive entropy regularization where magnitudes of the entropy gradients are dynamically rescaled based on the current policy stochasticity to encourage consistent levels of exploration. Our algorithm, referred to as LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.