Revisiting the Gumbel-Softmax in MADDPG

TL;DR

This paper investigates alternative gradient estimators to Gumbel-Softmax in MADDPG for discrete action spaces, demonstrating that some alternatives significantly improve performance and convergence in grid-world tasks.

Contribution

It introduces and evaluates several alternative estimators to Gumbel-Softmax within MADDPG, showing improved performance in discrete multi-agent environments.

Findings

One estimator achieves up to 55% higher returns.

Faster convergence observed with certain estimators.

Alternatives outperform Gumbel-Softmax in discrete scenarios.

Abstract

MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that extends the popular single-agent method, DDPG, to multi-agent scenarios. Importantly, DDPG is an algorithm designed for continuous action spaces, where the gradient of the state-action value function exists. For this algorithm to work in discrete action spaces, discrete gradient estimation must be performed. For MADDPG, the Gumbel-Softmax (GS) estimator is used -- a reparameterisation which relaxes a discrete distribution into a similar continuous one. This method, however, is statistically biased, and a recent MARL benchmarking paper suggests that this bias makes MADDPG perform poorly in grid-world situations, where the action space is discrete. Fortunately, many alternatives to the GS exist, boasting a wide range of properties. This paper explores several of these alternatives and integrates them into MADDPG for discrete grid-world scenarios. The corresponding impact on various performance metrics is then measured and analysed. It is found that one of the proposed estimators performs significantly better than the original GS in several tasks, achieving up to 55% higher returns, along with faster convergence.