Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning

TL;DR

This paper introduces ATD($5$), an adaptive TD($5$) method for multi-agent reinforcement learning that dynamically adjusts the bias-variance trade-off using a likelihood-free density ratio estimator, improving performance.

Contribution

It proposes a novel likelihood-free density ratio estimator with dual replay buffers to adaptively set TD($5$) parameters in multi-agent RL, addressing policy distribution calculation challenges.

Findings

ATD($5$) outperforms static 5$ approaches on SMAC and Gfootball benchmarks.

The method demonstrates consistent or superior performance compared to baseline algorithms.

Adaptive 5$ improves value estimation accuracy in multi-agent settings.

Abstract

TD($\lambda$) in value-based MARL algorithms or the Temporal Difference critic learning in Actor-Critic-based (AC-based) algorithms synergistically integrate elements from Monte-Carlo simulation and Q function bootstrapping via dynamic programming, which effectively addresses the inherent bias-variance trade-off in value estimation. Based on that, some recent works link the adaptive $\lambda$ value to the policy distribution in the single-agent reinforcement learning area. However, because of the large joint action space from multiple number of agents, and the limited transition data in Multi-agent Reinforcement Learning, the policy distribution is infeasible to be calculated statistically. To solve the policy distribution calculation problem in MARL settings, we employ a parametric likelihood-free density ratio estimator with two replay buffers instead of calculating statistically. The two replay buffers of different sizes store the historical trajectories that represent the data distribution of the past and current policies correspondingly. Based on the estimator, we assign Adaptive TD($\lambda$), \textbf{ATD($\lambda$)}, values to state-action pairs based on their likelihood under the stationary distribution of the current policy. We apply the proposed method on two competitive baseline methods, QMIX for value-based algorithms, and MAPPO for AC-based algorithms, over SMAC benchmarks and Gfootball academy scenarios, and demonstrate consistently competitive or superior performance compared to other baseline approaches with static $\lambda$ values.