Distributed TD Tracking with Linear Function Approximation over Directed Communication Networks

TL;DR

This paper introduces PP-DTD, a novel distributed policy evaluation algorithm for multi-agent reinforcement learning over directed networks, achieving fast convergence and robustness.

Contribution

It presents the first distributed TD-based policy evaluation algorithm for directed graphs with proven linear convergence rates.

Findings

PP-DTD achieves linear convergence to a neighborhood of the optimum.

The algorithm demonstrates a convergence rate of O(T^{-1}) with decaying step-sizes.

Numerical experiments show robustness and effectiveness in cooperative tasks.

Abstract

We study the policy evaluation problem in multi-agent reinforcement learning (MARL) over directed communication networks, where agents cooperate with each other to explore an unknown environment and accomplish a specific task. We propose a Push-Pull-type distributed algorithm, named PP-DTD, for policy evaluation in MARL within the framework of temporal difference (TD) learning with linear function approximation. PP-DTD integrates TD learning with the Push-Pull mechanism to accommodate directed communication networks, and further utilizes variance reduction techniques to enhance both algorithmic stability and convergence rate. We show that PP-DTD achieves linear convergence to a neighborhood of the optimum under constant step-sizes and a convergence rate of $\mathcal{O}({T^{-1}})$ under decaying step-sizes when the sample is independent and identically distributed or Markovian. To the best of our knowledge, PP-DTD is the first distributed algorithm for policy evaluation in MARL over directed graphs that achieves a comparable convergence rate to single-agent TD. The numerical experiments on cooperative navigation tasks demonstrate the robustness and effectiveness of PP-DTD.