A primal-dual perspective for distributed TD-learning

TL;DR

This paper introduces a novel distributed TD-learning algorithm based on primal-dual optimization that converges efficiently without requiring doubly stochastic communication matrices, applicable to various network scenarios.

Contribution

It develops a primal-dual ODE-based approach for distributed TD-learning that relaxes common network assumptions and analyzes convergence under different step-size and observation models.

Findings

Algorithm converges exponentially under various conditions.

Does not require doubly stochastic communication matrices.

Applicable to both i.i.d. and Markovian data models.

Abstract

The goal of this paper is to investigate distributed temporal difference (TD) learning for a networked multi-agent Markov decision process. The proposed approach is based on distributed optimization algorithms, which can be interpreted as primal-dual Ordinary differential equation (ODE) dynamics subject to null-space constraints. Based on the exponential convergence behavior of the primal-dual ODE dynamics subject to null-space constraints, we examine the behavior of the final iterate in various distributed TD-learning scenarios, considering both constant and diminishing step-sizes and incorporating both i.i.d. and Markovian observation models. Unlike existing methods, the proposed algorithm does not require the assumption that the underlying communication network structure is characterized by a doubly stochastic matrix.