Distributed Value Function Approximation for Collaborative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces novel distributed gradient-based algorithms for multi-agent reinforcement learning that enable off-policy value function approximation with limited inter-agent communication, providing convergence guarantees and variance reduction insights.

Contribution

The paper presents new distributed gradient algorithms with convergence analysis for multi-agent off-policy learning under strict communication constraints.

Findings

Algorithms converge to the solutions of associated ODEs.

Variance reduction effects are demonstrated through stochastic differential equation analysis.

Simulation results illustrate the algorithms' superior properties.

Abstract

In this paper we propose several novel distributed gradient-based temporal difference algorithms for multi-agent off-policy learning of linear approximation of the value function in Markov decision processes with strict information structure constraints, limiting inter-agent communications to small neighborhoods. The algorithms are composed of: 1) local parameter updates based on single-agent off-policy gradient temporal difference learning algorithms, including eligibility traces with state dependent parameters, and 2) linear stochastic time varying consensus schemes, represented by directed graphs. The proposed algorithms differ by their form, definition of eligibility traces, selection of time scales and the way of incorporating consensus iterations. The main contribution of the paper is a convergence analysis based on the general properties of the underlying Feller-Markov processes and the stochastic time varying consensus model. We prove, under general assumptions, that the parameter estimates generated by all the proposed algorithms weakly converge to the corresponding ordinary differential equations (ODE) with precisely defined invariant sets. It is demonstrated how the adopted methodology can be applied to temporal-difference algorithms under weaker information structure constraints. The variance reduction effect of the proposed algorithms is demonstrated by formulating and analyzing an asymptotic stochastic differential equation. Specific guidelines for communication network design are provided. The algorithms' superior properties are illustrated by characteristic simulation results.