Asynchronous stochastic approximations with asymptotically biased errors and deep multi-agent learning

TL;DR

This paper establishes conditions for the stability and convergence of asynchronous stochastic approximations with biased errors, with applications to multi-agent learning algorithms like policy gradient and value iteration.

Contribution

It provides verifiable conditions ensuring stability and convergence of asynchronous SAs with biased approximation errors, extending analysis to multi-agent learning schemes.

Findings

Stability is unaffected by asymptotically bounded biased errors.

Convergence relates to the limiting set and approximation errors.

Experimental results support the theoretical analysis.

Abstract

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control problem becomes more tractable, function approximations affect stability and convergence. In this paper, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze Policy Gradient methods and noisy Value Iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between the limiting set and the approximation errors is established. Finally, experimental results are presented that support the theory.