Approximate Global Convergence of Independent Learning in Multi-Agent Systems

TL;DR

This paper provides the first finite-sample analysis demonstrating approximate global convergence guarantees for independent learning algorithms in multi-agent systems, with implications for their sample complexity and practical deployment.

Contribution

It introduces a novel analytical approach for independent learning, establishing finite-sample convergence bounds and demonstrating practical relevance through experiments.

Findings

Sample complexity of (^{-2}}) up to an error term.

Constructed a separable MDP for convergence analysis.

Validated theoretical results with experiments on synthetic and real-world scenarios.

Abstract

Independent learning (IL), despite being a popular approach in practice to achieve scalability in large-scale multi-agent systems, usually lacks global convergence guarantees. In this paper, we study two representative algorithms, independent $Q$-learning and independent natural actor-critic, within value-based and policy-based frameworks, and provide the first finite-sample analysis for approximate global convergence. The results imply a sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-2})$ up to an error term that captures the dependence among agents and characterizes the fundamental limit of IL in achieving global convergence. To establish the result, we develop a novel approach for analyzing IL by constructing a separable Markov decision process (MDP) for convergence analysis and then bounding the gap due to model difference between the separable MDP and the original one. Moreover, we conduct numerical experiments using a synthetic MDP and an electric vehicle charging example to verify our theoretical findings and to demonstrate the practical applicability of IL.