Personalized Multi-Agent Average Reward TD-Learning via Joint Linear Approximation

TL;DR

This paper introduces a novel approach for personalized multi-agent average reward TD learning that leverages shared linear structures to improve convergence and mitigate conflicting signals, inspired by federated learning techniques.

Contribution

It proposes a cooperative single-timescale TD learning method that estimates a common subspace among agents, addressing heterogeneity and Markovian sampling challenges.

Findings

Achieves linear speedup in convergence.

Effectively filters out conflicting signals.

Demonstrates benefits through experiments.

Abstract

We study personalized multi-agent average reward TD learning, in which a collection of agents interacts with different environments and jointly learns their respective value functions. We focus on the setting where there exists a shared linear representation, and the agents' optimal weights collectively lie in an unknown linear subspace. Inspired by the recent success of personalized federated learning (PFL), we study the convergence of cooperative single-timescale TD learning in which agents iteratively estimate the common subspace and local heads. We showed that this decomposition can filter out conflicting signals, effectively mitigating the negative impacts of ``misaligned'' signals, and achieving linear speedup. The main technical challenges lie in the heterogeneity, the Markovian sampling, and their intricate interplay in shaping error evolutions. Specifically, not only are the error dynamics of multiple variables closely interconnected, but there is also no direct contraction for the principal angle distance between the optimal subspace and the estimated subspace. We hope our analytical techniques can be useful to inspire research on deeper exploration into leveraging common structures. Experiments are provided to show the benefits of learning via a shared structure to the more general control problem.