Concurrent Learning with Aggregated States via Randomized Least Squares Value Iteration

TL;DR

This paper proves that randomized least squares value iteration with aggregated states enables multiple agents to explore efficiently in reinforcement learning, achieving optimal regret rates and lower space complexity.

Contribution

It introduces a novel concurrent learning framework with aggregated states, providing theoretical regret bounds and improved space efficiency over prior methods.

Findings

Per-agent regret decreases at the optimal rate of 1/√N.

Algorithm achieves polynomial worst-case regret bounds.

Significantly reduces space complexity compared to prior work.

Abstract

Designing learning agents that explore efficiently in a complex environment has been widely recognized as a fundamental challenge in reinforcement learning. While a number of works have demonstrated the effectiveness of techniques based on randomized value functions on a single agent, it remains unclear, from a theoretical point of view, whether injecting randomization can help a society of agents {\it concurently} explore an environment. The theoretical results %that we established in this work tender an affirmative answer to this question. We adapt the concurrent learning framework to \textit{randomized least-squares value iteration} (RLSVI) with \textit{aggregated state representation}. We demonstrate polynomial worst-case regret bounds in both finite- and infinite-horizon environments. In both setups the per-agent regret decreases at an optimal rate of $\Theta\left(\frac{1}{\sqrt{N}}\right)$, highlighting the advantage of concurent learning. Our algorithm exhibits significantly lower space complexity compared to \cite{russo2019worst} and \cite{agrawal2021improved}. We reduce the space complexity by a factor of $K$ while incurring only a $\sqrt{K}$ increase in the worst-case regret bound, compared to \citep{agrawal2021improved,russo2019worst}. Additionally, we conduct numerical experiments to demonstrate our theoretical findings.