Regret Analysis of Multi-task Representation Learning for Linear-Quadratic Adaptive Control

TL;DR

This paper analyzes the regret of multi-task representation learning in linear-quadratic control, showing benefits of multiple agents and sharing representations in dynamic environments with changing goals.

Contribution

It introduces a novel regret analysis for multi-task representation learning in linear-quadratic control, addressing challenges of misspecification and dynamic environments.

Findings

Regret scales as  O((\u221a{T/H})) for benign exploration.

Regret scales as  O((_u d_ heta) ( T) + T^{3/4}/H^{1/5}) in difficult exploration.

Sharing representations reduces effective task-specific parameters, improving regret bounds.

Abstract

Representation learning is a powerful tool that enables learning over large multitudes of agents or domains by enforcing that all agents operate on a shared set of learned features. However, many robotics or controls applications that would benefit from collaboration operate in settings with changing environments and goals, whereas most guarantees for representation learning are stated for static settings. Toward rigorously establishing the benefit of representation learning in dynamic settings, we analyze the regret of multi-task representation learning for linear-quadratic control. This setting introduces unique challenges. Firstly, we must account for and balance the $\textit{misspecification}$ introduced by an approximate representation. Secondly, we cannot rely on the parameter update schemes of single-task online LQR, for which least-squares often suffices, and must devise a novel scheme to ensure sufficient improvement. We demonstrate that for settings where exploration is "benign", the regret of any agent after $T$ timesteps scales as $\tilde O(\sqrt{T/H})$, where $H$ is the number of agents. In settings with "difficult" exploration, the regret scales as $\tilde O(\sqrt{d_u d_\theta} \sqrt{T} + T^{3/4}/H^{1/5})$, where $d_x$ is the state-space dimension, $d_u$ is the input dimension, and $d_\theta$ is the task-specific parameter count. In both cases, by comparing to the minimax single-task regret $O(\sqrt{d_x d_u^2}\sqrt{T})$, we see a benefit of a large number of agents. Notably, in the difficult exploration case, by sharing a representation across tasks, the effective task-specific parameter count can often be small $d_\theta < d_x d_u$. Lastly, we provide numerical validation of the trends we predict.