Leveraging Offline Data from Similar Systems for Online Linear Quadratic Control

TL;DR

This paper develops a Bayesian algorithm for online LQR control that leverages offline data from a similar system, reducing regret and improving stability despite the sim2real gap.

Contribution

It introduces a Thompson sampling-based method that incorporates offline trajectory data to enhance online control in unknown linear systems.

Findings

Achieves sublinear Bayes regret depending on system dissimilarity.

Outperforms naive strategies when offline data is similar.

Regret scales as (S,M_elta)((T/S)).

Abstract

``Sim2real gap", in which the system learned in simulations is not the exact representation of the real system, can lead to loss of stability and performance when controllers learned using data from the simulated system are used on the real system. In this work, we address this challenge in the linear quadratic regulator (LQR) setting. Specifically, we consider an LQR problem for a system with unknown system matrices. Along with the state-action pairs from the system to be controlled, a trajectory of length $S$ of state-action pairs from a different unknown system is available. Our proposed algorithm is constructed upon Thompson sampling and utilizes the mean as well as the uncertainty of the dynamics of the system from which the trajectory of length $S$ is obtained. We establish that the algorithm achieves $\tilde{\mathcal{O}}({f(S,M_{\delta})\sqrt{T/S}})$ Bayes regret after $T$ time steps, where $M_{\delta}$ characterizes the \emph{dissimilarity} between the two systems and $f(S,M_{\delta})$ is a function of $S$ and $M_{\delta}$. When $M_{\delta}$ is sufficiently small, the proposed algorithm achieves $\tilde{\mathcal{O}}({\sqrt{T/S}})$ Bayes regret and outperforms a naive strategy which does not utilize the available trajectory.