Approximate Thompson Sampling for Learning Linear Quadratic Regulators with $O(\sqrt{T})$ Regret

TL;DR

This paper introduces a new Thompson sampling algorithm for learning linear quadratic regulators that achieves an optimal $O(\sqrt{T})$ regret bound by using Langevin dynamics and an excitation mechanism to improve posterior sampling.

Contribution

The paper presents a novel Thompson sampling method for LQR with a carefully designed excitation and Langevin dynamics, achieving regret bounds without restrictive assumptions.

Findings

Achieves $O(\sqrt{T})$ regret bound for LQR learning.

Develops a Langevin dynamics-based posterior sampling approach.

Provides concentration properties of approximate posteriors.

Abstract

We propose a novel Thompson sampling algorithm that learns linear quadratic regulators (LQR) with a Bayesian regret bound of $O(\sqrt{T})$. Our method leverages Langevin dynamics with a carefully designed preconditioner and incorporates a simple excitation mechanism. We show that the excitation signal drives the minimum eigenvalue of the preconditioner to grow over time, thereby accelerating the approximate posterior sampling process. Furthermore, we establish nontrivial concentration properties of the approximate posteriors generated by our algorithm. These properties enable us to bound the moments of the system state and attain an $O(\sqrt{T})$ regret bound without relying on the restrictive assumptions that are often used in the literature.