Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems

TL;DR

This paper provides a theoretical analysis of a reinforcement learning policy for continuous-time linear-quadratic systems, demonstrating fast learning and regret bounds using a randomized certainty equivalent approach.

Contribution

It introduces a novel regret analysis for the randomized certainty equivalent policy in continuous-time stochastic control, highlighting its efficiency and fundamental challenges.

Findings

Square-root of time regret bounds established

Linear scaling of regret with number of parameters shown

Policy learns optimal control quickly from a single trajectory

Abstract

This work theoretically studies a ubiquitous reinforcement learning policy for controlling the canonical model of continuous-time stochastic linear-quadratic systems. We show that randomized certainty equivalent policy addresses the exploration-exploitation dilemma in linear control systems that evolve according to unknown stochastic differential equations and their operating cost is quadratic. More precisely, we establish square-root of time regret bounds, indicating that randomized certainty equivalent policy learns optimal control actions fast from a single state trajectory. Further, linear scaling of the regret with the number of parameters is shown. The presented analysis introduces novel and useful technical approaches, and sheds light on fundamental challenges of continuous-time reinforcement learning.