Reinforcement Learning Policies in Continuous-Time Linear Systems

TL;DR

This paper develops online reinforcement learning policies for continuous-time linear systems with model uncertainty, providing theoretical guarantees, stability analysis, and demonstrating effectiveness in a flight-control application.

Contribution

It introduces the first comprehensive approach to exploration-exploitation trade-offs in continuous-time systems, including novel analysis methods and performance guarantees.

Findings

Regret bound grows with square-root of time and parameters

Policies effectively learn optimal actions in flight-control task

Sharp stability results for inexact system dynamics

Abstract

Linear dynamical systems that obey stochastic differential equations are canonical models. While optimal control of known systems has a rich literature, the problem is technically hard under model uncertainty and there are hardly any results. We initiate study of this problem and aim to learn (and simultaneously deploy) optimal actions for minimizing a quadratic cost function. Indeed, this work is the first that comprehensively addresses the crucial challenge of balancing exploration versus exploitation in continuous-time systems. We present online policies that learn optimal actions fast by carefully randomizing the parameter estimates, and establish their performance guarantees: a regret bound that grows with square-root of time multiplied by the number of parameters. Implementation of the policy for a flight-control task demonstrates its efficacy. Further, we prove sharp stability results for inexact system dynamics and tightly specify the infinitesimal regret caused by sub-optimal actions. To obtain the results, we conduct a novel eigenvalue-sensitivity analysis for matrix perturbation, establish upper-bounds for comparative ratios of stochastic integrals, and introduce the new method of policy differentiation. Our analysis sheds light on fundamental challenges in continuous-time reinforcement learning and suggests a useful cornerstone for similar problems.