Model-Free Linear Quadratic Control via Reduction to Expert Prediction

TL;DR

This paper introduces a novel model-free control algorithm for linear quadratic systems that achieves sublinear regret with polynomial computational complexity by reducing control to an expert prediction problem.

Contribution

It presents the first provably sublinear regret model-free algorithm for adaptive LQ control using a reduction to expert prediction.

Findings

Achieves regret of $O(T^{\xi+2/3})$ for small 

Outperforms standard policy iteration empirically

Worse performance than model-based approaches

Abstract

Model-free approaches for reinforcement learning (RL) and continuous control find policies based only on past states and rewards, without fitting a model of the system dynamics. They are appealing as they are general purpose and easy to implement; however, they also come with fewer theoretical guarantees than model-based RL. In this work, we present a new model-free algorithm for controlling linear quadratic (LQ) systems, and show that its regret scales as $O(T^{\xi+2/3})$ for any small $\xi>0$ if time horizon satisfies $T>C^{1/\xi}$ for a constant $C$. The algorithm is based on a reduction of control of Markov decision processes to an expert prediction problem. In practice, it corresponds to a variant of policy iteration with forced exploration, where the policy in each phase is greedy with respect to the average of all previous value functions. This is the first model-free algorithm for adaptive control of LQ systems that provably achieves sublinear regret and has a polynomial computation cost. Empirically, our algorithm dramatically outperforms standard policy iteration, but performs worse than a model-based approach.