Adapt and Stabilize, Then Learn and Optimize: A New Approach to Adaptive LQR

TL;DR

This paper introduces a new adaptive LQR algorithm that overcomes practical implementation barriers by combining direct MRAC with an epoch-based approach, achieving low regret without requiring initial stabilization or exploration.

Contribution

It proposes a novel adaptive control algorithm that addresses key practical drawbacks of existing methods, with theoretical guarantees and improved regret performance.

Findings

Simulations show comparable regret to existing methods under certain conditions.

Regret is significantly smaller when initial stabilization or exploration conditions are not met.

The approach achieves high-probability regret bounds similar to prior work.

Abstract

This paper focuses on adaptive control of the discrete-time linear quadratic regulator (adaptive LQR). Recent literature has made significant contributions in proving non-asymptotic convergence rates, but existing approaches have a few drawbacks that pose barriers for practical implementation. These drawbacks include (i) a requirement of an initial stabilizing controller, (ii) a reliance on exploration for closed-loop stability, and/or (iii) computationally intensive algorithms. This paper proposes a new algorithm that overcomes these drawbacks for a particular class of discrete-time systems. This algorithm leverages direct model-reference adaptive control (direct MRAC) and combines it with an epoch-based approach in order to address the drawbacks (i)-(iii) with a provable high-probability regret bound comparable to existing literature. Simulations demonstrate that the proposed approach yields regrets that are comparable to those from existing methods when the conditions (i) and (ii) are met, and yields regrets that are significantly smaller when either of these two conditions is not met.