Regret Analysis of Policy Optimization over Submanifolds for Linearly Constrained Online LQG

TL;DR

This paper introduces a Riemannian-based online Newton algorithm for the online LQG problem, providing regret bounds and demonstrating effectiveness through simulations, with a focus on controllers constrained by physical conditions.

Contribution

It develops the ONM algorithm for online LQG over manifolds with constraints, offering theoretical regret analysis and empirical validation.

Findings

Regret bounds depend on the path-length of the optimal controller sequence.

The ONM algorithm effectively tracks the optimal controller in simulations.

The approach handles physical constraints like sparsity in control design.

Abstract

Recent advancement in online optimization and control has provided novel tools to study online linear quadratic regulator (LQR) problems, where cost matrices are time-varying and unknown in advance. In this work, we study the online linear quadratic Gaussian (LQG) problem over the manifold of stabilizing controllers that are linearly constrained to impose physical conditions such as sparsity. By adopting a Riemannian perspective, we propose the online Newton on manifold (ONM) algorithm, which generates an online controller on-the-fly based on the second-order information of the cost function sequence. To quantify the algorithm performance, we use the notion of regret, defined as the sub-optimality of the algorithm cumulative cost against a (locally) minimizing controller sequence. We establish a regret bound in terms of the path-length of the benchmark minimizer sequence, and we further verify the effectiveness of ONM via simulations.