Model-free Learning for Risk-constrained Linear Quadratic Regulator with Structured Feedback in Networked Systems

TL;DR

This paper introduces a model-free learning algorithm for risk-constrained LQR problems with structured feedback, suitable for networked systems, and demonstrates its convergence and effectiveness through numerical tests.

Contribution

It develops a stochastic gradient-based algorithm for risk-constrained LQR with structured feedback, addressing model-free settings and convergence guarantees.

Findings

Converges to a stationary point with high probability.

Effectively incorporates risk constraints in networked control.

Achieves near-optimal control performance compared to classical LQR.

Abstract

We develop a model-free learning algorithm for the infinite-horizon linear quadratic regulator (LQR) problem. Specifically, (risk) constraints and structured feedback are considered, in order to reduce the state deviation while allowing for a sparse communication graph in practice. By reformulating the dual problem as a nonconvex-concave minimax problem, we adopt the gradient descent max-oracle (GDmax), and for modelfree setting, the stochastic (S)GDmax using zero-order policy gradient. By bounding the Lipschitz and smoothness constants of the LQR cost using specifically defined sublevel sets, we can design the stepsize and related parameters to establish convergence to a stationary point (at a high probability). Numerical tests in a networked microgrid control problem have validated the convergence of our proposed SGDmax algorithm while demonstrating the effectiveness of risk constraints. The SGDmax algorithm has attained a satisfactory optimality gap compared to the classical LQR control, especially for the full feedback case.