Power-Constrained Policy Gradient Methods for LQR

TL;DR

This paper explores power-constrained policy gradient methods for LQR, optimizing gradient transmission over noisy channels with power limits, leading to improved convergence in reinforcement learning control tasks.

Contribution

It introduces an optimal power allocation algorithm for noisy gradient transmission and demonstrates its benefits for LQR policy gradient convergence.

Findings

Adaptive power allocation improves convergence rate.

Optimal power allocation minimizes expected optimality error.

Application to LQR enhances reinforcement learning control.

Abstract

Consider a discrete-time Linear Quadratic Regulator (LQR) problem solved using policy gradient descent when the system matrices are unknown. The gradient is transmitted across a noisy channel over a finite time horizon using analog communication by a transmitter with an average power constraint. This is a simple setup at the intersection of reinforcement learning and networked control systems. We first consider a communication-constrained optimization framework, where gradient descent is applied to optimize a non-convex function under noisy gradient transmission. We provide an optimal power allocation algorithm that minimizes an upper bound on the expected optimality error at the final iteration and show that adaptive power allocation can lead to better convergence rate as compared to standard gradient descent with uniform power distribution. We then apply our results to the LQR setting.