Data-Driven LQR using Reinforcement Learning and Quadratic Neural Networks

TL;DR

This paper presents a novel data-driven reinforcement learning method for designing linear quadratic regulators using quadratic neural networks trained via convex optimization, enabling analytical policy improvement without a system model.

Contribution

It introduces the first use of convex-optimized quadratic neural networks as Q-function approximators in RL for LQR, allowing analytical policy updates.

Findings

Converges to optimal control under controllability and stabilizing initial policy.

Effective in quadrotor control example.

First to employ convex optimization for QNN training in RL.

Abstract

This paper introduces a novel data-driven approach to design a linear quadratic regulator (LQR) using a reinforcement learning (RL) algorithm that does not require a system model. The key contribution is to perform policy iteration (PI) by designing the policy evaluator as a two-layer quadratic neural network (QNN). This network is trained through convex optimization. To the best of our knowledge, this is the first time that a QNN trained through convex optimization is employed as the Q-function approximator (QFA). The main advantage is that the QNN's input-output mapping has an analytical expression as a quadratic form, which can then be used to obtain an analytical expression for policy improvement. This is in stark contrast to the available techniques in the literature that must train a second neural network to obtain policy improvement. The article establishes the convergence of the learning algorithm to the optimal control, provided the system is controllable and one starts from a stabilitzing policy. A quadrotor example demonstrates the effectiveness of the proposed approach.