Model-free stochastic linear quadratic design by semidefinite programming

TL;DR

This paper introduces a novel model-free semidefinite programming approach for designing stochastic linear quadratic controllers, linking dual problem optimality with Q-functions to enhance reinforcement learning methods.

Contribution

It develops a new SDP-based algorithm for model-free SLQ control, providing theoretical insights and practical tools for reinforcement learning applications.

Findings

The proposed algorithm effectively derives optimal control gains.

The approach offers a new perspective on Q-learning and RL algorithms.

Simulation results validate the method's effectiveness.

Abstract

In this article, we study a model-free design approach for stochastic linear quadratic (SLQ) controllers. Based on the convexity of the SLQ dual problem and the Karush-Kuhn-Tucker (KKT) conditions, we find the relationship between the optimal point of the dual problem and the Q-function, which can be used to develop a novel model-free semidefinite programming (SDP) algorithm for deriving optimal control gain. This study provides a new optimization perspective for understanding Q-learning algorithms and lays a theoretical foundation for effective reinforcement learning (RL) algorithms. Finally, the effectiveness of the proposed model-free SDP algorithm is demonstrated by two case simulations.