Approximate Midpoint Policy Iteration for Linear Quadratic Control

TL;DR

This paper introduces a midpoint policy iteration algorithm for linear quadratic control that achieves cubic convergence in model-based settings and can be approximately implemented in model-free scenarios using trajectory data, outperforming standard methods.

Contribution

The paper proposes a novel midpoint policy iteration algorithm with cubic convergence for LQ control, applicable in both model-based and model-free contexts, with theoretical and empirical validation.

Findings

Achieves cubic convergence in model-based LQ control.

Can be approximately implemented in model-free settings using trajectory data.

Numerical experiments confirm effectiveness and efficiency.

Abstract

We present a midpoint policy iteration algorithm to solve linear quadratic optimal control problems in both model-based and model-free settings. The algorithm is a variation of Newton's method, and we show that in the model-based setting it achieves cubic convergence, which is superior to standard policy iteration and policy gradient algorithms that achieve quadratic and linear convergence, respectively. We also demonstrate that the algorithm can be approximately implemented without knowledge of the dynamics model by using least-squares estimates of the state-action value function from trajectory data, from which policy improvements can be obtained. With sufficient trajectory data, the policy iterates converge cubically to approximately optimal policies, and this occurs with the same available sample budget as the approximate standard policy iteration. Numerical experiments demonstrate effectiveness of the proposed algorithms.