A Reduced Order Iterative Linear Quadratic Regulator (ILQR) Technique for the Optimal Control of Nonlinear Partial Differential Equations

TL;DR

This paper presents a reduced order ILQR-based reinforcement learning method for efficiently controlling nonlinear PDEs, demonstrating significant computational savings while maintaining performance.

Contribution

It introduces a novel reduced order ILQR technique using the Method of Snapshots for nonlinear PDE control, with convergence analysis and practical validation.

Findings

Significant reduction in computational burden compared to standard ILQR.

Effective control of nonlinear PDEs like viscous Burger's equation and phase-field models.

Convergence to a limit set influenced by truncation error.

Abstract

In this paper, we introduce a reduced order model-based reinforcement learning (MBRL) approach, utilizing the Iterative Linear Quadratic Regulator (ILQR) algorithm for the optimal control of nonlinear partial differential equations (PDEs). The approach proposes a novel modification of the ILQR technique: it uses the Method of Snapshots to identify a reduced order Linear Time Varying (LTV) approximation of the nonlinear PDE dynamics around a current estimate of the optimal trajectory, utilizes the identified LTV model to solve a time-varying reduced order LQR problem to obtain an improved estimate of the optimal trajectory along with a new reduced basis, and iterates till convergence. The convergence behavior of the reduced order approach is analyzed and the algorithm is shown to converge to a limit set that is dependent on the truncation error in the reduction. The proposed approach is tested on the viscous Burger's equation and two phase-field models for microstructure evolution in materials, and the results show that there is a significant reduction in the computational burden over the standard ILQR approach, without significantly sacrificing performance.