Data-Driven Near-Optimal Control of Nonlinear Systems Over Finite Horizon

TL;DR

This paper introduces a reinforcement learning approach for near-optimal control of nonlinear systems over finite horizons, decomposing the problem into infinite-horizon sub-problems to avoid complex equations.

Contribution

It presents a novel decomposition method using singular perturbation theory combined with policy iteration for finite-horizon nonlinear control.

Findings

Performance approaches model-based optimal as horizon increases

Decomposition simplifies solving time-varying HJB equations

Simulation results validate the effectiveness of the approach

Abstract

We examine the problem of two-point boundary optimal control of nonlinear systems over finite-horizon time periods with unknown model dynamics by employing reinforcement learning. We use techniques from singular perturbation theory to decompose the control problem over the finite horizon into two sub-problems, each solved over an infinite horizon. In the process, we avoid the need to solve the time-varying Hamilton-Jacobi-Bellman equation. Using a policy iteration method, which is made feasible as a result of this decomposition, it is now possible to learn the controller gains of both sub-problems. The overall control is then formed by piecing together the solutions to the two sub-problems. We show that the performance of the proposed closed-loop system approaches that of the model-based optimal performance as the time horizon gets long. Finally, we provide three simulation scenarios to support the paper's claims.