Convexity in Optimal Control Problems

TL;DR

This paper highlights the importance of Hamiltonian convexity in continuous-time nonlinear optimal control, showing it guarantees unique solutions and analyzing how discretization affects solution quality, validated through simulation comparisons.

Contribution

It demonstrates that Hamiltonian convexity ensures unique optimal trajectories and analyzes how discretization impacts solution uniqueness and convergence in optimal control problems.

Findings

Convex Hamiltonian guarantees unique optimal solutions.

Discretization can introduce multiple spurious optima.

ILQR converges to the global optimum, SQP may get stuck in local minima.

Abstract

This paper investigates the central role played by the Hamiltonian in continuous-time nonlinear optimal control problems. We show that the strict convexity of the Hamiltonian in the control variable is a sufficient condition for the existence of a unique optimal trajectory, and the nonlinearity/non-convexity of the dynamics and the cost are immaterial. The analysis is extended to discrete-time problems, revealing that discretization destroys the convex Hamiltonian structure, leading to multiple spurious optima, unless the time discretization is sufficiently small. We present simulated results comparing the "indirect" Iterative Linear Quadratic Regulator (iLQR) and the "direct" Sequential Quadratic Programming (SQP) approach for solving the optimal control problem for the cartpole and pendulum models to validate the theoretical analysis. Results show that the ILQR always converges to the "globally" optimum solution while the SQP approach gets stuck in spurious minima given multiple random initial guesses for a time discretization that is insufficiently small, while both converge to the same unique solution if the discretization is sufficiently small.