Superlinear Optimization Algorithms

TL;DR

This paper introduces novel superlinear optimization algorithms inspired by optimal control theory, capable of handling singular Hessians and reducing computational costs while outperforming gradient descent in experiments.

Contribution

The paper presents new optimization algorithms that are superlinear convergent, can handle singular Hessians, and reduce computational complexity compared to existing methods.

Findings

Algorithms are superlinear convergent with proper parameters.

Some algorithms avoid Hessian inversion or use only diagonal elements.

Numerical experiments show they outperform gradient descent.

Abstract

This paper proposes several novel optimization algorithms for minimizing a nonlinear objective function. The algorithms are enlightened by the optimal state trajectory of an optimal control problem closely related to the minimized objective function. They are superlinear convergent when appropriate parameters are selected as required. Unlike Newton's method, all of them can be also applied in the case of a singular Hessian matrix. More importantly, by reduction, some of them avoid calculating the inverse of the Hessian matrix or an identical dimension matrix and some of them need only the diagonal elements of the Hessian matrix. In these cases, these algorithms still outperform the gradient descent method. The merits of the proposed optimization algorithm are illustrated by numerical experiments.