Higher-Order Corrections to Optimisers based on Newton's Method

TL;DR

This paper introduces higher-order correction techniques based on geodesic acceleration to improve Newton-like optimization methods, significantly reducing steps and computation time in nonlinear least squares problems.

Contribution

It develops a differential equation framework for higher-order corrections, enabling 2nd to 4th order improvements to Levenberg--Marquardt and similar algorithms.

Findings

Higher-order methods reduce optimization steps.

Significant decrease in computation time.

Effective for ill-conditioned Jacobian problems.

Abstract

The Newton, Gauss--Newton and Levenberg--Marquardt methods all use the first derivative of a vector function (the Jacobian) to minimise its sum of squares. When the Jacobian matrix is ill-conditioned, the function varies much faster in some directions than others and the space of possible improvement in sum of squares becomes a long narrow ellipsoid in the linear model. This means that even a small amount of nonlinearity in the problem parameters can cause a proposed point far down the long axis of the ellipsoid to fall outside of the actual curved valley of improved values, even though it is quite nearby. This paper presents a differential equation that `follows' these valleys, based on the technique of geodesic acceleration, which itself provides a 2$^\mathrm{nd}$ order improvement to the Levenberg--Marquardt iteration step. Higher derivatives of this equation are computed that allow $n^\mathrm{th}$ order improvements to the optimisation methods to be derived. These higher-order accelerated methods up to 4$^\mathrm{th}$ order are tested numerically and shown to provide substantial reduction of both number of steps and computation time.