A robust BFGS algorithm for unconstrained nonlinear optimization problems

TL;DR

This paper introduces a modified BFGS algorithm that adaptively combines steepest descent and Newton methods, ensuring global convergence and super-linear local convergence for nonlinear optimization problems.

Contribution

The paper proposes a dynamically adaptive modified BFGS algorithm with proven convergence properties and comparable computational cost to standard BFGS, enhancing robustness and efficiency.

Findings

Proven global convergence for any twice differentiable nonlinear function.

Super-linear convergence near local optima with positive definite Hessian.

Numerical tests show competitive performance against existing algorithms.

Abstract

In this paper, a modified BFGS algorithm is proposed. The modified BFGS matrix estimates a modified Hessian matrix which is a convex combination of an identity matrix for the steepest descent algorithm and a Hessian matrix for the Newton algorithm. The coefficient of the convex combination in the modified BFGS algorithm is dynamically chosen in every iteration. It is proved that, for any twice differentiable nonlinear function (convex or non-convex), the algorithm is globally convergent to a stationary point. If the stationary point is a local optimizer where the Hessian is strongly positive definite in a neighborhood of the optimizer, the iterates will eventually enter and stay in the neighborhood, and the modified BFGS algorithm reduces to the BFGS algorithm in this neighborhood. Therefore, the modified BFGS algorithm is super-linearly convergent. Moreover, the computational cost of the modified BFGS in each iteration is almost the same as the cost of the BFGS. Numerical test on the CUTE test set is reported. The performance of the modified BFGS algorithm implemented in our MATLAB function is compared to the BFGS algorithm implemented in the MATLAB Optimization Toolbox function, a limited memory BFGS implemented as L-BFGS, a descent conjugate gradient algorithm implemented as CG-Descent 5.3, and a limited memory, descent and conjugate algorithm implemented as L-CG-Descent. This result shows that the modified BFGS algorithm may be very effective.