Hessian approximations

TL;DR

This paper presents the nested-set Hessian approximation, a new second-order derivative-free method with proven error bounds, capable of efficiently approximating the Hessian matrix using structured evaluation points.

Contribution

It introduces the nested-set Hessian, a novel second-order approximation method based on the generalized simplex gradient, with flexible evaluation set sizes and proven error bounds.

Findings

Fewer function evaluations needed with favorable point structure.

Error bound proportional to the maximal radius of evaluation sets.

Advantages demonstrated through calculus-based approximation techniques.

Abstract

This work introduces the nested-set Hessian approximation, a second-order approximation method that can be used in any derivative-free optimization routine that requires such information. It is built on the foundation of the generalized simplex gradient and proved to have an error bound that is on the order of the maximal radius of the two sets used in its construction. We show that when the points used in the computation of the nested-set Hessian have a favourable structure, (n+1)(n+2)/2 function evaluations are sufficient to approximate the Hessian. However, the nested-set Hessian also allows for evaluation sets with more points without negating the error analysis. Two calculus-based approximation techniques of the Hessian are developed and some advantages of the same are demonstrated.