A general framework for floating point error analysis of simplex derivatives

TL;DR

This paper develops a comprehensive framework to analyze floating point errors in various simplex derivatives used in derivative-free optimization, enhancing understanding of their accuracy limits.

Contribution

It introduces a general, adaptable framework for floating point error analysis of simplex derivatives, applicable to multiple variants and guiding optimal sample set choices.

Findings

Framework applies to generalized simplex derivatives

Provides error bounds for different simplex gradient variants

Suggests minimal sample set diameters for accuracy

Abstract

Gradient approximations are a class of numerical approximation techniques that are of central importance in numerical optimization. In derivative-free optimization, most of the gradient approximations, including the simplex gradient, centred simplex gradient, and adapted centred simplex gradient, are in the form of simplex derivatives. Owing to machine precision, the approximation accuracy of any numerical approximation technique is subject to the influence of floating point errors. In this paper, we provide a general framework for floating point error analysis of simplex derivatives. Our framework is independent of the choice of the simplex derivative as long as it satisfies a general form. We review the definition and approximation accuracy of the generalized simplex gradient and generalized centred simplex gradient. We define and analyze the accuracy of a generalized version of the adapted centred simplex gradient. As examples, we apply our framework to the generalized simplex gradient, generalized centred simplex gradient, and generalized adapted centred simplex gradient. Based on the results, we give suggestions on the minimal choice of approximate diameter of the sample set.