Efficient Calculation of Regular Simplex Gradients

TL;DR

This paper introduces an efficient method for calculating regular simplex gradients in high-dimensional spaces, reducing computational complexity from cubic to linear or quadratic, and enabling second-order approximations.

Contribution

It presents a novel approach to compute regular simplex gradients with significantly reduced overhead, including a method for second-order gradient approximation from first-order gradients.

Findings

Linear algebra overhead reduced to O(n) for regular simplexes

Second order gradient approximation can be cheaply obtained

Gradient computation complexity reduced to O(n^2) for aligned simplexes

Abstract

Simplex gradients are an essential feature of many derivative free optimization algorithms, and can be employed, for example, as part of the process of defining a direction of search, or as part of a termination criterion. The calculation of a general simplex gradient in $\mathbb{R}^n$ can be computationally expensive, and often requires an overhead operation count of $\mathcal{O}(n^3)$ and in some algorithms a storage overhead of $\mathcal{O}(n^2)$. In this work we demonstrate that the linear algebra overhead and storage costs can be reduced, both to $\mathcal{O}(n)$, when the simplex employed is regular and appropriately aligned. We also demonstrate that a second order gradient approximation can be obtained cheaply from a combination of two, first order (appropriately aligned) regular simplex gradients. Moreover, we show that, for an arbitrarily aligned regular simplex, the gradient can be computed in only $\mathcal{O}(n^2)$ operations.