Expected decrease for derivative-free algorithms using random subspaces

TL;DR

This paper analyzes how using low-dimensional random subspaces in derivative-free optimization algorithms can improve their expected decrease per function evaluation, especially as problem dimension grows.

Contribution

The paper provides a theoretical analysis connecting subspace dimension to algorithmic guarantees in derivative-free methods, supported by numerical evidence.

Findings

Lower subspace dimensions lead to better expected decrease.

Numerical computations confirm the advantage of low-dimensional subspaces.

The analysis applies to both direct-search and model-based approaches.

Abstract

Derivative-free algorithms seek the minimum of a given function based only on function values queried at appropriate points. Although these methods are widely used in practice, their performance is known to worsen as the problem dimension increases. Recent advances in developing randomized derivative-free techniques have tackled this issue by working in low-dimensional subspaces that are drawn at random in an iterative fashion. The connection between the dimension of these random subspaces and the algorithmic guarantees has yet to be fully understood. In this paper, we develop an analysis for derivative-free algorithms (both direct-search and model-based approaches) employing random subspaces. Our results leverage linear local approximations of smooth functions to obtain understanding of the expected decrease achieved per function evaluation. Although the quantities of interest involve multidimensional integrals with no closed-form expression, a relative comparison for different subspace dimensions suggest that low dimension is preferable. Numerical computation of the quantities of interest confirm the benefit of operating in low-dimensional subspaces.