Pushing the Limits of the Reactive Affine Shaker Algorithm to Higher Dimensions

TL;DR

This paper introduces the Reactive Affine Shaker, a simple stochastic local search algorithm for high-dimensional optimization, which adapts its search space through affine transformations without using function values directly.

Contribution

The paper presents a novel, simple stochastic search method that adapts the search region via affine transformations, achieving competitive results in high-dimensional optimization.

Findings

RAS performs comparably to state-of-the-art high-dimensional Bayesian Optimization.

The affine transformation approach effectively guides the search in very large dimensions.

The ablation study highlights the importance of directional probability distributions in RAS.

Abstract

Bayesian Optimization (BO) for the minimization of expensive functions of continuous variables uses all the knowledge acquired from previous samples (${\boldsymbol x}_i$ and $f({\boldsymbol x}_i)$ values) to build a surrogate model based on Gaussian processes. The surrogate is then exploited to define the next point to sample, through a careful balance of exploration and exploitation. Initially intended for low-dimensional spaces, BO has recently been modified and used also for very large-dimensional spaces (up to about one thousand dimensions). In this paper we consider a much simpler algorithm, called "Reactive Affine Shaker" (RAS). The next sample is always generated with a uniform probability distribution inside a parallelepiped (the "box"). At each iteration, the form of the box is adapted during the search through an affine transformation, based only on the point $\boldsymbol x$ position and on the success or failure in improving the function. The function values are therefore not used directly to modify the search area and to generate the next sample. The entire dimensionality is kept (no active subspaces). Despite its extreme simplicity and its use of only stochastic local search, surprisingly the produced results are comparable to and not too far from the state-of-the-art results of high-dimensional versions of BO, although with some more function evaluations. An ablation study and an analysis of probability distribution of directions (improving steps and prevailing box orientation) in very large-dimensional spaces are conducted to understand more about the behavior of RAS and to assess the relative importance of the algorithmic building blocks for the final results.