Decreasing the Computing Time of Bayesian Optimization using Generalizable Memory Pruning

TL;DR

This paper introduces a generalizable memory pruning wrapper for Bayesian optimization that significantly reduces computation time across various models and data sets without compromising convergence.

Contribution

It presents a novel, model-agnostic memory pruning method that decreases Bayesian optimization's computational complexity from polynomial to a non-increasing pattern, applicable to any surrogate model and acquisition function.

Findings

Reduced wall-clock time from polynomial to sawtooth pattern

Demonstrated effectiveness across multiple data sets and models

Maintained convergence performance despite pruning

Abstract

Bayesian optimization (BO) suffers from long computing times when processing highly-dimensional or large data sets. These long computing times are a result of the Gaussian process surrogate model having a polynomial time complexity with the number of experiments. Running BO on high-dimensional or massive data sets becomes intractable due to this time complexity scaling, in turn, hindering experimentation. Alternative surrogate models have been developed to reduce the computing utilization of the BO procedure, however, these methods require mathematical alteration of the inherit surrogate function, pigeonholing use into only that function. In this paper, we demonstrate a generalizable BO wrapper of memory pruning and bounded optimization, capable of being used with any surrogate model and acquisition function. Using this memory pruning approach, we show a decrease in wall-clock computing times per experiment of BO from a polynomially increasing pattern to a sawtooth pattern that has a non-increasing trend without sacrificing convergence performance. Furthermore, we illustrate the generalizability of the approach across two unique data sets, two unique surrogate models, and four unique acquisition functions. All model implementations are run on the MIT Supercloud state-of-the-art computing hardware.