A model aggregation approach for high-dimensional large-scale optimization

TL;DR

This paper introduces MamBO, a model aggregation-based Bayesian optimization method designed to efficiently solve high-dimensional, large-scale problems by combining subsampling, subspace embeddings, and model aggregation to enhance robustness and convergence.

Contribution

The paper proposes a novel model aggregation approach within Bayesian optimization to address surrogate model uncertainty in high-dimensional, large-scale settings, with theoretical convergence guarantees.

Findings

MamBO achieves superior or comparable performance to existing high-dimensional BO algorithms.

MamBO finds higher classification accuracy in face detection tasks.

MamBO is computationally faster than other high-dimensional BO methods.

Abstract

Bayesian optimization (BO) has been widely used in machine learning and simulation optimization. With the increase in computational resources and storage capacities in these fields, high-dimensional and large-scale problems are becoming increasingly common. In this study, we propose a model aggregation method in the Bayesian optimization (MamBO) algorithm for efficiently solving high-dimensional large-scale optimization problems. MamBO uses a combination of subsampling and subspace embeddings to collectively address high dimensionality and large-scale issues; in addition, a model aggregation method is employed to address the surrogate model uncertainty issue that arises when embedding is applied. This surrogate model uncertainty issue is largely ignored in the embedding literature and practice, and it is exacerbated when the problem is high-dimensional and data are limited. Our proposed model aggregation method reduces these lower-dimensional surrogate model risks and improves the robustness of the BO algorithm. We derive an asymptotic bound for the proposed aggregated surrogate model and prove the convergence of MamBO. Benchmark numerical experiments indicate that our algorithm achieves superior or comparable performance to other commonly used high-dimensional BO algorithms. Moreover, we apply MamBO to a cascade classifier of a machine learning algorithm for face detection, and the results reveal that MamBO finds settings that achieve higher classification accuracy than the benchmark settings and is computationally faster than other high-dimensional BO algorithms.