GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models

TL;DR

GIT-BO introduces a high-dimensional Bayesian optimization framework that leverages tabular foundation models and active subspace methods to improve efficiency and scalability without retraining.

Contribution

It combines a tabular foundation model with gradient-based subspace exploration for scalable, zero-shot Bayesian optimization in high dimensions.

Findings

Outperforms state-of-the-art GP-based BO methods in high-dimensional benchmarks.

Requires no online retraining, reducing computational overhead.

Effective across diverse real-world and synthetic problems up to 500 dimensions.

Abstract

Bayesian optimization (BO) struggles in high dimensions, where Gaussian-process surrogates demand heavy retraining and brittle assumptions, slowing progress on real engineering and design problems. We introduce GIT-BO, a Gradient-Informed BO framework that couples TabPFN v2, a tabular foundation model that performs zero-shot Bayesian inference in context, with an active-subspace mechanism computed from the model's own predictive-mean gradients. This aligns exploration to an intrinsic low-dimensional subspace via a Fisher-information estimate and selects queries with a UCB acquisition, requiring no online retraining. Across 60 problem variants spanning 20 benchmarks-nine scalable synthetic families and ten real-world tasks (e.g., power systems, Rover, MOPTA08, Mazda)-up to 500 dimensions, GIT-BO delivers a stronger performance-time trade-off than state-of-the-art GP-based methods (SAASBO, TuRBO, Vanilla BO, BAxUS), ranking highest in performance and with runtime advantages that grow with dimensionality. Limitations include memory footprint and dependence on the capacity of the underlying TFM.