BREATHE: Second-Order Gradients and Heteroscedastic Emulation based Design Space Exploration

TL;DR

BREATHE is a novel optimization framework that efficiently explores complex vector and graph-based design spaces using second-order gradients and heteroscedastic surrogate models, significantly outperforming existing methods.

Contribution

It introduces a constrained multi-objective optimization framework capable of handling both vector and graph-based spaces with improved sample efficiency and performance.

Findings

Achieves 64.1% higher performance than baseline in vector optimization.

Outperforms graph-based Bayesian optimization by up to 64.9%.

Up to 21.9× higher hypervolume in multi-objective tasks.

Abstract

Researchers constantly strive to explore larger and more complex search spaces in various scientific studies and physical experiments. However, such investigations often involve sophisticated simulators or time-consuming experiments that make exploring and observing new design samples challenging. Previous works that target such applications are typically sample-inefficient and restricted to vector search spaces. To address these limitations, this work proposes a constrained multi-objective optimization (MOO) framework, called BREATHE, that searches not only traditional vector-based design spaces but also graph-based design spaces to obtain best-performing graphs. It leverages second-order gradients and actively trains a heteroscedastic surrogate model for sample-efficient optimization. In a single-objective vector optimization application, it leads to 64.1% higher performance than the next-best baseline, random forest regression. In graph-based search, BREATHE outperforms the next-best baseline, i.e., a graphical version of Gaussian-process-based Bayesian optimization, with up to 64.9% higher performance. In a MOO task, it achieves up to 21.9$\times$ higher hypervolume than the state-of-the-art method, multi-objective Bayesian optimization (MOBOpt). BREATHE also outperforms the baseline methods on most standard MOO benchmark applications.