Beyond the training set: an intuitive method for detecting distribution shift in model-based optimization

TL;DR

This paper introduces a simple, intuitive method for detecting distribution shifts in model-based optimization by training a binary classifier to distinguish between training and design data, helping improve design quality.

Contribution

The paper proposes a straightforward classifier-based approach to identify distribution shifts in MBO, aiding practitioners in maintaining model reliability during optimization.

Findings

The method effectively detects distribution shifts in real-world MBO applications.

Distribution shift intensity correlates with the number of optimization steps.

Using the shift detection improves the reliability and quality of generated designs.

Abstract

Model-based optimization (MBO) is increasingly applied to design problems in science and engineering. A common scenario involves using a fixed training set to train models, with the goal of designing new samples that outperform those present in the training data. A major challenge in this setting is distribution shift, where the distributions of training and design samples are different. While some shift is expected, as the goal is to create better designs, this change can negatively affect model accuracy and subsequently, design quality. Despite the widespread nature of this problem, addressing it demands deep domain knowledge and artful application. To tackle this issue, we propose a straightforward method for design practitioners that detects distribution shifts. This method trains a binary classifier using knowledge of the unlabeled design distribution to separate the training data from the design data. The classifier's logit scores are then used as a proxy measure of distribution shift. We validate our method in a real-world application by running offline MBO and evaluate the effect of distribution shift on design quality. We find that the intensity of the shift in the design distribution varies based on the number of steps taken by the optimization algorithm, and our simple approach can identify these shifts. This enables users to constrain their search to regions where the model's predictions are reliable, thereby increasing the quality of designs.