Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations

TL;DR

This paper introduces a transfer learning method for surrogate models that combines domain warping and affine transformations, enabling effective adaptation to new tasks with limited data, including nonlinear variations.

Contribution

It extends previous work by addressing nonlinear input warping combined with affine transformations for surrogate transfer learning.

Findings

Transferred surrogate models outperform original models in data-scarce scenarios.

The method effectively handles nonlinear input transformations.

Validation on BBOB and real-world automotive tasks demonstrates significant improvements.

Abstract

Surrogate models provide efficient alternatives to computationally demanding real world processes but often require large datasets for effective training. A promising solution to this limitation is the transfer of pre-trained surrogate models to new tasks. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models, typically assuming an affine transformation between the source and target functions. This paper extends previous research by addressing a broader range of transformations, including linear and nonlinear variations. Specifically, we consider the combination of an unknown input warping, such as one modeled by the beta cumulative distribution function, with an unspecified affine transformation. Our approach achieves transfer learning by employing a limited number of data points from the target task to optimize these transformations, minimizing empirical loss on the transfer dataset. We validate the proposed method on the widely used Black-Box Optimization Benchmark (BBOB) testbed and a real-world transfer learning task from the automobile industry. The results underscore the significant advantages of the approach, revealing that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch using the transfer dataset, particularly in data-scarce scenarios.