Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis

TL;DR

This paper investigates when transfer learning effectively improves performance modeling of configurable systems, showing that small environmental changes benefit from linear transformations, while larger changes require more efficient sampling strategies.

Contribution

The study provides empirical insights into the conditions under which transfer learning enhances performance modeling, highlighting different strategies for small versus severe environmental changes.

Findings

Linear transformations help in small environmental changes

Sampling efficiency improves with transfer learning in severe changes

Transfer learning reduces modeling effort across various configurations

Abstract

Modern software systems provide many configuration options which significantly influence their non-functional properties. To understand and predict the effect of configuration options, several sampling and learning strategies have been proposed, albeit often with significant cost to cover the highly dimensional configuration space. Recently, transfer learning has been applied to reduce the effort of constructing performance models by transferring knowledge about performance behavior across environments. While this line of research is promising to learn more accurate models at a lower cost, it is unclear why and when transfer learning works for performance modeling. To shed light on when it is beneficial to apply transfer learning, we conducted an empirical study on four popular software systems, varying software configurations and environmental conditions, such as hardware, workload, and software versions, to identify the key knowledge pieces that can be exploited for transfer learning. Our results show that in small environmental changes (e.g., homogeneous workload change), by applying a linear transformation to the performance model, we can understand the performance behavior of the target environment, while for severe environmental changes (e.g., drastic workload change) we can transfer only knowledge that makes sampling more efficient, e.g., by reducing the dimensionality of the configuration space.