Predicting Software Performance with Divide-and-Learn

TL;DR

This paper introduces DaL, a divide-and-learn approach using local deep neural networks to improve software performance prediction accuracy in sparse configuration landscapes, outperforming state-of-the-art methods.

Contribution

The paper proposes a novel divide-and-learn method that effectively handles sample sparsity in software performance prediction using local models, enhancing accuracy and efficiency.

Findings

DaL outperforms existing methods in 33 out of 40 cases.

DaL achieves up to 1.94x improvement in prediction accuracy.

DaL requires fewer samples for comparable or better accuracy.

Abstract

Predicting the performance of highly configurable software systems is the foundation for performance testing and quality assurance. To that end, recent work has been relying on machine/deep learning to model software performance. However, a crucial yet unaddressed challenge is how to cater for the sparsity inherited from the configuration landscape: the influence of configuration options (features) and the distribution of data samples are highly sparse. In this paper, we propose an approach based on the concept of 'divide-and-learn', dubbed DaL. The basic idea is that, to handle sample sparsity, we divide the samples from the configuration landscape into distant divisions, for each of which we build a regularized Deep Neural Network as the local model to deal with the feature sparsity. A newly given configuration would then be assigned to the right model of division for the final prediction. Experiment results from eight real-world systems and five sets of training data reveal that, compared with the state-of-the-art approaches, DaL performs no worse than the best counterpart on 33 out of 40 cases (within which 26 cases are significantly better) with up to 1.94x improvement on accuracy; requires fewer samples to reach the same/better accuracy; and producing acceptable training overhead. Practically, DaL also considerably improves different global models when using them as the underlying local models, which further strengthens its flexibility. To promote open science, all the data, code, and supplementary figures of this work can be accessed at our repository: https://github.com/ideas-labo/DaL.