Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks

TL;DR

This paper introduces BDLPerf, a Bayesian neural network approach for predicting software system performance across configurations, effectively modeling uncertainty and providing reliable confidence intervals to improve prediction accuracy.

Contribution

It presents a novel Bayesian deep learning method that incorporates uncertainty calibration and efficient hyperparameter tuning for performance prediction in configurable software systems.

Findings

BDLPerf outperforms existing methods in accuracy.

It provides reliable confidence intervals for predictions.

The approach is validated on 10 real-world systems.

Abstract

Configurable software systems are employed in many important application domains. Understanding the performance of the systems under all configurations is critical to prevent potential performance issues caused by misconfiguration. However, as the number of configurations can be prohibitively large, it is not possible to measure the system performance under all configurations. Thus, a common approach is to build a prediction model from a limited measurement data to predict the performance of all configurations as scalar values. However, it has been pointed out that there are different sources of uncertainty coming from the data collection or the modeling process, which can make the scalar predictions not certainly accurate. To address this problem, we propose a Bayesian deep learning based method, namely BDLPerf, that can incorporate uncertainty into the prediction model. BDLPerf can provide both scalar predictions for configurations' performance and the corresponding confidence intervals of these scalar predictions. We also develop a novel uncertainty calibration technique to ensure the reliability of the confidence intervals generated by a Bayesian prediction model. Finally, we suggest an efficient hyperparameter tuning technique so as to train the prediction model within a reasonable amount of time whilst achieving high accuracy. Our experimental results on 10 real-world systems show that BDLPerf achieves higher accuracy than existing approaches, in both scalar performance prediction and confidence interval estimation.