Analyzing the Stationarity Process in Software Effort Estimation Datasets

TL;DR

This paper tests the stationarity assumption in software effort estimation datasets using kernel estimators and finds that nonstationary datasets are better modeled with uniform weighting, challenging the common assumption of data relevance over time.

Contribution

It introduces a method to test stationarity in effort estimation datasets and compares weighted and unweighted models, revealing the impact of data stationarity on model accuracy.

Findings

Nonstationary datasets favor uniform models over kernel-weighted models.

Stationarity affects the accuracy of effort estimation models.

Model accuracy is independent of the kernel estimator used.

Abstract

Software effort estimation models are typically developed based on an underlying assumption that all data points are equally relevant to the prediction of effort for future projects. The dynamic nature of several aspects of the software engineering process could mean that this assumption does not hold in at least some cases. This study employs three kernel estimator functions to test the stationarity assumption in five software engineering datasets that have been used in the construction of software effort estimation models. The kernel estimators are used in the generation of nonuniform weights which are subsequently employed in weighted linear regression modeling. In each model, older projects are assigned smaller weights while the more recently completed projects are assigned larger weights, to reflect their potentially greater relevance to present or future projects that need to be estimated. Prediction errors are compared to those obtained from uniform models. Our results indicate that, for the datasets that exhibit underlying nonstationary processes, uniform models are more accurate than the nonuniform models; that is, models based on kernel estimator functions are worse than the models where no weighting was applied. In contrast, the accuracies of uniform and nonuniform models for datasets that exhibited stationary processes were essentially equivalent. Our analysis indicates that as the heterogeneity of a dataset increases, the effect of stationarity is overridden. The results of our study also confirm prior findings that the accuracy of effort estimation models is independent of the type of kernel estimator function used in model development.