Measurement of software development effort estimation bias: Avoiding biased measures of estimation bias

TL;DR

This paper improves the measurement of effort estimation bias in software development by highlighting the importance of matching bias measures to estimate types and demonstrating limitations of common bias metrics.

Contribution

It introduces a refined approach to measuring estimation bias, emphasizing the importance of estimate type and analyzing limitations of existing bias measures.

Findings

Common bias measures can be misleading for mean effort estimates

Matching bias measures to estimate types improves accuracy

No practical bias measure exists for most likely effort estimates

Abstract

In this paper, we propose improvements in how estimation bias, e.g., the tendency towards under-estimating the effort, is measured. The proposed approach emphasizes the need to know what the estimates are meant to represent, i.e., the type of estimate we evaluate and the need for a match between the type of estimate given and the bias measure used. We show that even perfect estimates of the mean effort will not lead to an expectation of zero estimation bias when applying the frequently used bias measure: (actual effort - estimated effort)/actual effort. This measure will instead reward under-estimates of the mean effort. We also provide examples of bias measures that match estimates of the mean and the median effort, and argue that there are, in general, no practical bias measures for estimates of the most likely effort. The paper concludes with implications for the evaluation of bias of software development effort estimates.