An Assessment of the Consistency for Software Measurement Methods

TL;DR

This paper evaluates the consistency of software measurement methods, proposing a statistical approach to compare different methods and assess improvements in their reliability across different raters.

Contribution

It introduces a homogeneous statistic for measuring consistency and provides a statistical framework to evaluate and compare the consistency of various software measurement methods.

Findings

Proposed a new statistic to quantify measurement consistency.

Developed a statistical analysis method for comparing measurement methods.

Facilitated empirical evaluation of revisions to improve consistency.

Abstract

Consistency, defined as the requirement that a series of measurements of the same project carried out by different raters using the same method should produce similar results, is one of the most important aspects to be taken into account in the measurement methods of the software. In spite of this, there is a widespread view that many measurement methods introduce an undesirable amount of subjectivity in the measurement process. This perception has made several organizations develop revisions of the standard methods whose main aim is to improve their consistency by introducing some suitable modifications of those aspects which are believed to introduce a greater degree of subjectivity.Each revision of a method must be empirically evaluated to determine to what extent is the aim of improving its consistency achieved. In this article we will define an homogeneous statistic intended to describe the consistency level of a method, and we will develop the statistical analysis which should be carried out in order to conclude whether or not a measurement method is more consistent than other one.