Ranking with Ties based on Noisy Performance Data

TL;DR

This paper addresses the challenge of ranking objects based on noisy performance data, proposing methods to compute partial rankings that handle overlaps and incomparabilities.

Contribution

It introduces three methodologies for computing partial rankings in noisy, overlapping performance data scenarios, addressing the non-transitivity issue.

Findings

Developed three methods for partial ranking computation.

Analyzed the properties of these ranking methodologies.

Demonstrated how to investigate performance differences using partial rankings.

Abstract

We consider the problem of ranking a set of objects based on their performance when the measurement of said performance is subject to noise. In this scenario, the performance is measured repeatedly, resulting in a range of measurements for each object. If the ranges of two objects do not overlap, then we consider one object as 'better' than the other, and we expect it to receive a higher rank; if, however, the ranges overlap, then the objects are incomparable, and we wish them to be assigned the same rank. Unfortunately, the incomparability relation of ranges is in general not transitive; as a consequence, in general the two requirements cannot be satisfied simultaneously, i.e., it is not possible to guarantee both distinct ranks for objects with separated ranges, and same rank for objects with overlapping ranges. This conflict leads to more than one reasonable way to rank a set of objects. In this paper, we explore the ambiguities that arise when ranking with ties, and define a set of reasonable rankings, which we call partial rankings. We develop and analyse three different methodologies to compute a partial ranking. Finally, we show how performance differences among objects can be investigated with the help of partial ranking.