Ranking Under Uncertainty

TL;DR

This paper introduces an analytical approach to evaluate how noise affects the reliability of ranking objects, especially in noisy biological data, revealing that current experiment sizes are insufficient for stable rankings.

Contribution

It provides a novel analytical framework for assessing ranking reliability under noise, comparing similarity measures, and applying it to gene selection in cancer microarray data.

Findings

Top-K-List overlap is more sensitive to noise than Kendall's tau.

Gene rankings in microarray experiments are unreliable with current sample sizes.

Larger sample sizes are needed for stable and reliable rankings.

Abstract

Ranking objects is a simple and natural procedure for organizing data. It is often performed by assigning a quality score to each object according to its relevance to the problem at hand. Ranking is widely used for object selection, when resources are limited and it is necessary to select a subset of most relevant objects for further processing. In real world situations, the object's scores are often calculated from noisy measurements, casting doubt on the ranking reliability. We introduce an analytical method for assessing the influence of noise levels on the ranking reliability. We use two similarity measures for reliability evaluation, Top-K-List overlap and Kendall's tau measure, and show that the former is much more sensitive to noise than the latter. We apply our method to gene selection in a series of microarray experiments of several cancer types. The results indicate that the reliability of the lists obtained from these experiments is very poor, and that experiment sizes which are necessary for attaining reasonably stable Top-K-Lists are much larger than those currently available. Simulations support our analytical results.