Sequential rank agreement methods for comparison of ranked lists

TL;DR

This paper introduces sequential rank agreement methods to compare multiple ranked lists, providing an intuitive, flexible measure of agreement that can handle incomplete or censored data, with applications in genetics and cancer studies.

Contribution

It develops a new agreement measure based on standard deviation of ranks, applicable to multiple lists and capable of identifying change-points in agreement.

Findings

Effective in gene ranking comparisons

Useful for assessing classifier agreement in cancer studies

Can visualize agreement changes as list depth varies

Abstract

The comparison of alternative rankings of a set of items is a general and prominent task in applied statistics. Predictor variables are ranked according to magnitude of association with an outcome, prediction models rank subjects according to the personalized risk of an event, and genetic studies rank genes according to their difference in gene expression levels. This article constructs measures of the agreement of two or more ordered lists. We use the standard deviation of the ranks to define a measure of agreement that both provides an intuitive interpretation and can be applied to any number of lists even if some or all are incomplete or censored. The approach can identify change-points in the agreement of the lists and the sequential changes of agreement as a function of the depth of the lists can be compared graphically to a permutation based reference set. The usefulness of these tools are illustrated using gene rankings, and using data from two Danish ovarian cancer studies where we assess the within and between agreement of different statistical classification methods.