Using the bootstrap to quantify the authority of an empirical ranking

TL;DR

This paper critically examines the bootstrap method for ranking confidence, revealing its limitations and proposing improved variants, especially in high-dimensional and dependent data contexts like genomics.

Contribution

It demonstrates the inconsistency of the standard bootstrap for empirical ranks and introduces an independent component bootstrap to address dependence issues.

Findings

Standard bootstrap can be inconsistent for ranks.

Bootstrap intervals may be accurate in order but not in magnitude.

Independent component bootstrap improves performance with dependent data.

Abstract

The bootstrap is a popular and convenient method for quantifying the authority of an empirical ordering of attributes, for example of a ranking of the performance of institutions or of the influence of genes on a response variable. In the first of these examples, the number, $p$, of quantities being ordered is sometimes only moderate in size; in the second it can be very large, often much greater than sample size. However, we show that in both types of problem the conventional bootstrap can produce inconsistency. Moreover, the standard $n$-out-of-$n$ bootstrap estimator of the distribution of an empirical rank may not converge in the usual sense; the estimator may converge in distribution, but not in probability. Nevertheless, in many cases the bootstrap correctly identifies the support of the asymptotic distribution of ranks. In some contemporary problems, bootstrap prediction intervals for ranks are particularly long, and in this context, we also quantify the accuracy of bootstrap methods, showing that the standard bootstrap gets the order of magnitude of the interval right, but not the constant multiplier of interval length. The $m$-out-of-$n$ bootstrap can improve performance and produce statistical consistency, but it requires empirical choice of $m$; we suggest a tuning solution to this problem. We show that in genomic examples, where it might be expected that the standard, ``synchronous'' bootstrap will successfully accommodate nonindependence of vector components, that approach can produce misleading results. An ``independent component'' bootstrap can overcome these difficulties, even in cases where components are not strictly independent.