Ranking and combining multiple predictors without labeled data

TL;DR

This paper introduces a spectral method to rank and combine multiple classifiers without labeled data, enabling the construction of a more accurate ensemble classifier and providing robustness against malicious classifiers.

Contribution

It presents a novel spectral approach for ranking classifiers and constructing an ensemble without labeled data, including the Spectral Meta-Learner (SML) for improved accuracy.

Findings

SML outperforms most individual classifiers in accuracy.

The spectral method reliably ranks classifiers based on their balanced accuracy.

SML is robust to malicious classifiers designed to mislead the ensemble.

Abstract

In a broad range of classification and decision making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier accuracy can be assessed using available labeled data, and raises two questions: given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to a) reliably rank them; and b) construct a meta-classifier more accurate than most classifiers in the ensemble? Here we present a novel spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, as its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), a novel ensemble classifier whose weights are equal to this eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting, for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.