Estimating the Accuracies of Multiple Classifiers Without Labeled Data

TL;DR

This paper introduces efficient spectral algorithms to estimate classifier accuracies and build improved unsupervised ensemble classifiers using only unlabeled predictions, under independence assumptions.

Contribution

It presents novel spectral methods for accuracy estimation and ensemble construction without labeled data, with proven consistency and asymptotic analysis.

Findings

Algorithms are computationally efficient and scalable.

Methods achieve competitive accuracy in experiments.

Theoretical guarantees under classifier independence.

Abstract

In various situations one is given only the predictions of multiple classifiers over a large unlabeled test data. This scenario raises the following questions: Without any labeled data and without any a-priori knowledge about the reliability of these different classifiers, is it possible to consistently and computationally efficiently estimate their accuracies? Furthermore, also in a completely unsupervised manner, can one construct a more accurate unsupervised ensemble classifier? In this paper, focusing on the binary case, we present simple, computationally efficient algorithms to solve these questions. Furthermore, under standard classifier independence assumptions, we prove our methods are consistent and study their asymptotic error. Our approach is spectral, based on the fact that the off-diagonal entries of the classifiers' covariance matrix and 3-d tensor are rank-one. We illustrate the competitive performance of our algorithms via extensive experiments on both artificial and real datasets.