The Relationship Between Agnostic Selective Classification Active Learning and the Disagreement Coefficient

TL;DR

This paper explores the relationship between agnostic selective classification, active learning, and the disagreement coefficient, introducing an improved algorithm that achieves fast rejection rates without strong assumptions.

Contribution

It introduces ILESS, an improved PCS learning algorithm that achieves fast rejection rates in the agnostic setting without strong assumptions, and establishes an equivalence with the disagreement coefficient.

Findings

ILESS achieves fast rejection rates without assumptions.

Fast rejection rate linked to poly-logarithmic disagreement coefficient.

Exponential speedup in active learning with ActiveiLESS.

Abstract

A selective classifier (f,g) comprises a classification function f and a binary selection function g, which determines if the classifier abstains from prediction, or uses f to predict. The classifier is called pointwise-competitive if it classifies each point identically to the best classifier in hindsight (from the same class), whenever it does not abstain. The quality of such a classifier is quantified by its rejection mass, defined to be the probability mass of the points it rejects. A "fast" rejection rate is achieved if the rejection mass is bounded from above by O(1/m) where m is the number of labeled examples used to train the classifier (and O hides logarithmic factors). Pointwise-competitive selective (PCS) classifiers are intimately related to disagreement-based active learning and it is known that in the realizable case, a fast rejection rate of a known PCS algorithm (called Consistent Selective Strategy) is equivalent to an exponential speedup of the well-known CAL active algorithm. We focus on the agnostic setting, for which there is a known algorithm called LESS that learns a PCS classifier and achieves a fast rejection rate (depending on Hanneke's disagreement coefficient) under strong assumptions. We present an improved PCS learning algorithm called ILESS for which we show a fast rate (depending on Hanneke's disagreement coefficient) without any assumptions. Our rejection bound smoothly interpolates the realizable and agnostic settings. The main result of this paper is an equivalence between the following three entities: (i) the existence of a fast rejection rate for any PCS learning algorithm (such as ILESS); (ii) a poly-logarithmic bound for Hanneke's disagreement coefficient; and (iii) an exponential speedup for a new disagreement-based active learner called ActiveiLESS.