Beyond Disagreement-based Agnostic Active Learning

TL;DR

This paper introduces a new algorithm for agnostic active learning that reduces label complexity by combining confidence-rated prediction with a novel predictor, applicable to general classification tasks.

Contribution

It presents a novel reduction from consistent active learning to confidence-rated prediction and introduces a new confidence-rated predictor, improving label efficiency in agnostic settings.

Findings

Achieves better label complexity in agnostic active learning

Provides a confidence-rated predictor with guaranteed error

Applicable to general classification problems

Abstract

We study agnostic active learning, where the goal is to learn a classifier in a pre-specified hypothesis class interactively with as few label queries as possible, while making no assumptions on the true function generating the labels. The main algorithms for this problem are {\em{disagreement-based active learning}}, which has a high label requirement, and {\em{margin-based active learning}}, which only applies to fairly restricted settings. A major challenge is to find an algorithm which achieves better label complexity, is consistent in an agnostic setting, and applies to general classification problems. In this paper, we provide such an algorithm. Our solution is based on two novel contributions -- a reduction from consistent active learning to confidence-rated prediction with guaranteed error, and a novel confidence-rated predictor.