Active Learning from Imperfect Labelers

TL;DR

This paper introduces an active learning algorithm that effectively handles labelers who can abstain or provide incorrect labels, achieving near-optimal query complexity under various noise and abstention conditions.

Contribution

It proposes a novel algorithm that utilizes abstention responses, analyzes its statistical properties, and demonstrates near-optimal query complexity under realistic labeler assumptions.

Findings

Algorithm is statistically consistent.

Achieves near-optimal query complexity.

Adapts to different noise and abstention rates.

Abstract

We study active learning where the labeler can not only return incorrect labels but also abstain from labeling. We consider different noise and abstention conditions of the labeler. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under fairly natural assumptions on the noise and abstention rate of the labeler. This algorithm is adaptive in a sense that it can automatically request less queries with a more informed or less noisy labeler. We couple our algorithm with lower bounds to show that under some technical conditions, it achieves nearly optimal query complexity.