Bayesian Pool-based Active Learning With Abstention Feedbacks

TL;DR

This paper introduces Bayesian algorithms for pool-based active learning with abstention feedbacks, effectively estimating abstention rates and achieving near-optimal utility guarantees, with strong empirical performance.

Contribution

Develops two greedy algorithms that learn classification and abstention rates simultaneously, with proven near-optimality guarantees.

Findings

Algorithms perform well in practical scenarios.

Achieve a ${(1-rac{1}{e})}$ approximation of optimal utility.

Effectively estimate unknown abstention rates.

Abstract

We study pool-based active learning with abstention feedbacks, where a labeler can abstain from labeling a queried example with some unknown abstention rate. This is an important problem with many useful applications. We take a Bayesian approach to the problem and develop two new greedy algorithms that learn both the classification problem and the unknown abstention rate at the same time. These are achieved by simply incorporating the estimated abstention rate into the greedy criteria. We prove that both of our algorithms have near-optimality guarantees: they respectively achieve a ${(1-\frac{1}{e})}$ constant factor approximation of the optimal expected or worst-case value of a useful utility function. Our experiments show the algorithms perform well in various practical scenarios.