Cost-Accuracy Aware Adaptive Labeling for Active Learning

TL;DR

This paper introduces a cost-accuracy aware active learning algorithm that optimally selects instances and labelers with varying costs and accuracies to improve generalization performance efficiently.

Contribution

It proposes a novel algorithm that balances labeling costs and accuracies using generalization bounds, addressing diverse labeler settings in active learning.

Findings

Achieves higher generalization accuracy at lower costs.

Demonstrates state-of-the-art performance on multiple datasets.

Effectively manages diverse labeler costs and accuracies.

Abstract

Conventional active learning algorithms assume a single labeler that produces noiseless label at a given, fixed cost, and aim to achieve the best generalization performance for given classifier under a budget constraint. However, in many real settings, different labelers have different labeling costs and can yield different labeling accuracies. Moreover, a given labeler may exhibit different labeling accuracies for different instances. This setting can be referred to as active learning with diverse labelers with varying costs and accuracies, and it arises in many important real settings. It is therefore beneficial to understand how to effectively trade-off between labeling accuracy for different instances, labeling costs, as well as the informativeness of training instances, so as to achieve the best generalization performance at the lowest labeling cost. In this paper, we propose a new algorithm for selecting instances, labelers (and their corresponding costs and labeling accuracies), that employs generalization bound of learning with label noise to select informative instances and labelers so as to achieve higher generalization accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.