A Robust UCB Scheme for Active Learning in Regression from Strategic Crowds

TL;DR

This paper introduces a Bayesian active learning framework for linear regression using crowdsourced labels, employing a robust UCB method for annotator selection, with theoretical guarantees and incentives for strategic annotators.

Contribution

It proposes a novel Bayesian model with variational inference, establishes equivalence of active learning criteria, and applies a robust UCB scheme for annotator selection with strategic incentives.

Findings

Decoupling of instance and annotator selection simplifies active learning.

Robust UCB scheme provides provable regret guarantees.

Incentive design effectively motivates strategic annotators.

Abstract

We study the problem of training an accurate linear regression model by procuring labels from multiple noisy crowd annotators, under a budget constraint. We propose a Bayesian model for linear regression in crowdsourcing and use variational inference for parameter estimation. To minimize the number of labels crowdsourced from the annotators, we adopt an active learning approach. In this specific context, we prove the equivalence of well-studied criteria of active learning like entropy minimization and expected error reduction. Interestingly, we observe that we can decouple the problems of identifying an optimal unlabeled instance and identifying an annotator to label it. We observe a useful connection between the multi-armed bandit framework and the annotator selection in active learning. Due to the nature of the distribution of the rewards on the arms, we use the Robust Upper Confidence Bound (UCB) scheme with truncated empirical mean estimator to solve the annotator selection problem. This yields provable guarantees on the regret. We further apply our model to the scenario where annotators are strategic and design suitable incentives to induce them to put in their best efforts.