Robust Assignment of Labels for Active Learning with Sparse and Noisy Annotations

TL;DR

This paper introduces two novel algorithms for unifying noisy and sparse annotations in active learning, improving label reliability and accuracy in scenarios with limited or faulty expert labels.

Contribution

The paper proposes new annotation unification algorithms that effectively handle sparse and noisy labels, requiring minimal overlap between annotators, and outperform existing methods.

Findings

Algorithms improve label reliability in noisy annotation settings.

Proposed methods outperform state-of-the-art in experiments.

Robustness demonstrated across four public datasets.

Abstract

Supervised classification algorithms are used to solve a growing number of real-life problems around the globe. Their performance is strictly connected with the quality of labels used in training. Unfortunately, acquiring good-quality annotations for many tasks is infeasible or too expensive to be done in practice. To tackle this challenge, active learning algorithms are commonly employed to select only the most relevant data for labeling. However, this is possible only when the quality and quantity of labels acquired from experts are sufficient. Unfortunately, in many applications, a trade-off between annotating individual samples by multiple annotators to increase label quality vs. annotating new samples to increase the total number of labeled instances is necessary. In this paper, we address the issue of faulty data annotations in the context of active learning. In particular, we propose two novel annotation unification algorithms that utilize unlabeled parts of the sample space. The proposed methods require little to no intersection between samples annotated by different experts. Our experiments on four public datasets indicate the robustness and superiority of the proposed methods in both, the estimation of the annotator's reliability, and the assignment of actual labels, against the state-of-the-art algorithms and the simple majority voting.