To Aggregate or Not? Learning with Separate Noisy Labels

TL;DR

This paper investigates whether to aggregate noisy labels from multiple annotators or to use them separately, providing theoretical analysis and empirical evidence favoring separation under high noise or limited annotations.

Contribution

It offers a theoretical comparison of label aggregation versus separation, revealing conditions where separation outperforms aggregation in noisy label learning.

Findings

Separation is better when noise rates are high.

Separation outperforms aggregation with limited annotations.

Empirical results support the theoretical conclusions.

Abstract

The rawly collected training data often comes with separate noisy labels collected from multiple imperfect annotators (e.g., via crowdsourcing). A typical way of using these separate labels is to first aggregate them into one and apply standard training methods. The literature has also studied extensively on effective aggregation approaches. This paper revisits this choice and aims to provide an answer to the question of whether one should aggregate separate noisy labels into single ones or use them separately as given. We theoretically analyze the performance of both approaches under the empirical risk minimization framework for a number of popular loss functions, including the ones designed specifically for the problem of learning with noisy labels. Our theorems conclude that label separation is preferred over label aggregation when the noise rates are high, or the number of labelers/annotations is insufficient. Extensive empirical results validate our conclusions.