Label Denoising through Cross-Model Agreement

TL;DR

This paper introduces DeCA, a cross-model agreement framework that improves learning from noisy labels by leveraging the similarity in predictions on clean data and differences on noisy data across models.

Contribution

The paper proposes a novel denoising method using cross-model agreement, effective for both binary and multi-label noisy data scenarios.

Findings

DeCA significantly outperforms standard training methods.

It improves robustness in recommendation and image classification tasks.

Cross-model agreement effectively reduces label noise impact.

Abstract

Learning from corrupted labels is very common in real-world machine-learning applications. Memorizing such noisy labels could affect the learning of the model, leading to sub-optimal performances. In this work, we propose a novel framework to learn robust machine-learning models from noisy labels. Through an empirical study, we find that different models make relatively similar predictions on clean examples, while the predictions on noisy examples vary much more across different models. Motivated by this observation, we propose \em denoising with cross-model agreement \em (DeCA) which aims to minimize the KL-divergence between the true label distributions parameterized by two machine learning models while maximizing the likelihood of data observation. We employ the proposed DeCA on both the binary label scenario and the multiple label scenario. For the binary label scenario, we select implicit feedback recommendation as the downstream task and conduct experiments with four state-of-the-art recommendation models on four datasets. For the multiple-label scenario, the downstream application is image classification on two benchmark datasets. Experimental results demonstrate that the proposed methods significantly improve the model performance compared with normal training and other denoising methods on both binary and multiple-label scenarios.