Relation Modeling and Distillation for Learning with Noisy Labels

TL;DR

This paper introduces RMDNet, a relation modeling and distillation framework that improves learning with noisy labels by modeling inter-sample relationships through self-supervised contrastive learning and knowledge distillation, enhancing robustness and generalization.

Contribution

The paper proposes RMDNet, a novel plug-and-play framework combining relation modeling and distillation to better handle noisy labels in representation learning.

Findings

RMDNet outperforms existing methods on benchmark datasets.

The relation modeling module effectively eliminates noise interference.

The framework improves model generalization in noisy label scenarios.

Abstract

Learning with noisy labels has become an effective strategy for enhancing the robustness of models, which enables models to better tolerate inaccurate data. Existing methods either focus on optimizing the loss function to mitigate the interference from noise, or design procedures to detect potential noise and correct errors. However, their effectiveness is often compromised in representation learning due to the dilemma where models overfit to noisy labels. To address this issue, this paper proposes a relation modeling and distillation framework that models inter-sample relationships via self-supervised learning and employs knowledge distillation to enhance understanding of latent associations, which mitigate the impact of noisy labels. Specifically, the proposed method, termed RMDNet, includes two main modules, where the relation modeling (RM) module implements the contrastive learning technique to learn representations of all data, an unsupervised approach that effectively eliminates the interference of noisy tags on feature extraction. The relation-guided representation learning (RGRL) module utilizes inter-sample relation learned from the RM module to calibrate the representation distribution for noisy samples, which is capable of improving the generalization of the model in the inference phase. Notably, the proposed RMDNet is a plug-and-play framework that can integrate multiple methods to its advantage. Extensive experiments were conducted on two datasets, including performance comparison, ablation study, in-depth analysis and case study. The results show that RMDNet can learn discriminative representations for noisy data, which results in superior performance than the existing methods.