A Survey of Label-noise Representation Learning: Past, Present and Future

TL;DR

This survey comprehensively reviews Label-Noise Representation Learning (LNRL), analyzing its theoretical foundations, categorizing existing methods, and proposing future research directions for robust deep learning with noisy labels.

Contribution

It provides a formal definition of LNRL, categorizes existing methods, discusses their strengths and weaknesses, and suggests new research avenues and datasets.

Findings

Categorized LNRL methods into three main directions.

Identified key components for robust LNRL.

Proposed future research directions including new datasets and noise types.

Abstract

Classical machine learning implicitly assumes that labels of the training data are sampled from a clean distribution, which can be too restrictive for real-world scenarios. However, statistical-learning-based methods may not train deep learning models robustly with these noisy labels. Therefore, it is urgent to design Label-Noise Representation Learning (LNRL) methods for robustly training deep models with noisy labels. To fully understand LNRL, we conduct a survey study. We first clarify a formal definition for LNRL from the perspective of machine learning. Then, via the lens of learning theory and empirical study, we figure out why noisy labels affect deep models' performance. Based on the theoretical guidance, we categorize different LNRL methods into three directions. Under this unified taxonomy, we provide a thorough discussion of the pros and cons of different categories. More importantly, we summarize the essential components of robust LNRL, which can spark new directions. Lastly, we propose possible research directions within LNRL, such as new datasets, instance-dependent LNRL, and adversarial LNRL. We also envision potential directions beyond LNRL, such as learning with feature-noise, preference-noise, domain-noise, similarity-noise, graph-noise and demonstration-noise.