Generation and Analysis of Feature-Dependent Pseudo Noise for Training Deep Neural Networks

TL;DR

This paper introduces a method to generate feature-dependent pseudo noisy datasets for training deep neural networks, enabling better study of noise effects and development of robust training techniques.

Contribution

The paper proposes a novel approach to create feature-dependent noisy datasets using DNN predictions, filling the gap left by synthetic random noise datasets.

Findings

Pseudo noisy datasets resemble real feature-dependent noise.

Learning is easier with feature-dependent noise than with random noise.

Current training methods are not robust to feature-dependent label noise.

Abstract

Training Deep neural networks (DNNs) on noisy labeled datasets is a challenging problem, because learning on mislabeled examples deteriorates the performance of the network. As the ground truth availability is limited with real-world noisy datasets, previous papers created synthetic noisy datasets by randomly modifying the labels of training examples of clean datasets. However, no final conclusions can be derived by just using this random noise, since it excludes feature-dependent noise. Thus, it is imperative to generate feature-dependent noisy datasets that additionally provide ground truth. Therefore, we propose an intuitive approach to creating feature-dependent noisy datasets by utilizing the training predictions of DNNs on clean datasets that also retain true label information. We refer to these datasets as "Pseudo Noisy datasets". We conduct several experiments to establish that Pseudo noisy datasets resemble feature-dependent noisy datasets across different conditions. We further randomly generate synthetic noisy datasets with the same noise distribution as that of Pseudo noise (referred as "Randomized Noise") to empirically show that i) learning is easier with feature-dependent label noise compared to random noise, ii) irrespective of noise distribution, Pseudo noisy datasets mimic feature-dependent label noise and iii) current training methods are not generalizable to feature-dependent label noise. Therefore, we believe that Pseudo noisy datasets will be quite helpful to study and develop robust training methods.