Addressing out-of-distribution label noise in webly-labelled data

TL;DR

This paper investigates label noise in webly-labeled datasets, analyzes existing methods, and proposes DSOS, a simple technique to improve robustness against out-of-distribution label noise in image classification.

Contribution

It introduces DSOS, a novel method to mitigate out-of-distribution label noise, validated on multiple datasets including real-world noisy datasets.

Findings

DSOS outperforms state-of-the-art methods on noisy datasets.

Web noise distribution significantly impacts model performance.

Proposed approach improves robustness to out-of-distribution label noise.

Abstract

A recurring focus of the deep learning community is towards reducing the labeling effort. Data gathering and annotation using a search engine is a simple alternative to generating a fully human-annotated and human-gathered dataset. Although web crawling is very time efficient, some of the retrieved images are unavoidably noisy, i.e. incorrectly labeled. Designing robust algorithms for training on noisy data gathered from the web is an important research perspective that would render the building of datasets easier. In this paper we conduct a study to understand the type of label noise to expect when building a dataset using a search engine. We review the current limitations of state-of-the-art methods for dealing with noisy labels for image classification tasks in the case of web noise distribution. We propose a simple solution to bridge the gap with a fully clean dataset using Dynamic Softening of Out-of-distribution Samples (DSOS), which we design on corrupted versions of the CIFAR-100 dataset, and compare against state-of-the-art algorithms on the web noise perturbated MiniImageNet and Stanford datasets and on real label noise datasets: WebVision 1.0 and Clothing1M. Our work is fully reproducible https://git.io/JKGcj