Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels

TL;DR

This paper re-labels the ImageNet training set with multi-label annotations using a strong classifier, improving model accuracy and robustness, and facilitating better transfer learning for related tasks.

Contribution

It introduces a scalable method to generate multi-label annotations for ImageNet training data, addressing label noise and mismatch issues in single-label training.

Findings

ResNet-50 achieves 78.9% top-1 accuracy with localized multi-labels

Accuracy improves to 80.2% with CutMix regularization

Models trained on re-labeled data outperform baselines on transfer tasks

Abstract

ImageNet has been arguably the most popular image classification benchmark, but it is also the one with a significant level of label noise. Recent studies have shown that many samples contain multiple classes, despite being assumed to be a single-label benchmark. They have thus proposed to turn ImageNet evaluation into a multi-label task, with exhaustive multi-label annotations per image. However, they have not fixed the training set, presumably because of a formidable annotation cost. We argue that the mismatch between single-label annotations and effectively multi-label images is equally, if not more, problematic in the training setup, where random crops are applied. With the single-label annotations, a random crop of an image may contain an entirely different object from the ground truth, introducing noisy or even incorrect supervision during training. We thus re-label the ImageNet training set with multi-labels. We address the annotation cost barrier by letting a strong image classifier, trained on an extra source of data, generate the multi-labels. We utilize the pixel-wise multi-label predictions before the final pooling layer, in order to exploit the additional location-specific supervision signals. Training on the re-labeled samples results in improved model performances across the board. ResNet-50 attains the top-1 classification accuracy of 78.9% on ImageNet with our localized multi-labels, which can be further boosted to 80.2% with the CutMix regularization. We show that the models trained with localized multi-labels also outperforms the baselines on transfer learning to object detection and instance segmentation tasks, and various robustness benchmarks. The re-labeled ImageNet training set, pre-trained weights, and the source code are available at {https://github.com/naver-ai/relabel_imagenet}.