Flaws of ImageNet, Computer Vision's Favourite Dataset

TL;DR

This paper critically examines the ImageNet-1k dataset, highlighting issues like incorrect labels and domain shifts, and advocates for dataset refinement to improve future computer vision research.

Contribution

It provides a detailed analysis of ImageNet-1k's flaws and initiates a discussion on refining this widely used dataset for better research outcomes.

Findings

Identification of label correctness issues

Detection of image duplicates and ambiguities

Discussion on domain shifts affecting evaluation

Abstract

Since its release, ImageNet-1k dataset has become a gold standard for evaluating model performance. It has served as the foundation for numerous other datasets and training tasks in computer vision. As models have improved in accuracy, issues related to label correctness have become increasingly apparent. In this blog post, we analyze the issues in the ImageNet-1k dataset, including incorrect labels, overlapping or ambiguous class definitions, training-evaluation domain shifts, and image duplicates. The solutions for some problems are straightforward. For others, we hope to start a broader conversation about refining this influential dataset to better serve future research.