Fine-Grained ImageNet Classification in the Wild

TL;DR

This paper investigates the robustness of fine-grained ImageNet classifiers in real-world, uncurated data settings, using hierarchical knowledge to evaluate and explain model performance and errors.

Contribution

It introduces a novel evaluation framework for assessing classifier robustness on uncurated data using hierarchical knowledge for analysis.

Findings

Models show varying robustness under real-world data conditions.

Hierarchical knowledge helps identify and explain misclassifications.

Transformer-based architectures perform differently than convolutional ones in this setting.

Abstract

Image classification has been one of the most popular tasks in Deep Learning, seeing an abundance of impressive implementations each year. However, there is a lot of criticism tied to promoting complex architectures that continuously push performance metrics higher and higher. Robustness tests can uncover several vulnerabilities and biases which go unnoticed during the typical model evaluation stage. So far, model robustness under distribution shifts has mainly been examined within carefully curated datasets. Nevertheless, such approaches do not test the real response of classifiers in the wild, e.g. when uncurated web-crawled image data of corresponding classes are provided. In our work, we perform fine-grained classification on closely related categories, which are identified with the help of hierarchical knowledge. Extensive experimentation on a variety of convolutional and transformer-based architectures reveals model robustness in this novel setting. Finally, hierarchical knowledge is again employed to evaluate and explain misclassifications, providing an information-rich evaluation scheme adaptable to any classifier.