Corner Cases: How Size and Position of Objects Challenge ImageNet-Trained Models

TL;DR

This paper investigates how object size and position biases in images influence the reliance of ImageNet-trained models on background features, revealing limitations of current mitigation methods and introducing a synthetic dataset to study these effects.

Contribution

It introduces Hard-Spurious-ImageNet, a synthetic dataset to analyze size and position biases, and demonstrates that existing methods fail to address these biases effectively.

Findings

Models rely on background features when objects are small and off-center.

Current mitigation methods do not account for size and position biases.

Biases affect worst-group accuracy in image classification.

Abstract

Backgrounds in images play a major role in contributing to spurious correlations among different data points. Owing to aesthetic preferences of humans capturing the images, datasets can exhibit positional (location of the object within a given frame) and size (region-of-interest to image ratio) biases for different classes. In this paper, we show that these biases can impact how much a model relies on spurious features in the background to make its predictions. To better illustrate our findings, we propose a synthetic dataset derived from ImageNet-1k, Hard-Spurious-ImageNet, which contains images with various backgrounds, object positions, and object sizes. By evaluating the dataset on different pretrained models, we find that most models rely heavily on spurious features in the background when the region-of-interest (ROI) to image ratio is small and the object is far from the center of the image. Moreover, we also show that current methods that aim to mitigate harmful spurious features, do not take into account these factors, hence fail to achieve considerable performance gains for worst-group accuracies when the size and location of core features in an image change. The dataset and implementation code are available at https://github.com/Mishalfatima/Corner_Cases.