FOCUS: Familiar Objects in Common and Uncommon Settings

TL;DR

FOCUS is a dataset designed to evaluate how well deep image classifiers generalize to uncommon settings, revealing biases and limitations in current models through detailed analysis.

Contribution

The paper introduces FOCUS, a new dataset for stress-testing model generalization to uncommon settings, and analyzes model performance and feature usage in these scenarios.

Findings

Models perform significantly worse on uncommon settings.

Deep features often rely on spurious correlations.

Dataset reveals biases in current image classifiers.

Abstract

Standard training datasets for deep learning often contain objects in common settings (e.g., "a horse on grass" or "a ship in water") since they are usually collected by randomly scraping the web. Uncommon and rare settings (e.g., "a plane on water", "a car in snowy weather") are thus severely under-represented in the training data. This can lead to an undesirable bias in model predictions towards common settings and create a false sense of accuracy. In this paper, we introduce FOCUS (Familiar Objects in Common and Uncommon Settings), a dataset for stress-testing the generalization power of deep image classifiers. By leveraging the power of modern search engines, we deliberately gather data containing objects in common and uncommon settings in a wide range of locations, weather conditions, and time of day. We present a detailed analysis of the performance of various popular image classifiers on our dataset and demonstrate a clear drop in performance when classifying images in uncommon settings. By analyzing deep features of these models, we show that such errors can be due to the use of spurious features in model predictions. We believe that our dataset will aid researchers in understanding the inability of deep models to generalize well to uncommon settings and drive future work on improving their distributional robustness.