Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains

TL;DR

This paper introduces ImageNet-ES, a new dataset capturing real-world environmental and sensor variations, revealing limitations of current OOD detection methods and showing that sensor control can enhance model robustness in computer vision.

Contribution

The paper presents a novel dataset, ImageNet-ES, that captures real-world distribution shifts, and demonstrates the impact of environment and sensor variations on robustness and OOD detection.

Findings

Existing OOD detection methods struggle with real-world covariate shifts.

Learning environment and sensor variations improves robustness on ImageNet-C and -ES.

Sensor control can significantly enhance model performance without increasing size.

Abstract

Computer vision applications predict on digital images acquired by a camera from physical scenes through light. However, conventional robustness benchmarks rely on perturbations in digitized images, diverging from distribution shifts occurring in the image acquisition process. To bridge this gap, we introduce a new distribution shift dataset, ImageNet-ES, comprising variations in environmental and camera sensor factors by directly capturing 202k images with a real camera in a controllable testbed. With the new dataset, we evaluate out-of-distribution (OOD) detection and model robustness. We find that existing OOD detection methods do not cope with the covariate shifts in ImageNet-ES, implying that the definition and detection of OOD should be revisited to embrace real-world distribution shifts. We also observe that the model becomes more robust in both ImageNet-C and -ES by learning environment and sensor variations in addition to existing digital augmentations. Lastly, our results suggest that effective shift mitigation via camera sensor control can significantly improve performance without increasing model size. With these findings, our benchmark may aid future research on robustness, OOD, and camera sensor control for computer vision. Our code and dataset are available at https://github.com/Edw2n/ImageNet-ES.