SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks

TL;DR

This paper introduces SNAP, a new benchmark dataset that evaluates how capture conditions like lighting and camera settings affect deep learning models in vision tasks such as classification, detection, and VQA, revealing significant biases and sensitivities.

Contribution

The paper presents SNAP, a comprehensive benchmark dataset for analyzing capture condition effects on vision models, and provides insights into dataset biases and model vulnerabilities.

Findings

Models are biased by capture conditions.

Models do not reach human accuracy on well-exposed images.

Capture variations significantly affect model performance.

Abstract

Generalization of deep-learning-based (DL) computer vision algorithms to various image perturbations is hard to establish and remains an active area of research. The majority of past analyses focused on the images already captured, whereas effects of the image formation pipeline and environment are less studied. In this paper, we address this issue by analyzing the impact of capture conditions, such as camera parameters and lighting, on DL model performance on 3 vision tasks -- image classification, object detection, and visual question answering (VQA). To this end, we assess capture bias in common vision datasets and create a new benchmark, SNAP (for $\textbf{S}$hutter speed, ISO se$\textbf{N}$sitivity, and $\textbf{AP}$erture), consisting of images of objects taken under controlled lighting conditions and with densely sampled camera settings. We then evaluate a large number of DL vision models and show the effects of capture conditions on each selected vision task. Lastly, we conduct an experiment to establish a human baseline for the VQA task. Our results show that computer vision datasets are significantly biased, the models trained on this data do not reach human accuracy even on the well-exposed images, and are susceptible to both major exposure changes and minute variations of camera settings. Code and data can be found at https://github.com/ykotseruba/SNAP