Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift

TL;DR

This paper analyzes how natural distribution shifts, such as weather changes, affect CNN performance in object detection and classification, benchmarking uncertainty methods and providing insights for robustness in autonomous driving scenarios.

Contribution

It introduces a new dataset and provides a detailed analysis of CNN robustness and confidence estimation under various real-world distribution shifts.

Findings

ConvNeXt-Tiny is more robust than EfficientNet-B0.

Heavy rain affects classification more than localization.

Selective MC-Dropout layers can improve performance and confidence estimation.

Abstract

Natural distribution shift causes a deterioration in the perception performance of convolutional neural networks (CNNs). This comprehensive analysis for real-world traffic data addresses: 1) investigating the effect of natural distribution shift and weather augmentations on both detection quality and confidence estimation, 2) evaluating model performance for both classification and object localization, and 3) benchmarking two common uncertainty quantification methods - Ensembles and different variants of Monte-Carlo (MC) Dropout - under natural and close-to-natural distribution shift. For this purpose, a novel dataset has been curated from publicly available autonomous driving datasets. The in-distribution (ID) data is based on cutouts of a single object, for which both class and bounding box annotations are available. The six distribution-shift datasets cover adverse weather scenarios, simulated rain and fog, corner cases, and out-of-distribution data. A granular analysis of CNNs under distribution shift allows to quantize the impact of different types of shifts on both, task performance and confidence estimation: ConvNeXt-Tiny is more robust than EfficientNet-B0; heavy rain degrades classification stronger than localization, contrary to heavy fog; integrating MC-Dropout into selected layers only has the potential to enhance task performance and confidence estimation, whereby the identification of these layers depends on the type of distribution shift and the considered task.