From Snow to Rain: Evaluating Robustness, Calibration, and Complexity of Model-Based Robust Training

TL;DR

This paper evaluates model-based training methods that use learned nuisance models to improve robustness and calibration of deep learning models against natural corruptions like snow and rain, demonstrating their effectiveness and trade-offs.

Contribution

It introduces hybrid model-based training strategies combining random and adversarial nuisance sampling, and provides comprehensive evaluation on robustness, calibration, and complexity.

Findings

Model-based adversarial training offers the strongest robustness.

Data augmentation with learned nuisance models achieves similar robustness with less computation.

Model-based methods outperform traditional baselines across corruption severities.

Abstract

Robustness to natural corruptions remains a critical challenge for reliable deep learning, particularly in safety-sensitive domains. We study a family of model-based training approaches that leverage a learned nuisance variation model to generate realistic corruptions, as well as new hybrid strategies that combine random coverage with adversarial refinement in nuisance space. Using the Challenging Unreal and Real Environments for Traffic Sign Recognition dataset (CURE-TSR), with Snow and Rain corruptions, we evaluate accuracy, calibration, and training complexity across corruption severities. Our results show that model-based methods consistently outperform baselines Vanilla, Adversarial Training, and AugMix baselines, with model-based adversarial training providing the strongest robustness under across all corruptions but at the expense of higher computation and model-based data augmentation achieving comparable robustness with $T$ less computational complexity without incurring a statistically significant drop in performance. These findings highlight the importance of learned nuisance models for capturing natural variability, and suggest a promising path toward more resilient and calibrated models under challenging conditions.