Using Pre-Training Can Improve Model Robustness and Uncertainty

TL;DR

Pre-training enhances model robustness and uncertainty estimation across various challenging scenarios, offering significant improvements even without task-specific adjustments, despite not always boosting traditional accuracy metrics.

Contribution

The paper demonstrates that pre-training notably improves robustness and uncertainty estimates, introduces adversarial pre-training, and highlights its importance beyond traditional performance metrics.

Findings

Pre-training yields large gains in adversarial robustness.

Pre-training improves uncertainty estimation and calibration.

Pre-training alone can surpass state-of-the-art in some robustness tasks.

Abstract

He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. Through extensive experiments on adversarial examples, label corruption, class imbalance, out-of-distribution detection, and confidence calibration, we demonstrate large gains from pre-training and complementary effects with task-specific methods. We introduce adversarial pre-training and show approximately a 10% absolute improvement over the previous state-of-the-art in adversarial robustness. In some cases, using pre-training without task-specific methods also surpasses the state-of-the-art, highlighting the need for pre-training when evaluating future methods on robustness and uncertainty tasks.