An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

TL;DR

This paper empirically evaluates how simple regularization methods impact the robustness and uncertainty estimation of deep neural networks, revealing some as effective baseline approaches for these issues.

Contribution

It provides comprehensive empirical analysis demonstrating that certain regularization techniques can serve as strong baselines for robustness and uncertainty in DNNs.

Findings

Some regularization methods improve robustness to input corruptions.

Certain regularization techniques enhance uncertainty estimation.

Regularization methods are effective and inexpensive baselines.

Abstract

Despite apparent human-level performances of deep neural networks (DNN), they behave fundamentally differently from humans. They easily change predictions when small corruptions such as blur and noise are applied on the input (lack of robustness), and they often produce confident predictions on out-of-distribution samples (improper uncertainty measure). While a number of researches have aimed to address those issues, proposed solutions are typically expensive and complicated (e.g. Bayesian inference and adversarial training). Meanwhile, many simple and cheap regularization methods have been developed to enhance the generalization of classifiers. Such regularization methods have largely been overlooked as baselines for addressing the robustness and uncertainty issues, as they are not specifically designed for that. In this paper, we provide extensive empirical evaluations on the robustness and uncertainty estimates of image classifiers (CIFAR-100 and ImageNet) trained with state-of-the-art regularization methods. Furthermore, experimental results show that certain regularization methods can serve as strong baseline methods for robustness and uncertainty estimation of DNNs.