Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty

TL;DR

This paper investigates the role of explicit regularization in neural networks for achieving well-calibrated predictive uncertainty, proposing a probabilistic calibration measure and exploring regularization techniques to enhance log-likelihood on unseen data.

Contribution

It introduces a new probabilistic calibration measure and demonstrates how explicit regularization can improve predictive uncertainty in neural networks, offering an efficient alternative to Bayesian methods.

Findings

Explicit regularization improves log-likelihood on unseen samples.

Proposed methods enhance calibration of neural network predictions.

Offers a scalable alternative to Bayesian neural networks.

Abstract

From the statistical learning perspective, complexity control via explicit regularization is a necessity for improving the generalization of over-parameterized models. However, the impressive generalization performance of neural networks with only implicit regularization may be at odds with this conventional wisdom. In this work, we revisit the importance of explicit regularization for obtaining well-calibrated predictive uncertainty. Specifically, we introduce a probabilistic measure of calibration performance, which is lower bounded by the log-likelihood. We then explore explicit regularization techniques for improving the log-likelihood on unseen samples, which provides well-calibrated predictive uncertainty. Our findings present a new direction to improve the predictive probability quality of deterministic neural networks, which can be an efficient and scalable alternative to Bayesian neural networks and ensemble methods.