Variational Deep Learning via Implicit Regularization

TL;DR

This paper introduces a novel regularization method for variational neural networks that leverages the implicit bias of gradient descent, leading to improved in- and out-of-distribution performance with minimal tuning.

Contribution

It provides a theoretical characterization of the implicit regularization effect in overparametrized models and empirically demonstrates its effectiveness without extra hyperparameter tuning.

Findings

Strong in-distribution performance

Robust out-of-distribution generalization

Minimal computational overhead

Abstract

Modern deep learning models generalize remarkably well in-distribution, despite being overparametrized and trained with little to no explicit regularization. Instead, current theory credits implicit regularization imposed by the choice of architecture, hyperparameters, and optimization procedure. However, deep neural networks can be surprisingly non-robust, resulting in overconfident predictions and poor out-of-distribution generalization. Bayesian deep learning addresses this via model averaging, but typically requires significant computational resources as well as carefully elicited priors to avoid overriding the benefits of implicit regularization. Instead, in this work, we propose to regularize variational neural networks solely by relying on the implicit bias of (stochastic) gradient descent. We theoretically characterize this inductive bias in overparametrized linear models as generalized variational inference and demonstrate the importance of the choice of parametrization. Empirically, our approach demonstrates strong in- and out-of-distribution performance without additional hyperparameter tuning and with minimal computational overhead.