SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates

TL;DR

SDE-Net introduces a novel approach to uncertainty quantification in deep neural networks by modeling their transformations as stochastic dynamical systems, effectively capturing epistemic uncertainty with theoretical guarantees.

Contribution

The paper proposes SDE-Net, a new neural stochastic differential equation model that captures epistemic uncertainty through a dynamical systems perspective, with theoretical analysis and improved performance.

Findings

Outperforms existing uncertainty estimation methods.

Provides theoretical guarantees on solution existence and uniqueness.

Effectively captures epistemic uncertainty in deep learning tasks.

Abstract

Uncertainty quantification is a fundamental yet unsolved problem for deep learning. The Bayesian framework provides a principled way of uncertainty estimation but is often not scalable to modern deep neural nets (DNNs) that have a large number of parameters. Non-Bayesian methods are simple to implement but often conflate different sources of uncertainties and require huge computing resources. We propose a new method for quantifying uncertainties of DNNs from a dynamical system perspective. The core of our method is to view DNN transformations as state evolution of a stochastic dynamical system and introduce a Brownian motion term for capturing epistemic uncertainty. Based on this perspective, we propose a neural stochastic differential equation model (SDE-Net) which consists of (1) a drift net that controls the system to fit the predictive function; and (2) a diffusion net that captures epistemic uncertainty. We theoretically analyze the existence and uniqueness of the solution to SDE-Net. Our experiments demonstrate that the SDE-Net model can outperform existing uncertainty estimation methods across a series of tasks where uncertainty plays a fundamental role.