Measuring Uncertainty through Bayesian Learning of Deep Neural Network Structure

TL;DR

This paper introduces a Bayesian approach to neural network structure learning, combining structure inference with weight uncertainty, resulting in competitive predictive performance and efficient inference.

Contribution

It proposes a novel Bayesian inference method on network structure inspired by neural architecture search, unifying structure and weight learning efficiently.

Findings

Competitive predictive performance across challenging scenarios

Effective Bayesian uncertainty quantification in network structure

Empirical validation supports the proposed modeling approach

Abstract

Bayesian neural networks (BNNs) augment deep networks with uncertainty quantification by Bayesian treatment of the network weights. However, such models face the challenge of Bayesian inference in a high-dimensional and usually over-parameterized space. This paper investigates a new line of Bayesian deep learning by performing Bayesian inference on network structure. Instead of building structure from scratch inefficiently, we draw inspirations from neural architecture search to represent the network structure. We then develop an efficient stochastic variational inference approach which unifies the learning of both network structure and weights. Empirically, our method exhibits competitive predictive performance while preserving the benefits of Bayesian principles across challenging scenarios. We also provide convincing experimental justification for our modeling choice.