Dependence between Bayesian neural network units

TL;DR

This paper investigates the dependence structure of hidden units in finite-width Bayesian neural networks, combining theoretical insights with empirical analysis of how depth and width influence these dependencies.

Contribution

It provides new theoretical understanding and empirical evaluation of hidden unit dependence in practical Bayesian neural networks of finite size.

Findings

Hidden units exhibit dependence in finite-width networks.

Depth and width significantly affect hidden units dependence.

Theoretical results align with empirical observations.

Abstract

The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years, with the flagship result that hidden units converge to a Gaussian process limit when the layers width tends to infinity. Underpinning this result is the fact that hidden units become independent in the infinite-width limit. Our aim is to shed some light on hidden units dependence properties in practical finite-width Bayesian neural networks. In addition to theoretical results, we assess empirically the depth and width impacts on hidden units dependence properties.