Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks

TL;DR

This paper explains how output-output correlations in finite Bayesian one-hidden-layer networks can be understood through kernel shape renormalization, bridging empirical observations with theoretical insights in the proportional limit of Bayesian deep learning.

Contribution

It introduces the concept of kernel shape renormalization to explain output correlations in finite networks, extending the understanding of Bayesian neural networks in the proportional limit.

Findings

Kernel shape renormalization accounts for output-output correlations.

Numerical experiments validate the predictive power of the kernel renormalization approach.

The approach explains phenomenology observed in finite Bayesian networks.

Abstract

Finite-width one hidden layer networks with multiple neurons in the readout layer display non-trivial output-output correlations that vanish in the lazy-training infinite-width limit. In this manuscript we leverage recent progress in the proportional limit of Bayesian deep learning (that is the limit where the size of the training set $P$ and the width of the hidden layers $N$ are taken to infinity keeping their ratio $\alpha = P/N$ finite) to rationalize this empirical evidence. In particular, we show that output-output correlations in finite fully-connected networks are taken into account by a kernel shape renormalization of the infinite-width NNGP kernel, which naturally arises in the proportional limit. We perform accurate numerical experiments both to assess the predictive power of the Bayesian framework in terms of generalization, and to quantify output-output correlations in finite-width networks. By quantitatively matching our predictions with the observed correlations, we provide additional evidence that kernel shape renormalization is instrumental to explain the phenomenology observed in finite Bayesian one hidden layer networks.