When narrower is better: the narrow width limit of Bayesian parallel branching neural networks

TL;DR

This paper reveals that in Bayesian parallel branching neural networks, narrower widths can lead to better or comparable performance to wider networks, especially in bias-limited scenarios, due to symmetry breaking and robust branch learning.

Contribution

It introduces a novel narrow width regime for Bayesian branching networks, showing improved performance and data-reflective readout norms in this regime.

Findings

Narrower BPB-NNs exhibit more robust learning due to symmetry breaking.

Performance in narrow width limit can surpass or match wide networks in bias-limited cases.

The phenomenon extends to various architectures like residual-MLP and graph neural networks.

Abstract

The infinite width limit of random neural networks is known to result in Neural Networks as Gaussian Process (NNGP) (Lee et al. (2018)), characterized by task-independent kernels. It is widely accepted that larger network widths contribute to improved generalization (Park et al. (2019)). However, this work challenges this notion by investigating the narrow width limit of the Bayesian Parallel Branching Neural Network (BPB-NN), an architecture that resembles neural networks with residual blocks. We demonstrate that when the width of a BPB-NN is significantly smaller compared to the number of training examples, each branch exhibits more robust learning due to a symmetry breaking of branches in kernel renormalization. Surprisingly, the performance of a BPB-NN in the narrow width limit is generally superior to or comparable to that achieved in the wide width limit in bias-limited scenarios. Furthermore, the readout norms of each branch in the narrow width limit are mostly independent of the architectural hyperparameters but generally reflective of the nature of the data. We demonstrate such phenomenon primarily in the branching graph neural networks, where each branch represents a different order of convolutions of the graph; we also extend the results to other more general architectures such as the residual-MLP and demonstrate that the narrow width effect is a general feature of the branching networks. Our results characterize a newly defined narrow-width regime for parallel branching networks in general.