On the expected behaviour of noise regularised deep neural networks as Gaussian processes

TL;DR

This paper explores how noise regularisation techniques like dropout affect deep neural networks when viewed as Gaussian processes, revealing their influence on model behaviour and prior assumptions.

Contribution

It establishes the relationship between noise regularisation in neural networks and their Gaussian process equivalents, highlighting optimal initialisation and prior effects.

Findings

Best performing NNGPs align with recent noise regularised ReLU initialisation schemes

Noise influences the covariance matrix, favoring simpler functions away from training data

Experimental validation on MNIST, CIFAR-10, and synthetic data supports theoretical insights

Abstract

Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation (e.g. dropout) on NNGPs, and relate their behaviour to signal propagation theory in noise regularised deep neural networks. For ReLU activations, we find that the best performing NNGPs have kernel parameters that correspond to a recently proposed initialisation scheme for noise regularised ReLU networks. In addition, we show how the noise influences the covariance matrix of the NNGP, producing a stronger prior towards simple functions away from the training points. We verify our theoretical findings with experiments on MNIST and CIFAR-10 as well as on synthetic data.