Deep Neural Networks as Point Estimates for Deep Gaussian Processes

TL;DR

This paper establishes a theoretical link between neural networks and deep Gaussian processes, showing how each can enhance the other's capabilities in uncertainty estimation and prediction accuracy, supported by experimental validation.

Contribution

It introduces a novel equivalence between neural networks and deep sparse Gaussian processes through activation functions and kernels, enabling improved uncertainty and accuracy.

Findings

Neural networks can be interpreted as deep Gaussian processes with specific kernel choices.

The proposed models improve uncertainty estimation in neural networks.

Experimental results demonstrate enhanced performance on regression and classification tasks.

Abstract

Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.