On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernel

TL;DR

This paper links deep Gaussian processes with deep trigonometric networks, revealing their covariance structures, expressivity, and neural tangent kernel, and provides insights into their inference and feature learning capabilities.

Contribution

It establishes a connection between DGPs and deep trig networks using Bochner's theorem, enabling exact MAP estimation and analysis of neural tangent kernels.

Findings

Deep DGPs can be represented as deep trig networks with equivalent covariance functions.

Varying priors over network parameters corresponds to different kernels.

Finite-width deep networks deviate from the limiting kernel, affecting feature learning.

Abstract

Deep Gaussian Process (DGP) as a model prior in Bayesian learning intuitively exploits the expressive power in function composition. DGPs also offer diverse modeling capabilities, but inference is challenging because marginalization in latent function space is not tractable. With Bochner's theorem, DGP with squared exponential kernel can be viewed as a deep trigonometric network consisting of the random feature layers, sine and cosine activation units, and random weight layers. In the wide limit with a bottleneck, we show that the weight space view yields the same effective covariance functions which were obtained previously in function space. Also, varying the prior distributions over network parameters is equivalent to employing different kernels. As such, DGPs can be translated into the deep bottlenecked trig networks, with which the exact maximum a posteriori estimation can be obtained. Interestingly, the network representation enables the study of DGP's neural tangent kernel, which may also reveal the mean of the intractable predictive distribution. Statistically, unlike the shallow networks, deep networks of finite width have covariance deviating from the limiting kernel, and the inner and outer widths may play different roles in feature learning. Numerical simulations are present to support our findings.