How Deep Are Deep Gaussian Processes?

TL;DR

This paper explores the structure, ergodicity, and inference methods of deep Gaussian processes, revealing how their depth affects their properties and proposing robust sampling techniques for hierarchical models.

Contribution

It introduces a unified framework for deep Gaussian processes, analyzes their ergodicity, and develops inference methods using Metropolis-within-Gibbs sampling.

Findings

Samples exhibit Markovian structure with respect to depth

Ergodicity determines the effective depth of the process

Proposed sampling methods are robust to resolution levels

Abstract

Recent research has shown the potential utility of Deep Gaussian Processes. These deep structures are probability distributions, designed through hierarchical construction, which are conditionally Gaussian. In this paper, the current published body of work is placed in a common framework and, through recursion, several classes of deep Gaussian processes are defined. The resulting samples generated from a deep Gaussian process have a Markovian structure with respect to the depth parameter, and the effective depth of the resulting process is interpreted in terms of the ergodicity, or non-ergodicity, of the resulting Markov chain. For the classes of deep Gaussian processes introduced, we provide results concerning their ergodicity and hence their effective depth. We also demonstrate how these processes may be used for inference; in particular we show how a Metropolis-within-Gibbs construction across the levels of the hierarchy can be used to derive sampling tools which are robust to the level of resolution used to represent the functions on a computer. For illustration, we consider the effect of ergodicity in some simple numerical examples.