Deep Q-Exponential Processes

TL;DR

This paper introduces deep Q-Exponential Processes, a novel hierarchical model that enhances expressiveness and regularization over traditional Gaussian processes, with scalable inference and improved performance on complex data.

Contribution

It generalizes Q-Exponential Processes to deep architectures, combining proper regularization with increased modeling flexibility through a hierarchical latent variable approach.

Findings

Demonstrates numerical advantages over state-of-the-art deep probabilistic models.

Provides scalable variational inference for deep Q-EP.

Shows improved expressiveness and regularization properties.

Abstract

Motivated by deep neural networks, the deep Gaussian process (DGP) generalizes the standard GP by stacking multiple layers of GPs. Despite the enhanced expressiveness, GP, as an $L_2$ regularization prior, tends to be over-smooth and sub-optimal for inhomogeneous subjects, such as images with edges. Recently, Q-exponential process (Q-EP) has been proposed as an $L_q$ relaxation to GP and demonstrated with more desirable regularization properties through a parameter $q>0$ with $q=2$ corresponding to GP. Sharing the similar tractability of posterior and predictive distributions with GP, Q-EP can also be stacked to improve its modeling flexibility. In this paper, we generalize Q-EP to deep Q-EP to enjoy both proper regularization and improved expressiveness. The generalization is realized by introducing shallow Q-EP as a latent variable model and then building a hierarchy of the shallow Q-EP layers. Sparse approximation by inducing points and scalable variational strategy are applied to facilitate the inference. We demonstrate the numerical advantages of the proposed deep Q-EP model by comparing with multiple state-of-the-art deep probabilistic models.