Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

TL;DR

This paper introduces Denoising Diffusion Variational Inference (DDVI), a novel approach for more accurate posterior inference of inducing points in Deep Gaussian Processes, improving efficiency and reducing bias.

Contribution

The paper proposes DDVI, combining diffusion SDEs and score matching, with a new variational lower bound for better inducing point inference in DGPs.

Findings

DDVI outperforms baseline methods in experiments

Improved posterior approximation accuracy

Enhanced model efficiency and reduced bias

Abstract

Deep Gaussian processes (DGPs) provide a robust paradigm for Bayesian deep learning. In DGPs, a set of sparse integration locations called inducing points are selected to approximate the posterior distribution of the model. This is done to reduce computational complexity and improve model efficiency. However, inferring the posterior distribution of inducing points is not straightforward. Traditional variational inference approaches to posterior approximation often lead to significant bias. To address this issue, we propose an alternative method called Denoising Diffusion Variational Inference (DDVI) that uses a denoising diffusion stochastic differential equation (SDE) to generate posterior samples of inducing variables. We rely on score matching methods for denoising diffusion model to approximate score functions with a neural network. Furthermore, by combining classical mathematical theory of SDEs with the minimization of KL divergence between the approximate and true processes, we propose a novel explicit variational lower bound for the marginal likelihood function of DGP. Through experiments on various datasets and comparisons with baseline methods, we empirically demonstrate the effectiveness of DDVI for posterior inference of inducing points for DGP models.