Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes

TL;DR

This paper introduces NOVI, a neural operator variational inference method for Deep Gaussian Processes that improves inference accuracy, convergence speed, and robustness by minimizing regularized Stein discrepancy.

Contribution

The paper proposes a novel NOVI approach that uses neural generators and Stein discrepancy minimization, offering a more expressive and computationally efficient inference method for DGPs.

Findings

Achieves 93.56% accuracy on CIFAR10, outperforming state-of-the-art Gaussian process methods.

Demonstrates faster convergence and robustness across datasets.

Provides theoretical guarantees on prediction error.

Abstract

Deep Gaussian Process (DGP) models offer a powerful nonparametric approach for Bayesian inference, but exact inference is typically intractable, motivating the use of various approximations. However, existing approaches, such as mean-field Gaussian assumptions, limit the expressiveness and efficacy of DGP models, while stochastic approximation can be computationally expensive. To tackle these challenges, we introduce Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a sampler and minimizes the Regularized Stein Discrepancy in L2 space between the generated distribution and true posterior. We solve the minimax problem using Monte Carlo estimation and subsampling stochastic optimization techniques. We demonstrate that the bias introduced by our method can be controlled by multiplying the Fisher divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm. Our experiments on datasets ranging from hundreds to tens of thousands demonstrate the effectiveness and the faster convergence rate of the proposed method. We achieve a classification accuracy of 93.56 on the CIFAR10 dataset, outperforming SOTA Gaussian process methods. Furthermore, our method guarantees theoretically controlled prediction error for DGP models and demonstrates remarkable performance on various datasets. We are optimistic that NOVI has the potential to enhance the performance of deep Bayesian nonparametric models and could have significant implications for various practical applications