Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

TL;DR

This paper proves convergence guarantees for minibatch SGD in Gaussian process hyperparameter estimation, showing it recovers parameters efficiently and offers empirical benefits over existing methods, especially in large-scale settings.

Contribution

It provides the first theoretical analysis of minibatch SGD convergence for GP hyperparameters under correlated data, with practical empirical validation.

Findings

Minibatch SGD converges to a critical point of the GP log-likelihood.

It recovers hyperparameters at a rate of O(1/K) for K iterations.

Empirical results show better generalization and reduced computational cost.

Abstract

Stochastic gradient descent (SGD) and its variants have established themselves as the go-to algorithms for large-scale machine learning problems with independent samples due to their generalization performance and intrinsic computational advantage. However, the fact that the stochastic gradient is a biased estimator of the full gradient with correlated samples has led to the lack of theoretical understanding of how SGD behaves under correlated settings and hindered its use in such cases. In this paper, we focus on hyperparameter estimation for the Gaussian process (GP) and take a step forward towards breaking the barrier by proving minibatch SGD converges to a critical point of the full log-likelihood loss function, and recovers model hyperparameters with rate $O(\frac{1}{K})$ for $K$ iterations, up to a statistical error term depending on the minibatch size. Our theoretical guarantees hold provided that the kernel functions exhibit exponential or polynomial eigendecay which is satisfied by a wide range of kernels commonly used in GPs. Numerical studies on both simulated and real datasets demonstrate that minibatch SGD has better generalization over state-of-the-art GP methods while reducing the computational burden and opening a new, previously unexplored, data size regime for GPs.