Sparse Variational Student-t Processes for Heavy-tailed Modeling

TL;DR

This paper introduces Sparse Variational Student-t Processes (SVTP), a scalable framework for robust heavy-tailed data modeling that outperforms sparse Gaussian processes in accuracy and efficiency.

Contribution

The paper develops the first scalable sparse Student-t process framework with novel inference algorithms and a natural gradient optimization exploiting a new Fisher information connection.

Findings

SVTP outperforms sparse GPs on heavy-tailed datasets.

Achieves up to 3x faster convergence and 40% lower prediction error.

Maintains efficiency on datasets with over 200,000 samples.

Abstract

The Gaussian process (GP) is a powerful tool for nonparametric modeling, but its sensitivity to outliers limits its applicability to data distributions with heavy-tails. Studentt processes offer a robust alternative for heavy tail modeling, but they lack the scalable developments of the GP to large datasets necessary for practical applications. We present Sparse Variational Student-t Processes (SVTP), the first principled framework that extends the sparse inducing point method to the Student-t process. We develop two novel inference algorithms, SVTP-UB and SVTP-MC, with theoretical guarantees, and derive a natural gradient optimization that exploits a previously unused connection between the Fisher information matrix of the multivariate Student-t distribution and the beta function (the 'beta link'). Experiments on UCI and Kaggle datasets demonstrate that SVTP significantly outperforms sparse GPs on when the data is contains outliers and heavy tails, achieving up to 3 times faster convergence and 40% lower prediction error while maintaining computational efficiency for datasets with over 200,000 samples.