Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes

TL;DR

This paper investigates how hyperparameter tuning via marginal likelihood affects uncertainty estimation in Gaussian processes, revealing monotonic improvements with input dimension and contrasting behaviors with cross-validation metrics that show double descent.

Contribution

It provides a theoretical analysis demonstrating monotonicity in marginal likelihood performance and contrasting it with double descent phenomena in cross-validation metrics, including effects of cold posteriors.

Findings

Marginal likelihood improves monotonically with input dimension when hyperparameters are tuned.

Cross-validation metrics exhibit double descent behavior as input dimension increases.

Cold posteriors intensify the observed phenomena.

Abstract

Despite their importance for assessing reliability of predictions, uncertainty quantification (UQ) measures for machine learning models have only recently begun to be rigorously characterized. One prominent issue is the curse of dimensionality: it is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyperparameters to maximize marginal likelihood (the empirical Bayes procedure), the performance, as measured by the marginal likelihood, improves monotonically} with the input dimension. On the other hand, we prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent. Cold posteriors, which have recently attracted interest due to their improved performance in certain settings, appear to exacerbate these phenomena. We verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.