Revisiting Penalized Likelihood Estimation for Gaussian Processes

TL;DR

This paper introduces a new cross-validation metric called decorrelated prediction error (DPE) for Gaussian process covariance parameter estimation, improving tuning parameter selection especially in small data or regularization scenarios.

Contribution

The paper proposes DPE, a novel CV metric inspired by Mahalanobis distance, enhancing penalized likelihood estimation for GPs over traditional methods.

Findings

DPE provides more reliable tuning parameter selection than traditional metrics.

DPE performs comparably to MLE when regularization isn't needed.

DPE outperforms traditional metrics in regularized scenarios, especially with the one-standard error rule.

Abstract

Gaussian processes (GPs) are popular as nonlinear regression models for expensive computer simulations, yet GP performance relies heavily on estimation of unknown covariance parameters. Maximum likelihood estimation (MLE) is common, but it can be plagued by numerical issues in small data settings. The addition of a nugget helps but is not a cure-all. Penalized likelihood methods may improve upon traditional MLE, but their success depends on tuning parameter selection. We introduce a new cross-validation (CV) metric called ``decorrelated prediction error'' (DPE), within the penalized likelihood framework for GPs. Inspired by the Mahalanobis distance, DPE provides more consistent and reliable tuning parameter selection than traditional metrics like prediction error, particularly for $K$-fold CV. Our proposed metric performs comparably to standard MLE when penalization is unnecessary and outperforms traditional tuning parameter selection metrics in scenarios where regularization is beneficial, especially under the one-standard error rule.