Empirical Gaussian Processes

TL;DR

Empirical Gaussian Processes introduce a data-driven approach to constructing flexible GP priors by empirically estimating mean and covariance functions, improving adaptivity and performance in regression tasks.

Contribution

The paper proposes a novel framework for data-driven GP priors using empirical estimation, with an EM algorithm for learning from multiple datasets, enhancing flexibility and practical applicability.

Findings

Achieves competitive results on learning curve extrapolation.

Demonstrates improved time series forecasting performance.

Provides theoretical convergence guarantees to data-generating processes.

Abstract

Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. We study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data generating process. Practically, we formulate the problem of learning the GP prior from independent datasets as likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.