Meta-Learning Mean Functions for Gaussian Processes

TL;DR

This paper investigates meta-learning the mean function in Gaussian process models to improve prior knowledge encoding, demonstrating its potential benefits and risks compared to kernel-focused approaches.

Contribution

It introduces the concept of meta-learning the mean function in Gaussian processes, an area less explored compared to kernel learning, and analyzes its advantages and challenges.

Findings

Meta-learning the mean function can enhance Gaussian process priors.

Overfitting risks are associated with mean function meta-learning.

Connections to model-agnostic meta-learning and functional PCA are established.

Abstract

When fitting Bayesian machine learning models on scarce data, the main challenge is to obtain suitable prior knowledge and encode it into the model. Recent advances in meta-learning offer powerful methods for extracting such prior knowledge from data acquired in related tasks. When it comes to meta-learning in Gaussian process models, approaches in this setting have mostly focused on learning the kernel function of the prior, but not on learning its mean function. In this work, we explore meta-learning the mean function of a Gaussian process prior. We present analytical and empirical evidence that mean function learning can be useful in the meta-learning setting, discuss the risk of overfitting, and draw connections to other meta-learning approaches, such as model agnostic meta-learning and functional PCA.