Model Selection for Gaussian Process Regression by Approximation Set Coding

TL;DR

This paper introduces an approximation set coding framework for selecting the best kernel in Gaussian process regression, offering a competitive alternative to traditional methods like marginal likelihood and cross-validation.

Contribution

The paper proposes a novel model selection criterion based on approximation set coding specifically for Gaussian process kernels, addressing the challenge of choosing the appropriate kernel structure.

Findings

Approximation set coding competes with maximum evidence and leave-one-out cross-validation.

The framework effectively ranks different kernels for Gaussian process regression.

Experimental results demonstrate promising performance of the proposed method.

Abstract

Gaussian processes are powerful, yet analytically tractable models for supervised learning. A Gaussian process is characterized by a mean function and a covariance function (kernel), which are determined by a model selection criterion. The functions to be compared do not just differ in their parametrization but in their fundamental structure. It is often not clear which function structure to choose, for instance to decide between a squared exponential and a rational quadratic kernel. Based on the principle of approximation set coding, we develop a framework for model selection to rank kernels for Gaussian process regression. In our experiments approximation set coding shows promise to become a model selection criterion competitive with maximum evidence (also called marginal likelihood) and leave-one-out cross-validation.