Gaussian Processes and Limiting Linear Models

TL;DR

This paper explores the relationship between Gaussian processes and linear models, developing a prior that enhances modeling flexibility and efficiency, especially for data with partial linearity, demonstrated on synthetic and real datasets.

Contribution

It introduces a novel prior for Gaussian processes that explicitly incorporates linear models, enabling flexible, efficient nonstationary modeling beyond traditional approaches.

Findings

The proposed prior improves modeling of partially linear data.

Linear features can be extracted per dimension.

Enhanced nonstationary modeling with combined treed partition models.

Abstract

Gaussian processes retain the linear model either as a special case, or in the limit. We show how this relationship can be exploited when the data are at least partially linear. However from the perspective of the Bayesian posterior, the Gaussian processes which encode the linear model either have probability of nearly zero or are otherwise unattainable without the explicit construction of a prior with the limiting linear model in mind. We develop such a prior, and show that its practical benefits extend well beyond the computational and conceptual simplicity of the linear model. For example, linearity can be extracted on a per-dimension basis, or can be combined with treed partition models to yield a highly efficient nonstationary model. Our approach is demonstrated on synthetic and real datasets of varying linearity and dimensionality.