Inference at the data's edge: Gaussian processes for modeling and inference under model-dependency, poor overlap, and extrapolation

TL;DR

This paper advocates using Gaussian Processes for modeling and inference in scenarios requiring extrapolation, emphasizing their ability to quantify uncertainty and handle model-dependency and poor data overlap.

Contribution

It introduces an accessible approach to Gaussian Processes for inference, including an automated hyperparameter selection method and practical applications in treatment effect and regression discontinuity analysis.

Findings

GPs provide wider, more realistic uncertainty intervals during extrapolation.

The R package gpss facilitates practical implementation of GPs for causal inference.

GPs effectively capture counterfactual uncertainty in various applied settings.

Abstract

Many inferential tasks involve fitting models to observed data and predicting outcomes at new covariate values, requiring interpolation or extrapolation. Conventional methods select a single best-fitting model, discarding fits that were similarly plausible in-sample but would yield sharply different predictions out-of-sample. Gaussian Processes (GPs) offer a principled alternative. Rather than committing to one conditional expectation function, GPs deliver a posterior distribution over outcomes at any covariate value. This posterior effectively retains the range of models consistent with the data, widening uncertainty intervals where extrapolation magnifies divergence. In this way, the GP's uncertainty estimates reflect the implications of extrapolation on our predictions, helping to tame the "dangers of extreme counterfactuals" (King & Zeng, 2006). The approach requires (i) specifying a covariance function linking outcome similarity to covariate similarity, and (ii) assuming Gaussian noise around the conditional expectation. We provide an accessible introduction to GPs with emphasis on this property, along with a simple, automated procedure for hyperparameter selection implemented in the R package gpss. We illustrate the value of GPs for capturing counterfactual uncertainty in three settings: (i) treatment effect estimation with poor overlap, (ii) interrupted time series requiring extrapolation beyond pre-intervention data, and (iii) regression discontinuity designs where estimates hinge on boundary behavior.