Progression: an extrapolation principle for regression

TL;DR

This paper introduces a new extrapolation principle for regression based on tail dependence, enabling better out-of-distribution predictions by transforming data and assuming a simple boundary relationship.

Contribution

It proposes a semi-parametric method called progression that leverages tail dependence theory for improved regression extrapolation beyond training data.

Findings

Effective integration with random forests and additive models.

Guarantees on approximation error beyond training data.

Improved out-of-distribution extrapolation performance.

Abstract

The problem of regression extrapolation, or out-of-distribution generalization, arises when predictions are required at test points outside the range of the training data. In such cases, the non-parametric guarantees for regression methods from both statistics and machine learning typically fail. Based on the theory of tail dependence, we propose a novel statistical extrapolation principle. After a suitable, data-adaptive marginal transformation, it assumes a simple relationship between predictors and the response at the boundary of the training predictor samples. This assumption holds for a wide range of models, including non-parametric regression functions with additive noise. Our semi-parametric method, progression, leverages this extrapolation principle and offers guarantees on the approximation error beyond the training data range. We demonstrate how this principle can be effectively integrated with existing approaches, such as random forests and additive models, to improve extrapolation performance on out-of-distribution samples.