Towards Understanding Extrapolation: a Causal Lens

TL;DR

This paper provides a theoretical framework for understanding and achieving extrapolation under distribution shifts using causal models, even with minimal target data outside the training support.

Contribution

It introduces a causal latent-variable model for extrapolation, offering conditions for identification with limited off-support target samples, and guides practical algorithm design.

Findings

Identification is possible with a single off-support sample under certain conditions.

Theoretical insights inform the design of practical extrapolation algorithms.

Experimental validation on synthetic and real data supports the theory.

Abstract

Canonical work handling distribution shifts typically necessitates an entire target distribution that lands inside the training distribution. However, practical scenarios often involve only a handful of target samples, potentially lying outside the training support, which requires the capability of extrapolation. In this work, we aim to provide a theoretical understanding of when extrapolation is possible and offer principled methods to achieve it without requiring an on-support target distribution. To this end, we formulate the extrapolation problem with a latent-variable model that embodies the minimal change principle in causal mechanisms. Under this formulation, we cast the extrapolation problem into a latent-variable identification problem. We provide realistic conditions on shift properties and the estimation objectives that lead to identification even when only one off-support target sample is available, tackling the most challenging scenarios. Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties. We showcase how our theoretical results inform the design of practical adaptation algorithms. Through experiments on both synthetic and real-world data, we validate our theoretical findings and their practical implications.