Benign Overfitting in Out-of-Distribution Generalization of Linear Models

TL;DR

This paper investigates benign overfitting in out-of-distribution generalization for linear models, providing theoretical guarantees and identifying key covariance conditions that influence performance under covariate shift.

Contribution

It extends understanding of benign overfitting to OOD scenarios, offering non-asymptotic guarantees for ridge regression and analyzing the impact of covariance structures.

Findings

Benign overfitting occurs in OOD regimes under certain covariance conditions.

Standard ridge regression can achieve fast rates with specific covariance structures.

PCR outperforms ridge regression in some OOD settings by achieving faster convergence.

Abstract

Benign overfitting refers to the phenomenon where an over-parameterized model fits the training data perfectly, including noise in the data, but still generalizes well to the unseen test data. While prior work provides some theoretical understanding of this phenomenon under the in-distribution setup, modern machine learning often operates in a more challenging Out-of-Distribution (OOD) regime, where the target (test) distribution can be rather different from the source (training) distribution. In this work, we take an initial step towards understanding benign overfitting in the OOD regime by focusing on the basic setup of over-parameterized linear models under covariate shift. We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression, even under the OOD regime when the target covariance satisfies certain structural conditions. We identify several vital quantities relating to source and target covariance, which govern the performance of OOD generalization. Our result is sharp, which provably recovers prior in-distribution benign overfitting guarantee [Tsigler and Bartlett, 2023], as well as under-parameterized OOD guarantee [Ge et al., 2024] when specializing to each setup. Moreover, we also present theoretical results for a more general family of target covariance matrix, where standard ridge regression only achieves a slow statistical rate of $O(1/\sqrt{n})$ for the excess risk, while Principal Component Regression (PCR) is guaranteed to achieve the fast rate $O(1/n)$, where $n$ is the number of samples.