Sample Weight Averaging for Stable Prediction

TL;DR

This paper introduces SAmple Weight Averaging (SAWA), a method that reduces variance in sample reweighting algorithms to improve stable prediction under covariate shift, supported by theoretical proof and empirical results.

Contribution

SAWA is a simple, universal strategy that enhances existing reweighting methods by decreasing variance and estimation error, leading to better covariate-shift generalization.

Findings

SAWA improves stability and accuracy in covariate shift scenarios.

Theoretical analysis confirms the variance reduction benefits of SAWA.

Empirical results show SAWA outperforms baseline methods on synthetic and real datasets.

Abstract

The challenge of Out-of-Distribution (OOD) generalization poses a foundational concern for the application of machine learning algorithms to risk-sensitive areas. Inspired by traditional importance weighting and propensity weighting methods, prior approaches employ an independence-based sample reweighting procedure. They aim at decorrelating covariates to counteract the bias introduced by spurious correlations between unstable variables and the outcome, thus enhancing generalization and fulfilling stable prediction under covariate shift. Nonetheless, these methods are prone to experiencing an inflation of variance, primarily attributable to the reduced efficacy in utilizing training samples during the reweighting process. Existing remedies necessitate either environmental labels or substantially higher time costs along with additional assumptions and supervised information. To mitigate this issue, we propose SAmple Weight Averaging (SAWA), a simple yet efficacious strategy that can be universally integrated into various sample reweighting algorithms to decrease the variance and coefficient estimation error, thus boosting the covariate-shift generalization and achieving stable prediction across different environments. We prove its rationality and benefits theoretically. Experiments across synthetic datasets and real-world datasets consistently underscore its superiority against covariate shift.