Understand the Effect of Importance Weighting in Deep Learning on Dataset Shift

TL;DR

This paper investigates how importance weighting influences deep learning models under dataset shifts, revealing limited benefits in complex scenarios and emphasizing the importance of regularization and data complexity.

Contribution

It provides an empirical evaluation of importance weighting effects in deep neural networks under different types of dataset shifts, highlighting its limited practical utility.

Findings

Weighting affects decision boundaries early in training.

L2 regularization helps preserve weighting effects.

Importance weighting shows no significant gain under covariate shift.

Abstract

We evaluate the effectiveness of importance weighting in deep neural networks under label shift and covariate shift. On synthetic 2D data (linearly separable and moon-shaped) using logistic regression and MLPs, we observe that weighting strongly affects decision boundaries early in training but fades with prolonged optimization. On CIFAR-10 with various class imbalances, only L2 regularization (not dropout) helps preserve weighting effects. In a covariate-shift experiment, importance weighting yields no significant performance gain, highlighting challenges on complex data. Our results call into question the practical utility of importance weighting for real-world distribution shifts.