Importance Tempering: Group Robustness for Overparameterized Models

TL;DR

This paper introduces importance tempering, a novel technique to enhance group robustness in overparameterized models, effectively addressing distribution shifts and minority collapse, with theoretical justification and state-of-the-art empirical results.

Contribution

It proposes importance tempering as a new method to improve robustness of overparameterized models against distribution shifts and class imbalance, with theoretical analysis and empirical validation.

Findings

Importance tempering improves decision boundary robustness.

Proper group temperature selection varies under different shift types.

Achieves state-of-the-art results on worst group classification.

Abstract

Although overparameterized models have shown their success on many machine learning tasks, the accuracy could drop on the testing distribution that is different from the training one. This accuracy drop still limits applying machine learning in the wild. At the same time, importance weighting, a traditional technique to handle distribution shifts, has been demonstrated to have less or even no effect on overparameterized models both empirically and theoretically. In this paper, we propose importance tempering to improve the decision boundary and achieve consistently better results for overparameterized models. Theoretically, we justify that the selection of group temperature can be different under label shift and spurious correlation setting. At the same time, we also prove that properly selected temperatures can extricate the minority collapse for imbalanced classification. Empirically, we achieve state-of-the-art results on worst group classification tasks using importance tempering.