Boosting Test Performance with Importance Sampling--a Subpopulation Perspective

TL;DR

This paper introduces importance sampling as an effective approach to address subpopulation issues in machine learning, providing a unified theoretical framework and demonstrating state-of-the-art empirical results on benchmarks.

Contribution

It offers a new systematic formulation of the subpopulation problem, clarifies connections among existing methods, and introduces a flexible estimator applicable in attribute-known and unknown scenarios.

Findings

Achieves state-of-the-art results on benchmark datasets.

Provides a theoretical connection between existing subpopulation methods.

Demonstrates the effectiveness of importance sampling in practical settings.

Abstract

Despite empirical risk minimization (ERM) is widely applied in the machine learning community, its performance is limited on data with spurious correlation or subpopulation that is introduced by hidden attributes. Existing literature proposed techniques to maximize group-balanced or worst-group accuracy when such correlation presents, yet, at the cost of lower average accuracy. In addition, many existing works conduct surveys on different subpopulation methods without revealing the inherent connection between these methods, which could hinder the technology advancement in this area. In this paper, we identify important sampling as a simple yet powerful tool for solving the subpopulation problem. On the theory side, we provide a new systematic formulation of the subpopulation problem and explicitly identify the assumptions that are not clearly stated in the existing works. This helps to uncover the cause of the dropped average accuracy. We provide the first theoretical discussion on the connections of existing methods, revealing the core components that make them different. On the application side, we demonstrate a single estimator is enough to solve the subpopulation problem. In particular, we introduce the estimator in both attribute-known and -unknown scenarios in the subpopulation setup, offering flexibility in practical use cases. And empirically, we achieve state-of-the-art performance on commonly used benchmark datasets.