Minimax Regret Learning for Data with Heterogeneous Subgroups

TL;DR

This paper introduces a minimax regret learning framework for data with heterogeneous subgroups, aiming to improve robustness and generalization to unseen populations, especially under noise heterogeneity.

Contribution

It develops a distribution-free robust hierarchical model using minimax regret, with theoretical guarantees and specialization to linear and generalized linear models.

Findings

Empirical MMR guarantees regret bounds on training and unseen populations.

Specialized MMR framework reveals geometric properties distinct from other robust methods.

Demonstrated effectiveness through simulations and image recognition applications.

Abstract

Modern complex datasets often consist of various sub-populations with known group information. In the presence of sub-population heterogeneity, it is crucial to develop robust and generalizable learning methods that (1) can enjoy robust performance on each of the training populations, and (2) is generalizable to an unseen testing population. While various min-max formulations have been proposed to achieve (1) in the robust learning literature, their generalization to an unseen testing is less explored. Moreover, a general min-max formulation can be sensitive to the noise heterogeneity, and, in the extreme case, be degenerate such that a single high-noise population dominates. The min-max-regret (MMR) can mitigate these challenges. In this work, we consider a distribution-free robust hierarchical model for the generalization from multiple training populations to an unseen testing population. Under the robust hierarchical model, the empirical MMR can enjoy the regret guarantees on each of the training populations as well as the unseen testing population. We further specialize the general MMR framework to linear regression and generalized linear model, where we characterize the geometry of MMR and its distinction from other robust methods. We demonstrate the effectiveness of MMR through extensive simulation studies and an application to image recognition.