Beyond Minimax Rates in Group Distributionally Robust Optimization via a Novel Notion of Sparsity

TL;DR

This paper introduces a new sparsity concept in group distributionally robust optimization, enabling algorithms to achieve better sample complexity by exploiting the structure of group risks, with practical validation on datasets.

Contribution

It proposes a novel $(\lambda,eta)$-sparsity notion and develops algorithms that improve sample complexity bounds by leveraging this structure in GDRO.

Findings

Improved sample complexity depending on $eta$ instead of $K$

Algorithms adapt to the best sparsity condition in data

Validated effectiveness on synthetic and real datasets

Abstract

The minimax sample complexity of group distributionally robust optimization (GDRO) has been determined up to a $\log(K)$ factor, where $K$ is the number of groups. In this work, we venture beyond the minimax perspective via a novel notion of sparsity that we dub $(\lambda, \beta)$-sparsity. In short, this condition means that at any parameter $\theta$, there is a set of at most $\beta$ groups whose risks at $\theta$ all are at least $\lambda$ larger than the risks of the other groups. To find an $\epsilon$-optimal $\theta$, we show via a novel algorithm and analysis that the $\epsilon$-dependent term in the sample complexity can swap a linear dependence on $K$ for a linear dependence on the potentially much smaller $\beta$. This improvement leverages recent progress in sleeping bandits, showing a fundamental connection between the two-player zero-sum game optimization framework for GDRO and per-action regret bounds in sleeping bandits. We next show an adaptive algorithm which, up to log factors, gets a sample complexity bound that adapts to the best $(\lambda, \beta)$-sparsity condition that holds. We also show how to get a dimension-free semi-adaptive sample complexity bound with a computationally efficient method. Finally, we demonstrate the practicality of the $(\lambda, \beta)$-sparsity condition and the improved sample efficiency of our algorithms on both synthetic and real-life datasets.