Group Distributionally Robust Optimization with Flexible Sample Queries

TL;DR

This paper introduces a flexible sample size approach to group distributionally robust optimization (GDRO), enabling dynamic sample queries and providing theoretical guarantees and empirical validation.

Contribution

It develops a novel GDRO algorithm that supports varying sample sizes per iteration with high-probability regret bounds, extending existing fixed-sample methods.

Findings

Achieves a high-probability optimization error bound of O(1/t * sqrt(sum_{j=1}^t (m/r_j) log m))

Demonstrates a sample complexity of O(m log(m)/ε^2) for fixed sample size r

Validates the approach on synthetic and real-world datasets.

Abstract

Group distributionally robust optimization (GDRO) aims to develop models that perform well across $m$ distributions simultaneously. Existing GDRO algorithms can only process a fixed number of samples per iteration, either 1 or $m$, and therefore can not support scenarios where the sample size varies dynamically. To address this limitation, we investigate GDRO with flexible sample queries and cast it as a two-player game: one player solves an online convex optimization problem, while the other tackles a prediction with limited advice (PLA) problem. Within such a game, we propose a novel PLA algorithm, constructing appropriate loss estimators for cases where the sample size is either 1 or not, and updating the decision using follow-the-regularized-leader. Then, we establish the first high-probability regret bound for non-oblivious PLA. Building upon the above approach, we develop a GDRO algorithm that allows an arbitrary and varying sample size per round, achieving a high-probability optimization error bound of $O\left(\frac{1}{t}\sqrt{\sum_{j=1}^t \frac{m}{r_j}\log m}\right)$, where $r_t$ denotes the sample size at round $t$. This result demonstrates that the optimization error decreases as the number of samples increases and implies a consistent sample complexity of $O(m\log (m)/\epsilon^2)$ for any fixed sample size $r\in[m]$, aligning with existing bounds for cases of $r=1$ or $m$. We validate our approach on synthetic binary and real-world multi-class datasets.