Re-evaluating Group Robustness via Adaptive Class-Specific Scaling

TL;DR

This paper introduces a class-specific and adaptive scaling method to improve group robustness in machine learning models, effectively balancing robust and average accuracies without additional training, and providing a new unified evaluation metric.

Contribution

It proposes a simple, adaptable scaling strategy that enhances existing debiasing algorithms and reveals that naive ERM can match advanced methods, offering new insights into robustness trade-offs.

Findings

Naive ERM with class-specific scaling outperforms some debiasing methods.

Adaptive scaling improves both robust and average accuracies.

A new metric quantifies the trade-off between accuracies.

Abstract

Group distributionally robust optimization, which aims to improve robust accuracies -- worst-group and unbiased accuracies -- is a prominent algorithm used to mitigate spurious correlations and address dataset bias. Although existing approaches have reported improvements in robust accuracies, these gains often come at the cost of average accuracy due to inherent trade-offs. To control this trade-off flexibly and efficiently, we propose a simple class-specific scaling strategy, directly applicable to existing debiasing algorithms with no additional training. We further develop an instance-wise adaptive scaling technique to alleviate this trade-off, even leading to improvements in both robust and average accuracies. Our approach reveals that a na\"ive ERM baseline matches or even outperforms the recent debiasing methods by simply adopting the class-specific scaling technique. Additionally, we introduce a novel unified metric that quantifies the trade-off between the two accuracies as a scalar value, allowing for a comprehensive evaluation of existing algorithms. By tackling the inherent trade-off and offering a performance landscape, our approach provides valuable insights into robust techniques beyond just robust accuracy. We validate the effectiveness of our framework through experiments across datasets in computer vision and natural language processing domains.