The Effect of Model Size on Worst-Group Generalization

TL;DR

This study systematically examines how increasing model size affects worst-group generalization in ERM, finding that larger pre-trained models improve performance even without subgroup labels across vision and NLP tasks.

Contribution

It provides the first comprehensive analysis of model size effects on worst-group generalization across multiple architectures, domains, and initialization methods.

Findings

Larger models do not harm and may improve worst-group test accuracy.

Pre-trained larger models consistently outperform smaller ones on Waterbirds and MultiNLI.

Increasing model size is recommended when subgroup labels are unavailable.

Abstract

Overparameterization is shown to result in poor test accuracy on rare subgroups under a variety of settings where subgroup information is known. To gain a more complete picture, we consider the case where subgroup information is unknown. We investigate the effect of model size on worst-group generalization under empirical risk minimization (ERM) across a wide range of settings, varying: 1) architectures (ResNet, VGG, or BERT), 2) domains (vision or natural language processing), 3) model size (width or depth), and 4) initialization (with pre-trained or random weights). Our systematic evaluation reveals that increasing model size does not hurt, and may help, worst-group test performance under ERM across all setups. In particular, increasing pre-trained model size consistently improves performance on Waterbirds and MultiNLI. We advise practitioners to use larger pre-trained models when subgroup labels are unknown.