Are Sample-Efficient NLP Models More Robust?

TL;DR

This study investigates whether sample-efficient NLP models are inherently more robust out-of-distribution, finding that the relationship varies across tasks and datasets, and that universal robustness improvements are unlikely.

Contribution

It provides a comprehensive empirical analysis across multiple NLP tasks and interventions, revealing the complex and dataset-dependent relationship between sample efficiency and robustness.

Findings

Sample efficiency correlates with robustness only in some cases.

Lower sample efficiency models can sometimes be more robust.

Universal methods for improving robustness via sample efficiency are unlikely.

Abstract

Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution performance. However, it is unclear how broadly these trends hold. We conduct a large empirical study across three tasks, three broadly-applicable modeling interventions (increasing model size, using a different adaptation method, and pre-training on more data), and 14 diverse datasets to investigate the relationship between sample efficiency (amount of data needed to reach a given ID accuracy) and robustness (how models fare on OOD evaluation). We find that higher sample efficiency is only correlated with better average OOD robustness on some modeling interventions and tasks, but not others. On individual datasets, models with lower sample efficiency can even be more robust. These results suggest that general-purpose methods for improving sample efficiency are unlikely to yield universal OOD robustness improvements, since such improvements are highly dataset- and task-dependent. Even in an era of large, multi-purpose pretrained models, task-specific decisions may often be necessary for OOD generalization.