Robustness as an Emergent Property of Task Performance

TL;DR

This paper demonstrates that model robustness naturally emerges as models achieve high performance on tasks, driven mainly by task-specific competence rather than inherent robustness properties.

Contribution

It provides empirical evidence that robustness correlates strongly with task performance, challenging the view that robustness must be explicitly engineered.

Findings

Robustness increases with task performance across diverse datasets.

Task-specific competence is the main driver of robustness, not inherent model properties.

Robustness is more reliable on easier, well-saturated tasks.

Abstract

Robustness is often regarded as a critical future challenge for real-world applications, where stability is essential. However, as models often learn tasks in a similar order, we hypothesize that easier tasks will be easier regardless of how they are presented to the model. Indeed, in this paper, we show that as models approach high performance on a task, robustness is effectively achieved. Through an empirical analysis of multiple models across diverse datasets and configurations (e.g., paraphrases, different temperatures), we find a strong positive correlation. Moreover, we find that robustness is primarily driven by task-specific competence rather than inherent model-level properties, challenging current approaches that treat robustness as an independent capability. Thus, from a high-level perspective, we may expect that as new tasks saturate, model robustness on these tasks will emerge accordingly. For researchers, this implies that explicit efforts to measure and improve robustness may warrant reduced emphasis, as such robustness is likely to develop alongside performance gains. For practitioners, it acts as a sign that indeed the tasks that the literature deals with are unreliable, but on easier past tasks, the models are reliable and ready for real-world deployment.