Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI

TL;DR

This paper critically examines the 'bigger-is-better' AI paradigm, challenging assumptions about scale-driven performance, highlighting sustainability issues, and discussing societal and ethical implications of increasing AI model sizes.

Contribution

It refutes the notions that larger models always outperform smaller ones and that all AI problems require large-scale models, emphasizing sustainability and societal concerns.

Findings

Performance gains are not solely due to increased scale

Scaling demands are unsustainable environmentally and economically

Large models concentrate power and limit diverse applications

Abstract

With the growing attention and investment in recent AI approaches such as large language models, the narrative that the larger the AI system the more valuable, powerful and interesting it is is increasingly seen as common sense. But what is this assumption based on, and how are we measuring value, power, and performance? And what are the collateral consequences of this race to ever-increasing scale? Here, we scrutinize the current scaling trends and trade-offs across multiple axes and refute two common assumptions underlying the 'bigger-is-better' AI paradigm: 1) that performance improvements are driven by increased scale, and 2) that all interesting problems addressed by AI require large-scale models. Rather, we argue that this approach is not only fragile scientifically, but comes with undesirable consequences. First, it is not sustainable, as, despite efficiency improvements, its compute demands increase faster than model performance, leading to unreasonable economic requirements and a disproportionate environmental footprint. Second, it implies focusing on certain problems at the expense of others, leaving aside important applications, e.g. health, education, or the climate. Finally, it exacerbates a concentration of power, which centralizes decision-making in the hands of a few actors while threatening to disempower others in the context of shaping both AI research and its applications throughout society.