Meek Models Shall Inherit the Earth
Hans Gundlach, Jayson Lynch, Neil Thompson

TL;DR
This paper suggests that due to diminishing returns on compute scaling, smaller 'meek' AI models will eventually match the performance of larger models, impacting AI strategy and policy.
Contribution
It introduces a model showing diminishing returns to compute and argues that meek models will inherit AI capabilities, challenging the focus on scaling for performance.
Findings
Diminishing returns to compute reduce advantage of larger models.
Proxies like training loss correlate with capability measures.
Empirical data supports convergence of model capabilities.
Abstract
The past decade has seen incredible scaling of AI systems by a few companies, leading to inequality in AI model performance. This paper argues that, contrary to prevailing intuition, the diminishing returns to compute scaling will lead to a convergence of AI model capabilities. In other words, meek models (those with limited computation budget) shall inherit the earth, approaching the performance level of the best models overall. We develop a model illustrating that under a fixed-distribution next-token objective, the marginal capability returns to raw compute shrink substantially. Given current scaling practices, we argue that these diminishing returns are strong enough that even companies that can scale their models exponentially faster than other organizations will eventually have little advantage in capabilities. As part of our argument, we give several reasons that proxies like…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEthics and Social Impacts of AI · Big Data and Business Intelligence · Economic and Technological Innovation
