Position: Enough of Scaling LLMs! Lets Focus on Downscaling

TL;DR

This paper argues for shifting focus from scaling up large language models to downscaling them, aiming to reduce resource use and environmental impact while maintaining performance.

Contribution

It introduces a holistic framework and practical strategies for downscaling LLMs, challenging the emphasis on increasing model size based on scaling laws.

Findings

Downscaling can maintain performance levels of large models.

Downscaling reduces computational and environmental costs.

Proposes practical methods for resource-efficient LLM development.

Abstract

We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.