Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs

TL;DR

This paper investigates why large language models experience performance plateaus despite increasing size, emphasizing the roles of data quality, density, and training strategies, and proposing a new scaling law for sub-scaling regimes.

Contribution

It introduces a sub-optimal scaling law that accounts for data density and training strategies, improving performance prediction in sub-scaling regimes.

Findings

High data density causes diminishing returns due to redundancy.

Optimal resource allocation is essential for continued performance gains.

A new scaling law better predicts performance in sub-scaling regimes.

Abstract

Traditional scaling laws in natural language processing suggest that increasing model size and training data enhances performance. However, recent studies reveal deviations, particularly in large language models, where performance improvements decelerate, which is a phenomenon known as sub-scaling. This paper revisits these scaling laws by examining the impact of data quality and training strategies on model performance. Through extensive empirical analysis of over 400 models, we identify high data density and non-optimal resource allocation as key factors contributing to sub-scaling. High data density leads to diminishing returns due to redundant information, while optimal resource allocation is crucial for sustained performance improvements. We propose a sub-optimal scaling law that better predicts performance in sub-scaling regimes, highlighting the importance of data quality and diversity.