To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis

TL;DR

This paper investigates the effects of data repetition in pre-training large language models, revealing overfitting issues, key factors influencing degradation, and the limited effectiveness of regularization techniques, with MoE offering scalable tuning benefits.

Contribution

It provides empirical insights into multi-epoch degradation in LLMs, identifies main contributing factors, and evaluates regularization and MoE strategies for efficient training.

Findings

Data repetition causes overfitting and performance degradation.

Dataset size, model parameters, and training objectives are key factors.

Dropout can mitigate degradation if carefully tuned, while other regularizations are less effective.

Abstract

Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically investigate three key aspects under this approach. First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist of dataset quality and model FLOPs. Finally, we explore whether widely used regularization can alleviate multi-epoch degradation. Most regularization techniques do not yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, we discover that leveraging mixture-of-experts (MoE) enables cost-effective and efficient hyper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on a broader scale.