Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases

TL;DR

This paper demonstrates that repeating smaller datasets can accelerate training by leveraging sampling biases and layer-wise growth, offering a new perspective on data efficiency in machine learning.

Contribution

It introduces the concept that smaller datasets with repetitions can be intentionally used to improve training speed through induced sampling biases and layer-wise growth.

Findings

Repeating smaller datasets speeds up training across tasks and architectures.

Sampling biases enable layer-wise growth that accelerates learning.

Smaller datasets with repetitions can be a proactive strategy, not just a fallback.

Abstract

This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.