Scaling Laws for Deep Learning

TL;DR

This paper investigates the scaling laws governing deep learning, demonstrating their predictability across models and tasks, analyzing their theoretical origins, and proposing methods to approach the fundamental error limits.

Contribution

It establishes predictable scaling laws for deep learning training and pruning, analyzes their theoretical basis, and suggests a pathway to reduce errors to fundamental limits.

Findings

Scaling laws are predictable for state-of-the-art models and tasks.

Deep learning errors are dominated by sources far from the theoretical minimum.

A conjectural approach using Nyquist learners could reach the generalization error lower limit.

Abstract

Running faster will only get you so far -- it is generally advisable to first understand where the roads lead, then get a car ... The renaissance of machine learning (ML) and deep learning (DL) over the last decade is accompanied by an unscalable computational cost, limiting its advancement and weighing on the field in practice. In this thesis we take a systematic approach to address the algorithmic and methodological limitations at the root of these costs. We first demonstrate that DL training and pruning are predictable and governed by scaling laws -- for state of the art models and tasks, spanning image classification and language modeling, as well as for state of the art model compression via iterative pruning. Predictability, via the establishment of these scaling laws, provides the path for principled design and trade-off reasoning, currently largely lacking in the field. We then continue to analyze the sources of the scaling laws, offering an approximation-theoretic view and showing through the exploration of a noiseless realizable case that DL is in fact dominated by error sources very far from the lower error limit. We conclude by building on the gained theoretical understanding of the scaling laws' origins. We present a conjectural path to eliminate one of the current dominant error sources -- through a data bandwidth limiting hypothesis and the introduction of Nyquist learners -- which can, in principle, reach the generalization error lower limit (e.g. 0 in the noiseless case), at finite dataset size.