Scaling Laws for Transfer

TL;DR

This paper investigates how transfer learning scales with model size and data, revealing power-law relationships that quantify the effective data transfer and the generality of models in unsupervised fine-tuning.

Contribution

It introduces a framework to measure effective data transfer in transfer learning, demonstrating power-law scaling laws that relate model size and dataset size to transfer efficiency.

Findings

Effective data transfer follows a power-law with model and dataset size.

Pre-training multiplies the effective size of the fine-tuning dataset.

Transfer performance scales predictably with parameters, data, and compute.

Abstract

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.