Universal pre-training by iterated random computation

TL;DR

This paper explores using randomly generated data for pre-training models, providing theoretical justification and empirical evidence that such pre-training enhances zero-shot learning, especially when combined with fine-tuning.

Contribution

It introduces a theoretical framework for pre-training with synthetic data and demonstrates its effectiveness in improving model performance and convergence.

Findings

Synthetic pre-training enables zero-shot in-context learning.

Model performance improves with scale of synthetic data.

Fine-tuning after synthetic pre-training leads to faster convergence.

Abstract

We investigate the use of randomly generated data for the sake of pre-training a model. We justify this approach theoretically from the perspective of algorithmic complexity, building on recent research that shows that sequence models can be trained to approximate Solomonoff induction. We derive similar, but complementary theoretical results. We show empirically that synthetically generated data can be used to pre-train a model before the data is seen. We replicate earlier results that models trained this way show zero-shot in-context learning across a variety of datasets, and that this performance improves with scale. We extend earlier results to real-world data, and show that finetuning a model after pre-training offers faster convergence and better generalization.