Learning Universal Predictors

TL;DR

This paper investigates the limits of meta-learning by training neural networks to approximate Solomonoff Induction using data generated from Universal Turing Machines, aiming for universal prediction capabilities.

Contribution

It introduces a method to leverage UTM-generated data for meta-learning neural networks capable of universal prediction, combining theoretical analysis and extensive experiments.

Findings

UTM data enhances meta-learning for universal prediction

Neural networks can approximate Solomonoff Induction with UTM data

Meta-training protocols influence the universality of learned predictors

Abstract

Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.