Fundamental limits to learning closed-form mathematical models from data

TL;DR

This paper investigates the fundamental limits of learning closed-form mathematical models from noisy data, revealing a phase transition between learnability and unlearnability and highlighting the effectiveness of probabilistic model selection.

Contribution

It characterizes the phase transition in model learning from noisy data and compares the performance of probabilistic model selection with standard machine learning methods.

Findings

A phase transition exists between learnable and unlearnable regimes.

Probabilistic model selection achieves optimal generalization in both phases.

Standard neural networks are limited in the low-noise phase by their interpolation ability.

Abstract

Given a finite and noisy dataset generated with a closed-form mathematical model, when is it possible to learn the true generating model from the data alone? This is the question we investigate here. We show that this model-learning problem displays a transition from a low-noise phase in which the true model can be learned, to a phase in which the observation noise is too high for the true model to be learned by any method. Both in the low-noise phase and in the high-noise phase, probabilistic model selection leads to optimal generalization to unseen data. This is in contrast to standard machine learning approaches, including artificial neural networks, which in this particular problem are limited, in the low-noise phase, by their ability to interpolate. In the transition region between the learnable and unlearnable phases, generalization is hard for all approaches including probabilistic model selection.