How Does Machine Learning Manage Complexity?

TL;DR

This paper analyzes the complexity management capabilities of machine learning models through a computational complexity perspective, emphasizing their ability to model complex distributions.

Contribution

It introduces a complexity-theoretic framework for understanding machine learning, focusing on computable distributions and their relation to model power and limitations.

Findings

Models producing distributions close to pseudorandom generators must be nearly uniform.

Machine learning models can manage complexity by approximating computable distributions.

The framework abstracts from specific learning algorithms, focusing on distribution properties.

Abstract

We provide a computational complexity lens to understand the power of machine learning models, particularly their ability to model complex systems. Machine learning models are often trained on data drawn from sampleable or more complex distributions, a far wider range of distributions than just computable ones. By focusing on computable distributions, machine learning models can better manage complexity via probability. We abstract away from specific learning mechanisms, modeling machine learning as producing P/poly-computable distributions with polynomially-bounded max-entropy. We illustrate how learning computable distributions models complexity by showing that if a machine learning model produces a distribution $\mu$ that minimizes error against the distribution generated by a cryptographic pseudorandom generator, then $\mu$ must be close to uniform.