Information-Theoretic Framework for Understanding Modern Machine-Learning

TL;DR

This paper presents an information-theoretic framework that models learning as universal prediction, providing insights into the success of modern architectures like deep neural networks and transformers through complexity and spectral analysis.

Contribution

It introduces a novel complexity measure based on model volume and spectral properties, unifies various learning settings, and explains the effectiveness of over-parameterized architectures.

Findings

Broad complexity range is key to successful architectures.

Spectral properties relate to model capacity and flat minima.

Framework applies across multiple learning paradigms.

Abstract

We introduce an information-theoretic framework that views learning as universal prediction under log loss, characterized through regret bounds. Central to the framework is an effective notion of architecture-based model complexity, defined by the probability mass or volume of models in the vicinity of the data-generating process, or its projection on the model class. This volume is related to spectral properties of the expected Hessian or the Fisher Information Matrix, leading to tractable approximations. We argue that successful architectures possess a broad complexity range, enabling learning in highly over-parameterized model classes. The framework sheds light on the role of inductive biases, the effectiveness of stochastic gradient descent, and phenomena such as flat minima. It unifies online, batch, supervised, and generative settings, and applies across the stochastic-realizable and agnostic regimes. Moreover, it provides insights into the success of modern machine-learning architectures, such as deep neural networks and transformers, suggesting that their broad complexity range naturally arises from their layered structure. These insights open the door to the design of alternative architectures with potentially comparable or even superior performance.