TASI Lectures on Physics for Machine Learning

TL;DR

This paper provides a comprehensive overview of neural network theory from a physics perspective, covering classical and recent results like the universal approximation theorem, neural tangent kernel, and connections to field theory.

Contribution

It offers a unified, physics-inspired framework for understanding neural network expressivity, dynamics, and statistical properties, integrating recent advances and classical results.

Findings

Neural tangent kernel characterizes neural network training dynamics.

Maximal update parameterization enhances feature learning.

Connections between neural networks and field theory are elucidated.

Abstract

These notes are based on lectures I gave at TASI 2024 on Physics for Machine Learning. The focus is on neural network theory, organized according to network expressivity, statistics, and dynamics. I present classic results such as the universal approximation theorem and neural network / Gaussian process correspondence, and also more recent results such as the neural tangent kernel, feature learning with the maximal update parameterization, and Kolmogorov-Arnold networks. The exposition on neural network theory emphasizes a field theoretic perspective familiar to theoretical physicists. I elaborate on connections between the two, including a neural network approach to field theory.