Optimal Transport for Machine Learners

TL;DR

This paper provides a comprehensive overview of Optimal Transport theory, its mathematical foundations, numerical methods, and applications in machine learning, especially for generative models and neural network training.

Contribution

It offers a detailed synthesis of OT's mathematical principles, numerical algorithms, and diverse applications in modern machine learning.

Findings

Explains fundamental OT formulations and the Bures metric.

Introduces numerical methods like entropic regularization.

Highlights applications in neural network training and generative models.

Abstract

Optimal Transport is a foundational mathematical theory that connects optimization, partial differential equations, and probability. It offers a powerful framework for comparing probability distributions and has recently become an important tool in machine learning, especially for designing and evaluating generative models. These course notes cover the fundamental mathematical aspects of OT, including the Monge and Kantorovich formulations, Brenier's theorem, the dual and dynamic formulations, the Bures metric on Gaussian distributions, and gradient flows. It also introduces numerical methods such as linear programming, semi-discrete solvers, and entropic regularization. Applications in machine learning include topics like training neural networks via gradient flows, token dynamics in transformers, and the structure of GANs and diffusion models. These notes focus primarily on mathematical content rather than deep learning techniques.