A Gaussian Comparison Theorem for Training Dynamics in Machine Learning

TL;DR

This paper introduces a Gaussian comparison theorem to analyze training dynamics in machine learning models, providing non-asymptotic results and refinement schemes for better understanding of model evolution.

Contribution

It presents a novel non-asymptotic analysis connecting training dynamics to a surrogate system using Gordon’s theorem, and extends dynamic mean-field theory to non-asymptotic settings.

Findings

Validated the dynamic mean-field expressions in asymptotic regimes

Developed an iterative scheme for non-asymptotic accuracy

Analyzed perceptron training with fluctuation parameters

Abstract

We study training algorithms with data following a Gaussian mixture model. For a specific family of such algorithms, we present a non-asymptotic result, connecting the evolution of the model to a surrogate dynamical system, which can be easier to analyze. The proof of our result is based on the celebrated Gordon comparison theorem. Using our theorem, we rigorously prove the validity of the dynamic mean-field (DMF) expressions in the asymptotic scenarios. Moreover, we suggest an iterative refinement scheme to obtain more accurate expressions in non-asymptotic scenarios. We specialize our theory to the analysis of training a perceptron model with a generic first-order (full-batch) algorithm and demonstrate that fluctuation parameters in a non-asymptotic domain emerge in addition to the DMF kernels.