Global Convergence in Training Large-Scale Transformers

TL;DR

This paper provides a rigorous theoretical analysis of the convergence properties of large-scale Transformers during training, establishing conditions under which gradient flow converges to a global minimum in the mean-field limit.

Contribution

It introduces novel mean-field techniques for analyzing Transformers, extending existing tools to partial homogeneity and local Lipschitz smoothness assumptions.

Findings

Gradient flow converges to a global minimum with small weight decay.

Mean-field limit of large Transformers is characterized by a Wasserstein PDE.

New analytical techniques for Transformers may be useful for future research.

Abstract

Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks (Lu et al., 2020) that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only $\textit{partial homogeneity}$ and $\textit{local Lipschitz smoothness}$. These new techniques may be of independent interest.