The Fine-Grained Complexity of Gradient Computation for Training Large Language Models

TL;DR

This paper establishes the computational complexity limits of gradient computation in training large language models, showing that certain steps cannot be performed faster than quadratic time unless a major complexity hypothesis fails.

Contribution

It extends previous results to the gradient computation of one-layer attention networks, fully characterizing the fine-grained complexity of LLM training steps.

Findings

Gradient computation for one-layer attention networks is nearly as hard as the forward pass.

No truly sub-quadratic algorithms exist for these computations under SETH.

The results fully characterize the complexity of training large language models.

Abstract

Large language models (LLMs) have made fundamental contributions over the last a few years. To train an LLM, one needs to alternatingly run `forward' computations and `backward' computations. The forward computation can be viewed as attention function evaluation, and the backward computation can be viewed as a gradient computation. In previous work by [Alman and Song, NeurIPS 2023], it was proved that the forward step can be performed in almost-linear time in certain parameter regimes, but that there is no truly sub-quadratic time algorithm in the remaining parameter regimes unless the popular hypothesis SETH is false. In this work, we show nearly identical results for the harder-seeming problem of computing the gradient of loss function of one layer attention network, and thus for the entire process of LLM training. This completely characterizes the fine-grained complexity of every step of LLM training.