Perfect diffusion is $\mathsf{TC}^0$ -- Bad diffusion is Turing-complete

TL;DR

This paper investigates the computational limits of diffusion-based language models, showing that perfect diffusion is computationally limited to the class $ ext{TC}^0$, while bad diffusion can simulate any Turing machine, revealing a stark contrast in capabilities.

Contribution

It establishes a theoretical dichotomy demonstrating that the quality of the diffusion network determines whether it is computationally limited or Turing-complete.

Findings

Perfect diffusion models are limited to the $ ext{TC}^0$ complexity class.

Bad diffusion models can simulate any Turing machine.

Theoretical insights into the capabilities and limitations of diffusion models.

Abstract

This paper explores the computational complexity of diffusion-based language modeling. We prove a dichotomy based on the quality of the score-matching network in a diffusion model. In one direction, a network that exactly computes the score function of some initial distribution can only perform language modeling within the $\mathsf{TC}^0$ complexity class, reflecting limitations tied to rapid convergence. In the other direction, we show that if there is no requirement for the network to match any score function, then diffusion modeling can simulate any Turing machine in a certain sense. This dichotomy provides a theoretical lens on the capabilities and limitations of diffusion models, particularly concerning tasks requiring sequential computation. We conjecture extensions of our theoretical results, including for the case where the diffusion model is not perfect, but merely good. We also discuss the wider context and practical implications, and hypothesize that a machine learning architecture that can interpolate between sequential and parallel modes of operation would be superior to both Transformers and diffusion models.