Critical Phase Transition in Large Language Models

TL;DR

This paper investigates whether large language models exhibit phase transitions, revealing that at certain temperature settings, their behavior changes qualitatively, akin to physical phase transitions, with implications for understanding their dynamics.

Contribution

The study provides the first analysis of phase transition phenomena in LLMs, identifying critical points and behaviors similar to natural phase transitions in physical systems.

Findings

Divergent statistical quantities at specific temperature points

Power-law decay of correlations near the transition

Slow convergence to stationary states in LLMs

Abstract

Large Language Models (LLMs) have demonstrated impressive performance. To understand their behaviors, we need to consider the fact that LLMs sometimes show qualitative changes. The natural world also presents such changes called phase transitions, which are defined by singular, divergent statistical quantities. Therefore, an intriguing question is whether qualitative changes in LLMs are phase transitions. In this work, we have conducted extensive analysis on texts generated by LLMs and suggested that a phase transition occurs in LLMs when varying the temperature parameter. Specifically, statistical quantities have divergent properties just at the point between the low-temperature regime, where LLMs generate sentences with clear repetitive structures, and the high-temperature regime, where generated sentences are often incomprehensible. In addition, critical behaviors near the phase transition point, such as a power-law decay of correlation and slow convergence toward the stationary state, are similar to those in natural languages. Our results suggest a meaningful analogy between LLMs and natural phenomena.