Phase Transitions in the Output Distribution of Large Language Models
Julian Arnold, Flemming Holtorf, Frank Sch\"afer, Niels L\"orch
TL;DR
This paper explores phase transitions in large language models by applying statistical methods to detect abrupt changes in output distributions, revealing new behaviors and transitions as models evolve.
Contribution
It adapts physics-inspired statistical techniques to automatically identify phase transitions in language models without prior system-specific knowledge.
Findings
Detection of previously unknown phase transitions
Application of distributional distance measures
Discovery of new behavioral regimes in language models
Abstract
In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. Analogous phenomena have recently been observed in large language models. Typically, the task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. These methods are largely system agnostic and, as shown here, can be adapted to study the behavior of large language models. In particular, we quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens. This versatile approach is…
Peer Reviews
Decision·Submitted to ICLR 2025
- The paper is innovative, linking physics concepts with language models - The paper studies multiple open LLMs - The proposed metric is theoretically grounded in a link to the Fisher information - I found the execution to be rigorous and writing to be clear - Findings can be of potential interest, e.g. it might be surprising that there phase transitions are identified at some training epochs
From my perspective, the primary weakness of the paper is the lack of any external validation of the proposed approach. While the paper proposes a method for measuring phase transitions, it remained unclear to me how to validate the findings. The findings might in principle depend a lot on the choice of the divergence. Is there any independent way of validating that the outcomes of the proposed method are useful or interesting? Is there a clear use case? Is there a principled relation to a theor
The connection between phase transitions in physics and large language models is interesting!
The paper's main challenge and contribution are unclear. While the authors cite several previous works that have analyzed phase transitions, their contribution appears to be proposing new statistical detection methods. However, if this is their primary contribution, the experimental validation is insufficient - they use only a limited number of prompts and test cases. More extensive experiments with diverse prompts would be needed to validate their methods. Alternatively, if the paper aims to e
This paper addresses an interesting problem: whether changes in output distributions of LLMs, as one varies an underlying parameter, can be used to detect interesting changes in the underlying mechanisms. It reviews some potentially useful measures from statistical physics, that place widely used measures in NLP and ML such as the KL divergence in a broader context. The authors have looked at many different models and checkpoints, and obtaining the numbers reported in this paper / creating the
The first half of the paper is devoted to reviewing mathematical tools to measure distances between distributions. In the second half of the paper, the distance measures are used in some simple case studies. The main weakness of the paper is that each of these case studies stops where it starts to become interesting, such that we learn nothing new about the behaviour or underlying mechanisms of LLMs, and that the usefulness of the mathematical tools from the first half is never shown. I don't h
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling