Forking Paths in Neural Text Generation
Eric Bigelow, Ari Holtzman, Hidenori Tanaka, Tomer Ullman
TL;DR
This paper introduces a novel method to identify key forking tokens in neural text generation, revealing how small changes at specific points can lead to vastly different outcomes, which enhances uncertainty estimation in LLMs.
Contribution
It presents a flexible, model-agnostic approach to analyze uncertainty dynamics at the token level in large language models, without requiring fine-tuning or access to model weights.
Findings
Many forking tokens identified, including punctuation marks.
LLMs can produce very different outputs from a single token change.
Method applied across diverse tasks and domains.
Abstract
Estimating uncertainty in Large Language Models (LLMs) is important for properly evaluating LLMs, and ensuring safety for users. However, prior approaches to uncertainty estimation focus on the final answer in generated text, ignoring intermediate steps that might dramatically impact the outcome. We hypothesize that there exist key forking tokens, such that re-sampling the system at those specific tokens, but not others, leads to very different outcomes. To test this empirically, we develop a novel approach to representing uncertainty dynamics across individual tokens of text generation, and applying statistical models to test our hypothesis. Our approach is highly flexible: it can be applied to any dataset and any LLM, without fine tuning or accessing model weights. We use our method to analyze LLM responses on 7 different tasks across 4 domains, spanning a wide range of typical use…
Peer Reviews
Decision·ICLR 2025 Poster
1. The research question in this work is compelling because forking tokens are important for understanding model behaviors and steering model generation. 2. The approach of detecting forking tokens with CPD models is also innovative.
The core contribution of this paper is already intriguing, so the following weakness is likely minor. The method section is somewhat difficult to follow. In Section 2.2, I struggled due to insufficient explanation of the connection between the definition of $o_t$ and the subsequent detection method at the beginning of Sec. 2.2. While the high-level concept in lines 247–259 is more understandable, some details remain unclear. For instance, defining $\tau_{i-1}$ and $\tau_i$ as the start and end
1. The paper formulate a interesting and (I think) novel hypothesis about forking tokens, that there are a few sparse but critical tokens that will determine the trajectory of the generation, and uncertainty estimation should depend on these critical tokens. 2. The estimation method (finding the critical forking token) seems statistically motivated and sound.
1. I think the method section is written in a very unclear way. For example, the Bayesian formulation in line 258 can be better described. The Gibbs sampling step is also very unclear. There seem to be lots of details missing. A related question: why use linear regression for the CPD? The math in the survival analysis part makes sense, but still lacks all the execution details: what is d? and what are the pros and cons of these two approaches? 2. I agree that the forking theory is interesting,
* The assumption in this article is quite important, and the pipeline constructed to validate the assumption and the evaluation metrics are very interesting * The experimental analysis in this article is detailed and comprehensive. * The assumption in this article is very interesting and significant.
The evaluation of the uncertainty of language models designed in this article requires sampling a large number of generated results for different tokens and conducting evaluation analysis. Therefore, the cost of evaluating a single sample is also enormous, which may affect the scalability of this work. However, this does not negate the innovativeness of this work.
Videos
Taxonomy
TopicsTopic Modeling
MethodsFocus