Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics

TL;DR

This paper investigates whether language models internally represent alternative reasoning paths during text generation by analyzing hidden activations, revealing correlations between uncertainty, controllability, and future outcomes.

Contribution

It demonstrates that hidden activations can predict a model's uncertainty and future outcomes, indicating internal representation of multiple reasoning paths during generation.

Findings

Uncertainty correlates with controllability via activation interventions.

Hidden activations predict future outcome distributions.

Models implicitly represent multiple reasoning paths.

Abstract

When a language model generates text, the selection of individual tokens might lead it down very different reasoning paths, making uncertainty difficult to quantify. In this work, we consider whether reasoning language models represent the alternate paths that they could take during generation. To test this hypothesis, we use hidden activations to control and predict a language model's uncertainty during chain-of-thought reasoning. In our experiments, we find a clear correlation between how uncertain a model is at different tokens, and how easily the model can be steered by controlling its activations. This suggests that activation interventions are most effective when there are alternate paths available to the model -- in other words, when it has not yet committed to a particular final answer. We also find that hidden activations can predict a model's future outcome distribution, demonstrating that models implicitly represent the space of possible paths.