Looking Beyond The Top-1: Transformers Determine Top Tokens In Order

TL;DR

This paper reveals that Transformers predict tokens in order of their ranking, with saturation events occurring sequentially, and introduces a new early-exit method based on this insight to improve efficiency.

Contribution

It uncovers the ordered nature of saturation events in Transformers across modalities and proposes a task transition mechanism and an effective early-exit strategy.

Findings

Saturation events occur in token ranking order.

Transformers exhibit this behavior across architectures and modalities.

The proposed early-exit method improves efficiency.

Abstract

Understanding the inner workings of Transformers is crucial for achieving more accurate and efficient predictions. In this work, we analyze the computation performed by Transformers in the layers after the top-1 prediction has become fixed, which has been previously referred to as the "saturation event". We expand the concept of saturation events for top-k tokens, demonstrating that similar saturation events occur across language, vision, and speech models. We find that these saturation events happen in order of the corresponding tokens' ranking, i.e., the model first decides on the top ranking token, then the second highest ranking token, and so on. This phenomenon seems intrinsic to the Transformer architecture, occurring across different architectural variants (decoder-only, encoder-only, and to a lesser extent full-Transformer), and even in untrained Transformers. We propose an underlying mechanism of task transition for this sequential saturation, where task k corresponds to predicting the k-th most probable token, and the saturation events are in fact discrete transitions between the tasks. In support of this we show that it is possible to predict the current task from hidden layer embedding. Furthermore, using an intervention method we demonstrate that we can cause the model to switch from one task to the next. Finally, leveraging our findings, we introduce a novel token-level early-exit strategy, which surpasses existing methods in balancing performance and efficiency.