A transformer architecture alteration to incentivise externalised reasoning

TL;DR

This paper introduces a transformer modification with an early-exit mechanism and reinforcement learning training to make large language models more verbose reasoners, reducing unnecessary computation while maintaining performance.

Contribution

It presents a novel architectural change and training pipeline that incentivizes models to truncate forward passes early, optimizing reasoning efficiency.

Findings

Models learn to adaptively reduce computation across tokens

Early-exit mechanism maintains task performance

Potential to minimize excess computation in reasoning models

Abstract

We propose a new architectural change, and post-training pipeline, for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We augment an existing transformer architecture with an early-exit mechanism at intermediate layers and train the model to exit at shallower layers when the next token can be predicted without deep computation. After a calibration stage, we incentivise the model to exit as early as possible while maintaining task performance using reinforcement learning. We provide preliminary results to this effect for small reasoning models, showing that they learn to adaptively reduce computations across tokens. We predict that, applied at the right scale, our approach can minimise the amount of excess computation that reasoning models have at their disposal to perform non-myopic planning using their internal activations, reserving this only for difficult-to-predict tokens.