Confident Adaptive Language Modeling

TL;DR

CALM is a framework that dynamically adjusts compute during language model inference by early exiting based on confidence, significantly reducing computational costs while maintaining high performance across diverse tasks.

Contribution

This work introduces CALM, a novel method for adaptive compute allocation in language models through confidence-based early exits, addressing key challenges in implementation.

Findings

Potential speedup of up to 3x in inference

Maintains high performance with reduced compute

Effective across diverse text generation tasks

Abstract

Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $\times 3$ -- while provably maintaining high performance.