Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference

TL;DR

Entropy Adaptive Decoding (EAD) dynamically switches between models during language generation based on uncertainty, significantly reducing computational costs while maintaining high performance levels across various model sizes.

Contribution

This paper introduces EAD, a novel method that adaptively switches models during inference based on entropy, enabling efficient computation with controlled output divergence.

Findings

Achieves 96.7% of 11B model performance using only 43% of tokens.

Reduces computational cost by 41.5% with LLaMA models.

Maintains high performance with minimal divergence across model pairs.

Abstract

We present Entropy Adaptive Decoding (EAD), a novel approach for efficient language model inference that dynamically switches between different-sized models based on prediction uncertainty. By monitoring rolling entropy in model logit distributions, our method identifies text regions where a smaller model suffices and switches to a larger model only when prediction uncertainty exceeds a threshold. Unlike speculative decoding approaches that maintain perfect output fidelity through verification, EAD accepts controlled output divergence in exchange for computational efficiency. Our experiments on the MATH benchmark demonstrate remarkable efficiency gains across different model families. Using the LLaMA family, we maintain 96.7\% of the 11B model's performance (50.4\% vs 52.1\%) while using it for only 43\% of tokens, decreasing computational cost by 41.5\%. These gains become more pronounced with larger size differentials in the Qwen family, where we achieve 92.9\% of the 14B model's performance (74.3\% vs 80.0\%) while using it for just 25\% of tokens, decreasing computational cost by 67\%. The consistency of these results across model pairs suggests that language model computation can be significantly optimized by selectively deploying model capacity based on local generation complexity. Our findings indicate that current approaches to model inference may be unnecessarily conservative in their pursuit of perfect output fidelity, and that accepting minor performance trade-offs can enable dramatic reductions in computational costs.