OverFill: Two-Stage Models for Efficient Language Model Decoding

TL;DR

OverFill introduces a two-stage decoding approach for large language models, decoupling prefill and decode phases to optimize efficiency and accuracy, significantly reducing training data needs and improving performance.

Contribution

It proposes a novel two-stage model architecture that decouples prefill and decode stages, enhancing efficiency and accuracy in language model inference.

Findings

OverFill outperforms pruned models by over 79% in accuracy.

It matches the performance of models trained from scratch with less data.

Significant latency improvements with minimal overhead.

Abstract

Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill.