RecycleGPT: An Autoregressive Language Model with Recyclable Module

TL;DR

RecycleGPT introduces a novel autoregressive language model that leverages recycled model states to significantly reduce inference latency, achieving up to 1.4x speedup while maintaining high performance.

Contribution

The paper proposes RecycleGPT, a new method that recycles pre-generated states to accelerate decoding without retraining or multiple model runs.

Findings

Achieves up to 1.4x speedup in inference

Maintains high language modeling performance

Reduces inference latency significantly

Abstract

Existing large language models have to run K times to generate a sequence of K tokens. In this paper, we present RecycleGPT, a generative language model with fast decoding speed by recycling pre-generated model states without running the whole model in multiple steps. Our approach relies on the observation that adjacent tokens in a sequence usually have strong correlations and the next token in a sequence can be reasonably guessed or inferred based on the preceding ones. Experiments and analysis demonstrate the effectiveness of our approach in lowering inference latency, achieving up to 1.4x speedup while preserving high performance.