L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models

TL;DR

L-MTP introduces a leap-based multi-token prediction method for large language models that improves long-range dependency capture and accelerates inference by predicting non-adjacent tokens in a single pass.

Contribution

It proposes a novel leap multi-token prediction technique that extends traditional methods, enhancing efficiency and long-range dependency modeling in LLMs.

Findings

Boosts inference speed significantly.

Improves performance on diverse benchmarks.

Effectively captures long-range dependencies.

Abstract

Large language models (LLMs) have achieved notable progress. Despite their success, next-token prediction (NTP), the dominant method for LLM training and inference, is constrained in both contextual coverage and inference efficiency due to its inherently sequential process. To overcome these challenges, we propose leap multi-token prediction~(L-MTP), an innovative token prediction method that extends the capabilities of multi-token prediction (MTP) by introducing a leap-based mechanism. Unlike conventional MTP, which generates multiple tokens at adjacent positions, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass. This structured leap not only enhances the model's ability to capture long-range dependencies but also enables a decoding strategy specially optimized for non-sequential leap token generation, effectively accelerating inference. We theoretically demonstrate the benefit of L-MTP in improving inference efficiency. Experiments across diverse benchmarks validate its merit in boosting both LLM performance and inference speed. The source code is available at https://github.com/Xiaohao-Liu/L-MTP.