Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters

TL;DR

This paper introduces a novel multilingual inference method using speculative decoding with language-specific draft models, significantly reducing inference time across diverse languages and outperforming previous approaches.

Contribution

It proposes a targeted pretrain-and-finetune strategy for language-specific draft models to accelerate multilingual LLM inference, a novel approach in this domain.

Findings

Substantial inference speedup in multiple languages

Effective out-of-domain inference acceleration

Validated improvements with GPT-4o evaluation

Abstract

Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.