Llasa+: Free Lunch for Accelerated and Streaming Llama-Based Speech Synthesis

TL;DR

Llasa+ is a novel streaming TTS model that significantly accelerates speech synthesis by predicting multiple tokens simultaneously and verifying them, achieving faster inference without quality loss.

Contribution

The paper introduces Llasa+, which employs multi-token prediction and a verification algorithm to speed up Llama-based TTS while maintaining high quality.

Findings

Achieves 1.48X speedup in speech synthesis

Maintains high quality despite acceleration

Applicable to other LLM-based models

Abstract

Recent progress in text-to-speech (TTS) has achieved impressive naturalness and flexibility, especially with the development of large language model (LLM)-based approaches. However, existing autoregressive (AR) structures and large-scale models, such as Llasa, still face significant challenges in inference latency and streaming synthesis. To deal with the limitations, we introduce Llasa+, an accelerated and streaming TTS model built on Llasa. Specifically, to accelerate the generation process, we introduce two plug-and-play Multi-Token Prediction (MTP) modules following the frozen backbone. These modules allow the model to predict multiple tokens in one AR step. Additionally, to mitigate potential error propagation caused by inaccurate MTP, we design a novel verification algorithm that leverages the frozen backbone to validate the generated tokens, thus allowing Llasa+ to achieve speedup without sacrificing generation quality. Furthermore, we design a causal decoder that enables streaming speech reconstruction from tokens. Extensive experiments show that Llasa+ achieves a 1.48X speedup without sacrificing generation quality, despite being trained only on LibriTTS. Moreover, the MTP-and-verification framework can be applied to accelerate any LLM-based model. All codes and models are publicly available at https://github.com/ASLP-lab/LLaSA_Plus.