On the Effectiveness of Acoustic BPE in Decoder-Only TTS

TL;DR

This paper investigates the use of acoustic byte-pair encoding (BPE) in decoder-only text-to-speech models, demonstrating its benefits in improving speech intelligibility and diversity through comprehensive experiments.

Contribution

It provides the first thorough analysis of acoustic BPE's impact on decoder-only TTS, clarifying optimal settings and its advantages over other methods.

Findings

Acoustic BPE improves speech intelligibility and diversity.

Different BPE settings influence speech features.

Acoustic BPE is a beneficial tool for decoder-only TTS.

Abstract

Discretizing speech into tokens and generating them by a decoder-only model have been a promising direction for text-to-speech (TTS) and spoken language modeling (SLM). To shorten the sequence length of speech tokens, acoustic byte-pair encoding (BPE) has emerged in SLM that treats speech tokens from self-supervised semantic representations as characters to further compress the token sequence. But the gain in TTS has not been fully investigated, and the proper choice of acoustic BPE remains unclear. In this work, we conduct a comprehensive study on various settings of acoustic BPE to explore its effectiveness in decoder-only TTS models with semantic speech tokens. Experiments on LibriTTS verify that acoustic BPE uniformly increases the intelligibility and diversity of synthesized speech, while showing different features across BPE settings. Hence, acoustic BPE is a favorable tool for decoder-only TTS.