Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer

TL;DR

GPST introduces a hierarchical transformer for efficient, high-quality speech generation that handles long acoustic sequences, supports multilingual and personalized speech synthesis, and outperforms existing models in key metrics.

Contribution

The paper presents GPST, a novel hierarchical transformer architecture that effectively models long speech sequences and enables versatile, high-quality speech synthesis including multilingual and personalized capabilities.

Findings

GPST significantly reduces word error rate compared to existing models.

GPST produces more natural and coherent speech from brief prompts.

GPST demonstrates superior speaker similarity and speech quality.

Abstract

While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce \textbf{G}enerative \textbf{P}re-trained \textbf{S}peech \textbf{T}ransformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. The code is available at \url{https://github.com/youngsheen/GPST}.