SpeechAccentLLM: A Unified Framework for Foreign Accent Conversion and Text to Speech

TL;DR

SpeechAccentLLM introduces a unified framework leveraging LLM techniques for foreign accent conversion and TTS, featuring novel models for speech tokenization, multitask training for data efficiency, and postprocessing for quality enhancement.

Contribution

The paper presents SpeechCodeVAE for speech tokenization with CTC integration, a multitask training strategy for improved FAC and TTS, and SpeechRestorer for postprocessing to reduce errors and improve prosody.

Findings

SpeechCodeVAE achieves optimal content and structural trade-offs.

Multitask training accelerates convergence and improves speech quality.

SpeechRestorer effectively reduces errors and enhances prosody.

Abstract

Foreign accent conversion (FAC) in speech processing remains a challenging task. Building on the remarkable success of large language models (LLMs) in Text-to-Speech (TTS) tasks, this study investigates the adaptation of LLM-based techniques for FAC, which we term SpeechAccentLLM. At the core of this framework, we introduce SpeechCodeVAE, the first model to integrate connectionist temporal classification (CTC) directly into codebook discretization for speech content tokenization. This novel architecture generates tokens with a unique "locality" property, as validated by experiments demonstrating optimal trade-offs among content faithfulness, temporal coherence, and structural recoverability. Then, to address data scarcity for the FAC module, we adopted a multitask learning strategy that jointly trains the FAC and TTS modules. Beyond mitigating data limitations, this approach yielded accelerated convergence and superior speech quality compared to standalone FAC training. Moreover, leveraging the salient properties of our discrete speech representations, we introduce SpeechRestorer, a postprocessing architecture designed to refine LLM-generated outputs. This module effectively mitigates stochastic errors prevalent in LLM inference pipelines while enhancing prosodic continuity, as validated by ablation experiments.