FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching

TL;DR

FELLE is a novel autoregressive speech synthesis model that combines language modeling with token-wise flow matching and a hierarchical coarse-to-fine approach to improve the quality and coherence of generated mel-spectrograms.

Contribution

It introduces a new autoregressive framework integrating flow matching with hierarchical coarse-to-fine generation for speech synthesis.

Findings

Significant improvements in TTS quality demonstrated.

Effective modeling of continuous-valued tokens with flow matching.

Enhanced temporal coherence and stability in synthesis.

Abstract

To advance continuous-valued token modeling and temporal-coherence enforcement, we propose FELLE, an autoregressive model that integrates language modeling with token-wise flow matching. By leveraging the autoregressive nature of language models and the generative efficacy of flow matching, FELLE effectively predicts continuous-valued tokens (mel-spectrograms). For each continuous-valued token, FELLE modifies the general prior distribution in flow matching by incorporating information from the previous step, improving coherence and stability. Furthermore, to enhance synthesis quality, FELLE introduces a coarse-to-fine flow-matching mechanism, generating continuous-valued tokens hierarchically, conditioned on the language model's output. Experimental results demonstrate the potential of incorporating flow-matching techniques in autoregressive mel-spectrogram modeling, leading to significant improvements in TTS generation quality, as shown in https://aka.ms/felle.