Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis

TL;DR

This paper introduces Shallow Flow Matching, a new technique for improving flow-based text-to-speech models by constructing intermediate states, leading to better naturalness and faster inference.

Contribution

The paper presents Shallow Flow Matching, a novel method that enhances flow matching in TTS models through intermediate state construction and adaptive inference, improving quality and speed.

Findings

Improves speech naturalness in TTS models.

Accelerates inference with adaptive-step ODE solvers.

Demonstrates consistent gains across multiple models.

Abstract

We propose Shallow Flow Matching (SFM), a novel mechanism that enhances flow matching (FM)-based text-to-speech (TTS) models within a coarse-to-fine generation paradigm. Unlike conventional FM modules, which use the coarse representations from the weak generator as conditions, SFM constructs intermediate states along the FM paths from these representations. During training, we introduce an orthogonal projection method to adaptively determine the temporal position of these states, and apply a principled construction strategy based on a single-segment piecewise flow. The SFM inference starts from the intermediate state rather than pure noise, thereby focusing computation on the latter stages of the FM paths. We integrate SFM into multiple TTS models with a lightweight SFM head. Experiments demonstrate that SFM yields consistent gains in speech naturalness across both objective and subjective evaluations, and significantly accelerates inference when using adaptive-step ODE solvers. Demo and codes are available at https://ydqmkkx.github.io/SFMDemo/.