LP-CFM: Perceptual Invariance-Aware Conditional Flow Matching for Speech Modeling

TL;DR

This paper introduces LP-CFM, a speech modeling method that incorporates perceptual invariances like amplitude and timing shifts, improving robustness especially in low-resource scenarios.

Contribution

It proposes a novel perceptual invariance-aware flow matching framework with vector calibrated sampling for more robust speech generation.

Findings

Consistently outperforms conventional CFM in neural vocoding tasks.

Shows strong gains in low-resource and few-step sampling scenarios.

Enhances robustness by modeling perceptual invariances explicitly.

Abstract

The goal of this paper is to provide a new perspective on speech modeling by incorporating perceptual invariances such as amplitude scaling and temporal shifts. Conventional generative formulations often treat each dataset sample as a fixed representative of the target distribution. From a generative standpoint, however, such samples are only one among many perceptually equivalent variants within the true speech distribution. To address this, we propose Linear Projection Conditional Flow Matching (LP-CFM), which models targets as projection-aligned elongated Gaussians along perceptually equivalent variants. We further introduce Vector Calibrated Sampling (VCS) to keep the sampling process aligned with the line-projection path. In neural vocoding experiments across model sizes, data scales, and sampling steps, the proposed approach consistently improves over the conventional optimal transport CFM, with particularly strong gains in low-resource and few-step scenarios. These results highlight the potential of LP-CFM and VCS to provide more robust and perceptually grounded generative modeling of speech.