Latent-Level Enhancement with Flow Matching for Robust Automatic Speech Recognition

TL;DR

This paper introduces a latent-level enhancement method using flow matching to refine distorted speech representations during ASR inference, improving robustness without retraining the entire model.

Contribution

The novel FM-Refiner module operates on ASR encoder latents, providing a plug-and-play solution that enhances recognition accuracy in noisy conditions.

Findings

Consistently reduces word error rate in noisy environments

Effective when combined with traditional speech enhancement methods

Lightweight and does not require fine-tuning the ASR model

Abstract

Noise-robust automatic speech recognition (ASR) has been commonly addressed by applying speech enhancement (SE) at the waveform level before recognition. However, speech-level enhancement does not always translate into consistent recognition improvements due to residual distortions and mismatches with the latent space of the ASR encoder. In this letter, we introduce a complementary strategy termed latent-level enhancement, where distorted representations are refined during ASR inference. Specifically, we propose a plug-and-play Flow Matching Refinement module (FM-Refiner) that operates on the output latents of a pretrained CTC-based ASR encoder. Trained to map imperfect latents-either directly from noisy inputs or from enhanced-but-imperfect speech-toward their clean counterparts, the FM-Refiner is applied only at inference, without fine-tuning ASR parameters. Experiments show that FM-Refiner consistently reduces word error rate, both when directly applied to noisy inputs and when combined with conventional SE front-ends. These results demonstrate that latent-level refinement via flow matching provides a lightweight and effective complement to existing SE approaches for robust ASR.