Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction

TL;DR

This paper enhances audio-visual target speech extraction by integrating linguistic constraints from pre-trained language models, leading to improved speech quality and robustness across languages and visual cue limitations without extra inference costs.

Contribution

It introduces a novel method of incorporating linguistic knowledge from PSLMs and PLMs into AV-TSE models as auxiliary supervision, improving performance.

Findings

Consistent improvement in speech quality and intelligibility.

Robust performance gains in multi-language and visual cue-impaired scenarios.

No additional computational cost during inference.

Abstract

Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains.