ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction

TL;DR

ELEGANCE introduces a framework that enhances audio-visual target speech extraction by integrating linguistic knowledge from large language models, significantly improving performance in challenging scenarios.

Contribution

It is the first to incorporate LLM-derived linguistic guidance into AV-TSE models, combining three strategies for improved speech extraction.

Findings

Improves extraction accuracy in visual cue impaired scenarios

Enhances performance on unseen languages and speaker switches

Effective across multiple AV-TSE backbones

Abstract

Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set. Demo page: https://alexwxwu.github.io/ELEGANCE/.