LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement

TL;DR

This paper introduces a reinforcement learning framework for audio-visual speech enhancement that uses a Large Language Model to generate interpretable feedback, improving speech quality metrics and subjective listening experience.

Contribution

It presents a novel RL-based AVSE approach utilizing LLM-generated natural language feedback as a reward, enhancing interpretability and performance over traditional scalar metrics.

Findings

Outperforms supervised and DNSMOS-based RL baselines in PESQ and STOI.

LLM feedback provides semantically rich, explicit speech quality improvements.

Achieves better subjective listening test results.

Abstract

In existing Audio-Visual Speech Enhancement (AVSE) methods, objectives such as Scale-Invariant Signal-to-Noise Ratio (SI-SNR) and Mean Squared Error (MSE) are widely used; however, they often correlate poorly with perceptual quality and provide limited interpretability for optimization. This work proposes a reinforcement learning-based AVSE framework with a Large Language Model (LLM)-based interpretable reward model. An audio LLM generates natural language descriptions of enhanced speech, which are converted by a sentiment analysis model into a 1-5 rating score serving as the PPO reward for fine-tuning a pretrained AVSE model. Compared with scalar metrics, LLM-generated feedback is semantically rich and explicitly describes improvements in speech quality. Experiments on the 4th COG-MHEAR AVSE Challenge (AVSEC-4) dataset show that the proposed method outperforms a supervised baseline and a DNSMOS-based RL baseline in PESQ, STOI, neural quality metrics, and subjective listening tests.