Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models

TL;DR

This paper introduces an adapter-based extension for pre-trained audio-visual speech recognition models, enabling efficient noise-specific adaptation with minimal parameter training while maintaining high performance.

Contribution

The work presents a novel adapter-based AVSR extension that allows noise-specific adaptation, reducing training parameters significantly compared to full fine-tuning, and leverages pre-trained models effectively.

Findings

Achieves up to 88.5% reduction in trainable parameters.

Maintains near state-of-the-art performance across various noise scenarios.

Supports extension with additional noise-specific adapters.

Abstract

We present an approach to Audio-Visual Speech Recognition that builds on a pre-trained Whisper model. To infuse visual information into this audio-only model, we extend it with an AV fusion module and LoRa adapters, one of the most up-to-date adapter approaches. One advantage of adapter-based approaches, is that only a relatively small number of parameters are trained, while the basic model remains unchanged. Common AVSR approaches train single models to handle several noise categories and noise levels simultaneously. Taking advantage of the lightweight nature of adapter approaches, we train noise-scenario-specific adapter-sets, each covering individual noise-categories or a specific noise-level range. The most suitable adapter-set is selected by previously classifying the noise-scenario. This enables our models to achieve an optimum coverage across different noise-categories and noise-levels, while training only a minimum number of parameters. Compared to a full fine-tuning approach with SOTA performance our models achieve almost comparable results over the majority of the tested noise-categories and noise-levels, with up to 88.5% less trainable parameters. Our approach can be extended by further noise-specific adapter-sets to cover additional noise scenarios. It is also possible to utilize the underlying powerful ASR model when no visual information is available, as it remains unchanged.