Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders

TL;DR

This paper introduces Prompt-aware Mixture (PaM), a method that combines multiple audio encoders with large language models, enabling improved performance across various audio understanding tasks by dynamically selecting task-specific features.

Contribution

The paper proposes a novel prompt-aware mixture approach that uses multiple audio encoders to extract task-specific features, outperforming single-encoder models and traditional fusion methods.

Findings

PaM enables a single Speech LLM to surpass all single-encoder models on multiple tasks.

PaM outperforms concatenation and averaging baselines in feature fusion.

The approach improves performance on ASR, speaker verification, and audio captioning tasks.

Abstract

Connecting audio encoders with large language models (LLMs) allows the LLM to perform various audio understanding tasks, such as automatic speech recognition (ASR) and audio captioning (AC). Most research focuses on training an adapter layer to generate a unified audio feature for the LLM. However, different tasks may require distinct features that emphasize either semantic or acoustic aspects, making task-specific audio features more desirable. In this paper, we propose Prompt-aware Mixture (PaM) to enhance the Speech LLM that uses multiple audio encoders. Our approach involves using different experts to extract different features based on the prompt that indicates different tasks. Experiments demonstrate that with PaM, only one Speech LLM surpasses the best performances achieved by all single-encoder Speech LLMs on ASR, Speaker Number Verification, and AC tasks. PaM also outperforms other feature fusion baselines, such as concatenation and averaging. Our code would be available at: https://github.com/shanweiqiao/PaM