PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs

TL;DR

This paper introduces PAL, a hybrid method for integrating audio encoders into large language models that improves efficiency and performance by combining attention-based and token-based approaches, enabling richer audio semantics transfer.

Contribution

The paper proposes PAL, a novel hybrid integration approach combining PLITS and LAL methods, significantly enhancing efficiency and effectiveness of audio-to-LLM transfer.

Findings

LAL matches or outperforms existing methods in multiple tasks.

PAL achieves comparable or better performance with higher efficiency.

Memory usage is reduced by about 60%, throughput increased by about 190%.

Abstract

Integration of audio perception into large language models (LLMs) is an emerging research area for enabling machine listening applications, yet efficient transfer of rich audio semantics from audio encoders to LLMs remains underexplored. The most widely used integration paradigm projects audio-encoder output tokens into the LLM input space (e.g., via an MLP or a Q-Former) and then prepends or inserts them into the text token sequence. We refer to this generic scheme as Prepend to the LLM's input token space (PLITS) integration. We propose an efficient alternative, Lightweight Audio LLM Integration (LAL). LAL injects audio representations solely through the attention mechanism at selected LLM layers, bypassing the feed-forward module. It encodes rich audio semantics at an appropriate level of abstraction for integration into different transformer blocks, substantially reducing computational overhead compared to existing approaches. We further introduce PAL, a hybrid integration approach for efficiently Probing Audio encoders via LLM. PAL applies PLITS only to a compact set of summary tokens while integrating the full audio token sequence via LAL. Under an identical training curriculum, LAL consistently matches or outperforms existing integration approaches across multiple base LLMs and tasks, with improvements of up to 30% over a strong PLITS baseline, while reducing memory usage by about 60% and increasing throughput by about 190%. Moreover, PAL matches or exceeds PLITS performance while offering substantially better computational and memory efficiency.