AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Models

TL;DR

This paper provides an in-depth analysis of how large audio-language models perceive auditory attributes internally, revealing their reliance on input queries and the evolution of attribute information across layers.

Contribution

It introduces the first detailed examination of auditory attribute perception in LALMs and proposes a method to enhance their performance based on these insights.

Findings

Attribute information decreases with layer depth when recognition fails.

Early layer attribute resolution correlates with better accuracy.

LALMs rely heavily on querying auditory inputs rather than hidden states.

Abstract

Understanding the internal mechanisms of large audio-language models (LALMs) is crucial for interpreting their behavior and improving performance. This work presents the first in-depth analysis of how LALMs internally perceive and recognize auditory attributes. By applying vocabulary projection on three state-of-the-art LALMs, we track how attribute information evolves across layers and token positions. We find that attribute information generally decreases with layer depth when recognition fails, and that resolving attributes at earlier layers correlates with better accuracy. Moreover, LALMs heavily rely on querying auditory inputs for predicting attributes instead of aggregating necessary information in hidden states at attribute-mentioning positions. Based on our findings, we demonstrate a method to enhance LALMs. Our results offer insights into auditory attribute processing, paving the way for future improvements.