CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models

TL;DR

This paper introduces Class-Conditional Sparse Attention Vectors, a method that learns class-specific importance weights for attention heads in large audio-language models, significantly improving few-shot classification performance.

Contribution

It proposes a novel class-conditional weighting scheme for attention heads, enabling specialization and improved ensemble predictions in large audio-language models.

Findings

Outperforms state-of-the-art uniform voting methods by up to 14.52% in accuracy.

Achieves up to 8.35% absolute gains in spoofing detection.

Demonstrates consistent improvements across multiple audio and audiovisual classification benchmarks.

Abstract

Large audio-language models (LALMs) exhibit strong zero-shot capabilities in multiple downstream tasks, such as audio question answering (AQA) and abstract reasoning; however, these models still lag behind specialized models for certain discriminative tasks (e.g., audio classification). Recent studies show that sparse subsets of attention heads within an LALM can serve as strong discriminative feature extractors for downstream tasks such as classification via simple voting schemes. However, these methods assign uniform weights to all selected heads, implicitly assuming that each head contributes equally across all semantic categories. In this work, we propose Class-Conditional Sparse Attention Vectors for Large Audio-Language Models, a few-shot classification method that learns class-dependent importance weights over attention heads. This formulation allows individual heads to specialize in distinct semantic categories and to contribute to ensemble predictions proportionally to their estimated reliability. Experiments on multiple few-shot audio and audiovisual classification benchmarks and tasks demonstrate that our method consistently outperforms state-of-the-art uniform voting-based approaches by up to 14.52%, 1.53%, 8.35% absolute gains for audio classification, audio-visual classification, and spoofing detection respectively.