MUKA: Multi Kernel Audio Adaptation Of Audio-Language Models

TL;DR

MUKA introduces a multi-kernel adaptation framework for large audio-language models, combining local and global representations to enable efficient few-shot adaptation without additional training, achieving state-of-the-art results.

Contribution

The paper proposes MUKA, a novel multi-kernel adaptation method that enhances audio-language model adaptation by integrating local and global semantic representations without extra training.

Findings

MUKA outperforms existing training-free adaptation methods on 11 audio datasets.

MUKA surpasses some training-based adapters in several scenarios.

The method maintains theoretical guarantees of kernel methods.

Abstract

Multimodal foundation models have demonstrated impressive generalization capabilities, yet efficiently adapting them to new tasks in a few-shot setting remains a critical challenge. In this work, we investigate the few-shot adaptation of Large Audio-Language Models (ALMs) through both training-based and training-free approaches. We introduce MUKA, a multi-kernel adaptation framework that combines the fine-grained, context-dependent representations of instruction-tuning based models like Pengi with the global semantic representations of contrastive pretraining methods like CLAP. By constructing a product kernel that aligns local similarity with global semantics, MUKA enhances representational power while preserving the theoretical guarantees of kernel methods and avoiding additional training. Extensive experiments across 11 diverse audio datasets demonstrate that MUKA achieves state-of-the-art performance among training-free methods and even surpasses training-based adapters in several scenarios, offering a compelling balance between adaptability and efficiency.