SICL-AT: Another way to adapt Auditory LLM to low-resource task

TL;DR

This paper introduces SICL-AT, a novel post-training method that enhances auditory LLMs' in-context learning ability, improving performance on low-resource speech and audio tasks without extensive fine-tuning.

Contribution

Proposes SICL-AT, a new post-training approach that strengthens auditory LLMs' in-context learning, outperforming fine-tuning in low-resource scenarios.

Findings

Vanilla ICL improves zero-shot performance across tasks

SICL-AT enhances model capabilities with high-resource speech data

Method outperforms direct fine-tuning in low-resource settings

Abstract

Auditory Large Language Models (LLMs) have demonstrated strong performance across a wide range of speech and audio understanding tasks. Nevertheless, they often struggle when applied to low-resource or unfamiliar tasks. In case of labeled in-domain data is scarce or mismatched to the true test distribution, direct fine-tuning can be brittle. In-Context Learning (ICL) provides a training-free, inference-time solution by adapting auditory LLMs through conditioning on a few in-domain demonstrations. In this work, we first show that \emph{Vanilla ICL}, improves zero-shot performance across diverse speech and audio tasks for selected models which suggest this ICL adaptation capability can be generalized to multimodal setting. Building on this, we propose \textbf{Speech In-Context Learning Adaptation Training (SICL-AT)}, a post-training recipe utilizes only high resource speech data intending to strengthen model's in-context learning capability. The enhancement can generalize to audio understanding/reasoning task. Experiments indicate our proposed method consistently outperforms direct fine-tuning in low-resource scenario.