In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads

TL;DR

This paper explores how in-context learning operates in speech language models, focusing on acoustic features, linguistic structure, and the causal role of induction heads in speech tasks.

Contribution

It is the first to analyze ICL in speech models, revealing the impact of speaking rate and the causal importance of induction heads in speech-based ICL.

Findings

Speaking rate significantly influences ICL performance and is mimicked in output.

Pitch range and intensity have minimal impact and are not consistently reproduced.

Ablating induction heads eliminates ICL ability, confirming their causal role.

Abstract

In-Context Learning (ICL) has been extensively studied in text-only Language Models, but remains largely unexplored in the speech domain. Here, we investigate how linguistic and acoustic features affect ICL in Speech Language Models. We focus on the Text-to-Speech (TTS) task, which allows us to analyze ICL from two angles: (1) how accurately the model infers the task from the demonstrations (i.e., generating the correct spoken content), and (2) to what extent the model mimics the acoustic characteristics of the demonstration speech in its output. We find that speaking rate strongly affects ICL performance and is also mimicked in the output, whereas pitch range and intensity have little impact on performance and are not consistently reproduced. Finally, we investigate the role of induction heads in speech-based ICL and show that these heads play a causal role: ablating the top-k induction heads completely removes the model's ICL ability, mirroring findings from text-based ICL.