How Far Do SSL Speech Models Listen for Tone? Temporal Focus of Tone Representation under Low-resource Transfer

TL;DR

This paper investigates how self-supervised speech models capture lexical tone across four diverse languages, revealing that tone transfer depends on the downstream task and varies in temporal focus.

Contribution

It provides the first analysis of tone representation in SSL speech models across multiple tone languages and shows how task type influences temporal focus in tone transfer.

Findings

Tone cues span approximately 100-180ms depending on language.

Tone transfer varies with downstream task, affecting temporal focus.

Task type influences how SSL models utilize tone information.

Abstract

Lexical tone is central to many languages but remains underexplored in self-supervised learning (SSL) speech models, especially beyond Mandarin. We study four languages with complex and diverse tone systems (Burmese, Thai, Lao, and Vietnamese) to ask how far such models "listen" for tone and how transfer operates in low-resource conditions. As a baseline reference, we estimate the temporal span of tone cues: approximately 100ms (Burmese/Thai) and 180ms (Lao/Vietnamese). Probes and gradient analysis on fine-tuned SSL models reveal that tone transfer varies by downstream task: automatic speech recognition fine-tuning aligns spans with language-specific tone cues, while prosody- and voice-related tasks bias toward overly long spans. These findings indicate that tone transfer is shaped by downstream task, highlighting task effects on temporal focus in tone modeling.