Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semantically

TL;DR

This paper investigates whether cross-lingual alignment in Whisper-style speech encoders is driven by semantics rather than phonetic similarity, demonstrating semantic alignment persists without phonetic cues.

Contribution

The study provides evidence that Whisper-style speech encoders align languages semantically, not just phonetically, especially in models trained with translation objectives and through early-exiting techniques.

Findings

Spoken translation retrieval remains above chance without phonetic cues.

Semantic alignment persists in final encoder layers trained with translation.

Early-exiting the encoder improves speech recognition for low-resource languages.

Abstract

Cross-lingual alignment in pretrained language models enables knowledge transfer across languages. Similar alignment has been reported in Whisper-style speech encoders, based on spoken translation retrieval using representational similarity. However, prior work does not control for phonetic overlap between equivalent utterances, which may artificially support retrieval. We conduct pronunciation-controlled experiments to test whether cross-lingual alignment arises from semantic rather than phonetic similarity. Results show that spoken translation retrieval remains strongly above chance without phonetic cues in the final layers of encoders trained with a speech translation objective, most clearly for models additionally trained on translation. We further test early-exiting the encoder to induce representations we hypothesize to be less tied to language-specific semantics. These experiments indeed reveal performance gains in automatic speech recognition on low-resource languages unseen during training.