Cross-Lingual Interleaving for Speech Language Models

TL;DR

This paper introduces a cross-lingual interleaving method for spoken language models that improves multilingual understanding and cross-lingual capabilities without textual supervision, supported by new datasets and benchmarks.

Contribution

The authors propose a novel cross-lingual interleaving technique for speech models and release new multilingual datasets and benchmarks for evaluation.

Findings

Interleaving improves monolingual semantic accuracy.

Enhances cross-lingual continuation robustness.

Strengthens cross-lingual hidden-state alignment.

Abstract

Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.