Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios

TL;DR

This paper introduces a phoneme-augmented Chain-of-Thought framework for speech-to-text translation, significantly improving low-resource and zero-resource language translation by leveraging phoneme recognition and curriculum learning.

Contribution

It presents a novel integration of phoneme representations into a multilingual CoT framework, enhancing cross-lingual transfer in low-resource scenarios.

Findings

Improves translation quality in low-resource settings.

Enables zero-resource translation for unseen languages.

Slightly reduces performance on high-resource languages.

Abstract

We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.