Instituto de Telecomunica\c{c}\~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning

TL;DR

This paper introduces a unified speech-to-text model for speech recognition, translation, and spoken question answering, emphasizing small-scale models, high-quality data, and instruction fine-tuning for improved speech processing tasks.

Contribution

It presents a novel approach combining a pre-trained speech encoder and text decoder with modality alignment and instruction fine-tuning, using small-scale models and high-quality data.

Findings

Effective speech-to-text performance on IWSLT 2025 tasks

Successful integration of modality alignment and instruction fine-tuning

Utilization of small-scale models with high-quality data enhances results

Abstract

This paper presents the IT-IST submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pre-trained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources.