PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs

TL;DR

PROST-LLM introduces a progressive fine-tuning and optimization framework to significantly improve speech-to-speech translation capabilities in large language models, addressing data scarcity issues through self-sampling and preference learning.

Contribution

It presents a novel multi-stage training approach combining tri-task learning, self-sampling, and preference optimization to enhance S2ST in LLMs.

Findings

Significant improvement in S2ST performance demonstrated

Effective use of self-sampling and back-translation for data augmentation

Enhanced translation quality confirmed through extensive experiments

Abstract

Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs.