Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024

TL;DR

This paper explores integrating large language models into cascaded speech translation systems, improving transcription and translation accuracy on standard test sets, but facing challenges in noisy, multi-speaker scenarios.

Contribution

It introduces a method to incorporate LLMs into cascaded speech translation, refining ASR and MT outputs through fine-tuning on N-best lists and document-level data.

Findings

0.3% absolute WER improvement on tst2019

0.65% COMET score increase on tst2019

Limited benefits in noisy, multi-speaker environments

Abstract

Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT's offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B\footnote{mistralai/Mistral-7B-Instruct-v0.1} into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of $0.3\%$ in Word Error Rate and $0.65\%$ in COMET for tst2019 test set. In challenging test sets with overlapping speakers and background noise, we find that integrating LLM is not beneficial due to poor ASR performance. Here, we use ASR with chunked long-form decoding to improve context usage that may be unavailable when transcribing with Voice Activity Detection segmentation alone.