AMPS: ASR with Multimodal Paraphrase Supervision

TL;DR

This paper introduces AMPS, a novel multimodal ASR training technique that leverages paraphrase supervision to enhance conversational speech recognition across multiple languages, achieving notable WER reductions.

Contribution

AMPS is the first to incorporate paraphrase-based supervision into a multilingual multimodal ASR system for improved conversational speech recognition.

Findings

Up to 5% relative WER reduction across languages.

Effective use of paraphrases improves recognition accuracy.

Detailed evaluation confirms system robustness.

Abstract

Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition (ASR) systems. In this work, we present a new technique AMPS that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages, including Hindi, Marathi, Malayalam, Kannada, and Nyanja. We use paraphrases of the reference transcriptions as additional supervision while training the multimodal ASR model and selectively invoke this paraphrase objective for utterances with poor ASR performance. Using AMPS with a state-of-the-art multimodal model SeamlessM4T, we obtain significant relative reductions in word error rates (WERs) of up to 5%. We present detailed analyses of our system using both objective and human evaluation metrics.