Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment

TL;DR

This paper introduces SPECTRA, a novel speech-text dialog pre-training model that effectively captures cross-modal alignment and dialog context, improving performance across various spoken dialog understanding tasks.

Contribution

SPECTRA is the first speech-text dialog pre-training model that incorporates explicit cross-modal alignment and contextual dialog understanding.

Findings

Outperforms existing models on four speech-text tasks.

Effectively captures speech-text temporal alignment.

Enhances multi-turn dialog comprehension.

Abstract

Recently, speech-text pre-training methods have shown remarkable success in many speech and natural language processing tasks. However, most previous pre-trained models are usually tailored for one or two specific tasks, but fail to conquer a wide range of speech-text tasks. In addition, existing speech-text pre-training methods fail to explore the contextual information within a dialogue to enrich utterance representations. In this paper, we propose Speech-text dialog Pre-training for spoken dialog understanding with ExpliCiT cRoss-Modal Alignment (SPECTRA), which is the first-ever speech-text dialog pre-training model. Concretely, to consider the temporality of speech modality, we design a novel temporal position prediction task to capture the speech-text alignment. This pre-training task aims to predict the start and end time of each textual word in the corresponding speech waveform. In addition, to learn the characteristics of spoken dialogs, we generalize a response selection task from textual dialog pre-training to speech-text dialog pre-training scenarios. Experimental results on four different downstream speech-text tasks demonstrate the superiority of SPECTRA in learning speech-text alignment and multi-turn dialog context.