A novel multimodal dynamic fusion network for disfluency detection in spoken utterances

TL;DR

This paper introduces a multimodal dynamic fusion network that effectively combines speech and text cues to improve disfluency detection in spoken language, outperforming existing models and reducing spurious correlations.

Contribution

It presents a novel multimodal architecture that integrates acoustic and prosodic cues with minimal additional parameters, achieving state-of-the-art results in disfluency detection.

Findings

Achieves state-of-the-art performance on English Switchboard dataset.

Outperforms prior unimodal and multimodal systems significantly.

Overcomes spurious correlation issues present in text-only systems.

Abstract

Disfluency, though originating from human spoken utterances, is primarily studied as a uni-modal text-based Natural Language Processing (NLP) task. Based on early-fusion and self-attention-based multimodal interaction between text and acoustic modalities, in this paper, we propose a novel multimodal architecture for disfluency detection from individual utterances. Our architecture leverages a multimodal dynamic fusion network that adds minimal parameters over an existing text encoder commonly used in prior art to leverage the prosodic and acoustic cues hidden in speech. Through experiments, we show that our proposed model achieves state-of-the-art results on the widely used English Switchboard for disfluency detection and outperforms prior unimodal and multimodal systems in literature by a significant margin. In addition, we make a thorough qualitative analysis and show that, unlike text-only systems, which suffer from spurious correlations in the data, our system overcomes this problem through additional cues from speech signals. We make all our codes publicly available on GitHub.