Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

TL;DR

This paper introduces a large-scale multimodal dataset and a novel gated fusion model with contrastive learning to improve turn-taking prediction in human-robot dialogue, addressing data imbalance and modality integration challenges.

Contribution

It presents a new large-scale dataset, a gated multimodal fusion mechanism, and a contrastive learning approach to enhance turn-taking prediction in dialogue systems.

Findings

The proposed model outperforms several state-of-the-art baselines.

Contrastive learning improves feature representations for turn-taking.

Data augmentation effectively addresses class imbalance.

Abstract

Turn-taking, aiming to decide when the next speaker can start talking, is an essential component in building human-robot spoken dialogue systems. Previous studies indicate that multimodal cues can facilitate this challenging task. However, due to the paucity of public multimodal datasets, current methods are mostly limited to either utilizing unimodal features or simplistic multimodal ensemble models. Besides, the inherent class imbalance in real scenario, e.g. sentence ending with short pause will be mostly regarded as the end of turn, also poses great challenge to the turn-taking decision. In this paper, we first collect a large-scale annotated corpus for turn-taking with over 5,000 real human-robot dialogues in speech and text modalities. Then, a novel gated multimodal fusion mechanism is devised to utilize various information seamlessly for turn-taking prediction. More importantly, to tackle the data imbalance issue, we design a simple yet effective data augmentation method to construct negative instances without supervision and apply contrastive learning to obtain better feature representations. Extensive experiments are conducted and the results demonstrate the superiority and competitiveness of our model over several state-of-the-art baselines.