Voice Activity Projection Model with Multimodal Encoders

TL;DR

This paper introduces a multimodal voice activity projection model using pre-trained audio and face encoders to better capture social cues, improving turn-taking prediction in human-machine interactions.

Contribution

The paper presents a novel multimodal VAP model with pre-trained encoders, enhancing turn-taking prediction by capturing subtle expressions and outperforming previous models.

Findings

Model performs competitively on turn-taking metrics.

Outperforms previous state-of-the-art models in some cases.

Source code and models are publicly available.

Abstract

Turn-taking management is crucial for any social interaction. Still, it is challenging to model human-machine interaction due to the complexity of the social context and its multimodal nature. Unlike conventional systems based on silence duration, previous existing voice activity projection (VAP) models successfully utilized a unified representation of turn-taking behaviors as prediction targets, which improved turn-taking prediction performance. Recently, a multimodal VAP model outperformed the previous state-of-the-art model by a significant margin. In this paper, we propose a multimodal model enhanced with pre-trained audio and face encoders to improve performance by capturing subtle expressions. Our model performed competitively, and in some cases, even better than state-of-the-art models on turn-taking metrics. All the source codes and pretrained models are available at https://github.com/sagatake/VAPwithAudioFaceEncoders.