Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

TL;DR

This paper introduces a real-time turn-taking prediction system using a voice activity projection model that leverages contrastive predictive coding and self-attention transformers to predict future dialogue voice activity from stereo audio.

Contribution

The paper presents a novel VAP model combining CPC and transformers for real-time turn-taking prediction directly from stereo audio.

Findings

System operates in real-time with CPU settings

Minimal performance degradation with varying input context length

Effective voice activity prediction from stereo audio

Abstract

A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.