Multilingual and Continuous Backchannel Prediction: A Cross-lingual Study

TL;DR

This study develops a multilingual, Transformer-based backchannel prediction model for Japanese, English, and Chinese, revealing language-specific cues and demonstrating cross-lingual differences in timing behavior through extensive experiments.

Contribution

It introduces a unified multilingual backchannel prediction model that captures language-universal and language-specific cues, and provides empirical cross-linguistic insights into backchannel timing.

Findings

Multilingual model matches or surpasses monolingual baselines.

Japanese relies more on short-term linguistic cues, while English and Chinese focus on silence and prosody.

The model operates in real-time with CPU-only inference.

Abstract

We present a multilingual, continuous backchannel prediction model for Japanese, English, and Chinese, and use it to investigate cross-linguistic timing behavior. The model is Transformer-based and operates at the frame level, jointly trained with auxiliary tasks on approximately 300 hours of dyadic conversations. Across all three languages, the multilingual model matches or surpasses monolingual baselines, indicating that it learns both language-universal cues and language-specific timing patterns. Zero-shot transfer with two-language training remains limited, underscoring substantive cross-lingual differences. Perturbation analyses reveal distinct cue usage: Japanese relies more on short-term linguistic information, whereas English and Chinese are more sensitive to silence duration and prosodic variation; multilingual training encourages shared yet adaptable representations and reduces overreliance on pitch in Chinese. A context-length study further shows that Japanese is relatively robust to shorter contexts, while Chinese benefits markedly from longer contexts. Finally, we integrate the trained model into a real-time processing software, demonstrating CPU-only inference. Together, these findings provide a unified model and empirical evidence for how backchannel timing differs across languages, informing the design of more natural, culturally-aware spoken dialogue systems.