Beyond Words: Multimodal LLM Knows When to Speak

TL;DR

This paper introduces a multimodal approach for large language models to better determine when to speak during conversations, using synchronized video, audio, and text cues to improve timing accuracy.

Contribution

It presents a novel multimodal strategy and dataset for enhancing LLMs' conversational timing awareness, significantly improving response type prediction performance.

Findings

Up to 3x improvement in response type prediction accuracy.

Multimodal perception enhances conversational naturalness.

A new dataset with aligned modalities and reaction annotations.

Abstract

Chatbots via large language models (LLMs) generate fluent responses but often struggle with when to speak, especially for brief, timely listener reactions during ongoing dialogue. We present a multimodal strategy for LLMs, which leverages synchronized video, audio, and text cues to improve conversational timing awareness. The strategy reformulates response timing as a dense response-type prediction task, enabling an agent to decide whether to remain silent, produce a short reaction, or start a full response under streaming constraints. Therefore, we introduce a curated multimodal dataset from real-world dyadic conversational videos with temporally aligned modalities and fine-grained reaction type annotations. Moreover, we design a multimodal strategy, MM-When2Speak, with a multimodal integration module on top of an LLM backbone. Experiments across various modality settings and strong LLM baselines show that MM-When2Speak achieves up to a 3x improvement in response type prediction performance, highlighting the importance of multimodal perception for natural and engaging conversational interaction.