OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

TL;DR

This paper introduces OmniResponse, a novel multimodal LLM for generating synchronized verbal and non-verbal listener responses in dyadic interactions, advancing online conversational AI with new datasets and evaluation benchmarks.

Contribution

The paper presents OmniResponse, a multimodal LLM with novel components for synchronized response generation, and introduces ResponseNet, a new dataset for training and evaluating such models.

Findings

OmniResponse outperforms baselines in content accuracy and synchronization.

ResponseNet enables detailed evaluation of multimodal response generation.

The approach effectively aligns speech, facial responses, and text in dyadic interactions.

Abstract

In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Text Markup, which precisely timestamps generated text tokens, and TempoVoice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Comprehensive evaluations on ResponseNet demonstrate that OmniResponse outperforms baseline models in terms of semantic speech content, audio-visual synchronization, and generation quality. Our dataset, code, and models are publicly available.