U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation

TL;DR

U-Mind is a comprehensive real-time multimodal interaction system that integrates language, speech, motion, and video synthesis, enabling natural and synchronized communication for intelligent agents.

Contribution

It introduces a unified framework with novel alignment and reasoning strategies for high-quality, real-time multimodal dialogue and interaction.

Findings

Achieves state-of-the-art results on multimodal interaction tasks

Demonstrates effective cross-modal synchronization and reasoning

Enables expressive, synchronized visual feedback in real-time

Abstract

Full-stack multimodal interaction in real-time is a central goal in building intelligent embodied agents capable of natural, dynamic communication. However, existing systems are either limited to unimodal generation or suffer from degraded reasoning and poor cross-modal alignment, preventing coherent and perceptually grounded interactions. In this work, we introduce U-Mind, the first unified system for high-intelligence multimodal dialogue that supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop. At its core, U-Mind implements a Unified Alignment and Reasoning Framework that addresses two key challenges: enhancing cross-modal synchronization via a segment-wise alignment strategy, and preserving reasoning abilities through Rehearsal-Driven Learning. During inference, U-Mind adopts a text-first decoding pipeline that performs internal chain-of-thought planning followed by temporally synchronized generation across modalities. To close the loop, we implement a real-time video rendering framework conditioned on pose and speech, enabling expressive and synchronized visual feedback. Extensive experiments demonstrate that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks, including question answering, instruction following, and motion generation, paving the way toward intelligent, immersive conversational agents.