A Unified Framework for Motion Reasoning and Generation in Human Interaction

TL;DR

This paper introduces VIM, a unified model that integrates language and motion understanding to generate and control interactive human motions in multi-turn conversations, supported by a large-scale dataset Inter-MT2.

Contribution

The paper presents VIM, a novel unified architecture for simultaneous motion and language processing, and introduces Inter-MT2, a large-scale dataset for interactive motion instruction tuning.

Findings

VIM effectively handles multiple interactive motion tasks.

Inter-MT2 enables training of versatile motion-language models.

VIM demonstrates strong performance across diverse motion understanding and generation tasks.

Abstract

Recent advancements in large language models (LLMs) have significantly improved their ability to generate natural and contextually relevant text, enabling more human-like AI interactions. However, generating and understanding interactive human-like motion, where multiple individuals engage in coordinated movements, remains challenging due to the complexity of modeling these interactions. Additionally, a unified and versatile model is needed to handle diverse interactive scenarios, such as chat systems that dynamically adapt to user instructions and assigned roles. To address these challenges, we introduce VIM, the Versatile Interactive Motion-language model, which integrates both language and motion modalities to effectively understand, generate, and control interactive motions in multi-turn conversational contexts. Unlike previous studies that primarily focus on uni-directional tasks such as text-to-motion or motion-to-text, VIM employs a unified architecture capable of simultaneously understanding and generating both motion and text modalities. Given the absence of an appropriate dataset to support this task, we introduce Inter-MT2, a large-scale instruction-tuning dataset containing 82.7K multi-turn interactive motion instructions, covering 153K interactive motion samples. Inter-MT2 spans diverse instructional scenarios, including motion editing, question answering, and story generation, leveraging off-the-shelf large language models and motion diffusion models to construct a broad set of interactive motion instructions. We extensively evaluate the versatility of VIM across multiple interactive motion-related tasks, including motion-to-text, text-to-motion, reaction generation, motion editing, and reasoning about motion sequences.