Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

TL;DR

Motion-Agent introduces a conversational framework utilizing a pre-trained language model to generate, edit, and understand 3D human motions interactively, achieving high performance with minimal fine-tuning.

Contribution

The paper presents Motion-Agent, a novel framework that leverages a pre-trained language model for versatile human motion generation and editing without extensive training.

Findings

Performance comparable to diffusion models with only 1-3% fine-tuning

Enables complex motion generation through multi-turn conversations with GPT-4

Supports a wide range of motion-language tasks interactively

Abstract

While previous approaches to 3D human motion generation have achieved notable success, they often rely on extensive training and are limited to specific tasks. To address these challenges, we introduce Motion-Agent, an efficient conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1--3\% of the model's parameters fine-tuned using adapters, MotionLLM delivers performance on par with diffusion models and other transformer-based methods trained from scratch. By integrating MotionLLM with GPT-4 without additional training, Motion-Agent is able to generate highly complex motion sequences through multi-turn conversations, a capability that previous models have struggled to achieve. Motion-Agent supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges. Project page: https://knoxzhao.github.io/Motion-Agent