FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models

TL;DR

This paper introduces FreeMotion, a novel approach for human motion synthesis that leverages multimodal large language models to generate diverse motions from natural language instructions without using motion capture data.

Contribution

The paper pioneers mocap-free human motion synthesis using MLLMs for open-set tasks, enabling general motion generation from natural language without prior motion data.

Findings

Effective generation of human motions from language instructions.

Supports a wide range of motion categories and environments.

Demonstrates promising results in general human motion synthesis.

Abstract

Human motion synthesis is a fundamental task in computer animation. Despite recent progress in this field utilizing deep learning and motion capture data, existing methods are always limited to specific motion categories, environments, and styles. This poor generalizability can be partially attributed to the difficulty and expense of collecting large-scale and high-quality motion data. At the same time, foundation models trained with internet-scale image and text data have demonstrated surprising world knowledge and reasoning ability for various downstream tasks. Utilizing these foundation models may help with human motion synthesis, which some recent works have superficially explored. However, these methods didn't fully unveil the foundation models' potential for this task and only support several simple actions and environments. In this paper, we for the first time, without any motion data, explore open-set human motion synthesis using natural language instructions as user control signals based on MLLMs across any motion task and environment. Our framework can be split into two stages: 1) sequential keyframe generation by utilizing MLLMs as a keyframe designer and animator; 2) motion filling between keyframes through interpolation and motion tracking. Our method can achieve general human motion synthesis for many downstream tasks. The promising results demonstrate the worth of mocap-free human motion synthesis aided by MLLMs and pave the way for future research.