IKMo: Image-Keyframed Motion Generation with Trajectory-Pose Conditioned Motion Diffusion Model

TL;DR

IKMo introduces a decoupled, diffusion-based human motion generation method utilizing trajectory and pose inputs, enhanced by a two-stage conditioning framework and MLLM-based pre-processing, achieving superior fidelity and controllability.

Contribution

The paper presents IKMo, a novel motion generation approach that decouples trajectory and pose inputs with a two-stage conditioning framework and integrates MLLM-based agents for improved control and realism.

Findings

Outperforms state-of-the-art on HumanML3D and KIT-ML datasets.

MLLM-based agents improve user satisfaction and motion alignment.

Enhanced motion fidelity and controllability demonstrated.

Abstract

Existing human motion generation methods with trajectory and pose inputs operate global processing on both modalities, leading to suboptimal outputs. In this paper, we propose IKMo, an image-keyframed motion generation method based on the diffusion model with trajectory and pose being decoupled. The trajectory and pose inputs go through a two-stage conditioning framework. In the first stage, the dedicated optimization module is applied to refine inputs. In the second stage, trajectory and pose are encoded via a Trajectory Encoder and a Pose Encoder in parallel. Then, motion with high spatial and semantic fidelity is guided by a motion ControlNet, which processes the fused trajectory and pose data. Experiment results based on HumanML3D and KIT-ML datasets demonstrate that the proposed method outperforms state-of-the-art on all metrics under trajectory-keyframe constraints. In addition, MLLM-based agents are implemented to pre-process model inputs. Given texts and keyframe images from users, the agents extract motion descriptions, keyframe poses, and trajectories as the optimized inputs into the motion generation model. We conducts a user study with 10 participants. The experiment results prove that the MLLM-based agents pre-processing makes generated motion more in line with users' expectation. We believe that the proposed method improves both the fidelity and controllability of motion generation by the diffusion model.