Video Diffusion Models are Training-free Motion Interpreter and Controller

TL;DR

This paper reveals that video diffusion models inherently encode motion features, introduces a training-free motion control method using these features, and demonstrates its effectiveness in generating natural, controllable video motion.

Contribution

It uncovers the existence of interpretable motion-aware features in video diffusion models and proposes a training-free framework for motion control leveraging these features.

Findings

Motion-aware features are inherently encoded in diffusion models.

The proposed MOFT method enables training-free motion extraction.

The framework achieves competitive results in natural motion generation.

Abstract

Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-based studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present a new MOtion FeaTure (MOFT) by eliminating content correlation information and filtering motion channels. MOFT provides a distinct set of benefits, including the ability to encode comprehensive motion information with clear interpretability, extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free video motion control framework. Our method demonstrates competitive performance in generating natural and faithful motion, providing architecture-agnostic insights and applicability in a variety of downstream tasks.