FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia and, Ziwei Liu

TL;DR
FreeTraj introduces a tuning-free method for trajectory-controllable video generation using diffusion models, by guiding noise and attention without additional training, enabling flexible and extended video creation.
Contribution
We propose FreeTraj, a novel tuning-free framework that controls video trajectories by modifying noise sampling and attention, eliminating the need for training-based methods.
Findings
Effective trajectory control demonstrated in experiments
Enables longer and larger video generation with controllable trajectories
Flexible trajectory input via manual or automatic planning
Abstract
Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomenons and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Coding and Compression Technologies
MethodsSoftmax · Attention Is All You Need · OPT · Focus · Diffusion
