FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models

TL;DR

FreeTraj introduces a tuning-free method for trajectory-controllable video generation using diffusion models, by guiding noise and attention without additional training, enabling flexible and extended video creation.

Contribution

We propose FreeTraj, a novel tuning-free framework that controls video trajectories by modifying noise sampling and attention, eliminating the need for training-based methods.

Findings

Effective trajectory control demonstrated in experiments

Enables longer and larger video generation with controllable trajectories

Flexible trajectory input via manual or automatic planning

Abstract

Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomenons and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models.