ATI: Any Trajectory Instruction for Controllable Video Generation

TL;DR

This paper introduces a unified framework for controllable video generation that integrates multiple motion types through trajectory-based inputs, enabling precise and semantically aligned motion control in generated videos.

Contribution

It presents a novel motion control method that projects user-defined trajectories into the latent space of pre-trained models, unifying camera, object, and local motions in a single framework.

Findings

Outperforms prior methods in controllability and visual quality

Supports diverse motion control tasks including stylized effects and viewpoint changes

Compatible with various state-of-the-art video generation models

Abstract

We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion using trajectory-based inputs. In contrast to prior methods that address these motion types through separate modules or task-specific designs, our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models via a lightweight motion injector. Users can specify keypoints and their motion paths to control localized deformations, entire object motion, virtual camera dynamics, or combinations of these. The injected trajectory signals guide the generative process to produce temporally consistent and semantically aligned motion sequences. Our framework demonstrates superior performance across multiple video motion control tasks, including stylized motion effects (e.g., motion brushes), dynamic viewpoint changes, and precise local motion manipulation. Experiments show that our method provides significantly better controllability and visual quality compared to prior approaches and commercial solutions, while remaining broadly compatible with various state-of-the-art video generation backbones. Project page: https://anytraj.github.io/.