Tora: Trajectory-oriented Diffusion Transformer for Video Generation

TL;DR

Tora introduces a trajectory-oriented diffusion transformer framework that enables controllable, high-fidelity video generation with precise motion guidance by integrating textual, visual, and trajectory conditions.

Contribution

It is the first to incorporate trajectory-based motion control into diffusion transformers for scalable, high-quality video synthesis.

Findings

Tora achieves higher motion fidelity than baseline DiT models.

It accurately simulates complex physical movements.

The framework supports diverse durations, aspect ratios, and resolutions.

Abstract

Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that concurrently integrates textual, visual, and trajectory conditions, thereby enabling scalable video generation with effective motion guidance. Specifically, Tora consists of a Trajectory Extractor (TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D motion compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos that accurately follow designated trajectories. Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate that Tora excels in achieving high motion fidelity compared to the foundational DiT model, while also accurately simulating the complex movements of the physical world. Code is made available at https://github.com/alibaba/Tora .