Tora2: Motion and Appearance Customized Diffusion Transformer for Multi-Entity Video Generation

TL;DR

Tora2 is a novel diffusion transformer model that enables simultaneous customization of appearance and motion for multiple entities in video generation, improving detail preservation and multimodal alignment.

Contribution

It introduces a decoupled personalization extractor, gated self-attention, and a contrastive loss, enabling multi-entity appearance and motion customization in video synthesis.

Findings

Achieves competitive performance with state-of-the-art methods.

Enhances motion control capabilities in multi-entity video generation.

Improves visual detail preservation and multimodal alignment.

Abstract

Recent advances in diffusion transformer models for motion-guided video generation, such as Tora, have shown significant progress. In this paper, we present Tora2, an enhanced version of Tora, which introduces several design improvements to expand its capabilities in both appearance and motion customization. Specifically, we introduce a decoupled personalization extractor that generates comprehensive personalization embeddings for multiple open-set entities, better preserving fine-grained visual details compared to previous methods. Building on this, we design a gated self-attention mechanism to integrate trajectory, textual description, and visual information for each entity. This innovation significantly reduces misalignment in multimodal conditioning during training. Moreover, we introduce a contrastive loss that jointly optimizes trajectory dynamics and entity consistency through explicit mapping between motion and personalization embeddings. Tora2 is, to our best knowledge, the first method to achieve simultaneous multi-entity customization of appearance and motion for video generation. Experimental results demonstrate that Tora2 achieves competitive performance with state-of-the-art customization methods while providing advanced motion control capabilities, which marks a critical advancement in multi-condition video generation. Project page: https://ali-videoai.github.io/Tora2_page/.