S2DiT: Sandwich Diffusion Transformer for Mobile Streaming Video Generation

TL;DR

S2DiT is a novel mobile-optimized diffusion transformer that enables real-time streaming video generation with high quality, using efficient attention mechanisms and a sandwich architecture optimized via dynamic programming.

Contribution

The paper introduces S2DiT, a new efficient diffusion transformer architecture with a sandwich design and novel attention mechanisms for mobile streaming video generation.

Findings

Achieves real-time streaming at over 10 FPS on an iPhone.

Maintains high video quality comparable to server models.

Effectively transfers large model capacity to compact mobile models.

Abstract

Diffusion Transformers (DiTs) have recently improved video generation quality. However, their heavy computational cost makes real-time or on-device generation infeasible. In this work, we introduce S2DiT, a Streaming Sandwich Diffusion Transformer designed for efficient, high-fidelity, and streaming video generation on mobile hardware. S2DiT generates more tokens but maintains efficiency with novel efficient attentions: a mixture of LinConv Hybrid Attention (LCHA) and Stride Self-Attention (SSA). Based on this, we uncover the sandwich design via a budget-aware dynamic programming search, achieving superior quality and efficiency. We further propose a 2-in-1 distillation framework that transfers the capacity of large teacher models (e.g., Wan 2.2-14B) to the compact few-step sandwich model. Together, S2DiT achieves quality on par with state-of-the-art server video models, while streaming at over 10 FPS on an iPhone.