StreamFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs

TL;DR

StreamFusion introduces a topology-aware, efficient sequence parallelism framework for distributed diffusion transformer inference on GPUs, overcoming communication bottlenecks and synchronization overheads to significantly improve performance.

Contribution

It proposes a novel topology-aware sequence parallelism method, Torus Attention, and a one-sided communication approach, enhancing distributed inference efficiency for diffusion transformers.

Findings

Achieves up to 1.77x speedup over state-of-the-art methods.

Reduces communication and synchronization overheads in distributed GPU inference.

Demonstrates improved scalability for high-resolution image and video generation.

Abstract

Diffusion Transformers (DiTs) have gained increasing adoption in high-quality image and video generation. As demand for higher-resolution images and longer videos increases, single-GPU inference becomes inefficient due to increased latency and large activation sizes. Current frameworks employ sequence parallelism (SP) techniques such as Ulysses Attention and Ring Attention to scale inference. However, these implementations have three primary limitations: (1) suboptimal communication patterns for network topologies on modern GPU machines, (2) latency bottlenecks from all-to-all operations in inter-machine communication, and (3) GPU sender-receiver synchronization and computation overheads from using two-sided communication libraries. To address these issues, we present StreamFusion, a topology-aware efficient DiT serving engine. StreamFusion incorporates three key innovations: (1) a topology-aware sequence parallelism technique that accounts for inter- and intra-machine bandwidth differences, (2) Torus Attention, a novel SP technique enabling overlapping of inter-machine all-to-all operations with computation, and (3) a one-sided communication implementation that minimizes GPU sender-receiver synchronization and computation overheads. Our experiments demonstrate that StreamFusion outperforms the state-of-the-art approach by an average of $1.35\times$ (up to $1.77\times$).