Accelerating Parallel Diffusion Model Serving with Residual Compression

TL;DR

CompactFusion is a novel compression framework that reduces communication overhead in parallel diffusion model inference by transmitting only compressed residuals, enabling faster and higher-quality image and video generation.

Contribution

It introduces Residual Compression for diffusion activations, effectively removing redundancy and supporting scalable, communication-efficient parallel inference without pipeline rework.

Findings

Achieves 3.0x speedup on 4xL20 with high fidelity

Supports sequence parallelism on slow networks, reaching 6.7x speedup

Maintains high generation quality while significantly reducing data transfer

Abstract

Diffusion models produce realistic images and videos but require substantial computational resources, necessitating multi-accelerator parallelism for real-time deployment. However, parallel inference introduces significant communication overhead from exchanging large activations between devices, limiting efficiency and scalability. We present CompactFusion, a compression framework that significantly reduces communication while preserving generation quality. Our key observation is that diffusion activations exhibit strong temporal redundancy-adjacent steps produce highly similar activations, saturating bandwidth with near-duplicate data carrying little new information. To address this inefficiency, we seek a more compact representation that encodes only the essential information. CompactFusion achieves this via Residual Compression that transmits only compressed residuals (step-wise activation differences). Based on empirical analysis and theoretical justification, we show that it effectively removes redundant data, enabling substantial data reduction while maintaining high fidelity. We also integrate lightweight error feedback to prevent error accumulation. CompactFusion establishes a new paradigm for parallel diffusion inference, delivering lower latency and significantly higher generation quality than prior methods. On 4xL20, it achieves 3.0x speedup while greatly improving fidelity. It also uniquely supports communication-heavy strategies like sequence parallelism on slow networks, achieving 6.7x speedup over prior overlap-based method. CompactFusion applies broadly across diffusion models and parallel settings, and integrates easily without requiring pipeline rework. Portable implementation demonstrated on xDiT is publicly available at https://github.com/Cobalt-27/CompactFusion