SDiT: Semantic Region-Adaptive for Diffusion Transformers

TL;DR

SDiT is a novel diffusion transformer method that adaptively allocates computational resources based on regional complexity, significantly accelerating text-to-image synthesis without sacrificing quality.

Contribution

It introduces a training-free, region-adaptive framework combining semantic clustering, complexity-driven scheduling, and boundary refinement for efficient diffusion transformers.

Findings

Achieves up to 3.0x speedup in inference

Maintains high perceptual and semantic quality

Operates without model retraining or architectural changes

Abstract

Diffusion Transformers (DiTs) achieve state-of-the-art performance in text-to-image synthesis but remain computationally expensive due to the iterative nature of denoising and the quadratic cost of global attention. In this work, we observe that denoising dynamics are spatially non-uniform-background regions converge rapidly while edges and textured areas evolve much more actively. Building on this insight, we propose SDiT, a Semantic Region-Adaptive Diffusion Transformer that allocates computation according to regional complexity. SDiT introduces a training-free framework combining (1) semantic-aware clustering via fast Quickshift-based segmentation, (2) complexity-driven regional scheduling to selectively update informative areas, and (3) boundary-aware refinement to maintain spatial coherence. Without any model retraining or architectural modification, SDiT achieves up to 3.0x acceleration while preserving nearly identical perceptual and semantic quality to full-attention inference.