Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers

TL;DR

SPRINT introduces a token dropping method for diffusion transformers that maintains high quality while significantly reducing training and inference costs through a novel fusion of shallow and deep layer outputs.

Contribution

It proposes a simple, effective token dropping technique with residual fusion and a two-stage training schedule to improve efficiency of diffusion transformers.

Findings

Achieves 9.8x training savings on ImageNet-1K 256x256.

Nearly halves FLOPs during inference with Path-Drop Guidance.

Maintains comparable image quality metrics.

Abstract

Diffusion Transformers (DiTs) deliver state-of-the-art generative performance but their quadratic training cost with sequence length makes large-scale pretraining prohibitively expensive. Token dropping can reduce training cost, yet na\"ive strategies degrade representations, and existing methods are either parameter-heavy or fail at high drop ratios. We present SPRINT, Sparse--Dense Residual Fusion for Efficient Diffusion Transformers, a simple method that enables aggressive token dropping (up to 75%) while preserving quality. SPRINT leverages the complementary roles of shallow and deep layers: early layers process all tokens to capture local detail, deeper layers operate on a sparse subset to cut computation, and their outputs are fused through residual connections. Training follows a two-stage schedule: long masked pre-training for efficiency followed by short full-token fine-tuning to close the train--inference gap. On ImageNet-1K 256x256, SPRINT achieves 9.8x training savings with comparable FID/FDD, and at inference, its Path-Drop Guidance (PDG) nearly halves FLOPs while improving quality. These results establish SPRINT as a simple, effective, and general solution for efficient DiT training.