Amber-Image: Efficient Compression of Large-Scale Diffusion Transformers

TL;DR

This paper introduces Amber-Image, a highly efficient compression framework for large diffusion transformers, enabling lightweight T2I models with minimal training and computational costs while maintaining high quality.

Contribution

The paper presents a novel compression method that reduces model size by 70% and enables cost-effective training of high-performance diffusion models without large-scale data engineering.

Findings

Amber-Image-10B uses depth pruning and distillation to compress models.

Amber-Image-6B employs a hybrid architecture with progressive distillation.

Models achieve high-fidelity synthesis comparable to larger models.

Abstract

Diffusion Transformer (DiT) architectures have significantly advanced Text-to-Image (T2I) generation but suffer from prohibitive computational costs and deployment barriers. To address these challenges, we propose an efficient compression framework that transforms the 60-layer dual-stream MMDiT-based Qwen-Image into lightweight models without training from scratch. Leveraging this framework, we introduce Amber-Image, a series of streamlined T2I models. We first derive Amber-Image-10B using a timestep-sensitive depth pruning strategy, where retained layers are reinitialized via local weight averaging and optimized through layer-wise distillation and full-parameter fine-tuning. Building on this, we develop Amber-Image-6B by introducing a hybrid-stream architecture that converts deep-layer dual streams into a single stream initialized from the image branch, further refined via progressive distillation and lightweight fine-tuning. Our approach reduces parameters by 70% and eliminates the need for large-scale data engineering. Notably, the entire compression and training pipeline-from the 10B to the 6B variant-requires fewer than 2,000 GPU hours, demonstrating exceptional cost-efficiency compared to training from scratch. Extensive evaluations on benchmarks like DPG-Bench and LongText-Bench show that Amber-Image achieves high-fidelity synthesis and superior text rendering, matching much larger models.