One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models

TL;DR

This paper introduces TiUE, a time-independent unified encoder for text-to-image diffusion models that reduces inference time, improves diversity and realism, and maintains high image quality through shared encoder features and regularization.

Contribution

The paper proposes the first time-independent encoder for T2I diffusion models, enabling parallel sampling and improved diversity in one-step generation.

Findings

TiUE outperforms state-of-the-art methods in diversity and realism.

TiUE significantly reduces inference time with shared encoder features.

Incorporating KL divergence improves perceptual quality.

Abstract

Text-to-Image (T2I) diffusion models have made remarkable advancements in generative modeling; however, they face a trade-off between inference speed and image quality, posing challenges for efficient deployment. Existing distilled T2I models can generate high-fidelity images with fewer sampling steps, but often struggle with diversity and quality, especially in one-step models. From our analysis, we observe redundant computations in the UNet encoders. Our findings suggest that, for T2I diffusion models, decoders are more adept at capturing richer and more explicit semantic information, while encoders can be effectively shared across decoders from diverse time steps. Based on these observations, we introduce the first Time-independent Unified Encoder TiUE for the student model UNet architecture, which is a loop-free image generation approach for distilling T2I diffusion models. Using a one-pass scheme, TiUE shares encoder features across multiple decoder time steps, enabling parallel sampling and significantly reducing inference time complexity. In addition, we incorporate a KL divergence term to regularize noise prediction, which enhances the perceptual realism and diversity of the generated images. Experimental results demonstrate that TiUE outperforms state-of-the-art methods, including LCM, SD-Turbo, and SwiftBrushv2, producing more diverse and realistic results while maintaining the computational efficiency.