Text Embedding Knows How to Quantize Text-Guided Diffusion Models

TL;DR

This paper introduces QLIP, a novel quantization method for text-guided diffusion models that uses text prompts to optimize layer precision, significantly reducing computational costs while maintaining image quality.

Contribution

The paper presents a new quantization approach that incorporates text prompts to improve efficiency in language-to-image diffusion models, adaptable to existing methods.

Findings

QLIP reduces computational complexity in diffusion models.

QLIP improves image quality in quantized diffusion models.

QLIP seamlessly integrates with existing quantization techniques.

Abstract

Despite the success of diffusion models in image generation tasks such as text-to-image, the enormous computational complexity of diffusion models limits their use in resource-constrained environments. To address this, network quantization has emerged as a promising solution for designing efficient diffusion models. However, existing diffusion model quantization methods do not consider input conditions, such as text prompts, as an essential source of information for quantization. In this paper, we propose a novel quantization method dubbed Quantization of Language-to-Image diffusion models using text Prompts (QLIP). QLIP leverages text prompts to guide the selection of bit precision for every layer at each time step. In addition, QLIP can be seamlessly integrated into existing quantization methods to enhance quantization efficiency. Our extensive experiments demonstrate the effectiveness of QLIP in reducing computational complexity and improving the quality of the generated images across various datasets.