LCM-LoRA: A Universal Stable-Diffusion Acceleration Module

TL;DR

This paper introduces LCM-LoRA, a universal acceleration module for Stable-Diffusion models, leveraging Latent Consistency Models and LoRA distillation to improve image generation efficiency and quality across various models without additional training.

Contribution

The paper extends Latent Consistency Models with LoRA distillation to larger models and introduces LCM-LoRA as a universal, plug-in accelerator for diverse Stable-Diffusion tasks.

Findings

LCM-LoRA achieves superior image quality with less memory.

It can be directly plugged into various models without retraining.

It outperforms previous numerical PF-ODE solvers like DDIM and DPM-Solver.

Abstract

Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps. LCMs are distilled from pre-trained latent diffusion models (LDMs), requiring only ~32 A100 GPU training hours. This report further extends LCMs' potential in two aspects: First, by applying LoRA distillation to Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded LCM's scope to larger models with significantly less memory consumption, achieving superior image generation quality. Second, we identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into various Stable-Diffusion fine-tuned models or LoRAs without training, thus representing a universally applicable accelerator for diverse image generation tasks. Compared with previous numerical PF-ODE solvers such as DDIM, DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that possesses strong generalization abilities. Project page: https://github.com/luosiallen/latent-consistency-model.