TLCM: Training-efficient Latent Consistency Model for Image Generation with 2-8 Steps

TL;DR

TLCM is a training-efficient, data-free latent consistency model that accelerates image generation with 2-8 inference steps, maintaining high quality and flexibility, and surpassing existing models in various benchmarks.

Contribution

The paper introduces a novel data-free, multi-step latent consistency distillation method that significantly reduces training time and inference steps while preserving image quality.

Findings

Achieves high-quality image generation with only 2-8 steps.

Surpasses existing accelerated models in CLIP and aesthetic scores.

Demonstrates versatility in style transfer and controllable generation.

Abstract

Distilling latent diffusion models (LDMs) into ones that are fast to sample from is attracting growing research interest. However, the majority of existing methods face two critical challenges: (1) They hinge on long training using a huge volume of real data. (2) They routinely lead to quality degradation for generation, especially in text-image alignment. This paper proposes a novel training-efficient Latent Consistency Model (TLCM) to overcome these challenges. Our method first accelerates LDMs via data-free multistep latent consistency distillation (MLCD), and then data-free latent consistency distillation is proposed to efficiently guarantee the inter-segment consistency in MLCD. Furthermore, we introduce bags of techniques, e.g., distribution matching, adversarial learning, and preference learning, to enhance TLCM's performance at few-step inference without any real data. TLCM demonstrates a high level of flexibility by enabling adjustment of sampling steps within the range of 2 to 8 while still producing competitive outputs compared to full-step approaches. Notably, TLCM enjoys the data-free merit by employing synthetic data from the teacher for distillation. With just 70 training hours on an A100 GPU, a 3-step TLCM distilled from SDXL achieves an impressive CLIP Score of 33.68 and an Aesthetic Score of 5.97 on the MSCOCO-2017 5K benchmark, surpassing various accelerated models and even outperforming the teacher model in human preference metrics. We also demonstrate the versatility of TLCMs in applications including image style transfer, controllable generation, and Chinese-to-image generation.