Beyond and Free from Diffusion: Invertible Guided Consistency Training

TL;DR

This paper introduces invertible Guided Consistency Training (iGCT), a data-driven framework for guided image generation that eliminates the need for costly diffusion model distillation, improving efficiency and quality.

Contribution

iGCT is the first training method enabling guided consistency models to operate without diffusion models, reducing compute costs and addressing guidance saturation artifacts.

Findings

iGCT significantly improves FID and precision over CFG.

At guidance scale 13, iGCT achieves 0.8 precision, outperforming diffusion models.

iGCT enables guidance and inversion in CMs without relying on DMs.

Abstract

Guidance in image generation steers models towards higher-quality or more targeted outputs, typically achieved in Diffusion Models (DMs) via Classifier-free Guidance (CFG). However, recent Consistency Models (CMs), which offer fewer function evaluations, rely on distilling CFG knowledge from pretrained DMs to achieve guidance, making them costly and inflexible. In this work, we propose invertible Guided Consistency Training (iGCT), a novel training framework for guided CMs that is entirely data-driven. iGCT, as a pioneering work, contributes to fast and guided image generation and editing without requiring the training and distillation of DMs, greatly reducing the overall compute requirements. iGCT addresses the saturation artifacts seen in CFG under high guidance scales. Our extensive experiments on CIFAR-10 and ImageNet64 show that iGCT significantly improves FID and precision compared to CFG. At a guidance of 13, iGCT improves precision to 0.8, while DM's drops to 0.47. Our work takes the first step toward enabling guidance and inversion for CMs without relying on DMs.