Enhancing Diffusion Model Guidance through Calibration and Regularization

TL;DR

This paper improves classifier-guided diffusion models by introducing calibration and divergence-aware sampling techniques, leading to better image generation quality and diversity without retraining the diffusion model.

Contribution

It presents a novel calibration method based on Smooth ECE and new guidance strategies that enhance diffusion model performance without retraining the underlying model.

Findings

Achieved an FID of 2.13 on ImageNet 128x128 with ResNet-101 classifier.

Improved classifier calibration with minimal fine-tuning.

Enhanced sampling methods maintain diversity and class guidance effectively.

Abstract

Classifier-guided diffusion models have emerged as a powerful approach for conditional image generation, but they suffer from overconfident predictions during early denoising steps, causing the guidance gradient to vanish. This paper introduces two complementary contributions to address this issue. First, we propose a differentiable calibration objective based on the Smooth Expected Calibration Error (Smooth ECE), which improves classifier calibration with minimal fine-tuning and yields measurable improvements in Frechet Inception Distance (FID). Second, we develop enhanced sampling guidance methods that operate on off-the-shelf classifiers without requiring retraining. These include tilted sampling with batch-level reweighting, adaptive entropy-regularized sampling to preserve diversity, and a novel f-divergence-based sampling strategy that strengthens class-consistent guidance while maintaining mode coverage. Experiments on ImageNet 128x128 demonstrate that our divergence-regularized guidance achieves an FID of 2.13 using a ResNet-101 classifier, improving upon existing classifier-guided diffusion methods while requiring no diffusion model retraining. The results show that principled calibration and divergence-aware sampling provide practical and effective improvements for classifier-guided diffusion.