Breaking the Likelihood-Quality Trade-off in Diffusion Models by Merging Pretrained Experts

TL;DR

This paper presents a plug-and-play method that merges two pretrained diffusion models by switching between them at different noise levels, improving both image quality and likelihood without retraining.

Contribution

It introduces a simple expert switching technique that combines pretrained diffusion models at different noise levels to overcome the likelihood-quality trade-off.

Findings

Merged models outperform individual experts on CIFAR-10 and ImageNet32.

The approach improves or maintains both likelihood and visual quality.

No retraining or fine-tuning required for the expert switching method.

Abstract

Diffusion models for image generation often exhibit a trade-off between perceptual sample quality and data likelihood: training objectives emphasizing high-noise denoising steps yield realistic images but poor likelihoods, whereas likelihood-oriented training overweights low-noise steps and harms visual fidelity. We introduce a simple plug-and-play sampling method that combines two pretrained diffusion experts by switching between them along the denoising trajectory. Specifically, we apply an image-quality expert at high noise levels to shape global structure, then switch to a likelihood expert at low noise levels to refine pixel statistics. The approach requires no retraining or fine-tuning -- only the choice of an intermediate switching step. On CIFAR-10 and ImageNet32, the merged model consistently matches or outperforms its base components, improving or preserving both likelihood and sample quality relative to each expert alone. These results demonstrate that expert switching across noise levels is an effective way to break the likelihood-quality trade-off in image diffusion models.