When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models

TL;DR

This paper investigates the effectiveness of ensembling diffusion models, finding improvements in likelihood but inconsistent gains in perceptual quality, and provides theoretical insights into score model aggregation.

Contribution

It systematically evaluates ensembling methods for diffusion models, revealing their impact on likelihood and quality, and offers theoretical analysis of score model summation techniques.

Findings

Ensembling improves score-matching loss and likelihood.

Ensembling does not consistently improve perceptual quality metrics.

A specific aggregation strategy outperforms others in tabular data.

Abstract

Diffusion models now generate high-quality, diverse samples, with an increasing focus on more powerful models. Although ensembling is a well-known way to improve supervised models, its application to unconditional score-based diffusion models remains largely unexplored. In this work we investigate whether it provides tangible benefits for generative modelling. We find that while ensembling the scores generally improves the score-matching loss and model likelihood, it fails to consistently enhance perceptual quality metrics such as FID on image datasets. We confirm this observation across a breadth of aggregation rules using Deep Ensembles, Monte Carlo Dropout, on CIFAR-10 and FFHQ. We attempt to explain this discrepancy by investigating possible explanations, such as the link between score estimation and image quality. We also look into tabular data through random forests, and find that one aggregation strategy outperforms the others. Finally, we provide theoretical insights into the summing of score models, which shed light not only on ensembling but also on several model composition techniques (e.g. guidance).