Bigger is not Always Better: Scaling Properties of Latent Diffusion Models

TL;DR

This paper investigates how the size of latent diffusion models affects sampling efficiency, revealing that smaller models often outperform larger ones under limited inference budgets, challenging common assumptions about model scaling.

Contribution

It provides an empirical analysis of model size impact on sampling efficiency in LDMs and demonstrates the generalizability of these findings across various tasks and samplers.

Findings

Smaller models can outperform larger ones at the same inference budget.

Model size has a complex, non-linear effect on sampling efficiency.

Findings are consistent across different diffusion samplers and downstream tasks.

Abstract

We study the scaling properties of latent diffusion models (LDMs) with an emphasis on their sampling efficiency. While improved network architecture and inference algorithms have shown to effectively boost sampling efficiency of diffusion models, the role of model size -- a critical determinant of sampling efficiency -- has not been thoroughly examined. Through empirical analysis of established text-to-image diffusion models, we conduct an in-depth investigation into how model size influences sampling efficiency across varying sampling steps. Our findings unveil a surprising trend: when operating under a given inference budget, smaller models frequently outperform their larger equivalents in generating high-quality results. Moreover, we extend our study to demonstrate the generalizability of the these findings by applying various diffusion samplers, exploring diverse downstream tasks, evaluating post-distilled models, as well as comparing performance relative to training compute. These findings open up new pathways for the development of LDM scaling strategies which can be employed to enhance generative capabilities within limited inference budgets.