Performance Plateaus in Inference-Time Scaling for Text-to-Image Diffusion Without External Models

TL;DR

This paper investigates the limits of inference-time scaling for text-to-image diffusion models without external evaluation models, revealing rapid performance plateaus and the efficiency of minimal optimization steps.

Contribution

It demonstrates that inference-time scaling quickly reaches a performance plateau in this setting, highlighting the limited gains from additional optimization steps without external models.

Findings

Performance plateaus are reached quickly during inference-time scaling.

A small number of optimization steps suffices for maximum performance.

External models are not necessary for effective inference-time optimization.

Abstract

Recently, it has been shown that investing computing resources in searching for good initial noise for a text-to-image diffusion model helps improve performance. However, previous studies required external models to evaluate the resulting images, which is impossible on GPUs with small VRAM. For these reasons, we apply Best-of-N inference-time scaling to algorithms that optimize the initial noise of a diffusion model without external models across multiple datasets and backbones. We demonstrate that inference-time scaling for text-to-image diffusion models in this setting quickly reaches a performance plateau, and a relatively small number of optimization steps suffices to achieve the maximum achievable performance with each algorithm.