Cost-Aware Routing for Efficient Text-To-Image Generation

TL;DR

This paper introduces a cost-aware routing framework for text-to-image diffusion models that dynamically allocates computational resources based on prompt complexity, balancing image quality and efficiency.

Contribution

It proposes a novel routing approach that selects among multiple models or denoising steps to optimize quality and cost for each prompt, outperforming single-model methods.

Findings

Achieves higher average image quality than individual models on COCO and DiffusionDB.

Effectively reserves high-cost computation for complex prompts, saving resources on simpler ones.

Demonstrates the benefit of multi-model routing over uniform cost reduction techniques.

Abstract

Diffusion models are well known for their ability to generate a high-fidelity image for an input prompt through an iterative denoising process. Unfortunately, the high fidelity also comes at a high computational cost due the inherently sequential generative process. In this work, we seek to optimally balance quality and computational cost, and propose a framework to allow the amount of computation to vary for each prompt, depending on its complexity. Each prompt is automatically routed to the most appropriate text-to-image generation function, which may correspond to a distinct number of denoising steps of a diffusion model, or a disparate, independent text-to-image model. Unlike uniform cost reduction techniques (e.g., distillation, model quantization), our approach achieves the optimal trade-off by learning to reserve expensive choices (e.g., 100+ denoising steps) only for a few complex prompts, and employ more economical choices (e.g., small distilled model) for less sophisticated prompts. We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine already-trained text-to-image models, our approach is able to deliver an average quality that is higher than that achievable by any of these models alone.