Prompt-Aware Scheduling for Efficient Text-to-Image Inferencing System

TL;DR

This paper presents a prompt-aware scheduling system for text-to-image models that optimally balances quality and efficiency under high load conditions by matching prompts to models at different approximation levels.

Contribution

It introduces a novel prompt-aware scheduling approach that improves inference efficiency and image quality in text-to-image generation systems under high load.

Findings

Enhanced inference efficiency during high loads

Maintained high image quality with prompt-model matching

Reduced model loading overheads

Abstract

Traditional ML models utilize controlled approximations during high loads, employing faster, but less accurate models in a process called accuracy scaling. However, this method is less effective for generative text-to-image models due to their sensitivity to input prompts and performance degradation caused by large model loading overheads. This work introduces a novel text-to-image inference system that optimally matches prompts across multiple instances of the same model operating at various approximation levels to deliver high-quality images under high loads and fixed budgets.