HADIS: Hybrid Adaptive Diffusion Model Serving for Efficient Text-to-Image Generation

TL;DR

HADIS introduces a hybrid diffusion model serving system that adaptively routes text-to-image queries to optimize latency and quality, significantly outperforming existing systems.

Contribution

The paper proposes a novel hybrid architecture and system, HADIS, for adaptive diffusion model serving that improves latency and response quality through query-aware routing and resource optimization.

Findings

Up to 35% improvement in response quality.

Latency violation rates reduced by 2.7-45 times.

Effective offline profiling for resource management.

Abstract

Text-to-image diffusion models have achieved remarkable visual quality but incur high computational costs, making latency-aware, scalable deployment challenging. To address this, we advocate a hybrid architecture that achieves query awareness when serving diffusion models. Unlike existing query-aware serving systems that cascade lightweight and heavyweight models with a fixed configuration, our hybrid architecture first routes each query directly to a suitable model variant, then reroutes it to a cascaded heavyweight model only if necessary. We theoretically analyze conditions for the hybrid architecture to outperform non-hybrid alternatives in latency and response quality. Building on this architecture, we design HADIS, a hybrid serving system for latency-aware diffusion models that jointly optimizes cascade model selection, query routing, and resource allocation. To reduce the complexity of resource management, HADIS uses an offline profiling phase to produce a Pareto-optimal cascade configuration table. At runtime, HADIS selects the best cascade configuration and GPU allocation given latency and workload constraints. Empirical evaluations on real-world traces demonstrate that HADIS improves response quality by up to 35% while reducing latency violation rates by 2.7-45$\times$ compared to state-of-the-art model serving systems.

Tables

Table 1. Table 1: Features and proposed rules of the rule-based router.

Feature	Description and Rule
Prompt length	Longer prompts often increase hardness
Token rarity	Rare words are harder to visualize, increasing hardness
Num. of objects	A Larger number of objects in the prompt makes it harder to visualize
Abstractness	Abstract concepts (e.g.,“freedom”, “hope”) are harder than concrete ones
Attribute density	More modifiers increase the hardness
Spatial relations	More relation counts (e.g., “in front of”, “next to”) increase compositional demand.
Action verbs	Dynamic scenes (e.g., “running”, “holding”) are harder than static ones
Named entities	Proper nouns may not be well-represented in training data for diffusion model (e.g., local brands), increasing hardness

Equations

$$\text{Quality:}{Q}_{RO}$$

$$\text{Cost:}{C}_{RO}$$

$$s.t.r_0^{{\theta }_R}$$

$$\text{Quality:}{Q}_{HY}$$

$$\text{Cost:}{C}_{HY}$$

$$s.t.r_0^{{\theta }_H}$$

$${\pi }^{{\theta }_H,{\tau }_H}$$

$$d_{TP}^{{\tau }_H}\ge$$

$$d_{FP}^{{\tau }_H}\le$$

$$\text{Quality:}{Q}_{DO}$$

$$\text{Cost:}{C}_{DO}$$

$$s.t.{\pi }^{{\tau }_D}$$

$$r_{TP}^{{\theta }_H}\ge \frac{\alpha (1-q)r_{FP}^{{\theta }_H}+q{d}_{TP}^{{\tau }_D}-\alpha }{q(d_{TP}^{{\tau }_H}-\alpha )},r_{FP}^{{\theta }_H}\le \frac{d_{FP}^{{\tau }_D}}{d_{FP}^{{\tau }_H}}$$

$$\sum _b{y}_{i,b}=1,\sum _k{z}_k=1,\sum _i{x}_i\le S\forall i$$

$$x_i\sum _b{y}_{i,b}{\mu }_{i,b}\ge \lambda \sum _k{z}_k{r}_{k,i}\forall i$$

$$\sum _i({\hat{L}}_i+D_i^{\text{queue}})\le T_{SLO}$$

$$F=\sum _k{z}_k{\mathrm{FID}}_k$$

$$\underset{\{z_k\},\{x_i\},\{y_{i,b}\},\{u_{i,b}\}}{\min }F\text{s.t. Eqs. (3)–(6)}$$

Full text

1 Introduction

Text-to-image diffusion models such as Stable Diffusion XL [32], DALL$\cdot$E3 [29], and the more recent Diffusion Transformer [31] have demonstrated remarkable capabilities in generating semantically rich, high-quality images from natural language prompts. These models now power emerging applications in creative design, e-commerce, and content generation. Commercial platforms such as Adobe FireFly [4] and Photoshop’s generative fill [5] have integrated text-to-image capabilities directly into interactive UIs, where users expect both realism and responsiveness as they iterate on designs.

However, the computational cost of these models remains a significant bottleneck for real-time and scalable deployments in production model serving systems. Generating a single image often requires multiple seconds of GPU time due to the iterative denoising process. While recent few-step diffusion variants such as SDXL-Lightning [26] and SDXL-Turbo [37] have been explicitly designed for real-time or near-real-time generation on commodity hardware, this work addresses the problem from the model serving perspective.

Model Serving Goals. A model serving system has two key goals. First, the system must use resources efficiently to serve queries (i.e., text prompts from the user) with high throughput. Second, since queries are served to users in real time, the system must guarantee service-level objectives (SLOs) that require the latency of response to be within a specified threshold, even as user queries vary widely in complexity. Simple queries like “a banana on a white plate” require minimal computation, whereas complex ones like “a surreal cityscape representing time and memory” demand deeper semantic understanding and finer generative detail. Serving all queries with a “heavyweight” model that has a high computational cost leads to excessive resource usage and response latencies, while using a low-cost “lightweight” model alone compromises the response quality. Therefore, a model serving system must dynamically adjust to query arrival rates and query difficulty to satisfy both throughput and SLO constraints.

Query-Aware Model Serving. Recently, query awareness has shown promises for improving model serving throughput while meeting SLO constraints. In query-aware model serving systems, such as FrugalGPT [11], DiffServe [8], Tabi [43], CascadeBert [25], and CascadeServe [24], each query is processed through a model cascade that is a set of models sequenced from lighter weight ones to heavier weight ones. Each query is routed sequentially through the models in the cascade till a response of acceptable quality is obtained. DiffServe [8] exemplifies such a system tailored to diffusion models. Its cascade uses two stages: every prompt is first processed by a lightweight model, and the generated image is assessed by a discriminator. If the image is deemed low-quality, the prompt is rerouted to a heavyweight model. Easier queries only require the first stage of the cascade, saving resources and latency, while harder queries are routed to the second.

Limitations of Current Solutions. The current query-aware serving systems using model cascades have two key limitations. First, model cascades inherently increase both resource usage and latency as some fraction of the queries need to get routed through multiple models. For instance, in a two-model cascade, all prompts are sent to a lightweight model, which wastes resources for queries that are hard and require heavyweight processing regardless. Second, the models chosen for a cascade may not be optimal for all query workloads. For instance, DiffServe relies on a fixed pair of lightweight and heavyweight models, even though different workloads and latency budgets may favor different combinations.

Our Main Ideas. These limitations motivate our main ideas which we list below.

(i) A hybrid routing architecture. We propose a hybrid architecture (Fig. 1) that combines a router and a discriminator to reduce resource wastage and latency. The architecture begins with a router that computes a query difficulty score using linguistic and semantic features, including token rarity, abstractness, spatial relations, etc. Queries that are predicted to be hard are routed directly to a heavyweight model, skipping the lightweight model(s) in the cascade. Queries that are classified as easy are processed by a lightweight model, whose outputs are then evaluated by an image discriminator for both semantic and aesthetic quality. If the generated images fail to meet the quality requirements, the query is sent to the heavyweight model. This hybrid architecture reduces computational overhead and latency for easy queries and avoids wasteful lightweight model inference on hard queries. We theoretically characterize the conditions for the router and discriminator performance such that the hybrid architecture is guaranteed to outperform the router-only and discriminator-only alternatives in response quality and latency.

(ii) Dynamic choice of models. As diffusion models proliferate, each variant offers different trade-offs between latency and quality. Statically fixing the model in a hybrid architecture across all query workload scenarios is inefficient. Therefore, we design a model serving system that dynamically chooses the models in the hybrid architecture to suit the properties of the current query workload.

(iii) Managing cascade complexity. The plethora of diffusion model variants raises the possibility of complex hybrid architecture consisting of many model variants, discriminators and router components. We show that such complexity is unnecessary and it is sufficient to dynamically choose a simple two-stage model cascade with a single router and discriminator to achieve the benefits of query awareness. This observation greatly reduces the space of architectural choices and enables efficient resource management.

(iv) Reducing the complexity of resource management. Although restricting the design to two-stage cascades simplifies the search space, the system must still decide among many configuration options, including the choice of model pairs, router and discriminator thresholds, model placements, and batch sizes. Moreover, the optimal configuration shifts as the query workload and SLO requirements change. Prior systems such as InferLine [12], INFaaS [34], Proteus [7], and DiffServe [8] do not cover the entire search space because they are not designed to operate over the hybrid architecture. HADIS reduces the combinatorial complexity as follows. We use offline profiling to benchmark all pairwise model combinations under different router and discriminator thresholds and generate a cascade configuration lookup table. The table prunes the search space by identifying Pareto-optimal configurations that offer the best quality-latency trade-offs. At runtime, the system leverages this table to select a configuration that meets the desired SLO constraints. To determine the optimal deployment strategy given a workload distribution and system constraints (e.g., GPU count, latency budget, etc), we formulate the configuration selection and resource allocation as an MILP (mixed integer linear programming) problem, jointly optimizing them to maximize overall serving efficiency and quality.

Our Contributions. Our specific contributions are below.

• •

We propose a hybrid serving architecture that combines a query router and a discriminator, and theoretically derive conditions under which this hybrid architecture achieves performance benefits.

• •

We empirically show that two-model cascades are sufficient for effective model scaling in text-to-image serving, and propose adaptive model cascade selection, dynamically selecting optimal pairwise model cascades based on various workload and latency constraints.

• •

We address the resource management problem for serving the hybrid architecture through profiling Pareto-optimal configurations offline and formulating a MILP that jointly optimizes architectural configurations and resource allocation.

• •

We implement these approaches in a production-ready system, HADIS. Experiments show that it improves response quality by up to 35% and SLO violation by 2.7-45$\times$ compared to the latest research prototype.

2 Background and Rationale for Design

Background for T2I diffusion models. Text-to-image (T2I) diffusion models embody a clear quality-latency trade-off (see Figure 2-left): more compute (e.g., deeper networks or transformers, more denoising steps, etc) typically yields better image quality but longer runtime (and latency) and lesser throughput (i.e., fewer queries processed per second). In our setting on L40S GPUs, the spectrum spans sub-second generation (e.g., SDXL-Lightning at 2 steps $\approx$ 0.5s) to tens of seconds (e.g., SD3.5-Large at 50 steps $\approx$ 27s) for an image of 1024$\times$1024 resolution.

Rationale for two-model cascades. In the context of T2I diffusion models, we show that two-model cascades offer comparable quality-latency trade-offs as model cascades with more model variants. Figure 2-right explores the design space of diffusion-model cascades using four model variants (orange stars in the left panel). For each candidate cascade we enumerate routing policies specified by the fraction of queries routed to the heavyweight model(s), and measure the resulting mean latency (x-axis) and FID (y-axis) on 5k prompts. The right panel plots all feasible cascade configurations in grey by varying model combinations and routing policies. The Pareto frontier111A Pareto frontier $P$ is a set of model cascades such that for every $m\in P$ there is no other model cascade $m^{\prime}\in P$ that has both lower FID and latency than $m$. In that sense, all model cascades in the Pareto frontier represent an optimal tradeoff between the two objectives where one objective can be made better only at the cost of the other. obtained by two-model and three-model cascades are colored in red and blue, respectively. Two empirical facts stand out. First, the red and blue frontiers coincide over the relevant latency range: for any latency budget, a two-model cascade attains essentially the same best FID as a three-model cascade. Second, while the blue frontier can fill in a few intermediate points (i.e., slightly finer granularity), it does not deliver significantly better quality at a given latency.

The implication is that adding a third stage offers little benefit but introduces system overhead: an extra first-stage inference pass, additional discriminator evaluations, increased queuing between stages, and more routing/placement decisions, each contributing to latency and control complexity. Given that two-model cascades already realize the tight Pareto trade-off, we adopt a two-stage design and focus our effort on adaptive model-pair selection and hybrid routing to minimize cascade cost while preserving response quality.

Rationale for dynamic model choice. Given that our system adopts a two-model cascade, it is natural to ask which pair of models can provide the best system performance. We empirically observe that none of the fixed lightweight/heavyweight model pairs remains optimal across varying query demands.

Figure 2-right shows the best two-model cascade configurations vary with the latency budget. It plots the Pareto frontier of two-model cascade configurations built from four candidates in the left panel. The orange dashed lines partition latency into three regimes. Within each regime, the frontier points arise from a different cascade, <SDXL-Lightning, SD3.5-Turbo> at low latency, <SD3.5-Turbo, SD3.5-Medium> at mid latency, and <SD3.5-Medium, SD3.5-Large> at high latency. We observe that different cascades offer different latencies and response qualities. Hence, when the allowed SLO is tighter, a faster cascade is preferred to satisfy deadlines; when the SLO relaxes, cascades of higher quality can be chosen to increase the response quality.

Figure 3 further shows the best two-model cascades vary with the query loads. It compares several two-model cascades under low demand (left panel) and high demand (right panel) with the SLO held fixed. Each cascade serves queries with the resource allocation algorithm in DiffServe [8]. At low load, all pairs keep SLO violations small. <SD3.5-Turbo, SD3.5-Medium> (red star) attains the best violation rate with competitive FID, while <SD3.5-Medium, SD3.5-Large> trades a slight increase in violations for the best FID (grey diamond). Under high load, as capacity constraints surface, pairs involving slower models (i.e., grey diamond) cannot meet throughput demands and incur large violations, whereas <SDXL-Lightning, SD3.5-Turbo> (blue down-pointing triangle) remains within SLO and delivers the best quality–latency balance for that demand.

The results demonstrate that no single light/heavy pair is universally optimal. The optimal pair varies with both the latency budget and the workload, because different pairs rest on different segments of the quality-latency Pareto frontier and saturate at different throughputs. This motivates dynamic model-pair selection at runtime.

Rationale for hybrid architecture. We argue for a hybrid query routing architecture (Fig. 1) by noting that both a router and a discriminator are needed and either one alone is insufficient, as they address complementary failure modes of the model cascade. If we want to solely use a router to evaluate the hardness of queries and route queries based on that, the router should be lightweight, introducing negligible overhead, and also accurate such that most of the queries will not be misclassified, i.e., false “hard” queries can cause unnecessary waste of GPU resources, and false "easy" queries can reduce the response quality. While query hardness has been studied in  [44, 15, 27], estimating query hardness for text-to-image generation with a router is still an open and non-trivial research problem. Building a router that classifies queries with high accuracy is difficult.

A discriminator in the model cascade is usually effective in determining the hardness of a query, which is done in prior work [8, 11]. However, if we use a discriminator alone, all queries will go through the lightweight model regardless of hardness, then be evaluated by the discriminator, which brings in additional overhead, especially when the first model of the cascade requires significant computation. For instance, in Fig. 2, when adopting model pair <SD3.5-Medium, SD3.5-Large>, of which the latency is $\sim$13s and $\sim$27s respectively, the additional overhead for the first model of the cascade can be 33% of the execution latency.

We therefore advocate a hybrid architecture, combining both a router and a discriminator. The rule-based router sends obviously hard queries directly to the heavyweight model, while the rest go to the lightweight model. The discriminator then evaluates the outputs from the lightweight model and catches false “easy” cases for re-routing to heavyweight models. This design avoids unnecessary first-stage work for clearly hard queries, preserves quality by correcting router mistakes, and keeps the added control cost minimal. Our evaluation in §5.4 shows this hybrid routing reduces SLO violations versus discriminator-only and improves FID versus router-only, delivering the best quality-latency trade-off under identical hardware resources.

3 Theoretical Analysis of Hybrid Architecture

This section formalizes the rationale for our hybrid architecture. We theoretically show that there exist parameter settings under which the hybrid architecture (HY) achieves quality better than Router-Only (RO) and Discriminator-Only (DO) at the same expected cost. We then interpret the resulting conditions geometrically as feasibility regions in the ROC planes of the router and the discriminator (see Fig. 4 and Fig. 5).

Preliminary. The analysis considers a hybrid architecture with four components: a cheap, fast model $M_{0}$, a slower, high-quality model $M_{1}$, a router, and a discriminator. The router decides whether each query is sent to $M_{0}$ or directly to $M_{1}$, while the discriminator decides, for queries that went through $M_{0}$, which outputs are good enough to return and which should be escalated to $M_{1}$, as illustrated in Fig. 1. Let $q$ denote the fraction of easy queries in the workload, so hard queries occupy mass $1-q$. Let $s_{0}$ and $s_{1}$ be the latencies of running $M_{0}$ and $M_{1}$ with $s_{0}<s_{1}$. We define the latency ratio $\alpha=\frac{s_{0}}{s_{1}}\in(0,1)$ which captures how much cheaper the lightweight model is relative to the heavyweight model.

For the router with threshold $\theta$, we define $r_{TP}^{\theta}$ as the probability an easy query is routed to $M_{0}$ (i.e., True Positive Rate, TPR), and $r_{FP}^{\theta}$ as the probability a hard query is routed to $M_{0}$ (i.e., False Positive Rate, FPR). As $\theta$ varies, the pair $(r_{FP}^{\theta},r_{TP}^{\theta})$ traces the router’s ROC curve. Similarly, for the discriminator with threshold $\tau$, we define $d_{TP}^{\tau}$ as the probability that an output from $M_{0}$ on an easy query is accepted (TPR), whereas $d_{FP}^{\tau}$ is the probability that an output on a hard query is incorrectly accepted and completed at $M_{0}$ (FPR). As $\tau$ varies, the pair $(d_{FP}^{\tau},d_{TP}^{\tau})$ traces the discriminator’s ROC curve. Together, these two ROC curves describe the operating space in which we will derive conditions for HY to dominate RO and DO under equal expected cost.

To reason about quality analytically, we define quality using a simple probabilistic model. We assume the heavyweight model $M_{1}$ always produces acceptable quality for both easy and hard queries, while the lightweight model $M_{0}$ is adequate only for easy queries. Under this abstraction, the only way to harm quality is to return an $M_{0}$ output on a hard query. The quality ($Q$) is hence defined as one minus the probability of this failure event. We model cost ($C$) as the expected latency for an architecture to process a query.

Comparison between HY and RO. In a RO architecture, a hard query is mishandled if and only if the router classifies it as easy and sends it to $M_{0}$, which occurs with probability $(1-q)\,r_{FP}^{\theta_{R}}$, yielding:

[TABLE]

where $\theta_{R}$ is the router threshold of RO architecture; $r_{0}$ and $r_{1}$ are the fractions of queries routed to $M_{0}$ and $M_{1}$, respectively.

In a HY architecture, a hard query is mishandled only if it is routed to $M_{0}$ and the discriminator incorrectly accepts the $M_{0}$ output. This happens with probability $(1-q)\,r_{FP}^{\theta_{H}}d_{FP}^{\tau_{H}}$:

[TABLE]

where $\pi$ is the fraction of queries that are first routed to $M_{0}$ and then rerouted to $M_{1}$.

To compare HY against RO, we impose the equal-cost constraint $C_{HY}=C_{RO}$ and analyze the quality gap. Note that only if $\theta_{H}>\theta_{R}$ can the cost be equal. Requiring $Q_{HY}\geq Q_{RO}$ (i.e., HY is at least as good as RO at the same cost) yields the following bounds on the discriminator ROC point $(d_{FP}^{\tau_{H}},d_{TP}^{\tau_{H}})$ of the HY architecture:

[TABLE]

The inequalities define a feasibility region in the ROC plane for configuring $\tau_{H}$ : as long as HY employs a discriminator whose $(\text{FPR},\text{TPR})$ lies inside this region, the HY architecture will match RO’s expected cost while achieving better response quality. We visualize such a region in Fig. 4 by fixing $q=0.5$ and traversing the feasible $(r^{\theta}_{TP},r^{\theta}_{FP})$ pairs. As $\alpha$ increases from $0.1$ to $0.2$, the feasible region shrinks. This is because when the lightweight model becomes more expensive relative to $M_{1}$, only discriminator thresholds with sufficiently high $d_{TP}$ and low $d_{FP}$ (i.e., making accurate re-routing decisions) allow HY to still dominate RO at equal cost. In practice, the discriminator ROC lies well above the router ROC, making these conditions practically attainable.

Comparison between HY and DO.. We now compare the hybrid architecture against a DO architecture, where every query first runs through $M_{0}$ and a discriminator with threshold $\tau_{D}$ decides whether to accept the $M_{0}$ output or reroute the query to $M_{1}$.

[TABLE]

Requiring $Q_{HY}\geq Q_{DO}$ and $C_{HY}=C_{DO}$ yields bounds on the router ROC point of our hybrid architecture (i.e., $\langle r_{FP}^{\theta_{H}},r_{TP}^{\theta_{H}}\rangle$) for a given discriminator ROC and latency ratio $\alpha$. With $d_{TP}^{\tau_{H}}>\alpha$, we obtain

[TABLE]

The above inequalities define a feasibility region in the router ROC plane: if HY employs a router whose $(\text{FPR},\text{TPR})$ lies inside this region, HY is guaranteed to match DO’s expected cost while achieving better response quality. Fig. 5 illustrates how the required router accuracy depends on $\alpha$. When the lightweight model is very cheap ($\alpha=0.1$), DO can already exploit $M_{0}$ aggressively at low cost, so the feasibility region is relatively small and the router must be quite accurate to provide additional benefit. As $\alpha$ increases, the region expands, indicating that even a moderately accurate router is sufficient for HY to dominate DO.

4 HADIS Design

We present HADIS, a diffusion model serving system that incorporates the architectural elements presented in §2. We first provide an overview (§4.1), then detail the main features of HADIS, including adaptive model cascading (§4.2), hybrid routing (§4.3), and resource management (§4.4).

4.1 Overview of HADIS

Fig. 6 shows the system architecture of HADIS. It has three major components: Controller, Query Router, and Workers. We explain how these components are involved in the control path and the data path. The control path is marked with dotted arrows in Fig. 6. HADIS is initialized with a set of diffusion models registered in the Controller. The control path takes mainly two actions: (1) The Controller profiles the registered models and pairwise model combinations offline to generate a cascade configuration lookup table. (2) At runtime, the Controller leverages this table to select the optimal cascade configuration. Each configuration includes a model pair and the associated thresholds. The data path is marked with solid arrows in Fig. 6. (1) A query from the client is sent to the query router. (2) The router estimates query hardness and routes the query to a worker hosting a suitable diffusion-model variant. (3) If the query is first processed by the lightweight model and the quality score of the generated response is below the threshold, the worker re-routes the query to another worker with a more heavyweight model. (4) The final output is returned to the client. We next describe each main system component.

Controller. A centralized Controller manages resources and coordinates system modules. It comprises: (1) A Resource Manager that periodically solves MILP to determine the model pair and thresholds (i.e., the router’s hardness threshold and the discriminator’s quality threshold) and to produce a resource allocation plan (i.e., model placement across workers and batch sizes); (2) A Load Balancer that generates routing tables so each query is directed to a worker, and updates these tables when model selection or placement changes. (3) A Metadata store that periodically pulls heartbeats from the Query Router and Workers to update execution statistics, including per-model queue lengths and demand at each worker. (4) A Model Profiler that supplies a lookup table of profiled cascade configurations to the Resource Manager. Only registered diffusion-model pairs and discriminators are eligible for adaptive model cascading. §4.2 describes construction of the lookup table and §4.4 presents the resource-management formulation based on the lookup table.

Workers. Each worker maintains a local queue and processes queued queries using its hosted model. A worker hosts either a lightweight model co-located with a discriminator or a heavyweight model at any given time; this role is dynamic and can be swapped at runtime to handle time-varying demand. The Controller determines the batch size, which model variant to host, and the quality threshold for each worker, while workers report status (e.g., queries received, queue lengths, timeouts) back for planning.

Query Router The router sits on the data path between clients and workers. It implements a rule-based prompt analyzer that assigns obviously hard queries directly to heavyweight models and the rest to lightweight models. The rule-based analyzer is part of the hybrid routing design and will be elaborated in §4.3.

4.2 Adaptive Model Cascading

Adaptive model cascading dynamically selects the appropriate model pairs in a cascade given the current demand and latency SLO. However, determining both the optimal cascade and resource-allocation plan is NP-hard. The search space can be prohibitively large due to numerous model candidates, decision thresholds, model placements, and batch-size choices. HADIS addresses this challenge by decomposing the problem into two stages: offline model profiling (this section) and online MILP-based resource management ($4.4).

Model candidate selection. In practice, the model pool is already constrained by user provision, so we do not face an unbounded number of variants. However, exhaustively profiling every possible cascade built from these models is still unnecessary. We therefore prune the pool with three heuristic, reproducible rules before any offline benchmarking: (1) If multiple models exhibit similar latency and FID, we retain a single representative and discard the others; (2) If one model Pareto-dominates another (no worse in latency and strictly better in quality, or vice versa), we keep the dominating model and drop the dominated ones; (3) If a model lies approximately on the line segment between two neighbors, we omit it, as a two-stage cascade can reproduce its trade-off by thresholding between the neighbors. Figure 2-left illustrates the retained (orange) and pruned (blue) models. In our running example, we start with 8 model candidates, yielding 28 two-model cascades and 2800 configurations (10 hardness thresholds and 10 quality thresholds per cascade). After pruning 4 models, only 4 remains, yielding 6 cascades and 600 configurations, reducing profiling cost by 79%.

Offline profiling and lookup-table construction. The theoretical analysis in §3 characterizes which router–discriminator operating points would be optimal if we knew the true easy/hard split and exact ROC curves. However, in practice there is no ground-truth notion of “easy vs. hard” or per-query correctness label for text-to-image tasks, so we cannot directly measure $q$, TPR, and FPR as in the analysis. Instead, after determining the two-model cascade <lightweight, heavyweight>, for each pair we sweep the hardness threshold and quality threshold over a fixed grid to form a cascade configuration:

• •

*model_pairs: * the pair instantiating the lightweight and heavyweight roles

• •

diff_thres: the router’s hardness threshold ($\theta$)

• •

qual_thres: the discriminator’s quality threshold ($\tau$)

• •

*route_ratio: * the measured fraction of queries processed by each model under these thresholds ($r_{0}$ and $r_{1}$)

• •

*FID: * the resulting response quality

We then prune dominated rows to retain only Pareto-optimal configurations. The union over all pairs yields a compact lookup table of <pair, thresholds> configurations that captures the relevant quality–latency trade-offs and induced routing fractions. This table is the sole input to the online phase, where the Resource Manager jointly select a configuration, model placement, and batch sizes via a fast MILP, enabling HADIS to adapt model pairs and thresholds at runtime without expensive online exploration.

4.3 Hybrid Routing

HADIS employs: (1) a rule-based router that estimates query hardness before any generation, and (2) a CLIP (Contrastive Language-Image Pre-training)-based discriminator that evaluates the generated image. While an explicit ROC to characterize the performance of the router and discriminator is infeasible due to the lack of ground truth labels, we empirically show that the design enables HADIS to outperform non-hybrid alternatives in §5.4.

Rule-based router. The Query Router includes a rule-based prompt analyzer that extracts and normalizes a compact feature vector from the query prompts, computing a hardness score using the weighted sum. The rules that estimate hardness are designed heuristically. Table 1 lists the features and corresponding rules used in the router. The feature weights are tuned via grid search, and we select coefficients that optimize the quality-latency objective, providing evidence that misrouting is reduced.

CLIP-based discriminator. The Discriminator is co-located with lightweight models on GPUs and determines whether their outputs are aesthetically acceptable. Accepted results are returned to the client, while rejected ones are re-routed to heavyweight models. We use a CLIP backbone [33] because it provides robust image embeddings for aesthetic evaluation [20]. The discriminator keeps the pretrained CLIP frozen and fine-tunes a multi-layer perceptron (MLP) head.

Training. We train the discriminator as a binary classifier to distinguish high-quality, real images (e.g., ImageNet [35]) from images generated by our candidate diffusion models. This teaches it to detect typical divergences, such as loss of sharpness, texture incoherence, and structural glitches, enabling accurate quality assessment within the cascade.

Inference. Given a prompt and its generated image, CLIP produces text and visual embeddings; the MLP head consumes the visual embedding and outputs a softmax score in $[0,1]$, used as the quality score (likelihood the image is “real-like” and thus high quality). The discriminator can also compute the CLIP score [21] (cosine similarity between visual and text embeddings) to assess semantic alignment at no extra inference cost. While we do not report semantic metrics in our evaluation because candidates differ little on CLIP score, the signal is useful in production and can be combined with the quality score as a safety check. The overhead of discriminator is analyzed at §5.8.

4.4 Resource Management

The Resource Manager adapts HADIS to the current workload by jointly (i) selecting the cascade configuration, including a lightweight/heavyweight model pair and thresholds, (ii) placing models across workers, and (iii) choosing batch size per model. It tunes thresholds (i.e., the router’s hardness threshold and discriminator’s quality threshold) to balance response quality and latency as demand shifts. For instance, during low demand, a lower hardness threshold or higher quality threshold prioritizes image quality, while at peak times, a higher hardness threshold or lower quality threshold ensures fewer queries are routed to heavyweight models, allowing minor quality compromises to meet latency deadlines. When demand exceeds the capacity of the current pair under its active thresholds, the Resource Manager shifts to a lighter configuration and vice versa. These changes are coordinated with model placement and batch-size updates so that workers are reallocated to match the new routing schedule and each model operates near its latency/throughput sweet spot.

The resource management problem. We formulate the problem with cascade configuration selection as a mixed-integer linear program (MILP) over a pre-profiled lookup table of cascade configurations (described in §4.2). At runtime, the Resource Manager selects one configuration and decides model placement and batch sizes per model so that demand is satisfied and the latency SLO is met while minimizing FID (maximizing response quality).

Optimization variables. For each model pair, we introduce a binary selector $z_{k}$ over the threshold configuration $k$ of that pair. A threshold configuration $k$ includes a hardness threshold and a quality threshold (i.e., $k=<\tau,\theta>$). For each model $i$ in the pair, we choose an integer worker count $x_{i}\geq 0$ and a one-hot batch size via binaries $y_{i,b}$ over a discrete set of batches $b$. The lookup table provides the routing fraction $r_{k,i}$ to model $i$ under configuration $k$, the profiled latency $L_{i,b}$ and throughput $\mu_{i,b}$ at batch size $b$, and the quality of the configuration FIDk. Let $\lambda$ denote the system demand, $S$ the available workers for this pair, and $T_{SLO}$ the latency deadline. Queueing delay for model $i$ is estimated online from the instantaneous queue length $Q_{i}$ and the arrival rate $\lambda_{i}$.

Constraints. The variables must satisfy three groups of constraints. First, considering the resource availability, each model selects a single batch size, each cascade has exactly one threshold configuration for the pair, and worker usage is bounded by the budget:

[TABLE]

Second, considering the throughput constraint, per-model capacity must cover the share of demand routed to that model under the chosen configuration:

[TABLE]

Then, considering the latency constraint, end-to-end latency must remain within the SLO. Let the selected batch latency for model $i$ be $\widehat{L}_{i}=\sum_{b}y_{i,b}L_{i,b}$ and let $D^{\text{queue}}_{i}=\alpha\frac{Q_{i}}{\lambda_{i}}$ denote the queueing delay estimate given Little’s law [40] where $\alpha$ is a safety factor. We impose a conservative path bound, making sure the latency constraint is met:

[TABLE]

Eventually, the objective quality variable equals the FID of the selected configuration:

[TABLE]

MILP formulation. The resource manager identifies the optimal configuration and allocation by

[TABLE]

Solving the MILP. The optimization problem is solved periodically by invoking an MILP solver to identify a global optimal cascade configuration, model allocation, and model batch sizes. We estimate query demand $\lambda$ using an exponentially weighted moving average (EWMA) on the recent demand history. Note that the time overhead to solve the MILP does not lie on the critical path of query serving as the MILP is called asynchronously. We provide a detailed analysis of the overhead in Section 5.8.

5 Evaluation

This section evaluates the efficacy of HADIS. We first outline the common experimental setup (§5.1). We then present end-to-end results and analysis of system performance on real-world workloads (§5.2) and synthetic workloads (§5.3). We quantify the benefit of hybrid routing and the effectiveness of MILP-based resource allocator via controlled ablation studies (§5.4 and §5.5). We study sensitivity of HADIS to various latency SLOs (§5.6) and to workload hardness composition of queries (§5.7). Finally, we report the runtime overheads of each system component (§5.8).

5.1 Experiment settings

Implementation of HADIS. We implement HADIS on a testbed cluster to evaluate performance on real GPU hardware. The system implementation consists of $\sim$5K lines of Python code. We use the HuggingFace [41] and PyTorch frameworks [30] to execute the diffusion models and discriminator, and use spaCy [23] to extract prompt features for the rule-based query router. Our cluster consists of 16 workers with NVIDIA L40S GPU, and system components including Controller, Load Balancer, Worker, and Client, communicate via gRPC for low-latency coordination. We use Gurobi [19] to solve our MILP optimization. Unless otherwise noted, all results reported in this paper are collected on this testbed.

Model Selection. We apply our model-selection rules to choose four diffusion models that provide diverse latency-quality trade-offs, including SDXL-Lightning, SD3.5-Turbo, SD3.5-Medium, and SD3.5-Large. We use the default number of steps for each model, ensuring high-quality generation. SDXL-Lightning [26] is a fast generative model that produces an image in 2 steps, yielding $\sim$0.5s latency per 1024$\times$1024 image on an L40S GPU. Stable Diffusion 3.5 [9] is a diffusion transformer family that can produce images of high quality. We configure SD3.5-Turbo with 4 steps ($\sim$1.3s per image on L40S), while SD3.5-Medium and SD3.5-Large use 50 steps, with latencies of $\sim$13s and $\sim$27s, respectively, on the same hardware. Unless otherwise noted, the latency SLO is 60s, chosen as a multiple of the slowest model’s runtime, and all models render at 1024$\times$1024 resolution.

Datasets and workloads. For quality evaluation, we use ImageNet-1k, a widely-used subset of ImageNet [35], with associated captions. We take the first 5k caption-image pairs, using captions as prompts and the corresponding images as the reference set for FID computation. To drive system load, we use the Microsoft Azure Functions trace [38] as a representative real-world workload, applying shape-preserving scaling to match our cluster’s capacity.

Evaluation Metrics. We assess system performance using two primary metrics. (1) Response quality FID measures the distance between the distribution of generated images and a reference set of real images (lower is better). For each system configuration, we run all queries through the system and compute FID against the reference images from our evaluation set to evaluate the quality of generated images. (2) SLO violation ratio is the fraction of queries whose end-to-end latency exceeds the latency deadline or are proactively dropped because they are predicted to miss it.

Baselines. We compare HADIS against a spectrum of baselines ranging from static (Clipper) to dynamic resource allocation (Proteus) and query-aware cascading (DiffServe).

• •

Clipper [13] is a static baseline, which does not dynamically select models, allocate resources, or route queries given query complexity. We implement two variants, Clipper-Light, which minimizes SLO violations with the lightest model (SDXL-Lightning), and Clipper-Heavy, which maximizes response quality with the heaviest model (SD3.5-Large). Note that Clipper also represents other static serving systems (e.g., TensorFlow-Serving [28]), which rely on application developers to allocate resources for models.

• •

Proteus [7] is a high-throughput serving system that selects among multiple model variants as workload changes. It scales quality by switching to faster variants from the four diffusion model candidates under high load and switching to higher-quality variants when capacity allows. Proteus runs single-stage inference without a per-query cascade or discriminator, where queries are randomly routed to each worker with equal probability.

• •

DiffServe [8] is a text-to-image serving system built around a model-cascading pipeline. DiffServe uses an MILP planner to adapt to demand by jointly setting resource allocations and the quality threshold of the discriminator. Unlike HADIS, it employs a fixed pair of light and heavyweight models. In our implementation, we fix the pair to SD3.5-Turbo and SD3.5-Large, which outperformed other pairings on the query workload.

5.2 Performance on Real-World Traces

Figure 7 reports time series of demand, FID, and SLO violations, plus their averages over the trace. Clipper-Light has the lowest quality (highest FID) and low violations because it processes all queries with the fastest model Clipper-Heavy uses the highest-quality model, achieving the best FID among baselines but incurring significant violations (up to 90% at peak, 77% on average). Proteus dynamically tunes the resource allocation according to demand changes, selecting higher-quality models under low load and shifting toward faster models under high load. However, as it is query-agnostic, in which all queries are routed to the same model with equal probability, easy queries could be over-served by heavy models while hard queries could be under-served by light models, thus its quality improvement over Clipper-Light is limited and suffers from high SLO violations (up to 12% during the peak, 7% on average). DiffServe offers a better response quality and a lower SLO violation ratio due to its dynamic resource allocation and query-awareness. It routes queries to the heavyweight model given the decision made by the discriminator, thus improving the quality over Proteus by up to 12% in the timeseries and 4% on average.

HADIS offers the best performance as it couples query-aware routing, adaptive model selection, and MILP-based resource allocation, choosing the best pair of models with routing thresholds, and allocating GPU resources given the current workload. As demand rises, it shifts traffic toward faster model cascade variants and tightens usage of heavyweight models to protect latency; as demand decreases, it reallocates GPUs to model cascade variants with better response quality and relaxes thresholds to recover quality. Consequently, HADIS improves quality by up to 10%-35% over baselines while maintaining low violations. During peak, it offers better quality than all baselines except Clipper-Heavy, which yields significantly high SLO violations. As shown in the average statistics, HADIS achieves a 7%-23% reduction in FID compared to baselines, and reduces the average SLO violation ratio by 2.7$\times$, 4.2$\times$, and 45$\times$ compared to DiffServe, Proteus, and Clipper-Heavy.

5.3 Performance on Synthetic Traces

This section evaluates HADIS on a controlled synthetic workload and explains how the system adapts its routing, model selection, and resource plan over time (Figure 8).

We replay a piecewise-constant demand trace with four load levels (P1-P4) arranged as rising, peak, and falling phases. Each level is held long enough for the system to settle before the next step change. This shape mimics bursty traffic to evaluate the responsiveness of resource allocation under both load increases and decreases. Across all phases, HADIS adapts to workload changes dynamically, yielding high response quality with low SLO violations.

In Figure 8(a), as demand climbs to P3/P4, FID increases modestly, representing response quality decreases; when demand falls back to P2/P1, response quality (FID) improves. HADIS trades some quality for higher throughput under high load and recovers quality when capacity returns. Throughout the time-varying demands, the SLO violation ratios remain low (1%$\sim$5%). Notably, SLO violations are relatively high during P1/P2 because the models in use are mainly SD3.5-Large and SD3.5-Medium, as shown in Figure 8(b), which have long execution latency in return for higher quality, leading to higher SLO violation rates.

In Figure 8(b)-left, we can observe per-model throughputs (processed queries per second, QPS) and percentages of queries shared by each model. The traffic mix shifts toward faster models (i.e., SDXL-Lightning and SD3.5-Turbo) during P3/P4 and back toward higher-quality models (i.e., SD3.5-Medium and SD3.5-Large>) during P1. This is because the MILP chooses model combinations from the profiled table that meet the active SLO under the current load. When QPS rises, HADIS responses by adjusting the hardness and quality threshold to reduce the fraction of query re-routing, leading to an increased percentage of queries on lightweight paths. The observation is more obvious in Figure 8(b)-right, which shows the GPU workers allocated and the ratio of queries routed to the heavyweight model. Note that in P3 and P4, HADIS is able to achieve high quality with heavier models while meeting latency SLO, because SDXL-Lightning is applied periodically to drain the queued queries accumulated by those models, preventing timeouts.

5.4 Effectiveness of Hybrid Routing

We now show an ablation study of our hybrid routing approach in Figure 9 to understand the effect of the rule-based router and the image discriminator. We compare HADIS against three variants that disable parts of the hybrid routing pipeline: (1) Random: assign random hardness and quality scores to each query and routes based on these scores. (2) Router-only: routing decisions are made by the rule-based router only without discriminator-based re-routing. (3) Discriminator-only: routing decisions are made by the discriminator only without the router. All variants are evaluated on the same real-world trace with the same MILP-based resource allocator.

Figure 9 shows that HADIS achieves the lowest average FID and SLO violation ratio across all variants. Random and Router-only suffer worse FID at similar SLO violation levels since many hard queries are served by lightweight models due to misclassification. Discriminator-only improves FID but incurs higher violations, as it forces every query through the light model, paying extra latency. In contrast, HADIS adopts hybrid routing, skipping the light stage for obviously hard queries and catching false “easy” cases with discriminator. Thus, it preserves low latency and achieves higher response quality.

5.5 Evaluation of Resource Allocation

We ablate the resource planner by implementing two variants, Cache-{D+Q} and Cache-{D}. Unlike HADIS which solves an online MILP to seek optimal resource allocation plans at runtime, these cached planners reuse previously computed plans on a cache hit. Cache-{D} indexes plans by a discretized range of system demand, whereas Cache-{D+Q} uses a two-dimensional key over range of system demand and range of worker queue length. At runtime, we quantize the observed demand (and queue length for Cache-{D+Q}) to their ranges and probe the cache. If the measurements fall within any keyed range that has an associated plan, we declare a cache hit and apply that plan. If not, we solve the MILP and insert the new plan under the corresponding range key.

Figure 10 shows the timeseries results and average statistics. Cache-{D} performs the worst, with early spikes in FID and SLO violations because it lags behind load shifts and is agnostic about real-time system workload. Cache-{D+Q} is more responsive by incorporating queue length into the key to capture congestion. We empirically observe that both caching variants exhibit a startup spike in SLO violations since their initial plans are either biased or misaligned with the actual demands, but violations stabilize once the workload settles. In contrast, HADIS with online MILP adapts plans in real time, rebalancing workers and thresholds as demand changes. It prevents prolonged queue growth, keeps violations low, and maintains comparable or better FID, avoiding under- or over-provision when workload fluctuates.

The results also reveal the potential of caching. Moving from Cache-{D} to Cache-{D+Q} improves both FID and SLO violation ratio, suggesting that better keys help. A hybrid planner using MILP online to correct mismatches and to populate a cache could deliver MILP-level performance with lower optimization overhead, improving scalability for large-scale deployments.

5.6 Sensitivity to Latency SLO

We vary the SLO multiplier from 0.25$\times$ to 1.75$\times$ and run HADIS and all baselines on the real-world trace. Figure 11 reports the average FID and SLO violation ratio.

As the SLO relaxes, all methods improve on both metrics, but HADIS outperforms baselines across the entire range. Under tight budgets (0.25-0.75$\times$), HADIS sustains low violations while achieving lower FID than Proteus and DiffServe (up to 11% and 7%). Clipper-Light maintains $\sim$0 violations but delivers the worst FID, whereas Clipper-Heavy attains the best FID yet exhibits very high SLO violation ratios even with higher SLOs. The SLO violation ratio of DiffServe is nearly zero at the 0.25$\times$ multiplier because the latency SLO (15s) is below the runtime of the heavyweight model ($\sim$27s), making the MILP allocate all workers to lightweight models. Although HADIS could also select a lighter cascade, sacrificing FID for near-zero SLO violations, this would violate its objective of maximizing quality under SLO constraints. Thus, the planner avoids such degenerate allocations. For SLO multipliers above $1\times$, all methods become demand-bound rather than deadline-bound, where performance is limited by the offered QPS. Results show that HADIS guarantees low SLO violations and high quality over a broad range of SLOs.

5.7 Sensitivity to Hardness of Queries

We study how sensitive HADIS is to shifts in the hardness distribution of incoming queries. We isolate the effect of hardness from demand by fixing the query demand and constructing a four-phase trace: the first and third phases contain only Easy queries, and the second and fourth contain only Hard queries. Queries are categorized using weighted scores over textual features. As shown in Figure 12, during Easy phases, the system settles into a relatively dynamic phase where the lightweight model serves the majority of traffic. At each transition to a Hard phase, the router bypasses lightweight inference for a larger fraction of queries, sharply increasing the heavy-class intake. The heavy queue rises, triggering workers assigned to the heavyweight model. These results confirm that HADIS adaptively concentrates resources on heavyweight models precisely when hard queries predominate, highlighting the value of our query-side bypass design.

5.8 Overhead Analysis

HADIS introduces three sources of overhead: (1) a query router that performs rule-based feature extraction and a score computation (5ms per query), (2) a CLIP-based discriminator that computes quality score for generated images (7ms per query on an L40S GPU), and (3) an MILP-based resource allocator whose measured average runtime is $\sim$30ms per planning cycle. The aggregated overhead of router and discriminator is $\sim$12ms, which is negligible relative to diffusion inference of SDXL-Lightning (2.4%), SD3.5-Turbo (0.9%), SD3.5-Medium (0.01%), and SD3.5-Large (<0.01%). Moreover, since the MILP solver runs periodically and applies updates asynchronously, this overhead is off the critical path and does not affect individual queries.

6 Related Work

Model serving systems aim to provide a unified abstraction to the user, hiding details of the underlying ML frameworks, data preprocessing, and performance optimization. Systems such as SageMaker [36], Triton [1], TensorFlow Serving [3], TorchServe [2], and research prototypes like Nexus [39], BATCH [10], and Clockwork [16] typically require users to manually specify which models to deploy while the system handles resource management accordingly. More recent systems tend to jointly optimize accuracy, latency, and cost together for more efficient resource allocation. Clipper [14], Rafiki [42], and Cocktail [17] ensemble models to improve quality, while INFaaS [34], Model Switching [45], and Sommelier [18] adaptively select variants under changing load. Proteus [7] further exploits accuracy–efficiency trade-offs across variants. However, these systems react primarily to system state rather than per-query difficulty, missing optimization opportunities when requests vary in semantic complexity. Cascaded architectures have therefore been proposed (e.g., CascadeBERT [25], Tabi [43]), where easy queries are handled by lightweight models and harder ones are deferred to heavier models. DiffServe [8] extends this idea to diffusion models, constructing model cascades and formulating resource allocation as an MILP to jointly optimize efficiency and response quality. NIRVANA [6] reduces cost by caching intermediate denoising states, which is complementary to our approach. All these cascading frameworks, however, fix the cascade pair and always begin with a lightweight pass, which wastes computation when many prompts are clearly difficult. In contrast, HADIS adaptively selects both the model pair and decision thresholds based on time-varying query demands, and uses a prompt-side router to send obviously hard prompts directly to heavyweight models, thereby improving generation quality and reducing cost by avoiding unnecessary lightweight inference.

7 Conclusion

In this work, we presented HADIS, a query-aware text-to-image diffusion model serving system that addresses the fundamental tension between response quality, latency, and resource efficiency in diffusion model deployments. It advocates a hybrid routing architecture that integrates a predictive router with a generation-aware discriminator, reducing unnecessary computation on hard queries while preserving high-quality outputs for easy ones. HADIS reduces resource management complexity through offline profiling and a Pareto-optimal lookup table, and jointly optimize architectural and resource decisions with an MILP formulation. Evaluated on real-world workload traces, HADIS outperforms state-of-the-art baselines, improving response quality and reducing SLO violations.