EC-Diff: Fast and High-Quality Edge-Cloud Collaborative Inference for Diffusion Models

TL;DR

EC-Diff is a novel framework that accelerates diffusion model inference by optimizing cloud-edge collaboration, reducing cloud computation, and maintaining high-quality generation with up to 2x speedup.

Contribution

It introduces a gradient-based noise estimation method and a greedy search algorithm for optimal cloud-edge inference switching in diffusion models.

Findings

Achieves up to 2x inference speedup over cloud-only methods.

Improves generation quality compared to pure edge inference.

Effectively balances inference speed and output quality in diffusion models.

Abstract

Diffusion Models have shown remarkable proficiency in image and video synthesis. As model size and latency increase limit user experience, hybrid edge-cloud collaborative framework was recently proposed to realize fast inference and high-quality generation, where the cloud model initiates high-quality semantic planning and the edge model expedites later-stage refinement. However, excessive cloud denoising prolongs inference time, while insufficient steps cause semantic ambiguity, leading to inconsistency in edge model output. To address these challenges, we propose EC-Diff that accelerates cloud inference through gradient-based noise estimation while identifying the optimal point for cloud-edge handoff to maintain generation quality. Specifically, we design a K-step noise approximation strategy to reduce cloud inference frequency by using noise gradients between steps and applying cloud inference periodically to adjust errors. Then we design a two-stage greedy search algorithm to efficiently find the optimal parameters for noise approximation and edge model switching. Extensive experiments demonstrate that our method significantly enhances generation quality compared to edge inference, while achieving up to an average $2\times$ speedup in inference compared to cloud inference. Video samples and source code are available at https://ec-diff.github.io/.