Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model
Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam
TL;DR
ECAD introduces a genetic algorithm-based caching method that significantly accelerates diffusion model inference, improves speed-quality trade-offs, and generalizes across models and resolutions without modifying network parameters.
Contribution
The paper presents ECAD, a novel evolutionary caching approach that learns efficient caching schedules for diffusion models, outperforming prior heuristics and enabling broad applicability.
Findings
ECAD achieves up to 2.58x inference speedup.
ECAD outperforms previous methods by 4.47 COCO FID.
Learned schedules generalize across resolutions and model variants.
Abstract
Diffusion-based image generation models excel at producing high-quality synthetic content, but suffer from slow and computationally expensive inference. Prior work has attempted to mitigate this by caching and reusing features within diffusion transformers across inference steps. These methods, however, often rely on rigid heuristics that result in limited acceleration or poor generalization across architectures. We propose Evolutionary Caching to Accelerate Diffusion models (ECAD), a genetic algorithm that learns efficient, per-model, caching schedules forming a Pareto frontier, using only a small set of calibration prompts. ECAD requires no modifications to network parameters or reference images. It offers significant inference speedups, enables fine-grained control over the quality-latency trade-off, and adapts seamlessly to different diffusion models. Notably, ECAD's learned…
Peer Reviews
Decision·ICLR 2026 Poster
1. **Novel framing and methodological clarity.** The paper introduces a principled optimization perspective (Pareto frontier with NSGA-II) that replaces heuristic-based caching rules. This reframing is conceptually elegant and well-motivated. 2. **Training-free and model-agnostic.** ECAD operates without any weight updates or retraining, making it practical for various diffusion models—especially for users without access to large compute resources. 3. **Comprehensive experi
1. **Compute inefficiency during optimization.** Although no network training is required, ECAD’s optimization process (hundreds of generations, thousands of images per prompt) might still be computationally expensive. The paper could better quantify this cost relative to real-world savings. 2. **Reliance on automatic quality metrics.** The optimization depends on Image Reward or similar automatic metrics, whose correlation to human preference can be imperfect or domain-specific
- framing diffusion caching as a multi-objective Pareto optimization problem to balance quality and efficiency is novel. - the proposed method is lightweight, and can adapt across prompts, model variants, and resolutions. - strong empirical performance that achieves state-of-the-art inference acceleration with competitive image quality across several models. - the paper is well-written and the comprehensive ablation studies in the appendix provide supports for the design choices.
- the performance is dependent on the choice of the quality metric used during optimization, and the caching schedules may not generalize perfectly to all possible evaluation metrics. - the experiment setup (e.g., 100 prompts) may not be representative of complex prompts from real-world applications.
+ The proposed evolutIionary caching framework is novel and effective.
- The authors emphasize in the introduction that "there are no restrictions on batch size." However, it is unclear whether they measure generation performance per image and acceleration ratio under different batch sizes. Doing so would provide strong evidence to support this claim. - For the strongest model, FLUX, it would be more compelling to evaluate its performance on more comprehensive benchmarks, such as GenEval and DPG-Bench, rather than relying solely on toy datasets.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Computer Graphics and Visualization Techniques · Image Enhancement Techniques
MethodsContrastive Language-Image Pre-training · Sparse Evolutionary Training · Diffusion