DiffBench Meets DiffAgent: End-to-End LLM-Driven Diffusion Acceleration Code Generation

TL;DR

This paper introduces DiffBench, a comprehensive benchmark, and DiffAgent, an LLM-driven system that automatically generates and refines acceleration codes for diffusion models, significantly improving efficiency.

Contribution

The paper presents a novel LLM-based framework with DiffBench and DiffAgent for automated diffusion model acceleration code generation and evaluation, advancing the state-of-the-art in model optimization.

Findings

DiffAgent outperforms existing LLMs in generating effective acceleration strategies.

DiffBench provides a thorough evaluation pipeline for diffusion model codes.

The framework reduces computational overhead in diffusion model inference.

Abstract

Diffusion models have achieved remarkable success in image and video generation. However, their inherently multiple step inference process imposes substantial computational overhead, hindering real-world deployment. Accelerating diffusion models is therefore essential, yet determining how to combine multiple model acceleration techniques remains a significant challenge. To address this issue, we introduce a framework driven by large language models (LLMs) for automated acceleration code generation and evaluation. First, we present DiffBench, a comprehensive benchmark that implements a three stage automated evaluation pipeline across diverse diffusion architectures, optimization combinations and deployment scenarios. Second, we propose DiffAgent, an agent that generates optimal acceleration strategies and codes for arbitrary diffusion models. DiffAgent employs a closed-loop workflow in which a planning component and a debugging component iteratively refine the output of a code generation component, while a genetic algorithm extracts performance feedback from the execution environment to guide subsequent code refinements. We provide a detailed explanation of the DiffBench construction and the design principles underlying DiffAgent. Extensive experiments show that DiffBench offers a thorough evaluation of generated codes and that DiffAgent significantly outperforms existing LLMs in producing effective diffusion acceleration strategies.