Efficient Score Pre-computation for Diffusion Models via Cross-Matrix Krylov Projection

TL;DR

This paper introduces a cross-matrix Krylov projection method to accelerate diffusion models by efficiently solving large linear systems, significantly reducing computation time and enabling high-quality image generation under limited resources.

Contribution

The novel cross-matrix Krylov projection technique exploits matrix similarities to speed up diffusion model computations, achieving substantial time savings and improved efficiency.

Findings

Achieves 15.8% to 43.7% time reduction over standard solvers.

Up to 115× speedup in denoising tasks compared to DDPM baselines.

Enables high-quality image generation within limited computational budgets.

Abstract

This paper presents a novel framework to accelerate score-based diffusion models. It first converts the standard stable diffusion model into the Fokker-Planck formulation which results in solving large linear systems for each image. For training involving many images, it can lead to a high computational cost. The core innovation is a cross-matrix Krylov projection method that exploits mathematical similarities between matrices, using a shared subspace built from ``seed" matrices to rapidly solve for subsequent ``target" matrices. Our experiments show that this technique achieves a 15.8\% to 43.7\% time reduction over standard sparse solvers. Additionally, we compare our method against DDPM baselines in denoising tasks, showing a speedup of up to 115$\times$. Furthermore, under a fixed computational budget, our model is able to produce high-quality images while DDPM fails to generate recognizable content, illustrating our approach is a practical method for efficient generation in resource-limited settings.