Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Caching

TL;DR

This paper introduces E$^2$-CRF, a caching method that accelerates frequency domain diffusion models by adaptively reusing features based on spectral properties, achieving over 2x speedup without quality loss.

Contribution

It proposes a novel error-feedback event-driven caching strategy exploiting spectral localization and mirror symmetry to speed up diffusion models.

Findings

Achieves approximately 2.2x speedup in inference.

Maintains sample quality comparable to baseline models.

Effective across 5 different datasets.

Abstract

Diffusion models achieve remarkable success in time series generation. However, slow inference limits their practical deployment. We propose E$^2$-CRF (Error-Feedback Event-Driven Cumulative Residual Feature caching) to accelerate frequency domain diffusion models. Our method exploits two structural properties: (1) spectral localization, where signal energy concentrates in low frequencies, and (2) mirror symmetry, which halves the effective frequency dimension. E$^2$-CRF uses a closed-loop error-feedback system that adaptively caches transformer KV features across diffusion steps. We trigger recomputation using event-driven residual dynamics instead of fixed schedules. Our method selectively recomputes high-energy or rapidly-changing tokens while reusing cached features for stable high-frequency components. E$^2$-CRF achieves ~2.2 speedup while maintaining sample quality. We demonstrate effectiveness on 5 datasets. Our caching strategy naturally aligns with the diffusion process's structure-to-detail progression. We include sufficient-condition error and complexity bounds under standard regularity assumptions (Appendix), alongside empirical validation. Our code is available at https://github.com/NoakLiu/FastFourierDiffusion and is also integrated in https://github.com/NoakLiu/FastCache-xDiT.