FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning

TL;DR

FeRA introduces a frequency energy-based fine-tuning framework for diffusion models, improving adaptation stability and effectiveness by aligning parameter updates with the models' intrinsic frequency energy progression.

Contribution

The paper proposes FeRA, a novel frequency-driven fine-tuning method that leverages a frequency energy framework for stable and effective diffusion model adaptation.

Findings

FeRA achieves stable diffusion model adaptation across different backbones.

It improves adaptation effectiveness by aligning with frequency energy progression.

FeRA seamlessly integrates with adapter-based tuning schemes.

Abstract

Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapt large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency energy mechanism governing this process. Building upon this observation, we propose FeRA, a frequency driven fine tuning framework that aligns parameter updates with the intrinsic frequency energy progression of diffusion. FeRA establishes a comprehensive frequency energy framework for effective diffusion adaptation fine tuning, comprising three synergistic components: (i) a compact frequency energy indicator that characterizes the latent bandwise energy distribution, (ii) a soft frequency router that adaptively fuses multiple frequency specific adapter experts, and (iii) a frequency energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency energy mechanism, FeRA provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation.