Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models

TL;DR

This paper introduces SODA, a spectrum-aware fine-tuning method for diffusion models that adjusts singular values and basis vectors, improving parameter efficiency and representation capacity.

Contribution

It proposes a novel spectrum-aware adaptation framework using Kronecker products and Stiefel optimizers, enhancing fine-tuning of generative models.

Findings

SODA outperforms existing fine-tuning methods on text-to-image diffusion tasks.

The method effectively balances computational efficiency and model capacity.

Extensive evaluations demonstrate SODA's superior adaptation performance.

Abstract

Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods.