Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets
Ahmet Bilican, M. Ak{\i}n Y{\i}lmaz, A. Murat Tekalp, R. G\"okberk Cinbi\c{s}
TL;DR
Wavelet Fine-Tuning (WaveFT) introduces a highly sparse, wavelet domain-based PEFT method that enables extreme parameter efficiency and outperforms existing methods like LoRA in personalized text-to-image generation tasks.
Contribution
WaveFT is a novel PEFT approach that learns sparse updates in the wavelet domain, allowing fine-grained control and superior performance at very low parameter counts.
Findings
WaveFT outperforms LoRA in low-parameter regimes.
WaveFT achieves higher subject fidelity and image diversity.
WaveFT enables precise control of trainable parameters.
Abstract
Efficiently adapting large foundation models is critical, especially with tight compute and memory budgets. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA offer limited granularity and effectiveness in few-parameter regimes. We propose Wavelet Fine-Tuning (WaveFT), a novel PEFT method that learns highly sparse updates in the wavelet domain of residual matrices. WaveFT allows precise control of trainable parameters, offering fine-grained capacity adjustment and excelling with remarkably low parameter count, potentially far fewer than LoRA's minimum, ideal for extreme parameter-efficient scenarios. Evaluated on personalized text-to-image generation using Stable Diffusion XL as baseline, WaveFT significantly outperforms LoRA and other PEFT methods, especially at low parameter counts; achieving superior subject fidelity, prompt alignment, and image diversity.
Peer Reviews
Decision·Submitted to ICLR 2026
1. Sparse coefficients in a transform domain with one-shot merge keep inference cost unchanged and expose a precise capacity knob. 2. Theoretical lemmas justify why sparse (transform-domain) updates can be high-rank, in contrast to LoRA’s low-rank subspace, the empirical rank plots support this at SDXL scales. 3. On SDXL personalization (30 subjects), WaveFT consistently improves subject fidelity (DINO/CLIP-I) at very small budgets, while remaining competitive on text alignment/diversity.
1. Despite framing WaveFT as broadly applicable, substantive experiments are confined to image generation (SDXL personalization). There has been no results on controllable images generations which are common tasks in this field. 2. Transformed-parameterization positioning, incomplete baselines. The paper compares to FourierFT and SHiRA, but not to orthogonal/rotation-constrained adapters (OFT-style) that are explicitly discussed in Related Work; these belong to the same “change the weight geome
1. The paper introduces a genuinely novel approach to PEFT by leveraging sparse updates in the wavelet domain. The motivation is clear and compelling: it directly addresses the granularity limitation of LoRA's rank-based parameterization and proposes an elegant solution that allows for precise, continuous control over the parameter budget, even below LoRA's minimum. The use of a transformed domain is an insightful direction for PEFT research. 2. A significant strength is the inclusion of a theo
1. The high-rank updates enabled by sparse parameterization (also verified in SaRA) are a double-edged sword—while expanding representational capacity, they excessively enhance the model’s fitting ability, making it more prone to overfitting, which is unacceptable for parameter-efficient fine-tuning tasks that prioritize generalization. 2. As evidenced by the CLIP-T scores in Figure 5, both SHiRA and WaveFT exhibit a sharp decline in prompt alignment performance as the number of trainable param
- the author proposes an extremely parameter efficienct parameterization utlizing the wavelet transform, opening many interesting questions to explore - the author included an extensive limitation section to discuss the limitation of the waveft method - the author includes theoretical analysis of sparse finetuning methods
- the waveft seems to be a generic peft method and not constrained for finetuning diffusion models, and the author also did not claim that it is only valid for diffusion models, therefore it should include at least one finetuning tasks from other domains, for example finetuning large language models - waveft, because of its efficient parameterization using wavelet, seems to be extremly parameter efficient, however, parameter-efficiency does not directly translate to compute- and memory-efficienc
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Neural Network Applications · Image Enhancement Techniques
MethodsDiffusion