LiteDiff
Ruchir Namjoshi, Nagasai Thadishetty, Vignesh Kumar, Hemanth Venkateshwara

TL;DR
LiteDiff introduces a lightweight fine-tuning method for diffusion models, enabling efficient domain adaptation in medical imaging with minimal data and computational resources.
Contribution
The paper proposes LiteDiff, a novel approach that freezes the base diffusion model and only trains small residual adapters, reducing computational cost and overfitting.
Findings
LiteDiff outperforms full fine-tuning in efficiency and performance.
Ablation studies identify optimal adapter placement in U-Net.
Effective in low-data chest X-ray domain adaptation.
Abstract
In recent years, diffusion models have demonstrated remarkable success in high-fidelity image synthesis. However, fine-tuning these models for specialized domains, such as medical imaging, remains challenging due to limited domain-specific data and the high computational cost of full model adaptation. In this paper, we introduce Lite-Diff (Lightweight Diffusion Model Adaptation), a novel finetuning approach that integrates lightweight adaptation layers into a frozen diffusion U-Net while enhancing training with a latent morphological autoencoder (for domain-specific latent consistency) and a pixel level discriminator(for adversarial alignment). By freezing weights of the base model and optimizing only small residual adapter modules, LiteDiff significantly reduces the computational overhead and mitigates overfitting, even in minimal-data settings. Additionally, we conduct ablation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
