LiteDiff

TL;DR

LiteDiff introduces a lightweight fine-tuning method for diffusion models, enabling efficient domain adaptation in medical imaging with minimal data and computational resources.

Contribution

The paper proposes LiteDiff, a novel approach that freezes the base diffusion model and only trains small residual adapters, reducing computational cost and overfitting.

Findings

LiteDiff outperforms full fine-tuning in efficiency and performance.

Ablation studies identify optimal adapter placement in U-Net.

Effective in low-data chest X-ray domain adaptation.

Abstract

In recent years, diffusion models have demonstrated remarkable success in high-fidelity image synthesis. However, fine-tuning these models for specialized domains, such as medical imaging, remains challenging due to limited domain-specific data and the high computational cost of full model adaptation. In this paper, we introduce Lite-Diff (Lightweight Diffusion Model Adaptation), a novel finetuning approach that integrates lightweight adaptation layers into a frozen diffusion U-Net while enhancing training with a latent morphological autoencoder (for domain-specific latent consistency) and a pixel level discriminator(for adversarial alignment). By freezing weights of the base model and optimizing only small residual adapter modules, LiteDiff significantly reduces the computational overhead and mitigates overfitting, even in minimal-data settings. Additionally, we conduct ablation studies to analyze the effects of selectively integrating adaptation layers in different U-Net blocks, revealing an optimal balance between efficiency and performance. Experiments on three chest X-ray datasets - (1) Kaggle Chest X-Ray Pneumonia, (2) NIH Chest X-ray14 and (3) VinBigData Chest X_ray demonstrate that LiteDiff achieves superior adaptation efficiency compared to naive full fine-tuning. Our framework provides a promising direction for transfer learning in diffusion models, facilitating their deployment in diverse low data domains.