Pixels Under Pressure: Exploring Fine-Tuning Paradigms for Foundation Models in High-Resolution Medical Imaging

TL;DR

This paper systematically evaluates various fine-tuning methods for high-resolution (512x512) medical image generation using diffusion models, demonstrating how specific strategies enhance image quality and downstream classification performance.

Contribution

It provides a comprehensive benchmark of fine-tuning techniques for high-res medical image synthesis, highlighting their impact on quality metrics and downstream tasks.

Findings

Certain fine-tuning methods improve FID and Vendi scores.

Fine-tuning enhances prompt-image alignment.

Synthetic images boost classifier performance in data-scarce scenarios.

Abstract

Advancements in diffusion-based foundation models have improved text-to-image generation, yet most efforts have been limited to low-resolution settings. As high-resolution image synthesis becomes increasingly essential for various applications, particularly in medical imaging domains, fine-tuning emerges as a crucial mechanism for adapting these powerful pre-trained models to task-specific requirements and data distributions. In this work, we present a systematic study, examining the impact of various fine-tuning techniques on image generation quality when scaling to high resolution 512x512 pixels. We benchmark a diverse set of fine-tuning methods, including full fine-tuning strategies and parameter-efficient fine-tuning (PEFT). We dissect how different fine-tuning methods influence key quality metrics, including Fr\'echet Inception Distance (FID), Vendi score, and prompt-image alignment. We also evaluate the utility of generated images in a downstream classification task under data-scarce conditions, demonstrating that specific fine-tuning strategies improve both generation fidelity and downstream performance when synthetic images are used for classifier training and evaluation on real images. Our code is accessible through the project website - https://tehraninasab.github.io/PixelUPressure/.