Exploring connections of spectral analysis and transfer learning in medical imaging

TL;DR

This paper investigates how spectral analysis reveals transfer learning behaviors and frequency shortcuts in medical imaging models, highlighting differences based on pre-training data and proposing data editing to mitigate overfitting.

Contribution

It introduces spectral analysis as a tool to understand transfer learning and shortcut learning in medical imaging models, and demonstrates how data editing can improve model robustness.

Findings

Models pre-trained on natural images differ from those on medical images in learning priorities.

Frequency shortcuts can cause models to overfit to artifacts.

Source data editing can reduce shortcut learning and improve model robustness.

Abstract

In this paper, we use spectral analysis to investigate transfer learning and study model sensitivity to frequency shortcuts in medical imaging. By analyzing the power spectrum density of both pre-trained and fine-tuned model gradients, as well as artificially generated frequency shortcuts, we observe notable differences in learning priorities between models pre-trained on natural vs medical images, which generally persist during fine-tuning. We find that when a model's learning priority aligns with the power spectrum density of an artifact, it results in overfitting to that artifact. Based on these observations, we show that source data editing can alter the model's resistance to shortcut learning.