FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation

TL;DR

FreMIM introduces a frequency domain approach to masked image modeling, enhancing self-supervised pre-training for medical image segmentation by capturing both global and local features effectively.

Contribution

It proposes a novel frequency-based MIM framework that leverages high- and low-frequency components for improved medical image segmentation.

Findings

Outperforms previous state-of-the-art MIM methods on three benchmarks.

Consistently improves model performance over training from scratch.

Effectively captures structural and semantic information through multi-stage supervision.

Abstract

The research community has witnessed the powerful potential of self-supervised Masked Image Modeling (MIM), which enables the models capable of learning visual representation from unlabeled data. In this paper, to incorporate both the crucial global structural information and local details for dense prediction tasks, we alter the perspective to the frequency domain and present a new MIM-based framework named FreMIM for self-supervised pre-training to better accomplish medical image segmentation tasks. Based on the observations that the detailed structural information mainly lies in the high-frequency components and the high-level semantics are abundant in the low-frequency counterparts, we further incorporate multi-stage supervision to guide the representation learning during the pre-training phase. Extensive experiments on three benchmark datasets show the superior advantage of our FreMIM over previous state-of-the-art MIM methods. Compared with various baselines trained from scratch, our FreMIM could consistently bring considerable improvements to model performance. The code will be publicly available at https://github.com/Rubics-Xuan/FreMIM.