Cross-D Conv: Cross-Dimensional Transferable Knowledge Base via Fourier Shifting Operation

TL;DR

This paper introduces Cross-D Conv, a Fourier domain-based operation that enables transfer learning between 2D and 3D medical imaging models, improving 3D model performance using abundant 2D data.

Contribution

It proposes a novel Fourier shifting operation for cross-dimensional transfer learning, bridging 2D and 3D convolutional models in medical imaging.

Findings

Achieves comparable or superior performance in feature assessment.

Enhances 3D model training with 2D data.

Maintains computational efficiency of 2D training.

Abstract

In biomedical imaging analysis, the dichotomy between 2D and 3D data presents a significant challenge. While 3D volumes offer superior real-world applicability, they are less available for each modality and not easy to train in large scale, whereas 2D samples are abundant but less comprehensive. This paper introduces Cross-D Conv operation, a novel approach that bridges the dimensional gap by learning the phase shifting in the Fourier domain. Our method enables seamless weight transfer between 2D and 3D convolution operations, effectively facilitating cross-dimensional learning. The proposed architecture leverages the abundance of 2D training data to enhance 3D model performance, offering a practical solution to the multimodal data scarcity challenge in 3D medical model pretraining. Experimental validation on the RadImagenet (2D) and multimodal volumetric sets demonstrates that our approach achieves comparable or superior performance in feature quality assessment. The enhanced convolution operation presents new opportunities for developing efficient classification and segmentation models in medical imaging. This work represents an advancement in cross-dimensional and multimodal medical image analysis, offering a robust framework for utilizing 2D priors in 3D model pretraining while maintaining computational efficiency of 2D training. The code is available on https://github.com/convergedmachine/Cross-D-Conv.