Reinventing 2D Convolutions for 3D Images

TL;DR

This paper introduces ACS convolutions, a novel approach that enables 2D CNNs to learn 3D representations by splitting kernels into three views, leveraging pretrained weights, and outperforming traditional 2D and 3D methods on medical imaging tasks.

Contribution

The study presents ACS convolutions, allowing 2D CNNs to be converted into 3D models with pretrained weights, improving performance and efficiency in medical image analysis.

Findings

ACS CNNs outperform 2D and 3D CNNs on benchmarks.

Pretrained ACS CNNs show consistent superiority across tasks.

ACS convolution is a plug-and-play replacement for 3D convolutions.

Abstract

There have been considerable debates over 2D and 3D representation learning on 3D medical images. 2D approaches could benefit from large-scale 2D pretraining, whereas they are generally weak in capturing large 3D contexts. 3D approaches are natively strong in 3D contexts, however few publicly available 3D medical dataset is large and diverse enough for universal 3D pretraining. Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D / 3D parts still exist. In this study, we bridge the gap between 2D and 3D convolutions by reinventing the 2D convolutions. We propose ACS (axial-coronal-sagittal) convolutions to perform natively 3D representation learning, while utilizing the pretrained weights on 2D datasets. In ACS convolutions, 2D convolution kernels are split by channel into three parts, and convoluted separately on the three views (axial, coronal and sagittal) of 3D representations. Theoretically, ANY 2D CNN (ResNet, DenseNet, or DeepLab) is able to be converted into a 3D ACS CNN, with pretrained weight of a same parameter size. Extensive experiments on several medical benchmarks (including classification, segmentation and detection tasks) validate the consistent superiority of the pretrained ACS CNNs, over the 2D / 3D CNN counterparts with / without pretraining. Even without pretraining, the ACS convolution can be used as a plug-and-play replacement of standard 3D convolution, with smaller model size and less computation.