HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training

TL;DR

HySparK introduces a hybrid CNN-Transformer pre-training method using sparse masking for large-scale 3D medical images, improving representation learning and transferability in downstream tasks.

Contribution

This work presents a novel hybrid sparse masking pre-training strategy combining CNNs and Transformers for 3D medical images, which was not previously explored.

Findings

Robust transferability to supervised downstream tasks.

Effective dense multi-scale feature reconstruction.

Promising results on large-scale 3D medical datasets.

Abstract

The generative self-supervised learning strategy exhibits remarkable learning representational capabilities. However, there is limited attention to end-to-end pre-training methods based on a hybrid architecture of CNN and Transformer, which can learn strong local and global representations simultaneously. To address this issue, we propose a generative pre-training strategy called Hybrid Sparse masKing (HySparK) based on masked image modeling and apply it to large-scale pre-training on medical images. First, we perform a bottom-up 3D hybrid masking strategy on the encoder to keep consistency masking. Then we utilize sparse convolution for the top CNNs and encode unmasked patches for the bottom vision Transformers. Second, we employ a simple hierarchical decoder with skip-connections to achieve dense multi-scale feature reconstruction. Third, we implement our pre-training method on a collection of multiple large-scale 3D medical imaging datasets. Extensive experiments indicate that our proposed pre-training strategy demonstrates robust transfer-ability in supervised downstream tasks and sheds light on HySparK's promising prospects. The code is available at https://github.com/FengheTan9/HySparK