Large-Scale 3D Medical Image Pre-training with Geometric Context Priors

TL;DR

This paper introduces VoCo, a contrastive learning framework leveraging geometric context priors in 3D medical images, enabling effective large-scale pre-training without annotations and improving performance across multiple medical tasks.

Contribution

The paper proposes a novel self-supervised learning method, VoCo, that exploits geometric context priors in 3D medical images for large-scale pre-training, along with a new dataset and comprehensive benchmark.

Findings

VoCo outperforms existing methods on various medical tasks.

The study provides scaling laws and guidelines for model size adaptation.

The PreCT-160K dataset enables large-scale pre-training in medical imaging.

Abstract

The scarcity of annotations poses a significant challenge in medical image analysis. Large-scale pre-training has emerged as a promising label-efficient solution, owing to the utilization of large-scale data, large models, and advanced pre-training techniques. However, its development in medical images remains underexplored. The primary challenge lies in harnessing large-scale unlabeled data and learning high-level semantics without annotations. We observe that 3D medical images exhibit consistent geometric context, i.e., consistent geometric relations between different organs, which leads to a promising way for learning consistent representations. Motivated by this, we introduce a simple-yet-effective Volume Contrast (VoCo) framework to leverage geometric context priors for self-supervision. Given an input volume, we extract base crops from different regions to construct positive and negative pairs for contrastive learning. Then we predict the contextual position of a random crop by contrasting its similarity to the base crops. In this way, VoCo encodes the inherent geometric context into model representations, facilitating high-level semantic learning without annotations. Specifically, we (1) introduce the largest medical pre-training dataset PreCT-160K; (2) investigate scaling laws and propose guidelines for tailoring different model sizes to various medical tasks; (3) build a benchmark encompassing 48 medical tasks. Extensive experiments highlight the superiority of VoCo. Codes at https://github.com/Luffy03/Large-Scale-Medical.