Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning

TL;DR

This paper introduces a structure-aware self-supervised learning framework for 3D medical images that leverages semantic consistency and discrepancy at the structural level to improve representation quality across diverse datasets.

Contribution

It proposes the $S^2DC$ framework that enforces semantic discrepancy and consistency based on anatomical structures, addressing limitations of fixed patch partitioning in prior methods.

Findings

Outperforms state-of-the-art methods across 10 datasets

Effective in 4 different medical imaging tasks

Works across 3 imaging modalities

Abstract

3D medical image self-supervised learning (mSSL) holds great promise for medical analysis. Effectively supporting broader applications requires considering anatomical structure variations in location, scale, and morphology, which are crucial for capturing meaningful distinctions. However, previous mSSL methods partition images with fixed-size patches, often ignoring the structure variations. In this work, we introduce a novel perspective on 3D medical images with the goal of learning structure-aware representations. We assume that patches within the same structure share the same semantics (semantic consistency) while those from different structures exhibit distinct semantics (semantic discrepancy). Based on this assumption, we propose an mSSL framework named $S^2DC$, achieving Structure-aware Semantic Discrepancy and Consistency in two steps. First, $S^2DC$ enforces distinct representations for different patches to increase semantic discrepancy by leveraging an optimal transport strategy. Second, $S^2DC$ advances semantic consistency at the structural level based on neighborhood similarity distribution. By bridging patch-level and structure-level representations, $S^2DC$ achieves structure-aware representations. Thoroughly evaluated across 10 datasets, 4 tasks, and 3 modalities, our proposed method consistently outperforms the state-of-the-art methods in mSSL.